(#4) test: migrate to matrix + remove legacy E2E

[SieDeta]

Summary

This PR completes the high-risk migration from legacy E2E tests to the matrix-based execution framework introduced in previous phases. It removes duplicated legacy infrastructure while preserving existing test entry points and behavior.

The primary goal of this migration is to consolidate previously independent E2E implementations behind a single execution engine (tests/e2e_matrix.py), reducing maintenance burden while ensuring that all E2E coverage continues to exercise the same underlying runner and comparator paths.

What's Included

Wrapper Support (tests/e2e_matrix.py)

matrix_argv_from_env()

Translates legacy E2E_* environment variables into a single-cell matrix invocation, ensuring compatibility with existing workflows.

Supported legacy inputs include:

• Model selection
• --enforce-eager
• Dtype configuration
• Ring transport options
• Hook selection
• Tolerance settings
• Database configuration
• Other existing E2E parameters

This preserves the behavior of the previous E2E tests while routing execution through the matrix framework.

---

run_single()

Provides a thin compatibility layer that:

• Parses matrix arguments
• Verifies that exactly one matrix cell is requested
• Executes that cell
• Returns the resulting CellResult

Any invocation expanding to multiple cells is rejected.

This guarantees that existing pytest tests continue to represent a single logical assertion, matching the semantics of the legacy implementations.

Migration to Matrix-Based Execution

Prior to this PR, each E2E test suite implemented its own orchestration logic, resulting in duplicated code for:

• Runner invocation
• Comparator selection
• Environment translation
• Result parsing
• Reporting

The new architecture centralizes these responsibilities in tests/e2e_matrix.py and shared libraries under tests/lib/.

The migrated tests now act as thin wrappers that specify the desired matrix configuration while delegating execution to the common matrix engine.

Conceptually, the migration changes the structure from:

HF tests
├── runner orchestration
├── comparison logic
└── reporting

vLLM tests
├── runner orchestration
├── comparison logic
└── reporting

to:

                    tests/e2e_matrix.py
                             │
           ┌─────────────────┼─────────────────┐
           │                 │                 │
    HF wrappers       vLLM wrappers     Future wrappers

This enables future expansion across:

• Backends (HF, vLLM)
• Models
• Execution modes
• Hook selections
• Tensor parallel configurations
• Comparison standards

without introducing additional duplicated E2E infrastructure.

Legacy E2E Removal

test_e2e_correctness_vs_hf.py

Reduced from approximately 1667 lines → 85 lines.

Active Tests Migrated

The remaining active tests now act as thin wrappers around matrix execution:

• HF / eager / allclose
• HF / CUDA graphs / allclose (tol = 0.5)

---

Removed Legacy Code

Deleted the following obsolete components:

• Both uncollected _legacy implementations retained "for reference"

• Permanently disabled:

  @pytest.mark.skipif(True)
  def test_e2e_correctness_hf_cuda_graphs(...)

• Unused in-process helper functions

Note: The disabled CUDA-graphs test relied on a compiled-rollout reference that had already been superseded. No active assertions or passing coverage were lost.

vLLM Test Migration

test_vllm_identical.py

Reduced from 262 → ~55 lines.

Migrated to a matrix wrapper executing:

vllm / bitwise

comparison cells.

---

test_vllm_rowcnt.py

Reduced from 117 → ~50 lines.

Migrated to a matrix wrapper executing:

vllm / row_count

comparison cells.

Safety Checks

Public Test Names Preserved

All four externally visible test names remain unchanged.

This preserves compatibility with:

• Existing shell wrappers
• CI configurations
• Node ID–based test invocations

---

Precise Requirement-Based Skips

Tests now use explicit requirement guards from tests/_requirements.py:

• require_cuda()
• require_vllm()
• require_clickhouse()

This improves failure reporting and prevents ambiguous environment-related failures.

---

Static Matrix-Parity Verification

Verified that each wrapper dispatches the exact same runner + comparator chain used by the corresponding legacy implementation.

Parity was confirmed using matrix dry-run dispatch output.

This migration intentionally changes the orchestration layer only; the underlying inference runners and comparison logic remain unchanged.

Verification

• ✅ CPU test suite: 116 passed
• ✅ Added 6 wrapper translation unit tests
• ✅ All wrapper translation tests pass
• ✅ Static dispatch parity verified for all migrated wrappers
• ✅ No changes to public test node IDs
• ✅ Zero newly introduced failures

⚠️ Merge Gate

Per the migration plan (PR #58 – PR #59 convention), this PR requires a numeric GPU/ClickHouse parity run before merge.

The purpose of this validation is to demonstrate that the matrix wrappers produce numerically equivalent behavior to the legacy implementations under real GPU execution.

The parity run should compare:

• Legacy E2E implementation output
• Matrix-wrapper output

using the same:

• Model
• Backend
• Execution mode
• Hook selection
• Tensor-parallel configuration

The comparison should verify:

• Identical runner dispatch behavior
• Equivalent ClickHouse transport output
• Matching comparator verdicts
• Consistent numeric metrics (max_abs, mean_abs, first-difference position)
• No unexpected token divergence

This validation provides evidence that the migration changed only the execution framework and not the functional correctness guarantees of the E2E suite.

Note: Static dispatch parity has been verified in this PR. Dynamic GPU/ClickHouse parity execution could not be performed in the current environment and must be completed prior to merge.

[Samfisheryu]

Closing this as superseded by #58. The migration-wrapper commits from this PR have been cherry-picked into the combined #58 branch for review together with the shared matrix work.