(#6) ci: wire CPU/GPU/multi-GPU/nightly + docs

[SieDeta]

Summary

This PR implements the CI infrastructure and testing workflow described in the project plan, introducing a standardized test matrix, reusable setup action, and testing documentation.

Merge the PR #56 first.

Key Changes

• Registered the nightly pytest marker

  • Added the nightly marker in pyproject.toml (matching the assumption made in the plan's open question).

  • Enables the full sweep command:

    pytest -m "slow or nightly"

    without triggering unknown-marker warnings or errors.

• Introduced a shared CI setup action

  • Added .github/actions/setup-dmi/action.yml, a composite action used across all CI jobs.
  • The action:
    • Installs Python dependencies
    • Installs the required Transformers fork
    • Builds the ClickHouse C++ client
    • Builds the DMI native backend (.so)
    • Optionally installs vLLM
    • Performs a smoke check to ensure monitoring imports correctly

• Added the CI test workflow

  • Introduced .github/workflows/tests.yml with four jobs corresponding directly to the canonical test commands.

Deliverables

pyproject.toml

• Registers the nightly pytest marker.
• Ensures pytest -m "slow or nightly" executes cleanly without unknown-marker errors.

.github/actions/setup-dmi/action.yml

Reusable composite action responsible for environment setup across all CI jobs.

Responsibilities:

• Install Python dependencies
• Install the Transformers fork
• Build the ClickHouse C++ client
• Build the DMI native backend
• Optionally install vLLM
• Verify monitoring imports successfully

.github/workflows/tests.yml

Defines four CI jobs mapping 1:1 to the canonical test commands:

Job	Trigger	Runner	Command
cpu	Push / PR	[self-hosted, linux, dmi-cpu]	not gpu and not e2e and not manual
gpu-smoke	Push / PR (after cpu)	[self-hosted, linux, gpu]	gpu and not multi_gpu and not slow
multi-gpu	Push / PR (after cpu)	[self-hosted, linux, multi-gpu]	multi_gpu
nightly	07:00 UTC cron / workflow dispatch	[self-hosted, linux, gpu]	slow or nightly

docs/testing.md

Adds testing documentation covering:

• The four canonical test commands
• Complete marker taxonomy
• Skip-guard conventions
• CI job mappings

Key Decisions

• Native backend build in CPU jobs

  • The CPU suite also requires building the native backend.
  • Monitoring imports load the prebuilt .so at import time (JIT disabled), so CPU tests require the backend to be compiled.
  • This requires nvcc during setup, but does not require GPU hardware at runtime.
  • The setup-dmi action handles this automatically, and the rationale is documented in testing.md.

• Self-hosted GPU runners

  • GitHub-hosted runners do not provide GPU resources.
  • GPU-dependent jobs (gpu-smoke, multi-gpu, and nightly) therefore target capability-labelled self-hosted runners appropriate for this lab repository.

• Exit-code-5 tolerance on GPU-tier jobs

  • multi_gpu currently selects zero tests because tensor-parallel tests are introduced in later phases.
  • Pytest exits with code 5 when no tests are collected, which would incorrectly fail the workflow.
  • Exit code 5 is therefore tolerated for:
    • gpu-smoke
    • multi-gpu
    • nightly
  • The CPU job remains a strict gate and does not mask empty selections.

Verification

• Both workflow YAML files parse successfully.
• The workflow contains the expected four jobs with the correct triggers.
• The nightly marker registers correctly:
  • Appears in pytest --markers
  • Produces no warnings.

Marker Selection Validation

Marker Expression	Selected Tests
not gpu and not e2e and not manual	78
gpu and not multi_gpu and not slow	18
multi_gpu	2
slow or nightly	12

Additional validation:

• All referenced build paths exist:
  • monitoring/Makefile
  • libs/clickhouse-cpp
  • integration/*

[Samfisheryu]

I rebased this PR onto main and removed the stale chained-branch state. The PR is now cleanly based on the current test-marker work, and the accidental submodule pointer changes are gone.

The CI design still needs to be realigned with #55. Please restructure it around the intended three-stage plan:

1. Stage 1: lightweight CPU PR gate
   This should be a true CPU-only gate using the current cpu marker. It should stay dependency-light and should not require the native backend build or self-hosted GPU infrastructure.

2. Stage 2: GPU / native regression
   The heavier native backend build, GPU tests, multi-GPU tests, ClickHouse, vLLM, and model-cache-dependent checks should run on the GPU/self-hosted tier. This likely requires setting up the self-hosted runner environment on the Frootlab server, including runner labels, CUDA/toolchain availability, ClickHouse/model-cache assumptions, and trigger policy.

3. Stage 3: packaging
   The native backend build should be wired into the install/package flow, so the workflow is not just manually running make in CI. If this PR does not implement that stage yet, please scope the documentation accordingly.

In short, please redesign this PR to follow the CPU gate → GPU/native regression on the Frootlab self-hosted server → packaging split from #55. Right now those responsibilities are still mixed together, especially in the shared setup path.

[Samfisheryu]

This is a security blocker from #55. With pull_request enabled here and the jobs below running on self-hosted runners, a fork PR can execute untrusted checked-out code on the Frootlab runner (checkout with recursive submodules, pip install -e, make, and the composite action). Docker would only be defense in depth; the primary fix is to avoid auto-running self-hosted jobs for untrusted fork PRs. The PR path should be GitHub-hosted CPU-only, and self-hosted jobs should be limited to trusted triggers such as schedule, workflow_dispatch, or internal PRs with a maintainer-controlled label.

[Samfisheryu]

These GPU-tier jobs should not run on every PR/push. if: github.event_name != 'schedule' means they run for normal PR updates and pushes after CPU, which conflicts with #55 Stage 2: GPU/native regression should be nightly plus explicit/manual maintainer opt-in, for example workflow_dispatch or a run-gpu label on trusted internal PRs. Please gate gpu-smoke and multi-gpu accordingly instead of making every PR consume self-hosted GPU resources.

[Samfisheryu]

@XbzOnGit Any comment on CICD design?

[SieDeta]

Issue 1 — Restructure around the three-stage plan

• Stage 1 (CPU gate): Job now runs on ubuntu-latest (GitHub-hosted). SKIP_NATIVE_BUILD=1 is passed via the setup-dmi action so no nvcc/cmake is required. Only the integration/transformers submodule is initialised (vLLM fork excluded — 3.7 GB). CPU gate marker expression tightened to not gpu and not e2e and not manual and not native_backend — excludes tests that require the compiled .so at runtime.
• Stage 2 (GPU/native regression): gpu-smoke, multi-gpu, and nightly run on [self-hosted, linux, gpu|multi-gpu]. The native backend is compiled via pip install -e . --no-build-isolation (calls setup.py NativeBuildExt) — no manual make in CI steps.
• Stage 3 (Packaging): Added setup.py with NativeBuildExt that shells out to cmake + make -C monitoring. The --no-build-isolation requirement is documented in both pyproject.toml and setup.py. No .so files are committed to git.

---

Issue 2 — Security blocker (fork PRs on self-hosted runners)

• CPU job runs on ubuntu-latest — no self-hosted runner is touched by any PR, including forks.
• All self-hosted jobs (gpu-smoke, multi-gpu, nightly) carry an explicit fork guard: github.event.pull_request.head.repo.full_name == github.repository
• Fork PRs can never trigger execution on the Frootlab runner.

---

Issue 3 — GPU jobs firing on every PR/push

• gpu-smoke and multi-gpu are now gated to: schedule (nightly) OR workflow_dispatch (manual) OR pull_request with the run-gpu label on a non-fork internal PR only.
• cancel-in-progress: false on all self-hosted jobs so a running GPU test is never killed mid-flight.

---

Additional fixes (from CI test runs)

• monitoring/ring_transport.py: Wrapped the native .so load in try/except ImportError so the module is importable on CPU-only runners without the backend built. _HOOK_DEFS falls back to () — all derived constants become empty. GPU monitoring calls fail at invocation time (not import time), allowing mocked tests to collect and run cleanly.
• monitoring/selection.py: Import of named HOOK_TYPE_* constants wrapped in try/except ImportError (they are not injected when
  _HOOK_DEFS = ()). Alias registrations (hidden-states, logits, etc.) guarded by if _id_by_short: to prevent KeyError when the backend is absent.
• tests/test_moe_v1_routing_hooks.py: Fork-specific top-level imports (qwen2_moe_compare, qwen2_moe_p) wrapped in try/except ImportError with pytest.skip(..., allow_module_level=True). Marker changed from cpu to framework_fork — these tests require the vendored DMI transformers fork, not just a CPU runner.
• tests/test_per_hook_isolation.py: test_round_trip_byte_identical now skips when framework == "vllm" — the vLLM _compare model files (gpt2_compare.py, qwen3_compare.py, llama_compare.py) are not yet vendored into integration/vllm/.
• integration/transformers submodule: Restored to ce5095aa (DMI fork, ring_offload branch). The previous pointer 3aa21543 was an upstream HuggingFace commit accidentally included in the "fix CICD design" commit — it lacks all DMI-specific model files (gpt2_compare, llama_compare, qwen3_compare, etc.) required by the test suite.
• requirements.txt: Added regex>=2023.12.25 (required by the transformers fork at import time). Pinned tokenizers>=0.22.1,<=0.23.0 (the fork's dependency_versions_check.py rejects tokenizers==0.23.1). Bumped huggingface-hub to >=1.2.1,<2.0.
• conftest.py: Added huggingface_hub.is_offline_mode shim for compatibility with huggingface-hub>=1.0 which removed the top-level export.