Runtime reconfiguration (node-toggle): low-overhead per-hook monitoring switch (CUDA-graph node enable/disable) by zhaoyboo · Pull Request #69 · ProjectDMX/DMI

zhaoyboo · 2026-06-19T04:47:03Z

Runtime reconfiguration (node-toggle): low-overhead per-hook monitoring switch

Summary

Enable/disable individual monitoring (producer) hooks at runtime, between
CUDA-graph replays, via cudaGraphNodeSetEnabled on the captured producer
kernel nodes — no re-capture, no re-instantiation, no model changes. This is the
enabling primitive for adaptive monitoring (graduated hallucination monitoring,
per-layer profiler sweeps, on-demand debugging): you pay only for the hooks you
turn on, and you can change the set per step.

The feature is independently switchable — with dmx_node_toggle=false (default)
the path is identical to stock DMI.

Design

A monitored decode step runs as a captured CUDA graph; each hook is a producer
kernel node in that graph. Toggling a hook = flipping its node's enabled bit on
the instantiated graph exec (cudaGraphNodeSetEnabled); a disabled node becomes
an empty-node traversal (~0.18 µs residual), so idle is near-baseline.

The one invariant that drives the design: every step, three lanes must agree on
which hooks fire —

capacity-reserve (adaptor_base._compute_step_plan),
meta-push (ring_transport.pre_push_all_metas),
device node-enable (the SetEnabled state),

or the ring desyncs (reserve ≠ writes, or meta ≠ payload). All three read one
source of truth, transport.specs_for_step().

Two execution modes:

eager (set_active_hooks): flip every bound exec immediately; safe at a
quiescent point (static config after warmup).
lazy (set_active_hooks_lazy): defer the device flip to each graph's next
replay, event-guarded; for per-step / adaptive reconfigure.

Decode-gated, prefill-passthrough. The toggle governs only producers inside
a replayed decode graph. Prefill / eager steps run producers ungated, so on
those steps all three lanes use the full active set (stock behavior); only
decode-graph steps use the toggle subset. The per-step selector reads vLLM's own
FULL-graph dispatch condition.

Implementation

Toggle policy and state live in dedicated, toggle-owned modules; the core files
keep only seams and one-line delegates.

monitoring/csrc/ring/toggle_registry.{h,cu} — ToggleRegistry: node
registry (graph → producer nodes), graph↔exec bindings, the enabled set,
eager apply_all, lazy per-graph ensure_current + replay-event guard, and a
registry-version guard cache. CUDA-runtime-only; own mutex.
monitoring/node_toggle.py — NodeToggleController: activation guards
(fail-closed on incomplete/non-uniform registry), the enabled→effective set
recompute (memoized), eager/lazy reconfigure.
integration/vllm_node_toggle.py — vLLM wiring: keep_graph patch +
replay-time graph guard, config parse, captured-graph bind, post-warmup
activation.
Capture-time node recording lives in the producer op
(ring_torch_op.cpp) — fail-closed: anything it can't validate (non-kernel
tail, unreadable capture state, unsupported producer) flags an anomaly and
refuses activation.

Composes with gpu_padding_strip (basic + prefix producers record their node);
the chunked producer is not toggle-managed and fails closed.

Performance (H100, Qwen3-8B, 36 hooks)

Config	Per-step overhead	Note
toggle off	+0.10%	≈ baseline; 3.7× cheaper idle than `null_mode`
per enabled hook	~3.5 µs	linear in #enabled
full (all hooks)	≈ plain always-on	~13 µs machinery over always-on
reconfigure (lazy)	4.4 µs host-only	flat in #graphs (registry-version cache)

Files changed

New (toggle-owned):

File	Lines	Role
`monitoring/csrc/ring/toggle_registry.{h,cu}`	+100 / +279	C++ registry + CUDA-graph ops
`integration/vllm_node_toggle.py`	+253	vLLM wiring (patch, guard, bind, activation)
`monitoring/node_toggle.py`	+198	host-side controller (guards, caches, reconfigure)

Modified (core — the coupling surface):

File	Δ	What
`monitoring/csrc/ring/ring_engine_py.cu`	+138 / −4	method delegates + `prepare_step` reserve clamp
`monitoring/csrc/ring/ring_engine_py.h`	+86	declarations
`integration/vllm_adapter.py`	+102 / −3	config reads + flag-gated lifecycle + per-step predicate
`monitoring/ring_transport.py`	+93 / −2	`effective_specs`/`specs_for_step` seam + delegates
`monitoring/csrc/ring/ring_torch_op.cpp`	+79	capture-time node recorder
`monitoring/csrc/bindings.cpp`	+73 / −1	pybind surface
`monitoring/hook_points.py`	+12	eager-path gate
`monitoring/adaptor_base.py`	+10 / −2	capacity seam
`ring_torch_op.h` +6, `Makefile` +2	minor	declaration + build

Plus 16 ring-gate tests (+2119) and 3 docs (node_toggle_usage,
node_toggle_implementation, node_toggle_migration_scope).

Coupling interface (behavior when toggle is OFF)

With the toggle off, the only new code that executes is below — each
identity-equivalent to stock (main's unit suite passes unmodified).

Surface (runs even when OFF)	Off behavior	Per-step cost
`specs_for_step()` seam (capacity + meta)	returns `active_specs` (same object)	2 property reads
`prepare_step` reserve clamp	identical to main's subtraction in every reachable state; clamps only states main cannot reach	2 compares
`dmx_record_capture_node` in producer ops	first-line early-return on a flag only toggle wiring sets	1 bool / producer call
`_gated_step` compute (vLLM adapter)	gated on `toggle_gate_active` — not computed when off	0
`hook_points` eager gate	inside the `force_eager` branch + `toggle_gate_active` short-circuit	0
vLLM wiring	behind the `dmx_node_toggle` flag; `CUDAGraph` not patched when off	0

The one unavoidable intrusion. Node-toggle cannot be purely additive: the
lockstep invariant forces _compute_step_plan (reserve) and pre_push_all_metas
(meta) to read specs_for_step() instead of active_specs — a real edit to
main's per-step path that runs every step, toggle on or off. It is safe by
construction (off → returns the same active_specs; on → all three lanes read
the one set), and is the seam to review hardest. Any future per-step quantity
derived from "which hooks fire" must read specs_for_step() for the same reason.

Tests

15 ring gates (binding, eager/lazy/multigraph reconfigure, reserve-invariant,
graph guard, lazy fault-injection, negative-desync, prefix + reserve, chunked
fail-closed, reconfigure cache, prefill-passthrough, force-eager gate) + a
self-checking FULL-decode vLLM smoke (Qwen3-0.6B → ClickHouse; partial toggle
delivers exactly the enabled layers). Main's unit suite (46) passes unmodified.

Notes

Requires cudagraph_mode=FULL for decode (the worker raises if a subset is
requested but no full-decode graph is bound). If an unregistered graph ever
reaches replay, the guard fails loud rather than silently desyncing.
Chunked-producer toggle support is deferred until a chunked producer is
dispatched and ring-space reclamation lands; until then it is fail-closed.

…, prepare_step clamp) Re-applied the node-toggle C++ layer onto the post-#40/#51 Ring2 backend: - ring_torch_op: g_toggle_capture + capture-record block on the basic producer only (option B: toggle runs with gpu_padding_strip=False, so every hook dispatches to producer, not producer_prefix/producer_chunked). - ring_engine_py: toggle registry in Impl, set_enabled_hooks/apply_toggle, is_hook_enabled + batched effective_enabled_mask (#14 guard), Phase-4 lazy ensure_graph_current/record_replay_event, two-sided clamp merged into main's prepare_step (reserve-invariant safety). - bindings: pybind for the 14 toggle methods. - test_toggle_binding.py: adapted to main's 4-arg producer op; drives real capture->record->bind->apply, all pass.

…s[_lazy], meta-gate) Re-applied the ring_transport.py toggle surface onto main's post-#42 transport: - effective_specs property = single source (active ∩ enabled ∩ registered), read by meta-push (here) and capacity-reserve (Layer 3, adaptor_base). - set_active_hooks (eager) / set_active_hooks_lazy (Phase 4) + _recompute via one batched effective_enabled_mask pybind call; clear_toggle teardown; ensure_graph_current / record_replay_event wrappers. - pre_push_all_metas now iterates effective_specs (preserving main's actual_q_len/strip + flags handling). - bindings: restored the Python-callable SubmitFn sink (test/custom-consumer path main had dropped) so the reconfigure guardrails can observe slices. Gates (full pipeline producer->drain->p2p->submit, all pass): test_reconfig_sequence_e2e (eager, all transitions), test_reconfig_lazy_e2e (lazy per-graph + event guard, 2 graphs), test_reconfig_multigraph_e2e (uniform-graph + per-graph routing).

…invariant) - adaptor_base._compute_step_plan walks transport.effective_specs instead of active_specs -> capacity-reserve, meta-push and device-enable all read the one source. No-op when toggle inactive (effective_specs == active_specs). Reserving for the full active set while only the enabled subset fires would drift the ring head/tail (the reserve-invariant bug); this closes it in lockstep with main's actual_q_len/strip byte sizing (untouched). - get_stats() + RingFlushStats: live head/tail probe (deferred from Layer 1), now exposed to validate the invariant. Gate test_reserve_invariant_e2e: drives the real _compute_step_plan -> prepare_step -> pre_push_all_metas -> replay -> drain across full/empty/partial reconfigures; every round n_hooks == #enabled AND the ring fully drains (payload_gap == task_gap == 0) -> reserve == actual writes, no drift. All 5 toggle ring gates green.

Re-targeted the toggle vLLM lifecycle onto main's thin DMXGPUWorker + VLLMAdaptor: - module-level keep_graph monkeypatch (_KeepGraphCUDAGraph) with the Phase-4 lazy replay() override (ensure_graph_current + record_replay_event), gated by _DMX_LAZY_REPLAY_HOOK; _parse_enabled_hooks. - init_device: read dmx_node_toggle / dmx_enabled_hooks / dmx_lazy_toggle. - OPTION B: force gpu_padding_strip=False when node-toggle is on, so every hook dispatches to the basic producer (the only op whose node is toggle-recorded); prefix/chunked producers would bind 0 nodes otherwise. Warns if user asked strip=True explicitly. - compile_or_warm_up_model: open capture window (clear registry + enable_toggle_capture + keep_graph patch) before super(); after warmup bind each captured graph's exec, close the window (#1), and apply the static enabled set eagerly or lazily. Guards #1/#2 (raise if subset requested but nothing bound). - _dmx_bind_captured_graphs: walk CUDAGraphWrapper.concrete_cudagraph_entries, instantiate-once (#3), bind (graph -> exec). Gate test_toggle_keepgraph_patch (model-independent): import + parse + keep_graph patch all pass. NOTE: full server smoke (bind N graphs + serve + toggle) is gated on Layer 0 -- the integration/vllm submodule is pinned at the old branch commit (3-arg producer); main's backend is 4-arg, so the fork must move to main's pin (4-arg-compatible hooked models) before an end-to-end run.

…ess-only #1 (vllm_adapter replay): ensure_graph_current()'s result was discarded and the graph replayed unconditionally. Since before_forward already pushed this step's metas for the new enabled set, replaying after a failed/partial device apply desyncs the ring. Now: nonzero result -> FATAL RuntimeError (the FIFO is already dirty for this step, so it is unrecoverable; the worker must terminate, must not be caught and resumed). #2 (ensure_graph_current): applied_version[g] was bumped unconditionally, so a partial cudaGraphNodeSetEnabled failure marked the graph 'current' and the unapplied nodes were never retried. Now the version is bumped ONLY on full success; per-node last_enabled already makes a retry idempotent. Build clean; lazy e2e success path regression passes. The CUDA-error path itself is not unit-tested (cudaGraphNodeSetEnabled can't be forced to fail from Python without fault injection) -- guaranteed structurally (raise-on-nonzero + version-on-success-only). Testable failure paths land with #3/#4/#5.

…s check #3 runtime-captured graphs: vLLM's CUDAGraphWrapper can capture a NEW graph at runtime (batch_descriptor absent at warmup). Such a graph has no recorded producer nodes and no bound exec -> its producers run default-ON while the meta gate filters to the enabled subset -> desync. Added a replay-time guard armed whenever a toggle gate is active (eager AND lazy, via _DMX_TOGGLE_REPLAY_GUARD): the keep_graph replay() override calls the new read-only is_graph_ready(raw) and RAISES FATAL if the graph isn't registered+bound. Eager = validation only (apply happened at config time); lazy = validate + ensure_graph_current (separate APIs, per the eager/lazy separation -- eager never runs lazy apply at replay). The raise is unrecoverable by design: before_forward already pushed this step's metas. Note: vLLM's first use of a new descriptor captures-and-returns without replay(), so the guard trips on the graph's SECOND use (covered by the test). #4 registry completeness: set_active_hooks[_lazy] now also requires toggle_registry_complete() -- the recorded-node graph set must EXACTLY match the bound-exec set. A partial/mismatched bind previously passed (only count>0 + uniform were checked) and would desync the unbound graph. New engine APIs: is_graph_ready (read-only, no version/event/mutation), toggle_registry_complete. Gate test_toggle_graph_guard_e2e covers both failure paths (incomplete registry raises; unknown graph raises at replay; ready graph replays clean). All 7 toggle gates green.

…tent-hook hack) _parse_enabled_hooks now distinguishes three meanings instead of overloading the empty string: - not provided / '' -> None : toggle gate INACTIVE (all hooks fire) - 'none' -> [] : explicit empty set -> toggle-0 (all OFF) - '0:1,0:2' -> [...] : that specific set compile_or_warm_up_model already activates on 'is not None', so [] -> set_active_hooks([]) -> gate active, effective set empty -> every node disabled. Removes the need for the '0:99' nonexistent-layer workaround to express all-off. Cases covered in test_toggle_keepgraph_patch.

…zy for dynamic) Make the in-flight-race contract explicit (per review): eager set_active_hooks -> apply_toggle mutates bound execs immediately and does NOT wait on replay events, so it is only safe at a quiescent point (post-warmup static config). Dynamic / per-step reconfigure must use the lazy path, whose ensure_graph_current waits on the prior replay event before mutating. No behaviour change.

…jected gate Review follow-ups: (High) lazy event guard could fail silently -> UB/desync. Fixed: - ensure_graph_current now checks cudaEventSynchronize; on failure it does NOT touch the exec (can't confirm the prior replay finished) and returns the error. - record_replay_event now returns the CUDA error from event create/record (was void, swallowed both). A failed/missing event would let a later ensure mutate a still-executing exec; the replay hook now treats nonzero as FATAL. (Medium) #1/#2 had only success-path coverage. Added a real fault-injected gate (test_toggle_lazy_failpath_e2e) via a test-only _test_force_apply_error hook + _test_applied_current introspector: verifies apply-failure leaves the graph stale (not marked current) and that a failed lazy apply at replay RAISES FATAL (no silent desync), plus recovery on success. (Low) corrected the no-graph-bound message: node-toggle needs cudagraph_mode=FULL; PIECEWISE graphs hold a model subset and fail the uniform-hook-set requirement. #3 (real two-phase runtime-capture) remains a server-smoke concern, not a code gap. All 8 toggle ring gates green.

…eout_us (#6) Real end-to-end smoke (tests/ring/smoke_toggle_vllm.py, offline LLM, in-process worker, full pipeline -> ClickHouse): Qwen3-0.6B, cudagraph FULL, node-toggle ON with a PARTIAL set (last 4 of 28 hidden-state hooks). Result: - bound 35 FULL decode graphs, 980 nodes (35x28) registered; - option B: gpu_padding_strip auto-forced off (warned); - active hooks [(0,24),(0,25),(0,26),(0,27)] applied (eager); - replay guard active (mode=eager) -- the #3 guard runs in vivo; - coherent generation, no desync/crash; - ClickHouse got 760 rows for EXACTLY layers 24-27, ZERO for the other 24 -> partial-toggle lockstep proven on a real model. Wired dmx_drain_flush_timeout_us (#6): the RingConfig field existed + was bound but init_device never set it, so low-volume/sparse-hook data never timer-flushed (the smoke's first runs landed 0 rows until this). Default 0 = legacy. Observation (not a regression): vLLM auto-downgraded FULL->FULL_AND_PIECEWISE because FlashAttention only supports UNIFORM_BATCH full graphs here; node-toggle still bound the 35 FULL *decode* graphs and served correctly. If a piecewise graph ever replayed unregistered the guard would fail loud (no silent desync).

#1 exit code: clickhouse-client returncode now checked; strict asserts (total>0, layers == enabled set EXACTLY, hidden dim correct) each exit NONZERO via fail(); SMOKE_OK + exit 0 only on full success. (Verified it gates: an over-strict shape assertion made it exit 1.) Previously it printed SMOKE_OK + exit 0 unconditionally. #2 data isolation: the script now DROPs its table at start (and end), so any rows seen are provably from THIS run -- historical rows for the same layers can no longer mask a no-export run. Asserts total rows + layer set + shape. #3 corrected the shape expectation: hidden-states rows are [q_len, HIDDEN] -- q_len=1 (decode) AND >1 (prefill chunks), not only [1,1024]. Docstring now says this validates the FULL-DECODE path (graphs node-toggle binds); vLLM may auto-downgrade FULL->FULL_AND_PIECEWISE, so it is not full piecewise coverage. Re-run: EXIT 0, 768 rows, layers exactly [24-27], shapes [1,1024]+[N,1024].

#2 drain_flush default: was 0 on v2 (would buffer sparse partial-toggle data until the ring fills/shutdown); the original toggle branch defaulted 50ms. Now conditional: 50000us when dmx_node_toggle is on (prompt export without manual tuning), 0 otherwise (preserves main's non-toggle behaviour). Override still via dmx_drain_flush_timeout_us. The smoke now OMITS the explicit override so it exercises this default (re-run: EXIT 0, 672 rows, layers exactly [24-27]). #4 docs: added docs/node_toggle_usage.md (user-facing config/API: dmx_node_toggle, enabled-set semantics incl. 'none'=all-off, eager vs lazy + the static/dynamic constraint, FULL-decode requirement + FULL_AND_PIECEWISE downgrade note, gpu_padding_strip auto-off, sparse-hook timeout, runtime API, fail-loud guards). Also tracks the previously-untracked design docs (implementation, migration_scope).

…, negative test) Three proportionate items from the design review (skipped the premature CUDA-API optimizations and the already-fail-loud 'gaps'): 1. clear_toggle now RE-ENABLES disabled device nodes before dropping the registry -- otherwise a graph replaying after a runtime clear (host gate back to push-all-metas, device nodes still OFF) would desync. Best-effort, safe at the quiescent call sites; doc clarifies runtime revert should use set_active_hooks(full_set), not clear_toggle. 2. Capture-node validation (fail-closed): the capture site now checks the tail dependency is a kernel node (cudaGraphNodeGetType) before registering it; a non-kernel tail (future multi-op producer / capture event-join / odd topology) is counted as an anomaly and set_active_hooks[_lazy] then REFUSES to activate rather than toggling a wrong node. Hardens the deps[nd-1] assumption. 3. Negative desync test (test_toggle_negative_desync_e2e): deliberately pushes a REVERSED meta order and asserts ALL rows mis-associate (layer != marker) -> proves the FIFO-order lockstep is load-bearing; positive control with correct order stays aligned. Plus a capture-anomaly fail-closed gate in test_toggle_graph_guard_e2e. Doc: noted current scope is engine/step-level (not per-request-within-batch). All 9 toggle ring gates green.

Comment/docstring-only, no behaviour change: - remove Phase X labels (Phase B / Phase 4 / Phase 3b) from the toggle code - remove review/issue tags (#1/#2/#3/#4/#5/#14, Guard #N) - remove dev-history narration (original branch / matches / fix / today's behavior / verbatim / deferred from Phase 1.5) - tighten the set_active_hooks docstring Comments now describe what the code does + why (correctness), not its history.

Toggle now records the prefix producer's kernel node too, so it works with gpu_padding_strip=True (the common dense hidden-states case) instead of forcing strip off: - ring_torch_op: extract dmx_record_capture_node helper; basic + prefix producers record their node (supported=true, kernel-node validated); chunked/MoE producer flags an anomaly (supported=false) so set_active_hooks fails loud rather than leaving a fired-but-unregistered producer (silent desync). - vllm_adapter: drop the forced gpu_padding_strip=False under toggle. - ring_transport: anomaly message names the chunked/MoE cause + the remedy (dmx_gpu_padding_strip=False). - usage doc updated. Chunked/MoE deliberately out of scope (reserve accounting under variable per-chunk bytes is the trickiest combo + needs an MoE model to test) -> fail-closed. Gates: new test_toggle_prefix_e2e (prefix node recorded, no anomaly, reconfigure lockstep + correct stripped prefix). All 10 ring gates pass. Real vLLM smoke re-run with strip ON (default): bound 35 graphs/980 prefix nodes, 672 rows for exactly the enabled layers.

… fixes Review follow-ups on the prefix-path Option A: - (#1) test_toggle_prefix_reserve_e2e: drives the real capacity path (_compute_step_plan -> prepare_step -> prefix replay -> drain) and asserts the ring fully drains (payload_gap == task_gap == 0) every round while varying BOTH the enabled subset AND the actual row count -> proves CPU-reserved bytes == prefix-kernel written bytes (strip byte-accounting in lockstep). gap==0 holds. - (#2) test_toggle_chunked_failclosed_e2e: captures a mixed basic+chunked graph, asserts the chunked node is NOT registered + flags an anomaly, and that set_active_hooks / _lazy both RAISE (naming chunked + the remedy). Direct 'no silent desync' gate. - (#3) wording: 'chunked/MoE' -> 'chunked' everywhere; the vLLM adapter never dispatches chunked (rb>0 always -> prefix), so MoE hooks go through prefix, not chunked. Clarified chunked is dormant infra in comments/messages/usage doc. - (#4) dmx_record_capture_node is now fully fail-closed: capture-info read failure OR (active && cap_nd<1) flags an anomaly; only the legitimate not-actively-capturing eager case returns silently. All 12 toggle ring gates pass.

Remove misalignment-prone narration from branch-added comments: - scheme codenames (Option A/B, prefix-path implementation, Layer N, Phase N) - review/PR tags (fixes #1-#4, post-#40/#51 backend) - time-anchored claims (today, dormant infra, legacy, pre-Layer-3 bug) - cross-layer behavior claims (vLLM captures all hooks in every graph) - already-stale facts (binding test claimed only the basic producer records nodes; prefix-test docstring conflated chunked with MoE) Comment-only; backend rebuilt, guard gates re-run green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…its) The earlier comment sweeps also rewrote six comment lines that belong to main (Phase 1.5 / today's-behavior wording in vllm_adapter, ring_transport, ring_engine_py.h, ring_torch_op.cpp). Those lines are main's to own; restore them verbatim so the PR diff carries only node-toggle changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ph class If vLLM ever resolves torch.cuda.CUDAGraph before (or without) the keep_graph patch -- e.g. an import-time binding in a future version -- graphs are created with keep_graph=False: the template is freed on instantiate, the capture-recorded node handles dangle, and SetEnabled on them is UB that no existing guard or test can catch (the smoke can pass despite it). Check each graph's type for the patch marker at bind time, before any handle is touched, and fail loud with the root cause. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…egistry version Per-step reconfigure is the access pattern of the planned consumers (graduated hallucination monitoring, profiler layer sweeps, debugging), and its dominant cost at production scale was re-validating an immutable registry: the set_active_hooks[_lazy] guards are O(graphs x hooks), measured 9.9/15.5/36.6us host-only at 1/8/35 bound graphs. The engine now keeps a registry_version bumped by every registry mutation (capture flag, node recording, anomaly, bind, clear); the transport memoizes on it: a guard PASS is cached per version (failures never cached), and effective_specs is memoized per (version, enabled set, active_specs identity). Fail-loud semantics unchanged -- any mutation forces full re-validation, gated by test_toggle_reconfig_cache_e2e (preset equivalence, active_specs-swap miss, anomaly-after-cached-PASS, clear+incomplete-rebind). Measured after: lazy reconfigure 4-6us flat, independent of graph count (35 graphs: 36.6 -> 4.4us). All 13 ring gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Two violations of main's own consumer contracts, found by running main's unit suite against the branch: - effective_specs assumed __init__ ran; main's test_hook_spec_flags builds a minimal RingTransport via __new__ and pre_push_all_metas crashed on the missing gate attributes. getattr defaults now map 'attributes absent' to 'toggle inactive' (the correct semantics for any minimal transport). - is_graph_ready's docstring said 'vLLM', violating ring_transport.py's framework-neutrality hygiene test; rephrased. (The test's two remaining violations, lines 175/424, are main's own pre-existing lines -- it is red on origin/main itself.) 13 ring gates + main's unit tests (43) green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Mechanical move, zero semantic change. The toggle's Python logic now lives in two toggle-owned files: - monitoring/node_toggle.py -- NodeToggleController: registry guards (with the version-keyed verdict cache), effective-set recompute (with the preset memo), eager/lazy reconfigure, clear, replay-time hooks. - integration/vllm_node_toggle.py -- keep_graph patch + replay guard, dmx_enabled_hooks parser, captured-graph bind walker (incl. the unpatched-class check), post-warmup activation. Core files keep only the seam and one-line delegates: ring_transport.py is +74 lines vs main (was ~+190), vllm_adapter.py +54 (was ~+265). The replay guard now reads transport.toggle_gate_active / toggle_lazy_active properties instead of private attrs. Tests repointed to the new modules. 13 ring gates + main's unit suite (43) + the full vLLM smoke green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Mechanical move, zero semantic change. All toggle state + CUDA-graph operations (node registry, exec bindings, enabled set, lazy versions + replay events, anomaly counter, registry version, the internal mutex) move from RingEnginePy::Impl into a self-contained ToggleRegistry class (toggle_registry.{h,cu}, CUDA-runtime-only dependency -- the engine passes the ATen stream in). RingEnginePy keeps thin delegates plus the two glue points that belong to the engine: the process-global capture flag (ring_set_toggle_capture) and the current-stream lookup. ring_engine_py.cu shrinks from +355 to +142 vs main (delegates + the prepare_step clamp + get_stats); pybind surface unchanged. 13 ring gates + main's unit suite (43) + full vLLM smoke green (35 graphs / 980 nodes bound, 672 rows exactly layers 24-27). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… desync Bug (found by off-path audit): the meta-push + capacity-reserve lanes read the toggle subset on EVERY step, but the toggle's SetEnabled only gates producers inside a replayed decode graph. A prefill/eager step runs producers ungated (all active hooks fire), so it reserved/pushed K metas while firing N producers -> a permanent surplus of (N-K) orphan task entries. The p2p drain pairs metas to payloads by flat FIFO 1:1 (p2p_thread.cpp do_post_processing), so the surplus CASCADES: every subsequent row is paired with a shifted payload (+ shifted step context). Layer labels come from metas (always look right), so the corruption was invisible to the old smoke; only size-boundary crossings emitted the [p2p] shape/bytes mismatch warnings. Fix: split the path. transport._gated_step selects the per-step spec set -- effective_specs (toggle subset) on a decode-graph step, full active_specs on a prefill/eager step. The vLLM adapter sets it from vLLM's own FULL-dispatch condition (uniform-decode batch within captured size; same private-attr risk surface as predict_padded_q_len), conservative-to-full on any unexpected shape. Prefill is unchanged from stock (full monitoring); decode is gated. Both lanes follow the split, so the ring stays in lockstep on both paths. Gate: test_toggle_prefill_passthrough_e2e (decode replay gated + eager prefill full, ring drains gap==0 across both, incl. a post-prefill decode still aligned; teeth-checked -- gating the prefill step reproduces task_gap=-(N-K) cascading into the next step). Smoke now distinguishes decode rows (==enabled) from prefill rows (full active set) by shape, and fails on any p2p mismatch warning (the previously-unwatched desync symptom). 14 ring gates + main unit suite (43) + full vLLM smoke (0 mismatch warnings) green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The eager safety net (HookPoint.forward force_eager branch) fires producers without a graph, so SetEnabled can't gate them. On a gated decode step that overflowed capacity -> force_eager, a disabled hook fired while meta-push counted only the enabled subset -> meta<->payload FIFO mispairing (content corruption; reserve_one keeps head/tail balanced so there's no gap symptom). Gate host-side in the eager branch (safe: never captured/compiled), guarded by toggle_gate_active so the non-toggle eager path (stock prefill) is untouched. Gate: test_toggle_force_eager_gate_e2e drives every HookPoint.forward eagerly with force_eager + a decode subset and checks each delivered row's payload MARKER matches its layer label (the gap-based check has no teeth here); teeth-verified -- neutering the gate yields mispaired=[(2,1)]. 15 ring gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- Only compute _gated_step when the toggle gate is active; with the gate off, specs_for_step() returns active_specs regardless, so the non-toggle path no longer pays for _step_uses_decode_graph every step (off-state decoupling). - Reword a ring_transport.py comment ('The vLLM adapter' -> 'serving-framework adapter') to satisfy the framework-neutrality lint. - Fix a rebase merge artifact: drain_flush_timeout_us was assigned twice (node-toggle's 50ms-when-on conditional, then main #65's 100ms default overriding it). Merge to one assignment: 50ms when toggle on, else main's 100ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

zhaoyboo and others added 26 commits June 18, 2026 16:58

zhaoyboo changed the title ~~Runtime node-toggle: low-overhead per-hook monitoring switch (CUDA-graph node enable/disable)~~ Runtime reconfiguration (node-toggle): low-overhead per-hook monitoring switch (CUDA-graph node enable/disable) Jun 19, 2026

Navigation Menu

Uh oh!

Runtime reconfiguration (node-toggle): low-overhead per-hook monitoring switch (CUDA-graph node enable/disable) - #69

Runtime reconfiguration (node-toggle): low-overhead per-hook monitoring switch (CUDA-graph node enable/disable)#69
zhaoyboo wants to merge 26 commits into
mainfrom
feature/node-toggle-v2

zhaoyboo commented Jun 19, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

zhaoyboo commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Runtime reconfiguration (node-toggle): low-overhead per-hook monitoring switch

Summary

Design

Implementation

Performance (H100, Qwen3-8B, 36 hooks)

Files changed

Coupling interface (behavior when toggle is OFF)

Tests

Notes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

zhaoyboo commented Jun 19, 2026 •

edited

Loading