scsd: re-emit op 0x02 to the coordinator at the ADD transition (vms-a84d) - #1029
scsd: re-emit op 0x02 to the coordinator at the ADD transition (vms-a84d)#1029baron-3dl wants to merge 2 commits into
Conversation
…84d) The booted-node cluster join stalls at barrier step 5/12: the second existing member (VAX1) never pushes its disk-DLM rebuild toward the joining OVMX node, so it withholds its own op-0b#5 and the coordinator never releases 0c#5. Root cause (directly observed on the inter-VAX channel already captured in the run-1025 pcap; three-way grounded against the vax3-2to3 reference): the coordinator emits its op-0x12 "rebuild toward the new member" relay to the other existing member ONLY in response to a FRESH joiner op-0x02 landed AT the transition. In the reference this is exact -- joiner op-0x02 -> COORD op-0x12 relay 0.3ms later -> that member rebuilds and clears its barrier step. OVMX's op-0x02 to the coordinator is one-shot on the JOIN_CFG2_DELAY_MS (4.9s) timer (cfg_sent/joiner_cfg2_sent), decoupled from the coordinator's transitions: it DID fire COORD's relay once, but for the pre-epoch transition; the OVMX-ADD epoch opens afterward and gets no fresh op-0x02, so COORD never relays OVMX into VAX1's rebuild set. Fix: when the coordinator opens an ADD transition (op-0x09 XITION) while we are the joiner, re-emit a fresh op-0x02 to the coordinator, once per ADD-epoch. The op-0x02 CONTENT is byte-identical to the deferred one (proven byte-equal to a real joiner's op-0x02); only its TIMING changes. No fabricated lock/held state -- INV-6 clean; the gate-clearing member rebuild is a pure cat-0x82 echo OVMX already performs for the coordinator. Joiner-path-guarded (rx->do_connect && !appeared_after_join), so the member/bystander and OVMX<->OVMX (H2) paths are untouched. H2 cannot exercise the real COORD->VAX1 relay, so CI-green is necessary-not- sufficient; the lab re-fire vs vaxlab-2 is the actual proof. All 26 vmsscs unit tests pass (incl. the 141k-frame scsd_wire replay). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PhM3QcmFEY3p8YNnHGaYwP
…arrier (vms-a84d) Lab re-fire #1029b tested the epoch re-emit in isolation (collision landed on the coordinator's VC, XITION opened, SCSD-I-CFG2REEMIT fired) and exposed a timing inversion: the re-emit was armed on the XITION but SENT from the poll-loop choke point, which fires on the JOIN_CFG2_DELAY_MS (4.9s) cadence -- so it landed at +35.26, ~4s AFTER the barrier had already run steps 1-5 and stalled at +31.2. The coordinator needs the fresh op-0x02 BEFORE it fans out the barrier; 4s late it is ignored, no op-0x12 relay, VAX1 never rebuilds, barrier stuck at 5/12. The reference (vax3-2to3) sends its relay-triggering op-0x02 0.04s BEFORE the barrier (+34.7634 op-0x02 -> +34.807 barrier). Match that: send the fresh op-0x02 SYNCHRONOUSLY in the XITION handler, the instant the ADD transition opens (~1s before the barrier fan-out), instead of deferring to the poll loop. Drops the poll-loop block and the cfg2_reemit_pending flag; keeps cfg2_reemit_epoch as the once-per-epoch guard. Content still byte-identical to the deferred op-0x02 (INV-6 clean); only the timing changes. Joiner-path-guarded. All 26 vmsscs unit tests pass. ⚠ Still a candidate: whether a correctly-timed fresh op-0x02 actually triggers the coordinator's op-0x12 relay is coordinator-internal and unproven -- the lab re-fire is the arbiter. #1029b could not test it because the send was 4s late. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PhM3QcmFEY3p8YNnHGaYwP
|
Held as draft — the op-0x02 avenue is refuted. Lab re-fire #1029c-1 confirmed the synchronous timing fix works mechanically (SCSD-I-CFG2REEMIT now fires 0.84s before the barrier, matching the reference), but a correctly-timed fresh op-0x02 to the coordinator produces no op-0x12 relay, no VAX1 rebuild — barrier still stalls at 5/12, CN=2. So op-0x02 does not trigger the transition-time VAX1 relay; the earlier +18.18 correlation was coincidental. Both parts of this PR (timing + would-be coordinator routing) are moot against a refuted mechanism. Holding as draft pending identification of the real coordinator-internal relay trigger; this branch will be reworked or closed once that lands. Cap-denial passed on every run — anti-fabrication guarantee intact throughout. |
The stall
The booted-node cluster join stalls at barrier step 5/12: the second existing member (VAX1) never pushes its disk-DLM rebuild toward the joining OVMX node, so it withholds its own
op-0b#5and the coordinator never releases0c#5.OVMXJ0reaches thememberflag but is never counted (CLUSTER_NODES stays 2).Root cause (directly observed on the inter-VAX channel)
The run-1025 pcap was captured on the lab bridge and already contains COORD↔VAX1 traffic. Mining it (three-way grounded against the
vax3-2to3reference):op-0x12"rebuild toward the new member" relay to the other existing member only in response to a FRESH joinerop-0x02landed AT the transition. Reference: joinerop-0x02→ COORDop-0x12relay 0.3 ms later → that member rebuilds and clears its barrier step.op-0x02to the coordinator is one-shot on theJOIN_CFG2_DELAY_MS(4.9 s) timer, decoupled from the coordinator's transitions. It did fire COORD's relay once — but for the pre-epoch transition. The OVMX-ADD epoch (0xE) opens afterward and gets no freshop-0x02, so COORD never relays OVMX into VAX1's rebuild set.Fix
When the coordinator opens an ADD transition (
op-0x09XITION) while we are the joiner, re-emit a freshop-0x02to the coordinator, once per ADD-epoch. Content is byte-identical to the deferredop-0x02(proven byte-equal to a real joiner's); only the timing changes.Honesty (INV-6)
No fabricated lock/held state. The gate-clearing member rebuild is a pure
cat-0x82echo OVMX already performs for the coordinator (the member originates its own value blocks; OVMX only reflects them). Two independent reference decodes confirmed the joiner never initiates held-state records.Scope / safety
Joiner-path-guarded (
rx->do_connect && !appeared_after_join), so the member/bystander and OVMX↔OVMX (H2) paths are untouched. No new opcode/enum/TU. All 26 vmsscs unit tests pass (incl. the 141k-framescsd_wirereplay).⚠ Verification bar
H2 cannot exercise the real COORD→VAX1 relay (the #1026 lesson), so CI-green is necessary-not-sufficient. The decider is the lab re-fire vs vaxlab-2, graded by eye on both consoles via the SDA CSB oracle: COORD relays VAX1 into OVMX's rebuild → VAX1 emits
0b#5→ barrier 5/12→12/12 → CLUB Nodes 2→3 →OVMXJ0 = MEMBERat CLUSTER_NODES=3, cap-denied. Milestone stays un-cut until that is observed sustained on both consoles.🤖 Generated with Claude Code