Skip to content

feat(orchestrator): startup hygiene — stale per-PID files + orphan detection - #8

Open
evannadeau wants to merge 2 commits into
SpawnBox-dev:mainfrom
evannadeau:feat/startup-hygiene
Open

feat(orchestrator): startup hygiene — stale per-PID files + orphan detection#8
evannadeau wants to merge 2 commits into
SpawnBox-dev:mainfrom
evannadeau:feat/startup-hygiene

Conversation

@evannadeau

Copy link
Copy Markdown

Summary

Two additive startup-time hygiene improvements in plugins/orchestrator/mcp/server.ts. Both run once per MCP startup, both are detection/cleanup (never auto-kill), both are no-ops in the steady state.

1. Reap stale per-PID active-session-<pid> files (b7f43b7)

Per-PID active-session-<pid> files (introduced in 0.30.19+) make session_id lookup race-free under concurrent sessions, but nothing has been reaping them when the owning claude process exits. On a developer machine with many short-lived sessions per day, they accumulate indefinitely — 8 stale files observed in one project on 2026-05-13, from claude PIDs long since dead.

They are cosmetic — the legacy single active-session file remains the primary lookup — but a slow directory listing eventually becomes a real cost.

This patch adds a startup sweep that walks <project>/.orchestrator-state/, matches files of shape active-session-<pid>, probes liveness via process.kill(pid, 0), and unlinks dead-PID entries. Cross-platform; cheap; idempotent; race-safe (only unlinks PIDs verified gone). Lost races with concurrent sessions tolerated — next startup retries.

2. Warn about likely-orphan sibling MCPs (a28388e)

Complements the existing orphan-bun watchdog (which catches "parent dies while I'm alive" for the current process). The watchdog only protects processes that LOADED the watchdog code — older bun processes whose in-memory bytecode predates a fix do not benefit, and can survive forever if their original parent claude died without triggering whatever watchdog they happen to be running.

Concretely observed 2026-05-13: an orphan bun survived ~30 minutes across multiple watchdog tick intervals before manual kill -9. The on-disk dist/server.js had been rebuilt while the orphan was running, so any subsequent watchdog improvements were invisible to it.

This patch adds a startup-time scan (Linux only) that walks /proc for bun processes whose cmdline references orchestrator/dist/server.js and whose parent chain contains no live claude process within 8 hops. Suspects are logged with diagnostic guidance:

[orchestrator] startup hygiene: detected N likely-orphan sibling MCP process(es): pid=A,B,C.
Their parent claude is no longer in the process tree, suggesting they outlived their owning session and may be running stale bytecode whose watchdog never fired.
Diagnose with 'pstree -ps <pid>'; clean up with 'kill -9 <pid>' if confirmed orphan.

Detection only — does NOT auto-kill. Sibling MCPs may co-own infrastructure shared across live sessions (the python sidecar is deliberately shared via .sidecar-port — killing an unrelated bun could take down a live session's embeddings). The operator decides whether to clean up.

Windows is unchanged — killOlderDuplicateMcps already handles a related case (siblings sharing our parent claude). Pure orphans on Windows are rare because parent death typically reaps children.

Why "detection, not auto-kill"

The sidecar reuse pattern at startSidecar() line 338–350 deliberately shares the python embedding server across MCPs. An auto-kill could take down a sidecar that a live session depends on. Surfacing the problem at startup gives the operator the information without the risk.

Files changed

  • plugins/orchestrator/mcp/server.ts — 2 new functions + 1 startup block wiring them in
  • plugins/orchestrator/dist/server.js — rebuilt via bun run build

Imports updated: added readdirSync and unlinkSync to the existing node:fs import line.

Tested

  • bun run typecheck — clean
  • bun test — 516 pass / 0 fail / 38 files / 1207 assertions (no test changes)
  • Manually verified the diagnostic recipe: 8 stale active-session-<pid> files in a real workspace were correctly identified as dead and could be removed by the same logic the reaper applies.

Test plan

  • Fresh install on a machine with no .orchestrator-state/ directory — both functions should be silent no-ops.
  • Fresh install on a machine where prior sessions left stale active-session-<pid> files — verify the startup log line reports the reap count and the files are gone.
  • Multiple live Claude Code sessions in the same project — verify the warner does NOT report them as orphans (each has a live claude ancestor).
  • Orphan reproduction (kill a parent claude process while the bun keeps running) — verify the warner reports the surviving bun PID at the next session startup.

🤖 Generated with Claude Code

evannadeau and others added 2 commits May 13, 2026 19:21
Per-PID active-session-<pid> files (introduced in 0.30.19+) make
session_id lookup race-free under concurrent sessions, but nothing has
been reaping them when the owning claude process exits. On a developer
machine with many short-lived sessions per day, they accumulate
indefinitely — 8 stale files observed in one project on 2026-05-13,
from claude PIDs long since dead.

The files are cosmetic in the sense that the legacy single
`active-session` file remains the primary lookup, but a slow directory
listing eventually becomes a real cost on a hot-spot workstation.

This patch adds a startup sweep that walks `<project>/.orchestrator-state/`,
matches files of shape `active-session-<pid>`, probes liveness via
`process.kill(pid, 0)`, and unlinks dead-PID entries. The probe is
cross-platform via Node's API. Cheap, idempotent, race-safe (we only
unlink files whose PID is verifiably gone). Lost races with concurrent
sessions are tolerated — next startup retries.

Runs once at MCP startup, unconditionally (even when the
no-claude-ancestor branch is about to exit, so future startups benefit).

Tested: bun run typecheck clean, bun test 516 pass / 0 fail.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Complements the existing orphan-bun watchdog (which catches "parent
dies while I'm alive" cases for the current process). The watchdog
only protects processes that LOADED the watchdog code - older bun
processes whose in-memory bytecode predates a fix do not benefit from
that fix, and can survive forever if their original parent claude
died without triggering whatever watchdog they happen to be running.

Concretely: on a developer machine that pulls plugin updates, an MCP
process loaded at time T1 may still be alive after the on-disk
`dist/server.js` is rebuilt at T2 > T1. If the parent claude that
spawned T1's bun dies after T2, the T1 bun's in-memory watchdog code
is the version from T1 - any later improvements to watchdog detection
are invisible to it. We observed this 2026-05-13: an orphan bun
survived ~30 minutes across multiple watchdog tick intervals before
manual cleanup via `kill -9`.

This patch adds a startup-time scan (Linux only) that walks /proc for
bun processes whose cmdline references `orchestrator/dist/server.js`
and whose parent chain contains no live `claude` process within 8
hops. Suspects are logged with diagnostic guidance; we do NOT
auto-kill, because sibling MCPs may co-own infrastructure shared
across live sessions (the python sidecar is deliberately shared via
`.sidecar-port`). Detection surfaces the issue; the operator decides.

Windows is unchanged - killOlderDuplicateMcps already handles a
related case (siblings sharing our parent claude). Pure orphans on
Windows are rare because parent death typically reaps children.

Tested: bun run typecheck clean, bun test 516 pass / 0 fail.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
evannadeau added a commit to evannadeau/claude-plugins that referenced this pull request May 31, 2026
…reaper regression (#4)

Two related fixes that share a root cause: a fresh single-claude launch can
end up with the MCP server's selfSession.session_id drifted from the harness
session_id, and nothing reaps the per-PID active-session-<pid> files that
feed the drift.

Bug 1 (phantom-sibling false positive):

On a cold start with no /resume, if a stale active-session-<pid> file from
a prior session collides on PID with the new claude process (or the legacy
single-file fallback wins a race), the MCP self-registers in
agent_channel.db under an id8 that is NOT the harness session_id. The
every-turn cross-session injection then compares against the harness id
only and reports the MCP's own row as "1 sibling session active" for the
whole session lifetime - heartbeat keeps advancing so the bug never
self-clears.

Fix: live_sessions.ts gains a process-wide self-id filter. server.ts
registers the MCP's selfSession id at startAgentChannel time AND on the
first explicit session_id observed via resolveSessionId. getLive-
OtherSessionIds excludes BOTH the caller id and the registered self id,
preventing the phantom even when the two disagree.

Bug 2 (stale per-PID active-session file reaper):

The per-PID active-session-<pid> scheme (0.30.19+) makes session_id lookup
race-free under concurrent sessions, but nothing reaps the files when the
owning claude process exits. They accumulate indefinitely - one project
hit 30 stale files in ~12 days. Worse, PID reuse hands the fallback
resolver a session_id that is no longer live, feeding directly into Bug 1.

Fix: extract reapStaleActiveSessionFiles into its own engine module
(testable in isolation with an injectable liveness probe) and wire it at
MCP server startup. Cheap, idempotent, race-safe via process.kill(pid, 0).
Originally proposed upstream as spawnbox-dev/claude-plugins PR SpawnBox-dev#8; that PR
never merged so this fork carries the fix.

Tests:
- tests/engine/live_sessions_phantom_self.test.ts (3 cases) - covers
  drift, genuine siblings still surface, idempotent self-id updates.
- tests/engine/startup_hygiene.test.ts (6 cases) - covers reap of dead-
  PID files, non-PID files ignored, missing dir no-op, empty dir no-op,
  real process.kill probe smoke test, pid<=0 rejection.
- Full suite: 599 pass / 0 fail (was 590 baseline).
- tsc --noEmit clean.

Version: 0.30.52 -> 0.30.53.

Co-authored-by: Evan Nadeau <1878498+evannadeau@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
SpawnBox-dev pushed a commit that referenced this pull request Aug 8, 2026
…e vector (0.46.0)

Semantic search did not work on this corpus, and it is not clear it ever has.
Measured 2026-08-08 against the live 7148-note KB.

THE EVIDENCE. Five paraphrase probes - same idea, deliberately different
vocabulary - ranked the correct note #205, #967, #1206, #1688 and #3247 out of
7148. The control, querying by each note's OWN words, returned #1 every time,
so the data was findable and only meaning-matching was broken. That is the one
capability a 1.8GB embedding model exists to provide.

WHAT IT WAS NOT. The pipeline is correct: self-retrieval returns rank #1 at
score 1.000 and stored vectors are unit-normalised. Anisotropy was the obvious
suspect - unrelated notes scored 0.73-0.79 cosine against each other, where
healthy is ~0.1-0.3 - so mean-centering was tried, the standard fix. It
repaired the score collapse (0.79 -> 0.33) and made every single rank WORSE
(mean 1463 -> 2678). Hypothesis refuted and recorded as such.

WHAT IT WAS, in two parts.

1. TRUNCATION. sidecar/embed_server.py caps the tokenizer at 512 tokens
   (~2000 chars) while 3402 of 7148 notes are longer; the largest note had
   ~97% of its text discarded before its vector was computed. bge-m3 supports
   8192 tokens, so the model was chosen for long context and configured to use
   6% of it.

2. DILUTION, which is the bigger half. Even VERBATIM text from inside the
   window failed to retrieve its own note (ranks #17-#1961). One 1024-dim
   vector cannot represent a 5000-word note covering twenty separate claims;
   every specific idea averages into the document mean. That also explains the
   0.73-0.79 similarity between unrelated notes - all document averages look
   alike.

THE FIX. Embed PASSAGES. A note is split into ~1500-char overlapping chunks
(mcp/engine/chunking.ts) and each is embedded into the new note_chunks table; a
note's vector score becomes its BEST-matching chunk. Max-pool, not mean -
averaging would re-introduce the dilution being fixed.

Chunks sit comfortably inside the existing 512-token window, so this needed no
change to the Python sidecar; the truncation limit simply stops mattering.

VALIDATED BEFORE SHIPPING. Controlled A/B on a 300-note subset, same model,
same probes, only the representation varying: chunked max-pool won 5/5 probes,
mean rank 63.2 -> 32.0. Honest limits: the probes are adversarial (near-zero
lexical overlap), the remaining ranks are #8-#98 of 300 rather than top-3, and
the vector leg is RRF-fused with BM25 in practice, so it contributes rather
than dominates. This is a real improvement, not a silver bullet.

MIGRATION IS DELIBERATE, NOT AUTOMATIC. New and edited notes are chunked on
write. Existing notes get chunks via backfillChunks(), which NOTHING calls on
its own - a test asserts server.ts never references it. That is the direct
lesson of 0.44.0/0.45.1, where a sweep wired into startup halted this machine
twice in one afternoon. It is resumable by construction (population = notes
with no chunk rows, recomputed per call), takes an optional limit so it can be
done in sessions, and processes longest notes first so an interrupted run still
delivers the biggest wins. Until a note is backfilled it keeps scoring off its
note-level vector, so search degrades rather than dropping notes.

`embeddings` is intentionally retained: the near-duplicate gate and auto-linker
ask a whole-document question, which is what that vector answers.

Guards: tests/engine/chunking.test.ts (8), tests/engine/chunked-retrieval.test.ts
(13), including wiring assertions for max-pool, the no-chunks fallback, and the
opt-in-only property. Suite 1082 pass / 0 fail.

Also fixes a test fixture that returned one vector regardless of input - it
only worked while embedIfAvailable sent exactly one text.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MD4kPkrZLWbUxe4arhdwii
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant