startMount: two unwrapped execs make a transport rejection unclassifiable (100% JIT provision failure in prod)

[khaliqgant]

Impact

Every forceProvision JIT sandbox provision through agentrelay.com is currently failing at the Relayfile mount. Measured 5/5 against production today:

5/5  HTTP 503  mountElapsedMs 112143 / 131990 / 122960 / 133527 / 117733
     phase: relayfile_mount_invocation_unknown

With the mount disabled (mountRelayfile:false), the same route is 5/5 HTTP 201 in 21–28s. The mount is the sole blocker. This has been blocking factory-cloud dispatch end-to-end for ~24h.

What the timings show

Production cloud-web-worker logs, one representative request:

phase	elapsed
sandbox_provision (Daytona create)	1.34s
cli_credential_setup	9.19s
enrollment_token_mint	1.40s
relayfile_token_mint	0.76s
relayfile_credentials_seed	0.19s
startMount	122.0s → throw

Daytona itself is fast. The credentials seed completes at mountElapsedMs 951, so the ~121s hang begins essentially immediately inside startMount.

The defect

orchestrator.js#startMount in 0.1.12 wraps some of its execs so a rejection becomes a classified error, but not all:

exec	wrapped?	on rejection
mkdir mount path	yes (Failed to create relayfile mount path:)	classified
initial-sync launch	no	unclassified
status probe (poll loop)	yes (Failed to check relayfile initial sync status:)	classified
daemon start	no	unclassified

Every throw that embeds sandbox command output is one the caller's classifier recognises. So an unclassified error can only have come from one of those two unwrapped runScript calls rejecting — and ~122s matches the ~120s Daytona proxy read timeout your own comment in this file cites:

// The initial sync can outlive any single exec (Daytona's proxy read
// timeout is ~120s and callers wrap execs in client-side fail-fasts), so
// it runs detached in the sandbox ...

The detached launch shell itself looks correct (nohup sh -c '<runner>' >/dev/null 2>&1 & echo $! — fds redirected, setsid where available), which is why this needs the cause to be readable before guessing further.

Asks

1. Wrap the two remaining execs the way mkdir and the status probe already are, so a transport rejection lands on a named phase (relayfile_mount_*_not_invoked / _invoked_exited) instead of falling through to the caller's unknown bucket. This is the cheap part and it is what makes the next failure self-describing.
2. Then determine which of the two is hanging and why. Given the launch shell detaches correctly, the daemon start (buildRelayfileMountStartShell) is worth looking at first — a foreground daemon that never closes its exec channel would hang exactly this way.

Consumer-side context: cloud discards the cause on this path today, which is why it read as opaque. AgentWorkforce/cloud#3244 fixes cloud's half so the cause is logged; this issue is the actual fault.

Deliberately filed without the factory readiness label — this needs a human to pick the approach, not an auto-dispatch.

Reported by cloud-ensure-fix-r1.

[khaliqgant]

Correction: the root cause in the issue body above is wrong

I filed this inferring a transport rejection on one of the two unwrapped execs, from the ~122s timing matching Daytona's ~120s proxy read timeout. That was inference, not measurement, and it was wrong.

AgentWorkforce/cloud#3244 shipped the missing diagnostic and production immediately reported the real cause:

phase:       relayfile_mount_invocation_unknown
causeName:   Error
causeDetail: Relayfile initial sync paused before complete readiness

The execs return fine. startMount's poll loop sees the sync exit with code 75 (RELAYFILE_INITIAL_SYNC_INCOMPLETE_EXIT_CODE) after ~88–122s of real work, and raises the bare sentence from orchestrator.ts:330.

What actually fails

buildRelayfileMountInitialSyncCompletionGuardShell reads <localDir>/.relay/state.json per exact-layout root and requires both:

state.bootstrap == null
typeof state.lastSuccessfulReconcileAt === "string" && state.lastSuccessfulReconcileAt.trim().length > 0

Any root failing either predicate exits 75. So the guard is working as designed — relayfile is checkpointing a bootstrap that never reaches a completed public reconcile.

Probable mechanism, flagged as a hypothesis rather than a finding: AgentWorkforce/relayfile#412 documents that this exact .relay/state.json has two independent writers emitting disjoint schemas to the same path, overwriting each other ~1/sec. A guard keying on lastSuccessfulReconcileAt would then pass or fail depending on which writer wrote last. What would confirm it: read .relay/state.json out of a sandbox that just exited 75 and see which schema is present. I have not done that, so I am not asserting it.

Two things this changes

1. PR fix(mount): classify transport rejections from the launch and daemon-start execs #46 does not fix this. It is correct hardening for the transport case and I stand by it, but the execs are not where this dies. Nobody should wait on fix(mount): classify transport rejections from the launch and daemon-start execs #46 expecting this path to clear.

2. Exit 75 is sysexits.h TEMPFAIL — this package's own comment says "the checkpoint is resumable on the same sandbox." But the consumer treats it as terminal: cloud's required-mount path tears the whole sandbox down and returns 503. A resumable checkpoint being handled as a fatal error is arguably its own defect, and it is worth deciding whether startMount should resume/retry the sync rather than throwing.

Cloud-side follow-ups: AgentWorkforce/cloud#3248 maps the bare message to a named phase so this stops reporting as "unknown".

Leaving this issue open, retitled in spirit: the hang is a sync that exits incomplete, not a wedged exec.