[factory] publishPullRequest never creates the head ref on the cloud path, then 422-loops forever and pins the only batch slot

[khaliqgant]

Summary

On the deployed cloud writeback path, publishPullRequest never creates the head
ref, then opens a PR against it. GitHub answers 422 Validation Failed, the retry
is unbounded, the dispatch lifecycle never reaches a terminal phase, and with
batchSize: 1 the pinned slot stops the entire factory.

This is the live cause of the current outage: candidates: 20, dispatched: 0, skipped: 20 while /healthz reports ok: true.

The code

src/mount/relayfile-github-connection-write.ts (confirmed identical in the
deployed @agent-relay/factory@0.1.83 dist/):

const headSha = input.headSha ?? (input.clonePath && !input.headRef
  ? await this.#gitValue(['-C', input.clonePath, 'rev-parse', 'HEAD'], 'HEAD commit')
  : undefined)
const draftName = githubDraftName(headRef, headSha)
// ...
// A remote implementer already pushed its branch. For the legacy local-clone
// path, create the branch before opening the PR.
if (headSha) {
  await this.#writeAndConfirm(createRefPath, { ref: fullHeadRef, sha: headSha })
  // ...
}
const pullRequestPath = `${repoRoot}/pull-requests/${draftName}.json`
await this.#writeAndConfirm(pullRequestPath, { head: headRef, base: input.baseRef, ... })

A remote implementer supplies headRef and no headSha. So !input.headRef is
false, the ternary yields undefined, headSha is undefined, and if (headSha)
does not run. Ref creation is skipped in exactly the case it is needed, and the PR is
then opened against a branch GitHub has never seen.

The ?? 'pushed' sentinel in githubDraftName means the failure signs its own
artifact — every wedged writeback path ends in -pushed.json, which decodes as
"no sha was available".

Measured, from /evidence at 2026-09-02T09:15Z

[factory] durable dispatch lifecycle retry failed {"issue":"186","error":
  "Writeback operation failed for /github/repos/AgentWorkforce/factory/pull-requests/
   factory-factory-186-agentworkforce-factory-808843c4-pushed.json:
   GitHub writeback failed with status 422: Validation Failed"}

git ls-remote confirms factory/186-agentworkforce-factory-808843c4 does not
exist on origin. Six work units were starved behind that one row, all naming it:

{"issue":"427","attempts":275,"waitedMs":8227919,"batchSize":1,"occupiedBy":["186"]}
{"issue":"417","attempts":274,"waitedMs":8228052,"batchSize":1,"occupiedBy":["186"]}
{"issue":"416","attempts":274,"waitedMs":8228329,"batchSize":1,"occupiedBy":["186"]}
{"issue":"394","attempts":274,"waitedMs":8228788,"batchSize":1,"occupiedBy":["186"]}
{"issue":"413","attempts":274,"waitedMs":8228632,"batchSize":1,"occupiedBy":["186"]}
{"issue":"412","attempts":83, "waitedMs":2339996,"batchSize":1,"occupiedBy":["186"]}

2.3 h on that boot alone; the row survived container replacement, so the true age is
longer. The newest factory/* branch in this repo is factory/273, dated
2026-08-16 — no dispatch has produced a branch in over two weeks.

Contrast: the sibling path does not have this hole

src/writeback/github.ts (the gh-CLI writeback) does both of the things the
relayfile path omits:

const headSha = input.headSha ?? (input.clonePath          // no `&& !input.headRef`
  ? await this.#gitValue([...'rev-parse', 'HEAD'], 'HEAD commit')
  : undefined)

if (input.clonePath && !input.headRef) {
  await this.#git(['-C', input.clonePath, 'push', 'origin', `HEAD:refs/heads/${headRef}`])
}

The relayfile path has no push at all — the gated createRefPath writeback was
the only mechanism that could put the branch on GitHub, and it is gated on the one
value that is never present when it is needed.

Two defects, please treat separately

1. The ref is never created (this issue). Whatever the upstream cause, the
   publish step must not open a PR against a head it has not confirmed exists.
   It should confirm the ref and fail loudly and terminally if it cannot,
   rather than 422-looping.

2. A deterministic writeback failure is retried forever. A 422 Validation Failed is not transient — no number of retries fixes an absent head ref — yet
   the lifecycle retry has no dead-letter, so one poisoned row pins the batch slot
   indefinitely. #419/#423/#429 address the reaper; this is upstream of the
   reaper, because the row is running and legitimately owned the whole time. A
   non-retryable writeback status should terminalise the lifecycle and release the
   slot.

Open question for whoever picks this up

Whether the implementer should have pushed and could not (no authenticated
GitHub actor in the sandbox — the "C5 gap"), or whether it is never asked to push
and the orchestrator was always meant to create the ref. The 422 proves the ref is
absent; it does not by itself say which half is wrong. Both readings still require
fix (1): the publish step must verify rather than assume.

Repro

Dispatch any issue through the deployed cloud path with github.identity: "app".
The lifecycle reaches running, the implementer reports a headRef, and the
publish writeback 422s on .../pull-requests/factory-<branch>-pushed.json forever.

---

Found while driving one issue end-to-end from lane/cloud-ensure-fix-r1.
Slot unpinned in production by clearing factory:186 via reset-stuck-dispatches
(run 33614083863) and removing its factory-ready label; see #186 for that note.

[khaliqgant]

Sharpening: on the remote path the ref-creation branch is unreachable by construction

I traced the only call site. src/orchestrator/factory.ts (#publishGithubPullRequest,
~L11380-11425 on main) is the sole caller of publisher.publishPullRequest:

const remoteBranch = implementer.result?.locality === 'remote' && implementer.spec.branch
  ? implementer.spec.branch
  : undefined
if (!remoteBranch && !implementer.spec.clonePath) {
  throw new Error(`GitHub PR publication requires a pushed branch or configured clone path for ...`)
}
// ...
const result = await publisher.publishPullRequest({
  repo,
  ...(remoteBranch ? { headRef: remoteBranch } : { clonePath: implementer.spec.clonePath }),
  expectedHeadRef,
  baseRef,
  title, body,
  ...(implementer.sessionRef ? { sessionRef: implementer.sessionRef } : {}),
})

Two facts that close the argument:

1. headSha is never passed. There is no headSha key in the object literal, and this
   is the only call site in the codebase.
2. headRef and clonePath are mutually exclusive — the spread picks exactly one.

So on the remote path (headRef set, clonePath absent), inside
relayfile-github-connection-write.ts:

const headSha = input.headSha ?? (input.clonePath && !input.headRef ? ...rev-parse... : undefined)
//              ^ never passed      ^ undefined

headSha is undefined unconditionally — not "usually", not "when the implementer
omits it". It is therefore not merely that the && !input.headRef guard is wrong; even
with that clause removed, input.clonePath is absent on the remote path and the ternary
still yields undefined. if (headSha) { create ref } is dead code whenever the
implementer is remote, which is every cloud dispatch.

That makes the load-bearing line the comment, not the condition:

// A remote implementer already pushed its branch.

The publish step has no mechanism to create the ref on the remote path and no check that
it exists. It is a pure assumption, and when it is false the result is a deterministic
422 retried without bound.

Consequence for the fix

Removing && !input.headRef does not fix this — there is no clone to rev-parse and
no sha to write. The remote path needs one of:

• the implementer to report its head sha, so the existing createRefPath writeback can
  run (and then the guard genuinely does need widening); or
• an explicit existence check on refs/heads/<headRef> before the PR create, failing
  terminally rather than 422-looping; or
• the sandbox to actually be able to push (the C5 gap), which is the upstream half.

The first two are worth doing regardless of C5, because they convert a silent
factory-wide stall into a single failed dispatch that releases its slot.

Live confirmation

After clearing the #186 row the slot freed and a fresh dispatch ran through
claim → sandbox → agent placement (placedAgents 0 → 1) within ~90 s, so every stage
before publish is healthy. I am reporting where that dispatch ends separately.

[khaliqgant]

The open question is settled: the sandbox genuinely cannot push (AgentWorkforce/cloud#3220)

I left this above as undecided — whether the implementer should have pushed and could
not, or was never asked to. It is the former, and it is measured, not inferred.
cloud#3220 records it:

Gate 0 run b7e5deaa proved none of that can work: C5-dev-deploy-authority FAILED
because the sandbox has no authenticated gh. The same auth boundary means the
sandbox cannot git push either […]

gh is installed in the Daytona snapshot (#3163) but deliberately uncredentialed.
agent-relay cloud run exposes no --env or --secret flag, so there is no
submit-time path to inject a credential either.

So the two halves interlock exactly:

1. The sandbox cannot push → the branch and its commits never reach GitHub (cloud#3220 / C5).
2. publishPullRequest assumes it did and cannot create the ref either → PR opened
   against a non-existent head → deterministic 422 (this issue).
3. The 422 is retried without bound → the slot is pinned → the whole factory stops.

This narrows the fix, and it is worth being blunt about the consequence

Fix (2) from the body — confirm the ref exists, fail terminally instead of 422-looping —
is still worth doing and is the one that stops a single bad dispatch from taking the
factory down with it. But on its own it does not produce a branch. It converts an
infinite loop into a clean, slot-releasing failure. That is a large operational win and
zero progress on actually shipping work.

Fix (1) as originally written — "let the existing createRefPath writeback run" — is
not sufficient either, and I want to correct my own framing above. Creating a ref
requires a sha that already exists on GitHub. If the sandbox cannot push, the commit
objects are not there, so there is no sha to point a ref at. Supplying headSha from the
implementer would not help; the objects are the missing thing, not the pointer.

Therefore: no cloud dispatch can produce a branch until C5 is settled. Everything in
this repo is downstream of that. The nearest thing to a Factory-side fix is to make the
failure honest and cheap instead of silent and total.

Suggested split

• Here (factory), shippable now: treat a non-retryable writeback status (422, and
  the 4xx payload-shape rejections like [factory] Orphaned lifecycle label can never be cleared: labels payload must be a non-empty string array #411) as terminal for the lifecycle. Release the
  slot, mark the dispatch failed, surface the reason. Stops one poisoned row from being a
  factory-wide outage.
• Also here: verify refs/heads/<headRef> exists before opening the PR, so the error
  names the real cause ("implementer branch was never pushed") instead of a bare
  422 Validation Failed that reads like a payload bug.
• cloud#3220 / factory#228: the actual path to a branch. See my note on cloud#3220 —
  the app-actor write path is currently returning 422, not 403, on this repo, which is
  a negative result for the narrow-repositories hypothesis that issue is waiting on.