feat(cloud-agent-next): report pre-dispatch failures durably - #6069
Open
eshurakov wants to merge 2 commits into
Open
feat(cloud-agent-next): report pre-dispatch failures durably#6069eshurakov wants to merge 2 commits into
eshurakov wants to merge 2 commits into
Conversation
A message admitted through the control-plane API could fail during preparation, attachment, or dispatch without ever appearing in cloud_agent_session_runs, because only the legacy DO emitted run.state reports and its emitter swallowed the send. Operators could not identify the accepted session/message that had failed before the wrapper started. Add a durable, alarm-driven report obligation to the control-plane SandboxSession DO and commit it in the same transaction as admission, the accepted/terminal common commit, and deletion snapshots. The obligation carries the latest cumulative lifecycle snapshot and is delivered through the existing report queue; a bounded outbox handles retry, capacity, and expiry without changing dispatch or alarm timing or failing a healthy run. Failures are classified from bounded cause facts at the committing transition, observed acceptance is never fabricated, and a trusted first-admission anchor lets the consumer create the reporting parent for empty/worktree chats. Report snapshots and alarms are preserved through both deletion purge paths. Extend focused queue/lifecycle, report-store, report-consumer, and control-plane failure tests, plus a real DO reporting integration test.
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Contributor
Code Review SummaryStatus: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)SUGGESTION
Files Reviewed (7 files)
Fix these issues in Kilo Cloud Previous Review Summary (commit a1762eb)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit a1762eb)Status: 3 Issues Found | Recommendation: Address before merge Executive Summary
Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (18 files)
Reviewed by deepseek-v4.1-flash · Input: 0 · Output: 0 · Cached: 0 Review guidance: REVIEW.md from base branch |
… post-dispatch An accepted run that failed as provider_unknown was reported as pre_dispatch/sandbox_connect_failed while also carrying dispatchAcceptedAt. Branch on the observed dispatch state like the sibling reasons, so an accepted run reports post_dispatch_no_activity/wrapper_disconnected. Also drop a tautological assertion in the reports integration test and apply oxfmt to the files CI flagged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An accepted control-plane message could fail during preparation, attachment, or dispatch and never appear in
cloud_agent_session_runs: only the legacyCloudAgentSessionDO emittedrun.statereports, and its emitter swallowed queue-send failures. Operators could not identify the session/message that failed before the wrapper started.This adds a durable, alarm-driven report obligation to the control-plane
SandboxSessionDO. The obligation is committed in the same DO transaction as admission, the accepted/terminal common commit, and deletion snapshots, and carries the latest cumulative lifecycle snapshot (identity, queued/accepted/terminal timestamps, evidence-based classification). Delivery reuses the existing report queue andcloud_agent_session_runsprojection; a bounded outbox handles retry, capacity, and expiry without changing dispatch/alarm timing or failing a healthy run. A trusted first-admission anchor lets the consumer create the reporting parent for empty/worktree chats, and report snapshots/alarms survive both deletion purge paths.Key design points:
telemetry/control-plane-failure.ts); the classifying transition owns the decision. Wrapper/operation-result failure reasons are untrusted text and fall through tounknown/unclassified.dispatchAcceptedAtis never fabricated fromapplyMessageOutcome's completion-timeacceptedAt.Verification
Manual E2E against a real
wrangler devworker (restarted after the last edit), the in-process report queue consumer, and real Postgres:POST /trpc/startforworkspace_355fb352…/msg_08d5046a…with the sandbox killed during preparation.cloud_agent_session_runsshowedstatus=failed,queued_at/terminal_atset,dispatch_accepted_atNULL,failure_stage=pre_dispatch,failure_code=sandbox_connect_failed,failure_responsibility=platform,failure_reason=sandbox_connectivity, and a bounded diagnostic with future expiry. Reproduced on a second session.completedrow withdispatch_accepted_atpopulated.cloud_agent_sessionsrow on first admission, withcreated_at= runqueued_atandinitial_message_id= the first message.listHealthErrorSessionsreturned both failed sessions with the classification.Automated (not manual):
worker-utils662 passed, cloud-agent-next focused unit 467 passed, full integration 664 passed, typecheck and lint clean.Note: the
missing_metadatavariant is unreachable through the real create API, so the pre-dispatch path was exercised viaenvironment_failed → sandbox_connect_failed.Visual Changes
N/A
Reviewer Notes
report-storeidentity/conflict behavior is covered with a faked DB chain, not real SQL.