Skip to content

fix(agent): unify live model retry budget - #2211

Merged
bobleer merged 3 commits into
GCWing:mainfrom
bobleer:bob/unify-live-agent-error-retries
Aug 11, 2026
Merged

fix(agent): unify live model retry budget#2211
bobleer merged 3 commits into
GCWing:mainfrom
bobleer:bob/unify-live-agent-error-retries

Conversation

@bobleer

@bobleer bobleer commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Make the live Agent round executor own one bounded 10-attempt retry budget across request opening and stream processing.
  • Retry every model failure until that budget is exhausted, including HTTP 4xx/5xx responses, TTFT and idle timeouts, malformed SSE/schema errors, structured provider errors, context overflow, and failures after partial output. Cancellation remains immediate.
  • Preserve Retry-After only as a backoff hint and prevent adapter retries from multiplying the live Agent budget to 100 requests.
  • Add CLI coverage for malformed-SSE recovery and assert that disconnect plus persistent provider failure totals exactly 10 requests.
  • Harden macOS dispatch cancellation against zombie leaders after the first CI run exposed that existing race.

Type and Areas

Type: regression fix

Areas: Rust core, AI adapters, CLI runtime/tests

Motivation / Impact

Live sessions could terminate immediately for error categories excluded by RoundExecutor string/category checks even though BitFun already had a retry mechanism. At the same time, transport retries could nest with round retries and produce up to 10 x 10 attempts. The live Agent now gives every model error the same single retry budget, improving task completion without unbounded or multiplied requests.

The first CI run also showed that macOS kill(pid, 0) treats an unreaped zombie dispatch leader as present. That could incorrectly authorize SIGKILL escalation after TERM; the guard now requires a non-zombie process state.

Verification

  • cargo test -p bitfun-core-types (32 tests passed across unit/integration targets)
  • cargo test -p bitfun-ai-adapters (221 unit tests plus adapter integration tests passed)
  • cargo test -p bitfun-agent-stream (66 tests passed)
  • cargo test -p bitfun-core --no-default-features --features agent-runtime --lib agentic::execution::round_executor::tests (18 tests passed)
  • cargo test -p bitfun-cli --test cli_command_contracts stream_json_malformed_sse_retries_then_completes -- --nocapture
  • cargo test -p bitfun-cli --test cli_command_contracts stream_json_provider_ -- --nocapture
  • cargo test -p bitfun-cli --test cli_command_contracts stream_json_disconnect_then_exhausted_retry_failure_emits_one_error_terminal -- --nocapture
  • cargo test -p bitfun-cli --bin bitfun dispatch::runner::tests -- --nocapture (6 tests passed)
  • The formerly failing macOS cancellation test passed 20 additional consecutive direct runs.
  • node scripts/check-core-boundaries.mjs
  • pnpm run fmt:rs
  • git diff --check

Reviewer Notes

AiProviderError gains an optional retryAfterMs transport hint. It is backward-compatible on the wire and does not affect retry admission. This PR was AI-assisted and is fully tested at the focused unit, adapter-integration, runtime, and CLI end-to-end levels.

Checklist

  • This PR is focused and does not include secrets, temporary prompts, generated scratch files, or unrelated artifacts.
  • Relevant verification is recorded above, or skipped checks are explained.
  • User-facing strings, docs, and locales are updated where applicable (no user-facing copy changed).

@bobleer

bobleer commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

CI follow-up: the first run exposed a real macOS-only zombie-process race in an existing dispatch cancellation guard. kill(pid, 0) reports an unreaped zombie as present, so the guard could incorrectly authorize SIGKILL escalation after TERM. Commit 201ad29 now requires a non-zombie macOS process state before escalation. The formerly failing test passed once through Cargo, 20 additional consecutive direct runs, and the full six-test dispatch::runner module locally. A fresh CI run has been triggered.

@bobleer
bobleer merged commit 3911dc7 into GCWing:main Aug 11, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant