CI performance, resilience & prebuilt distribution - #6
Merged
Conversation
…lias fallback Three opt-in improvements aimed at CI adoption, none changing default behavior: - runner: add Options.Concurrency and `augur run -concurrency/-j N` to run agent invocations in parallel. Each invocation's stdout/stderr is buffered and flushed as one block so parallel output never interleaves; the Summary is assembled in deterministic task order regardless of completion order. Defaults to 1 (the original strictly-sequential path is unchanged). The proxy's per-(scenario,run) state, the trace writer, and the cassette are already mutex-guarded, so parallel invocations record correctly. - proxy: replace the hardcoded 10m client timeout with a per-request context deadline governed by Server.Timeout (flag -timeout on run/proxy) and overridable per call via the X-Augur-Timeout header, for slow reasoning models. Deriving the deadline from the inbound request context keeps client cancellation propagating to the upstream call (no orphaned tokens). -timeout 0 disables the deadline. - cost: add opt-in Pricing.PrefixFallback + Resolve, matching a model absent from the snapshot to its longest dash-delimited prefix (gpt-4o-2024-08-06 -> gpt-4o) with a dash-boundary rule so a base name can't swallow an unrelated one. aggregate surfaces every fallback in Result.ModelAliases; `-normalize-models` on aggregate/gate enables it and warns per normalized model so it is never silent. Off by default: an un-priced call is still a hard ErrUnknownModel. Tests: runner concurrency (real overlap, ordering, non-interleaved output, abort/continue-on-error), proxy timeout (fires, header override, header stripped), cost Resolve (exact/fallback/longest-match/dash-boundary), aggregate alias surfacing. README: new "Tuning for CI" section. Full suite green.
Ship augur as prebuilt, checksum-verified binaries so agent repos (Python/TS/ Node) run the cost gate without a Go toolchain in CI. - .goreleaser.yaml: cross-compile linux/darwin/windows x amd64/arm64 (static, CGO disabled), stable version-free archive names (augur_<os>_<arch>.tar.gz), sha256 checksums.txt, and a GitHub release per v* tag. Build metadata is injected via -ldflags into main.version/commit/date. - version_cmd.go: `augur version` reports the injected build info (from-source builds honestly say "dev"); it's also the handle the Action uses to confirm a downloaded binary runs. Registered in the dispatch table + usage order. - .github/workflows/release.yml: on a v* tag, run tests then GoReleaser (release --clean) to publish the binaries. CI gains a goreleaser-check job so the config is validated on every push. - action.yml: new "Resolve augur binary" step downloads the release asset matching the pinned tag (github.action_ref) and runner os/arch, verifies its sha256 against checksums.txt (tolerant of coreutils/BSD format variants), and extracts it. Falls back to the existing source build for branch/SHA pins, unbuilt platforms, or a download failure. New `version` input forces a tag or `source`. The from-source path is unchanged, just gated behind need-build. - README: document prebuilt distribution, the version input, and how to cut a release. .gitignore: ignore GoReleaser's /dist/. The download/checksum/os-arch/version-resolution logic was simulated locally (valid + corrupt archive, binary- and two-space checksum formats, tag/branch/ source refs). GoReleaser config is schema-checked in CI. Full suite green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five opt-in improvements aimed at CI adoption and lower resource use. None change default behavior. Two commits:
1. CI perf & resilience (
b5bf69d)augur run -concurrency N(alias-j N) runs agent invocations in parallel through the in-process proxy. Per-invocation stdout/stderr is buffered and flushed as a block (no interleaving); theSummaryis assembled in deterministic task order. Defaults to1(the original sequential path is untouched). Proxy state, trace writer and cassette are already mutex-guarded.-timeoutonrun/proxy), overridable per call via theX-Augur-Timeoutheader for slow reasoning models (o1/o3/r1). Deriving the deadline from the inbound request context keeps client cancellation propagating to the upstream call (no orphaned tokens).-timeout 0disables it.--normalize-modelsonaggregate/gateprices a model absent from the snapshot via its longest dash-delimited prefix (gpt-4o-2024-08-06→gpt-4o), with a dash-boundary rule so a base name can't swallow an unrelated one. Off by default (an un-priced call is still a hardErrUnknownModel); every fallback is surfaced inResult.ModelAliasesand warned to stderr, so it's never silent.2. Prebuilt distribution (
a7fef9d)augur_<os>_<arch>.tar.gz+checksums.txtto a GitHub release perv*tag. Build metadata is injected via-ldflags.augur versionreports the injected build info (from-source builds saydev).action.ymlnow downloads the release binary matching the pinned tag (github.action_ref) and runner os/arch, sha256-verifies it, and extracts it — so Python/TS/Node agent repos run the gate without a Go toolchain. Transparent source-build fallback for branch/SHA pins, unbuilt platforms, or download failures; newversioninput forces a tag orsource.release.yml) onv*tags; CI gains agoreleaser-checkjob validating the config on every push.Not included (from the original proposal)
Deliberately skipped with rationale: context-cancellation propagation was already implemented (
proxy.gousedNewRequestWithContext(r.Context())); theio.ReadAll→LimitReaderstreaming-hash idea rests on an inaccurate premise (the body must be buffered in full to forward and record it);internal/restructure and OpenTelemetry ingest are larger, lower-ROI items left for follow-ups.Verification
go test ./...green; new tests for runner concurrency (real overlap, ordering, non-interleaved output, abort/continue-on-error), proxy timeout (fires, header override, header stripped),cost.Resolve(exact/fallback/longest-match/dash-boundary), aggregate alias surfacing, and theversioncommand.🤖 Generated with Claude Code