test: evaluate gpt-5.6-sol for Bonk (do not merge) - #150
Closed
andre-j3sus wants to merge 8 commits into
Closed
Conversation
Bonk has failed on every run since 2026-08-18; the last success was
2026-07-31. Each run creates a session, sends the message, logs a single
"loop { step: 0 }", then emits zero tokens and exits 1 -- the provider never
initializes.
bonk.yml has not changed since it was added, and cloudflare/workers-sdk
fails identically while pinning a March action SHA, so neither this repo's
config nor the action code is the cause. What both broken repos share is the
1.15.13 pin. ask-bonk#226 tracked the same provider-initialization failures
on 2026-08-18 and verified a working run on 1.18.18.
Keeping the version pinned rather than moving to latest, since floating is
what produced this drift in the first place.
Bumping OpenCode surfaced the error 1.15.13 was swallowing: Model not found: cloudflare-ai-gateway/anthropic/claude-opus-4-8. Did you mean: anthropic/claude-opus-4.5, anthropic/claude-opus-4.6, anthropic/claude-opus-4.7? The version component is dot-separated, and 4.8 does not exist. Moving to 4.7, the newest the gateway offers.
Fixing the model id got past "Model not found" and reached the provider,
which then returned:
AI_APICallError: Invalid Anthropic API Key
{"code":"authentication_error","message":"Invalid Anthropic API Key"}
The three CF_AI_GATEWAY_* secrets are fine -- they authenticate to the
gateway. What is invalid is the gateway's upstream Anthropic key, so every
anthropic/* model fails the same way regardless of id. That also explains
why cloudflare-docs' bigbonk.yml (anthropic) has never once executed while
its bonk.yml (Workers AI) is green.
Adopting that working pair verbatim: kimi-k2.6 over Workers AI, which needs
no Anthropic credential, on the 1.17.7 they run it against.
kimi-k2.6 came from copying the cloudflare-docs config. kimi-k2.7-code is the code-optimized variant of the same family, at identical input and output pricing: +21.8% on Kimi Code Bench v2, +11.0% on Program Bench, and 30% fewer reasoning tokens. Cached input is $0.19/M against $0.16/M, which the token reduction should more than cover. Upstream states API usage is unchanged, and k2.6 is not deprecated, so reverting is a one-token change if this regresses.
Records the two things that are not evident from the workflow file: the Anthropic provider is unusable through this gateway, and comment-triggered runs execute main's copy of the workflow, so changes to it cannot be verified with /bonk on a PR.
k2.7-code wins on coding benchmarks but does not follow the review prompt. Measured on this PR: k2.6 2-line diff 10 loops full structured review k2.7-code 2-line diff 5 loops "LGTM!" k2.7-code 20-line diff 7 loops "LGTM!" The prompt asks it to summarize the change and flag issues; k2.7-code skips the summary regardless of diff size. Its advertised 30% reduction in reasoning tokens appears to cost exactly the analysis we want. Benchmarks measured coding ability, not instruction-following, and the latter is what a review bot needs.
Test branch, not for merge. Isolates the model: everything else matches ajesus/fix-bonk-opencode-version, so the review this produces is directly comparable to the kimi-k2.6 one on #149. variant is deliberately left unset even though workerd and cloudflare-os pair GPT with variant: high, so reasoning effort stays a separate axis to test rather than a confound here.
|
APIError: Function tools with reasoning_effort are not supported for gpt-5.6-sol in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'. |
2 similar comments
|
APIError: Function tools with reasoning_effort are not supported for gpt-5.6-sol in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'. |
|
APIError: Function tools with reasoning_effort are not supported for gpt-5.6-sol in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'. |
|
@andre-j3sus Bonk workflow failed. Check the logs for details. View workflow run · To retry, trigger Bonk again. |
Replaces gpt-5.6-sol, which returns 400 for function tools whenever reasoning_effort is set on /v1/chat/completions. That combination is fatal for a reviewer that has to read files, and explains why sol has no working run anywhere: workerd's bigbonk is 0 for 100, and cloudflare-os's successes are all prefilter no-ops that never reach the model. terra with variant: high on the dev build is the one GPT pairing with a real review behind it. Also adopts workerd's OPENCODE_CONFIG_CONTENT guardrails, denying interactive questions and doom loops. The action is pinned to current main rather than workerd's SHA, which is nine days behind what @main already resolved to here. Push rights stay NO_PUSH.
Collaborator
Author
|
Superseded: sol is unusable (400 on function tools with reasoning_effort). Reopening on terra to fire the opened trigger. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test PR, not for merge — #149 is the real change and stays open.
#149 revealed that the AI Gateway's Anthropic key is invalid, so Bonk moved to
Workers AI on
kimi-k2.6. The gateway's OpenAI key does work, though: workerd,cf, workers-py, cloudflare-os and vinext all review through
cloudflare-ai-gateway/openai/gpt-5.6-*, using the sameCF_AI_GATEWAY_*secret names and OIDC endpoint we do.
This branch is #149 with one line changed, the model, so the review it produces
is directly comparable to the
kimi-k2.6one on #149.variantstays unseteven though workerd and cloudflare-os pair GPT with
variant: high— reasoningeffort is worth testing, but as its own axis rather than a confound here.
Close once we have picked a model.