fix: probe a Provider's reviewed chat model, not the first ID it lists - #95
Merged
Merged
Conversation
Closes #93. With an empty probe model the connection test used the live /v1/models listing outright, so whatever an aggregator returned first became the model that got a chat payload. The catalogues of the Providers shipped by default are mostly generators: a t2v model rejecting a chat request produced a failure the user read as "my key is broken", during first-run setup. resolveProviderModel now prefers the Provider's manifest model when the endpoint actually serves it. That value is a reviewed, known-chat ID; the previous order reached it only when discovery failed, so the good value was bypassed exactly when the list was available to poison it. PickChatModel remains the last resort, for a custom endpoint or a Provider that no longer serves its manifest model. A model the user typed still wins over both: that field is their override, and a failure on it is the answer they asked for. nonChatModel gains video, audio and music terms — wan, kling, sora, veo, hailuo, seedance, cogvideox, t2v/i2v/v2v, bark, music, voice and others. This is defence in depth rather than the fix: a denylist over third-party IDs always lags, which is why the ordering above matters more. The pattern is tested in both directions, because a term broad enough to catch a generator family can also swallow a real chat model and push the probe onto models[0], reintroducing the bug from the other side. The probe result now reports which model was used and whether OneAgent chose it, and a failure on an auto-selected model names it and points at the override field. Before this the message could not distinguish a rejected key from a model we picked badly — the same text for two unrelated causes. It stays quiet when the key itself was rejected, which is about the key whatever model carried it. Verified the report's classification table against the live regex before changing anything: all ten video and audio IDs it lists were indeed treated as chat models, and PickChatModel did return the t2v model when it came first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #93.
The bug
With an empty probe model, the connection test used the live
/v1/modelslisting outright, so whatever an aggregator returned first became the model that got a chat payload. The catalogues of the Providers we ship by default are mostly generators, so at2vmodel rejecting a chat request produced a failure the user reads as "my key is broken" — during first-run setup, about credentials that are fine.I verified the report's classification table against the live regex before changing anything. All ten video and audio IDs it lists were indeed classified as chat models, and
PickChatModel([t2v, chat])did return thet2vone. The report is accurate throughout.Fix
Ordering, which is the real fix.
resolveProviderModelnow prefers the Provider's manifest model when the endpoint actually serves it. That value is a reviewed, known-chat ID — and the previous order reached it only when discovery failed, so the good value was bypassed exactly when the list was available to poison it.PickChatModelstays as the last resort, for a custom endpoint or a Provider that no longer serves its manifest model. Checking membership in the returned list matters: probing a manifest model the endpoint has dropped would fail for a reason the user cannot act on.A model the user typed still wins over both. That field is their override, and a failure on it is the answer they asked for.
Denylist, as defence in depth.
nonChatModelgainswan,kling,sora,veo,hailuo,seedance,cogvideox,t2v/i2v/v2v,bark,music,voiceand others. Secondary on purpose: a denylist over third-party IDs always lags. It is tested in both directions, because a term broad enough to catch a generator family can also swallow a real chat model and push the probe ontomodels[0]— reintroducing the bug from the other side.cogvideoinitially missedcogvideox-5b; the two-way test caught it.Failure message. The probe result now reports which model was used and whether OneAgent chose it. A failure on an auto-selected model names it and points at the override field. Previously one message covered two unrelated causes. It stays quiet when the key itself was rejected, which is about the key whatever model carried the request.
Testing
go test ./...— all packages pass;go vet ./...cleango test -race ./internal/{app,provider,binding}— cleanpnpm run test— 33 files / 244 tests passpnpm run test:e2e— 5 passedpnpm run build(includestsc --noEmit) — passNew coverage: an app-level test asserting a listing whose first two entries are video models still probes the manifest chat model; one asserting a user-typed model is never replaced (and that discovery does not even run);
PickChatModelagainst all thirteen generator IDs from the report plus ten real chat models; and fourConnectionStatuscases for the new message.Confirmed the two-line failure layout against computed styles in dark mode: icon top-aligned, error text red, explanation in secondary grey at 12px rather than inheriting the alarm colour. Regenerated
frontend/bindingswith the pinned CLI frombuild/Taskfile.yml; the only change is the two new fields.Not done
Issue point 3 ("do not silently overwrite a model the user typed") needed no change —
resolveProviderModelalready returned a non-empty model untouched, and there is now a test pinning it.🤖 Generated with Claude Code