Skip to content

fix: probe a Provider's reviewed chat model, not the first ID it lists - #95

Merged
yujiezhang-ops merged 1 commit into
mainfrom
fix/probe-model-selection
Aug 7, 2026
Merged

yujiezhang-ops merged 1 commit into
mainfrom
fix/probe-model-selection

Conversation

@yujiezhang-ops

Copy link
Copy Markdown
Collaborator

Closes #93.

The bug

With an empty probe model, the connection test used the live /v1/models listing outright, so whatever an aggregator returned first became the model that got a chat payload. The catalogues of the Providers we ship by default are mostly generators, so a t2v model rejecting a chat request produced a failure the user reads as "my key is broken" — during first-run setup, about credentials that are fine.

I verified the report's classification table against the live regex before changing anything. All ten video and audio IDs it lists were indeed classified as chat models, and PickChatModel([t2v, chat]) did return the t2v one. The report is accurate throughout.

Fix

Ordering, which is the real fix. resolveProviderModel now prefers the Provider's manifest model when the endpoint actually serves it. That value is a reviewed, known-chat ID — and the previous order reached it only when discovery failed, so the good value was bypassed exactly when the list was available to poison it. PickChatModel stays as the last resort, for a custom endpoint or a Provider that no longer serves its manifest model. Checking membership in the returned list matters: probing a manifest model the endpoint has dropped would fail for a reason the user cannot act on.

A model the user typed still wins over both. That field is their override, and a failure on it is the answer they asked for.

Denylist, as defence in depth. nonChatModel gains wan, kling, sora, veo, hailuo, seedance, cogvideox, t2v/i2v/v2v, bark, music, voice and others. Secondary on purpose: a denylist over third-party IDs always lags. It is tested in both directions, because a term broad enough to catch a generator family can also swallow a real chat model and push the probe onto models[0] — reintroducing the bug from the other side. cogvideo initially missed cogvideox-5b; the two-way test caught it.

Failure message. The probe result now reports which model was used and whether OneAgent chose it. A failure on an auto-selected model names it and points at the override field. Previously one message covered two unrelated causes. It stays quiet when the key itself was rejected, which is about the key whatever model carried the request.

Testing

  • go test ./... — all packages pass; go vet ./... clean
  • go test -race ./internal/{app,provider,binding} — clean
  • pnpm run test — 33 files / 244 tests pass
  • pnpm run test:e2e — 5 passed
  • pnpm run build (includes tsc --noEmit) — pass

New coverage: an app-level test asserting a listing whose first two entries are video models still probes the manifest chat model; one asserting a user-typed model is never replaced (and that discovery does not even run); PickChatModel against all thirteen generator IDs from the report plus ten real chat models; and four ConnectionStatus cases for the new message.

Confirmed the two-line failure layout against computed styles in dark mode: icon top-aligned, error text red, explanation in secondary grey at 12px rather than inheriting the alarm colour. Regenerated frontend/bindings with the pinned CLI from build/Taskfile.yml; the only change is the two new fields.

Not done

Issue point 3 ("do not silently overwrite a model the user typed") needed no change — resolveProviderModel already returned a non-empty model untouched, and there is now a test pinning it.

🤖 Generated with Claude Code

Closes #93.

With an empty probe model the connection test used the live /v1/models listing
outright, so whatever an aggregator returned first became the model that got a
chat payload. The catalogues of the Providers shipped by default are mostly
generators: a t2v model rejecting a chat request produced a failure the user read
as "my key is broken", during first-run setup.

resolveProviderModel now prefers the Provider's manifest model when the endpoint
actually serves it. That value is a reviewed, known-chat ID; the previous order
reached it only when discovery failed, so the good value was bypassed exactly
when the list was available to poison it. PickChatModel remains the last resort,
for a custom endpoint or a Provider that no longer serves its manifest model. A
model the user typed still wins over both: that field is their override, and a
failure on it is the answer they asked for.

nonChatModel gains video, audio and music terms — wan, kling, sora, veo, hailuo,
seedance, cogvideox, t2v/i2v/v2v, bark, music, voice and others. This is defence
in depth rather than the fix: a denylist over third-party IDs always lags, which
is why the ordering above matters more. The pattern is tested in both directions,
because a term broad enough to catch a generator family can also swallow a real
chat model and push the probe onto models[0], reintroducing the bug from the
other side.

The probe result now reports which model was used and whether OneAgent chose it,
and a failure on an auto-selected model names it and points at the override
field. Before this the message could not distinguish a rejected key from a model
we picked badly — the same text for two unrelated causes. It stays quiet when the
key itself was rejected, which is about the key whatever model carried it.

Verified the report's classification table against the live regex before changing
anything: all ten video and audio IDs it lists were indeed treated as chat
models, and PickChatModel did return the t2v model when it came first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@yujiezhang-ops
yujiezhang-ops merged commit ac0eb4b into main Aug 7, 2026
4 checks passed
@yujiezhang-ops
yujiezhang-ops deleted the fix/probe-model-selection branch August 7, 2026 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Connection probe can auto-select a video or audio model and fail

1 participant