feat: DeepSeek as a supported provider

[babaliauskas]

Adds DeepSeek (deepseek-flash, deepseek-v4-pro) as a first-class provider.

Draft because the live check against the DeepSeek API hasn't run yet (no key was available). The checklist is below.

What changes

• Registry.
  • New provider deepseek, with DEEPSEEK_API_KEY checked before run/compare and shown as a row in doctor.
  • Bare deepseek-* ids resolve to deepseek/…. This is what a capture records when the app calls api.deepseek.com through the OpenAI client; before, LiteLLM couldn't route these ids.
  • The judge-family warning now works for a DeepSeek judge grading a DeepSeek arm.
  • DeepSeek served by another host (azure_ai/, hosted_vllm/, openrouter/) stays other, so it isn't asked for DEEPSEEK_API_KEY.
• Tool calls. Any id containing deepseek is parsed as OpenAI-shaped. Previously every DeepSeek tool eval raised ModelError.
• Pricing. Recorded calls are priced under the provider-prefixed key first. Bare deepseek-flash was priced at $0 before. Side effect: a few niche Gemini keys change price (gemini-exp-1206 now prices at $0, and image-model cache/tier prices differ); text-model prices are unchanged.
• Thinking mode (new models/deepseek.py). Both models think by default. Thinking mode silently ignores temperature, and it rejects tool requests with a 400 unless earlier assistant turns carry reasoning_content.
  • DeepSeek arms, and a DeepSeek judge, now get the report's non-determinism banner.
  • Replayed assistant turns (tool rounds and chat history) get the single-space placeholder DeepSeek accepts.
  • EvalShift never switches thinking off.
• init --provider deepseek. Scaffolds deepseek-flash as source and deepseek-v4-pro as target and judge. The semantic block ships commented out because DeepSeek has no embedding endpoint.
• Docs. DOCS.md, llms-full.txt, docs/ and the CHANGELOG are updated, including a FAQ entry "Does EvalShift work with DeepSeek?". New docs-currency tests make the docs name every registry key env var and every init --provider choice.

Plan: docs/superpowers/plans/2026-09-30-deepseek-provider.md. Each task had a spec and quality review, and the whole branch had a final review.

CI note

Rebased onto main after #21 (the SQLAlchemy 2.1 cache fix) merged. CI is green on Python 3.11–3.14.

Live verification checklist (needs DEEPSEEK_API_KEY; the judge run also needs GEMINI_API_KEY)

• Smoke script. Run DEEPSEEK_API_KEY=... uv run python scripts/smoke_live_tools.py. For each DeepSeek model, expect:
  • tool calls and a cost above $0 for the tool prompts;
  • round 1: with no 400, which proves the backfill;
  • history: text=True, which shows the placeholder is harmless on a request without tools.
  • Commit the new tests/unit/fixtures/tool_responses/deepseek/*_live.json files.
• Backfill proof.
  • Temporarily disable the if "messages" in kwargs and thinking_by_default(canonical): block in src/evalshift_cli/models/client.py (around lines 739–744).
  • Rerun the script for deepseek-flash only. round 1 should now FAIL with DeepSeek's "reasoning_content … must be passed back" 400.
  • Restore the block. If round 1 passes without it, DeepSeek has relaxed the rule; keep the block and note it here.
• test-call. Run DEEPSEEK_API_KEY=... uv run evalshift test-call --model deepseek-flash --max-tokens 2048 and expect a reply and a non-zero cost.
  • Also try the default --max-tokens 256. If it returns an empty, thinking-only reply, reasoning tokens count toward max_tokens, and the FAQ should say so.
• Compare. Run cd examples/agent && DEEPSEEK_API_KEY=... GEMINI_API_KEY=... uv run evalshift compare --from deepseek-flash --to deepseek-v4-pro --yes. Expect:
  • the report is written;
  • the banner names both DeepSeek arms;
  • tool-call scores are populated;
  • evalshift doctor shows DEEPSEEK_API_KEY set.
• DeepSeek judge. Copy examples/agent to a scratch directory, set judge_model: deepseek-v4-pro on an llm_judge entry, and rerun the compare. Expect:
  • the verdicts are populated;
  • the banner lists deepseek/deepseek-v4-pro exactly once.

Release follows the plan's Phase E: CLI 1.2.0, then the action pin PR, then the sibling PRs and the site npm run sync:llms.

🤖 Generated with Claude Code