perf: parallelize the code indexing pipeline

[GoodbyePlanet]

Closes #67.

Why

IndexPipeline.index_service processed one file at a time: blob fetch → parse → dense embed → sparse embed → upsert. Network round-trips never overlapped, and since a single file yields only a handful of symbols, the pipeline sent dozens-to-hundreds of tiny embedding requests to providers that accept 128 inputs each.

What changed

All three recommendations from the issue:

1. Concurrent blob fetch — a producer task fetches and parses behind a bounded semaphore (_FETCH_CONCURRENCY = 10, matching the existing _TREE_WALK_CONCURRENCY / _DIFF_CONCURRENCY), feeding a bounded queue so downloads continue while the previous group embeds.
2. Cross-file batching — symbols accumulate across files up to the provider's batch size, exposed as a new EmbeddingProvider.batch_size property (128 for Voyage/OpenAI/Jina hosted, 32 for self-hosted Jina TEI and Ollama).
3. Gathered embeds — dense and sparse now run under one asyncio.gather. Dense is network-bound and sparse runs in a thread executor, so they overlap for free.

Plus two follow-ups found while implementing:

• _gh_get retried only rate limits. Transport errors, timeouts and 5xx fell straight through — exactly what concurrent fetching makes more likely, and what the embedding path has handled since post_with_retry landed. Rate limits still wait for GitHub's own Retry-After / X-RateLimit-Reset window; 5xx and transport errors use a short fixed backoff.
• Qdrant writes now overlap across files within a batch (_WRITE_CONCURRENCY = 4).

Design notes

The unit of store mutation stays one file. Batching happens only in the embedding stage, which touches no store state — that is what preserves per-file error isolation and the upsert-before-delete invariant. A batch-level embedding failure falls back to embedding that batch file by file, so one bad file costs an extra round-trip instead of dropping every file batched alongside it.

parse_file deliberately stays on the event loop. registry.py shares one parser instance per language and tree-sitter parsers are not safe for concurrent use, so a thread hop there would be a data race. Commented in-code so it doesn't look like a missed optimization.

Progress current is now a monotone count of files resolved rather than a loop index, so it reaches total instead of jumping over skipped files. The frame contract is unchanged.

Testing

389 tests pass; ruff check / format clean. All 26 pre-existing pipeline tests pass byte-identical — a deliberate constraint, since needing to edit one would have signalled semantic drift.

Each new test was mutation-checked rather than trusted on a green run. Reusing one slice for every file, a semaphore of 1, dropping the per-file fallback, removing the progress dedupe, serializing writes, flipping delete-before-upsert, and removing either retry path each fail exactly the intended test and nothing else.

Not yet run: the end-to-end make index-code against live repos, so the wall-clock improvement is so far unmeasured.

Docs

Updated ingestion.md and the /reindex frame docs. This also corrects two entries that were already stale: the upsert section still described delete-before-upsert, which the pipeline stopped doing in 2cdf49c.

🤖 Generated with Claude Code