Retry a Hub server error instead of ending the adapter's run

[borgr]

What happens today

mt_bench failed in the 2026-09-10 nightly on this, and nothing about it was the adapter's fault:

error: could not write to evaleval/EEE_raw: HfHubHTTPError: Server error
'500 Internal Server Error' for url 'https://huggingface.co/api/datasets/
evaleval/EEE_raw/commit/main'

The retry loop in cron/store.py is guarded by is_commit_conflict, which recognises 409 and 412 and nothing else. A 500 is therefore not retryable, so the very first attempt raises StoreError and the job dies — with eight attempts and jittered backoff sitting unused right beside it. One bad minute on the Hub costs that source its whole day.

The change

A 5xx means the Hub failed to apply the commit rather than refusing one, so nothing landed and replaying the same operations is safe. That is a different question from "did I lose a race", so it gets its own predicate rather than widening is_commit_conflict past what its name and docstring claim:

• is_transient_hub_error — 500, 502, 503, 504
• is_retryable_commit_error — either that or a conflict

Three call sites now gate on the combined predicate: RawStore.commit, DatastoreSubmitter.publish, and FlatPublisher._commit_batch in the flat rebuild that just landed. The submitter's existing guard is untouched, so it still retries only where _landed_anyway proved the batch absent and a retry cannot duplicate records.

Status matching goes against the Hub's Server error '5xx clause rather than a bare number. These messages carry request ids like Root=1-6aa264f2-3b5a4040556d7a837e54029b, and a digit run inside one must not read as a status — test_a_request_id_holding_a_status_number_is_not_a_status pins that.

Tests

1422 passed, 1 skipped, ruff clean. Four new, one amended:

• a 500 that clears on the second attempt commits, and waits exactly once
• a 503 that never clears still fails after the full budget
• the two predicates answer only their own question, and a 403 is retryable by neither
• a 403 whose request id contains 500 is not a server fault
• the existing "nothing landed" submitter test now uses a 502 in the Hub's real phrasing and takes the retry_waits fixture, since it retries before reporting

Scope

This does not fix the other two nightly failures. hal is #289 and terminal_bench_2 is #290, both still awaiting review. Those three are the whole of the current nightly redness, and the nightly being red also means the Flat rebuild workflow's workflow_run trigger never fires — only its 01:14 catch-up schedule does.