Stop retrying rejected callbacks, and cut the retry budget to one

[maximusunc]

The finish_query worker holds the whole decompressed TRAPI response in memory for every second a callback POST is in flight. With a 120s httpx timeout, a 3-attempt budget and TASK_LIMIT=10, that is up to ten full payloads resident for ~6 minutes each -- the peak that gets the pod OOM-killed (an uncatchable SIGKILL, which is why the crash leaves no traceback and no drain log behind).

Two changes to the retry loop, both aimed at that residency:

• Don't retry a 4xx. The receiver understood the request and rejected it, so an identical retry gets an identical answer; we were spending a full extra round trip, plus backoff, holding the payload, for a response that could not change. Observed in production as the same 400 logged three times for one message. 408 and 429 are kept retryable -- they describe a transient condition rather than a bad request. Everything else (5xx, timeouts, connect/protocol failures, and any exception we don't recognize) is retried exactly as before, so the loop is narrowed only where a retry is known to be pointless.

• CALLBACK_RETRIES 3 -> 1. It now means retries after the first attempt, with the loop bounded by the derived CALLBACK_ATTEMPTS, so the name matches what it counts and the budget is two POSTs rather than three.

The give-up log line now reports the attempts actually spent instead of the configured budget, which would misreport an early bail-out, and the span carries callback.retryable to separate "gave up" from "ran out".

Claude-Session: https://claude.ai/code/session_011RUcnjJx7gebTdjzG1MWFp

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.97%. Comparing base (8940967) to head (4f207ed).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Coverage Δ
workers/finish_query/worker.py	87.05% <100.00%> (+1.25%)	⬆️

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f73bbc7...4f207ed. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.