fix(retention): batch every delete — the sweep had never once succeeded

[thejefflarson]

watcher has been serving 503 on /healthz:

{"db":true,"retention_last_success_age_secs":null,"retention_stalled":true,"status":"unhealthy"}

null, not stale — retention had never once completed in the life of the process. Two independent bugs, both in the history-table loop.

1. The deletes were unbatched

Each history table was pruned with a single DELETE FROM <t> WHERE <col> < cutoff. Against a backlog that exceeds the pool's 60s statement_timeout the statement is cancelled and rolls back entirely — the sweep deletes nothing, the backlog grows, the next attempt is slower. It cannot recover on its own.

Measured in production:

DELETE FROM logs WHERE time < now() - make_interval(days => $1)
  elapsed = 60.01s   rows_affected = 0
  retention sweep failed: canceling statement due to statement timeout

logs held 11,488,281 rows past its 7-day window — essentially the whole 15 GB table, with 5 indexes including a GIN on attributes.

prune_raw_metrics already solved exactly this with ctid-batching, and its own doc says "this is exactly how the table once reached 35 GB" (JEF-425). The module comment assumed per-table deletes were small enough to skip it. They aren't. All four tables now share prune_batched.

2. One failing table starved the rest

The loop used ?, so the first failure aborted the sweep. metric_series_rollups is ordered after logs — so once the logs delete began timing out hourly, the rollups table was never swept again. That's why it reached 20 GB / 14.8M rows.

Per-table errors are now collected. The sweep still fails as a whole (so /healthz keeps reporting the stall, correctly), but every other table gets pruned first.

Also: a seq scan every minute

selfmon runs max(bucket) on metric_series_rollups once a minute. Both existing indexes lead with name, so a bare max(bucket) can use neither:

	plan	time
before	Parallel Seq Scan, 14.8M rows, 966k buffer reads	2907 ms
after	Index Only Scan, 1 row	0.142 ms

Added to the online-DDL lane, not a migration — ADR 0021: CONCURRENTLY can't run inside sqlx::migrate!'s transaction. Already applied by hand in production while diagnosing; declared here so a fresh database gets it.

Tests

The existing retention_prunes_old_rows inserts two rows per table and asserts deleted >= 4 — it passes against the broken code, which is how this shipped. Two regression tests added that fail against the old shape:

• retention_history_tables_drain_in_batches — batch = 1 forces the loop; a prune that stops after one statement fails it
• retention_one_failing_table_does_not_starve_the_rest — a BEFORE DELETE trigger fails spans (first in the list) and asserts logs (after it) is still pruned, and that the sweep still reports failure overall

Local gates green: cargo fmt --check, cargo clippy --all-targets -- -D warnings. Integration tests need Postgres and run on CI's sidecar — Docker isn't running on my machine, and I deliberately did not point DATABASE_URL at production, since pool_or_skip truncates every telemetry table.

Not fixed here

selfmon's pg_visibility_map_summary call fails with permission denied — the DB role lacks pg_stat_scan_tables. That's a role grant on the operator-managed cluster, not an app change.

🤖 Generated with Claude Code

https://claude.ai/code/session_013cYVqzH7Xfwea7fAozdQK7