feat(sinks): chunk sink deliveries so full-history replays stay under intake limits - #39
feat(sinks): chunk sink deliveries so full-history replays stay under intake limits#39Donemmanuelo wants to merge 1 commit into
Conversation
… intake limits (onelrian#36) A fresh install, lost cursor, or restored old cursor replays the whole audit history in one poll. BATCH_SIZE (default 500, 0 disables) splits each sink's pending batch into chunks that each get the existing retry/backoff; the per-sink watermark advances only when every chunk delivers, so a partial failure retries the full set next cycle and nothing is skipped. Documented in CONFIGURATION.md with tests covering chunk splitting, the 0 = off case, and watermark behavior on a failed chunk.
| }; | ||
|
|
||
| let mut all_delivered = true; | ||
| for chunk in pending.chunks(chunk_size) { |
There was a problem hiding this comment.
Chunking by event count doesn't bound bytes - a chunk of 500 events with large meta/user fields can still exceed an intake cap, so the docs' "payloads stay under intake limits" overpromises. Either chunk by estimated encoded size or qualify the claim. Also: the syslog sink opens a fresh TCP connection per send(), so a big replay now costs N connections (one per chunk) instead of one - fine, but worth knowing.
|
|
||
| > [!NOTE] | ||
| > A fresh install, a lost cursor, or a restored old cursor delivers the whole | ||
| > audit history in one poll. `BATCH_SIZE` splits that replay into |
There was a problem hiding this comment.
Partial-failure semantics worth stating explicitly: if chunk 2 of 3 fails, chunk 1 is already delivered but the watermark stays put, so the next cycle re-sends chunk 1 - real duplicates, and #38's same-timestamp ID tracking does not cover them (they're older than the watermark). At-least-once is fine, just say it: "a partial failure re-delivers the earlier chunks next cycle".
| sinks: &[Box<dyn Sink>], | ||
| cursors: &mut HashMap<String, Option<DateTime<Utc>>>, | ||
| retry_cfg: &RetryConfig, | ||
| batch_size: usize, |
There was a problem hiding this comment.
2 participants
Closes #36.
User-facing impact
A fresh install, lost
CURSOR_FILE, or restored old cursor replays theaccount's entire audit history in one poll — previously one oversized
request per sink that Loki's push API and many HTTP/SIEM intake endpoints
reject outright, then retry with backoff forever.
BATCH_SIZE(default500,0disables) splits each sink's pending batch into chunks, each withthe existing retry/backoff, so a large replay now succeeds in bounded
payloads. The per-sink watermark advances only when every chunk delivers, so
a partial failure retries the whole set next cycle and nothing is skipped.
Verification
cargo fmt --check,cargo clippy --all-targets -- -D warnings, andcargo test --lockedall pass (32 tests, +4 new: chunk splitting,0disables chunking, watermark held on a failed chunk via wiremock
up_to_n_times, andBATCH_SIZEenv parsing/defaulting).docs/CONFIGURATION.mddocuments the variable and the replay scenario.