feat(agent): keep samples and events on disk until the control plane accepts them - #32
Merged
Merged
Conversation
…accepts them - Keep two queues in the state directory: samples.json (at most 1440, the oldest goes first) and events.json. Each change is a crash-safe write. - Make each event id (a ULID) when the event goes into the queue, so a resend has the same id. Keep only the last agent.started. - Keep each value in the range of the contract before it goes into a queue: one bad value makes a 400, and a 400 drops the whole request. - Send the events first, then the samples with the status, at most 100 events and 240 samples in one request, oldest first. - Follow the reply rules: drop on 200, 400 and 422 (samples); keep on 401 (retry each 5 minutes), 429 (Retry-After), 5xx and network errors (wait 1, 2, 4, 8, then 10 minutes). Apply report_interval from each reply. - Send agent.started at once at start, not at the next tick. Refs #27
…ange Fixes from the adversarial review of this layer. - Cap each integer at the signed 64-bit limit: the control plane (PHP) fails a larger value with a 500, and the agent would resend the same sample for 24 hours. - Events and metrics wait on their own after a failure. A broken events route no longer stops the metrics. (The poll for commands, in the next layer, still waits until the events are sent.) - A wait counts from the start of the report, so a 5 minute wait ends at the tick 5 minutes later, also when the control plane is slow. - Samples that could not go are tried again at the next tick that their wait allows, not only after the next full report interval. - When the time of a report runs out, the rest goes with the next report. That is not a failure of the control plane and makes no backoff. - Log the start of an error reply, for example the validation errors of a 400. Log each sample or event that a full queue drops. - Keep a queue file that cannot be read as <name>.corrupt. - Remove an event's command_id that is not a ULID, so a bad id cannot make a 400 that drops the other events. Refs #27
nabil1440
added this pull request to stack #38
September 22, 2026 10:40
After the warnings of a failure (a 5xx or a network error, a 401 or a 429), the first request that succeeds logs one Info line with the time since the first failure. The events and the samples log apart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer 2 of 7 of the monitoring agent (#31 → #37).
Summary
agent.startedat once.failing_for. The events and the samples log apart.Change
internal/agent/outbox.go:samples.json(at most 1440) andevents.json.internal/agent/send.go: events first, then samples with the status; at most 100 events and 240 samples in each request; the reply rules of the contract (200, 400, 401, 422, 429, 5xx); a wait of 1, 2, 4, 8, then 10 minutes after failures.internal/agent/clean.go: the value ranges of the contract.internal/agent/wire: the JSON bodies of the contract.Adversarial review
The first version was refuted. These items are fixed in this PR:
<name>.corrupt. A command id that is not a ULID is removed.Tests
testing/synctestwith a fake control plane: each reply code, the backoff steps, resends with the same id, batches, a slow control plane, metrics while events fail, and the waits.agent.startedwith the token.Closes #26
Closes #27