Skip to content

feat(agent): install signed releases by itself, one time each day - #40

Merged
nabil1440 merged 10 commits into
agent/10-install-shfrom
agent/39-auto-update
Sep 28, 2026
Merged

nabil1440 merged 10 commits into
agent/10-install-shfrom
agent/39-auto-update

Conversation

@nabil1440

@nabil1440 nabil1440 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Closes #39.

Part of the monitoring agent stack. This PR is on top of #37.

What changes

  • Release signature. A release can now have checksums.txt.sig. The file names the key, the tag and the time of the signature, and it has an ed25519 signature. The signature covers these lines and the bytes of checksums.txt.
  • Signing tool. tools/releasesign makes a key pair, signs and verifies. make release-key makes the key. The maintainer signs on their own computer. The private key is never in GitHub.
  • Sign only a release that builds again. make sign-release runs tools/sign-release.sh. It shows the commit of the local tag and asks to type the tag. It builds the release again from the local tag, with the Go version of the CI binaries, and compares the binaries byte for byte. If GitHub serves a different archive, or the tag on GitHub moved, the script stops before the signature. For this, the build date is now the date of the commit.
  • Trusted keys. internal/release/keys.go holds the public keys. A build without a key does not update itself.
  • Auto-update in the agent. One time each day, at a time from the server id, the agent gets the latest release. It installs the release only if all of these are true:
    • The release is newer. The agent never goes back to an older version.
    • The signature is correct.
    • The signature is more than 24 hours old.
  • Off switch. FLY_AGENT_AUTO_UPDATE=off in /etc/fly/agent.env stops the check on one server. An unknown value gives a warning, and the agent continues.
  • No auto-update for local builds. A make build after a tag (v0.2.0-3-gabcdef1, -dirty) does not update itself: the release would replace newer code.
  • State fix. The state file now keeps all fields. Before, a new report interval would remove the time of the last check.
  • Contract v0.3.1. The README pins v0.3.1. The agent sends updates_total and updates_security as null until apt-check gives a count. Before, it sent 0: a false "no updates". A later failed count sends the last counts again (v0.3.1 states this).
  • Clock fix (for feat(agent): add fly agent run, the base of the monitoring agent #31). After the wall clock stepped back by more than a minute, the agent sent nothing for the size of the step. Now it skips a repeated minute only after a step of less than 2 minutes, and else follows the new clock. A repeated minute is harmless: the control plane keeps one sample for each minute.

agent.update does not change. It stays the fast path, and the recovery path if the key is lost or stolen.

Before release

  • Done: internal/release/keys.go trusts the key 21ec14c790e96d62 ("server-cli release key for flywp"). A test fails if the list is empty.
  • After the release workflow publishes a release, run make sign-release VERSION=<tag> KEY=<file or ->.

Tests

  • Signature: round trip, a wrong key, a wrong tag, a changed checksum file, a changed time, a time in the future, a bad format.
  • Agent (fake clock): one check each day at the slot, the 24 hour wait, no downgrade, no signature, a bad signature, GitHub down, a restart, a clock that was ahead, the off switch, a check while the events fail, and the state fix.
  • End to end (Linux): a real fly agent run installs a signed release from a test GitHub, then exits.
  • Sign script: a Linux build and a macOS build of one tag are the same. A swapped archive, an extra name in checksums.txt, and a missing local tag stop the script, and no signature is written.
  • Clock: steps back of 90 seconds, one hour and one year.
  • make check passes.

@nabil1440
nabil1440 added this pull request to stack #38 September 23, 2026 05:42
@nabil1440 nabil1440 self-assigned this Sep 23, 2026
@nabil1440 nabil1440 changed the title agent/39 auto update feat(agent): install signed releases by itself, one time each day Sep 23, 2026
Add the signature file checksums.txt.sig: the key id, the tag, the time of
the signature and an ed25519 signature over these lines and the bytes of
checksums.txt. SignedChecksum checks it and takes the sum from the same
bytes. The tool tools/releasesign makes the key and signs a release; make
release-key and make sign-release run it. keys.go holds the trusted
public keys.

Refs #39
Each day, at a time from the server id, the agent gets the latest release.
It installs the release only when it is newer, its checksums.txt has a valid
signature, and the signature is more than 24 hours old. It then exits, and
systemd starts the new binary. The check runs on each tick, not only on a
report tick, and it saves its time before it starts.

FLY_AGENT_AUTO_UPDATE=off stops the check on one server. A build without a
release version, or without a trusted key, does not check. agent.update
works as before.

The state file now keeps the whole state: a new report interval no longer
removes the time of the last check.

Closes #39
make sign-release signed whatever checksums.txt GitHub served. Someone who
controls GitHub could swap an archive and its sum before the signature.

tools/sign-release.sh now shows the commit of the local tag and asks to
type the tag. It checks the archives against checksums.txt, builds the
release again from the local tag with the Go version of the CI binaries,
and compares the binaries byte for byte. The binary holds its commit, so
this also proves that CI built the local tag. checksums.txt must name
exactly the built archives. The script signs and verifies with the tool
and the keys of the tag.

The build date is now the date of the commit, so the same commit and Go
version give the same binary on any computer.

Refs #39
- A make build after a tag (v0.2.0-3-gabcdef1, -dirty) sorts before the
  tag, so the release would replace newer code. release.IsLocalBuild
  turns auto-update off for these builds.
- Read the release JSON through a 1 MiB limit: the agent reads it each
  day without a person.
- Test that the tag and the key lines are part of the signed bytes.

Refs #39
After the wall clock stepped back by more than a minute, the loop waited
for the old tick time: the agent sent nothing for the size of the step.
For example, a VM that booted with its clock one hour ahead went silent
for one hour after NTP corrected it.

The loop now skips a repeated minute only after a step back of less than
2 minutes. After a larger step it follows the new clock. A minute that
runs two times is harmless: the control plane keeps one sample for each
minute.
make release-key takes COMMENT (default: "server-cli release key for
flywp"). The comment goes in the private key file as a PEM header and in
the output, and keys.go names each key with it. A comment must be one
line, so that it cannot add an other PEM header.

Refs #39
In "make release-key KEY=~/key" the shell does not expand the "~": it is
not at the start of a word. The tool and the sign script now replace a
leading "~/" with the home directory.

Refs #39
Add the public key 21ec14c790e96d62 ("server-cli release key for flywp").
The private key is kept outside GitHub. A test now fails when the list of
trusted keys is empty, so that no release ships without a key.

Refs #39
… counts

Contract v0.3.0 records the update by release, and permits null for the
update counts. The agent now sends updates_total and updates_security as
null until apt-check gives a count. Before, it sent 0: a false "no
updates". A later failure keeps the last counts, as before.

The README pin is v0.3.0.

Refs #39
The contract now states that a failed update count keeps the last counts,
and that null means no count since the agent started. The agent already
does this; only the pin changes.
@nabil1440
nabil1440 merged commit 2ef985b into develop Sep 28, 2026
1 of 2 checks passed
@nabil1440
nabil1440 deleted the agent/39-auto-update branch September 28, 2026 03:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature: Let the agent install signed releases on its own

1 participant