From de1b78b12cb11e21ed5cbd6b39d15b27b31d12ce Mon Sep 17 00:00:00 2001
From: nabil1440 <52530910+nabil1440@users.noreply.github.com>
Date: Mon, 28 Sep 2026 10:18:21 +0600
Subject: [PATCH 1/2] docs: rewrite the README and add the reference docs of
v0.2.0
The README is short: install, the commands, the agent in one paragraph,
and links. The details move to docs/: the monitoring agent, development,
releasing and the design decisions.
---
README.md | 226 +++++++++------------------------------
docs/decisions.md | 90 ++++++++++++++++
docs/development.md | 60 +++++++++++
docs/monitoring-agent.md | 163 ++++++++++++++++++++++++++++
docs/releasing.md | 104 ++++++++++++++++++
5 files changed, 465 insertions(+), 178 deletions(-)
create mode 100644 docs/decisions.md
create mode 100644 docs/development.md
create mode 100644 docs/monitoring-agent.md
create mode 100644 docs/releasing.md
diff --git a/README.md b/README.md
index 91d5260..e271a39 100644
--- a/README.md
+++ b/README.md
@@ -1,208 +1,78 @@
-# server-cli
+# fly — the FlyWP server CLI
-Easy CLI tool for servers managed by FlyWP.
+`fly` manages the sites on a server that [FlyWP](https://flywp.com) provisions. It also runs the
+FlyWP monitoring agent.
-Conforms to the FlyWP monitoring agent contract v0.5.0.
-
-## Installation
-
-### Prerequisites
-
-- [Docker](https://www.docker.com/get-started)
-- [Docker Compose](https://docs.docker.com/compose/install/)
-
-### Quick Install
-
-You can easily install the `fly` CLI tool using the following command. This will download and run the `install.sh` script, which will automatically detect your operating system and architecture, download the latest release, and install it to `/usr/local/bin`:
+## Install
```bash
curl -fsSL https://raw.githubusercontent.com/flywp/server-cli/main/install.sh | sudo bash
```
-
-
-Manual Installation
-
-### Manual Installation
-
-If you prefer to manually download and install the binary, follow these steps:
-
-1. Download the precompiled binaries from the [Releases](https://github.com/flywp/server-cli/releases) page. Choose the version suitable for your operating system and architecture.
-
-1. Download the [latest tarball]((https://github.com/flywp/server-cli/releases)) for your platform:
-
- ```bash
- wget https://github.com/flywp/server-cli/releases/download/v0.1.0/fly-linux-amd64.tar.gz
- ```
-
-2. Extract the tarball:
- ```bash
- tar -xzf fly-linux-amd64.tar.gz
- ```
-
-3. Move the binary to a directory in your PATH:
- ```bash
- sudo mv fly-linux-amd64 /usr/local/bin/fly
- ```
-
-4. Verify the installation:
- ```bash
- fly version
- ```
-
-
-
-## Usage
-
-### Base Docker Compose
-
-FlyWP has a base Docker Compose configuration for running MySQL, Redis, Ofelia, and Nginx Proxy that are shared for all sites hosted on the server. The base Docker Compose must be started before a site can be created.
-
-```bash
-fly base start # starts the base services (mysql, redis, nginx-proxy)
-fly base stop # stops the base services
-fly base restart # restarts the base services
-```
-
-### Site Operations
-
-You can run the following commands from anywhere inside a site folder or by specifying the domain name.
-
-```bash
-fly start --domain example.com # starts the website
-fly stop --domain example.com # stops the website
-fly restart --domain example.com # restarts the website
-fly --domain example.com wp # execute WP-CLI commands
-fly logs --domain example.com # view logs from all containers or a single one
-fly restart --domain example.com # restart a container
-fly --domain example.com exec [container] # execute commands inside a container. Default: the PHP container
-```
-
-Or run the commands from within the site directory without specifying the domain:
-
-```bash
-fly start # starts the website
-fly stop # stops the website
-fly restart # restarts the website
-fly wp # execute WP-CLI commands
-fly logs [container] # view logs from all containers or a single one
-fly logs -f [container] # follow the logs (--tail N shows the last N lines)
-fly restart # restart a container
-fly exec [container] # execute commands inside a container. Default: the PHP container
-```
-
-### WP-CLI
-
-**wp-cli**: To access `wp-cli`, use the following command from anywhere in the website folder or specify the domain name. The CLI will find the appropriate WordPress folder to execute the `wp` command.
-
-```bash
-fly --domain example.com wp plugin list --format=json
-```
-
-All arguments after the WP-CLI command (or after the command for `fly exec`) go to that command unchanged, flags included. Put `--domain` before the command. To pass a flag as the first argument, put `--` before it, for example `fly wp -- --info`.
+The script installs the latest release for your architecture (Linux amd64 or arm64) to
+`/usr/local/bin/fly`. It checks the download against `checksums.txt` first.
-### Monitoring agent
+To update later, run `sudo fly update`.
-`fly agent run` is the FlyWP monitoring agent. It runs all the time under systemd (`fly-agent.service`, as the server user, not root), and FlyWP installs it. Each minute it measures CPU, load, memory, swap, disk and network traffic, the pressure (PSI) and the disk activity. It reads the server each 10 seconds, so each minute also has its peaks. It also measures the CPU, the memory and the disk use of each site: each Docker Compose project in the home folder of the server user. It measures the disk use at most one time each hour, at the lowest I/O priority. It sends the values and the server status (restart needed, waiting updates, OS, kernel, uptime, CPU count, Docker state and version) to FlyWP. It keeps unsent data on disk for up to 24 hours. FlyWP can update and restart the agent through it, without SSH. The agent does not need Docker. When Docker runs, the agent reads its socket with two requests only: `GET /version` and `GET /containers/json`.
-
-It reads `FLY_AGENT_URL` (https), `FLY_AGENT_TOKEN` and `FLY_AGENT_SERVER_ID` from `/etc/fly/agent.env`, and keeps its state in `STATE_DIRECTORY` (`/var/lib/fly-agent`).
-
-The agent also updates itself. Once a day, at a time set by the server id, it checks the latest release on GitHub. It installs the release only when:
-
-- the release is newer than the running version (the agent never downgrades)
-- the release has a valid signature from the FlyWP release key (see [Releasing](#releasing))
-- the signature is more than 24 hours old
-
-To turn this off on one server, add `FLY_AGENT_AUTO_UPDATE=off` to `/etc/fly/agent.env` and restart the agent. Updates that FlyWP sends still work.
-
-```bash
-systemctl status fly-agent # is the agent running?
-journalctl -u fly-agent -f # the agent log
-```
-
-### Global Commands
-
-A few helper commands to debug the server installation and start/stop all sites.
-
-```bash
-fly status # shows the status of the system
-fly sites start # starts all sites
-fly sites stop # stops all sites
-fly sites restart # stops and starts all sites
-```
-
-## Development
-
-Go 1.27 or later is required (`go.mod` selects the toolchain). The Makefile holds the common tasks:
-
-```bash
-make build # builds bin/fly with the version from git
-make test # go test ./... -race
-make lint # golangci-lint (pinned version, built with the module's Go)
-make vuln # govulncheck
-make check # fmt-check, vet, lint, test and vuln (CI runs the same)
-make release # static linux/amd64 and linux/arm64 archives + checksums.txt in build/
-make help # lists all targets
-```
-
-`make release VERSION=v0.2.0` stamps a specific version. The build date is the date of the commit, so the same commit and Go version give the same binary on any computer. The release archives must keep the names `fly-linux-.tar.gz` with the binary `fly-linux-` inside: installed CLIs look for these names when they run `fly update`.
-
-CI runs `make check` and `make release` on every pull request and on every push to `develop` and `main`.
-
-### Releasing
-
-`main` is the release branch. To publish a release, tag a commit on `main` and push the tag:
+
+Install by hand
```bash
-git tag -a v0.2.0 -m "v0.2.0"
-git push origin v0.2.0
+arch=amd64 # or arm64
+base=https://github.com/flywp/server-cli/releases/latest/download
+curl -fsSLO "$base/fly-linux-$arch.tar.gz" && curl -fsSLO "$base/checksums.txt"
+sha256sum -c --ignore-missing checksums.txt
+tar -xzf "fly-linux-$arch.tar.gz" && sudo install -m 0755 "fly-linux-$arch" /usr/local/bin/fly
+fly version
```
-The Release workflow checks that the tag is on `main`, runs `make check`, builds the archives with `make release`, and creates the GitHub release with both archives and `checksums.txt`. A tag with a pre-release suffix, such as `v0.2.0-rc.1`, becomes a pre-release, so installed CLIs do not update to it.
+
-`install.sh` and `fly update` install only a release that has `checksums.txt`. Releases before v0.2.0 have none, so push the tag right after the merge into `main`: until the release is published, `install.sh` from `main` stops.
+## Use
-After the workflow publishes the release, sign it on your own computer:
+The site commands need Docker and Docker Compose. Run them inside a site folder, or name the site
+with `--domain`.
```bash
-make sign-release VERSION=v0.2.0 KEY= # KEY=- reads the key from stdin
+fly base start|stop|restart # the shared services: MySQL, Redis, Ofelia, Nginx Proxy
+fly start|stop|restart # the site
+fly restart # one container of the site
+fly wp # WP-CLI, for example: fly wp plugin list
+fly exec [container] # a command in a container (default: PHP)
+fly logs [-f] [container] # the logs of the site, or of one container
+
+fly sites start|stop|restart # all sites
+fly status # the state of the server and its services
+fly update # install the latest release
+fly version
```
-The signature says "this release is the code of my tag", so the command signs only what it can build again:
-
-1. It shows the commit of your **local** tag and asks you to type the tag. Review that commit first: the signature is the approval.
-2. It downloads the archives and `checksums.txt`, and checks the archives against the sums.
-3. It builds the release again from your local tag, with the Go version of the CI build, and compares the binaries byte for byte. A binary holds its commit, so this also proves that CI built your tag.
-4. It signs `checksums.txt`, checks the signature with the keys of the tag, and uploads `checksums.txt.sig`.
+Put `--domain` before the command: `fly --domain example.com wp plugin list`. Everything after
+the command goes to it unchanged. To pass a flag as the first argument, put `--` before it:
+`fly wp -- --info`.
-If GitHub serves a swapped archive, or the tag on GitHub moved, step 2 or 3 stops before the signature. Agents install a release by themselves only when it has a valid signature, and only 24 hours after it was signed. Each server then installs it at its own time of day. To stop a bad release in those 24 hours, mark it as a pre-release on GitHub.
+## Monitoring agent
-The signing key is kept outside GitHub, so that a push to GitHub alone cannot reach every server. `make release-key KEY=` makes a key and prints its public key line for `internal/release/keys.go`. `COMMENT=` names the key, in the key file and next to its line in `keys.go` (the default is "server-cli release key for flywp"). Keep the private key in a password manager, with a backup. Never commit it, and never put it in a GitHub secret. If the key is lost or leaked, ship a binary with a new key through a FlyWP update: that path does not use the signature.
-
-### Dev pre-releases
-
-To test a branch on real servers before it merges, publish a dev pre-release of its current commit:
+FlyWP installs `fly agent run` as the systemd service `fly-agent`. Each minute, it sends the health
+of the server and of each site to FlyWP. It runs as the server user, not as root, and it updates
+itself only to signed releases.
```bash
-make dev-version # prints the tag, for example v0.2.0-dev.1a2b3c4
-make dev-release # tags the commit and pushes the tag; CI publishes the pre-release
+systemctl status fly-agent
+journalctl -u fly-agent -f
```
-The version is the next minor version after the latest release, plus the short commit hash (`DEV_BASE=v0.1.2` overrides the first part). Pre-release tags can come from any branch. `make dev-release` refuses uncommitted changes, commits that are not pushed, and commits whose release workflow would publish the tag as a full release.
-
-`fly update` and `install.sh` only install the latest full release, so install a dev pre-release on a test server by hand:
+See [docs/monitoring-agent.md](docs/monitoring-agent.md).
-```bash
-tag=v0.2.0-dev.1a2b3c4 arch=amd64 # arch: amd64 or arm64 (uname -m: x86_64 or aarch64)
-base=https://github.com/flywp/server-cli/releases/download/$tag
-curl -fsSLO "$base/fly-linux-$arch.tar.gz" && curl -fsSLO "$base/checksums.txt"
-sha256sum -c --ignore-missing checksums.txt
-tar -xzf "fly-linux-$arch.tar.gz" && sudo install -m 0755 "fly-linux-$arch" /usr/local/bin/fly
-fly version
-```
+## Documentation
-To build the same version locally without publishing it, run `make release VERSION=$(make -s dev-version)`.
+| | |
+|---|---|
+| [Monitoring agent](docs/monitoring-agent.md) | What it measures, its settings and files, updates, security |
+| [Development](docs/development.md) | Build, tests, code layout |
+| [Releasing](docs/releasing.md) | Release, signing, dev pre-releases, rollback |
+| [Design decisions](docs/decisions.md) | Why the agent works the way it does |
## License
-This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
+MIT. See [LICENSE](LICENSE).
diff --git a/docs/decisions.md b/docs/decisions.md
new file mode 100644
index 0000000..5035c2c
--- /dev/null
+++ b/docs/decisions.md
@@ -0,0 +1,90 @@
+# Design decisions
+
+Each decision of the monitoring agent, with its reason. Change one only when new evidence
+contradicts the reason.
+
+## Shape
+
+- **One binary.** The agent is `fly agent run`, part of `fly`, not a separate program. One release,
+ one install, one update path.
+- **One loop.** One goroutine wakes every 10 seconds, reads the server, and at the tick of each
+ minute makes the sample and, every few minutes, sends. Only the disk walk of the sites runs apart.
+ The work is small, and one loop is easy to reason about after a crash.
+- **Plain JSON files for state**, written to a temporary file, synced and renamed. No database: the
+ queues are small, and a crash must never leave half a file. Event ids are ULIDs, made when the
+ event is queued, so a resend keeps its id and FlyWP can drop duplicates.
+- **HTTPS only**, except for a loopback host, which tests and local development need. Redirects are
+ refused: a redirect could turn a POST into a GET, or send the token over plain http.
+
+## Measurements
+
+- **Peaks, not only averages.** A minute average hides a 10-second spike: 10 seconds at 100% CPU and
+ 50 seconds idle average to about 17%. So the agent reads every 10 seconds and sends the peak
+ window of the minute next to the minute value. The minute values stay as they were, because alerts
+ read them: a sustained value is an incident, a 10-second peak is not.
+- **No peak of load or disk space.** The kernel already smooths the load over a minute, and disk
+ space changes over minutes, not seconds.
+- **Pressure (PSI)**, from the `total=` counters, not the kernel's `avg10`/`avg60` (moving averages
+ that do not match the windows of a sample). "How full" is not "under pressure": memory that makes
+ tasks wait is a problem at any percent.
+- **Disk activity, not a busy percent.** An SSD or a cloud disk serves many requests at once, so
+ "100% busy" does not mean full. The agent sends bytes and operations, and I/O pressure shows the
+ waiting. Only whole hardware disks count: partitions, loop, device-mapper and RAID devices would
+ count the same I/O twice.
+- **Network:** only interfaces with a hardware device (and no `master`), else the default-route
+ interface. Virtual interfaces would count the same traffic twice.
+- **Short windows are dropped.** A reading less than 5 seconds from its neighbour is left out, and a
+ first minute shorter than 10 seconds after a fresh start sends no sample. A window of a few milliseconds, or the load of
+ the start itself, would show as a false spike.
+- **`null`, not 0,** when a value is unknown: after a reboot, a counter that went back, or a missing
+ kernel feature. A 0 would draw a false dip.
+- **Waiting updates** come from `apt-check`, once an hour and not in the send budget. A failed count
+ keeps the last one.
+
+## Sites
+
+- **A site is a Docker Compose project** whose folder is directly in the home folder of the server
+ user. That is where FlyWP puts sites, and the Compose label gives the folder without guessing.
+- **CPU and memory from cgroup v2**, not `docker stats`: the agent reads files and makes no extra
+ Docker requests. A container counts for CPU only when both ticks saw the same container **and the
+ same cgroup**: `docker restart` keeps the container id but starts a new cgroup whose CPU time
+ begins at 0.
+- **Only two Docker requests** (`GET /version`, `GET /containers/json`), with a 5-second timeout.
+ The socket is powerful; the agent uses the least of it.
+- **Disk use once an hour, in the background, at idle I/O priority**, with a 5-minute limit for each
+ folder. A walk of a large site must never delay a sample or slow the sites.
+
+## Sending
+
+- **Keep data until FlyWP accepts it**, up to 24 hours of samples. A 400 drops the batch (it will
+ never be accepted); a 401, 429, 5xx or network error keeps it.
+- **Values are checked before they are queued.** One bad value makes FlyWP refuse the whole batch,
+ so the agent clamps each value to the range of the contract first.
+- **Samples and events have separate backoffs.** A failing event must not stop the metrics.
+
+## Updates
+
+- **The agent never downgrades.** An update to an older release fails with the reason. A bad
+ release is fixed with a newer one. This keeps agents that updated themselves from being moved back.
+- **Two update paths, and both stay:**
+ - FlyWP's `agent.update` with a pinned sha256. It is fast, and it is the recovery path: it does
+ not read the signature.
+ - The daily automatic update, which needs a signature. Without it, a person who controls the GitHub
+ repository alone could put code on every server.
+- **Offline ed25519 signatures.** The maintainer signs `checksums.txt` on their own computer after
+ rebuilding the release from their local tag. The key is never on GitHub or in CI. The public keys
+ are compiled in.
+- **A 24-hour wait after the signature**, then each server at its own time of day. A bad release can
+ be stopped in that time by marking it as a pre-release.
+- **A signature does not expire.**
+- **Why the command path must stay:** if the key leaks, the release that removes it would itself
+ need a signature, and only the leaked key could make one. The pinned-sha256 path is the way out.
+- **`fly update` stays.** It restarts the agent, and it keeps the owner of the binary, so the agent
+ can still replace it.
+
+## Security
+
+- **The agent never runs as root.** It runs as the server user under systemd.
+- **Root never writes into the server user's folder.** When `install.sh` or `fly update` replaces
+ the agent's binary, the file is written by the server user or changed through the open file, never
+ by a path that the server user could swap for a link.
diff --git a/docs/development.md b/docs/development.md
new file mode 100644
index 0000000..42e5318
--- /dev/null
+++ b/docs/development.md
@@ -0,0 +1,60 @@
+# Development
+
+Go 1.27 or later (`go.mod` selects the toolchain). `make help` lists every target.
+
+```bash
+make build # bin/fly, with the version from git
+make test # go test ./... -race
+make check # fmt-check, vet, lint, test and govulncheck: the gate before each merge
+make release # static linux/amd64 and linux/arm64 archives and checksums.txt in build/
+```
+
+CI runs `make check` and `make release` on each pull request and on each push to `develop` and
+`main`.
+
+## Tests on Linux
+
+The agent reads `/proc`, `/sys` and cgroups, so some tests build only on Linux
+(`*_linux_test.go`, for example the test against the real `/proc`). On macOS, `make check` skips
+them. Before you push a change to `internal/metrics` or `internal/agent`, run the tests on Linux as
+a non-root user:
+
+```bash
+docker run --rm -u "$(id -u):$(id -g)" -e HOME=/tmp -v "$PWD":/src -w /src golang:1.27 go test ./...
+```
+
+A few tests need root (the owner of the binary after `sudo fly update`). CI runs them with `sudo`.
+
+## Run the agent locally
+
+The agent accepts plain `http://` for a loopback host, so you can point it at a local control
+plane:
+
+```bash
+make build
+STATE_DIRECTORY=$(mktemp -d) FLY_AGENT_URL=http://127.0.0.1:8080 \
+FLY_AGENT_TOKEN=test FLY_AGENT_SERVER_ID=1 FLY_AGENT_AUTO_UPDATE=off bin/fly agent run
+```
+
+## Reproducible builds
+
+`make release` gives the same bytes for the same commit and Go version, on any computer: the build
+date is the commit date, paths are trimmed, and the archives are written with owner 0:0.
+`make sign-release` depends on this. See [releasing.md](releasing.md).
+
+## Code layout
+
+| Path | |
+|---|---|
+| `cmd/` | The commands (cobra). `cmd/agent.go` is `fly agent run`. |
+| `internal/agent` | The agent: settings, lock, schedule, queues, sending, commands, auto-update. |
+| `internal/agent/wire` | The JSON types of the contract. |
+| `internal/metrics` | The Linux measurements: `/proc`, PSI, disks, Docker, sites. |
+| `internal/dockerapi` | A minimal Docker client over the unix socket (two GET requests). |
+| `internal/release` | Release download, sha256 check, binary replacement, signatures, trusted keys. |
+| `internal/service` | The `fly-agent` systemd unit: restart after an update. |
+| `internal/statefile` | JSON files written atomically. |
+| `internal/docker` | Docker Compose calls and the Docker checks of the site commands. |
+| `internal/utils` | Finds the site folder from the current folder or `--domain`. |
+| `tools/releasesign`, `tools/sign-release.sh` | Key generation and release signing. |
+| `install.sh` | The installer. |
diff --git a/docs/monitoring-agent.md b/docs/monitoring-agent.md
new file mode 100644
index 0000000..bf8c2b4
--- /dev/null
+++ b/docs/monitoring-agent.md
@@ -0,0 +1,163 @@
+# The monitoring agent
+
+`fly agent run` is the FlyWP monitoring agent. It measures the server each minute and sends the
+values to FlyWP. FlyWP shows them as charts and alerts, and can restart or update the agent without
+SSH.
+
+The agent follows the FlyWP monitoring agent contract **v0.5.0**. The contract defines the requests,
+the fields and the rules on both sides.
+
+## How it runs
+
+| | |
+|---|---|
+| Service | `/etc/systemd/system/fly-agent.service`, `Restart=always`. FlyWP installs it. |
+| User | The server user (for example `fly`), never root. |
+| Binary | `~/.fly/bin/fly` of the server user. `/usr/local/bin/fly` is a link to it. |
+| Settings | `/etc/fly/agent.env` (root, mode 0600) |
+| State | `/var/lib/fly-agent` (`STATE_DIRECTORY` of systemd) |
+| Log | `journalctl -u fly-agent` |
+
+Only one agent can run with one state folder. The agent does not need Docker.
+
+### Settings
+
+| Variable | |
+|---|---|
+| `FLY_AGENT_URL` | The FlyWP control plane. It must be `https://`. Plain `http://` is allowed only for a loopback host (tests and local development). |
+| `FLY_AGENT_TOKEN` | The token of this server. The agent never logs it. |
+| `FLY_AGENT_SERVER_ID` | The id of the server in FlyWP. It also sets the second of the minute at which the agent works, so that servers do not all send at the same time. |
+| `FLY_AGENT_AUTO_UPDATE` | `off` stops the automatic updates on this server. See [Updates](#updates). |
+
+After a change, run `sudo systemctl restart fly-agent`.
+
+### State files
+
+| File | |
+|---|---|
+| `samples.json` | Samples not yet accepted by FlyWP. At most 24 hours (1440 samples); then the oldest go. |
+| `events.json` | Events not yet accepted (at most 1000). |
+| `ran.json` | The commands that ran, so that no command runs two times. |
+| `counters.json` | The last counters, so that a restart does not lose a minute of traffic. |
+| `state.json` | The report interval, and the time of the last release check. |
+| `lock` | Stops a second agent. |
+
+Each file is written to a temporary file first, then renamed, so a crash never leaves half a file.
+
+## What it sends
+
+The agent reads the server every 10 seconds. Each minute, it sends one sample with the value of the
+minute and, where it makes sense, the **peak** of the minute. A peak shows a short spike that the
+average of the minute hides.
+
+| Group | Values |
+|---|---|
+| CPU | use (%) and its peak, load (1 min) |
+| Memory | used and total, swap used and total, with the peaks of the used values |
+| Disk space | used and total of `/` (the same as `df`) |
+| Network | bytes in and out of the physical interfaces, and the peak bytes each second |
+| Pressure (PSI) | the share of time that tasks waited for CPU, memory or disk I/O, and its peak. Not sent when the kernel has no PSI. |
+| Disk activity | bytes and operations read and written on the physical disks, and the peaks each second |
+| Sites | the CPU, memory and disk use of each site (see below) |
+
+With the samples, the agent sends the **status** of the server: restart needed, waiting updates
+(and security updates), OS, kernel, uptime, architecture, CPU count, Docker state (`running`,
+`not_running`, `not_installed`) and Docker version.
+
+### Sites
+
+A site is a Docker Compose project whose folder is directly in the home folder of the server user.
+For each site, the agent sends:
+
+- **CPU:** the CPU time of its containers in the minute, as a share of all CPUs.
+- **Memory:** the memory of its containers, without the page cache that the kernel can free (the
+ same as `docker stats`).
+- **Disk:** the size of its folder. The agent measures it at most once an hour, in the background, at
+ the lowest I/O priority. A folder that the server user cannot list is skipped, so the value can be
+ lower than the real use (for example, the database files in `~/.fly` belong to the container
+ user).
+
+The agent reads Docker through its socket with two requests only: `GET /version` and
+`GET /containers/json`. It reads CPU and memory from cgroup v2. Without Docker or cgroup v2, it
+sends no site values.
+
+### Gaps and null values
+
+- A value that the agent cannot measure is `null`, not 0.
+- The first sample after a fresh start has no traffic (`net_counters_reset` is true) and no
+ pressure or disk activity values: they need two readings. A restart within 90 seconds keeps
+ them.
+- When the first minute after a fresh start is shorter than 10 seconds, the agent sends no sample
+ for it. That short minute is mostly the load of the start itself.
+- A reading that fails skips that minute. The agent never stops for a measurement error.
+
+## Sending
+
+- The agent sends every 1 to 10 minutes. FlyWP sets the interval in each reply.
+- When FlyWP cannot be reached, the agent keeps the data on disk and tries again: after 1, 2, 4, 8,
+ then every 10 minutes. It keeps measuring meanwhile. After a 401 it tries each 5 minutes; after a
+ 429 it waits as FlyWP asks.
+- When FlyWP accepts the data again, the agent logs one line: "the control plane accepts the
+ requests again".
+- Samples are sent even while events fail, and the other way around.
+
+## Commands from FlyWP
+
+| Command | What the agent does |
+|---|---|
+| `agent.restart` | Exits; systemd starts it again. |
+| `agent.update` | Downloads the version that FlyWP names, checks its sha256 against the value that FlyWP sends, replaces the binary and exits. |
+
+Each command runs at most one time, also after a crash. The result goes back as an event
+(`command.completed` or `command.failed`). The agent never installs an older version: an update to
+an older release fails with the reason.
+
+## Updates
+
+There are three ways to a new version. All keep the binary owned by the server user and restart the
+agent.
+
+1. **FlyWP sends `agent.update`** with the version and its sha256. This path does not need a
+ signature, so it also works when the release key is lost.
+2. **The agent updates itself.** Once a day, at a time set by the server id, it checks the latest
+ release on GitHub. It installs it only when:
+ - the release is newer than the running version;
+ - `checksums.txt.sig` has a valid signature from a FlyWP release key that this build trusts;
+ - the signature is more than 24 hours old.
+
+ Each server installs at its own time of day. A dev build, or a build without a trusted key,
+ never updates itself. `FLY_AGENT_AUTO_UPDATE=off` stops this path on one server.
+3. **`sudo fly update`** or `install.sh` install the latest release after checking its sha256 in
+ `checksums.txt`.
+
+## Security
+
+- The agent runs as the server user. It never runs as root, and it refuses to start as root.
+- It sends data only to `FLY_AGENT_URL`, over HTTPS, and it does not follow redirects. Downloads
+ are HTTPS too.
+- The token is never logged, and an error never shows it.
+- Every download is checked against a sha256 before it replaces the binary.
+- The release signing key is kept offline, never on GitHub. A person who controls the GitHub
+ repository alone cannot make every agent install their code. See [releasing.md](releasing.md).
+
+## Troubleshooting
+
+| Log line | Meaning |
+|---|---|
+| "must be an https URL" | `FLY_AGENT_URL` is plain http. |
+| "does not accept the token" | The token in `/etc/fly/agent.env` does not match FlyWP. The agent keeps its data and tries each 5 minutes. |
+| "asks the agent to wait" | FlyWP rate-limits the agent (429). |
+| "a different agent is running" | A second `fly agent run` uses the same state folder. |
+| "no sample for this minute" | The first minute after a start was too short. Normal. |
+| "some files of a site cannot be read" | The disk walk skipped folders; the site value is lower than the real use. Logged once per folder after each start. |
+| "a new release installs after its wait" | Normal: a signed release waits 24 hours. |
+| "a new release waits for its signature" | The latest release is not signed yet. |
+| "auto-update is off: this build trusts no release key" | A local or dev build. Use a release build. |
+
+To remove the agent from a server:
+
+```bash
+sudo systemctl disable --now fly-agent
+sudo rm -rf /etc/systemd/system/fly-agent.service /etc/fly /var/lib/fly-agent
+sudo systemctl daemon-reload
+```
diff --git a/docs/releasing.md b/docs/releasing.md
new file mode 100644
index 0000000..17a7c18
--- /dev/null
+++ b/docs/releasing.md
@@ -0,0 +1,104 @@
+# Releasing
+
+`develop` is the development branch. `main` is the release branch: a release tag must be on `main`.
+
+## A release
+
+1. **Merge `develop` into `main`** with a pull request and a **merge commit** (not a squash, so the
+ two branches keep a common history).
+2. **Tag and push at once:**
+
+ ```bash
+ git checkout main && git pull --ff-only
+ git tag -a v0.2.0 -m "v0.2.0"
+ git push origin v0.2.0
+ ```
+
+ The Release workflow checks that the tag is on `main`, runs `make check`, builds the archives
+ with `make release`, and publishes the release with `fly-linux-amd64.tar.gz`,
+ `fly-linux-arm64.tar.gz` and `checksums.txt`. Do not rename these files: installed CLIs look for
+ them.
+
+3. **Sign it**, on your own computer (see [Signing](#signing)):
+
+ ```bash
+ make sign-release VERSION=v0.2.0 KEY= # KEY=- reads the key from stdin
+ ```
+
+4. **Watch it for 24 hours.** Agents install a signed release by themselves only 24 hours after the
+ signature, each at its own time of day. First update one test server at once with
+ `sudo fly update --yes`, and watch its log.
+5. **To stop a bad release** within those 24 hours, mark it as a pre-release on GitHub. Agents no
+ longer see it. After that, fix forward: the agent never downgrades.
+
+A tag with a suffix, such as `v0.3.0-rc.1`, becomes a pre-release. `fly update`, `install.sh` and
+the agents ignore pre-releases.
+
+## Signing
+
+The signature says: "this release is the code of my tag". Agents check it before they update
+themselves. The key never goes to GitHub, so control of the GitHub repository alone is not enough to
+reach every server.
+
+`make sign-release` signs only what it can build again:
+
+1. It shows the commit of your **local** tag and asks you to type the tag. Review that commit first:
+ the signature is your approval.
+2. It downloads the archives and `checksums.txt`, and checks the archives against the sums.
+3. It builds the release again from your local tag, with the Go version of the CI build, and
+ compares the binaries byte for byte. A binary holds its commit, so this also proves that CI built
+ your tag.
+4. It signs `checksums.txt`, checks the signature with the keys of the tag, and uploads
+ `checksums.txt.sig`.
+
+If an archive on GitHub was swapped, or the tag moved, step 2 or 3 stops before anything is signed.
+`UPLOAD=0` does every check and writes the signature without uploading it, which is useful for a
+rehearsal on a dev pre-release.
+
+### Keys
+
+- `make release-key KEY=` makes a key pair. It prints the public key line
+ for `internal/release/keys.go`. `COMMENT="..."` names the key.
+- Keep the private key in a password manager or vault, with a backup. Never commit it, and never put
+ it in a GitHub secret or CI.
+- `keys.go` can list more than one key. One valid signature from any listed key is enough. To
+ replace a key, list both for one release, then remove the old one.
+- A change to `keys.go` takes effect only in the binaries of the next release.
+
+**If the key is lost or leaked:** make a new key, put its line in `keys.go` (remove the leaked
+line), and release. Ship that release through a FlyWP `agent.update`: that path checks the sha256
+that FlyWP sends and does not read the signature. This is why the command path must stay.
+
+## Dev pre-releases
+
+To test a branch on a real server before it merges:
+
+```bash
+make dev-version # prints the tag, for example v0.2.0-dev.1a2b3c4
+make dev-release # tags the pushed commit; CI publishes a pre-release
+```
+
+The version is the next minor version after the latest release, plus the short commit hash
+(`DEV_BASE=v0.2.1` overrides the first part). `make dev-release` refuses uncommitted changes and
+commits that are not pushed. Pre-release tags can come from any branch.
+
+Install one on a test server by hand:
+
+```bash
+tag=v0.2.0-dev.1a2b3c4 arch=amd64
+base=https://github.com/flywp/server-cli/releases/download/$tag
+curl -fsSLO "$base/fly-linux-$arch.tar.gz" && curl -fsSLO "$base/checksums.txt"
+sha256sum -c --ignore-missing checksums.txt
+tar -xzf "fly-linux-$arch.tar.gz" && sudo install -m 0755 "fly-linux-$arch" /usr/local/bin/fly
+```
+
+On a server that runs the agent, FlyWP can install a dev pre-release with `agent.update` instead.
+
+## Roll back and stop
+
+| Need | Do |
+|---|---|
+| Stop a signed release before agents take it | Within 24 hours of the signature, mark it as a pre-release on GitHub. |
+| Fix a bad release | Release a newer, fixed version. The agent never downgrades. |
+| Stop automatic updates on one server | Add `FLY_AGENT_AUTO_UPDATE=off` to `/etc/fly/agent.env` and restart `fly-agent`. FlyWP updates still work. |
+| Stop the agent on one server | `sudo systemctl disable --now fly-agent` |
From c558008a4a088f13f20710338766debaee465e6f Mon Sep 17 00:00:00 2001
From: nabil1440 <52530910+nabil1440@users.noreply.github.com>
Date: Mon, 28 Sep 2026 10:21:30 +0600
Subject: [PATCH 2/2] docs: a dev pre-release updates itself; say why the tag
goes out at once
---
docs/monitoring-agent.md | 8 +++++---
docs/releasing.md | 3 +++
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/docs/monitoring-agent.md b/docs/monitoring-agent.md
index bf8c2b4..32a0951 100644
--- a/docs/monitoring-agent.md
+++ b/docs/monitoring-agent.md
@@ -125,8 +125,9 @@ agent.
- `checksums.txt.sig` has a valid signature from a FlyWP release key that this build trusts;
- the signature is more than 24 hours old.
- Each server installs at its own time of day. A dev build, or a build without a trusted key,
- never updates itself. `FLY_AGENT_AUTO_UPDATE=off` stops this path on one server.
+ Each server installs at its own time of day. A dev pre-release updates itself too. A build
+ without a release version (`make build` between tags, `go build`) never does.
+ `FLY_AGENT_AUTO_UPDATE=off` stops this path on one server.
3. **`sudo fly update`** or `install.sh` install the latest release after checking its sha256 in
`checksums.txt`.
@@ -152,7 +153,8 @@ agent.
| "some files of a site cannot be read" | The disk walk skipped folders; the site value is lower than the real use. Logged once per folder after each start. |
| "a new release installs after its wait" | Normal: a signed release waits 24 hours. |
| "a new release waits for its signature" | The latest release is not signed yet. |
-| "auto-update is off: this build trusts no release key" | A local or dev build. Use a release build. |
+| "auto-update is off: this build has no release version" | A local build. Install a release or a dev pre-release. |
+| "auto-update is off: the agent cannot write the folder of its binary" | The binary is not in a folder of the server user. Install again with `install.sh`. |
To remove the agent from a server:
diff --git a/docs/releasing.md b/docs/releasing.md
index 17a7c18..b912dc7 100644
--- a/docs/releasing.md
+++ b/docs/releasing.md
@@ -14,6 +14,9 @@
git push origin v0.2.0
```
+ Push the tag right after the merge: `install.sh` from `main` installs only a release with
+ `checksums.txt`, so until the release exists it stops.
+
The Release workflow checks that the tag is on `main`, runs `make check`, builds the archives
with `make release`, and publishes the release with `fly-linux-amd64.tar.gz`,
`fly-linux-arm64.tar.gz` and `checksums.txt`. Do not rename these files: installed CLIs look for