Skip to content

feat(agent): measure the server each minute - #33

Merged
nabil1440 merged 2 commits into
agent/27-outboxfrom
agent/28-metrics
Sep 28, 2026
Merged

nabil1440 merged 2 commits into
agent/27-outboxfrom
agent/28-metrics

Conversation

@nabil1440

@nabil1440 nabil1440 commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Layer 3 of 7 of the monitoring agent (#31 → #37).

Summary

  • This PR measures the server each minute: CPU, load, memory, swap, disk and network traffic. The agent reads Linux files and needs no root.
  • The agent counts only the network cards of the server. It does not count Docker traffic two times.
  • The agent sends the server status: restart necessary, updates that wait, OS, kernel, uptime and CPU type.
  • The agent counts the updates one time each hour, because this takes some seconds.
  • If a measurement fails, the agent writes a log line and continues.

Change

  • internal/metrics: /proc/stat, /proc/loadavg, /proc/meminfo, statfs("/") and /proc/net/dev, as the contract section "What 'used' means" defines them.
  • Network: only the interfaces with a hardware device that are not a port of an other interface. Without such an interface, the default route.
  • The last counters are saved with the boot id. After a restart the next sample continues; after a reboot or a gap, the sample has net_counters_reset: true.

Adversarial review

The first version was refuted. These items are fixed in this PR:

  • On Ubuntu 24.04, apt-check can write warnings before its result, and both counts became 0. Only the last line counts now.
  • A failed count keeps the last counts.
  • The count runs with the sample of the minute, not in the time of the sends.
  • Without a counted interface, the traffic is "not known" (net_counters_reset), not 0.
  • The default route must have mask 0, so a VPN route 0.0.0.0/1 is not taken.
  • A port of an other interface (a bond or bridge port, or the Azure VF) is not counted.

Tests

  • Parsers and the collector with fixture files (any OS), a test on the real /proc (Linux CI), the interface rules, the reset rules and apt-check output with warnings.
  • On a test server (Ubuntu 24.04): disk and memory agree with df and free.

Closes #28

@nabil1440
nabil1440 added this pull request to stack #38 September 22, 2026 10:40
@nabil1440 nabil1440 self-assigned this Sep 22, 2026
@nabil1440 nabil1440 changed the title agent/28 metrics feat(agent): measure the server each minute Sep 22, 2026
- Measure CPU, load, memory, swap, disk and network from /proc and
  statfs("/"). Used memory is MemTotal - MemAvailable; used disk is
  (blocks - free blocks), as the contract defines them.
- Count only the network interfaces that have a hardware device, so that
  container traffic through veth, the bridges and docker0 is not counted two
  or three times. Without such an interface, count the default route.
- Save the last counters with the boot id. After an agent restart the next
  sample continues from them. After a reboot, a counter that went back or a
  reading older than 90 seconds, send 0 with net_counters_reset.
- Send the status in each report: reboot required, the update counts from
  apt-check (each hour; 0 and a warning when it fails), the OS name, the
  kernel, the uptime and the arch.
- A measurement that fails skips that minute. The agent continues.

Refs #28
…4.04

Fixes from the adversarial review of this layer.

- apt-check can write warnings before its result, for example for a source
  that is configured two times. Read only the last line. Before, both counts
  became 0, and a server with security updates looked up to date.
- A failed count keeps the last counts. The counts are 0 only when apt-check
  never gave a result.
- Count the updates with the sample of the minute, before the sends of a
  report, so a slow apt-check cannot use the time of the sends.
- When no interface is counted, mark the traffic as not known
  (net_counters_reset), not as 0, and log it one time.
- The default route must have mask 0, so a VPN route 0.0.0.0/1 is not taken.
- Leave out an interface that is a port of an other interface (a bond or a
  bridge port, or the Azure VF under netvsc): its traffic is also in the
  interface above it.

Refs #28
@nabil1440
nabil1440 merged commit 3097f02 into develop Sep 28, 2026
1 of 2 checks passed
@nabil1440
nabil1440 deleted the agent/28-metrics branch September 28, 2026 03:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

enhancement: Measure the server each minute for the monitoring agent

1 participant