containers.rules.yaml:59-69:
rate(container_cpu_cfs_throttled_periods_total{name!=""}[15m])
/ clamp_min(rate(container_cpu_cfs_periods_total{name!=""}[15m]), 1) > 0.25
CFS accounting only exists when a CPU quota exists. No service in compose.yaml
sets cpus or deploy.resources, so the cgroup reports nr_periods=0 and
nr_throttled=0 forever and the numerator can never be non-zero.
This is #63 exactly, one rule down in the same file. promtool check rules
passes it — it parses PromQL and never asks whether an expression can be true —
and it shows as loaded and healthy on the Prometheus status page. #63's fix came
with the repo's first promtool test rules cases; they cover ContainerHighMemory
and now PrometheusSizeRetentionActive, but not this one.
The tempting fix is the wrong one
Adding cpus: would make the rule live, and it should not be done. The whole
stack peaks at ~0.4 of 4 cores (alloy 0.20, loki 0.14, everything else under
0.05), and HostHighLoad fires at load15 > 2 × cores = 8. There is no
saturation to prevent, and CPU degrades gracefully anyway — CFS already fair-
shares, so a runaway produces slowness rather than death. Meanwhile a 0.20-core-
average service saturates a modest quota inside individual 100 ms CFS windows
during a scrape burst, so the limit would manufacture the alert fatigue this
repository goes out of its way to avoid, in order to justify a control that
prevents nothing.
What to do instead
- A
promtool test rules case asserting the expression can fire against
synthetic non-zero CFS series, paired with a quiet case, per the convention in
containers.test.yaml's header.
- A comment in the rule saying it is inert until a CPU quota exists, that one
deliberately does not, and what would have to change for it to become live.
That leaves a rule that is honest about its own state rather than one that looks
healthy and is not. If the conclusion is instead that an inert rule should not
ship at all, removing it is also a defensible answer — but it should be a
decision, not an oversight.
Found while doing #71 / #184.
containers.rules.yaml:59-69:CFS accounting only exists when a CPU quota exists. No service in
compose.yamlsets
cpusordeploy.resources, so the cgroup reportsnr_periods=0andnr_throttled=0forever and the numerator can never be non-zero.This is #63 exactly, one rule down in the same file.
promtool check rulespasses it — it parses PromQL and never asks whether an expression can be true —
and it shows as loaded and healthy on the Prometheus status page. #63's fix came
with the repo's first
promtool test rulescases; they coverContainerHighMemoryand now
PrometheusSizeRetentionActive, but not this one.The tempting fix is the wrong one
Adding
cpus:would make the rule live, and it should not be done. The wholestack peaks at ~0.4 of 4 cores (alloy 0.20, loki 0.14, everything else under
0.05), and
HostHighLoadfires atload15 > 2 × cores= 8. There is nosaturation to prevent, and CPU degrades gracefully anyway — CFS already fair-
shares, so a runaway produces slowness rather than death. Meanwhile a 0.20-core-
average service saturates a modest quota inside individual 100 ms CFS windows
during a scrape burst, so the limit would manufacture the alert fatigue this
repository goes out of its way to avoid, in order to justify a control that
prevents nothing.
What to do instead
promtool test rulescase asserting the expression can fire againstsynthetic non-zero CFS series, paired with a quiet case, per the convention in
containers.test.yaml's header.deliberately does not, and what would have to change for it to become live.
That leaves a rule that is honest about its own state rather than one that looks
healthy and is not. If the conclusion is instead that an inert rule should not
ship at all, removing it is also a defensible answer — but it should be a
decision, not an oversight.
Found while doing #71 / #184.