Found while going through the selfh.st directory for services the awesome-selfhosted list does not cover.
The gap
The estate monitors CPU, memory, load, storage capacity, network, containers, UPS and four SNMP devices. It does not monitor whether any disk is failing.
That is a real blind spot given what the hardware actually is, per docs/hardware.md:
| Host |
Disk |
Note |
Saruman |
2× 1 TB SAS HDD, RAID 1 |
Spinning enterprise disks. ADR-0007 states the constraint on this machine is storage, not compute — "sized against spindles, not RAM" |
oracle |
500 GB 5400 rpm HDD |
The slowest and oldest disk in the estate |
prometheus |
256 GB SSD |
A 2012 MacBook Pro SSD holding 30 days of metrics and all the logs |
morpheus |
1 TB SSD |
Holds the firewall config |
host.rules.yaml will fire when a filesystem fills. Nothing fires when a drive starts reallocating sectors, and those are entirely different events — the first is predictable and boring, the second is the one that costs the data.
This matters more than usual here because RAID 1 hides exactly this. Saruman's mirror will keep serving reads through a dying disk without complaint. The failure only becomes visible when the second one goes, which is the scenario the mirror exists to prevent and cannot, if nobody was told about the first.
Proposed
Scrutiny — SMART monitoring with a web UI, ~8.1k stars, actively maintained. It wraps smartmontools, tracks attribute history over time (which is the part raw smartctl does not give you), and exposes metrics.
Alternatively, and possibly better given what is already running: smartctl_exporter scraped by the existing Prometheus, with rules in host.rules.yaml alongside everything else. No new UI, no new datastore, and it fits ADR-0004 rather than adding a service. Scrutiny's advantage is the historical view and the fact that it is one container; the exporter's advantage is that it is the pattern this repo already uses everywhere.
Worth deciding between the two rather than defaulting to the one with the nicer screenshots.
Note on the alert
Whichever is chosen, the rule should alert on attribute trend, not just on SMART's overall pass/fail verdict — a disk can be days from failure while still reporting PASSED. Reallocated sector count, pending sectors and offline uncorrectable are the ones worth rules.
Also relevant to #114 and #71 as another thing that should have a bound, and to #95, which will add the first disks in the estate holding data whose loss actually matters.
Which VLAN this touches
🔴 99 — Winterfell (management)
🟢 30 — ImaginationLAN (lab) — Saruman, once #88 puts an agent there
Security review
Related
#88 (Alloy on Saruman and oracle), #81 (no dashboard for the stack itself), #95 (the NAS).
Found while going through the selfh.st directory for services the awesome-selfhosted list does not cover.
The gap
The estate monitors CPU, memory, load, storage capacity, network, containers, UPS and four SNMP devices. It does not monitor whether any disk is failing.
That is a real blind spot given what the hardware actually is, per
docs/hardware.md:Sarumanoracleprometheusmorpheushost.rules.yamlwill fire when a filesystem fills. Nothing fires when a drive starts reallocating sectors, and those are entirely different events — the first is predictable and boring, the second is the one that costs the data.This matters more than usual here because RAID 1 hides exactly this.
Saruman's mirror will keep serving reads through a dying disk without complaint. The failure only becomes visible when the second one goes, which is the scenario the mirror exists to prevent and cannot, if nobody was told about the first.Proposed
Scrutiny — SMART monitoring with a web UI, ~8.1k stars, actively maintained. It wraps
smartmontools, tracks attribute history over time (which is the part rawsmartctldoes not give you), and exposes metrics.Alternatively, and possibly better given what is already running:
smartctl_exporterscraped by the existing Prometheus, with rules inhost.rules.yamlalongside everything else. No new UI, no new datastore, and it fits ADR-0004 rather than adding a service. Scrutiny's advantage is the historical view and the fact that it is one container; the exporter's advantage is that it is the pattern this repo already uses everywhere.Worth deciding between the two rather than defaulting to the one with the nicer screenshots.
Note on the alert
Whichever is chosen, the rule should alert on attribute trend, not just on SMART's overall pass/fail verdict — a disk can be days from failure while still reporting PASSED. Reallocated sector count, pending sectors and offline uncorrectable are the ones worth rules.
Also relevant to #114 and #71 as another thing that should have a bound, and to #95, which will add the first disks in the estate holding data whose loss actually matters.
Which VLAN this touches
🔴 99 — Winterfell (management)
🟢 30 — ImaginationLAN (lab) —
Saruman, once #88 puts an agent thereSecurity review
Sarumansecrets/*.sops.yamldocs/network.mdwith an OUI-truncated MAC — n/aRelated
#88 (Alloy on
Sarumanandoracle), #81 (no dashboard for the stack itself), #95 (the NAS).