Self-host the alert delivery channel
Alerts are delivered to a hosted push service where the topic name is the only access
control. Anyone who learns it can read alerts and publish to them. Alert bodies carry
infrastructure detail — service names, mount points, environment labels — and the write
side matters as much as the read side: a forged notification is worse than a leaked one.
Proposal
Run the notification service on the monitoring host behind the existing tunnel, with its
own users and per-topic access control. One small container, same posture as the other
backends here, and access becomes something we can grant and revoke per person.
Notes
- Don't put it behind an SSO proxy: mobile clients can't complete the handshake and push
breaks silently. Its own auth over TLS is the control.
- It shares fate with the monitoring host. That's the accepted design — the external
dead-man's switch is the out-of-band detector and stays off this path.
- This isn't a routing problem, so it isn't solved by adding a routing engine. Grafana
already embeds one; what's missing is a private channel at the end of it.
Not urgent. The trigger is the team growing beyond one person, when per-person access
starts to matter.
Self-host the alert delivery channel
Alerts are delivered to a hosted push service where the topic name is the only access
control. Anyone who learns it can read alerts and publish to them. Alert bodies carry
infrastructure detail — service names, mount points, environment labels — and the write
side matters as much as the read side: a forged notification is worse than a leaked one.
Proposal
Run the notification service on the monitoring host behind the existing tunnel, with its
own users and per-topic access control. One small container, same posture as the other
backends here, and access becomes something we can grant and revoke per person.
compose.yml, signup disabled, default access denyinfra/ALERT_WEBHOOK_URLNotes
breaks silently. Its own auth over TLS is the control.
dead-man's switch is the out-of-band detector and stays off this path.
already embeds one; what's missing is a private channel at the end of it.
Not urgent. The trigger is the team growing beyond one person, when per-person access
starts to matter.