Skip to content

Prometheus, Loki and Alertmanager bind 0.0.0.0 with no authentication #70

Description

@Gerrrt

.env.example:11 ships BIND_ADDR=0.0.0.0, and the deployed .env on the
monitoring host has the same. Confirmed with ss -ltn on prometheus
(10.0.99.20):

LISTEN  0.0.0.0:9090     prometheus
LISTEN  0.0.0.0:9093     alertmanager
LISTEN  0.0.0.0:3100     loki
LISTEN  0.0.0.0:3000     grafana
UNCONN  0.0.0.0:1514     alloy (syslog)

Only Grafana authenticates. For the other three, anyone who can reach the host
on VLAN 99 can:

  • Prometheus — inject arbitrary metrics (--web.enable-remote-write-receiver)
    and force a config reload (--web.enable-lifecycle);
  • Loki — push arbitrary log lines and issue delete requests
    (auth_enabled: false);
  • Alertmanager — create silences, i.e. switch off any alert.

The last one is the interesting one: silencing an alert is a quiet way to
disable monitoring, and the stack's own record of that action is the thing being
disabled.

This is not currently exploitable from an untrusted segment

Default-deny between VLANs means only VLAN 50 can reach VLAN 99 at all
(docs/adr/0002). So the firewall is the entire control, and there is no
defence in depth behind it — a compromised workstation on the trusted segment
has unauthenticated write access to the metrics and log stores.

.env.example:9-11 already documents that BIND_ADDR can be set to the
management IP. The question is whether the shipped default should be the wide
one.

Options

  • Ship BIND_ADDR=127.0.0.1 and publish only Grafana, which is the only service
    with a login. Alloy's syslog listener has to stay reachable off-host and is
    separately bound.
  • Or keep it and record the residual in SECURITY.md, which already carries an
    exposure table.

Found while verifying #12.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions