Skip to content

Loki is now the only unbounded store on the disk #189

Description

@Gerrrt

#184 gave Prometheus a byte ceiling (--storage.tsdb.retention.size=12GB). That
makes Loki the only unbounded store on the disk, and it is fed by the same class
of endpoint:

  • auth_enabled: false, published on ${BIND_ADDR} (0.0.0.0 by default)
  • retention_period: 720h — time only, no size equivalent, because Loki has no
    size-based retention
  • limits_config sets none of ingestion_rate_mb, per_stream_rate_limit or
    max_global_streams_per_user, so the defaults apply

The volume is 461 MB today against 59 GiB free, so nothing is wrong right now.
The gap is that nothing bounds it, and #182 already tracks the fact that the
push endpoint is unauthenticated.

Be honest that rate limits do not fix this

The obvious response is to set ingestion_rate_mb. It does not solve the
problem it appears to solve. The lab produces roughly 0.012 MB/s of logs, so even
a deliberately tight 0.5 MB/s limit is 40x headroom — and 0.5 MB/s still fills
59 GiB inside a day and a half. Any limit loose enough not to drop real logs is
loose enough to fill the disk. There is no clean number.

That is the reason to file this rather than guess: the honest options are

  1. Authenticate the push endpoint (Authenticate the Prometheus and Loki ingest ports #182), which makes the volume bounded by who
    can write rather than by how fast
  2. Put loki-data on its own volume so filling it cannot take the host with it
  3. Record it as an accepted residual in SECURITY.md, which already carries an
    exposure table, and rely on HostDiskWillFillIn24h

and picking among them is a decision, not a config tweak. Option 1 is probably
the real answer, which makes this mostly a note that #182 has a second
justification.

Done when

  • One of the above chosen and written down, or the gap recorded explicitly in
    SECURITY.md alongside the existing Prometheus/Loki residual

Found while doing #71 / #184.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions