You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#184 gave Prometheus a byte ceiling (--storage.tsdb.retention.size=12GB). That
makes Loki the only unbounded store on the disk, and it is fed by the same class
of endpoint:
auth_enabled: false, published on ${BIND_ADDR} (0.0.0.0 by default)
retention_period: 720h — time only, no size equivalent, because Loki has no
size-based retention
limits_config sets none of ingestion_rate_mb, per_stream_rate_limit or max_global_streams_per_user, so the defaults apply
The volume is 461 MB today against 59 GiB free, so nothing is wrong right now.
The gap is that nothing bounds it, and #182 already tracks the fact that the
push endpoint is unauthenticated.
Be honest that rate limits do not fix this
The obvious response is to set ingestion_rate_mb. It does not solve the
problem it appears to solve. The lab produces roughly 0.012 MB/s of logs, so even
a deliberately tight 0.5 MB/s limit is 40x headroom — and 0.5 MB/s still fills
59 GiB inside a day and a half. Any limit loose enough not to drop real logs is
loose enough to fill the disk. There is no clean number.
That is the reason to file this rather than guess: the honest options are
Put loki-data on its own volume so filling it cannot take the host with it
Record it as an accepted residual in SECURITY.md, which already carries an
exposure table, and rely on HostDiskWillFillIn24h
and picking among them is a decision, not a config tweak. Option 1 is probably
the real answer, which makes this mostly a note that #182 has a second
justification.
Done when
One of the above chosen and written down, or the gap recorded explicitly in SECURITY.md alongside the existing Prometheus/Loki residual
#184 gave Prometheus a byte ceiling (
--storage.tsdb.retention.size=12GB). Thatmakes Loki the only unbounded store on the disk, and it is fed by the same class
of endpoint:
auth_enabled: false, published on${BIND_ADDR}(0.0.0.0 by default)retention_period: 720h— time only, no size equivalent, because Loki has nosize-based retention
limits_configsets none ofingestion_rate_mb,per_stream_rate_limitormax_global_streams_per_user, so the defaults applyThe volume is 461 MB today against 59 GiB free, so nothing is wrong right now.
The gap is that nothing bounds it, and #182 already tracks the fact that the
push endpoint is unauthenticated.
Be honest that rate limits do not fix this
The obvious response is to set
ingestion_rate_mb. It does not solve theproblem it appears to solve. The lab produces roughly 0.012 MB/s of logs, so even
a deliberately tight 0.5 MB/s limit is 40x headroom — and 0.5 MB/s still fills
59 GiB inside a day and a half. Any limit loose enough not to drop real logs is
loose enough to fill the disk. There is no clean number.
That is the reason to file this rather than guess: the honest options are
can write rather than by how fast
loki-dataon its own volume so filling it cannot take the host with itSECURITY.md, which already carries anexposure table, and rely on
HostDiskWillFillIn24hand picking among them is a decision, not a config tweak. Option 1 is probably
the real answer, which makes this mostly a note that #182 has a second
justification.
Done when
SECURITY.mdalongside the existing Prometheus/Loki residualFound while doing #71 / #184.