[HWORKS-2912] Service logs are silently dropped from .services: unbounded Kubernetes pod and node labels exhaust the 1000-field mapping limit - #638
Open
o-alex wants to merge 1 commit into
Conversation
o-alex
force-pushed
the
HWORKS-2912
branch
3 times, most recently
from
August 31, 2026 14:21
8d790c5 to
b21e294
Compare
…nded Kubernetes pod and node labels exhaust the 1000-field mapping limit https://hopsworks.atlassian.net/browse/HWORKS-2912 filebeat attaches Kubernetes metadata to every service log document, and the chart now keeps only an explicit set of pod, namespace and node labels so the shared `.services-*` index cannot exhaust OpenSearch's 1000-field mapping limit and start dropping logs. That makes the kept set part of a cluster's configuration surface: an operator who labels their own workloads for routing cannot tell from outside which keys survive collection, and a label that does not survive fails silently. Document the default sets as tables with a purpose per key, and the three lists each dimension takes: the mandatory keys the chart unions in regardless, the base list that replaces the rest, and the extra list that appends. Record the traps that produce silent failures rather than errors. A replacement drops the other default keys, each of which drives a routing branch. Overriding `logs_locations` drops `addKubernetesMetadata` with it, since Helm replaces lists, which leaves log lines arriving while every pipeline branch routing on `kubernetes.*` stops matching. Searching Dashboards for a dotted key returns nothing because label keys are dedotted on the document. Annotations and the deployment/cronjob names are covered as opt-ins, and Services Logs links here, since that is the page a reader arrives from. Signed-off-by: Alex Ormenisan <alex@logicalclocks.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Service Log Labels page documenting which Kubernetes labels are attached to service logs by default, and how to append to or replace that set.
Read
docs/setup_installation/admin/monitoring/services-logs-labels.md; themkdocs.ymlchange is its nav entry.Why
The chart change in logicalclocks/hopsworks-helm#2241 stops filebeat attaching every pod and node label to each log document, because each distinct key becomes a field in the shared
.services-*index and the index stops accepting documents once it passes 1000 fields. That makes the kept set part of a cluster's configuration surface: an operator who labels their own workloads for routing cannot tell from the outside which keys survive collection, and a label that does not survive fails silently.The page therefore documents the defaults as tables with a purpose per key (25 pod, 2 namespace, 1 node), the append and replace paths, annotations and the deployment/cronjob names, and the dedotting rule, since searching Dashboards for a dotted key such as
kubernetes.labels.app.kubernetes.io/namereturns nothing and gives no hint why.It also carries the warning that four defaults are read by the collection itself rather than by a pipeline, so replacing a base list without them stops filebeat excluding its own logs or stops project namespaces being collected, with no error either way.
Verification
npx markdownlint-cli2reports 0 errors on the new file. A strict build (uv run mkdocs build -s) completes with 0 errors and 0 warnings and emits the page. Note that the strict build requiresuv pip install "git+https://github.com/logicalclocks/hopsworks-api.git@main#subdirectory=python"first, per.claude/CLAUDE.md, otherwise it fails onhopsworks_commonin the API reference section.Risk and rollout
Documentation only. The page describes values that ship in the linked chart PR, so it should land with or after it.
Rollback: