CP-47363: Fit the clustered kuttl release on the Kind node - #1006
Merged
Conversation
Greptile SummaryThe PR reduces the resource footprint of the clustered KUTTL release and improves install-failure diagnostics.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Important Files Changed
Reviews (5): Last reviewed commit: "CP-47363: Make the clustered kuttl insta..." | Re-trigger Greptile |
evan-cz
force-pushed
the
CP-47363-kuttl-diag
branch
from
September 4, 2026 19:56
a2d7cb7 to
3d5427b
Compare
roberthocking
approved these changes
Sep 4, 2026
evan-cz
force-pushed
the
CP-47363-kuttl-diag
branch
from
September 4, 2026 20:53
3d5427b to
9de3248
Compare
evan-cz
force-pushed
the
CP-47363-kuttl-diag
branch
from
September 4, 2026 21:58
9de3248 to
182cef9
Compare
The clustered suite fails with a bare "context deadline exceeded", and kuttl deletes the namespace immediately afterwards, so nothing survives to say which container never became ready. That is why this suite went undiagnosed for months: the evidence is discarded before it can be read. Dump pod phases, container states, events, and Alloy logs when the install does not become ready. That dump is what identified the actual cause -- the Alloy container crash-looping on conflicting OpenTelemetry schema URLs, fixed separately in the Alloy fork -- rather than the timeout it presented as. Also shrink the release. The suite installs a second full-size stack beside the default one already on the node, and clustered mode fans the server out to three replicas. That is not what was failing here, since the suite passes without this change once Alloy starts, but the same overlay already exists for the webhookServer suites (CP-43292) and leaves headroom rather than relying on there being enough. The wait goes from three minutes to eight for the same reason as the dump: at three, a slow start and a broken install look identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
evan-cz
force-pushed
the
CP-47363-kuttl-diag
branch
from
September 4, 2026 22:28
182cef9 to
a1d78d6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1004.
The clustered kuttl suite never completes its install, so the whole k8s version matrix fails — and since
docker-buildis a rollup thatneeds: k8s-version-matrix-tests, that blocks merging anything.Cause
The suite installs a second full-size release alongside the default
cz-agentone already on the node. At chart defaults that is another ~1.3 CPU and ~2Gi of requests, and clustered mode fans the server out to three replicas. Pods stayPendingandhelm --waitexpires.Fix
CP-43292 already solved this for the webhookServer suites and added
tests/kuttl/webhookserver-min-footprint.yamlfor exactly this reason. Its own comment names our case:Both webhook suites use that overlay.
alloy-clustered-testnever adopted it. This applies it.This failure predates the stack
The suite's steps silently no-opped until the
commands:conversion (77b42c73, 2026-06-17), and the only matrix run recorded since then failed. This jobneeds: docker-build, which the image scan has been failing for weeks, so the matrix has beenskippedthroughout. Clearing the scan is what made it run again and surfaced this.Also included
State dump on failure — pod phases, container states, events, Alloy logs. kuttl deletes the namespace the moment a step fails, so every previous failure produced a bare
context deadline exceededand nothing else; that is why this went undiagnosed for three months. The wait also goes 3m → 8m, since at three minutes a slow start and a broken install are indistinguishable.🤖 Generated with Claude Code