Skip to content

CP-47363: Fit the clustered kuttl release on the Kind node - #1006

Merged
evan-cz merged 1 commit into
developfrom
CP-47363-kuttl-diag
Sep 5, 2026
Merged

CP-47363: Fit the clustered kuttl release on the Kind node#1006
evan-cz merged 1 commit into
developfrom
CP-47363-kuttl-diag

Conversation

@evan-cz

@evan-cz evan-cz commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Stacked on #1004.

The clustered kuttl suite never completes its install, so the whole k8s version matrix fails — and since docker-build is a rollup that needs: k8s-version-matrix-tests, that blocks merging anything.

Cause

The suite installs a second full-size release alongside the default cz-agent one already on the node. At chart defaults that is another ~1.3 CPU and ~2Gi of requests, and clustered mode fans the server out to three replicas. Pods stay Pending and helm --wait expires.

                    before overlay          after overlay
ksm            x1   100m / 256Mi            100m / 256Mi
server         x3   100m / 320Mi   →   x1    100m / 320Mi
aggregator     x3   200m / 128Mi   →   x1     20m / 128Mi
webhook        x3   100m / 128Mi   →   x1    100m / 128Mi
                    ~1.3 CPU, ~2Gi         ~320m, ~832Mi

Fix

CP-43292 already solved this for the webhookServer suites and added tests/kuttl/webhookserver-min-footprint.yaml for exactly this reason. Its own comment names our case:

Resource pressure. The default release already requests ~1.7 CPU; a second full-size stack pushed the Kind node past its allocatable CPU, so the new pods stayed Pending. … defaults.replicas: 1 → one replica per component (clustered mode would otherwise fan the server out to 3).

Both webhook suites use that overlay. alloy-clustered-test never adopted it. This applies it.

This failure predates the stack

The suite's steps silently no-opped until the commands: conversion (77b42c73, 2026-06-17), and the only matrix run recorded since then failed. This job needs: docker-build, which the image scan has been failing for weeks, so the matrix has been skipped throughout. Clearing the scan is what made it run again and surfaced this.

Also included

State dump on failure — pod phases, container states, events, Alloy logs. kuttl deletes the namespace the moment a step fails, so every previous failure produced a bare context deadline exceeded and nothing else; that is why this went undiagnosed for three months. The wait also goes 3m → 8m, since at three minutes a slow start and a broken install are indistinguishable.

🤖 Generated with Claude Code

@evan-cz
evan-cz requested a review from a team as a code owner September 4, 2026 19:33
@greptile-apps

greptile-apps Bot commented Sep 4, 2026

Copy link
Copy Markdown

Greptile Summary

The PR reduces the resource footprint of the clustered KUTTL release and improves install-failure diagnostics.

  • Applies the existing minimum-footprint Helm overlay to the clustered test release.
  • Extends the install wait and command timeout for cold Kind nodes.
  • Dumps pod, container, event, and Alloy log state when installation fails.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
tests/kuttl/alloy-clustered-test/steps/01-install-clustered-chart.yaml Applies the shared low-resource test overlay and adds guarded failure diagnostics without changing production chart behavior.

Reviews (5): Last reviewed commit: "CP-47363: Make the clustered kuttl insta..." | Re-trigger Greptile

@evan-cz
evan-cz force-pushed the CP-47363-kuttl-diag branch from a2d7cb7 to 3d5427b Compare September 4, 2026 19:56
@evan-cz evan-cz changed the title CP-47363: Make the clustered kuttl install diagnosable CP-47363: Fit the clustered kuttl release on the Kind node Sep 4, 2026
@evan-cz
evan-cz force-pushed the CP-47363-kuttl-diag branch from 3d5427b to 9de3248 Compare September 4, 2026 20:53
Base automatically changed from CP-47363-reloader to CP-47362-grype-output September 4, 2026 21:33
Base automatically changed from CP-47362-grype-output to CP-47350-go-1267 September 4, 2026 21:38
Base automatically changed from CP-47350-go-1267 to CP-47350-alloy-repin September 4, 2026 21:38
@evan-cz
evan-cz force-pushed the CP-47363-kuttl-diag branch from 9de3248 to 182cef9 Compare September 4, 2026 21:58
Base automatically changed from CP-47350-alloy-repin to develop September 4, 2026 22:27
The clustered suite fails with a bare "context deadline exceeded", and
kuttl deletes the namespace immediately afterwards, so nothing survives
to say which container never became ready. That is why this suite went
undiagnosed for months: the evidence is discarded before it can be read.

Dump pod phases, container states, events, and Alloy logs when the
install does not become ready. That dump is what identified the actual
cause -- the Alloy container crash-looping on conflicting OpenTelemetry
schema URLs, fixed separately in the Alloy fork -- rather than the
timeout it presented as.

Also shrink the release. The suite installs a second full-size stack
beside the default one already on the node, and clustered mode fans the
server out to three replicas. That is not what was failing here, since
the suite passes without this change once Alloy starts, but the same
overlay already exists for the webhookServer suites (CP-43292) and
leaves headroom rather than relying on there being enough.

The wait goes from three minutes to eight for the same reason as the
dump: at three, a slow start and a broken install look identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@evan-cz
evan-cz force-pushed the CP-47363-kuttl-diag branch from 182cef9 to a1d78d6 Compare September 4, 2026 22:28
@evan-cz
evan-cz added this pull request to the merge queue Sep 4, 2026
Merged via the queue into develop with commit efd4119 Sep 5, 2026
46 checks passed
@evan-cz
evan-cz deleted the CP-47363-kuttl-diag branch September 5, 2026 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants