CP-47394: Shard Alloy work across replicas

[evan-cz]

Alloy replicas each collect the entire cluster and remote_write all of it, so adding replicas does not reduce per-replica load. On a large cluster the HPA walks to maxReplicas and stays there, while the aggregator receives one copy of the cluster's metrics per replica.

Why clustering was inert

Every pipeline in the River config declares clustering { enabled = true } — but those blocks do nothing unless the process is also started clustered, and replicas can only divide work if they can find each other. Neither held:

1. The flags could never render. In agent-deploy.yaml, --cluster.enabled and --cluster.name were guarded on the agent mode being federated, inside a block already guarded on the mode being clustered. Mutually exclusive — so they have never been rendered in any configuration since they were introduced.

2. There was no headless service, so peer discovery had nothing to resolve.

Each replica therefore formed a single-member consistent-hash ring and owned 100% of targets.

Confirmed against Alloy source rather than inferred:

discovery/peer_discovery.go:67	with neither join nor discover peers set, returns no discovery function
discovery/distributed_targets.go:64-68	belongsToLocal = … || peers[0].Self — one-member ring resolves every key to self
kubestate/sharding.go:50	shouldProcessUID routes into the same path, so the plugin inherits it

The fix

Correct the guard, and add the headless service peer discovery needs. Two properties matter and are not interchangeable with the existing server service:

• clusterIP: None — a normal ClusterIP load-balances to one backend, so a replica would discover at most one arbitrary peer. Headless DNS returns every pod IP.
• publishNotReadyAddresses: true — otherwise a full restart deadlocks: no pod joins the service until ready, and the ring cannot form from pods that cannot see each other.

Port 9090 because Alloy gossips over its HTTP listen port (cmd_run.go passes ListenAddress: fr.httpListenAddr into the cluster service), so it must track --server.http.listen-addr.

Fail-safe by design

--cluster.wait-for-size is deliberately not set. cluster_readonly.go:134 returns Ready() unconditionally true unless that flag is also set, so replicas that cannot reach peers fall back to a single-member ring and process everything — degrading to today's behaviour, never to emitting nothing. Setting it would mean a failed ring produces no metrics at all.

What this does not fix

The per-replica baseline. Every replica still runs informers receiving the full object stream, so the memory floor still scales with cluster size. What now scales horizontally is the processing and emission above that floor — large clusters will still need a higher memory request, but the autoscaler should converge instead of saturating.

Ring identity comes from the resource name

--cluster.name is a gossip label: a node refuses to join peers whose label differs. It is not taken from .Values.clusterName — that value identifies the Kubernetes cluster to CloudZero for cost attribution, is optional, and defaults to empty, so keying the ring on it would label most installations with nothing. It is derived from cloudzero-agent.server.fullname instead, which is release-scoped and always present.

Service name cannot collide with the server Service

The headless Service and the regular server Service are different contracts, so sharing a name means a duplicate resource on install, or an upgrade asking Kubernetes to convert an existing ClusterIP Service to a headless one — rejected as an immutable field change.

Reserving room for the suffix is not sufficient, because appending to a truncated base can rebuild the base. Both a 63-character and a 62-character name ending in -cluster reconstruct themselves, and shortening further just moves the problem to another length. So when the derived name matches, the helper falls back to a different suffix, which is distinct by construction: that branch is only reached when the base ends in -cluster, and the fallback ends in -peers.

Verified across every shape, including a base already ending in the fallback suffix:

63 'a'                  ok
55 'a' + -cluster (63)  ok
54 'a' + -cluster (62)  ok
40 'a'                  ok
57 'a' + -peers   (63)  ok

Verified on a live cluster

Chart tests prove the flags render; only a cluster proves they work. On a 3-replica AKS install of the same topology (mode: clustered, webhookServer: false, kubeState on):

BEFORE   pods re-sharded  total=73  local=73    (one replica owns everything)
AFTER    pods re-sharded  total=75  local=35    peers_count=4

That deployment also surfaced the empty --cluster.name defect, which no amount of chart testing would have caught.

Testing

679/679 helm unit tests, helm lint, schema + kubeconform. The rendered chart was checked to confirm the service selector actually matches the deployment's pod labels — a silent mismatch there would look fine and fix nothing.

The existing unit test asserted the flags were absent, encoding the bug as intended behaviour. It now asserts the opposite, and the new service has its own coverage.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

The PR enables Alloy clustering for clustered-mode deployments and adds a headless Service for peer discovery.

• Derives a release-scoped ring identity and collision-safe discovery Service name.
• Adds Helm coverage for clustering flags, discovery configuration, and naming boundaries.
• Extends the KUTTL suite to deploy two replicas and verify peer membership and divided ownership at runtime.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
helm/templates/_helpers.tpl	Adds a bounded, collision-safe helper for the Alloy peer-discovery Service name.
helm/templates/agent-cluster-service.yaml	Adds a clustered-mode headless Service that publishes all replica addresses before readiness.
helm/templates/agent-deploy.yaml	Enables Alloy clustering and configures release-scoped identity and DNS peer discovery.
helm/tests/alloy_cluster_service_test.yaml	Covers Service rendering, discovery properties, selectors, ports, and prior naming-collision boundaries.
tests/kuttl/alloy-clustered-test/steps/03-verify-clustering.yaml	Adds runtime assertions for a multi-member ring and strict-subset target ownership.

_{Reviews (15): Last reviewed commit: "CP-47394: Shard Alloy work across replic..." | Re-trigger Greptile}

[evan-cz]

Right again, and this exposed that the approach itself was wrong rather than just the constant.

Reproduced the 62-character case: server.fullnameOverride of 54 'a' plus -cluster. Trimming to 55 cuts the separator, trimSuffix removes the trailing dash, and appending rebuilds the base — and the fallback at 54 does exactly the same thing. Shortening can never fix this, because the reconstruction just recurs at another length.

The fallback now uses a different suffix instead of a shorter one. That is distinct by construction: the branch is only reached when the base ends in -cluster, and the fallback ends in -peers, so they cannot be equal for any input.

Verified across every shape, including a base that already ends in the fallback suffix:

base 63 'a'                -> ok
base 55 'a' + -cluster (63) -> ok
base 54 'a' + -cluster (62) -> ok
base 40 'a'                -> ok
base 57 'a' + -peers (63)  -> ok

Both boundary cases are pinned by tests; 679/679 passing.

@greptileai review