feat(stacks): karpenter node isolation per organization/stack

[sylr]

Context

Formance runs many customer stacks in a shared Kubernetes cluster. Today there's no way to attribute compute cost to a specific customer/organization/stack — all pods land on shared nodes. This adds the operator's ability to provision dedicated Karpenter node pools per organization (shared <org> pool) or per stack (<org>-<stack> pool), so EC2 instances carry formance.com/{customer,organization,stack} tags for cost allocation and a stack's workloads run only on its dedicated nodes.

What it does

Driven by karpenter.* Settings and the formance.com/organization + formance.com/customer Stack labels:

1. Decides isolation level — karpenter.isolation: organization | stack.
2. Creates an EC2NodeClass by cloning a named cluster-scoped reference (karpenter.ec2-node-class.reference) and injecting spec.tags.
3. Creates a NodePool by cloning a reference (karpenter.node-pool.reference), pointing it at the cloned EC2NodeClass, and injecting node labels + taints.
4. Pins workloads — injects nodeSelector + tolerations into stack Deployments (required, since Karpenter is demand-driven).

Settings

Key	Meaning
karpenter.enabled	master switch (per-stack or *)
karpenter.isolation	organization (default) or stack
karpenter.ec2-node-class.reference	reference EC2NodeClass name to clone
karpenter.node-pool.reference	reference NodePool name to clone
karpenter.api.{ec2-node-class,node-pool}.group-version	GVK override (defaults karpenter.k8s.aws/v1, karpenter.sh/v1)

Design notes

• No new CRD / no second controller — integrated into the existing stacks reconciler (a second For(&Stack{}) would collide on the auto-derived controller name).
• Unstructured, not typed Karpenter deps — avoids the v1beta1→v1 churn; the feature only materializes a configured spec.
• Label-driven cleanup (app.kubernetes.io/managed-by=formance-operator) — removeAllModulesOwnedObjects namespace-filters and can't see cluster-scoped objects, so WithOwn cleanup would silently orphan pools.
• Shared org pools use additive non-controller owner refs (Stack → pool; both cluster-scoped, valid) and survive until the last owning stack is gone. A sweep releases/deletes stale pools on mode change, org relabel, or disable, under optimistic concurrency.
• Scheduling keys are pool-identity only (organization[/stack]) — customer is a tag/node-label for cost attribution but never a taint, since a shared org pool may see differing customer values that would break scheduling.
• Optional CRDs — no-ops when Karpenter isn't installed; a CustomResourceDefinition watch re-enables the feature without an operator restart.

Review

Two independent Codex review passes were incorporated (plan-level and implementation-level): label-driven cleanup for cluster-scoped objects, watch RBAC verb for the CRD watch, mutator idempotency + availability gating, optimistic-concurrency cleanup, and identity-only scheduling keys.

Testing

• go build ./..., go vet, gofmt clean
• Unit tests: resolve_test.go (modes/defaults/pending), karpenter_test.go (clone injection, owner refs, GC on mode change / disable of cluster-scoped objects, org pool surviving a second owner), mutator idempotency test in applications

Out of scope (follow-ups)

• Per-deployment / stack-wide scheduling override keys
• Pinning migration/reindex Jobs (taints only repel, so they still run on shared nodes)
• envtest integration against real Karpenter CRD fixtures

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 28060b85-0434-43ab-9b97-91eabb70925b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/karpenter-node-isolation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gfyrag]

The underlying need needs to be stated more precisely before this becomes an operational contract. If the goal is cost attribution for long-lived Deployments, this is a useful direction. If the goal is node isolation for a customer/stack, the current implementation is incomplete: migration/reindex Jobs explicitly remain on shared nodes, and “dedicated” is therefore not an isolation boundary. Please choose and document one of those contracts rather than using both interchangeably.

I found four blocking convergence/ownership issues:

1. An organization pool is shared, but its full spec and customer tag are recomputed from each stack’s labels and stack-scoped Settings. Two stacks in the same organization can continuously overwrite the same objects with different customer/reference configurations (inline).
2. CreateOrUpdate adopts and rewrites any existing cluster-scoped Karpenter object whose name happens to equal the organization/pool name, even if it was not created by this operator (inline).
3. The CRD and Stack-label watches enqueue only the Stack controller. They do not reconcile the module Deployments where withNodeIsolation actually changes selectors/tolerations, so installing/removing Karpenter or relabeling a Stack leaves existing workloads on stale placement until an unrelated module event occurs (inline).
4. Cloning a reference NodePool replaces its complete taint list instead of injecting only the Formance-managed taints (inline).

There is also a current CI blocker: just pre-commit succeeds but the dirty-tree check reports three generated files missing from the branch:

• docs/09-Configuration reference/02-Custom Resource Definitions.md
• docs/09-Configuration reference/settings.catalog.json
• helm/crds/templates/crds/apiextensions.k8s.io_v1_customresourcedefinition_settings.formance.com.yaml

The focused Go tests pass locally and all Kubernetes E2E jobs in the recorded workflow passed; the workflow failed only on that generated drift.

Additional need question: if per-organization pools are truly shared, where is their single authoritative configuration supposed to live? Per-stack Settings cannot safely own a shared object unless equality is enforced across all participating stacks.

[gfyrag]

Blocking — a shared organization pool cannot be reconciled from stack-local desired state. cfg.PoolName is only the organization in organization mode, but spec, the reference names, and formance.com/customer all come from the currently reconciling stack. The PR explicitly allows stacks in one organization to have different customer values; the second reconcile therefore overwrites the first stack’s EC2 tag/node label, and differing per-stack references make the entire spec last-writer-wins. This can flap indefinitely as either Stack reconciles. Please define one organization-scoped source of truth, enforce that every participating Stack resolves an identical configuration, or include all differing identity/config dimensions in the pool identity.

[gfyrag]

Blocking — do not adopt an arbitrary cluster-scoped object by name. In organization mode the generated name is the raw organization label. If an EC2NodeClass/NodePool with that name already exists but lacks the Formance managed label/owners, CreateOrUpdate still replaces its complete spec, stamps managed-by=formance-operator, and adds an owner reference. A naming collision can therefore destroy a platform-owned Karpenter configuration. Use a collision-resistant operator prefix and/or reject existing objects unless they already carry an explicit ownership marker belonging to this feature.

[gfyrag]

Blocking — this watch does not reconcile workload placement. It enqueues only the Stack reconciler, which creates/cleans Karpenter objects, while selectors and tolerations are applied inside each module’s Deployment reconciliation. A Stack organization relabel therefore creates the new pool and removes the old one but leaves existing Deployments selecting the old labels. The CRD watch above has the same gap on post-start installation/removal. Please fan these events out to the affected modules/Deployments (and test the end-to-end transition), or move placement reconciliation under a controller that actually owns those workloads.

[gfyrag]

Blocking — preserve the reference NodePool’s unrelated taints. The feature is described as cloning a reference and injecting isolation constraints, but SetNestedSlice replaces spec.template.spec.taints wholesale. Baseline taints used for architecture, capacity class, or other scheduling policy disappear from every clone. Read the existing taints and upsert/remove only the keys in ManagedSchedulingKeys, matching the idempotent behavior used for Deployment tolerations.