test(k8s): pin Kubernetes workload identity against a real cluster

[lakhansamani]

Asked to make sure nothing breaks in Kubernetes and to add k3d/kind tests. The
tests found something bigger than the change that prompted them.

The finding

In-cluster Kubernetes workload identity cannot work. Not TokenReview
specifically — the whole path.

Measured against a real kind cluster:

What the cluster publishes	Address	SafeHTTPClient
issuer	https://kubernetes.default.svc.cluster.local	ClusterIP → refused
jwks_uri	https://172.27.0.2:6443/openid/v1/jwks	RFC 1918 → refused
apiserver	https://127.0.0.1:61411	loopback → refused

validators.SafeHTTPClient rejects private, loopback and link-local addresses
unconditionally, and clientauth has no SafeHTTPClientAllowPrivate escape
hatch (the two that exist are gated on --env=e2e and belong to oauth_sso and
webhook delivery).

Meanwhile performTokenReview reads Authorizer's own in-cluster
ServiceAccount token (/var/run/secrets/kubernetes.io/serviceaccount/token) to
authenticate to an apiserver it can never dial. The two assumptions contradict
each other.

performTokenReview already documented half of this. The JWKS half was
undocumented and is the wider problem: it applies with or without TokenReview,
because static_jwks_url and oidc_discovery both point at the cluster's own
private address. That is why this is not a kind artefact — kubernetes.default.svc
is always a ClusterIP, and jwks_uri always points at the apiserver.

The operator-visible symptom, captured by the test:

400 {"error":"invalid_client","error_description":"Client authentication failed"}

Nothing points at the refused fetch. Worth fixing whenever the address problem is.

Why these tests, and why they assert failure

They assert current behaviour, deliberately. They are a pin, not an
aspiration: if someone changes the SSRF policy, the suite says precisely which
Kubernetes behaviour they changed. TestK8sWorkloadAuthenticationEndToEnd names
itself as the assertion to invert once this is fixed.

Three cases, and the middle one earns its place: TestK8sProjectedTokenIsWellFormed
proves the token side is correct — real iss, sub, audience-bound aud — so
the end-to-end failure is attributable to the fetch address and nothing else.
Without it a reader could reasonably blame the token.

Mechanics

• scripts/k8s-e2e.sh — kind by default, K8S_RUNTIME=k3d for k3d, K8S_KEEP=1
  to leave the cluster up. Always tears down, including on failure.
• make test-k8s
• .github/workflows/k8s.yml — scheduled weekly plus PRs touching the code that
  decides cluster reachability. Not per-push: it provisions a cluster to guard a
  limitation that changes rarely.
• Behind the k8s build tag. Verified make test is unaffected — 43 packages,
  0 FAIL, and the file is not compiled into normal runs.

Verification

Cold run from no cluster: create → test → teardown, exit 0, no cluster left behind.

go build ./...                          OK
go vet ./... && go vet -tags k8s ./...  OK
make test                               exit 0 — 43 packages, 0 FAIL
make lint                               exit 0
make test-k8s                           exit 0 — 3 tests, real kind cluster

Not fixed here

Making in-cluster workload identity actually work needs a security decision, not
a patch: a scoped SSRF exemption for an operator-declared apiserver/JWKS host
with CA pinning, or an explicit in-cluster transport. SafeHTTPClientAllowPrivate
asks for careful review before a third caller, and this would be one. README now
states the constraint instead of listing the feature unqualified.