fix(queue): push tenant account claims to the NATS resolver - #294
Merged
Conversation
/queue/new minted a per-tenant account JWT and handed it to the default no-op ResolverPusher in common/queueprovider/nats, so nats-server never learned the account existed. Against an auth_required server every issued credential failed at CONNECT with "Authorization Violation" — issued-looking credentials that do not work. internal/natsresolver is the missing publisher: one long-lived SYS-account connection opened at boot (auto-reconnecting) that request/replies the signed claim on $SYS.REQ.CLAIMS.UPDATE and requires a positive ack. A non-ack, an unparseable reply, a reply naming a different account, and a timeout are all errors — a claim that may not be installed is never reported as installed. The push is bounded (5s) so a NATS outage 503s instead of hanging the synchronous handler. Wiring fails loudly, never open: - buildQueueProvider attaches the pusher after Factory() through a type assertion that simply misses for non-nats backends. - With NATS_OPERATOR_SEED set, a pusher that cannot be built or cannot reach NATS is a hard error; NewQueueHandler then installs a provider that refuses to issue, so /queue/new answers 503 rather than downgrading to a legacy_open URL the server would reject. - A push rejection at request time tears down the backend resource, marks the row failed and returns 503 (CLAUDE.md rule 2). New env, both secrets — never logged, scrubbed from error strings: NATS_SYSTEM_USER_JWT SYS-account user JWT NATS_SYSTEM_USER_SEED matching NKey seed NATS_SYSTEM_URL optional; default nats://$NATS_HOST:4222 Adds github.com/nats-io/nats.go v1.53.1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was broken
POST /queue/newreturnedauth_mode: legacy_openwith no credentials at all, against a NATSserver that answers anonymous connects with
-ERR 'Authorization Violation'. Every queue customergot a URL they could not connect with.
Three independent layers, all of which had to be fixed for any of it to work:
common/queueprovider/natssigns anaccount JWT and pushes it to
$SYS.REQ.CLAIMS.UPDATE, but the default pusher isnoopPusherand
rg -F SetResolverPusherfound zero production callers — the interface was dead code.The claim was signed, cached, and thrown away; the customer received a credential for an account
the server had never heard of.
resolver: MEMORYis static — it onlyever knows
resolver_preloadaccounts and does not service claims-update. Fixed separately ininfra (
b0f0f28): directory-backed resolver on the JetStream PVC, verified live(
[INF] Managing all jwt in exclusive directory /data/jetstream/jwt).queue_provider.gochoselegacy_open.What this PR adds
A real
ResolverPusherin a newinternal/natsresolverpackage: one long-lived SYS-accountconnection opened at boot with unlimited reconnect,
RequestWithContextagainst$SYS.REQ.CLAIMS.UPDATE, ack required — a non-ack is an error, never swallowed.Wired via
SetResolverPusherafterqueueprovider.Factory(...), behind a type assertion so anon-NATS backend cannot panic.
Fails closed, loudly. If the operator seed is present but the pusher cannot be built or cannot
reach NATS, the api logs
queue.cred_provider_init_failed_isolation_unavailableand/queue/newreturns 503. It does not silently downgrade to
legacy_open— a credential that looks issuedbut does not work is the exact bug being fixed, and reintroducing it one layer up would be worse
than the original.
A second bug found while wiring it
Both
issueTenantCredscall sites discarded the error:queue.go:352(anonymous) and:578(authenticated). So even once the pusher worked, a failedissuance would have produced a resource with unusable credentials and a 200. Both now route
through a shared
failQueueCredIssue— teardown, mark failed, 503. Sites found 2, touched 2.New env contract
NATS_SYSTEM_USER_JWT$SYS_USER_JWT) — secretNATS_SYSTEM_USER_SEEDSU…) — secretNATS_SYSTEM_URLnats://$NATS_HOST:4222Both secrets are
TrimSpaced — k8s--from-filesecrets carry trailing newlines and an untrimmedNKey seed will not parse. Neither is ever logged; every error from the package goes through
redact(). Distinct from the existingNATS_SYSTEM_ACCOUNT_PUBLIC_KEY, which is a public key andcannot authenticate.
These must land together with
NATS_OPERATOR_SEED. Seed without the SYS pair → api boots,logs the isolation-unavailable error, and
/queue/new503s. Deliberate, but it means all three goin as one operator step.
Verification
make gategreen — all 40 packages ok,GATE_EXIT=0.Coverage:
internal/natsresolver100%,internal/config100%,internal/handlers95.5%. Every added or modified function is 100% covered; the two new in-function guards were
verified block-by-block in the profile.
Test matrix covers ack, 2xx-without-account, 5xx data envelope, top-level error envelope, missing
envelope, wrong-account ack, unparseable, empty and nil replies, transport error (with redaction
asserted), timeout, caller cancellation, nil ctx, the non-NATS type-assertion path, and an
end-to-end assertion that issuing a credential actually publishes that account's claim.
Two earlier gate runs were red and both were environment rather than code — chased down rather
than worked around: a dirty
instant_dev_test(duplicate key users_github_id_key, reproduces ona stashed origin/master tree, fixed by
make test-db-reset), and a real-DNS TXT lookup sittingexactly on a 5000ms test budget (
digtakes 5.065s on this host).Known gaps, deliberately not in this diff
Both become material only now that isolation actually works:
TenantCreds.AccountSeedintoresources.queue_account_seed_encrypted(columnexists since mig 060, no writer) — revocation after an api restart is impossible.
RevokeTenantCredentials/RevokeWithSeed, so deleting or reaping a queue resourceleaves its tenant account valid in the resolver; with
allow_delete: falsethe claim persists.Also:
infra/NATS-AUTH-RUNBOOK.md:179is now stale — it tells the operator a no-op pusher isexpected under a MEMORY resolver. With the directory resolver live, an un-acked claim is a fault.
🤖 Generated with Claude Code