Skip to content

fix(queue): push tenant account claims to the NATS resolver - #294

Merged
mastermanas805 merged 1 commit into
masterfrom
fix/nats-resolver-pusher
Aug 13, 2026
Merged

fix(queue): push tenant account claims to the NATS resolver#294
mastermanas805 merged 1 commit into
masterfrom
fix/nats-resolver-pusher

Conversation

@mastermanas805

Copy link
Copy Markdown
Member

What was broken

POST /queue/new returned auth_mode: legacy_open with no credentials at all, against a NATS
server that answers anonymous connects with -ERR 'Authorization Violation'. Every queue customer
got a URL they could not connect with.

Three independent layers, all of which had to be fixed for any of it to work:

  1. The resolver could not learn about tenant accounts. common/queueprovider/nats signs an
    account JWT and pushes it to $SYS.REQ.CLAIMS.UPDATE, but the default pusher is noopPusher
    and rg -F SetResolverPusher found zero production callers — the interface was dead code.
    The claim was signed, cached, and thrown away; the customer received a credential for an account
    the server had never heard of.
  2. The server wouldn't have accepted the push anyway. resolver: MEMORY is static — it only
    ever knows resolver_preload accounts and does not service claims-update. Fixed separately in
    infra (b0f0f28): directory-backed resolver on the JetStream PVC, verified live
    ([INF] Managing all jwt in exclusive directory /data/jetstream/jwt).
  3. The api never supplied the operator seed, so queue_provider.go chose legacy_open.

What this PR adds

A real ResolverPusher in a new internal/natsresolver package: one long-lived SYS-account
connection opened at boot with unlimited reconnect, RequestWithContext against
$SYS.REQ.CLAIMS.UPDATE, ack required — a non-ack is an error, never swallowed.

Wired via SetResolverPusher after queueprovider.Factory(...), behind a type assertion so a
non-NATS backend cannot panic.

Fails closed, loudly. If the operator seed is present but the pusher cannot be built or cannot
reach NATS, the api logs queue.cred_provider_init_failed_isolation_unavailable and /queue/new
returns 503. It does not silently downgrade to legacy_open — a credential that looks issued
but does not work is the exact bug being fixed, and reintroducing it one layer up would be worse
than the original.

A second bug found while wiring it

Both issueTenantCreds call sites discarded the error:

tenantCreds, _ := h.issueTenantCreds(ctx, tokenStr, creds.SubjectPrefix)

queue.go:352 (anonymous) and :578 (authenticated). So even once the pusher worked, a failed
issuance would have produced a resource with unusable credentials and a 200. Both now route
through a shared failQueueCredIssue — teardown, mark failed, 503. Sites found 2, touched 2.

New env contract

Var Meaning
NATS_SYSTEM_USER_JWT SYS-account user JWT (runbook's $SYS_USER_JWT) — secret
NATS_SYSTEM_USER_SEED matching user NKey seed (SU…) — secret
NATS_SYSTEM_URL optional; defaults to nats://$NATS_HOST:4222

Both secrets are TrimSpaced — k8s --from-file secrets carry trailing newlines and an untrimmed
NKey seed will not parse. Neither is ever logged; every error from the package goes through
redact(). Distinct from the existing NATS_SYSTEM_ACCOUNT_PUBLIC_KEY, which is a public key and
cannot authenticate.

These must land together with NATS_OPERATOR_SEED. Seed without the SYS pair → api boots,
logs the isolation-unavailable error, and /queue/new 503s. Deliberate, but it means all three go
in as one operator step.

Verification

make gate green — all 40 packages ok, GATE_EXIT=0.

Coverage: internal/natsresolver 100%, internal/config 100%, internal/handlers
95.5%. Every added or modified function is 100% covered; the two new in-function guards were
verified block-by-block in the profile.

Test matrix covers ack, 2xx-without-account, 5xx data envelope, top-level error envelope, missing
envelope, wrong-account ack, unparseable, empty and nil replies, transport error (with redaction
asserted), timeout, caller cancellation, nil ctx, the non-NATS type-assertion path, and an
end-to-end assertion that issuing a credential actually publishes that account's claim.

Two earlier gate runs were red and both were environment rather than code — chased down rather
than worked around: a dirty instant_dev_test (duplicate key users_github_id_key, reproduces on
a stashed origin/master tree, fixed by make test-db-reset), and a real-DNS TXT lookup sitting
exactly on a 5000ms test budget (dig takes 5.065s on this host).

Known gaps, deliberately not in this diff

Both become material only now that isolation actually works:

  • Nothing persists TenantCreds.AccountSeed into resources.queue_account_seed_encrypted (column
    exists since mig 060, no writer) — revocation after an api restart is impossible.
  • Nothing calls RevokeTenantCredentials/RevokeWithSeed, so deleting or reaping a queue resource
    leaves its tenant account valid in the resolver; with allow_delete: false the claim persists.

Also: infra/NATS-AUTH-RUNBOOK.md:179 is now stale — it tells the operator a no-op pusher is
expected under a MEMORY resolver. With the directory resolver live, an un-acked claim is a fault.

🤖 Generated with Claude Code

/queue/new minted a per-tenant account JWT and handed it to the
default no-op ResolverPusher in common/queueprovider/nats, so
nats-server never learned the account existed. Against an
auth_required server every issued credential failed at CONNECT with
"Authorization Violation" — issued-looking credentials that do not
work.

internal/natsresolver is the missing publisher: one long-lived
SYS-account connection opened at boot (auto-reconnecting) that
request/replies the signed claim on $SYS.REQ.CLAIMS.UPDATE and
requires a positive ack. A non-ack, an unparseable reply, a reply
naming a different account, and a timeout are all errors — a claim
that may not be installed is never reported as installed. The push
is bounded (5s) so a NATS outage 503s instead of hanging the
synchronous handler.

Wiring fails loudly, never open:
- buildQueueProvider attaches the pusher after Factory() through a
  type assertion that simply misses for non-nats backends.
- With NATS_OPERATOR_SEED set, a pusher that cannot be built or
  cannot reach NATS is a hard error; NewQueueHandler then installs a
  provider that refuses to issue, so /queue/new answers 503 rather
  than downgrading to a legacy_open URL the server would reject.
- A push rejection at request time tears down the backend resource,
  marks the row failed and returns 503 (CLAUDE.md rule 2).

New env, both secrets — never logged, scrubbed from error strings:
  NATS_SYSTEM_USER_JWT   SYS-account user JWT
  NATS_SYSTEM_USER_SEED  matching NKey seed
  NATS_SYSTEM_URL        optional; default nats://$NATS_HOST:4222

Adds github.com/nats-io/nats.go v1.53.1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mastermanas805
mastermanas805 merged commit cb8d2f8 into master Aug 13, 2026
17 of 18 checks passed
@mastermanas805
mastermanas805 deleted the fix/nats-resolver-pusher branch August 13, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant