fix(auth): never crash the api on an orphaned kratos identity; self-heal missing accounts

[islandbitcoin]

Root cause

On 2026-09-01 all three prod api replicas crash-looped (restart count 10, CrashLoopBackOff 20:37–20:47Z) because of one logged-in user.

1. The Kratos after-registration web_hook runs with response.parse: false: the identity is committed whether or not POST /kratos/registration managed to write the account. During today's signup wave the registration write failed 26× with DuplicateKeyForPersistError and 13× with IbexError. At least one identity (created 20:11:35Z, phone_no_password_v0) ended up with a valid session and no Mongo account. A sweep of all 6,208 identities found 19 such orphans (17 older, 2 from today).
2. sessionPublicContext threw mapError(CouldNotFindAccountFromKratosIdError) on every request carrying that session.
3. setGqlContext is an async Express middleware with no error path. The throw became an unhandled promise rejection, and Node 24 exits the process on those (--unhandled-rejections=throw is the default). The user's app retried on a timer, so all replicas died within a second of each other, repeatedly.

Crash tail from the previous container:

UnexpectedClientError: Unexpected error occurred, please try again or contact support if it persists (code: CouldNotFindAccountFromKratosIdError: ebbe2b32-…)
    at mapError (/app/lib/graphql/error-map.js:784:20)
    at sessionPublicContext (/app/lib/servers/middlewares/session.js:33:44)
    at async setGqlContext (/app/lib/servers/graphql-main-server.js:27:24)
Node.js v24.14.0

Changes

1. setGqlContext → src/servers/middlewares/gql-context.ts, wrapped. Never rejects. An AuthenticationError from session resolution is answered as HTTP 200 + GraphQL error NOT_AUTHENTICATED (same reasoning as apiKeyRateLimitMiddleware: the federation router swallows non-2xx subgraph responses into an opaque SUBREQUEST_HTTP_ERROR); anything else is a 500. Logged at error with kratosUserId and recorded on the span.
2. sessionPublicContext self-heals orphans. On CouldNotFindAccountFromKratosIdError it loads the Kratos identity and re-runs createAccountWithPhoneIdentifier from its phone trait — the same write the registration webhook should have done. If the identity has no phone (device accounts) or the write fails again, the original error stands and is answered per (1).
3. mapError: CouldNotFindAccountFromKratosIdError → AuthenticationError instead of the "unexpected error, please try again" catch-all, which invited the client to retry the exact request that crashed the api.
4. unhandledRejection handler in the api entrypoint: logs + records, no exit. No uncaughtException handling added.
5. parseRepositoryError keeps the driver message on DuplicateKeyForPersistError so a failed registration write can be attributed to the colliding index (accounts.kratosUserId vs users.phone). Today's 26 failures could not be.

Tests

New: servers/middlewares/gql-context.spec.ts (resolves + continues; 200/NOT_AUTHENTICATED without rejecting; 500 without rejecting), servers/middlewares/session.spec.ts (existing account; anon; orphan repaired from phone trait; re-raised when no phone / identity load fails / repair write fails), services/mongoose/parse-repository-error.spec.ts, plus a regression case in graphql/error-map.spec.ts.

Test Suites: 226 passed, 226 total
Tests:       3 skipped, 2431 passed, 2434 total
tsc-check: exit 0, 0 errors
eslint (changed files): clean

Known limitation

The repair re-runs the pre-existing registration sequence (users.update → accounts.persistNew → wallet init). If wallet init fails after persistNew, the account exists without wallets — the same partial state the webhook path already produces today. Not made worse here, but not fixed either.

Follow-ups (out of scope)

• Close the orphan-creation side: set can_interrupt: true on the Kratos after-registration web_hook (charts/flash/values.yaml + deployments/tf-modules/flash/kratos-postgres/postgres-values.tmpl.yaml, Kratos is v1.0.0) so a failed account write aborts registration instead of committing an identity.
• src/servers/ws-server.ts still calls sessionPublicContext unguarded inside the graphql-ws context callback.
• Reconcile the 17 pre-existing orphans (or let this PR's self-heal do it on their next login).

🤖 Generated with Claude Code

https://claude.ai/code/session_01EtQzyzQ38thfC2F8BLK4u8

[islandbitcoin]

Simon review: 3 rounds. Round 1 blocker (global CouldNotFindAccountFromKratosIdError → AuthenticationError remap leaking into admin lookups) and its 3 should-fixes were applied in b4dd5e6; round 2's finding (repair dropped phoneMetadata) in 1071a38; round 3's last finding (billed carrier lookup + flat 60s retry on unrepairable orphans) in the commit above: exponential backoff capped at 1h and a per-identity lookup cache. Changed specs: 5 suites / 35 tests green; tsc only the 2 pre-existing Express errors; eslint clean.

🤖 Generated with Claude Code