feat(observability): send metrics and logs to Grafana Cloud

[nourshoreibah]

Why

The lambdas have carried OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS since 2dacc6d, but nothing in the repo ever read them — so the Grafana Cloud account has been receiving nothing. Observability today is Sentry (uncaught throws) plus unstructured console.* in CloudWatch, which can't answer "is the API slow", "which endpoint is erroring", or "how long does a report take".

What

A new file:-linked package, @branch/lambda-telemetry, exporting OTLP metrics and structured logs — wired into the two places every request already passes through, so all six services are covered without per-service work.

dispatch() now records, for every route in every service:

Metric	Kind	Labels
http.server.requests	counter	method, route, status
http.server.request.duration	histogram (ms)	method, route, status
faas.cold_starts	counter	—
http.server.auth_failures	counter	method, route, unauthenticated | forbidden
http.server.unhandled_errors	counter	method, route
db.client.operation.duration	histogram (ms)	statement kind, outcome

The last one comes from log: kyselyTelemetryLog on each lambda's Kysely instance, which also logs slow (>500ms) and failed queries.

Domain metrics, recorded explicitly from controllers:

• branch.auth.logins — a failure spike is the credential-stuffing signal
• branch.auth.registrations — invitation_required shows people bouncing off the invite gate
• branch.users.invited
• branch.projects.changed — deletes cascade into expenditures, reports and S3
• branch.expenditures.changed / .amount — the approval funnel and spend by category
• branch.donations.recorded / .amount
• branch.reports.generated / .generation.duration / .size — the slowest path in the backend, nearest the 30s timeout

Logs go to stdout as JSON (CloudWatch unchanged) and to Loki, carrying request id, service, route, status, duration, cold-start flag and caller. serverError() routes through the same logger, so every 500 is structured; an Error is flattened rather than serialised to {}.

Design notes

• No Terraform change. The env vars were already on the functions.
• Telemetry never breaks a request. Every recorder and the flush swallow their own failures; the flush is capped at 2s so a slow Grafana can't become a 30s lambda timeout.
• No unbounded labels. Route patterns not paths, statement kinds not SQL, no ids or email addresses.
• Delta temporality — cumulative counters from short-lived Lambda containers read as a reset storm to Mimir.
• Off is a valid state. With no endpoint the OTel SDK is never required, so local dev and tests neither export nor pay the load cost. NODE_ENV=test also drops LOG_LEVEL to error so the access log doesn't drown jest output.
• Preview stacks inherit the OTLP vars (preview-env.yml copies the live lambda env), so their telemetry arrives tagged pr-<N> — the same treatment their Sentry errors get.

Design doc: docs/superpowers/specs/2026-09-06-grafana-telemetry-design.md

Cost

OTel adds 245 KB minified per bundle (528 KB total, 136 KB zipped).

Testing

• Shared packages green in CI's own dependency order: lambda-telemetry -> rbac -> types -> lambda-auth -> lambda-http — 28 / 30 / 30 / 52 tests. 13 of those are new tests covering the dispatch instrumentation, and 28 cover the telemetry package (including one that builds the real OTel providers, so an SDK API change fails loudly instead of silently dropping metrics).
• All six lambdas typecheck and bundle.
• Lambda unit suites: 81 auth, 61 projects, 80 expenditures, 41 users, 86 reports.
• Not run: the e2e suites — they need the Postgres container, and Docker wasn't available in this environment.

🤖 Generated with Claude Code