Summary
When the login path saturates (#1469), PgDog is completely silent: TCP accept succeeds, the client gives up mid-startup on its own connect_timeout, and nothing appears in logs, pgdog.errors, or any OpenMetrics series. During our worst production event (~130k client-side connect timeouts over ~30 min) every PgDog-side signal was green. We could only diagnose it from client-side APM and kernel TCP counters.
Feature request
Login/startup observability on the OpenMetrics endpoint:
pgdog_login_duration histogram — accept → ReadyForQuery, ideally with a label or separate series splitting the phases (startup read, auth exchange, pool params fetch), so a saturated phase is identifiable.pgdog_logins_in_flight gauge — connections accepted but not yet ReadyForQuery. This is the leading indicator of login-path saturation: it climbs minutes before clients start timing out.pgdog_logins_abandoned counter — client disconnected (or socket error) before login completed. This is exactly the event that was invisible: today it's indistinguishable from an LB health check (which handle_client deliberately ignores on ConnectionReset). Even counting them under a single counter, without logging, would have turned a 30-minute mystery into a one-glance diagnosis.- Optionally
pgdog_logins_total (rate of completed logins) — cheap and makes capacity planning against the per-instance login ceiling trivial.
Why metrics and not logs
At storm rates (thousands/s), per-event logging is unusable and itself a hazard; counters/histograms are the right shape. Log lines for abandoned logins could stay behind log_connections.
Same production deployment as #1452; happy to contribute the implementation if the shape above (or some variant) is acceptable.
Summary
When the login path saturates (#1469), PgDog is completely silent: TCP accept succeeds, the client gives up mid-startup on its own
connect_timeout, and nothing appears in logs,pgdog.errors, or any OpenMetrics series. During our worst production event (~130k client-side connect timeouts over ~30 min) every PgDog-side signal was green. We could only diagnose it from client-side APM and kernel TCP counters.Feature request
Login/startup observability on the OpenMetrics endpoint:
pgdog_login_durationhistogram — accept → ReadyForQuery, ideally with a label or separate series splitting the phases (startup read, auth exchange, pool params fetch), so a saturated phase is identifiable.pgdog_logins_in_flightgauge — connections accepted but not yet ReadyForQuery. This is the leading indicator of login-path saturation: it climbs minutes before clients start timing out.pgdog_logins_abandonedcounter — client disconnected (or socket error) before login completed. This is exactly the event that was invisible: today it's indistinguishable from an LB health check (whichhandle_clientdeliberately ignores onConnectionReset). Even counting them under a single counter, without logging, would have turned a 30-minute mystery into a one-glance diagnosis.pgdog_logins_total(rate of completed logins) — cheap and makes capacity planning against the per-instance login ceiling trivial.Why metrics and not logs
At storm rates (thousands/s), per-event logging is unusable and itself a hazard; counters/histograms are the right shape. Log lines for abandoned logins could stay behind
log_connections.Same production deployment as #1452; happy to contribute the implementation if the shape above (or some variant) is acceptable.