Test/ram first equivalence - #166
Open
KillerX wants to merge 14 commits into
Open
Conversation
Measured against a 10,000-user quiz spike on bare metal (6-core Ryzen 5 3600,
local PostgreSQL 17). A CPU profile of the burst attributed ~25% of all server
CPU to RSA signing and showed the per-user challenge caches being destroyed
continuously. Fixing both cut p95 from 2.16s to 947ms and p99 from 2.81s to
1.21s on a front-loaded arrival curve; with GC tuning the same profile now
reaches p95 734ms / p99 897ms, and p99 passes its 1000ms target.
Firebase token warmer (internal/services/firebase_token_warmer.go):
Minting a custom token is a local RSA-2048 signature costing ~1ms of CPU.
Cheap once, ruinous in aggregate: 10,000 cold users hitting firebaseToken
within seconds becomes thousands of signatures per second, and bigmod
dominated the profile. The warmer keeps a token cached for every user in the
current project and rotates them on a staggered 40-minute schedule, so the
request path is a pure cache hit. Steady state is ~5 signatures/sec for
13k users. Boot pass mints 10,989 tokens in ~3s at concurrency 4; after that
RSA is absent from the profile entirely and GetFirebaseToken p95 fell from
701ms to 118ms. ES256 would be ~35x cheaper but is not an option: Firebase
custom tokens require RS256 signed with the service-account key.
The cache key now comes from services.FirebaseTokenCacheKey so the warmer and
the firebaseToken resolver cannot drift apart.
cache.InvalidateUserChallengeEnrollment (internal/cache/invalidation.go):
Self-enrollment called InvalidateChallenge, which has no reverse index from a
challenge to its enrolled users and so falls back to DeletePrefix over five
per-user prefixes -- discarding the enrollment, completion, enrolled-challenge
and quiz-access caches for EVERY user. That blast radius is fine when an admin
edits a challenge and ruinous when thousands of users self-enroll at once:
each enrollment throws away the caches the very next ChallengePage request
needs, so the cache never survives to serve anything. A self-enrollment only
changes state scoped to (userID, projectID, challengeID), and all five keys are
directly addressable, so no prefix sweep is needed.
Also in EnrollInChallenge:
- remove a left-in fmt.Printf("DEBUG: ...") that wrote to stdout on every
enrollment; syscall write time was 8.4% of profiled CPU
- resolve the quiz through the cached QuizByChallengeIDLoader instead of a
direct GetQuizByChallengeID query, since this runs on every enrollment
Assisted by Claude Opus 5 (1M context) via Claude Code
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
freetext-quiz-spike.js uses a flat arrival rate, which turns out to hide the
system's real failure mode. Same 10,000 users and same 10-second window, but
front-loading the arrivals degrades p95 by 7x (303ms -> 2.16s) because arrival
rate briefly exceeds service rate and the backlog takes longer to drain than the
burst took to arrive. Total load is not the variable that matters; the shape of
the curve is.
This variant models a scan-the-poster curve, steep at the front and tapering,
hitting these cumulative arrival counts: 1k by second 1, 5k by second 2, 8k by
second 4, 10k by second 10. It is built as four time-phased
constant-arrival-rate scenarios because k6 ramping interpolates between targets,
which makes exact cumulative counts awkward to express.
Notes for anyone extending it:
- iterationInTest is per-scenario, so each phase needs its own TOKEN_BASE or
every phase restarts from token 0 and replays the same users
- token slices carry ~30% headroom: constant-arrival-rate can overshoot its
nominal total, and an overrun reuses a user who has already completed the
quiz, surfacing as "submission already completed" rather than as an obvious
harness fault. The overrun is now detected and reported explicitly.
- needs >= 12,100 tokens, so run tokengen with -limit 13000
- EXPECT_POINTS defaults to 1, so the quiz awards points and the score_journal
write and its trigger fan-out are actually exercised. The flat spike
deliberately uses a 0-point quiz, which means the scoring path had never
been load tested at all.
- RAMP_SCALE scales every phase's rate while preserving the curve, for
separating server cost from load-generator contention
Assisted by Claude Opus 5 (1M context) via Claude Code
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
k6 co-located with the server peaked at 1.7x the server's CPU and skewed
every measurement; this adds the off-box generator setup the ram-first
report calls for. scripts/loadtest-remote.sh mints tokens on the box,
preps + samples it over ssh, and runs k6 locally (make
loadtest-remote-{config,smoke,rampspike}). Spike scripts accept a
BASE_URL override, and the box-only bench artifacts (gendata, fixtures,
on-box runners) are committed under cmd/loadtest/bench/.
Assisted by Claude Fable 5 via Claude Code
…s, web UI - loadtest-remote.sh: LOADTEST_GENERATOR runs k6 on a remote VM (full 10k from a laptop dies in NAT/DDoS middleboxes; a same-DC VM does not) - rampspike: THINK_SCALE compresses per-question think time; token slices scale with RAMP_SCALE so >1x runs don't collide users - cmd/loadtestui: single-binary web UI on the generator VM to fire runs (label/ramp/think/note) and browse history parsed from results/ Assisted by Claude Fable 5 via Claude Code
Parse avg/min/med/max/p90/p95 for http_req_duration plus every
{ name:Op } line; main table shows the full distribution and req/s,
selecting a run shows a per-operation stats table.
Assisted by Claude Fable 5 via Claude Code
Inline-SVG charts, no deps: p50/p90/p95 per run on a log scale across the run history (points carry tooltips), and the cumulative user-arrival curve for the selected run derived from its RAMP_SCALE. Assisted by Claude Fable 5 via Claude Code
The 5-arg update_{team,church,superteam}_leaderboard functions re-ran a
COUNT(DISTINCT) membership aggregate on every point award. member_count
is already maintained by the membership triggers (00080/00081), so the
award path now UPDATEs with the stored value and only COUNTs when the
row is first created.
Measured on the bench box (10k rampspike, think 0.08), together with
GIN_MODE=release: overall p95 1.13s -> 255ms, FinalizeQuiz p95 -> 354ms,
postgres peak CPU 726% -> 624%, first fully passing thresholds run.
Leaderboard deltas verified exact vs score_journal; member_count drift 0.
Applied directly on the bench DB; needs a goose migration for the repo.
Assisted by Claude Fable 5 via Claude Code
…in in prod - New gqlgen ResponseCache extension: queries whose every top-level field is user-independent (currentProject/myCurrentProject) are served as a cached whole response keyed by raw query + language, 30s TTL. Skips field resolution, dataloaders and marshalling on hits. - Production router is gin.New()+Recovery instead of gin.Default(): the access logger writes one synchronous journald line per request even in release mode (~8k/s during spikes). Measured (10k rampspike, think 0.08, off-box): CurrentProject p95 452ms -> 52ms, overall p95 255ms -> 200ms, med 47ms -> 12ms. Assisted by Claude Fable 5 via Claude Code
…s export Replaces the production access log's observability: - middleware.HTTPStats aggregates per-route counts, status classes and latency histograms in memory; logs only 5xx and slow (>2s) requests. - GET /metrics/http serves the cumulative snapshot (p50/p90/p95/p99). - HTTP_STATS_FILE / HTTP_STATS_INTERVAL enable per-interval JSONL dumps: each line covers exactly one window (drained and reset), so averages and percentiles describe that minute alone; idle windows are skipped. - loadtestui: /api/export streams results/ as tar.gz (download link in the header) so run history can be viewed locally with the same binary. Assisted by Claude Fable 5 via Claude Code
… chart The link floated inside #status, which doesn't expand around floats, so the chart rendered on top of it. Status line is now a flex row with the link right-aligned in normal flow. Assisted by Claude Fable 5 via Claude Code
…ics exposure Fixes the divergences pinned by the A/B equivalence run (see notes/ram-first-equivalence-2026-08-25.md): - Response cache: key now includes a variables hash and, whenever the selection touches any user-scoped field (walked recursively with a conservative allowlist, fragments included), the calling user's ID. Cross-user sharing remains only for selections proven user-independent. Fixes the cross-user leak (B1) and variable collisions (B2). - Read-your-own-writes: any successful mutation drops the caller's per-user response entries; user/enrollment invalidation clears them too, and project invalidation clears all response entries (B3 local staleness, B4). - InvalidateUserChallengeEnrollment now broadcasts the same narrow (user, project, challenge) invalidation to other replicas instead of relying on a broad InvalidateUser the resolver no longer sends (B3 cross-replica), keeping the measured low blast radius. - /metrics/http is loopback-only (B5); read it on-host or via SSH tunnel. Verified against main on the test box with the diffcmp harness: warm-path read battery 61/61 MATCH, probes P1-P4 no longer reproduce, /metrics/http externally 404 on both sides. Assisted by Claude Fable 5 via Claude Code
The blank import of pressly/goose/v3/cmd/goose in tools.go pulled in drivers for every dialect goose supports (YDB, ClickHouse, MSSQL, Vertica, Turso/libsql) plus their transitive deps. Migrations run through the goose library and our own cmd/migrate, so the CLI was never needed. Removes ~165 lines from go.mod/go.sum. Assisted by Claude Opus 4.8 via Claude Code
Lockstep GraphQL/webhook replay comparator proving functional equivalence between two server builds on cloned databases, with pinned probes for the response-cache and invalidation bugs found on worktree-ram-first (tagged ram-first-2026-08-25). Findings and method in the notes report. Assisted by Claude Fable 5 via Claude Code
Warm battery 61/61 MATCH, probes P1-P4 no longer reproduce, /metrics/http externally 404, perf smoke within noise of main. Probe fixtures re-pointed for reruns against a database that already carries the first pass's writes. Assisted by Claude Fable 5 via Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.