Import DDNet official stats: points, records, teams, and history - #37
Merged
Conversation
Adds a worker pipeline over the daily ddnet.org dumps (resumable backfill to R2 Parquet + curated Postgres tables, nightly incremental import, online-status history since 2014), and surfaces it in the frontend: map records/checkpoints/thumbnails, player points and best times, mapper pages, proven teammates with duo ranks, finish-based activity back to 2013, and DDNet breakdowns on the home chart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…alysis # Conflicts: # apps/frontend/app/page.tsx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ingests the daily ddnet.org dumps through three new BullMQ jobs: a resumable backfill that streams the 69M-finish dump into month-partitioned Parquet on R2 and drains it into curated Postgres tables (bests with checkpoint splits, points/ranks, records, team runs, partner aggregates, per-day finish counts), an hourly incremental import with watermark dedup and self-healing recounts, and a byte-offset tailer for the since-2014 online-player feeds.
The frontend surfaces it with minimal additions: map headers gain mapper/stars/points/thumbnail and a Records tab (WR-progression chart, solo/team/duo leaderboards, checkpoint splits with player comparison), players gain a DDNet points header segment and tab plus proven-teammates with duo ranks, mapper pages are new, activity calendars extend back to 2013 with a distinct finish ramp, and the home chart gets DDNet total/by-country/by-mod sources. The pipeline was verified end-to-end against a fixture slice of the real dump with exact ground-truth parity, and
/statusreports backfill/import health.Before deploying:
fly volumes extendon teerankio-postgres2 by +20 GB (~14.5 GB of new tables at steady state).🤖 Generated with Claude Code