Make the globe global: translingual feed, NWS rebalance, drop dead indices - #3
Open
brianlo06 wants to merge 3 commits into
Open
Make the globe global: translingual feed, NWS rebalance, drop dead indices#3brianlo06 wants to merge 3 commits into
brianlo06 wants to merge 3 commits into
Conversation
The English lastupdate.txt runs ~53% US. Measured over a live 15-minute slot, the translingual feed — GDELT's machine translation of 65 other languages — runs 5.4%, and its top countries are Ukraine, Italy, Germany, Russia, Mexico, Brazil, Turkey and Spain rather than a US/UK duopoly. It also carries comparable volume (1499 insertable articles vs 1049), so running both takes the combined stream to roughly 25% US. Same format, so ingest_gkg_once() takes translation=True and switches index URL, watermark and source label together; the two feeds never share a cursor. The one wrinkle is that GDELT publishes the translingual GKG up to ~45 minutes behind the timestamp its own index advertises, so the advertised file usually 404s. first_available() walks back in 15-minute slots to the newest file that exists, rather than losing the cycle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TDChXPFeJKSfoVudk2GKwP
NWS is a US-only firehose: ~225 alerts active at any moment, ~1700 in a 48h window, against ~1000 articles per GDELT slot. On the old scale a *Minor* alert scored 0.40 — above the frontend's 0.3 floor and above the median GKG article — so routine US weather dominated the globe and the cluster list. Before this change /events/recent returned 1740 NWS rows out of 1945, and the five largest clusters were all thunderstorm warnings. Drop Minor and Unknown at ingest (~60% of the feed, and not world news), and rescale the rest so only genuinely severe weather competes: Extreme 0.95 -> 0.85, Severe 0.80 -> 0.60, Moderate 0.60 -> 0.35. Tornado and hurricane warnings still surface as breaking; a routine thunderstorm advisory no longer outranks a foreign election. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TDChXPFeJKSfoVudk2GKwP
Stooq retired its /q/l/ quote-CSV endpoint. All 16 instruments 404 with "The page you requested does not exist or has been moved" — verified against every symbol, with both a bot and a browser User-Agent, on stooq.com and stooq.pl. Their history CSV routes now serve a JavaScript anti-bot challenge, so there is no drop-in URL fix. Worse than the log noise: the rows were still being served. /markets was returning 14 index quotes last updated 2026-06-07, rendered on the globe as if live. Delete the ingester and its runner, and drop the stale rows. The 17 Frankfurter FX pins are unaffected — they use "FX:" symbols, cover the same financial centres, and are working. Numbered 019 to leave 018 free for the probes migration that currently exists only on the prod box. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TDChXPFeJKSfoVudk2GKwP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes that together address why the globe reads as US-only. Separate commits, independently revertable.
1. Ingest GDELT's translingual feed
lastupdate.txtis GDELT's English-only index and runs ~53% US on its own. The translingual feed — machine translation of 65 other languages — was measured over a live slot at 5.4% US:Running both should take the combined stream to roughly 25% US.
Same format, so
ingest_gkg_once(translation=True)switches index URL, watermark and source label together — the feeds never share a cursor.The wrinkle: GDELT publishes the translingual GKG up to ~45 minutes behind the timestamp its own index advertises, so the advertised file usually 404s.
first_available()walks back in 15-minute slots to the newest file that exists. Verified live: index said20260904220000, walk-back resolved20260904210000, 1375 insertable articles, 5.3% US.2. Stop NWS crowding out world news
NWS is US-only and uncapped: ~225 alerts active at any moment, ~1700 per 48h, against ~1000 articles per GDELT slot. On the old scale a Minor alert scored 0.40 — above the frontend's 0.3 floor and above the median GKG article (0.55). Measured before this change:
Drop
MinorandUnknownat ingest (~60% of the feed), and rescale the rest so only severe weather competes: Extreme0.95 -> 0.85, Severe0.80 -> 0.60, Moderate0.60 -> 0.35. Tornado and hurricane warnings still surface as breaking.3. Remove the dead Stooq ingester
All 16 instruments 404 — verified per symbol, with bot and browser User-Agents, on both
stooq.comandstooq.pl. Their history CSV routes now serve a JS anti-bot challenge, so there's no URL fix.Worse than log noise: the stale rows were still being served.
/marketsreturned 14 index quotes last updated2026-06-07, drawn on the globe as if live.sql/019drops them. The 17 Frankfurter FX pins are unaffected and working.Numbered 019 to leave 018 free for the probes migration that currently exists only on the prod box.
Verification
231 passed, 10 subtests passed— 220 existing plus 11 new covering the slot walk-back (including network errors mid-walk and the window boundary) and the severity scale.Capacity note before deploying
The translingual feed roughly doubles articles per cycle, and every one gets embedded. The ingest container is capped at
cpus: 1.0on a 2 GB box and a run already takes ~17 minutes of CPU against a 15-minute loop. Worth watchingdocker compose logs ingestfor cycle time after deploying — if runs stretch past the interval, the fix is to raise the sleep or cap articles per slot, not to drop the feed.🤖 Generated with Claude Code
https://claude.ai/code/session_01TDChXPFeJKSfoVudk2GKwP