feat(api): route stream resolution through named hosts during the chain migration - #1029
Open
rickyrombo wants to merge 1 commit into
Open
feat(api): route stream resolution through named hosts during the chain migration#1029rickyrombo wants to merge 1 commit into
rickyrombo wants to merge 1 commit into
Conversation
…in migration Plays are recorded by whichever node serves the audio -- logTrackListen runs at the top of mediorum's serveBlob, before it 307s to storage -- and they never travel through the relay, so the queue that carries ManageEntity writes to the new chain does not carry them. That matters for days during the genesis migration. Nodes move to the new chain in batches while the indexer still reads the old one, so every play recorded by an already-migrated node lands on a chain nobody is indexing. The fleet migration is measured in days, and roughly a quarter of plays would be lost across that window. playRoutingHosts, when set, puts those hosts first when resolving a stream URL. Naming nodes that stay on the old chain keeps plays on the chain the indexer is reading. Unset, this is inert. The original url and mirrors are kept behind the routing hosts rather than replaced: store-all nodes hold nearly everything, but a fresh upload has not necessarily replicated, and such a track must still stream -- with the play following whichever node serves it. tryFindWorkingUrl probes in order, so a routing host that cannot serve costs one request and falls through. Bandwidth is not a concern: those nodes 307 to presigned storage rather than streaming bytes, so the added cost is one request and a URL signature per play. Tests cover ordering, fallback retention, dedupe against existing mirrors, that the path and the signature mediorum parses for attribution survive the host rewrite, bare-host and full-URL forms, and that an unparseable link degrades to current behaviour. Confirmed failing with the routing disabled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runbook step 5 (
cmd/genesis-writer/ROLLOUT.md). Inert unless configured.The problem it solves
Plays are recorded by whichever node serves the audio —
logTrackListenruns at the top of mediorum'sserveBlob, before it 307s to storage — and they never travel through the relay. Sonew_chain_queue, which carriesManageEntityLegacywrites across to the new chain, does not carry plays.During the migration the fleet is split across two chains for days (step 10 moves ~10 nodes per jail cycle, ~5 hours apart). Every play recorded by an already-migrated node lands on the new chain while the indexer is still reading the old one — indexed by nobody. Roughly a quarter of plays across that window, feeding trending and rewards.
What it does
playRoutingHostsputs the named hosts first when resolving a stream URL. Point it at nodes that stay on the old chain —creatornode.audius.coandv.monophonic.digital, which are store-all and are held back as rollback anchors anyway — and every play lands on the chain the indexer is reading. Cleared at step 13, after which plays follow the serving node again.Design notes
Original hosts are kept as fallbacks, not replaced. Store-all nodes hold essentially everything, but replication of a fresh upload isn't instant, and a track they don't have yet must still be streamable — with the play following whichever node serves it.
tryFindWorkingUrlprobes in order, so a routing host that can't serve costs one request and falls through. Streaming degrades to current behaviour rather than failing.Bandwidth is not a concern. Those nodes return a 307 to a presigned storage URL rather than streaming bytes (
serve_blob.go,BlobStorageStreaming), so the added load is one request plus a URL signature per play — not audio egress. This matters because the same two nodes are also serving state-sync snapshots and are the rollback anchors.tryFindWorkingUrlis untouched. The candidate list is rewritten before it's called, so this doesn't conflict with #1027.Tests
signaturesurvive the host rewrite — mediorum parses that signature to attribute the listen, so losing it would misattribute or drop the playConfirmed failing with the routing disabled. Existing stream and download tests still pass.
Sequencing
Land before step 10 begins, since that's when the fleet starts splitting. Independent of #1018 and #1028; touches the same file as #1027 but not the same lines.