Conversation
An adapter whose upstream is archived is marked unrunnable, so the nightly skips it and whatever it once published stays at whichever schema was current then. Nothing in the repo could move those records forward, which is 4,574 of the 9,751 records stranded on schema 0.2.2 for hfopenllm_v2 alone. `cron run --backfill` reaches exactly those adapters. It refuses one the schedule owns, because publishing outside the cadence skips the freshness checks that set it, and points at --force-full instead. The refusal for an unrunnable adapter now names the way out, so the next person reaching for it does not edit the catalog to get past the gate. The workflow goes through the cron rather than calling an adapter module directly. That matters for a reason worth stating plainly: record filenames come from uuid4, so publishing outside the cron appends a second copy of every record instead of overwriting, which is how a schema migration doubled this datastore once already. Routing through the cron records fingerprints in the raw store, and the ledger makes a second dispatch a no-op.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
cron run --backfilland a parameterizedAdapter backfillworkflow, so an adapter whose upstream is archived can be carried into the datastore once.The gap
The nightly runs what the catalog marks runnable. An adapter whose source is frozen is deliberately excluded, so whatever it published once stays at whichever schema was current then and nothing can move it forward. Measured across the datastore today: 9,751 records stranded on schema
0.2.2with no current-schema counterpart, of which 4,574 arehfopenllm_v2and 2,428 more sit in 21 other collections that do have adapters.The gate, both ways
--backfillreaches an unrunnable adapter, and is refused for a runnable one, which belongs on the schedule — publishing outside the cadence skips the freshness checks that set it, so the refusal points at--force-fullinstead. The unrunnable refusal now names--backfillas the way out, so nobody edits the catalog to get past it. The override prints the reason it is overriding, into the run log rather than only the catalog.Why it routes through the cron
This is the load-bearing detail. Record filenames come from
uuid.uuid4()athelpers/io.py:597, so publishing outside the cron appends a second copy of every record rather than overwriting. That is exactly how this datastore came to hold two copies of much of its content when the schema moved from0.2.2to0.3.0. Going through the cron writes fingerprints to the raw store, and that ledger makes a second dispatch a no-op instead of a duplication.Related
#293 fixes the
hfopenllm_v2adapter bug this exposed. Datastore PR 215 carries that collection's 4,575 records, produced before this path existed — see the note there about its records being absent from the ledger.1309 tests pass.