Skip to content

feat: add --resume for incremental snapshot exports - #11

Open
gacevicljubisa wants to merge 33 commits into
mainfrom
feat/resume-flag
Open

feat: add --resume for incremental snapshot exports#11
gacevicljubisa wants to merge 33 commits into
mainfrom
feat/resume-flag

Conversation

@gacevicljubisa

@gacevicljubisa gacevicljubisa commented Aug 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Adds export --resume for incremental snapshots: continue a previous export from where it stopped instead of re-exporting every block from the contract's start.

# Recommended: new snapshot from the archived one — the previous file is never modified
batch-export export --resume snapshots/2026-07.ndjson.gzip --output snapshots/2026-08.ndjson.gzip

# In-place variant (omit --output, or name the input)
batch-export export --resume export.ndjson.gzip

Design

  • --resume (input) composes with --output (destination). Copy mode raw-copies the input's clean content and appends newly fetched entries; the input is opened read-only. The same file under any spelling — absolute vs relative, case-insensitive filesystem, symlink, hardlink — is detected via os.SameFile and treated as in-place, so copy mode can never truncate its own input.
  • Strict trust model. A resume input is this tool's own output, validated before anything is written. The only tolerated irregularity is the tool's own interrupted final write (one unterminated trailing line / one gzip member without its trailer) — not copied, or truncated in place with a warning, then re-fetched from the chain. Anything else — a non-log line, foreign data, an alien gzip member, a checksum mismatch — is refused (ErrNotAnExport, naming a byte offset for plain files or member+line for gzip) rather than repaired.
  • Inclusive re-query + skip filter: the cursor block is re-queried and entries already present are dropped — no gaps, no duplicates. --end works as usual (pin it for a deterministic snapshot; default 0 = latest block).
  • Gzip continuation appends a new member (RFC 1952: concatenated members are one stream — gzcat, gunzip, Go, Python all read it transparently). Measured on the real 90 MB export: a year of monthly continuations costs +0.021% in size. gzcat old.gzip | gzip > fresh.gzip consolidates any time.
  • Format by magic bytes, never extension; output format = input format; --compress is always ignored on resume (a plain-twin recompression would overwrite an independently continued archive). Gzip stores no filename inside, so the README recommends the *.ndjson.gzip double extension — extraction then yields a .ndjson file.
  • Failures reach the exit code: the saver's error cancels the fetch, every exit path waits for the final flush, and a failed save exits non-zero — verified empirically (read-only destination, SIGINT mid-run with a clean gzcat afterwards, ENOSPC on a tiny volume). An internal save failure is reported as itself, never dressed up as a user cancellation.

Backward compatibility

Verified against the real pre-branch archives: the full 90 MB export.ndjson / 15 MB export.ndjson.gzip (157,922 entries) pass strict validation with the correct cursor, resume cleanly in both modes, and round-trip byte-identically.

Testing

First tests in the repo: strict-validation and refusal tables (13+ refusal shapes across both formats, incl. CRC corruption), copy/in-place round trips, interrupted-write recovery for the three crash shapes, the same-file-spelling guard (RED-verified against the unguarded code), gzip member-boundary exactness, and an end-to-end append-resume cycle. go test -race, go vet, golangci-lint clean. cmd/export.go has no unit harness (needs live RPC); its three exit-code guarantees were verified manually as above.

Known limitations (documented)

  • Resume assumes the same chain and contract; a wrong --endpoint is not detected (a header would change the format batch-archive consumes).
  • The input is not re-validated between the cursor read and the append, and two concurrent in-place resumes of the same file are unsupported — recommended follow-up: size/mtime revalidation in PrepareOutput.
  • Plain-file validation covers the tail; gzip validates every line as a side effect of decoding.

🤖 Generated with Claude Code

gacevicljubisa and others added 30 commits August 26, 2026 16:07
@gacevicljubisa @claude
docs: add implementation plan for the --resume flag
dbb0b96 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa
feat(resume): read the last saved log from a previous export
8b1c89f
@gacevicljubisa
test(resume): cover a valid line straddling a window boundary
47dfba6 
TestReadCursor never exercised a valid log line split across the
64 KiB backward-read window boundary in lastCursorPlain; every prior
case had the target line either wholly inside the last window or was
garbage that fails to parse regardless of reconstruction order. Add a
case that deterministically positions a real line across the boundary
and asserts the straddle before running, so a broken carry
concatenation order regresses this case specifically.
@gacevicljubisa
feat(gzipstore): add AppendWriter for appending a gzip member
e216130
@gacevicljubisa
feat(filestore): add AppendLogsAsync with a skip filter
532ffb1 
Adds AppendWriter and AppendLogsAsync alongside the existing
SaveLogsAsync so a resumed export can append to an existing NDJSON
file instead of overwriting it, dropping logs a skip function (e.g.
resume.Cursor.Skip) reports as already written. filestore takes a
bare func(types.Log) bool rather than importing pkg/resume, keeping
the dependency one-directional.

Test helper fix beyond the brief: feed() must give each log a
non-nil Topics slice — go-ethereum's generated Log.UnmarshalJSON
rejects a null "topics" field as missing, so a nil Topics (the
brief's literal feed()) fails blocksIn's round trip regardless of
the implementation under test.
@gacevicljubisa
test(resume): add append-resume round trip covering filestore and gzi…
35eb9f1 
…pstore

Closes a gap in the plan: no task specified the append-then-re-read
round trip promised for pkg/resume, even though it needs the
filestore.AppendLogsAsync/AppendWriter and gzipstore.AppendWriter this
task completes. Adds TestAppendResumeRoundTrip, table-driven over
plain NDJSON and gzip, that builds a starting export with a genuinely
partial final block, resumes it through resume.Read + the matching
append writer + AppendLogsAsync(skip: cursor.Skip), and asserts:
  - the cursor lands on the last saved log
  - a boundary-block log at a higher index than the cursor is written,
    not skipped, since it was never saved
  - the full sequence after append is exactly original + new logs,
    with no duplicates and nothing dropped
  - the original bytes remain an unchanged prefix of the appended
    file (for gzip, proof a genuine second member was added rather
    than the archive being decompressed and rewritten)
  - resume.Read on the appended file advances to the newly written
    tail, so a resumed export is itself resumable
@gacevicljubisa
feat(export): add --resume to continue a previous export
441a66f
@gacevicljubisa
docs: document the --resume flag
a473b3d
@gacevicljubisa @claude
fix(resume): never report a partially written export as a clean end
b2e8654 
An interrupted run can leave a tail that no reader can trust: half an
NDJSON line, a line that never got its newline, or a gzip member that
never got its CRC and length trailer. Read tolerated all three and told
its caller nothing, so the write path appended straight onto them and
silently destroyed logs. A valid but unterminated last line was the
worst case: the cursor pointed at it, so a resumed query skipped it,
while the append fused it with the next log into one unparseable line
and the entry was gone for good.

The cursor now names the last entry that is complete and properly
terminated, and reports the offset at which the file's recoverable
content ends (CleanSize) together with whether anything follows it
(Truncated). Safety does not rest on recovery: where no boundary can be
positively identified, Read refuses with ErrNoCleanBoundary instead of
guessing one.

For gzip, walk the file a member at a time with Multistream(false) plus
Reset over an io.ByteReader, so the reader is never wrapped in a bufio
of its own and the byte count stays in step with its real position.
That gives exact member boundaries with no new dependency. A member
that ends mid-line is not treated as a boundary at all, since appending
after it would glue the next log onto an unterminated one.

lastCursorGzip also stops discarding scanner errors. Returning the last
cursor is still right, but a truncated member, a checksum mismatch or
an over-long line now leave the walk at the last clean boundary rather
than yielding a too-early cursor with a nil error. That was the single
line that made the corruption silent and endlessly repeatable: every
later resume appended more unreachable data while reporting success.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
fix(filestore): report the error from closing the log destination
8b7c593 
AppendLogsAsync dropped the error from Close, though its doc comment
promised a buffered destination is always flushed. On the gzip path
Close writes the deflate terminator and the member footer, so a failure
there -- a full disk is entirely plausible for a 90 MB export -- left a
truncated member behind while the CLI printed "all logs have been
saved". Join the close error into the result instead; errors.Join keeps
errors.Is working, so a cancelled context still reads as
context.Canceled.

SaveLogsAsync now opens through CreateWriter and shares the same path,
which fixes the same dropped Close for a fresh export and lets a caller
open the destination before it starts producing logs.

memberWriter.Close names the file it failed on, as CompressFile does;
that error is about to become visible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
fix(export): recover a partial write and open the destination up front
894ba47 
Discard whatever follows the resume file's clean boundary before the
writer is opened, logging the offset and how many bytes went. Nothing
recoverable is lost: everything discarded sits at or after the cursor,
so the resumed query fetches it again. Only the offset the reader
positively identified is ever truncated to, and never past it. This
runs after the RPC connection is up, so an unreachable endpoint leaves
the file untouched.

Open the writer synchronously before GetLogs as well. Opening it inside
the saving goroutine meant a failure there logged and returned while
fetchLogs kept pushing onto the 100-slot channel; once full it blocked
until cancellation, errorChan never closed, and RunE looped for ever
printing "still retrieving logs..." while nothing was saved. It now
returns the error immediately, and saveLogs is left as a dispatch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
docs: describe what resuming does with a partial write
add5d12 
The claim that "entries already in the file are skipped, so resuming
never duplicates or drops a log" held only for a file that ends
cleanly. Say so, and document what happens when a run killed mid-write
leaves a partial entry behind: the tool truncates to the last complete
entry and re-fetches from there, or refuses outright when no such point
can be identified.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
docs: scope what resuming discards and re-fetches
e837601 
Clarify that the resume boundary is the last newline-terminated line
that parses as a log entry, not just any newline-terminated line. And
scope the guarantee about re-fetching: for export content (partial
writes), discarded data is re-fetched; for foreign data that was never
a log entry, it is simply removed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
docs: reflow the partial-write paragraphs to the file's wrap width
3119609 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
docs: remove the implementation plan from the repo
91389f1 
The plan was scaffolding for building the feature, not reference material
for using it. The design rationale it carried now lives in the README and
in the PR description.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
refactor: condense comments to the non-obvious rationale
b53a9c3 
Comments had grown into design essays restating what the code already
says. Keep only the reasoning that stops the code being broken by a
well-meaning simplification -- why countingReader must implement
io.ByteReader, why Multistream is switched off per member, why a line
without a newline means an interrupted write, why the close error is
joined -- and drop the narration around it.

Also removes the step-by-step narration in CompressFile and stale
references to the review that produced the tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
docs: spec for resume as incremental snapshots (input/output model)
6836193 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
docs: implementation plan for resume as incremental snapshots
ee2a64c 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa
feat(resume): validate strictly, refuse files this tool did not write
85e61b6
@gacevicljubisa
feat(resume): add PrepareOutput for in-place and copy-mode continuation
a0251e6
@gacevicljubisa @claude
refactor(filestore): drop SaveLogsAsync, callers compose CreateWriter…
3413d96 
… and AppendLogsAsync

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mxruRkPfuH8dd5nVxgB8S
@gacevicljubisa
feat(export): compose --resume with --output for copy-mode continuation
a5c3ed3
@gacevicljubisa
fix(export): surface save failures in the exit code and never skip th…
cc96d16 
…e final flush
@gacevicljubisa
fix(export): report a saver-caused stop as the save error, not a fetc…
96a43f3 
…h error
@gacevicljubisa
docs: reframe resume around incremental snapshots
5c6be38
@gacevicljubisa @claude
fix(resume): treat any spelling of the same file as in-place, not copy
c134c4c 
PrepareOutput compared filepath.Clean(inputPath) == filepath.Clean(outputPath)
to decide copy vs. in-place. A relative path against an absolute one, a
symlink or hardlink, or a case-insensitive filesystem folding two names
together all pass that check as "different", so os.Create(outputPath)
truncated the input to zero bytes before io.CopyN ever got to read from it —
destroying the file copy mode promises to leave untouched.

Add an os.SameFile (device+inode) check as a second guard, falling back to
prepareInPlace whenever the paths name the same file by any route. Add a
table test covering an absolute/relative spelling pair and a symlink, over
both a clean input and one with an interrupted final write, and amend the
spec's §3 table row to describe the os.SameFile detection instead of leaving
"distinct spellings" as the operator's problem.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mxruRkPfuH8dd5nVxgB8S
@gacevicljubisa @claude
fix(resume): pinpoint gzip refusals and stop misclassifying decode er…
3ae7d0d 
…rors

Three related diagnostic fixes to gzip resume validation (§5):

- Thread a 1-based member index and per-member line number through
  scanMember so a refusal on a 150k+-line archive names where the offending
  line is ("member 4, line 812 is not a log entry") instead of nothing.

- Drop gzip.ErrChecksum from truncationShaped: an interrupted write cannot
  produce a full-length member body plus a complete 8-byte trailer holding
  the wrong CRC, so that shape is corruption or tampering, not truncation.
  It now falls into the non-truncation-shaped branch, which is reclassified
  from a bare wrapped error to ErrNotAnExport (documented tradeoff: a
  genuine I/O error shaped like corruption is now also refused rather than
  surfaced distinctly, since the two are indistinguishable at this layer and
  §5 favors refusal).

- Classify gzip.NewReader's first-header failure the same way the plain
  format's equivalent is handled: truncation-shaped (file cut inside the
  10-byte header) is ErrNoLogs; anything else is ErrNotAnExport. Previously
  it returned a bare wrapped error outside the two-sentinel taxonomy.

Also reword lastCursorPlain's "N bytes without a newline" refusal, which
named the whole file size even when the newline was simply outside the
examined window, to name the window and its starting offset instead.

New tests: a CRC-flip in a valid member's trailer, and a file cut to 5 bytes
inside the gzip header. Spec §5 corrected to say the plain read window is
2*maxLineBytes, not maxLineBytes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mxruRkPfuH8dd5nVxgB8S
@gacevicljubisa @claude
fix(export): do not mask a real save error behind a joined cancellation
4189448 
The saver goroutine treated any errors.Is(err, context.Canceled) as pure
cancellation and returned without recording saveErr. But AppendLogsAsync can
return errors.Join(context.Canceled, closeErr) — SIGINT racing a failing
gzip-member flush on Close — and errors.Is only asks "does this contain
context.Canceled anywhere," which such a joined error satisfies even though
it also carries a real failure. saveErr stayed nil, so on a fresh --compress
run compressFunc would run over a file whose final write had failed.

Add solelyCanceled, which recurses through Join and single-wrap chains and
only returns true when cancellation is the ENTIRE error, and use it in place
of errors.Is for that check. The errorChan branch's own
errors.Is(err, context.Canceled) is untouched — it classifies the fetcher's
echo of the same cancellation, not a joined save error.

No cmd unit harness exists for this path (per the design doc, §10); verified
by build, go vet, and tracing both error shapes through solelyCanceled: a
bare context.Canceled returns true, Join(context.Canceled, ENOSPC) returns
false.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mxruRkPfuH8dd5nVxgB8S
@gacevicljubisa @claude
docs(resume): fix stale wording in README, spec and a test comment
b6af15b 
- README: Go 1.24 -> 1.25 (matches go.mod); the --end flag-table line's
  default text was a stale copy from --start's default and never matched the
  code (default 0 = latest block, cobra prints no "(default N)" suffix for
  it); add the copy-mode overwrite behavior, note --start is ignored on
  resume, and add the recovery one-liner for a failed copy-mode run.
- pkg/resume/resume_test.go: reword the stale comment above manyLogs, which
  described the deleted multi-window backward-read walk; it now earns its
  keep via the gzip path's per-member buffered scan. Also assert
  PrepareOutput's discarded-count return exactly in
  TestResumeAfterInterruptedWrite instead of discarding it with _.
- Spec: §8 "A fusnote" -> "A footnote"; §9's "pkg/gzipstore: unchanged" no
  longer describes this branch, which added AppendWriter to it earlier.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mxruRkPfuH8dd5nVxgB8S
@gacevicljubisa @claude
docs: remove the implementation plan, keep the design spec
b0f54cf 
The spec documents the durable design; the plan was execution
scaffolding.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa gacevicljubisa changed the title feat: add --resume to continue an interrupted export feat: add --resume for incremental snapshot exports Aug 27, 2026
gacevicljubisa and others added 3 commits August 27, 2026 15:39
@gacevicljubisa @claude
docs(spec): record the fix-wave behaviors and the deferred revalidati…
f411eef 
…on non-goal

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
docs: recommend the double extension for snapshots and fix flag help
a837a44 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gacevicljubisa @claude
docs: remove the design spec from the repo
012edf0 
The design rationale lives in the PR description and the README's
behavior documentation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gacevicljubisa