Prepare bounded FTW beta shadow collection and release packages - #36
Conversation
…ery, and range-query performance. Co-authored-by: Cursor <cursoragent@cursor.com>
…d skip unchanged series during maintain. Co-authored-by: Cursor <cursoragent@cursor.com>
Prod-beta cannot keep every historical point in RAM: after seal, queries merge immutable segments with the unsealed tail, open replays only that tail, and dirty maintain materializes a window instead of scanning all time. Co-authored-by: Cursor <cursoragent@cursor.com>
…pen without dropping historical points. Co-authored-by: Cursor <cursoragent@cursor.com>
…lvage. Query methods return Result instead of panicking on corrupt blocks; integrity checks verify block payloads, salvage can drop orphan segments, and the CLI and shadow sidecar gain seal/maintain operations with periodic background upkeep. Co-authored-by: Cursor <cursoragent@cursor.com>
…tes. Smoke only cut after sync; this records a durable ACK watermark during Always ingest, verifies recovery against that watermark, rebuilds rollups after salvage, and exposes health/service-log ops fields plus a snapshot backup runbook. Co-authored-by: Cursor <cursoragent@cursor.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4927c1c. Configure here.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4927c1cda8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let series_points = database.query_history(series_id, span.start, span.end)?; | ||
| report.scanned_points = report | ||
| .scanned_points | ||
| .checked_add(series_points.len()) |
There was a problem hiding this comment.
For a reconciliation window that overlaps a large sealed segment, query_history fully decompresses matching blocks and materializes the complete filtered result before max_scanned_points is checked. Moreover, series_points.len() counts only points surviving timestamp filtering, not all entries visited, so many wide overlapping blocks can consume unbounded time and memory while the reported scan count remains below the limit. Apply the limit during block/index traversal or expose a bounded query iterator.
Useful? React with 👍 / 👎.
| if segment.stats().points != descriptor.points { | ||
| return Err(Error::Corruption { | ||
| offset: 0, | ||
| reason: format!( | ||
| "raw segment {} point count does not match the manifest", | ||
| descriptor.file | ||
| ), | ||
| }); | ||
| } |
There was a problem hiding this comment.
When a referenced .wseg is replaced or mixed up with another internally valid segment having the same point count, this check accepts it and the store subsequently serves the replacement history as valid. The manifest records minimum and maximum timestamps but those fields are never compared anywhere, and it stores no content digest, so check_integrity cannot detect the substitution either. Store and verify a segment checksum or otherwise bind all relevant segment contents to the descriptor.
Useful? React with 👍 / 👎.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
Approved: Cursor Bugbot completed successfully and the prior durability-watermark finding is resolved, with no unresolved Bugbot issues needing human review. No additional reviewers were assigned.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_968c336c-f424-4635-b5c8-20725d5bddba) |
1 participant
FTW's beta needs a bounded copy of live history while SQLite/Parquet keep their current role. This prepares FTWDB
0.1.0-alpha.2as an opt-in shadow collector with a separate process, a fixed protocol, resource limits, and usable release packages.Paired Core integration: srcfl/ftw#1096. Core owns the bounded history queue, loss/status reporting and opt-in Compose overlay. Core head
f305fce493f346c39c7a212676517c9c7be87adfpins FTWDB7bbae63532f695b10aca548bf4ee58c6d7ebb3a8in both Compose and CI; its Go-to-Rust contract check passes.Changes
Overloadedand degraded health while continuing to serve exact stored receipts. Disable automatic rollups/sealing/retention on this bounded collection path. Add process tests for both limits, invalid settings, conflicts, restart, and recovery after changing the budget.Validation
cargo audit --deny warnings; no ignored advisories.7bbae63532f695b10aca548bf4ee58c6d7ebb3a8passes all eight CI jobs: https://github.com/srcfl/ftwdb/actions/runs/34130603839. Native tests and all three package builds pass on Linux AMD64, Linux ARM64 and macOS. The downloaded push-run archives identify this exact clean source; all archive/binary SHA-256 values and ELF/Mach-O architectures match. Local Linux ARM64 container start/stop and native macOS archive checks also pass.Review follow-up
The two scan-limit and segment-binding findings now have regressions that exercise the failure before reading a block, filtered points, limits across seals/live data, valid replacement files, and refusal of unsafe restore/salvage. The flush comment does not reproduce:
Database::flushalready callsmark_ingress_durableafter a successful sync. Existing storage and multi-source runtime tests verify both source watermarks; grouped append sync also advances prior sources.The final Cursor Security Agent check passed. The final Bugbot rerun hit the account usage limit and returned a neutral error, so it supplies no new review.
Rollout limits
This PR does not publish a release or switch an authoritative read path. The Core integration must pin this FTWDB source and share the frozen v1 fixtures. Collection can have gaps after restart, overload, or outage; it does not claim complete replication or backfill.
A standalone FTWDB beta still needs target-box soak, physical SD-card power cuts, resource/write-volume measurements, and an off-card backup/restore drill. Process and NBD tests cannot substitute for those checks. The new exact-retry receipt index retains payload bytes; long-term space/replay budgets still need measurement before any read promotion.
Rollback: stop the source copy and sidecar, retain the new shadow store, then use a pre-upgrade snapshot or a fresh empty shadow store. Keep SQLite/Parquet in place. Filesystem free-space checks cannot reserve blocks against other processes, so keep host disk alerts and the service/container memory and CPU limits.