feat(storage): add an optional FTWDB beta candidate - #1096
Conversation
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_eb92b769-7be2-444a-a47e-43a956e67567) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_1d6999c0-8450-4c48-92c5-a5502b755b42) |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dd49a76575
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ftw: | ||
| environment: | ||
| FTWDB_SHADOW_SOCKET: /run/ftwdb-shadow/shadow.sock | ||
| volumes: | ||
| - ftwdb-shadow-run:/run/ftwdb-shadow |
There was a problem hiding this comment.
When a tester enables the candidate with this nonstandard overlay and then performs an in-app Core update, the updater recreates ftw without these environment and volume entries, silently ending the experiment. The base service fixes FTW_UPDATER_COMPOSE to docker-compose.yml, while discoverOverrides in go/cmd/ftw-updater/main.go only adds conventionally named override files, and the update path runs compose up -d from that resulting list. Include this overlay in the updater's Compose inputs (or otherwise persist its settings) so updates retain the socket and mount.
Useful? React with 👍 / 👎.
1 participant
Add an optional FTWDB process to the beta so test boxes can measure it alongside the current history store. Core copies five numeric fields after a live SQLite history transaction commits. SQLite and Parquet continue to serve all reads; config, forecasts, learned models and control keep their current stores.
Paired with FTWDB #36, pinned to
7bbae63532f695b10aca548bf4ee58c6d7ebb3a8in both the Compose overlay and contract CI. That candidate supplies the wire protocol, always-sync receipts, disk guards and packaging. This PR starts from v2.17.1-beta.1 and can land separately from calendar removal #1093 and SQLite config #1094. The planner and forecast owners have the scope; no solver or learning logic changes./api/health. Nothing reports a sent batch as durable.Coverage is a bounded live session, not a complete replica or backup. It excludes old data, driver samples, SQL imports, retention deletes, the energy ledger and forecast archives. A restart can discard pending work; a new session makes that boundary explicit. A complete migration would need a persisted change sequence and separate reconciliation work before changing any read path.
Validation: full Core CI and the Go–Rust contract workflow pass.
make verify-allalso passed with #1093, #1094 and this PR applied together on v2.17.1-beta.1, including Linux ARM64/AMD64 and Windows cross-builds. Race tests against the pinned Rust process passed with byte fixtures, late and repeated timestamps, failed SQLite commits, overload, lost acknowledgement, exact retry, SIGKILL, restart and offline content reconciliation. A running local Core returned HTTP 200 with a missing sidecar. The pinned Git-context image built and the actual Compose sidecar passed a UID, socket, resource-limit, shutdown and volume-isolation smoke test in a temporary project.Before wider use, run a 72-hour test on a real box and measure control latency, CPU, RSS, disk growth, sync rate and gaps. Physical power loss and shared SD-card pressure remain untested. No merge, release or physical-box rollout is part of this PR.
Note
Medium Risk
Touches the post-commit history path and adds async I/O, but design explicitly bounds memory, drops overload work, and refuses unsafe sidecars without blocking SQLite; still warrants monitoring on real hardware for control-loop and disk effects.
Overview
Adds an optional FTWDB beta candidate that shadows committed live site history to a local Rust sidecar over a Unix socket, while SQLite and Parquet remain authoritative for reads, control, config, and forecasting.
Core wiring:
-ftwdb-shadow-socket/FTWDB_SHADOW_SOCKETstartsftwdbshadow.Betain a background goroutine (no startup dependency on the sidecar). After successful SQLite history commits,stateemits ticks through a bounded 256-slotHistoryFeed(drops on overload, never blocks writers). The beta batches up to 128 ticks every 30s, maps five numeric fields (grid/PV/battery/load power + SoC) into the frozen shadow protocol v1, and commits with idempotent exact-byte retries and durable-ack requirements (always-sync, 512 MiB pause).GET /api/healthgainsftwdb_shadowwith session, queue/gap stats, durable receipts, and sidecar ops.New package
go/internal/ftwdbshadow: wire codec, Unix client, health ops validation, golden fixtures vendored for Go–Rust contract CI.Ops/docs:
docker-compose.ftwdb-shadow.ymloverlay (pinned FTWDB commit, hardened sidecar profile),.github/workflows/ftwdb-shadow-contract.yml, anddocs/ftwdb-shadow.md/ architecture note. Disabled by default.Reviewed by Cursor Bugbot for commit dd49a76. Bugbot is set up for automated code reviews on this repo. Configure here.