feat(storage): use embedded DuckDB for primary history - #1129
Conversation
1f5deb8 to
a512755
Compare
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_22e14306-0d5a-492c-8736-fe81a2e33466) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1bd31f6b3f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if s.history == nil { | ||
| return nil | ||
| } | ||
| ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute) |
There was a problem hiding this comment.
On installations with large histories or slow Raspberry Pi storage, the mandatory full backup will abort after 30 minutes even if it is making progress. This export performs an individual SQLite ExecContext for every DuckDB row and then rereads each destination table for verification, so mature databases containing tens or hundreds of millions of samples can legitimately exceed this fixed deadline; the resulting backup failure also blocks update flows that require a full backup. Use the caller's cancellable context or a configurable operation deadline rather than imposing an unconditional 30-minute cap.
Useful? React with 👍 / 👎.
1 participant
Core now writes and reads time-series data through embedded DuckDB. SQLite keeps configuration, device identity, learned state and cache. This removes the FTWDB shadow client and the live SQLite/Parquet query split.
History snapshots, metric samples and energy accounting commit together. A bounded queue keeps history I/O outside the control tick; health distinguishes accepted, committed and rejected ticks. The serial writer retires an earlier retry receipt only after observing its successful commit, retaining the current receipt across an uncertain result. Repeated samples keep their first value, history corrections keep their last value, and invalid floating-point values fail at the storage boundary.
Startup imports SQLite from one read snapshot in bounded transactions, checks every table's count and value hash, then binds the completed DuckDB generation to configuration. Legacy sample Parquet files use staging that can spill to disk, an ordered window check for duplicate keys, and verified 2,048-row commits. Checkpoints run between files and after 131,072 imported rows within a file. Each new file opens a fresh native DuckDB instance on the same durable primary to release retained table buffers. The production constructor finishes this import before it exposes the Store or starts the writer; verified restarts keep one instance. A persistent source hash prevents a changed or missing file from completing an interrupted import; full backups reject pending imports. Full backups export a consistent DuckDB snapshot to portable SQLite and omit the imported sample files so older Core does not read duplicates. Schema 3 requires a full backup. Image-only downgrade and incomplete config-only restore are refused.
Core and backup builds use official DuckDB Go bindings with CGO on Linux amd64/arm64 and Windows amd64. The Linux images include the C++ runtime. Windows uses the pinned MinGW 14.2.0 UCRT toolchain used for the upstream DuckDB archive. Linked-library notices have been checked against the actual artifacts.
Validation completed locally:
Readback now uses 2,048-row keyset pages. On the same 20.8-million-sample box source, startup with profiling completed in 47.2 seconds with 221 MiB peak RSS; an independent reader verified all eleven tables. Earlier materialized readback used about 852 MiB. These are individual Mac host runs, not Raspberry Pi timings, and DuckDB's 128 MB buffer setting is not a process memory limit.
All 17 CI checks passed on 1bd31f6, including Windows config/state/backup tests, all command builds, Core/backup startup with only System32 in PATH, Linux amd64/arm64 builds, Go, web, drivers and full-stack tests. The expanded database fixtures exceeded the old two-minute Go package deadline on shared Linux runners; the candidate gives each package five minutes and retains focused latency/cancellation assertions.
The full fresh-source import on that same commit completed in 387.49 seconds with 249.6 MiB peak process RSS and under 9 MiB Go heap. It used the production 128 MB DuckDB setting with two threads and no persistent native observer. After the final reopen, tracked ART memory was 0.75 MiB and base-table memory was 2.75 MiB. These are Mac host measurements, not Raspberry Pi timings. The independent union verifier used a separate 512 MB setting and its memory is not part of the production result.
Ready for physical beta validation at the repository owner's request. The complete source import, independent data comparison and final-head CI have passed. A fresh verified full backup of the current physical box is preserved. Full-scale portable export/restore and a long writer soak remain unverified; they are not claimed as completed. The next step is the normal beta build and installation, followed by physical writes, queries, forecast, plan and browser checks. CODEOWNERS review still applies to the running beta before stable promotion. Earlier full-source tests exposed memory growth in staging indexes and native table buffers; the staging, checkpoint and session changes above fixed the import failure on this fresh source. The primary key remains: separate checkpoint/reopen probes released its ART buffers. Beta v3.2.0-beta.1 is published from version commit 5b37898. Its normal updater is now creating the pre-install rollback point on the physical box; installation is in progress.
This branch is based on master after the planner fix in #1130 and its private Energyplan counterpart merged. The next beta must validate both changes together on the physical box.
Dependency overlap: #1085 also touches go.mod/go.sum; this change retains the existing direct dependency versions apart from adding DuckDB. #1033 proposes an Alpine Go builder update; this change needs a glibc builder for the official DuckDB archive.
Raspberry Pi evidence: an isolated f659 build passed Core start/hold/restart and the anonymous SQLite migration with eleven-table comparison. Its five-minute workload committed all 60 ticks without rejection, but 3 of 2,746 concurrent queries exceeded their three-second context under a deliberate 0.5-CPU quota. The test stopped there, so its subsequent checkpoint/readback and ARM64 backup/restore steps did not run. Peak combined QA RSS was 172.1 MiB. The QA container was removed and the existing Core was unchanged. Normal-CPU candidate checks follow during physical installation.