fix: keep live collection running during DuckDB history import

[frahlg]

A large first DuckDB import kept Core offline beyond the old updater's 30-minute limit, which could restore only the old image over changed data. On the physical box, we stopped that updater before it rolled back. Core now imports the catalog, site history and energy accounting first, then starts live collection while SQLite raw samples and Parquet import in the background. The API and UI show incomplete history until all sources pass verification.

Addresses #1164. Keep that release blocker open until the candidate passes physical-box checks.

• Commit each bounded source chunk and its resume cursor together. Keep prior primary values on overlap, retain original sources, and reuse beta.1 generations, sequences and file receipts. Raw retention waits for import completion; full-history backup refuses partial coverage.
• Stage Parquet in a separate DuckDB instance. Reopen the primary native instance only when every SQL connection has closed; Rows, transactions, prepared statements and Raw calls retain their connection. Live-writer maintenance also releases retained buffers after bounded rows or time. Maintenance failures appear in health and retry.
• Remove the seed's ALTER DEFAULT operation after an abrupt-exit test exposed a DuckDB WAL replay failure. Catalog inserts use the seeded sequences directly.
• Require a read-only updater capability before Core loads config or opens data when self-update is enabled. Old, unknown or unavailable updaters stop startup with “update ftw-updater first”. Native installations with self-update disabled still start normally.
• Save the previous image identity before replacement. Allow six hours for real Core readiness and leave Core/data in place on failure. Document updater-first installation and recovery after a stopped updater; never mark an interrupted job successful without evidence.
• Include the startup page, background-import banner and Update Center progress from the coordinated UI work.

Validation:

• make verify passed; web tests passed, 591/591.
• Race-enabled tests passed for real abrupt child-process exits during SQLite and Parquet import, held SQL lifetimes, live writes during import, beta.1 resume compatibility, deferred retention, and first-hop startup with unchanged config/data.
• The updater builds for Linux ARM64 with CGO disabled.
• The local memory regression reached 85,516,657 source rows and 74/131 files without OOM, then hit the harness's 20-minute limit. Peak RSS was 355,500,032 bytes on macOS ARM64. Resume is running from the same durable cursor, with 240 earlier live commits preserved and Store ready in 69 ms; final results will be added here. DuckDB settings are 128 MB primary and 64 MB staging, not an OS process cap.
• The coordinated UI task checked startup, ready, background, disconnected, failed, complete and setup states in a browser and native accessibility tree, including Reload focus across polling. The owner handles final human and physical-box checks.

No merge, release or physical-box change is part of this PR work. The local timeout above did not stop the production import. The previous full-data hash review belongs to an earlier synchronous implementation and is not claimed as proof of this patch.

Coordination: #1171 owns snapshot restore paths in the updater; this PR leaves those functions alone. #1176 changes a separate API parsing hunk. #1175 and #1177 do not share edited files with this change.

---

Note

High Risk
Changes first-boot DuckDB migration, persistent history layout, and self-update failure behavior—mistakes can strand sites or lose history without a matching backup restore.

Overview
Fixes the failure mode where a long first DuckDB migration let the updater roll back only the Core image and leave mismatched data (#1164).

Updater & startup guard: Core with self-update enabled now calls GET /capabilities on the updater socket before config or state open, requiring protocol 1 and preserve_core_on_readiness_failure. On readiness timeout or failure, the updater no longer auto-reverts the image; it persists previous_image_id, extends the Core wait to six hours, and fails with guidance to restore from a verified backup. Docs describe updater-first install and recovery if the sidecar stops mid-migration.

Background history import: Core seeds catalog, site history, and energy accounting, then starts the API and live collection while SQLite ts_samples and cold Parquet import in bounded, resumable chunks. HistoryMigrationStatus drives boot boot.html, health JSON, dashboard banners, and Update Center during startup. History stays incomplete until every source verifies; full backups and raw retention are blocked until import finishes. Parquet stages in a separate DuckDB file; the primary uses a rotating native connector so buffer release waits for closed SQL connections, with live-writer maintenance and explicit sequence-based catalog IDs.

UI: Treats API 503 + error: starting as boot/migration progress (not “self-update disabled”); another client’s in-flight update can resume in Update Center.

^{Reviewed by Cursor Bugbot for commit 7b17737. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_f1c87d25-995a-4331-9ec4-7a9f4ee73f2e)

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-08T10:23:30.237661Z	7b17737	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.