peat: agent memory as a fold (+ three fold fixes: Hnsw stranding, checkpointing, lazy recovery) - #18
Closed
flowerornament wants to merge 37 commits into
Closed
peat: agent memory as a fold (+ three fold fixes: Hnsw stranding, checkpointing, lazy recovery)#18flowerornament wants to merge 37 commits into
flowerornament wants to merge 37 commits into
Conversation
- new-project.sh scaffold + serde_json/clap/minijinja deps; cargo check green - hooks/README.md: verified Claude Code stdin contract (no env vars; jq over stdin; SessionStart stdout becomes context; .peat/current-session for obs) - tests/fixtures/transcript-nx-rs-planread.jsonl: real 62-line transcript, secret-scanned, for the parser golden test
…ctor Hnsw::push kept pending work keyed by K alone, last-writer-winning the value while accumulating the delta. KeyedStream::upsert emits remove(old) + insert(new) in one transaction; the pair netted to zero, commit's 0 arm skipped it, and the old vector survived in both the graph and the store (bogkit's own search example hits this; found and documented by the salience branch, examples/salience/README.md). Pending is now keyed by (key, value) — the Bm25 posting discipline — so distinct values never cancel, while the intended same-value insert+retract net-out still holds. Removals guard on the stored value so a same-tx replacement wins in either drain order (WriteTx::get reads uncommitted writes). Regression test proven red against the unpatched sink.
… partial) Ledger schema (versioned Envelope, (session,seq) idempotent ids), fold pipeline (day digest, file<->session multimap, Bm25 + ese->Hnsw over obs/final/user text, newest-wins subjects, session summaries), transcript parser (never-fatal, ISO-8601), and the three verbs. Smoke-tested against the fixture transcript end to end.
…bog-a-thon-36e, -410) Oracle 1: revising an event makes its old text unfindable in Bm25 and lose vector-nearest to a control doc, surviving reopen/rebuild; an #[ignore]d twin asserts the opposite and is proven to FAIL when run (red-capable). Oracle 2: fold(prefix) == independent scan prediction at five cut points, plus batching-unobservability. Fixture golden test, never-fatal parser test, ISO round-trip (epoch shell-verified).
…og-a-thon review) Equal-timestamp obs now resolve by (ts, seq) in both the fold and the oracle-2 predictor, with an 8x repeated regression test against drain-order flakiness. OBS_SEQ_BASE moved to 1<<31 so line-derived capture seqs can never collide on long transcripts. capture --final-msg ingests the Stop hook's authoritative last_assistant_message (spec amendment 2). Brief header renders the local calendar date (render boundary; UTC-labeled fallback). Retraction asymmetry in subj_step/sess_step declared in doc comments as intended for peat's write paths.
Several worktree agents can point PEAT_DB at one database; fold is single-writer, so opens retry with backoff (~45s) instead of dying when another agent's hook holds the lock. The brief gains an 'active in the last hour' section from the sessions view — one agent's brief shows what the other worktrees are doing, from the same fold that holds the long-term memory.
Found during the murail backfill: without Stream::checkpoint() the fjall journal grows unboundedly (76MB for one afternoon) and is replayed on every open, dominating brief latency after bulk ingest.
…ushes memtables The pipeline gains a Table<EventId, Envelope> ledger mirror, making the event stream readable back out of the store. 'peat asof YYYY-MM-DD [task]' filters the mirror at the cutoff and folds the prefix through the same pipeline into a scratch db — the brief as it would have read that day, ages computed relative to the cutoff. Deterministic replay (oracle 2) is what makes the result the truth of that day. New test pins the contract. fold::Stream::checkpoint now rotates every keyspace's memtable before persisting, so the journal can be retired; without it the journal grows for the database's lifetime and is replayed in full on every open.
Hnsw::init eagerly rebuilt the whole graph from persisted vectors on every Stream open — O(n) 512-dim inserts before any verb ran, 8.7s on a 44k-event store. init now marks the graph stale and first use rebuilds it (the exact path aborted transactions already take), so opens that never search pay nothing: peat brief dropped 8.7s -> 0.09s. peat side: vectors are for beliefs and session summaries (obs + final messages); the user-message firehose stays in Bm25 but is no longer embedded, bounding the graph a query rebuild must reconstruct.
|
Verdict: ✅ Looks eligible Hi @flowerornament — thanks for the submission! This is a clear, substantive use of BogKit. BogKit usage (all three crates, meaningfully called):
So imports + real API calls are all present, not just Cargo.toml entries. Structure
Template
Note (for the human judge, not a blocker): this PR also modifies the Partial review: A human judge makes the final call. Nice work! 🌱 Automated first pass by bog-auto-judge (claude-opus-4-8). Human judges make the final call. |
…yle filters stdout is an API (SessionStart injects it; --json is machine-read), so all decoration gates on is_terminal per stream: spinners and phase timings on stderr only; color on stdout only at an interactive prompt, via minijinja identity-filters (h1/dim/warn/accent) so hook, test, and piped output stay byte-for-byte unstyled. The lock-wait — up to 45s of silence before — now ticks elapsed time; ledger open, capture, and asof replay report measured cost when they exceed 300ms. deps: indicatif, console.
'asof 2026-08-16' was cutting at UTC midnight — 4:59pm on the US west coast — silently excluding the evening. The cutoff now subtracts the local offset (date +%z, render boundary only; the fold stays timezone-free). A zero-event replay also says why on stderr instead of rendering a bare header, which reads as broken.
'peat recall <query>' is search-only hybrid recall: BM25 and HNSW hits
RRF-fused, hydrated, printed with kind/cited/age disposition — the
query surface for humans and agent tools, distinct from brief's
orientation bundle.
Embedding now includes user messages up to 400 bytes: directives
('ground in the formal model') are short by nature, pasted walls are
long, so the cutoff keeps semantic recall over the instructions that
matter without re-inflating the graph a query rebuild pays for.
…e test) Two field bugs from the first cross-desk user. (1) obs resolved .peat/current-session relative to CWD only — from a jj workspace desk with its own .peat, that finds nothing even though PEAT_DB points at the shared anchor. Lookup now falls back to the file beside the shared db. A desk is not the anchor; the recurring workspace-path hazard, in miniature. (2) the lock-wait cap (45s) could expire under a legitimate long write (bulk capture) and die via raw panic; default is now 120s, PEAT_LOCK_WAIT_SECS overrides, and giving up prints what is happening and exits EX_TEMPFAIL instead of panicking.
…view)
First fresh-agent review nailed the recall gap: only user/final/obs text
was indexed, so operational conclusions ('root cause was X') living in
mid-session assistant messages were unfindable. Assistant text blocks
>= 80 bytes are now Said events (capped 1200B), keyword-indexed for
recall but not embedded — coverage without re-inflating the vector
graph a semantic query rebuilds. The closing message stays FinalMsg
(its Said duplicate is dropped). EVENT_VERSION 2; additive.
…murail-1a repro) murail-1a proved concurrent READERS panic-trace each other: fjall's single-writer db is exclusive even for reads, and catch_unwind does not silence the default panic hook — every caught-and-retried open printed a full backtrace, indistinguishable from a crash. The hook is now swapped out for the retry window and restored after; non-Locked panics restore it and fail immediately instead of burning the whole wait on an unretryable error; the give-up message says reads are exclusive too. 3 concurrent recalls: zero traces, all succeed.
Deposit today's judgment dated to the day it is about (noon local of --at YYYY-MM-DD): asof briefs for that day then carry beliefs, not just activity. Provenance stays honest — the depositing session id rides along, so a retrofill is visibly a retrofill.
…ock-once obs CompactSummary events (additive, v2) preserve the compaction's own distillation of the context it replaced — the closest thing to an observation compaction can produce, indexed for keyword and semantic recall. hooks/README documents the two new wirings: Stop block-once (the agent deposits observations in one extra turn while context is hot; stop_hook_active guards the loop) and PreCompact mechanical salvage capture.
Brief and recall JSON were clipping text before serialization, so the full stored fact was unreachable from the CLI. Clipping is display concern: it now lives in a template 'clip' filter (and recall's pretty printer), and --json is the API — full text, always.
--kind/--since/--session narrow recall; --subject reads a subject's full evidence trail straight from the evidence multimap (the claims-register read the reviewer asked for). 'subjects' lists the register newest-first with support and citation status — the anti-fragmentation view. 'show <session-prefix> <seq>' prints one event in full with any observations citing it. Full text everywhere; truncation only in terminal pretty printers.
h1/accent/dim/warn (bold headers, cyan identities, receded metadata, red distrust) now style subjects, recall, show, obs, and every receipt and error the same way the brief template does — same roles, same colors. Receipts (captured/recorded/near-subjects) are dim notes on stderr; errors get a red peat: prefix. All identity when the stream is piped.
One line per event (date, session, seq, kind, payload summary), filters for --session/--kind/--since, --json for one-object-per-line dumps. On a terminal the output pages through less -RFX (git-log style), so the ledger is scrollable; piped output is plain. This is the 'show me the actual data' view: capture something, then watch it sit in the ledger that every brief and belief is a fold of.
|
A recall hit could be found but not expanded: the pretty printer omitted the event's address, so 'peat show' had nothing to take. Hits now end with a dim (session seq) pair that pastes straight into show.
Structural: brief assembly (Brief/assemble/render/emit + the single rrf fusion) moves to brief.rs; path/session policy and the open-with-retry (panic hook now restored by a Drop guard, not three manual calls) to db.rs; display helpers and the pager join ui.rs, whose tty/color gates are now probed once per process instead of per styled string; the civil- date math consolidates beside iso_to_ms (the date subprocess for the brief header is gone); Event::tag/summary live with the enum so a new variant is a one-file change. main.rs is CLI dispatch: 1068 -> 581 lines. Dedup: one RRF implementation, one make_brief! for the Brief/Asof twins, one flattened clap Filter (help-text drift between recall and events fixed), typed recall Hit rows instead of a JSON round-trip, DayDelta struct-update literals, capture reuses transcript::override_final_msg. Efficiency: obs is one transaction (was four, with per-probe read_tx); tool inputs are borrowed, not deep-cloned per block; the closing message is capped once, not per assistant turn; show's citer walk runs only on a hit; events streams via fmt::Write. Also: the .peat data dir now writes its own .gitignore on open (cargo's target/ pattern) — jj snapshots non-ignored files into the working commit on the next command, so shipping the rule with the tool beats asking every repo to remember it (found live by murail-1a). Skipped by decision: SessStats.start_ms removal (view-schema change breaks deployed dbs), typed streaming transcript parse (capture is not latency-facing), named reader struct (hard-codes fold's HNSW consts; the tuple + macros remain the idiom).
Collapsible-ifs and a filter_map simplification auto-applied; the panic- hook type gets a named alias; redundant closures, lossless casts, and doc backticks from the pedantic set. Deliberately kept, per lint-pedantic-selective: must_use/panics-doc suggestions (bin crate, not a library API) and the u64<->i64 casts at the timezone boundary (wrap is impossible for any date this side of year 292471).
…t; tx panics no longer poison the store (bog-a-thon-14y, -p3u, -s7a, -958, -8wg, -9zp) Adversarial review of this branch's own earlier fixes found the (key, value)-keyed Hnsw pending traded one bug for three: deletes silently no-op'd when the caller's recomputed embedding didn't byte-match the store (undeletable rows); same-key double inserts resolved by HashMap drain order; and every push paid a value encode + clone. Pending is now per-key with in-memory resolution — net>0 or a trailing positive push indexes the latest record, net<0 deletes by key unconditionally, cancelled inserts no-op — order-independent across keys, push-ordered within one, zero store reads. Bm25 had the same disease worse: per-(term,doc) net deltas written as absolute frequencies corrupted every term shared between a replaced document's old and new text (executed repro: tf 2->1 nets to -1 and DELETES the posting — the document vanished from queries for its own words). Postings now resolve per document at commit: old-only terms removed, new frequencies written absolutely, doclen last-writer-wins. Stream::wtx guarded only the user closure; a panic in the pipeline's final commit (store reads, deserialization, lazy index rebuilds all live there now) skipped abort() and left nodes' pending state to replay into the next transaction as orphans. Commit now aborts-on-panic — and both panic paths drop the fjall tx BEFORE resuming the unwind, because unwinding through the live guard poisoned the writer lock for the rest of the process. checkpoint() documents its sharp edges (doc(hidden) fjall surface, wait-loop wedges if a flush worker died, L0 shredding under frequent small seals); peat's obs no longer checkpoints per event. All three new regression tests proven red against the pre-fix code via stash; 23 fold + 9 doc + 8 peat green.
Author
|
Post-submission wrap-up. Since the initial entry, this branch grew from dogfooding peat against ~59k real events across six agent desks. That surfaced four independent fold fixes, each in its own commit with a regression test proven red against the pre-fix code:
The example itself ( |
Transcript parsing becomes N frontends over the neutral event IR: formats detected by POSITIVE signature (Codex rollouts open with session_meta; Claude lines carry sessionId) and unknown formats are rejected, never guessed. The Codex adapter maps message/function_call/custom_tool_call/ compacted; its editorial choices (reasoning and developer-role skipped, compact summary taken from payload.message or the continuation handoff in replacement_history) live in the adapter and nowhere else. The contract that makes adapter evolution cheap is now documented at the module top: seq assignment is a pure function of the file and part of the ledger contract (add slots, never renumber), transcripts are the outer ledger, and capture is idempotent — so improving an adapter is fix + re-capture, with changed mappings replacing and new slots inserting. Smoke: a real 100MB rollout yields 1,850 events in 3s; recall surfaces cross-agent tmux traffic and Codex's own conclusions; golden test pins the mapping plus unknown-format rejection.
The brief now covers the entire past in bounded lines: 'further back' renders calendar bands that widen geometrically with distance (2 weeks, 2 months, 2 quarters, years, then one terminal band), each an extractive digest of the materialized day table ending in its own descent handle. Pure read-time regrouping — no new sinks, nothing stored, asof-correct for free; --budget / PEAT_BRIEF_BUDGET re-slices without recomputing. The learnable surface shrinks to two verbs: bare peat orients, and peat <thing> infers window / session / search from the argument's shape, with the header naming its interpretation. zoom descends year→months→ weeks→days→sessions with lane-B texts (finals, compact summaries, obs) revealed in range; show without a seq is one session's overview. All explicit subcommands remain visible spellings of the same reads. ladder.rs is the pure core (civil-date math shared with date_label, tiling and budget invariants unit-tested); README documents the surface and hooks/README gains the moment-coverage matrix.
- .peat/redirect (the beads convention): a worktree desk points at its anchor's .peat, so bare peat in any desk reads the shared memory — fixes the empty brief in murail-1a; desk-local files stay put - one counting grammar everywhere: 'N tools (M fail) · N commits', zero counts suppressed, large counts humanized (14.0k), spans on clipped bands; last-session age uses the same <1h/7h/33d labels as every tag - every read's lines end in the command that goes deeper: recall hits and belief lines gained handles; bare peat <subject> now reads the full evidence trail (briefs clip at ~120 chars — trails never do — and long deposits earn a split-this nudge at obs time) - session overview dedupes branch/worktree noise; zoom headers carry their dates - --help is first-class: shape-dispatch table in after_help, accurate hook references (capture runs at Stop/PreCompact/SessionEnd), budget flag explained in plain words, tagline no longer names bogkit
…a-thon-t5s, -lum) Nested phases (asof's replay wraps a scratch ledger open) interleaved two spinners and printed the opening receipt twice — a global phase count now keeps inner phases silent, verified under a real pty: one spinner line per story. README multi-agent section and hooks/README teach the desk redirect; PEAT_DB stays the explicit hook-side override.
…vations) An audit of every observation across three project ledgers found quality tracks house culture: herald's obs (timeless rules, populations counted, commits cited) are the model; the tool author's are the worst (deixis, episode names, deployment status deposited as knowledge). obs now prints a non-blocking style note on deictic and status-log phrasing, and the guidance blurb teaches the reader test with a real bad/good pair drawn from the corpus — the bad one is the author's own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
category
team
what you built
Agent memory as a fold. Every Claude Code session deposits its mechanical exhaust — user asks, tool calls, file touches, commits, the final message — plus small agent-written observations into one append-forever event ledger (
fold::KeyedStream, idempotent by(session, seq)). Every readable surface is a materialized bogkit view over that ledger, maintained incrementally:Aggregate→Table)InvertedIndex)foldfull-text) fused with ese → anny HNSW vector searchAggregate)Design decision worth judging: vectors index only distilled text — observations and final messages go through ese → HNSW; the firehose (user messages, tool calls) stays BM25-only. Embedding is the expensive lane, so it is reserved for the text with the highest signal density.
Verbs close the loop with Claude Code hooks:
peat captureingests the session transcript at Stop;peat briefrenders a session-start prompt from one snapshot at SessionStart (whose stdout is injected into context);peat obsdeposits a one-line judged observation;peat asof <date> [query]replays the ledger to any past day and renders that day's actual world, ages computed relative to it — time travel as a fold, ~14k events in 1.2s. The brief template is hot-editable; the pipeline never changes for prompt iteration. Replay determinism and retraction-visibility are tested with red-capable oracles.Bonus, riding in this PR — three fold 0.0.1 fixes, each red-proven or measured (multi-agent dogfooding on the hackathon's own sessions surfaced all three):
640ff6f):Hnsw::pushkeyed pending work by key alone, soKeyedStream::upsert's retract+insert pair netted to zero and the old vector silently survived in graph and store — reachable through bogkit's ownsearchexample (first documented by the salience work). Fixed by keying pending work by (key, value) — Bm25's posting discipline — with a stored-value guard on removals; regression test proven red against the unpatched sink.c9d1c4e,eea58ae): streams never retired their journal, so every open replayed every write ever made;checkpoint()now flushes memtables and peat calls it after capture writes.d25014a):Hnsw::initeagerly rebuilt the whole HNSW graph on every open — every CLI invocation paid O(corpus); recovery is now deferred until a search needs it.Measured on the murail ledger (~44k events):
peat brief8.7s → 0.13s; semantic query 0.18s;asoffull replay 1.2s.how to run
Hook wiring for live capture: see
examples/peat/hooks/README.md(stdin-JSON contract verified against current Claude Code docs). The Stop hook passes--final-msgfrom stdin'slast_assistant_message— authoritative over transcript tail parsing, which can lag at Stop time.demo / notes
peat capture <transcript.jsonl>— idempotent; re-run is a no-oppeat obs <subject> "<claim>" --from <seqs>— prints near-subject hint and support countpeat brief <task words>— renders five sections, in trust order: recent activity (per-day tools/fails/commits + touched files), last session (closing message, age, branch), recently touched files (file → sessions), possibly relevant (BM25 ⊕ HNSW fused hits, each tagged[kind · age]with origin and citation status inline), current understanding (agent-asserted subjects, newest wins, obs count + age shown)peat asof 2026-07-10 formal model— that day's real commits, files, and beliefs, ages relative to that day (replay, not a snapshot store)brief.tmpl(or drop an override in.peat/brief.tmpl) → re-brief — zero pipeline work(session, seq)event id (re-captured transcript or corrected obs), and oracle 1 proves the replaced text becomes unfindable in both indexes.ts_ms / 86_400_000) — "today" rolls at UTC midnight, deliberately clock-free inside the fold.examples/peat/tests/fixtures/holds a real (sanitized) transcript for the parser golden test.checklist