Longhaul and Thread to letter — two builds on the SuperDocs API - #200
Open
vishvamsinh28 wants to merge 14 commits into
Open
Longhaul and Thread to letter — two builds on the SuperDocs API#200vishvamsinh28 wants to merge 14 commits into
vishvamsinh28 wants to merge 14 commits into
Conversation
A multi-step edit over a document set that survives being killed. It plans the job, commits after every step, and on restart resumes from the last committed step rather than replaying — no step twice, no step skipped. The design turns on one thing: a crash between sending a request and recording its answer leaves local state not stale but unknowing. Each step's instruction carries an idempotency token, so the act of the turn landing echoes it into the session transcript — which, unlike jobs, never expires. "Did that happen?" becomes a question the server can answer indefinitely. Two windows, treated differently on purpose: a chat turn is never retried (a retry is a second edit), an approve always is (it names changes that already exist). When the evidence genuinely does not settle it, the run halts and says so rather than guessing in either direction. Proven: - 119/119 kill points byte-identical, each a real SIGKILL to a real subprocess, compared on document HTML, exported bytes and the commit log - Live against api.superdocs.app: 8 steps, 8 jobs, 0 duplicate turns, 8 ops - 64 tests, all offline, no API key needed The live run found three bugs the offline proof could not, and they are fixed and covered here: a turn can need several rounds of review; the ledger can be behind the server (killing a process leaves the ledger intact, so no number of kill points reaches it); and the commit log double-counted changes across rounds. A fourth finding is a rough edge in the product — export has no per-document selector and silently ignores one — worked around and reported. Human gate on three surfaces (terminal, browser, MCP) sharing one decision path. Documents that try to issue instructions are reported as data, never obeyed. Full account of what broke, including what the offline proof was blind to, in docs/LIVE-RESULTS.md. Built for the SuperDocs engineering task, Round 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ened
Two things were wrong with the previous state, and both were found by pointing
this at the real API instead of admiring it offline.
The demo pointed at a fake. `make demo` and the review page both talked to a
local stand-in, so the thing you actually looked at showed simulated data and
the integration was invisible. The live path was buried in a script.
- `make live` is now the flagship: the console against api.superdocs.app.
- The console shows every REST call it makes — method, path, status, duration,
billable or not — so the integration is visible rather than asserted.
- It opens idle and spends nothing until an operator presses Run, with the
budget set in the same click.
- Documents export to real .docx/.pdf/.md from the browser.
- `make console` keeps the offline version for when you have no key.
And the runner would say "done" for work that never happened. A step told to
replace a string returned no proposed changes for a document that plainly
contained it. Across one eight-step live run this happened twice. Nothing in
the ledger was wrong — it faithfully recorded that the model proposed nothing.
That is the problem: "there was nothing to do" and "I missed it" are
indistinguishable from the inside, and the run reported success while two
documents still said the thing the job existed to remove.
Every step now carries an expectation taken from its own instruction, and the
document the server holds is checked against it once the step finishes. Reading
a document is free, so the check costs nothing. A run with an unmet expectation
and nothing proposed reports blocked, never completed. Rejections are told
apart from misses, and a step adopted after a ledger loss says the record is
gone rather than guessing.
Also: turns are counted as issued the moment they leave, separately from what
the server confirms it billed; job polling backs off instead of asking every
second; latency is reported as median/p95/slowest rather than a sum.
75 tests, 119/119 kill points. Live: 486 operations left of 500.
Built for the SuperDocs engineering task, Round 2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
From the round's shared open list. Turns a messy email thread into a formal letter that states only what people actually undertook, attached back to a draft reply in the original conversation. The easy version of this writes a polite summary, which is worse than nothing: a summary turns "we should be able to have it by the 14th" into "Calder will deliver by the 14th", and now a document exists saying somebody promised something they did not promise. So every candidate is classified — firm, superseded, hedged, second-hand, proposed — and every one carries a verbatim quote from the thread. If it cannot be quoted it is not a commitment, it is the model's own prose in quotation marks, and it never reaches the letter. The rejects are kept and shown rather than silently filtered. A human ticks what goes in; the letter is written from those alone. A thread where nobody agreed anything produces a letter saying so. What the first live run found: it correctly marked a delivery date superseded and then never listed the commitment that superseded it. The letter would have named the payment terms and omitted the vendor's actual delivery date, while looking complete. Fixed in the prompt, and guaranteed by a structural check that fails the run when a superseded commitment has no replacement — a prompt is a request, the check is the guarantee. Honest limits, stated in the README rather than buried: the Gmail add-on in gmail-addon/ is written and not verified against live Gmail; its prompts are generated from the Python source with a test that fails on drift, so the sideloaded path cannot quietly diverge from the tested one. And the tool attributes rather than authorises — an injected "Northwind will pay 500,000" really is in the thread and quotes cleanly, so what protects the user is that the row points at the sender it came from. There is a hostile thread fixture and tests pinning that limit down. 40 tests, no API key needed. Two billable operations per run; parsing, quote checking, export and the reply are all free. Built for the SuperDocs engineering task, Round 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pipeline was terminal-only, which makes the interesting part — deciding which candidates are real commitments — invisible. That decision is the whole tool, so it now has a screen. One FastAPI app and one self-contained page. It shows every call it makes to SuperDocs with timings and which ones bill, so the integration is visible rather than asserted, and it opens idle: extraction is one operation, the letter is another, each waiting for a click. The middle of the screen is the gate. Every candidate with its status chip, its verbatim quote, the message it came from, and a checkbox. Firm and verified ones arrive ticked; everything else does not. What you tick is what the letter says, and the letter renders underneath with download buttons for the document and the reply draft. Also: collapse duplicate rows. A model asked for a table will sometimes list one commitment twice — same party, same obligation, same date, same status, with the quote trimmed differently — which makes a reviewer check the same thing twice and makes the counts lie. Rows differing in any of party, obligation, date or status are left alone: a date that moved is two commitments, not one, and collapsing those would hide exactly what this build exists to surface. 43 tests, no API key needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The setup note in Code.gs said "Extensions -> Apps Script", which is wrong: that makes a bound script, and a Gmail add-on has to be a standalone project created at script.google.com. A bound script cannot carry a Gmail add-on manifest, so anyone following that would have hit a wall immediately. gmail-addon/README.md now covers both install paths (browser-only, and clasp), what each of the four OAuth scopes is for and why none of them can send mail, what the add-on does screen by screen, what it costs, and the failures worth expecting on a first install — including the unverified-app interstitial that Google shows for any add-on you have not published. The status warning stays at the top: written, not verified against live Gmail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Written but untested" was a weaker claim than it needed to be. Apps Script is
ordinary JavaScript against a handful of Google globals, so stubbing GmailApp,
CardService and UrlFetchApp is enough to run Code.gs and Prompts.gs unmodified —
same files, same prompts, same live SuperDocs calls.
make addon
reads a real thread, builds the real sidebar card, extracts and verifies the
commitments, renders the checklist with firm-and-verified pre-ticked, drafts the
letter from what is ticked, and creates the draft reply with the attachment.
Two operations, and it prints every call it made.
This is not a mock of the add-on. It is the add-on, with Gmail replaced by an
.mbox and CardService replaced by something that prints. What remains unproven
is narrower and now stated as such: whether Google's runtime renders these
cards and delivers these events.
The Gmail path was pointed the wrong way. It shipped install instructions for
something nobody is going to install, and the thing you could actually show was
a terminal. For a demo that is backwards.
So the demo is now a simulated mail client — inbox, threads, and the add-on
panel where it sits in Gmail — driving the real SuperDocs API:
make demo
Click a thread. The panel reads it, asks SuperDocs what was actually committed
to, and shows each candidate with the words it came from: hedged and superseded
and second-hand left unticked, firm and verified ticked. Tick what is real,
press the button, and a compose window opens with the draft reply and the .docx
attached. Send is disabled; nothing is ever sent. Every API call is listed
underneath with timings and which ones bill.
Removed the clasp and sideload instructions entirely. They were answering a
question nobody asked, and their presence implied this was a deployment when
the deliverable is the demo. gmail-addon/ keeps the Apps Script itself, and
`make addon` still runs those exact files against the live API, so the claim
"this is what it would be in Gmail" has something behind it.
Three more invented threads so the inbox looks like an inbox, including one
where nobody commits to anything and one carrying a prompt injection.
Checking the build against its card found memo reachable from the CLI and hard-coded to "letter" in the UI — and since the UI is now the demo, the card's "letter or memo" was half met. There is a toggle in the add-on panel now. Testing that turned up two real bugs, both about session hygiene. The session id was derived from the thread's content alone, so every run of the same thread reused one conversation and turns piled up. After two dozen, a live run returned commitments about people who were not in the thread at all: the model answering from the session's history instead of the evidence. Every one of them failed quote verification and nothing reached the letter — the safety net worked exactly as designed — but it should not be load-bearing for something this avoidable. Each run gets a fresh session. And the letter was drafted in the extraction's session, which contains the whole thread. The module docstring claimed the drafting turn is given the confirmed commitments "not the thread"; that was false as implemented. A live memo invented a FROM: line by reading the sender out of the conversation the reviewer had just filtered. Drafting now happens in its own session, and the same memo correctly writes [SENDER NAME] because it cannot see who sent anything. 48 tests.
"make demo" meant offline in one build and live in the other. That is not a naming quibble — it means you cannot tell from the command whether it is about to spend your operations, which is the one thing a command like that has to be unambiguous about. Both builds now use the same two words, and every target says which it is when it starts: live — talks to api.superdocs.app. Uses your key. Costs operations. offline — talks to a local stand-in. No key, no network, no cost. longhaul: live / offline for the console, and the kill-and-resume demonstration is named for what it proves rather than for being a demo — proof for one kill point in six seconds, proof-all for all 119. Both offline, because determinism is only provable against a fixed backend and calling that "demo" hid the reason. thread-to-letter: live is the mail-client demo, offline is the terminal pipeline against the stand-in. READMEs follow, and every target either README mentions now exists.
"Find commitments" was unclickable, and the reason is embarrassing: the panel called innerHTML on itself every poll, which is every 900ms. The button was destroyed and rebuilt 1.1 times a second, so any click that straddled a rebuild landed on an element that no longer existed. It looked like nothing happened because nothing did. Now there is one delegated listener bound once to the document, dispatching on data-act, so re-rendering cannot unbind anything. Each region also carries a signature and is only rebuilt when what it shows has actually changed. A running turn shows a live indicator saying a turn takes 15-45 seconds, instead of leaving you to guess whether the click registered. And the fake chrome is gone. The search box was a div that did nothing; it is an input now and filters the inbox, with Esc to clear. The stars did nothing and are deleted. Starred, Snoozed and Sent went with them. Inbox and Drafts stay because they work: Drafts shows the draft the add-on wrote, and switches from state already in the page rather than after a round-trip. Verified by clicking, not by assuming: search filters, a thread opens, Find commitments runs and shows progress, a checkbox toggles, Write produces the draft with the right recipient, subject and attachment, and Drafts shows it.
…eset Three things you could not do, and one you could that you should not have been able to. With candidates on screen and nothing ticked, pressing Write drafted a letter recording that nothing was agreed — and charged an operation for it. But "nothing is ticked" and "nothing was agreed" are different statements, and only a person turns the first into the second. The button is disabled in that state now, with the reason next to it, and there is a separate opt-in that says exactly what it will do and asks before doing it. The refusal is enforced in the endpoint too, before the worker starts, so it cannot be reached by any other caller either. A draft could not be thrown away. Discard sits in the draft header, clears the letter and keeps the review, so another can be written without paying again. And there was no way back to a clean slate. Reset clears the open thread, the review, the draft and the call log. Costs nothing. Also: a model wraps its quote in quote marks, and storing them meant every display added another pair and the reader saw ""like this"". Stripped on the way in, with a test that quotes inside a sentence are left alone.
confirm() and alert() look like exactly what they are: a browser interrupting a
demo to announce a port number. Replaced with an in-app dialog and a toast that
match the rest of the page, close on Escape, confirm on Enter, and dismiss by
clicking away.
Writing them turned up a bug of the kind that only shows on screen: the scrim
sets display:grid, which beats the hidden attribute's display:none, so the
dialog rendered on page load with no title, no body and an empty button. There
is a global [hidden]{display:none !important} now, because that trap is waiting
for every element that gets a display and a hidden attribute.
Also tightened two endpoints. Asking for a letter with no thread open used to
return 200 and fail inside the worker, so the error arrived by polling rather
than as an answer. Both extract and letter now refuse up front — nothing should
be spent, or waited on, discovering what could have been checked for free.
Verified by clicking: the dialog opens on Reset, resets on confirm, and the
toast carries a server refusal instead of a browser alert.
Send was disabled with a note saying nothing is ever sent. Technically honest, practically a dead button in the middle of the demo. It sends now, into the simulated mailbox this already is: the reply moves to Sent with its attachment, Drafts goes back to empty, and the message stays downloadable from there. The confirmation says plainly that nothing leaves the machine and points at Save .eml for really sending one. Sent is back in the nav, because it does something now. That was the rule the stars and Snoozed failed. Also fixed a false alarm that showed up while testing this. The superseded-without-replacement warning matched parties as opaque strings, so a row attributed to "Meridian Legal / Northwind" matched neither, and the demo warned that a replacement was missing when it was there. Joint parties are now split on separators that are unambiguous — but never on "and" or "&", because "Calder and Vance" is one firm and nothing in the string distinguishes it from two. Tearing a company name in half to match the wrong party would be a worse failure than the missed join it fixes, and there is a test pinning that down. 56 tests.
Nine tests failed at once, all for one reason. This machine started reporting
its hostname as "Unknown_86:ae:73:f8:f0:67" after a network change. The lease
owner id is host:pid:nonce and was parsed with split(":", 2), so a hostname
containing colons made every owner unparseable — _owner_is_alive returned
"unknown", every dead lease looked alive, and no crashed run could reclaim one.
Resume was broken on this machine and would be on anyone else's whose hostname
looks like that.
Parsed from the right now: the pid and nonce are always the last two fields,
whatever the host calls itself. Not sanitised at write time, because leases
written by an earlier version are still in ledgers and have to stay readable.
Two tests cover it: liveness against a colon-laden host directly, and the
end-to-end consequence that a crashed runner's lease is reclaimable on the
machine it crashed on.
Worth recording how it surfaced. Nothing in the code changed — the environment
did. A pre-merge run caught it because the suite exercises real processes and
real leases rather than mocking them, which is the whole argument for testing
that way.
77 tests, 119/119 kill points.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Vishvamsinh Vaghela
Two builds for the SuperDocs engineering task, Round 2.
use-cases/vishvamsinh28/longhaul— my assignedbuild. A durable runner for editing jobs that outlast one sitting: it plans a multi-step
edit across a document set, commits after every step, and on restart resumes from the last
committed step. No step runs twice, none is skipped, and it reports which it skipped and
how it knows.
extensions/vishvamsinh28/thread-to-letter—from the shared open list. Turns a messy email thread into a formal letter or memo stating
only what people actually committed to, attached back to a draft reply. Every candidate
carries a verbatim quote from the thread; no quote, no commitment.
What's proven
SIGKILLto a real subprocess,compared on document HTML, exported bytes and the commit log.
api.superdocs.app: 8 steps → 8 jobs → 0 duplicate turns.How Longhaul works. Each step's instruction carries an idempotency token, so a turn
landing echoes it into the session transcript — which, unlike jobs, never expires. "Did
that happen?" becomes a question the server can answer after a crash. A chat turn is never
retried (a retry is a second edit); an approve always is (it names changes that already
exist). When the evidence genuinely doesn't settle it, the run halts and says so rather
than guessing.
What running them live found. Both have a "what broke" section, and the findings are
the most useful part of this PR. Longhaul: the model silently skipped 2 of 8 replacements
while reporting success — it now checks the document afterwards and reports
blockedrather than claiming work it didn't do. Thread to letter: an extraction dropped the
commitment that superseded an earlier one, which would have produced a letter missing the
obligation that actually stands.
Both use upload/chat/approve/export plus multi-document sessions. Human gate on every
path — terminal, browser and MCP for Longhaul; browser and terminal for Thread to letter.
Full account in
docs/LIVE-RESULTS.md.Each folder has its own README.
make liveruns either against the real API;make proofand
make offlineneed no key.