Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ line below is a mistake made in this codebase, not a hypothetical:
The one reading that wears a state colour without being an agent is the usage
bar: amber from 75% and red (`--danger`) from 90%, because a limit about to
run out is the same verb — something that needs you to act — about the account
rather than about one worktree. It is three 3px tracks in the corner of the
rather than about one worktree. It is two 3px tracks in the corner of the
top bar, nowhere near the row the rule is written to protect.
- **Two faces, one job each.** `--font-mono` for the terminal, patch lines, and
identifiers read character by character. `--font-ui` for everything the
Expand Down
53 changes: 53 additions & 0 deletions server/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ session/ engine (sessions, attachments, sizing) -> tmux -> node-pty
attention.ts idle / working / needs-you
readiness.ts whether it is safe to TYPE into a session -- a stricter question
dispatch.ts hands queued todos to Claude when it comes to rest
resume.ts parks a `continue` behind a Claude that stopped on a usage limit
claude.ts transcripts: --continue, the last prompt, and turn boundaries
(the prompt reader is incremental -- see "What a worktree is
working on")
Expand Down Expand Up @@ -197,6 +198,53 @@ sending it**, and a Return pressed on a screen we misread is not recoverable.
`SWB_DEBUG_DISPATCH=1` logs every verdict change, which is how the predicate was
checked against a real Claude before it was allowed to type anything.

## After a usage limit

When a limit runs out, Claude Code ends the turn and waits for a person, so an
agent working on its own stops and stays stopped after the limit resets.
Nothing inside Claude can undo that -- its `StopFailure` hook fires and its
output is ignored -- so `resume.ts` does it from here, and **types nothing
itself**: it parks a todo, `continue`, queued with `notBefore` set to the
reset, and the dispatcher types it exactly as it types any todo.

- **The stop is read from the transcript, not the screen or a hook.** Measured
in a live transcript: the turn closes with an assistant record carrying
`"error":"rate_limit"` and `"isApiErrorMessage":true`, whose text is `You've
hit your session limit · resets 4:10pm (UTC)`, then `turn_duration`. Going
back from the end, the first prompt or assistant record decides; a prompt
means somebody has spoken since, which is also what makes a retry that stops
again a *new* stop with a record of its own.
- **The reset comes from that sentence first, `/usage` second.** The sentence
has a time and no date, rounded, so it means the next such time with an
hour's grace backwards. Two minutes are added for the rounding, and the wait
is never under five minutes *from the stop* -- an early `continue` is answered
with another stop whose reset has passed, which would otherwise be continued
every fifteen seconds. From the stop rather than from now, so a server that
was down through the reset continues its agents as soon as it is back.
- **Answered stops are remembered in `state.json`** (`limitStops`), apart from
the todo, because the todo can be deleted while the stop is still the last
thing in the transcript.
- **It goes first and holds the queue.** `head` puts it ahead of todos queued
before the stop: it continues the turn they were queued to follow, and they
would only be stopped by the same limit.
- **It is the server's until a person touches it.** Typing into that Claude
drops it rather than handing it back with an error -- measured on a scratch
instance: one keystroke, and the todo was gone. Editing, moving or re-queuing
it makes it an ordinary todo that keeps its wait.

**It can be switched off**, from the top bar: `PUT /api/auto-continue`, kept in
`state.json` as `autoContinue` and on unless switched off. The limit is the
account's, so the switch is passed on to every linked machine (best effort --
one that is off or too old is skipped) and not on again from there, since a
request carrying the peer header is a gateway's. Measured with two scratch
instances: switched off at the gateway, the peer read `false`. Off parks
nothing and withdraws the `continue`s still waiting, forgetting their stops,
so switching it back on answers the agents that are still sitting there.

It looks every fifteen seconds, at the panes' own directories from
`engine.claudeDirs()`: resolving worktrees would run `git status` in every one
of them on a clock that runs with no browser open.

## Worktrees and ids

Worktrees are **discovered** from `git worktree list --porcelain -z` on every
Expand Down Expand Up @@ -496,6 +544,11 @@ Print mode is why this is small: `-p` answers a slash command as plain text, so
there is no pty to drive and no TUI to scrape. `--bare` does **not** work — it
skips whatever handles the command and prints a cost summary instead.

`--no-session-persistence` is load-bearing: without it every reading leaves a
session transcript behind, measured as 1,077 of them (12MB) under the state
directory's project in `~/.claude/projects`. With it, none -- checked by
listing that directory after a reading.

The reading costs a `claude` process, measured 4.3–4.6s, so it is cached for
five minutes and there is no timer: the browser polls on the same interval and a
poll inside the window is answered from the last reading, which also means
Expand Down
40 changes: 24 additions & 16 deletions server/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { StateStore } from './state.js'
import { Workspace } from './workspace.js'
import { registerApi } from './routes/api.js'
import { startDispatcher } from './session/dispatch.js'
import { startResumeWatcher } from './session/resume.js'
import { registerWs, wsPluginOptions } from './routes/ws.js'
import { registerGate } from './gate.js'
import { registerSecurityHeaders } from './headers.js'
Expand Down Expand Up @@ -88,29 +89,36 @@ engine.onSessionGone(() => {
broadcastInvalidate()
})

/** A worktree's absolute path, for reading its transcript. */
const pathFor = async (worktreeId: string): Promise<string | undefined> => {
try {
return (await workspace.resolve(worktreeId)).worktree.path
} catch {
// The worktree is gone or its project is unreadable; its queue waits.
return undefined
}
}

const onTodosChange = (): void => {
workspace.invalidate()
broadcastInvalidate()
}

/*
* Hand queued todos to Claude as it comes to rest.
*
* Started unconditionally, and on its own clock: unlike the worktree poll
* below, this must keep working with every browser closed -- a queue that only
* drains while someone is watching it is a queue you have to watch.
*/
startDispatcher({
store,
engine,
onChange: () => {
workspace.invalidate()
broadcastInvalidate()
},
pathFor: async (worktreeId) => {
try {
return (await workspace.resolve(worktreeId)).worktree.path
} catch {
// The worktree is gone or its project is unreadable; its queue waits.
return undefined
}
},
})
startDispatcher({ store, engine, onChange: onTodosChange, pathFor })

/*
* Park a `continue` behind every Claude that stops on a usage limit, for the
* dispatcher to type once the limit resets. Unconditional for the same reason:
* an agent that ran out overnight is exactly one nobody is watching.
*/
startResumeWatcher({ store, engine, onChange: onTodosChange })

/**
* Notice when an agent changes the repository.
Expand Down
2 changes: 2 additions & 0 deletions server/src/remote/proxy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,8 @@ const ALWAYS_LOCAL: ReadonlySet<string> = new Set([
'/api/health',
'/api/ui',
'/api/servers',
// A switch for this machine, which passes it on to the others itself.
'/api/auto-continue',
/*
* The credential routes, and this is not a formality. `POST /api/login?host=B`
* would forward the password to whatever machine B is -- you would be typing
Expand Down
22 changes: 22 additions & 0 deletions server/src/routes/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,28 @@ export const registerApi = (app: FastifyInstance, deps: ApiDeps): void => {
*/
app.get('/api/usage', async () => usage())

/*
* Whether agents stopped on a usage limit are continued once it resets.
*
* The limit is the account's, not this machine's, so the switch is too: it
* is passed on to every linked machine, each of which continues its own
* agents. Not passed on again from there -- linking is not transitive, and a
* request carrying the peer header is a gateway's -- and a machine that is
* off or too old to have the route is skipped rather than failing the
* switch, since this machine's own agents are answered either way.
*/
app.put('/api/auto-continue', async (request) => {
const { on } = z.object({ on: z.boolean() }).strict().parse(request.body)
store.setAutoContinue(on)
if (request.headers[PEER_READ_HEADER] === undefined) {
await Promise.allSettled(
workspace.peers().map((peer) => peer.request('PUT', '/api/auto-continue', { on }, 5_000)),
)
}
broadcastInvalidate()
return { on }
})

/*
* Whether this machine's Switchboard is behind origin, and updating it.
*
Expand Down
102 changes: 102 additions & 0 deletions server/src/session/claude.ts
Original file line number Diff line number Diff line change
Expand Up @@ -629,3 +629,105 @@ export const turnState = async (cwd: string, now = Date.now()): Promise<TurnStat
// is not still going; let the screen answer instead.
return now - newest.at > IN_TURN_STALE_MS ? 'unknown' : 'in-turn'
}

/** A turn that ended because a usage limit ran out, as the transcript has it. */
export interface LimitStop {
/** The record's own `uuid`, which is what makes one stop distinguishable from the next. */
id: string
/** What Claude said, e.g. `You've hit your session limit · resets 4:10pm (UTC)`. */
text: string
/** Epoch ms the record was written. */
at: number
}

/*
* The two kinds of record that can end the search below, prefiltered the way
* `INTERESTING` is: an assistant record can be hundreds of kilobytes of tool
* call, and only the newest one is ever parsed.
*/
const ASSISTANT = /"type"\s*:\s*"assistant"/

/**
* The record a limit stop is, if one is the newest thing said.
*
* Measured, in a live transcript where an agent ran out of its session limit
* mid-turn (Claude Code v2.1, July 2026): the turn closes with
*
* {"type":"assistant","error":"rate_limit","isApiErrorMessage":true,
* "message":{"content":[{"type":"text","text":"You've hit your session
* limit · resets 4:10pm (UTC)"}]},...}
* {"type":"system","subtype":"turn_duration",...}
*
* and nothing more until a person types. Read from the transcript rather than
* from the screen or a `StopFailure` hook: the `error` field is a structured
* value where the screen is prose, and a hook would mean writing into the
* settings of every Claude this starts -- attention is inferred here, never
* reported, and this is the same kind of question.
*
* Searching back from the end, the first prompt or assistant record decides:
* a prompt means somebody has spoken since (a retry, `/login`, "continue"), and
* any other assistant record means Claude is past it.
*/
const limitStopIn = (text: string): LimitStop | null => {
const lines = text.split('\n')
for (let index = lines.length - 1; index >= 0; index--) {
const line = lines[index]
if (line === undefined) continue
if (ASSISTANT.test(line)) {
let row: unknown
try {
row = JSON.parse(line)
} catch {
continue
}
const record = row as {
type?: unknown
error?: unknown
isSidechain?: unknown
uuid?: unknown
timestamp?: unknown
message?: { content?: unknown }
}
if (record.type !== 'assistant') continue
// A subagent's limit is its parent's to report, and the parent writes
// its own record when it does.
if (record.isSidechain === true) continue
if (record.error !== 'rate_limit' || typeof record.uuid !== 'string') return null
const content = record.message?.content
const said = Array.isArray(content)
? content
.map((part) => (part as { text?: unknown } | null)?.text)
.filter((part): part is string => typeof part === 'string')
.join(' ')
: typeof content === 'string'
? content
: ''
const at = typeof record.timestamp === 'string' ? Date.parse(record.timestamp) : NaN
return { id: record.uuid, text: said.trim(), at: Number.isFinite(at) ? at : Date.now() }
}
if (markOf(line)?.kind === 'prompt') return null
}
return null
}

/** The last answer per directory, kept while the file has not moved; see `marks`. */
const stops = new Map<string, { path: string; at: number; stop: LimitStop | null }>()

/**
* Whether this worktree's Claude is sitting on a usage-limit stop, and which.
*
* Null for anything else, including no transcript at all.
*/
export const limitStop = async (cwd: string): Promise<LimitStop | null> => {
const newest = await newestTranscript(transcriptDir(cwd))
if (newest === null) return null
const cached = stops.get(cwd)
if (cached !== undefined && cached.path === newest.path && cached.at === newest.at) {
return cached.stop
}
const text = await readTail(newest.path)
if (text === null) return null
const stop = limitStopIn(text)
stops.set(cwd, { path: newest.path, at: newest.at, stop })
return stop
}
25 changes: 22 additions & 3 deletions server/src/session/dispatch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -81,14 +81,26 @@ export const startDispatcher = (opts: {
const notBefore = new Map<string, number>()
const reasons = new Map<string, NotReady | 'no-session' | 'sent'>()

/** The head of a worktree's queue: the todo whose RUN NEXT was pressed first. */
/**
* The head of a worktree's queue: the todo whose RUN NEXT was pressed first --
* unless Claude stopped on a usage limit, and then the `continue` that
* resumes it (`resume.ts`). That one goes first because what it continues is
* the turn the limit interrupted, which everything queued behind it was
* queued to follow; and because it waits for the reset, it holds the rest of
* the queue back with it rather than letting a todo be typed into an account
* that would only stop it again.
*/
const head = (worktreeId: string): WorktreeTodo | undefined =>
store.todos
.filter(
(t) =>
t.worktreeId === worktreeId && t.queuedAt !== undefined && t.dispatchingAt === undefined,
)
.sort((a, b) => (a.queuedAt ?? 0) - (b.queuedAt ?? 0))[0]
.sort(
(a, b) =>
Number(b.limitStop !== undefined) - Number(a.limitStop !== undefined) ||
(a.queuedAt ?? 0) - (b.queuedAt ?? 0),
)[0]

/** The first line of the prompt, which is what the box should be showing. */
const opening = (prompt: string): string =>
Expand Down Expand Up @@ -222,7 +234,7 @@ export const startDispatcher = (opts: {
...state,
turn,
queuedAt: todo.queuedAt ?? 0,
notBefore: notBefore.get(worktreeId) ?? 0,
notBefore: Math.max(notBefore.get(worktreeId) ?? 0, todo.notBefore ?? 0),
})
if (!verdict.ready && verdict.why === 'typed-after-queue') {
/*
Expand All @@ -234,6 +246,13 @@ export const startDispatcher = (opts: {
for (const t of store.todos) {
if (t.worktreeId !== worktreeId || t.queuedAt === undefined) continue
if (t.dispatchingAt !== undefined || t.queuedAt >= state.lastUserInputAt) continue
// The server's own `continue` is not handed back: the human has
// already done what it was waiting to do, and a stray "continue"
// left in the list would read as something they had written.
if (t.limitStop !== undefined) {
store.removeTodo(t.id)
continue
}
store.patchTodo(t.id, {
queuedAt: undefined,
lastError: 'You typed into Claude after queueing this, so it was not sent. Queue it again?',
Expand Down
14 changes: 14 additions & 0 deletions server/src/session/engine.ts
Original file line number Diff line number Diff line change
Expand Up @@ -714,6 +714,20 @@ export class SessionEngine {
return [...this.sessions.values()].map((live) => live.toRecord())
}

/**
* Every running Claude's worktree and directory, for reading its transcript.
*
* The directory is the pane's, from tmux, which is what `refreshAttention`
* reads the turn record from; asking `workspace.resolve` instead would list
* every project's worktrees with `git status` in each, on a clock that runs
* with no browser open.
*/
claudeDirs(): { worktreeId: string; cwd: string }[] {
return [...this.sessions.values()]
.filter((live) => live.record.kind === 'claude' && !live.dead && live.cwd !== '')
.map((live) => ({ worktreeId: live.record.worktreeId, cwd: live.cwd }))
}

listForWorktree(worktreeId: string): Session[] {
return this.list().filter((s) => s.worktreeId === worktreeId)
}
Expand Down
Loading
Loading