Extract shared shell tokenizer from permission and display modules

[TheGreatAxios]

Summary

The permission gate and TUI display each carried their own shell tokenizer with verbatim duplicate helpers (isRedirectAmpersand, heredoc scanners, quote-state tracking). Two separate state machines over the same grammar is a security risk: the approval dialog can segment a command differently than the gate that authorizes it.

Changes

• New src/permission/shell-tokenizer.ts — shared module containing isRedirectAmpersand, parseHeredocOpener, and updateQuoteState (the three helpers that were duplicated)
• Modified src/permission/command.ts — imports from shell-tokenizer.ts instead of defining its own copies
• Modified src/tui/command-display.ts — imports from shell-tokenizer.ts, replacing parseHeredocMarker with parseHeredocOpener (with null fallback for backward compatibility)
• New src/permission/shell-tokenizer-property.test.ts — property test asserting that for a corpus of chained commands (quotes, heredocs, subshells, operators), display segments are a coarse merge of authz segments (never a different parse)

Verification

• Full gate passes: lint, typecheck, build, test (5918 tests, 0 failures)
• All existing permission and display test suites pass unmodified
• 17-entry corpus property test covers the agreement invariant

Fixes CL-6808

[linear-code]

CL-6808

[TheGreatAxios]

No blocking correctness findings.\n\nCould we add to the agreement corpus? The existing here-string case stays on one line, so it does not cover the newline boundary that motivated rejecting as a heredoc opener. This will retain coverage for the authorization/display alignment in that security-sensitive case.

[TheGreatAxios]

No blocking correctness findings. Could we add the newline-shaped here-string case (cat <<< payload followed by a newline and rm -rf /) to the agreement corpus? The existing here-string case stays on one line, so it does not cover the boundary that motivated rejecting a here-string as a heredoc opener. This retains coverage for authorization/display alignment in that security-sensitive case.

[TheGreatAxios]

Review · Blocking

The permission gate and TUI share shell-tokenizer primitives so their command segmentation stays aligned.

Finding

• src/permission/shell-tokenizer.ts:20 — parseHeredocOpener rejects <<< only at its first <, but the caller retries at the second < and interprets the remaining << as a heredoc. For cat <<< payload\nrm -rf /, both paths return one segment and the gate can offer a reusable cat * scope covering the destructive tail. Could we reject heredoc recognition at every position inside a here-string and assert the concrete two-segment result?

The agreement test at src/permission/shell-tokenizer-property.test.ts:25 currently passes because both consumers share the same incorrect parse. GitHub CI is green, but this authorization-scope defect should be resolved before merge.

[TheGreatAxios]

Review · Blocking

The here-string boundary is fixed at the current head, but the full shared-tokenizer diff still has one authorization boundary defect.

• src/permission/command.ts:44 — After a real heredoc terminator, the parser clears heredocMarker and consumes the terminator newline without pushing the completed command. cat <<EOF\nsafe\nEOF\nrm -rf build remains one authorization segment while display shows two, allowing a reusable cat * scope to cover the destructive tail. Could we split at that terminator boundary and replace the known-divergence exemption with concrete authorization/display and grant-scope assertions?

Targeted tests and bun run check pass, so this requires a new regression rather than relying on the existing agreement corpus.

[TheGreatAxios]

Pausing this implementation rather than continuing iterative tokenizer patches. The branch remains available, but it is not safe to merge in its current form and needs a fresh scope/design review before any further work.