From 5fc679d941663be1616b2d2197ed54c895d07d93 Mon Sep 17 00:00:00 2001 From: Raghu Betina Date: Sat, 29 Aug 2026 04:03:54 -0500 Subject: [PATCH 1/9] Define root Compilation output --- docs/commands.md | 52 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 45 insertions(+), 7 deletions(-) diff --git a/docs/commands.md b/docs/commands.md index b5332ff..4c97530 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -153,9 +153,41 @@ the accepted source SHA-256 from the saved ETag, hashes the current local bytes, ### Materialize a direct Compilation -With `--output`, the CLI requires an explicit absent destination beneath an existing real directory. It validates -that destination before pushing the Plan, checks it again after analysis, and never overwrites, merges into, or -repairs an existing path. +With `--output`, the CLI accepts either an explicit absent destination beneath an existing real directory or the +exact current directory (`.`). It validates that destination before pushing the Plan and checks it again after +analysis. Other existing destinations remain invalid, so `--output ./application` retains its absent-directory +contract. + +`--output .` is the noninteractive root-adoption mode. It works in an arbitrary real current directory and does not +recognize Drawing Board or another repository layout specially. Before starting Compilation, the CLI requires: + +- the current directory and its parent to be real directories, with the current directory not a filesystem root; +- no existing top-level path whose portable, case-insensitive name is `design`; +- only regular files and real directories in the existing tree, with no symbolic links, special files, nested Git + repositories, or mounted subtrees; and +- when the current directory is a Git worktree, a clean tracked worktree and index. Untracked and ignored design + material may remain present. + +After the accepted Plan and matching valid Analysis, the CLI captures the exact preexisting path, type, mode, and +file-byte inventory immediately before it starts Compilation. It also captures the worktree's untracked and ignored +paths without changing the index. A filesystem change while Compilation or artifact download is in progress stops +materialization instead of guessing which state to preserve. + +The complete generated artifact is written and verified in a uniquely created sibling transaction directory on the +same filesystem before any existing path moves. The artifact may not own `design` or `.git`. The transaction then +moves every preexisting non-Git entry, including the private `.firstdraft` authoring state, under the fixed +`./design` directory; keeps an existing top-level `.git` repository at the root; and installs the exact artifact as +the root application. If the root contains no entry other than `.git`, it does not create an empty `design` +directory. A failed rename or post-install verification reverses the completed renames and removes only the +transaction directory. A rollback failure leaves the owned transaction journal intact and reports +`materialization_failed` rather than deleting either copy. + +Git history is preserved, but the CLI does not rewrite or stage the index: formerly tracked design paths therefore +appear as root deletions plus untracked paths beneath `design` until the caller deliberately stages the new layout. +Previously untracked paths remain untracked. Every previously ignored path must still be ignored at its relocated +`design/...` path after the switch; otherwise the CLI rolls the whole transaction back. This preserves ordinary +relative `.gitignore` protection without editing `.git/info/exclude` or a configured global excludes file. A +non-Git root remains non-Git. After valid analysis, the CLI requests one Compilation for that exact reviewed Head and never starts GitHub Publication. It validates that the `202` response identifies the same Project, graph version, Head, Analysis, @@ -166,10 +198,12 @@ and materialization failures retain the last validated Compilation projection so without starting duplicate work. Follow the [direct Compilation recovery procedure](errors.md#direct-compilation-recovery). Success writes one JSON object to stdout containing the validated Project, Compilation, and absolute output path. -The installed directory contains exactly the artifact files and modes; the CLI does not add a Git repository, run a -formatter, or repair generated source. When the output is nested inside another Git worktree, initialize the -application as its own repository before running generated checks that inspect Git; otherwise Git resolves to the -parent worktree. Progress on stderr reports analysis and Compilation only. +An absent output directory contains exactly the artifact files and modes. Root adoption additionally contains the +preserved `design` directory and an existing root `.git`, when present; every artifact-owned path remains exact. The +CLI does not add a Git repository, run a formatter, or repair generated source. When an absent output is nested +inside another Git worktree, initialize the application as its own repository before running generated checks that +inspect Git; otherwise Git resolves to the parent worktree. Progress on stderr reports analysis and Compilation +only. ### Publish through GitHub @@ -232,6 +266,10 @@ Materialize an already successful Compilation into an absent path: firstdraft compilation download 01900000-0000-7000-8000-000000000001 --output ../movie-catalog ``` +The same command accepts `--output .` and applies the root-adoption transaction above. This is the recovery path +when a retained direct Compilation succeeded but its earlier root materialization did not complete; it never starts +replacement work. + The command validates the UUID and output path before network access, makes one status `GET`, requires `succeeded`, and makes one artifact `GET`. It never starts work or polls. Historical artifact validation uses the retained `compilation.head_source_sha256`, not the current local Plan or ETag, to pin the artifact's exact From 6bb2e2682b93dff6dde3cf03da93efd82ca61820 Mon Sep 17 00:00:00 2001 From: Raghu Betina Date: Sat, 29 Aug 2026 04:16:10 -0500 Subject: [PATCH 2/9] Close root output transaction gaps --- docs/commands.md | 101 ++++++++++++++++++++++++++++++----------------- docs/errors.md | 13 ++++-- 2 files changed, 74 insertions(+), 40 deletions(-) diff --git a/docs/commands.md b/docs/commands.md index 4c97530..a7126c7 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -153,41 +153,60 @@ the accepted source SHA-256 from the saved ETag, hashes the current local bytes, ### Materialize a direct Compilation -With `--output`, the CLI accepts either an explicit absent destination beneath an existing real directory or the -exact current directory (`.`). It validates that destination before pushing the Plan and checks it again after -analysis. Other existing destinations remain invalid, so `--output ./application` retains its absent-directory -contract. - -`--output .` is the noninteractive root-adoption mode. It works in an arbitrary real current directory and does not -recognize Drawing Board or another repository layout specially. Before starting Compilation, the CLI requires: - -- the current directory and its parent to be real directories, with the current directory not a filesystem root; -- no existing top-level path whose portable, case-insensitive name is `design`; -- only regular files and real directories in the existing tree, with no symbolic links, special files, nested Git - repositories, or mounted subtrees; and -- when the current directory is a Git worktree, a clean tracked worktree and index. Untracked and ignored design - material may remain present. - -After the accepted Plan and matching valid Analysis, the CLI captures the exact preexisting path, type, mode, and -file-byte inventory immediately before it starts Compilation. It also captures the worktree's untracked and ignored -paths without changing the index. A filesystem change while Compilation or artifact download is in progress stops -materialization instead of guessing which state to preserve. - -The complete generated artifact is written and verified in a uniquely created sibling transaction directory on the -same filesystem before any existing path moves. The artifact may not own `design` or `.git`. The transaction then -moves every preexisting non-Git entry, including the private `.firstdraft` authoring state, under the fixed -`./design` directory; keeps an existing top-level `.git` repository at the root; and installs the exact artifact as -the root application. If the root contains no entry other than `.git`, it does not create an empty `design` -directory. A failed rename or post-install verification reverses the completed renames and removes only the -transaction directory. A rollback failure leaves the owned transaction journal intact and reports -`materialization_failed` rather than deleting either copy. - -Git history is preserved, but the CLI does not rewrite or stage the index: formerly tracked design paths therefore -appear as root deletions plus untracked paths beneath `design` until the caller deliberately stages the new layout. -Previously untracked paths remain untracked. Every previously ignored path must still be ignored at its relocated -`design/...` path after the switch; otherwise the CLI rolls the whole transaction back. This preserves ordinary -relative `.gitignore` protection without editing `.git/info/exclude` or a configured global excludes file. A -non-Git root remains non-Git. +With `--output`, the CLI accepts either an explicit absent destination beneath an existing real directory or a path +that resolves to the physical current directory. It validates that destination before pushing the Plan and checks +it again after analysis. Other existing destinations remain invalid, so `--output ./application` retains its +absent-directory contract. + +`--output .` is the noninteractive root-adoption mode. `./`, an absolute spelling of the current directory, and +another spelling that resolves to that same physical directory select the same mode. It works at any real current +directory that meets the preconditions below and does not recognize Drawing Board or another repository layout +specially. Before starting Compilation, the CLI requires: + +- the current directory to be a real, writable, non-filesystem-root directory; +- no existing top-level path whose portable, case-insensitive name is `design` or the reserved + `.firstdraft-root-output` transaction path; +- every top-level entry other than `.git` to be a regular file or real directory on the current directory's + filesystem. Interior symlinks, dependency trees, sockets, and nested repositories move opaquely with their + top-level directory; the CLI neither follows nor repairs them; and +- when the current directory is the root of a Git worktree, a clean tracked worktree and index with no unmerged + entries, sparse checkout, or in-progress merge, rebase, cherry-pick, or revert. Untracked and ignored design + material may remain present. A directory nested inside a higher Git worktree is refused rather than treated as + non-Git. A valid top-level `.git` file for a linked worktree is retained like a `.git` directory. + +Git-backed root adoption invokes the installed Git executable explicitly. Read-only discovery uses +`git --no-optional-locks` with stable NUL-delimited porcelain so it does not refresh the index. Before remote work, +the CLI verifies in a temporary preview that every currently ignored entry remains ignored after its path and +applicable worktree `.gitignore` files move beneath `design`; repository-local and configured global exclusions are +both honored. A refusal is `invalid_output_path` with a machine-readable `reason` and happens before Plan push. + +After the accepted Plan and matching valid Analysis, the CLI creates `.firstdraft-root-output` with exclusive +creation. That directory is both the single-writer lock and the owned transaction journal. It captures the exact +top-level entry identities immediately before it starts Compilation and rechecks them immediately before moving +anything. A detected top-level change stops materialization. Interior changes are not recursively hashed: the +top-level directory is moved intact at the transaction boundary. + +The complete generated artifact is written and verified inside that in-root transaction directory before any +existing path moves. Staging inside the destination makes every later rename same-filesystem even when the current +directory itself is a container mount point. The artifact may not own a top-level path whose portable, +case-insensitive name is `design` or `.firstdraft-root-output`; artifact validation already excludes `.git` at any +depth. + +The transaction creates `./design` with mode `0755` on POSIX, moves every preexisting non-Git top-level entry under +it, keeps an existing top-level `.git` file or directory at the root, and installs the artifact's top-level entries +at the root. If the root contains no entry other than `.git`, it does not retain an empty `design` directory. A +failed rename or post-install verification reverses the completed renames and removes only the owned transaction. +If rollback itself cannot finish, `materialization_failed` reports `reason: "root_rollback_incomplete"` and leaves +`.firstdraft-root-output` in place as the recovery journal rather than deleting either copy. Another root adoption +is refused until that state is reconciled. + +For a Git root, the same transaction atomically replaces the index with a prepared index that stages each formerly +tracked path at `design/` and stages every exact generated artifact path at the root. This handles +overlapping names such as `README.md` and `.gitignore` without leaving the old design blob indexed at a generated +path. Previously untracked and ignored paths are never added to the index; the preflighted ignore protection is +rechecked after the move. `HEAD`, refs, configuration, and history do not change. The caller should inspect and +commit this staged root-adoption change before using destructive worktree or index restoration commands. A non-Git +root remains non-Git and is not initialized. After valid analysis, the CLI requests one Compilation for that exact reviewed Head and never starts GitHub Publication. It validates that the `202` response identifies the same Project, graph version, Head, Analysis, @@ -198,6 +217,8 @@ and materialization failures retain the last validated Compilation projection so without starting duplicate work. Follow the [direct Compilation recovery procedure](errors.md#direct-compilation-recovery). Success writes one JSON object to stdout containing the validated Project, Compilation, and absolute output path. +Root adoption additionally reports `root_adoption.design_path` (or `null` when no design directory was needed), its +top-level moved-entry count, whether a Git repository was preserved, and whether its index was replaced. An absent output directory contains exactly the artifact files and modes. Root adoption additionally contains the preserved `design` directory and an existing root `.git`, when present; every artifact-owned path remains exact. The CLI does not add a Git repository, run a formatter, or repair generated source. When an absent output is nested @@ -267,8 +288,14 @@ firstdraft compilation download 01900000-0000-7000-8000-000000000001 --output .. ``` The same command accepts `--output .` and applies the root-adoption transaction above. This is the recovery path -when a retained direct Compilation succeeded but its earlier root materialization did not complete; it never starts -replacement work. +when a retained direct Compilation succeeded but an earlier root materialization failed _and fully rolled back_; it +never starts replacement work. An incomplete rollback leaves `.firstdraft-root-output` and requires journal +reconciliation before this command can run again. + +Successful root adoption is intentionally one-way. The original `.firstdraft` authoring state moves under +`design/.firstdraft`; run later Plan commands from `design`, not from the generated application root. Compiling a +later Plan revision does not overwrite an already adopted root: choose a new absent output and deliberately +reconcile it with application work. The command validates the UUID and output path before network access, makes one status `GET`, requires `succeeded`, and makes one artifact `GET`. It never starts work or polls. Historical artifact validation uses the retained diff --git a/docs/errors.md b/docs/errors.md index 18b87aa..9704578 100644 --- a/docs/errors.md +++ b/docs/errors.md @@ -68,7 +68,9 @@ materialization failure envelopes include that last validated projection as `cur - after `artifact_unavailable`, wait if appropriate and use `firstdraft compilation download --output `; and - after `materialization_failed`, repair the destination condition, then use the same lower-level download command - with a new absent path. + with a new absent path. A root-output attempt whose transaction fully rolled back may instead retry that retained + download with `--output .`; `reason: "root_rollback_incomplete"` requires reconciliation of the retained + `.firstdraft-root-output` journal before another root attempt. After `compilation_wait_timed_out`, retained work may still continue. Use `firstdraft compilation status ` for one read-only status check; do not rerun @@ -118,5 +120,10 @@ stopped without following the replacement. | `compilation status --wait` | `compilation_changed`, `compilation_wait_timed_out` | 1 | Retained identity/provenance changed or the wait ended. | | `compilation download` | `compilation_not_succeeded` | 1 | Status was not `succeeded`; no artifact request was made. | | Download commands | `artifact_unavailable`, `invalid_artifact` | 1 | Artifact transport or integrity validation failed; direct Compile post-start errors include `current`. | -| Download commands | `invalid_output_path` | 2 | The destination was not an absent path beneath an existing real directory. | -| Download commands | `materialization_failed` | 1 | The output raced or installation failed; direct Compile post-start errors include `current`. | +| Download commands | `invalid_output_path` | 2 | The absent destination or root-adoption preconditions failed; `reason` identifies the stable refusal. | +| Download commands | `materialization_failed` | 1 | The output changed or its transaction failed; `reason` identifies incomplete rollback when applicable. | + +Root-output `invalid_output_path.reason` values are `destination_exists`, `root_not_real`, `root_reserved_path`, +`root_entry_unsupported`, `root_enclosing_worktree`, `root_git_unavailable`, `root_git_unsupported`, +`root_git_dirty`, `root_ignore_not_preserved`, and `root_busy`. Root-output `materialization_failed.reason` values +are `output_changed`, `root_artifact_collision`, `root_transaction_failed`, and `root_rollback_incomplete`. From 1a89dcf2324cf209c473b2404e6b643032251bda Mon Sep 17 00:00:00 2001 From: Raghu Betina Date: Sat, 29 Aug 2026 04:23:54 -0500 Subject: [PATCH 3/9] Specify root output recovery --- docs/commands.md | 56 ++++++++++++++++++++++++++++++++---------------- docs/errors.md | 15 ++++++++----- 2 files changed, 47 insertions(+), 24 deletions(-) diff --git a/docs/commands.md b/docs/commands.md index a7126c7..fe0f226 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -171,8 +171,10 @@ specially. Before starting Compilation, the CLI requires: top-level directory; the CLI neither follows nor repairs them; and - when the current directory is the root of a Git worktree, a clean tracked worktree and index with no unmerged entries, sparse checkout, or in-progress merge, rebase, cherry-pick, or revert. Untracked and ignored design - material may remain present. A directory nested inside a higher Git worktree is refused rather than treated as - non-Git. A valid top-level `.git` file for a linked worktree is retained like a `.git` directory. + material may remain present. Submodules and tracked `.gitmodules` files are refused in this first root-adoption + contract rather than moved with broken Git wiring. A directory nested inside a higher Git worktree is refused + rather than treated as non-Git. A valid top-level `.git` file for a linked worktree is retained like a `.git` + directory. Git-backed root adoption invokes the installed Git executable explicitly. Read-only discovery uses `git --no-optional-locks` with stable NUL-delimited porcelain so it does not refresh the index. Before remote work, @@ -180,11 +182,13 @@ the CLI verifies in a temporary preview that every currently ignored entry remai applicable worktree `.gitignore` files move beneath `design`; repository-local and configured global exclusions are both honored. A refusal is `invalid_output_path` with a machine-readable `reason` and happens before Plan push. -After the accepted Plan and matching valid Analysis, the CLI creates `.firstdraft-root-output` with exclusive -creation. That directory is both the single-writer lock and the owned transaction journal. It captures the exact -top-level entry identities immediately before it starts Compilation and rechecks them immediately before moving -anything. A detected top-level change stops materialization. Interior changes are not recursively hashed: the -top-level directory is moved intact at the transaction boundary. +The CLI creates `.firstdraft-root-output` with exclusive creation during the pre-push output check and holds it +through analysis, Compilation, and materialization. That directory is both the single-writer lock and the owned +transaction journal, so a concurrent root adoption is refused before either command sends a request. Immediately +after acquiring it, the CLI captures every other top-level entry's exact name, entry type, device, and inode. It +rechecks that set immediately before moving anything. Size, modification time, and contents are deliberately not +part of this identity: interior changes are not recursively inventoried, and a top-level directory moves intact at +the transaction boundary. A replaced, added, or removed top-level entry stops materialization. The complete generated artifact is written and verified inside that in-root transaction directory before any existing path moves. Staging inside the destination makes every later rename same-filesystem even when the current @@ -195,18 +199,32 @@ depth. The transaction creates `./design` with mode `0755` on POSIX, moves every preexisting non-Git top-level entry under it, keeps an existing top-level `.git` file or directory at the root, and installs the artifact's top-level entries at the root. If the root contains no entry other than `.git`, it does not retain an empty `design` directory. A -failed rename or post-install verification reverses the completed renames and removes only the owned transaction. -If rollback itself cannot finish, `materialization_failed` reports `reason: "root_rollback_incomplete"` and leaves -`.firstdraft-root-output` in place as the recovery journal rather than deleting either copy. Another root adoption -is refused until that state is reconciled. - -For a Git root, the same transaction atomically replaces the index with a prepared index that stages each formerly -tracked path at `design/` and stages every exact generated artifact path at the root. This handles -overlapping names such as `README.md` and `.gitignore` without leaving the old design blob indexed at a generated -path. Previously untracked and ignored paths are never added to the index; the preflighted ignore protection is -rechecked after the move. `HEAD`, refs, configuration, and history do not change. The caller should inspect and -commit this staged root-adoption change before using destructive worktree or index restoration commands. A non-Git -root remains non-Git and is not initialized. +nested mount that cannot travel with its top-level directory may make its rename fail; that is a transactional +failure, not permission to copy or traverse the mount. + +For a Git root, the CLI first prepares a replacement index that stages each formerly tracked path at +`design/` and stages every exact generated artifact path at the root. This handles overlapping names such +as `README.md` and `.gitignore` without leaving the old design blob indexed at a generated path. Previously +untracked and ignored paths are never added to the index. After the worktree renames finish, the CLI installs the +prepared index through Git's actual index lock path and atomic lock-file commit protocol, including in a linked +worktree whose index is outside the adopted root. The transaction journal retains whether an index existed plus an +exact private copy, mode, and digest of its prior bytes until final verification succeeds. The preflighted ignore +protection is rechecked after the move. `HEAD`, refs, configuration, and history do not change. The caller should +inspect and commit this staged root-adoption change before using destructive worktree or index restoration commands. +A non-Git root remains non-Git and is not initialized. + +The journal is a versioned private JSON record plus owned staging files. It records the physical root and original +top-level identity set, the transaction phase, completed design and artifact renames, and, for Git, the resolved +index path and original and prepared index digests. Each irreversible phase is recorded before the next one starts. +On any failure after a move, index installation, or post-install verification, the CLI first restores the exact +prior index through the same Git lock boundary, then reverses artifact and design renames in journal order. A fully +successful rollback removes only the owned transaction. If rollback itself cannot finish, +`materialization_failed` reports `reason: "root_rollback_incomplete"` and includes +`recovery_path: ".firstdraft-root-output"`; it leaves the journal and owned copies in place rather than guessing. +Do not delete that directory or run Git restoration commands. Inspect the versioned journal, restore the listed +index and paths to its recorded original identities, verify that snapshot, and only then remove the transaction +directory. Another root adoption reports `root_busy` until that state is reconciled; a foreign preexisting directory +with the same reserved name reports `root_reserved_path`. After valid analysis, the CLI requests one Compilation for that exact reviewed Head and never starts GitHub Publication. It validates that the `202` response identifies the same Project, graph version, Head, Analysis, diff --git a/docs/errors.md b/docs/errors.md index 9704578..4a028af 100644 --- a/docs/errors.md +++ b/docs/errors.md @@ -70,7 +70,10 @@ materialization failure envelopes include that last validated projection as `cur - after `materialization_failed`, repair the destination condition, then use the same lower-level download command with a new absent path. A root-output attempt whose transaction fully rolled back may instead retry that retained download with `--output .`; `reason: "root_rollback_incomplete"` requires reconciliation of the retained - `.firstdraft-root-output` journal before another root attempt. + `.firstdraft-root-output` journal before another root attempt. Do not delete that journal or run Git restoration + commands: read its versioned record, restore the exact listed index and path identities, verify its original + snapshot, and then remove the transaction directory. After `.firstdraft` has moved successfully under `design`, + run retained status and later Plan commands from `design`, not from the generated application root. After `compilation_wait_timed_out`, retained work may still continue. Use `firstdraft compilation status ` for one read-only status check; do not rerun @@ -123,7 +126,9 @@ stopped without following the replacement. | Download commands | `invalid_output_path` | 2 | The absent destination or root-adoption preconditions failed; `reason` identifies the stable refusal. | | Download commands | `materialization_failed` | 1 | The output changed or its transaction failed; `reason` identifies incomplete rollback when applicable. | -Root-output `invalid_output_path.reason` values are `destination_exists`, `root_not_real`, `root_reserved_path`, -`root_entry_unsupported`, `root_enclosing_worktree`, `root_git_unavailable`, `root_git_unsupported`, -`root_git_dirty`, `root_ignore_not_preserved`, and `root_busy`. Root-output `materialization_failed.reason` values -are `output_changed`, `root_artifact_collision`, `root_transaction_failed`, and `root_rollback_incomplete`. +Root-output `invalid_output_path.reason` values are `destination_exists`, `root_not_real`, `root_not_writable`, +`root_reserved_path`, `root_entry_unsupported`, `root_enclosing_worktree`, `root_git_unavailable`, +`root_git_unsupported`, `root_git_dirty`, `root_ignore_not_preserved`, and `root_busy`. Git refusing discovery, +including a `safe.directory` refusal, is `root_git_unavailable`; a discovered but unsupported Git shape such as a +submodule is `root_git_unsupported`. Root-output `materialization_failed.reason` values are `output_changed`, +`root_artifact_collision`, `root_ignore_changed`, `root_transaction_failed`, and `root_rollback_incomplete`. From eded80fb841b563ad59806ea740d78d3f86e2fc8 Mon Sep 17 00:00:00 2001 From: Raghu Betina Date: Sat, 29 Aug 2026 04:28:52 -0500 Subject: [PATCH 4/9] Close root output lock lifecycle --- docs/commands.md | 23 +++++++++++++++++------ docs/errors.md | 20 +++++++++++--------- 2 files changed, 28 insertions(+), 15 deletions(-) diff --git a/docs/commands.md b/docs/commands.md index fe0f226..4569f41 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -154,14 +154,16 @@ the accepted source SHA-256 from the saved ETag, hashes the current local bytes, ### Materialize a direct Compilation With `--output`, the CLI accepts either an explicit absent destination beneath an existing real directory or a path -that resolves to the physical current directory. It validates that destination before pushing the Plan and checks -it again after analysis. Other existing destinations remain invalid, so `--output ./application` retains its -absent-directory contract. +that resolves to the physical current directory. It validates either destination before pushing the Plan. An absent +destination is checked again after analysis; root adoption instead holds its owned lock and performs the exact +pre-move identity recheck described below. Other existing destinations remain invalid, so +`--output ./application` retains its absent-directory contract. `--output .` is the noninteractive root-adoption mode. `./`, an absolute spelling of the current directory, and another spelling that resolves to that same physical directory select the same mode. It works at any real current directory that meets the preconditions below and does not recognize Drawing Board or another repository layout -specially. Before starting Compilation, the CLI requires: +specially. This first root-adoption contract supports POSIX filesystems; Windows retains absent-directory output +and refuses root adoption as `root_platform_unsupported`. Before starting Compilation, the CLI requires: - the current directory to be a real, writable, non-filesystem-root directory; - no existing top-level path whose portable, case-insensitive name is `design` or the reserved @@ -188,7 +190,14 @@ transaction journal, so a concurrent root adoption is refused before either comm after acquiring it, the CLI captures every other top-level entry's exact name, entry type, device, and inode. It rechecks that set immediately before moving anything. Size, modification time, and contents are deliberately not part of this identity: interior changes are not recursively inventoried, and a top-level directory moves intact at -the transaction boundary. A replaced, added, or removed top-level entry stops materialization. +the transaction boundary. A replaced, added, or removed top-level entry stops materialization. The reserved-path +precondition ignores only the transaction directory created and still held by this invocation. + +`compilation download --output .` acquires the same lock before its first status request and holds it through +artifact download and materialization. Either command removes its own transaction directory on every ordinary exit +before the journal records an irreversible move or index installation. Its signal handlers do the same when Node +dispatches the signal before that boundary. A journal whose phase records no irreversible operation is likewise +safe to remove; the manual reconciliation rule below applies only after `root_rollback_incomplete`. The complete generated artifact is written and verified inside that in-root transaction directory before any existing path moves. Staging inside the destination makes every later rename same-filesystem even when the current @@ -198,7 +207,9 @@ depth. The transaction creates `./design` with mode `0755` on POSIX, moves every preexisting non-Git top-level entry under it, keeps an existing top-level `.git` file or directory at the root, and installs the artifact's top-level entries -at the root. If the root contains no entry other than `.git`, it does not retain an empty `design` directory. A +at the root. Immediately before each artifact entry is installed, its root destination must still be absent; an +unexpected entry stops the transaction and is never overwritten. If the root contains no entry other than `.git`, +it does not retain an empty `design` directory. A nested mount that cannot travel with its top-level directory may make its rename fail; that is a transactional failure, not permission to copy or traverse the mount. diff --git a/docs/errors.md b/docs/errors.md index 4a028af..124f6f7 100644 --- a/docs/errors.md +++ b/docs/errors.md @@ -70,10 +70,11 @@ materialization failure envelopes include that last validated projection as `cur - after `materialization_failed`, repair the destination condition, then use the same lower-level download command with a new absent path. A root-output attempt whose transaction fully rolled back may instead retry that retained download with `--output .`; `reason: "root_rollback_incomplete"` requires reconciliation of the retained - `.firstdraft-root-output` journal before another root attempt. Do not delete that journal or run Git restoration - commands: read its versioned record, restore the exact listed index and path identities, verify its original - snapshot, and then remove the transaction directory. After `.firstdraft` has moved successfully under `design`, - run retained status and later Plan commands from `design`, not from the generated application root. + `.firstdraft-root-output` journal before another root attempt. Only that reason requires manual reconciliation: + do not delete its journal or run Git restoration commands; read its versioned record, restore the exact listed + index and path identities, verify its original snapshot, and then remove the transaction directory. A journal + whose phase records no irreversible operation is safe to remove. After `.firstdraft` has moved successfully under + `design`, run retained status and later Plan commands from `design`, not from the generated application root. After `compilation_wait_timed_out`, retained work may still continue. Use `firstdraft compilation status ` for one read-only status check; do not rerun @@ -127,8 +128,9 @@ stopped without following the replacement. | Download commands | `materialization_failed` | 1 | The output changed or its transaction failed; `reason` identifies incomplete rollback when applicable. | Root-output `invalid_output_path.reason` values are `destination_exists`, `root_not_real`, `root_not_writable`, -`root_reserved_path`, `root_entry_unsupported`, `root_enclosing_worktree`, `root_git_unavailable`, -`root_git_unsupported`, `root_git_dirty`, `root_ignore_not_preserved`, and `root_busy`. Git refusing discovery, -including a `safe.directory` refusal, is `root_git_unavailable`; a discovered but unsupported Git shape such as a -submodule is `root_git_unsupported`. Root-output `materialization_failed.reason` values are `output_changed`, -`root_artifact_collision`, `root_ignore_changed`, `root_transaction_failed`, and `root_rollback_incomplete`. +`root_platform_unsupported`, `root_reserved_path`, `root_entry_unsupported`, `root_enclosing_worktree`, +`root_git_unavailable`, `root_git_unsupported`, `root_git_dirty`, `root_ignore_not_preserved`, and `root_busy`. Git +refusing discovery, including a `safe.directory` refusal, is `root_git_unavailable`; a discovered but unsupported +Git shape such as a submodule is `root_git_unsupported`. Root-output `materialization_failed.reason` values are +`output_changed`, `root_artifact_collision`, `root_ignore_changed`, `root_transaction_failed`, and +`root_rollback_incomplete`. From 8632165038ae4ac309caa7dbecc85758a35e8daa Mon Sep 17 00:00:00 2001 From: Raghu Betina Date: Sat, 29 Aug 2026 04:51:32 -0500 Subject: [PATCH 5/9] Adopt current directory for compiled output Let agent-driven workflows place verified Compilation bytes at the current root while moving existing design material aside without losing Git history or recovery information. Keep absent-directory output and GitHub Publication unchanged. --- docs/commands.md | 1 + scripts/check-pack.js | 1 + src/cli.js | 30 +- src/commands/compilation.js | 237 +++--- src/commands/plan-compile.js | 82 +- src/compilation-artifact.js | 126 ++- src/root-output.js | 1435 ++++++++++++++++++++++++++++++++++ test/compilation.test.js | 69 +- test/plan-compile.test.js | 104 ++- test/plan-publish.test.js | 9 +- test/root-output.test.js | 474 +++++++++++ 11 files changed, 2390 insertions(+), 178 deletions(-) create mode 100644 src/root-output.js create mode 100644 test/root-output.test.js diff --git a/docs/commands.md b/docs/commands.md index 4569f41..5a92731 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -248,6 +248,7 @@ without starting duplicate work. Follow the [direct Compilation recovery procedu Success writes one JSON object to stdout containing the validated Project, Compilation, and absolute output path. Root adoption additionally reports `root_adoption.design_path` (or `null` when no design directory was needed), its top-level moved-entry count, whether a Git repository was preserved, and whether its index was replaced. +After final verification succeeds, the CLI removes its owned `.firstdraft-root-output` transaction directory. An absent output directory contains exactly the artifact files and modes. Root adoption additionally contains the preserved `design` directory and an existing root `.git`, when present; every artifact-owned path remains exact. The CLI does not add a Git repository, run a formatter, or repair generated source. When an absent output is nested diff --git a/scripts/check-pack.js b/scripts/check-pack.js index 0f9e4b9..af73c72 100644 --- a/scripts/check-pack.js +++ b/scripts/check-pack.js @@ -52,6 +52,7 @@ if (result.status !== 0) { "src/file-system.js", "src/plan-compile-progress.js", "src/plan-state.js", + "src/root-output.js", "src/uuid-v7.js", "src/version.js", ]); diff --git a/src/cli.js b/src/cli.js index f315eeb..4333ab8 100644 --- a/src/cli.js +++ b/src/cli.js @@ -181,11 +181,11 @@ const PLAN_COMPILE_HELP = `First Draft CLI Usage: firstdraft plan compile - firstdraft plan compile --output + firstdraft plan compile --output Options: - --output Materialize the generated application here - -h, --help Show help + --output Materialize the generated application here + -h, --help Show help Environment: FIRSTDRAFT_API_TOKEN Authenticate API requests @@ -195,7 +195,8 @@ The command submits the exact current whole-file Plan, waits for its analysis, and proceeds only when that analysis is valid. Without --output, it conditionally creates or replays the internal GitHub Publication lifecycle and prints the private repository URL. With --output, it starts one direct Compilation and -atomically materializes the verified artifact into an absent directory without +atomically materializes the verified artifact into an absent directory, or +preserves existing root material under ./design when the output is ., without starting GitHub Publication. Progress is written to stderr. `; @@ -232,18 +233,19 @@ cancelled terminal states are successful status reads. const COMPILATION_DOWNLOAD_HELP = `First Draft CLI Usage: - firstdraft compilation download --output + firstdraft compilation download --output Options: - --output Materialize the generated application here - -h, --help Show help + --output Materialize the generated application here + -h, --help Show help Environment: FIRSTDRAFT_API_TOKEN Authenticate API requests The command reads the retained Compilation once, requires it to have succeeded, downloads and verifies its exact artifact once, and atomically -renames the verified files into an absent output path. It never starts work. +materializes it into an absent output path or adopts the current directory +while preserving existing material under ./design. It never starts work. `; const PLAN_INIT_HELP = `First Draft CLI @@ -386,7 +388,7 @@ const COMPILATION_ARTIFACT_UNAVAILABLE_DETAIL = const COMPILATION_ARTIFACT_INVALID_DETAIL = "The downloaded Compilation artifact did not satisfy the integrity contract. No files were materialized."; const COMPILATION_MATERIALIZATION_FAILED_DETAIL = - "The validated Compilation artifact could not be materialized at the requested absent output path."; + "The validated Compilation artifact could not be materialized at the requested output path."; const PLAN_COMPILE_DIRECT_ARTIFACT_UNAVAILABLE_DETAIL = "Could not download the retained Compilation artifact. Use current.compilation.id with 'firstdraft compilation download'; do not start another Compilation."; const PLAN_COMPILE_DIRECT_ARTIFACT_INVALID_DETAIL = @@ -394,7 +396,7 @@ const PLAN_COMPILE_DIRECT_ARTIFACT_INVALID_DETAIL = const PLAN_COMPILE_DIRECT_MATERIALIZATION_FAILED_DETAIL = "The retained Compilation artifact was validated but could not be materialized. Use current.compilation.id with 'firstdraft compilation download' after repairing the output path; do not start another Compilation."; const COMPILATION_INVALID_OUTPUT_PATH_DETAIL = - "The compilation output path must be absent beneath an existing real directory. No network request was made."; + "The compilation output must be an absent path beneath an existing real directory or the eligible current directory. No network request was made."; const GENERATE_UUID_INVALID_ARGUMENTS_DETAIL = "Invalid arguments. Run 'firstdraft generate uuid --help' for usage."; const GENERATE_APPLICATION_KEY_INVALID_ARGUMENTS_DETAIL = @@ -994,6 +996,7 @@ async function runCompilationDownload({ writeJson(stderr, { error: "invalid_output_path", detail: COMPILATION_INVALID_OUTPUT_PATH_DETAIL, + ...(error.reason ? { reason: error.reason } : {}), }); return 2; } @@ -1002,6 +1005,8 @@ async function runCompilationDownload({ writeJson(stderr, { error: "materialization_failed", detail: COMPILATION_MATERIALIZATION_FAILED_DETAIL, + ...(error.reason ? { reason: error.reason } : {}), + ...(error.recoveryPath ? { recovery_path: error.recoveryPath } : {}), }); return 1; } @@ -1928,6 +1933,7 @@ function writePlanCompileError(writer, error) { writeJson(writer, { error: "invalid_output_path", detail: COMPILATION_INVALID_OUTPUT_PATH_DETAIL, + ...(compilationError.reason ? { reason: compilationError.reason } : {}), }); return 2; } @@ -1936,6 +1942,10 @@ function writePlanCompileError(writer, error) { writeJson(writer, { error: "materialization_failed", detail: PLAN_COMPILE_DIRECT_MATERIALIZATION_FAILED_DETAIL, + ...(compilationError.reason ? { reason: compilationError.reason } : {}), + ...(compilationError.recoveryPath + ? { recovery_path: compilationError.recoveryPath } + : {}), ...(retainedCompilation ? { current: retainedCompilation } : {}), }); return 1; diff --git a/src/commands/compilation.js b/src/commands/compilation.js index c895bb9..b4ca580 100644 --- a/src/commands/compilation.js +++ b/src/commands/compilation.js @@ -10,7 +10,8 @@ import { MAX_ARTIFACT_BYTES, materializeCompilationArtifact, parseCompilationArtifact, - resolveOutputTarget, + prepareCompilationOutputTarget, + releaseCompilationOutputTarget, } from "../compilation-artifact.js"; import { FirstDraftNetworkError, @@ -312,43 +313,47 @@ export async function downloadCompilation({ createRequestSignal = (timeoutMs) => AbortSignal.timeout(timeoutMs), }) { const context = readContext({ cwd, compilationId, fileSystem }); - const outputTarget = resolveOutputTarget({ cwd, output }); - const current = await readCompilationStatus({ - ...context, - fetchFunction, - createRequestSignal, - requestTimeout: REQUEST_TIMEOUT_MS, - }); - if (current.compilation.status !== "succeeded") { - throw new CompilationNotSucceededError(current); - } + const outputTarget = prepareCompilationOutputTarget({ cwd, output }); + try { + const current = await readCompilationStatus({ + ...context, + fetchFunction, + createRequestSignal, + requestTimeout: REQUEST_TIMEOUT_MS, + }); + if (current.compilation.status !== "succeeded") { + throw new CompilationNotSucceededError(current); + } - const metadata = - /** @type {{path: string, sha256: string, media_type: string, byte_size: number}} */ ( - current.compilation.artifact - ); - const source = await downloadArtifact({ - apiUrl: context.apiUrl, - metadata, - fetchFunction, - createRequestSignal, - }); - const artifact = parseCompilationArtifact(source, { - projectId: context.projectId, - compilationId: current.compilation.id, - graphVersion: current.compilation.graph_version, - headSourceSha256: current.compilation.head_source_sha256, - analysisRunId: current.compilation.analysis_run_id, - compilerRelease: current.compilation.compiler_release, - target: current.compilation.target, - }); - const materialized = materializeCompilationArtifact(artifact, outputTarget); + const metadata = + /** @type {{path: string, sha256: string, media_type: string, byte_size: number}} */ ( + current.compilation.artifact + ); + const source = await downloadArtifact({ + apiUrl: context.apiUrl, + metadata, + fetchFunction, + createRequestSignal, + }); + const artifact = parseCompilationArtifact(source, { + projectId: context.projectId, + compilationId: current.compilation.id, + graphVersion: current.compilation.graph_version, + headSourceSha256: current.compilation.head_source_sha256, + analysisRunId: current.compilation.analysis_run_id, + compilerRelease: current.compilation.compiler_release, + target: current.compilation.target, + }); + const materialized = materializeCompilationArtifact(artifact, outputTarget); - return { - project: current.project, - compilation: current.compilation, - output: materialized, - }; + return { + project: current.project, + compilation: current.compilation, + output: materialized, + }; + } finally { + releaseCompilationOutputTarget(outputTarget); + } } /** @@ -367,6 +372,7 @@ export async function downloadCompilation({ * @property {string} expectedEtag * @property {CurrentCompilationIdentity} expected * @property {string} output + * @property {string | import("../root-output.js").RootOutputTarget} [outputTarget] * @property {typeof globalThis.fetch} [fetchFunction] * @property {import("../plan-state.js").PlanStateFileSystem} [fileSystem] * @property {(timeoutMs: number) => AbortSignal} [createRequestSignal] @@ -387,6 +393,7 @@ export async function compileAndDownload({ expectedEtag, expected, output, + outputTarget: suppliedOutputTarget, fetchFunction = globalThis.fetch, fileSystem = DEFAULT_FILE_SYSTEM, createRequestSignal = (timeoutMs) => AbortSignal.timeout(timeoutMs), @@ -394,89 +401,101 @@ export async function compileAndDownload({ now = Date.now, onProgress = () => {}, }) { - const outputTarget = resolveOutputTarget({ cwd, output }); - const context = readStartContext({ - cwd, - expectedEtag, - expected, - fileSystem, - }); - const deadline = now() + WAIT_TIMEOUT_MS; - - onProgress({ phase: "compilation", status: "waiting" }); - const initial = await startCompilation({ - ...context, - expected, - fetchFunction, - createRequestSignal, - }); - let current = initial; - + const ownsOutputTarget = suppliedOutputTarget === undefined; + const outputTarget = + suppliedOutputTarget ?? prepareCompilationOutputTarget({ cwd, output }); try { - while (!TERMINAL_STATUSES.has(current.compilation.status)) { - const remaining = deadline - now(); - if (remaining <= 0) throw new CompilationTimeoutError(current); + const context = readStartContext({ + cwd, + expectedEtag, + expected, + fileSystem, + }); + const deadline = now() + WAIT_TIMEOUT_MS; + + onProgress({ phase: "compilation", status: "waiting" }); + const initial = await startCompilation({ + ...context, + expected, + fetchFunction, + createRequestSignal, + }); + let current = initial; + + try { + while (!TERMINAL_STATUSES.has(current.compilation.status)) { + const remaining = deadline - now(); + if (remaining <= 0) throw new CompilationTimeoutError(current); + + await sleep(Math.min(POLL_INTERVAL_MS, remaining)); + if (now() >= deadline) throw new CompilationTimeoutError(current); + + const next = await readCompilationStatus({ + apiUrl: context.apiUrl, + projectId: context.projectId, + compilationId: initial.compilation.id, + fetchFunction, + createRequestSignal, + requestTimeout: Math.max( + 1, + Math.min(REQUEST_TIMEOUT_MS, deadline - now()), + ), + }); + if ( + !sameCompilation(initial, next) || + !validTransition(current, next) + ) { + throw new CompilationChangedError(next); + } + current = next; + } - await sleep(Math.min(POLL_INTERVAL_MS, remaining)); - if (now() >= deadline) throw new CompilationTimeoutError(current); + if (current.compilation.status === "failed") { + throw new CompilationFailedError(current); + } + if (current.compilation.status === "cancelled") { + throw new CompilationCancelledError(current); + } - const next = await readCompilationStatus({ + onProgress({ phase: "compilation", status: "succeeded" }); + const metadata = + /** @type {{path: string, sha256: string, media_type: string, byte_size: number}} */ ( + current.compilation.artifact + ); + const source = await downloadArtifact({ apiUrl: context.apiUrl, - projectId: context.projectId, - compilationId: initial.compilation.id, + metadata, fetchFunction, createRequestSignal, - requestTimeout: Math.max( - 1, - Math.min(REQUEST_TIMEOUT_MS, deadline - now()), - ), }); - if (!sameCompilation(initial, next) || !validTransition(current, next)) { - throw new CompilationChangedError(next); - } - current = next; - } - - if (current.compilation.status === "failed") { - throw new CompilationFailedError(current); - } - if (current.compilation.status === "cancelled") { - throw new CompilationCancelledError(current); - } - - onProgress({ phase: "compilation", status: "succeeded" }); - const metadata = - /** @type {{path: string, sha256: string, media_type: string, byte_size: number}} */ ( - current.compilation.artifact + const artifact = parseCompilationArtifact(source, { + projectId: context.projectId, + compilationId: current.compilation.id, + graphVersion: current.compilation.graph_version, + headSourceSha256: current.compilation.head_source_sha256, + analysisRunId: current.compilation.analysis_run_id, + compilerRelease: current.compilation.compiler_release, + target: current.compilation.target, + }); + const materialized = materializeCompilationArtifact( + artifact, + outputTarget, ); - const source = await downloadArtifact({ - apiUrl: context.apiUrl, - metadata, - fetchFunction, - createRequestSignal, - }); - const artifact = parseCompilationArtifact(source, { - projectId: context.projectId, - compilationId: current.compilation.id, - graphVersion: current.compilation.graph_version, - headSourceSha256: current.compilation.head_source_sha256, - analysisRunId: current.compilation.analysis_run_id, - compilerRelease: current.compilation.compiler_release, - target: current.compilation.target, - }); - const materialized = materializeCompilationArtifact(artifact, outputTarget); - return { - project: current.project, - compilation: current.compilation, - output: materialized, - }; - } catch (error) { - if (requiresRetainedRecovery(error)) { - throw new CompilationRetainedError(error, current); - } + return { + project: current.project, + compilation: current.compilation, + output: materialized, + }; + } catch (error) { + if (requiresRetainedRecovery(error)) { + throw new CompilationRetainedError(error, current); + } - throw error; + throw error; + } + } finally { + if (ownsOutputTarget) releaseCompilationOutputTarget(outputTarget); } } diff --git a/src/commands/plan-compile.js b/src/commands/plan-compile.js index da1b743..1ee4196 100644 --- a/src/commands/plan-compile.js +++ b/src/commands/plan-compile.js @@ -1,6 +1,9 @@ import { publishPlan } from "./plan-publish.js"; import { compileAndDownload } from "./compilation.js"; -import { resolveOutputTarget } from "../compilation-artifact.js"; +import { + prepareCompilationOutputTarget, + releaseCompilationOutputTarget, +} from "../compilation-artifact.js"; import { PlanPushNetworkError, PlanPushProtocolError, @@ -152,45 +155,46 @@ export async function compilePlanToDirectory({ readStatus = readPlanStatus, compile = compileAndDownload, }) { - // Reject an unavailable destination before the Plan push can mutate remote - // state. The Compilation boundary checks it again after Analysis in case the - // filesystem changes while this command is waiting. - resolveOutputTarget({ cwd, output }); - - const prepared = await preparePlan({ - cwd, - apiUrl, - fetchFunction, - fileSystem, - createTemporaryId, - createRequestSignal, - analysisSleep, - analysisNow, - onProgress, - push, - readStatus, - }); - const body = prepared.status.body; + const outputTarget = prepareCompilationOutputTarget({ cwd, output }); + try { + const prepared = await preparePlan({ + cwd, + apiUrl, + fetchFunction, + fileSystem, + createTemporaryId, + createRequestSignal, + analysisSleep, + analysisNow, + onProgress, + push, + readStatus, + }); + const body = prepared.status.body; - return compile({ - cwd, - expectedEtag: prepared.pushed.etag, - expected: { - projectId: body.project.id, - graphVersion: body.project.graph_version, - headSourceSha256: body.analysis.head_source_sha256, - analysisRunId: body.analysis.id, - compilerRelease: body.analysis.compiler_release, - target: body.analysis.target, - }, - output, - fetchFunction, - fileSystem, - createRequestSignal, - sleep: compilationSleep, - now: compilationNow, - onProgress, - }); + return await compile({ + cwd, + expectedEtag: prepared.pushed.etag, + expected: { + projectId: body.project.id, + graphVersion: body.project.graph_version, + headSourceSha256: body.analysis.head_source_sha256, + analysisRunId: body.analysis.id, + compilerRelease: body.analysis.compiler_release, + target: body.analysis.target, + }, + output, + outputTarget, + fetchFunction, + fileSystem, + createRequestSignal, + sleep: compilationSleep, + now: compilationNow, + onProgress, + }); + } finally { + releaseCompilationOutputTarget(outputTarget); + } } /** diff --git a/src/compilation-artifact.js b/src/compilation-artifact.js index 397f2a5..61d0ebd 100644 --- a/src/compilation-artifact.js +++ b/src/compilation-artifact.js @@ -14,6 +14,14 @@ import path from "node:path"; import { isFileSystemError } from "./file-system.js"; import { isUuidV7 } from "./plan-state.js"; +import { + materializeRootOutput, + prepareRootOutput, + releaseRootOutput, + resolveRootOutputPath, + RootOutputMaterializationError, + RootOutputPathError, +} from "./root-output.js"; export const ARTIFACT_MEDIA_TYPE = "application/vnd.firstdraft.compilation-artifact+json"; @@ -61,8 +69,21 @@ const DIRECTORY_MODE = 0o755; const POSIX_MODE_BITS_SUPPORTED = process.platform !== "win32"; export class CompilationArtifactInvalidError extends Error {} -export class CompilationMaterializationError extends Error {} -export class CompilationOutputPathError extends Error {} +export class CompilationMaterializationError extends Error { + /** @param {string} message @param {{cause?: unknown, reason?: string, recoveryPath?: string}} [options] */ + constructor(message, options = {}) { + super(message, options); + this.reason = options.reason; + this.recoveryPath = options.recoveryPath; + } +} +export class CompilationOutputPathError extends Error { + /** @param {string} message @param {{cause?: unknown, reason?: string}} [options] */ + constructor(message, options = {}) { + super(message, options); + this.reason = options.reason; + } +} /** * @typedef {object} ValidatedCompilationProvenance @@ -201,11 +222,80 @@ export function resolveOutputTarget({ cwd, output }) { return target; } +/** + * Validate an ordinary absent output or reserve root adoption before requests. + * + * @param {object} options + * @param {string} options.cwd + * @param {string} options.output + */ +export function prepareCompilationOutputTarget({ cwd, output }) { + if ( + typeof output !== "string" || + output.length === 0 || + output.includes("\0") + ) { + throw new CompilationOutputPathError( + "The compilation output path is invalid.", + ); + } + + const root = resolveRootOutputPath({ cwd, output }); + if (root === null) return resolveOutputTarget({ cwd, output }); + + try { + return prepareRootOutput({ root }); + } catch (error) { + if (!(error instanceof RootOutputPathError)) throw error; + throw new CompilationOutputPathError(error.message, { + cause: error, + reason: error.reason, + }); + } +} + +/** @param {string | import("./root-output.js").RootOutputTarget} target */ +export function releaseCompilationOutputTarget(target) { + if (typeof target === "string") return; + try { + releaseRootOutput(target); + } catch (error) { + if (!(error instanceof RootOutputMaterializationError)) throw error; + throw new CompilationMaterializationError(error.message, { + cause: error, + reason: error.reason, + recoveryPath: error.recoveryPath, + }); + } +} + /** * @param {ValidatedCompilationArtifact} artifact - * @param {string} target + * @param {string | import("./root-output.js").RootOutputTarget} target */ export function materializeCompilationArtifact(artifact, target) { + if (typeof target !== "string") { + try { + return materializeRootOutput(target, artifact, { + writeArtifact: (root) => { + writeArtifactTree(artifact.files, root); + applyMode(root, DIRECTORY_MODE); + }, + verifyArtifact: (root, ignoredRootEntries) => + verifyArtifactTree(artifact.files, root, ignoredRootEntries, { + verifyRootMode: ignoredRootEntries === undefined, + }), + }); + } catch (error) { + if (!(error instanceof RootOutputMaterializationError)) throw error; + throw new CompilationMaterializationError(error.message, { + cause: error, + reason: error.reason, + recoveryPath: error.recoveryPath, + }); + } + } + const parent = path.dirname(target); const prefix = path.join(parent, `.firstdraft-${path.basename(target)}-`); let temporaryDirectory = null; @@ -222,10 +312,8 @@ export function materializeCompilationArtifact(artifact, target) { renameSync(temporaryDirectory, target); temporaryDirectory = null; } catch (error) { - if ( - error instanceof CompilationMaterializationError || - isFileSystemError(error) - ) { + if (error instanceof CompilationMaterializationError) throw error; + if (isFileSystemError(error)) { throw new CompilationMaterializationError( "The compilation artifact could not be materialized.", { cause: error }, @@ -502,8 +590,18 @@ function writeArtifactTree(files, root) { } } -/** @param {ValidatedArtifactFile[]} files @param {string} root */ -function verifyArtifactTree(files, root) { +/** + * @param {ValidatedArtifactFile[]} files + * @param {string} root + * @param {Set} [ignoredRootEntries] + * @param {{verifyRootMode?: boolean}} [options] + */ +function verifyArtifactTree( + files, + root, + ignoredRootEntries = new Set(), + { verifyRootMode = true } = {}, +) { const expectedFiles = new Map(files.map((file) => [file.path, file])); const expectedDirectories = new Set(); for (const file of files) { @@ -519,13 +617,13 @@ function verifyArtifactTree(files, root) { if ( !rootStat.isDirectory() || rootStat.isSymbolicLink() || - !hasExpectedMode(rootStat.mode, DIRECTORY_MODE) + (verifyRootMode && !hasExpectedMode(rootStat.mode, DIRECTORY_MODE)) ) { throw new CompilationMaterializationError( "The materialized compilation root is invalid.", ); } - walkTree(root, "", actualFiles, actualDirectories); + walkTree(root, "", actualFiles, actualDirectories, ignoredRootEntries); if ( !setsEqual(actualFiles, new Set(expectedFiles.keys())) || !setsEqual(actualDirectories, expectedDirectories) @@ -581,16 +679,18 @@ function hasExpectedMode(actual, expected) { * @param {string} relative * @param {Set} files * @param {Set} directories + * @param {Set} ignoredRootEntries */ -function walkTree(root, relative, files, directories) { +function walkTree(root, relative, files, directories, ignoredRootEntries) { const directory = relative ? path.join(root, ...relative.split("/")) : root; const entries = readdirSync(directory, { withFileTypes: true }); for (const entry of entries) { + if (relative === "" && ignoredRootEntries.has(entry.name)) continue; const entryRelative = relative ? `${relative}/${entry.name}` : entry.name; if (entry.isDirectory()) { directories.add(entryRelative); - walkTree(root, entryRelative, files, directories); + walkTree(root, entryRelative, files, directories, ignoredRootEntries); } else if (entry.isFile()) { files.add(entryRelative); } else { diff --git a/src/root-output.js b/src/root-output.js new file mode 100644 index 0000000..765a47f --- /dev/null +++ b/src/root-output.js @@ -0,0 +1,1435 @@ +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { + accessSync, + chmodSync, + closeSync, + constants, + fsyncSync, + lstatSync, + mkdirSync, + openSync, + readFileSync, + readdirSync, + realpathSync, + renameSync, + rmdirSync, + rmSync, + unlinkSync, + writeFileSync, +} from "node:fs"; +import path from "node:path"; + +import { isFileSystemError } from "./file-system.js"; + +export const ROOT_TRANSACTION_NAME = ".firstdraft-root-output"; +export const ROOT_DESIGN_NAME = "design"; + +const JOURNAL_FORMAT = "firstdraft.root-output-transaction/1"; +const JOURNAL_NAME = "journal.json"; +const JOURNAL_TEMPORARY_NAME = "journal.tmp"; +const PREVIEW_NAME = "ignore-preview"; +const ARTIFACT_STAGE_NAME = "artifact"; +const PREPARED_INDEX_NAME = "prepared.index"; +const ORIGINAL_INDEX_NAME = "original.index"; +const DIRECTORY_MODE = 0o755; +const PRIVATE_DIRECTORY_MODE = 0o700; +const PRIVATE_FILE_MODE = 0o600; +const MAX_GIT_OUTPUT_BYTES = 128 * 1024 * 1024; +const GIT_HASH_PATTERN = /^[0-9a-f]{40,64}$/; +const RESERVED_NAMES = new Set([ + ROOT_DESIGN_NAME.toLowerCase(), + ROOT_TRANSACTION_NAME.toLowerCase(), +]); + +export class RootOutputPathError extends Error { + /** @param {string} message @param {string} reason @param {{cause?: unknown}} [options] */ + constructor(message, reason, options = {}) { + super(message, options); + this.reason = reason; + } +} + +export class RootOutputMaterializationError extends Error { + /** @param {string} message @param {string} reason @param {{cause?: unknown, recoveryPath?: string}} [options] */ + constructor(message, reason, options = {}) { + super(message, options); + this.reason = reason; + this.recoveryPath = options.recoveryPath; + } +} + +/** + * @typedef {object} RootEntryIdentity + * @property {string} name + * @property {"file" | "directory"} kind + * @property {string} device + * @property {string} inode + */ + +/** + * @typedef {object} GitIndexEntry + * @property {string} mode + * @property {string} object + * @property {string} path + */ + +/** + * @typedef {object} RootGitContext + * @property {string} gitDirectory + * @property {string} indexPath + * @property {string} indexLockPath + * @property {GitIndexEntry[]} indexEntries + * @property {string[]} ignoredPaths + */ + +/** + * @typedef {object} RootOutputTarget + * @property {"root"} kind + * @property {string} path + * @property {string} transactionPath + * @property {string} transactionDevice + * @property {string} transactionInode + * @property {RootEntryIdentity[]} snapshot + * @property {RootGitContext | null} git + * @property {Record} journal + * @property {boolean} irreversible + * @property {boolean} released + * @property {Map void>} signalHandlers + */ + +/** + * Return the physical current directory when output selects root adoption. + * Other existing or absent outputs return null for the ordinary resolver. + * + * @param {object} options + * @param {string} options.cwd + * @param {string} options.output + * @returns {string | null} + */ +export function resolveRootOutputPath({ cwd, output }) { + let physicalCwd; + try { + physicalCwd = realpathSync(cwd); + } catch (error) { + if (!isFileSystemError(error)) throw error; + return null; + } + + const candidate = path.resolve(cwd, output); + let physicalCandidate; + try { + physicalCandidate = realpathSync(candidate); + } catch (error) { + if (!isFileSystemError(error)) throw error; + if (error.code === "ENOENT") return null; + return null; + } + + return physicalCandidate === physicalCwd ? physicalCwd : null; +} + +/** + * Validate and exclusively reserve one root adoption before any request. + * + * @param {object} options + * @param {string} options.root + * @param {NodeJS.Platform} [options.platform] + * @returns {RootOutputTarget} + */ +export function prepareRootOutput({ root, platform = process.platform }) { + if (platform === "win32") { + throw new RootOutputPathError( + "Root output is not supported on this platform.", + "root_platform_unsupported", + ); + } + + try { + root = realpathSync(root); + } catch (error) { + if (!isFileSystemError(error)) throw error; + throw new RootOutputPathError( + "The root output directory could not be resolved.", + "root_not_real", + { cause: error }, + ); + } + + let rootStat; + try { + rootStat = lstatSync(root, { bigint: true }); + } catch (error) { + if (!isFileSystemError(error)) throw error; + throw new RootOutputPathError( + "The root output directory could not be read.", + "root_not_real", + { cause: error }, + ); + } + if ( + !rootStat.isDirectory() || + rootStat.isSymbolicLink() || + root === path.dirname(root) + ) { + throw new RootOutputPathError( + "The root output directory is not a real non-root directory.", + "root_not_real", + ); + } + + try { + accessSync(root, constants.R_OK | constants.W_OK | constants.X_OK); + } catch (error) { + if (!isFileSystemError(error)) throw error; + throw new RootOutputPathError( + "The root output directory is not writable.", + "root_not_writable", + { cause: error }, + ); + } + + assertReservedPathsAvailable(root); + const transactionPath = path.join(root, ROOT_TRANSACTION_NAME); + try { + mkdirSync(transactionPath, { mode: PRIVATE_DIRECTORY_MODE }); + chmodSync(transactionPath, PRIVATE_DIRECTORY_MODE); + } catch (error) { + if (!isFileSystemError(error)) throw error; + if (error.code === "EEXIST") throw existingTransactionError(root); + throw new RootOutputPathError( + "The root output lock could not be created.", + "root_not_writable", + { cause: error }, + ); + } + + const transactionStat = lstatSync(transactionPath, { bigint: true }); + /** @type {RootOutputTarget} */ + const target = { + kind: "root", + path: root, + transactionPath, + transactionDevice: String(transactionStat.dev), + transactionInode: String(transactionStat.ino), + snapshot: [], + git: null, + journal: {}, + irreversible: false, + released: false, + signalHandlers: new Map(), + }; + + try { + target.snapshot = captureRootSnapshot(root, rootStat.dev); + target.git = inspectGitRoot(root, transactionPath); + target.journal = initialJournal(target); + writeJournal(target); + if (target.git !== null) assertIgnoreProtection(target); + installSignalHandlers(target); + return target; + } catch (error) { + removePreparedTransaction(target); + throw error; + } +} + +/** @param {RootOutputTarget} target */ +export function releaseRootOutput(target) { + removeSignalHandlers(target); + if (target.released || target.irreversible) return; + removePreparedTransaction(target); +} + +/** + * @param {RootOutputTarget} target + * @param {object} artifact + * @param {Array<{path: string, mode: 420 | 493}>} artifact.files + * @param {string} artifact.manifest_sha256 + * @param {object} callbacks + * @param {(root: string) => void} callbacks.writeArtifact + * @param {(root: string, ignoredRootEntries?: Set) => void} callbacks.verifyArtifact + * @param {(from: string, to: string) => void} [callbacks.rename] + */ +export function materializeRootOutput( + target, + artifact, + { writeArtifact, verifyArtifact, rename = renameSync }, +) { + assertOwnedTransaction(target); + const artifactStage = path.join(target.transactionPath, ARTIFACT_STAGE_NAME); + + try { + assertArtifactRootNames(artifact.files); + mkdirSync(artifactStage, { mode: DIRECTORY_MODE }); + chmodSync(artifactStage, DIRECTORY_MODE); + writeArtifact(artifactStage); + verifyArtifact(artifactStage); + recheckRootSnapshot(target); + + if (target.git !== null) prepareGitIndex(target, artifact.files); + performRootTransaction(target, artifact.files, rename); + verifyRootResult(target, artifact.files, verifyArtifact); + + target.journal.phase = "complete"; + target.journal.pending = null; + writeJournal(target); + removeSignalHandlers(target); + removeCompletedTransaction(target); + + const movedEntries = target.snapshot.filter( + (entry) => entry.name !== ".git", + ); + return { + path: target.path, + file_count: artifact.files.length, + manifest_sha256: artifact.manifest_sha256, + root_adoption: { + design_path: + movedEntries.length === 0 + ? null + : path.join(target.path, ROOT_DESIGN_NAME), + moved_entry_count: movedEntries.length, + git_repository_preserved: target.git !== null, + git_index_replaced: target.git !== null, + }, + }; + } catch (error) { + const materializationError = normalizeMaterializationError(error); + const rollbackComplete = rollbackRootTransaction(target, rename); + if (!rollbackComplete) { + target.irreversible = true; + target.journal.phase = "rollback_incomplete"; + writeJournalBestEffort(target); + removeSignalHandlers(target); + throw new RootOutputMaterializationError( + "The root output transaction could not be rolled back.", + "root_rollback_incomplete", + { + cause: materializationError, + recoveryPath: ROOT_TRANSACTION_NAME, + }, + ); + } + + removePreparedTransaction(target); + throw materializationError; + } +} + +/** @param {string} root */ +function assertReservedPathsAvailable(root) { + const names = readdirSync(root); + for (const name of names) { + const folded = name.toLowerCase(); + if (folded === ROOT_DESIGN_NAME.toLowerCase()) { + throw new RootOutputPathError( + "The root output design path is reserved.", + "root_reserved_path", + ); + } + if (folded !== ROOT_TRANSACTION_NAME.toLowerCase()) continue; + if (name === ROOT_TRANSACTION_NAME) throw existingTransactionError(root); + throw new RootOutputPathError( + "The root output transaction path is reserved.", + "root_reserved_path", + ); + } +} + +/** @param {string} root */ +function existingTransactionError(root) { + const journalPath = path.join(root, ROOT_TRANSACTION_NAME, JOURNAL_NAME); + try { + const journal = JSON.parse(readFileSync(journalPath, "utf8")); + if (journal?.format === JOURNAL_FORMAT && journal?.root === root) { + return new RootOutputPathError( + "Another root output transaction is active or retained.", + "root_busy", + ); + } + } catch (error) { + if (!(error instanceof SyntaxError || isFileSystemError(error))) + throw error; + } + + return new RootOutputPathError( + "The root output transaction path is reserved.", + "root_reserved_path", + ); +} + +/** @param {string} root @param {bigint} rootDevice */ +function captureRootSnapshot(root, rootDevice) { + /** @type {RootEntryIdentity[]} */ + const snapshot = []; + const names = readdirSync(root).sort(compareStrings); + for (const name of names) { + if (name === ROOT_TRANSACTION_NAME) continue; + if (RESERVED_NAMES.has(name.toLowerCase())) { + throw new RootOutputPathError( + "A reserved root output path appeared during preflight.", + "root_reserved_path", + ); + } + const entryPath = path.join(root, name); + const stat = lstatSync(entryPath, { bigint: true }); + const kind = stat.isFile() + ? "file" + : stat.isDirectory() + ? "directory" + : null; + if (kind === null || stat.isSymbolicLink() || stat.dev !== rootDevice) { + throw new RootOutputPathError( + "A root output entry is unsupported.", + "root_entry_unsupported", + ); + } + snapshot.push({ + name, + kind, + device: String(stat.dev), + inode: String(stat.ino), + }); + } + return snapshot; +} + +/** @param {RootOutputTarget} target */ +function recheckRootSnapshot(target) { + const rootStat = lstatSync(target.path, { bigint: true }); + const current = captureRootSnapshot(target.path, rootStat.dev); + if (JSON.stringify(current) !== JSON.stringify(target.snapshot)) { + throw new RootOutputMaterializationError( + "The root output changed while Compilation was running.", + "output_changed", + ); + } +} + +/** @param {RootOutputTarget} target */ +function initialJournal(target) { + return { + format: JOURNAL_FORMAT, + root: target.path, + phase: "prepared", + irreversible: false, + snapshot: target.snapshot, + pending: null, + design_created: false, + design_moves: [], + artifact_moves: [], + index: null, + }; +} + +/** @param {RootOutputTarget} target */ +function writeJournal(target) { + assertOwnedTransaction(target); + const temporary = path.join(target.transactionPath, JOURNAL_TEMPORARY_NAME); + const destination = path.join(target.transactionPath, JOURNAL_NAME); + const source = Buffer.from(`${JSON.stringify(target.journal, null, 2)}\n`); + let descriptor = null; + try { + descriptor = openSync(temporary, "wx", PRIVATE_FILE_MODE); + writeFileSync(descriptor, source); + fsyncSync(descriptor); + closeSync(descriptor); + descriptor = null; + renameSync(temporary, destination); + fsyncDirectory(target.transactionPath); + } finally { + if (descriptor !== null) closeSync(descriptor); + removePathIfPresent(temporary); + } +} + +/** @param {RootOutputTarget} target */ +function writeJournalBestEffort(target) { + try { + writeJournal(target); + } catch (error) { + if (!isFileSystemError(error)) throw error; + } +} + +/** @param {string} directory */ +function fsyncDirectory(directory) { + let descriptor = null; + try { + descriptor = openSync(directory, constants.O_RDONLY); + fsyncSync(descriptor); + } finally { + if (descriptor !== null) closeSync(descriptor); + } +} + +/** @param {RootOutputTarget} target */ +function installSignalHandlers(target) { + for (const signal of /** @type {NodeJS.Signals[]} */ ([ + "SIGINT", + "SIGTERM", + ])) { + const handler = () => { + try { + if (!target.irreversible) releaseRootOutput(target); + } finally { + process.removeListener(signal, handler); + process.kill(process.pid, signal); + } + }; + target.signalHandlers.set(signal, handler); + process.once(signal, handler); + } +} + +/** @param {RootOutputTarget} target */ +function removeSignalHandlers(target) { + for (const [signal, handler] of target.signalHandlers) { + process.removeListener(signal, handler); + } + target.signalHandlers.clear(); +} + +/** @param {RootOutputTarget} target */ +function assertOwnedTransaction(target) { + const stat = lstatSync(target.transactionPath, { bigint: true }); + if ( + !stat.isDirectory() || + stat.isSymbolicLink() || + String(stat.dev) !== target.transactionDevice || + String(stat.ino) !== target.transactionInode + ) { + throw new RootOutputMaterializationError( + "The owned root output transaction changed.", + "root_transaction_failed", + ); + } +} + +/** @param {RootOutputTarget} target */ +function removePreparedTransaction(target) { + removeSignalHandlers(target); + if (target.released) return; + assertOwnedTransaction(target); + rmSync(target.transactionPath, { recursive: true, force: true }); + target.released = true; +} + +/** @param {RootOutputTarget} target */ +function removeCompletedTransaction(target) { + assertOwnedTransaction(target); + rmSync(target.transactionPath, { recursive: true, force: true }); + target.released = true; +} + +/** @param {Array<{path: string}>} files */ +function assertArtifactRootNames(files) { + for (const file of files) { + const top = file.path.split("/", 1)[0] ?? ""; + if (RESERVED_NAMES.has(top.toLowerCase())) { + throw new RootOutputMaterializationError( + "The compilation artifact owns a reserved root path.", + "root_artifact_collision", + ); + } + } +} + +/** + * @param {RootOutputTarget} target + * @param {Array<{path: string, mode: 420 | 493}>} files + * @param {(from: string, to: string) => void} rename + */ +function performRootTransaction(target, files, rename) { + const designEntries = target.snapshot.filter( + (entry) => entry.name !== ".git", + ); + const designPath = path.join(target.path, ROOT_DESIGN_NAME); + const artifactStage = path.join(target.transactionPath, ARTIFACT_STAGE_NAME); + const artifactTopNames = [ + ...new Set(files.map((file) => file.path.split("/", 1)[0] ?? "")), + ].sort(compareStrings); + + if (designEntries.length > 0) { + startIrreversibleStep(target, { + kind: "create_design", + path: designPath, + }); + mkdirSync(designPath, { mode: DIRECTORY_MODE }); + target.journal.design_created = true; + chmodSync(designPath, DIRECTORY_MODE); + finishIrreversibleStep(target, "moving_design"); + + for (const entry of designEntries) { + const source = path.join(target.path, entry.name); + const destination = path.join(designPath, entry.name); + startIrreversibleStep(target, { + kind: "move_design_entry", + source, + destination, + }); + rename(source, destination); + /** @type {Array>} */ ( + target.journal.design_moves + ).push({ source, destination }); + finishIrreversibleStep(target, "moving_design"); + } + } + + for (const name of artifactTopNames) { + const source = path.join(artifactStage, name); + const destination = path.join(target.path, name); + if (!isAbsent(destination)) { + throw new RootOutputMaterializationError( + "An artifact root path appeared during materialization.", + "root_transaction_failed", + ); + } + startIrreversibleStep(target, { + kind: "install_artifact_entry", + source, + destination, + }); + rename(source, destination); + /** @type {Array>} */ ( + target.journal.artifact_moves + ).push({ source, destination }); + finishIrreversibleStep(target, "installing_artifact"); + } + + if (target.git !== null) { + refreshPreparedIndex(target); + installPreparedIndex(target); + } +} + +/** @param {RootOutputTarget} target @param {Record} pending */ +function startIrreversibleStep(target, pending) { + target.irreversible = true; + target.journal.irreversible = true; + target.journal.pending = pending; + writeJournal(target); +} + +/** @param {RootOutputTarget} target @param {string} phase */ +function finishIrreversibleStep(target, phase) { + target.journal.phase = phase; + target.journal.pending = null; + writeJournal(target); +} + +/** + * @param {RootOutputTarget} target + * @param {Array<{path: string}>} files + * @param {(root: string, ignoredRootEntries?: Set) => void} verifyArtifact + */ +function verifyRootResult(target, files, verifyArtifact) { + const ignored = new Set([ROOT_TRANSACTION_NAME]); + if (target.snapshot.some((entry) => entry.name === ".git")) { + ignored.add(".git"); + } + if (target.snapshot.some((entry) => entry.name !== ".git")) { + ignored.add(ROOT_DESIGN_NAME); + } + verifyArtifact(target.path, ignored); + + const designPath = path.join(target.path, ROOT_DESIGN_NAME); + for (const identity of target.snapshot) { + const currentPath = + identity.name === ".git" + ? path.join(target.path, identity.name) + : path.join(designPath, identity.name); + if (!sameIdentity(currentPath, identity)) { + throw new RootOutputMaterializationError( + "A preserved root entry changed during materialization.", + "root_transaction_failed", + ); + } + } + + if (target.git !== null) { + assertInstalledIndex(target); + assertMovedIgnoreProtection(target); + const worktree = invokeGit(target.path, [ + "diff-files", + "--quiet", + "--ignore-submodules=none", + "--", + ]); + if (worktree.status !== 0) { + const detail = invokeGit(target.path, [ + "diff-files", + "--name-status", + "--", + ]); + throw new RootOutputMaterializationError( + "The prepared Git index does not match the adopted worktree.", + "root_transaction_failed", + { + cause: new Error(detail.stdout.toString("utf8")), + }, + ); + } + } + + const expectedTopNames = new Set( + files.map((file) => file.path.split("/", 1)[0] ?? ""), + ); + for (const name of ignored) expectedTopNames.add(name); + const actualTopNames = new Set(readdirSync(target.path)); + if (!setsEqual(expectedTopNames, actualTopNames)) { + throw new RootOutputMaterializationError( + "The adopted root contains an unexpected entry.", + "root_transaction_failed", + ); + } +} + +/** @param {RootOutputTarget} target @param {(from: string, to: string) => void} rename */ +function rollbackRootTransaction(target, rename) { + let complete = true; + + if (!rollbackIndex(target)) complete = false; + + const artifactMoves = + /** @type {Array<{source: string, destination: string}>} */ ( + target.journal.artifact_moves ?? [] + ); + for (const move of [...artifactMoves].reverse()) { + try { + if (!isAbsent(move.source) || isAbsent(move.destination)) { + complete = false; + continue; + } + rename(move.destination, move.source); + } catch (error) { + if (!isFileSystemError(error)) throw error; + complete = false; + } + } + + const designMoves = + /** @type {Array<{source: string, destination: string}>} */ ( + target.journal.design_moves ?? [] + ); + for (const move of [...designMoves].reverse()) { + try { + if (!isAbsent(move.source) || isAbsent(move.destination)) { + complete = false; + continue; + } + rename(move.destination, move.source); + } catch (error) { + if (!isFileSystemError(error)) throw error; + complete = false; + } + } + + if (target.journal.design_created) { + try { + const designPath = path.join(target.path, ROOT_DESIGN_NAME); + if (readdirSync(designPath).length === 0) { + rmdirSync(designPath); + } else { + complete = false; + } + } catch (error) { + if (!isFileSystemError(error) || error.code !== "ENOENT") { + if (!isFileSystemError(error)) throw error; + complete = false; + } + } + } + + if (complete) { + target.irreversible = false; + target.journal.irreversible = false; + target.journal.phase = "rolled_back"; + target.journal.pending = null; + writeJournalBestEffort(target); + } + return complete; +} + +/** @param {unknown} error */ +function normalizeMaterializationError(error) { + if (error instanceof RootOutputMaterializationError) return error; + if (error instanceof RootOutputPathError) { + return new RootOutputMaterializationError( + "The root output preconditions changed during materialization.", + "root_transaction_failed", + { cause: error }, + ); + } + if (isFileSystemError(error)) { + return new RootOutputMaterializationError( + "The root output transaction failed.", + "root_transaction_failed", + { cause: error }, + ); + } + throw error; +} + +/** @param {string} root @param {string} transactionPath */ +function inspectGitRoot(root, transactionPath) { + const topLevel = invokeGit(root, ["rev-parse", "--show-toplevel"]); + if (topLevel.status !== 0) { + const hasGitEntry = pathExists(path.join(root, ".git")); + const detail = topLevel.stderr.toString("utf8"); + if (!hasGitEntry && detail.includes("not a git repository")) return null; + throw new RootOutputPathError( + "The Git worktree could not be discovered.", + "root_git_unavailable", + ); + } + + const discovered = topLevel.stdout.toString("utf8").trim(); + let physicalTopLevel; + try { + physicalTopLevel = realpathSync(discovered); + } catch (error) { + if (!isFileSystemError(error)) throw error; + throw new RootOutputPathError( + "The Git worktree root could not be read.", + "root_git_unavailable", + { cause: error }, + ); + } + if (physicalTopLevel !== root) { + throw new RootOutputPathError( + "Root output cannot adopt a directory inside another Git worktree.", + "root_enclosing_worktree", + ); + } + + for (const operation of [ + "MERGE_HEAD", + "CHERRY_PICK_HEAD", + "REVERT_HEAD", + "rebase-apply", + "rebase-merge", + ]) { + const operationPath = gitPath(root, operation); + if (pathExists(operationPath)) { + throw new RootOutputPathError( + "The Git worktree has an in-progress operation.", + "root_git_dirty", + ); + } + } + + if (gitBoolean(root, ["config", "--bool", "core.sparseCheckout"])) { + throw new RootOutputPathError( + "Sparse Git worktrees are not supported for root output.", + "root_git_unsupported", + ); + } + + for (const arguments_ of [ + ["diff", "--quiet", "--ignore-submodules=none", "--"], + ["diff", "--cached", "--quiet", "--ignore-submodules=none", "--"], + ]) { + const result = invokeGit(root, arguments_); + if (result.status === 1) { + throw new RootOutputPathError( + "The Git worktree or index is not clean.", + "root_git_dirty", + ); + } + if (result.status !== 0) { + throw new RootOutputPathError( + "The Git worktree could not be inspected.", + "root_git_unavailable", + ); + } + } + + const entriesResult = requiredGitPath(root, ["ls-files", "--stage", "-z"]); + const indexEntries = parseIndexEntries(entriesResult.stdout); + if ( + indexEntries.some( + (entry) => + entry.mode === "160000" || + entry.path.split("/").includes(".gitmodules"), + ) + ) { + throw new RootOutputPathError( + "Git submodules are not supported for root output.", + "root_git_unsupported", + ); + } + + const unmerged = requiredGitPath(root, ["ls-files", "--unmerged", "-z"]); + if (unmerged.stdout.length > 0) { + throw new RootOutputPathError( + "The Git index contains unmerged entries.", + "root_git_dirty", + ); + } + + const status = requiredGitPath(root, [ + "status", + "--porcelain=v1", + "-z", + "--untracked-files=all", + "--ignored=matching", + ]); + const ignoredPaths = parseIgnoredPaths(status.stdout).filter( + (candidate) => + candidate !== ROOT_TRANSACTION_NAME && + !candidate.startsWith(`${ROOT_TRANSACTION_NAME}/`), + ); + + const gitDirectory = requiredGitPath(root, [ + "rev-parse", + "--absolute-git-dir", + ]) + .stdout.toString("utf8") + .trim(); + const indexPath = gitPath(root, "index"); + const indexLockPath = gitPath(root, "index.lock"); + if (pathExists(indexLockPath)) { + throw new RootOutputPathError( + "The Git index is currently locked.", + "root_busy", + ); + } + + return { + gitDirectory, + indexPath, + indexLockPath, + indexEntries, + ignoredPaths, + transactionPath, + }; +} + +/** @param {string} root @param {string[]} arguments_ */ +function gitBoolean(root, arguments_) { + const result = invokeGit(root, arguments_); + if (result.status === 1) return false; + if (result.status !== 0) { + throw new RootOutputPathError( + "The Git configuration could not be inspected.", + "root_git_unavailable", + ); + } + return result.stdout.toString("utf8").trim() === "true"; +} + +/** @param {string} root @param {string} name */ +function gitPath(root, name) { + return requiredGitPath(root, [ + "rev-parse", + "--path-format=absolute", + "--git-path", + name, + ]) + .stdout.toString("utf8") + .trim(); +} + +/** @param {string} root @param {string[]} arguments_ */ +function requiredGitPath(root, arguments_) { + const result = invokeGit(root, arguments_); + if (result.status !== 0) { + throw new RootOutputPathError( + "The required Git inspection failed.", + "root_git_unavailable", + ); + } + return result; +} + +/** @param {Buffer} source */ +function parseIndexEntries(source) { + /** @type {GitIndexEntry[]} */ + const entries = []; + for (const record of nullRecords(source)) { + const tab = record.indexOf("\t"); + const header = tab === -1 ? "" : record.slice(0, tab); + const filePath = tab === -1 ? "" : record.slice(tab + 1); + const match = /^(\d+) ([0-9a-f]+) (\d)$/.exec(header); + if (match === null || match[3] !== "0" || filePath.length === 0) { + throw new RootOutputPathError( + "The Git index shape is unsupported.", + "root_git_unsupported", + ); + } + entries.push({ + mode: match[1] ?? "", + object: match[2] ?? "", + path: filePath, + }); + } + return entries; +} + +/** @param {Buffer} source */ +function parseIgnoredPaths(source) { + const ignored = []; + for (const record of nullRecords(source)) { + if (!record.startsWith("!! ")) continue; + const candidate = normalizeGitPath(record.slice(3)); + if (candidate !== "") ignored.push(candidate); + } + return [...new Set(ignored)].sort(compareStrings); +} + +/** @param {RootOutputTarget} target */ +function assertIgnoreProtection(target) { + if (target.git === null || target.git.ignoredPaths.length === 0) return; + const previewRoot = path.join(target.transactionPath, PREVIEW_NAME); + const previewWorktree = path.join(previewRoot, "worktree"); + mkdirSync(previewWorktree, { recursive: true, mode: PRIVATE_DIRECTORY_MODE }); + + try { + for (const ignoredPath of target.git.ignoredPaths) { + copyApplicableIgnoreFiles(target.path, previewWorktree, ignoredPath); + createPreviewEntry(target.path, previewWorktree, ignoredPath); + } + + const expected = target.git.ignoredPaths.map( + (candidate) => `${ROOT_DESIGN_NAME}/${candidate}`, + ); + const result = invokeGit( + target.path, + ["check-ignore", "--no-index", "-z", "--stdin"], + { + input: Buffer.from(`${expected.join("\0")}\0`), + gitDirectory: target.git.gitDirectory, + workTree: previewWorktree, + }, + ); + if (result.status !== 0 && result.status !== 1) { + throw new RootOutputPathError( + "Git ignore protection could not be previewed.", + "root_git_unavailable", + ); + } + const actual = new Set(nullRecords(result.stdout).map(normalizeGitPath)); + if (!expected.every((candidate) => actual.has(candidate))) { + throw new RootOutputPathError( + "A currently ignored path would become visible after root adoption.", + "root_ignore_not_preserved", + ); + } + } finally { + rmSync(previewRoot, { recursive: true, force: true }); + } +} + +/** @param {string} root @param {string} previewWorktree @param {string} ignoredPath */ +function copyApplicableIgnoreFiles(root, previewWorktree, ignoredPath) { + const components = ignoredPath.split("/"); + for (let depth = 0; depth < components.length; depth += 1) { + const ancestor = components.slice(0, depth); + const source = path.join(root, ...ancestor, ".gitignore"); + let stat; + try { + stat = lstatSync(source); + } catch (error) { + if (!isFileSystemError(error)) throw error; + if (error.code === "ENOENT") continue; + throw error; + } + if (!stat.isFile() || stat.isSymbolicLink()) continue; + + const destination = path.join( + previewWorktree, + ROOT_DESIGN_NAME, + ...ancestor, + ".gitignore", + ); + mkdirSync(path.dirname(destination), { + recursive: true, + mode: DIRECTORY_MODE, + }); + if (!pathExists(destination)) + writeFileSync(destination, readFileSync(source)); + } +} + +/** @param {string} root @param {string} previewWorktree @param {string} ignoredPath */ +function createPreviewEntry(root, previewWorktree, ignoredPath) { + const source = path.join(root, ...ignoredPath.split("/")); + const destination = path.join( + previewWorktree, + ROOT_DESIGN_NAME, + ...ignoredPath.split("/"), + ); + const stat = lstatSync(source); + if (stat.isDirectory()) { + mkdirSync(destination, { recursive: true, mode: DIRECTORY_MODE }); + } else { + mkdirSync(path.dirname(destination), { + recursive: true, + mode: DIRECTORY_MODE, + }); + if (!pathExists(destination)) writeFileSync(destination, Buffer.alloc(0)); + } +} + +/** @param {RootOutputTarget} target @param {Array<{path: string, mode: 420 | 493}>} files */ +function prepareGitIndex(target, files) { + const git = /** @type {RootGitContext} */ (target.git); + const preparedPath = path.join(target.transactionPath, PREPARED_INDEX_NAME); + const originalPath = path.join(target.transactionPath, ORIGINAL_INDEX_NAME); + + requiredGit(target.path, ["read-tree", "--empty"], { + indexFile: preparedPath, + }); + + const indexRecords = []; + for (const entry of git.indexEntries) { + indexRecords.push( + Buffer.from( + `${entry.mode} ${entry.object} 0\t${ROOT_DESIGN_NAME}/${entry.path}\0`, + ), + ); + } + + const artifactStage = path.join(target.transactionPath, ARTIFACT_STAGE_NAME); + for (const file of files) { + const source = path.join(artifactStage, ...file.path.split("/")); + const hashed = requiredGit(target.path, [ + "hash-object", + "-w", + "--no-filters", + source, + ]) + .stdout.toString("utf8") + .trim(); + if (!GIT_HASH_PATTERN.test(hashed)) { + throw new RootOutputMaterializationError( + "Git did not return a valid artifact object identity.", + "root_transaction_failed", + ); + } + const mode = file.mode === 0o755 ? "100755" : "100644"; + indexRecords.push(Buffer.from(`${mode} ${hashed} 0\t${file.path}\0`)); + } + + requiredGit(target.path, ["update-index", "-z", "--index-info"], { + indexFile: preparedPath, + input: Buffer.concat(indexRecords), + }); + + const preparedSource = readFileSync(preparedPath); + const originalExists = pathExists(git.indexPath); + const originalSource = originalExists ? readFileSync(git.indexPath) : null; + const originalMode = originalExists + ? lstatSync(git.indexPath).mode & 0o777 + : 0o644; + if (originalSource !== null) { + writeFileSync(originalPath, originalSource, { + flag: "wx", + mode: PRIVATE_FILE_MODE, + }); + chmodSync(originalPath, PRIVATE_FILE_MODE); + } + + target.journal.index = { + path: git.indexPath, + lock_path: git.indexLockPath, + original_exists: originalExists, + original_mode: originalMode, + original_sha256: originalSource === null ? null : sha256(originalSource), + prepared_sha256: sha256(preparedSource), + installed: false, + }; + writeJournal(target); +} + +/** @param {RootOutputTarget} target */ +function installPreparedIndex(target) { + const git = /** @type {RootGitContext} */ (target.git); + const preparedSource = readFileSync( + path.join(target.transactionPath, PREPARED_INDEX_NAME), + ); + const indexJournal = /** @type {Record} */ ( + target.journal.index + ); + startIrreversibleStep(target, { + kind: "install_git_index", + source: path.join(target.transactionPath, PREPARED_INDEX_NAME), + destination: git.indexPath, + }); + writeLockedIndex( + git.indexPath, + git.indexLockPath, + preparedSource, + Number(indexJournal.original_mode ?? 0o644), + ); + indexJournal.installed = true; + finishIrreversibleStep(target, "index_installed"); +} + +/** @param {RootOutputTarget} target */ +function refreshPreparedIndex(target) { + const preparedPath = path.join(target.transactionPath, PREPARED_INDEX_NAME); + requiredGit(target.path, ["update-index", "--refresh"], { + indexFile: preparedPath, + }); + const indexJournal = /** @type {Record} */ ( + target.journal.index + ); + indexJournal.prepared_sha256 = sha256(readFileSync(preparedPath)); + writeJournal(target); +} + +/** @param {string} indexPath @param {string} lockPath @param {Buffer} source @param {number} mode */ +function writeLockedIndex(indexPath, lockPath, source, mode) { + let descriptor = null; + let ownsLock = false; + try { + descriptor = openSync(lockPath, "wx", mode); + ownsLock = true; + writeFileSync(descriptor, source); + fsyncSync(descriptor); + closeSync(descriptor); + descriptor = null; + renameSync(lockPath, indexPath); + ownsLock = false; + chmodSync(indexPath, mode); + fsyncDirectory(path.dirname(indexPath)); + } finally { + if (descriptor !== null) closeSync(descriptor); + if (ownsLock) removePathIfPresent(lockPath); + } +} + +/** @param {RootOutputTarget} target */ +function assertInstalledIndex(target) { + const git = /** @type {RootGitContext} */ (target.git); + const indexJournal = /** @type {Record} */ ( + target.journal.index + ); + if ( + indexJournal.installed !== true || + !pathExists(git.indexPath) || + sha256(readFileSync(git.indexPath)) !== indexJournal.prepared_sha256 + ) { + throw new RootOutputMaterializationError( + "The prepared Git index was not installed exactly.", + "root_transaction_failed", + ); + } +} + +/** @param {RootOutputTarget} target */ +function rollbackIndex(target) { + if (target.git === null || target.journal.index === null) return true; + const git = target.git; + const indexJournal = /** @type {Record} */ ( + target.journal.index + ); + try { + const indexExists = pathExists(git.indexPath); + const currentDigest = indexExists + ? sha256(readFileSync(git.indexPath)) + : null; + const originalStatePresent = + indexJournal.original_exists === true + ? currentDigest === indexJournal.original_sha256 + : !indexExists; + if (originalStatePresent) { + indexJournal.installed = false; + return true; + } + if (currentDigest !== indexJournal.prepared_sha256) return false; + + if (indexJournal.original_exists === true) { + const source = readFileSync( + path.join(target.transactionPath, ORIGINAL_INDEX_NAME), + ); + if (sha256(source) !== indexJournal.original_sha256) return false; + writeLockedIndex( + git.indexPath, + git.indexLockPath, + source, + Number(indexJournal.original_mode), + ); + } else { + const descriptor = openSync(git.indexLockPath, "wx", PRIVATE_FILE_MODE); + closeSync(descriptor); + unlinkSync(git.indexPath); + unlinkSync(git.indexLockPath); + fsyncDirectory(path.dirname(git.indexPath)); + } + indexJournal.installed = false; + return true; + } catch (error) { + if (!isFileSystemError(error)) throw error; + return false; + } +} + +/** @param {RootOutputTarget} target */ +function assertMovedIgnoreProtection(target) { + if (target.git === null || target.git.ignoredPaths.length === 0) return; + const expected = target.git.ignoredPaths.map( + (candidate) => `${ROOT_DESIGN_NAME}/${candidate}`, + ); + const result = invokeGit( + target.path, + ["check-ignore", "--no-index", "-z", "--stdin"], + { input: Buffer.from(`${expected.join("\0")}\0`) }, + ); + if (result.status !== 0 && result.status !== 1) { + throw new RootOutputMaterializationError( + "Git ignore protection could not be rechecked.", + "root_transaction_failed", + ); + } + const actual = new Set(nullRecords(result.stdout).map(normalizeGitPath)); + if (!expected.every((candidate) => actual.has(candidate))) { + throw new RootOutputMaterializationError( + "A previously ignored path is no longer ignored.", + "root_ignore_changed", + ); + } +} + +/** + * @param {string} root + * @param {string[]} arguments_ + * @param {{input?: Buffer, indexFile?: string, gitDirectory?: string, workTree?: string}} [options] + */ +function requiredGit(root, arguments_, options = {}) { + const result = invokeGit(root, arguments_, options); + if (result.status !== 0) { + throw new RootOutputMaterializationError( + "The required Git operation failed.", + "root_transaction_failed", + ); + } + return result; +} + +/** + * @param {string} root + * @param {string[]} arguments_ + * @param {{input?: Buffer, indexFile?: string, gitDirectory?: string, workTree?: string}} [options] + */ +function invokeGit(root, arguments_, options = {}) { + /** @type {NodeJS.ProcessEnv} */ + const environment = { ...process.env, GIT_OPTIONAL_LOCKS: "0" }; + delete environment.GIT_DIR; + delete environment.GIT_WORK_TREE; + delete environment.GIT_INDEX_FILE; + if (options.gitDirectory !== undefined) { + environment.GIT_DIR = options.gitDirectory; + } + if (options.workTree !== undefined) + environment.GIT_WORK_TREE = options.workTree; + if (options.indexFile !== undefined) + environment.GIT_INDEX_FILE = options.indexFile; + + const result = spawnSync( + "git", + ["--no-optional-locks", "-C", root, ...arguments_], + { + env: environment, + input: options.input, + maxBuffer: MAX_GIT_OUTPUT_BYTES, + }, + ); + if (result.error !== undefined || result.status === null) { + throw new RootOutputPathError( + "The Git executable is unavailable.", + "root_git_unavailable", + { cause: result.error }, + ); + } + return { + status: result.status, + stdout: result.stdout ?? Buffer.alloc(0), + stderr: result.stderr ?? Buffer.alloc(0), + }; +} + +/** @param {Buffer} source */ +function nullRecords(source) { + const records = source.toString("utf8").split("\0"); + if (records.at(-1) === "") records.pop(); + return records; +} + +/** @param {string} candidate */ +function normalizeGitPath(candidate) { + return candidate.endsWith("/") ? candidate.slice(0, -1) : candidate; +} + +/** @param {string} candidate */ +function pathExists(candidate) { + try { + lstatSync(candidate); + return true; + } catch (error) { + if (!isFileSystemError(error)) throw error; + if (error.code === "ENOENT") return false; + throw error; + } +} + +/** @param {string} candidate @param {RootEntryIdentity} expected */ +function sameIdentity(candidate, expected) { + try { + const stat = lstatSync(candidate, { bigint: true }); + const kind = stat.isFile() + ? "file" + : stat.isDirectory() + ? "directory" + : null; + return ( + kind === expected.kind && + String(stat.dev) === expected.device && + String(stat.ino) === expected.inode + ); + } catch (error) { + if (!isFileSystemError(error)) throw error; + return false; + } +} + +/** @param {string} candidate */ +function isAbsent(candidate) { + try { + lstatSync(candidate); + return false; + } catch (error) { + if (!isFileSystemError(error)) throw error; + if (error.code === "ENOENT") return true; + throw error; + } +} + +/** @param {string} candidate */ +function removePathIfPresent(candidate) { + try { + unlinkSync(candidate); + } catch (error) { + if (!isFileSystemError(error) || error.code !== "ENOENT") throw error; + } +} + +/** @param {string} left @param {string} right */ +function compareStrings(left, right) { + return left < right ? -1 : left > right ? 1 : 0; +} + +/** @param {Set} left @param {Set} right */ +function setsEqual(left, right) { + return ( + left.size === right.size && [...left].every((value) => right.has(value)) + ); +} + +/** @param {Buffer | string} source */ +function sha256(source) { + return createHash("sha256").update(source).digest("hex"); +} diff --git a/test/compilation.test.js b/test/compilation.test.js index 85cce5b..c7503ac 100644 --- a/test/compilation.test.js +++ b/test/compilation.test.js @@ -5,6 +5,7 @@ import { mkdtempSync, mkdirSync, readFileSync, + realpathSync, rmSync, writeFileSync, } from "node:fs"; @@ -19,6 +20,7 @@ import { FOUNDATION_PLAN_FORMAT, MAX_ARTIFACT_BYTES, } from "../src/compilation-artifact.js"; +import { ROOT_TRANSACTION_NAME } from "../src/root-output.js"; const PROJECT_ID = "01900000-0000-7000-8000-000000003001"; const COMPILATION_ID = "01900000-0000-7000-8000-000000003002"; @@ -155,7 +157,7 @@ test("compilation status wait pins provenance and follows valid transitions", as }, ); - assert.equal(result.status, 0); + assert.equal(result.status, 0, result.stderr); assert.deepEqual(JSON.parse(result.stdout), succeeded); assert.deepEqual(delays, [1_000, 1_000]); assert.equal(calls.length, 3); @@ -231,7 +233,7 @@ test("compilation download distinguishes Head and Plan provenance without starti }, ); - assert.equal(result.status, 0); + assert.equal(result.status, 0, result.stderr); assert.equal(result.stderr, ""); assert.deepEqual( calls.map(({ input, init }) => [init?.method, String(input)]), @@ -250,6 +252,60 @@ test("compilation download distinguishes Head and Plan provenance without starti assert.equal(body.output.file_count, 1); }); +test("compilation download adopts the current directory without starting work", async (context) => { + const cwd = remoteDirectory(context); + writeFileSync(path.join(cwd, "product-notes.md"), "Design notes\n"); + const retainedState = readFileSync( + path.join(cwd, ".firstdraft/state.json"), + "utf8", + ); + const fixture = artifactFixture(); + const status = compilationBody("succeeded", { artifact: fixture }); + /** @type {FetchCall[]} */ + const calls = []; + const result = await invoke( + ["compilation", "download", COMPILATION_ID, "--output", "."], + { + cwd, + fetchFunction: sequenceFetch( + [jsonResponse(status), artifactResponse(fixture)], + calls, + ), + }, + ); + + assert.equal(result.status, 0, result.stderr); + assert.equal(result.stderr, ""); + assert.deepEqual( + calls.map(({ input, init }) => [init?.method, String(input)]), + [ + ["GET", `https://api.example.test${STATUS_PATH}`], + ["GET", `https://api.example.test${ARTIFACT_PATH}`], + ], + ); + assert.equal( + readFileSync(path.join(cwd, "README.md"), "utf8"), + "Movie Catalog\n", + ); + assert.equal( + readFileSync(path.join(cwd, "design/product-notes.md"), "utf8"), + "Design notes\n", + ); + assert.equal( + readFileSync(path.join(cwd, "design/.firstdraft/state.json"), "utf8"), + retainedState, + ); + assert.equal(existsSync(path.join(cwd, ROOT_TRANSACTION_NAME)), false); + const body = JSON.parse(result.stdout); + assert.equal(body.output.path, realpathSync(cwd)); + assert.deepEqual(body.output.root_adoption, { + design_path: path.join(realpathSync(cwd), "design"), + moved_entry_count: 2, + git_repository_preserved: false, + git_index_replaced: false, + }); +}); + test("download requires succeeded status and validates historical Head provenance", async (context) => { const cwd = remoteDirectory(context); /** @type {FetchCall[]} */ @@ -419,6 +475,15 @@ test("compilation syntax and output preflight fail before network access", async ); assertHandledFailure(invalidOutput, "invalid_output_path", 2); + mkdirSync(path.join(cwd, "design")); + const reservedRoot = await invoke( + ["compilation", "download", COMPILATION_ID, "--output", "."], + { cwd, fetchFunction: inaccessible }, + ); + assertHandledFailure(reservedRoot, "invalid_output_path", 2); + assert.equal(JSON.parse(reservedRoot.stderr).reason, "root_reserved_path"); + assert.equal(existsSync(path.join(cwd, ROOT_TRANSACTION_NAME)), false); + const help = await invoke(["compilation", "download", "--help"], { cwd: process.cwd(), apiToken: undefined, diff --git a/test/plan-compile.test.js b/test/plan-compile.test.js index 1d4300f..83d98b6 100644 --- a/test/plan-compile.test.js +++ b/test/plan-compile.test.js @@ -5,6 +5,7 @@ import { mkdtempSync, mkdirSync, readFileSync, + realpathSync, rmSync, writeFileSync, } from "node:fs"; @@ -18,6 +19,7 @@ import { ARTIFACT_MEDIA_TYPE, FOUNDATION_PLAN_FORMAT, } from "../src/compilation-artifact.js"; +import { ROOT_TRANSACTION_NAME } from "../src/root-output.js"; const PROJECT_ID = "01900000-0000-7000-8000-000000002001"; const ANALYSIS_ID = "01900000-0000-7000-8000-000000002002"; @@ -301,6 +303,106 @@ test("plan compile --output starts a direct Compilation without Publication", as ); }); +test("plan compile root output locks before push and releases after invalid analysis", async (context) => { + const cwd = localDirectory(context, PLAN_SOURCE); + writeFileSync(path.join(cwd, "notes.md"), "design notes\n"); + let compilations = 0; + const result = await invoke(["plan", "compile", "--output", "."], { + cwd, + planCompilePush: async () => { + assert.equal(existsSync(path.join(cwd, ROOT_TRANSACTION_NAME)), true); + return successfulPush(); + }, + planCompileReadStatus: async () => ({ + status: 200, + body: analysisBody("invalid"), + }), + planCompileDownload: async () => { + compilations += 1; + throw new Error("Compilation must remain untouched"); + }, + }); + + assertHandledFailure(result, "plan_not_valid"); + assert.equal(compilations, 0); + assert.equal(existsSync(path.join(cwd, ROOT_TRANSACTION_NAME)), false); + assert.equal( + readFileSync(path.join(cwd, "notes.md"), "utf8"), + "design notes\n", + ); +}); + +test("plan compile root output materializes directly without Publication", async (context) => { + const cwd = localDirectory(context, PLAN_SOURCE, { + api_url: "https://api.example.test", + foundation_plan_etag: ETAG, + }); + writeFileSync(path.join(cwd, "product-notes.md"), "Design notes\n"); + const artifact = directArtifactFixture(); + /** @type {{input: string | URL | Request, init: RequestInit}[]} */ + const calls = []; + const result = await invoke(["plan", "compile", "--output", "."], { + cwd, + planCompilePush: successfulPush, + planCompileReadStatus: async () => ({ + status: 200, + body: analysisBody("valid"), + }), + planCompilePublish: async () => { + throw new Error("Publication must remain untouched"); + }, + fetchFunction: sequenceFetch( + [ + jsonResponse(directCompilationBody("succeeded", artifact), 202, { + Location: directCompilationPath(), + }), + new Response(artifact.source, { + status: 200, + headers: { + "Content-Type": ARTIFACT_MEDIA_TYPE, + "Content-Length": String(artifact.source.byteLength), + "Cache-Control": "no-store, no-transform", + ETag: `"sha256:${artifact.sha256}"`, + }, + }), + ], + calls, + ), + }); + + assert.equal(result.status, 0, result.stderr); + assert.deepEqual( + calls.map((call) => [call.init.method, String(call.input)]), + [ + ["POST", `https://api.example.test${compilationCollectionPath()}`], + ["GET", `https://api.example.test${directArtifactPath()}`], + ], + ); + assert.equal( + readFileSync(path.join(cwd, "README.md"), "utf8"), + "Movie Catalog\n", + ); + assert.equal( + readFileSync(path.join(cwd, "design/product-notes.md"), "utf8"), + "Design notes\n", + ); + assert.equal( + readFileSync( + path.join(cwd, "design/.firstdraft/foundation-plan.json"), + "utf8", + ), + PLAN_SOURCE.toString("utf8"), + ); + assert.equal(existsSync(path.join(cwd, ROOT_TRANSACTION_NAME)), false); + const output = JSON.parse(result.stdout).output; + assert.equal(output.path, realpathSync(cwd)); + assert.equal( + output.root_adoption.design_path, + path.join(realpathSync(cwd), "design"), + ); + assert.equal(output.root_adoption.moved_entry_count, 2); +}); + test("plan compile --output rejects an existing destination before Plan mutation", async (context) => { const cwd = localDirectory(context, PLAN_SOURCE); mkdirSync(path.join(cwd, "application")); @@ -909,7 +1011,7 @@ test("help and invalid direct-output syntax have no prerequisites", async () => assert.equal(help.status, 0); assert.match( help.stdout, - /firstdraft plan compile --output /, + /firstdraft plan compile --output /, ); for (const argv of [ diff --git a/test/plan-publish.test.js b/test/plan-publish.test.js index 0183f02..998a320 100644 --- a/test/plan-publish.test.js +++ b/test/plan-publish.test.js @@ -83,11 +83,11 @@ const PLAN_COMPILE_HELP = `First Draft CLI Usage: firstdraft plan compile - firstdraft plan compile --output + firstdraft plan compile --output Options: - --output Materialize the generated application here - -h, --help Show help + --output Materialize the generated application here + -h, --help Show help Environment: FIRSTDRAFT_API_TOKEN Authenticate API requests @@ -97,7 +97,8 @@ The command submits the exact current whole-file Plan, waits for its analysis, and proceeds only when that analysis is valid. Without --output, it conditionally creates or replays the internal GitHub Publication lifecycle and prints the private repository URL. With --output, it starts one direct Compilation and -atomically materializes the verified artifact into an absent directory without +atomically materializes the verified artifact into an absent directory, or +preserves existing root material under ./design when the output is ., without starting GitHub Publication. Progress is written to stderr. `; diff --git a/test/root-output.test.js b/test/root-output.test.js new file mode 100644 index 0000000..fa08fec --- /dev/null +++ b/test/root-output.test.js @@ -0,0 +1,474 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { + chmodSync, + existsSync, + lstatSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + realpathSync, + renameSync, + rmSync, + symlinkSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import test from "node:test"; + +import { + materializeRootOutput, + prepareRootOutput, + releaseRootOutput, + ROOT_TRANSACTION_NAME, + RootOutputMaterializationError, + RootOutputPathError, +} from "../src/root-output.js"; + +/** @type {Array<{path: string, mode: 420 | 493, contents: string}>} */ +const ARTIFACT_FILES = [ + { path: "README.md", mode: 0o644, contents: "Generated application\n" }, + { path: "app/models/movie.rb", mode: 0o644, contents: "class Movie; end\n" }, + { path: "bin/setup", mode: 0o755, contents: "#!/bin/sh\n" }, +]; + +test("adopts an arbitrary non-Git root without traversing preserved interiors", (context) => { + const root = temporaryDirectory(context); + mkdirSync(path.join(root, ".firstdraft")); + writeFileSync( + path.join(root, ".firstdraft", "foundation-plan.json"), + "plan\n", + ); + mkdirSync(path.join(root, "materials")); + writeFileSync(path.join(root, "materials", "notes.md"), "notes\n"); + symlinkSync("notes.md", path.join(root, "materials", "latest")); + + const target = prepareRootOutput({ root }); + assert.equal(existsSync(path.join(root, ROOT_TRANSACTION_NAME)), true); + const result = materialize(target); + + assert.deepEqual(result.root_adoption, { + design_path: path.join(realpathSync(root), "design"), + moved_entry_count: 2, + git_repository_preserved: false, + git_index_replaced: false, + }); + assert.equal( + readFileSync(path.join(root, "README.md"), "utf8"), + "Generated application\n", + ); + assert.equal( + readFileSync( + path.join(root, "design/.firstdraft/foundation-plan.json"), + "utf8", + ), + "plan\n", + ); + assert.equal( + lstatSync(path.join(root, "design/materials/latest")).isSymbolicLink(), + true, + ); + assert.equal(existsSync(path.join(root, ROOT_TRANSACTION_NAME)), false); +}); + +test("preflight holds one root lock and releases it before any irreversible phase", (context) => { + const root = temporaryDirectory(context); + writeFileSync(path.join(root, "notes.md"), "notes\n"); + const first = prepareRootOutput({ root }); + assert.throws( + () => prepareRootOutput({ root }), + (error) => + error instanceof RootOutputPathError && error.reason === "root_busy", + ); + + releaseRootOutput(first); + assert.equal(existsSync(path.join(root, ROOT_TRANSACTION_NAME)), false); + const second = prepareRootOutput({ root }); + releaseRootOutput(second); +}); + +test("rejects unsafe root shapes before mutation", (context) => { + const windowsRoot = temporaryDirectory(context); + assert.throws( + () => prepareRootOutput({ root: windowsRoot, platform: "win32" }), + (error) => + error instanceof RootOutputPathError && + error.reason === "root_platform_unsupported", + ); + + const linkedRoot = temporaryDirectory(context); + writeFileSync(path.join(linkedRoot, "outside"), "outside\n"); + symlinkSync("outside", path.join(linkedRoot, "linked")); + assert.throws( + () => prepareRootOutput({ root: linkedRoot }), + (error) => + error instanceof RootOutputPathError && + error.reason === "root_entry_unsupported", + ); + assert.equal(existsSync(path.join(linkedRoot, ROOT_TRANSACTION_NAME)), false); + + const reservedRoot = temporaryDirectory(context); + mkdirSync(path.join(reservedRoot, "Design")); + assert.throws( + () => prepareRootOutput({ root: reservedRoot }), + (error) => + error instanceof RootOutputPathError && + error.reason === "root_reserved_path", + ); +}); + +test("preserves a Git worktree and installs an exact prepared index", (context) => { + const root = temporaryDirectory(context); + initializeGit(root); + writeFileSync(path.join(root, ".gitignore"), ".env\n"); + writeFileSync(path.join(root, "README.md"), "Design README\n"); + mkdirSync(path.join(root, ".firstdraft")); + writeFileSync( + path.join(root, ".firstdraft", "foundation-plan.json"), + "plan\n", + ); + writeFileSync(path.join(root, ".env"), "SECRET=value\n"); + writeFileSync(path.join(root, "notes.md"), "untracked\n"); + git(root, [ + "add", + ".gitignore", + "README.md", + ".firstdraft/foundation-plan.json", + ]); + git(root, ["commit", "-m", "Design application"]); + const originalHead = git(root, ["rev-parse", "HEAD"]).trim(); + + const target = prepareRootOutput({ root }); + const result = materialize(target); + + assert.equal(result.root_adoption.git_repository_preserved, true); + assert.equal(result.root_adoption.git_index_replaced, true); + assert.equal(git(root, ["rev-parse", "HEAD"]).trim(), originalHead); + assert.equal( + readFileSync(path.join(root, "README.md"), "utf8"), + "Generated application\n", + ); + assert.equal( + readFileSync(path.join(root, "design/README.md"), "utf8"), + "Design README\n", + ); + assert.equal( + readFileSync(path.join(root, "design/.env"), "utf8"), + "SECRET=value\n", + ); + assert.equal(gitStatus(root, ["check-ignore", "design/.env"]), 0); + assert.equal(gitStatus(root, ["diff-files", "--quiet", "--"]), 0); + + const indexed = git(root, ["ls-files", "-z"]) + .split("\0") + .filter(Boolean) + .sort(); + assert.deepEqual(indexed, [ + "README.md", + "app/models/movie.rb", + "bin/setup", + "design/.firstdraft/foundation-plan.json", + "design/.gitignore", + "design/README.md", + ]); + assert.equal(indexed.includes("design/.env"), false); + assert.equal(indexed.includes("design/notes.md"), false); +}); + +test("preserves a linked Git worktree without relocating its Git file", (context) => { + const holder = temporaryDirectory(context); + initializeGit(holder); + writeFileSync(path.join(holder, "README.md"), "Design README\n"); + git(holder, ["add", "README.md"]); + git(holder, ["commit", "-m", "Design application"]); + const root = path.join(holder, "linked"); + git(holder, ["worktree", "add", "--quiet", "--detach", root, "HEAD"]); + const gitFile = readFileSync(path.join(root, ".git")); + + const target = prepareRootOutput({ root }); + const result = materialize(target); + + assert.equal(result.root_adoption.git_repository_preserved, true); + assert.equal(lstatSync(path.join(root, ".git")).isFile(), true); + assert.equal(readFileSync(path.join(root, ".git")).equals(gitFile), true); + assert.equal(gitStatus(root, ["diff-files", "--quiet", "--"]), 0); + assert.equal( + readFileSync(path.join(root, "README.md"), "utf8"), + "Generated application\n", + ); + assert.equal( + readFileSync(path.join(root, "design/README.md"), "utf8"), + "Design README\n", + ); +}); + +test("refuses enclosing worktrees, dirty indexes, and submodule metadata", (context) => { + const enclosing = temporaryDirectory(context); + initializeGit(enclosing); + mkdirSync(path.join(enclosing, "child")); + assert.throws( + () => prepareRootOutput({ root: path.join(enclosing, "child") }), + (error) => + error instanceof RootOutputPathError && + error.reason === "root_enclosing_worktree", + ); + + const dirty = temporaryDirectory(context); + initializeGit(dirty); + writeFileSync(path.join(dirty, "tracked"), "before\n"); + git(dirty, ["add", "tracked"]); + git(dirty, ["commit", "-m", "Tracked"]); + writeFileSync(path.join(dirty, "tracked"), "after\n"); + assert.throws( + () => prepareRootOutput({ root: dirty }), + (error) => + error instanceof RootOutputPathError && error.reason === "root_git_dirty", + ); + + const submodule = temporaryDirectory(context); + initializeGit(submodule); + writeFileSync(path.join(submodule, ".gitmodules"), '[submodule "vendor"]\n'); + git(submodule, ["add", ".gitmodules"]); + git(submodule, ["commit", "-m", "Submodule metadata"]); + assert.throws( + () => prepareRootOutput({ root: submodule }), + (error) => + error instanceof RootOutputPathError && + error.reason === "root_git_unsupported", + ); +}); + +test("detects pre-move changes and rolls back a failed rename exactly", (context) => { + const changedRoot = temporaryDirectory(context); + writeFileSync(path.join(changedRoot, "notes.md"), "notes\n"); + const changed = prepareRootOutput({ root: changedRoot }); + writeFileSync(path.join(changedRoot, "new.md"), "new\n"); + assert.throws( + () => materialize(changed), + (error) => + error instanceof RootOutputMaterializationError && + error.reason === "output_changed", + ); + assert.equal( + existsSync(path.join(changedRoot, ROOT_TRANSACTION_NAME)), + false, + ); + assert.equal( + readFileSync(path.join(changedRoot, "notes.md"), "utf8"), + "notes\n", + ); + + const rollbackRoot = temporaryDirectory(context); + writeFileSync(path.join(rollbackRoot, "one"), "one\n"); + writeFileSync(path.join(rollbackRoot, "two"), "two\n"); + const rollback = prepareRootOutput({ root: rollbackRoot }); + let renames = 0; + assert.throws( + () => + materialize(rollback, { + rename(from, to) { + renames += 1; + if (renames === 2) { + const error = new Error("injected rename failure"); + Object.assign(error, { code: "EIO" }); + throw error; + } + renameSync(from, to); + }, + }), + (error) => + error instanceof RootOutputMaterializationError && + error.reason === "root_transaction_failed", + ); + assert.equal(readFileSync(path.join(rollbackRoot, "one"), "utf8"), "one\n"); + assert.equal(readFileSync(path.join(rollbackRoot, "two"), "utf8"), "two\n"); + assert.equal(existsSync(path.join(rollbackRoot, "design")), false); + assert.equal( + existsSync(path.join(rollbackRoot, ROOT_TRANSACTION_NAME)), + false, + ); +}); + +test("restores exact Git index bytes when post-install verification fails", (context) => { + const root = temporaryDirectory(context); + initializeGit(root); + writeFileSync(path.join(root, "README.md"), "Design README\n"); + git(root, ["add", "README.md"]); + git(root, ["commit", "-m", "Design application"]); + const indexPath = path.resolve( + root, + git(root, ["rev-parse", "--git-path", "index"]).trim(), + ); + const originalIndex = readFileSync(indexPath); + const target = prepareRootOutput({ root }); + let verifications = 0; + + assert.throws( + () => + materializeRootOutput( + target, + { files: ARTIFACT_FILES, manifest_sha256: "a".repeat(64) }, + { + writeArtifact, + verifyArtifact(artifactRoot, ignoredRootEntries) { + verifications += 1; + verifyArtifact(artifactRoot, ignoredRootEntries); + if (verifications === 2) { + throw new RootOutputMaterializationError( + "Injected post-install verification failure.", + "root_transaction_failed", + ); + } + }, + }, + ), + (error) => + error instanceof RootOutputMaterializationError && + error.reason === "root_transaction_failed", + ); + + assert.equal(readFileSync(indexPath).equals(originalIndex), true); + assert.equal(gitStatus(root, ["diff", "--quiet", "--"]), 0); + assert.equal(gitStatus(root, ["diff", "--cached", "--quiet", "--"]), 0); + assert.equal( + readFileSync(path.join(root, "README.md"), "utf8"), + "Design README\n", + ); + assert.equal(existsSync(path.join(root, "design")), false); + assert.equal(existsSync(path.join(root, ROOT_TRANSACTION_NAME)), false); +}); + +test("refuses artifact collisions with reserved root names", (context) => { + const root = temporaryDirectory(context); + writeFileSync(path.join(root, "notes.md"), "notes\n"); + const target = prepareRootOutput({ root }); + assert.throws( + () => + materializeRootOutput( + target, + { + files: [{ path: "design/README.md", mode: 0o644 }], + manifest_sha256: "a".repeat(64), + }, + { writeArtifact, verifyArtifact }, + ), + (error) => + error instanceof RootOutputMaterializationError && + error.reason === "root_artifact_collision", + ); + assert.equal(readFileSync(path.join(root, "notes.md"), "utf8"), "notes\n"); + assert.equal(existsSync(path.join(root, ROOT_TRANSACTION_NAME)), false); +}); + +test("retains a versioned journal only when rollback cannot complete", (context) => { + const root = temporaryDirectory(context); + writeFileSync(path.join(root, "one"), "one\n"); + writeFileSync(path.join(root, "two"), "two\n"); + const target = prepareRootOutput({ root }); + let renames = 0; + assert.throws( + () => + materialize(target, { + rename(from, to) { + renames += 1; + if (renames === 2 || renames === 3) { + const error = new Error("injected rename failure"); + Object.assign(error, { code: "EIO" }); + throw error; + } + renameSync(from, to); + }, + }), + (error) => + error instanceof RootOutputMaterializationError && + error.reason === "root_rollback_incomplete" && + error.recoveryPath === ROOT_TRANSACTION_NAME, + ); + const journal = JSON.parse( + readFileSync( + path.join(root, ROOT_TRANSACTION_NAME, "journal.json"), + "utf8", + ), + ); + assert.equal(journal.format, "firstdraft.root-output-transaction/1"); + assert.equal(journal.phase, "rollback_incomplete"); +}); + +/** @param {ReturnType} target @param {{rename?: (from: string, to: string) => void}} [options] */ +function materialize(target, options = {}) { + return materializeRootOutput( + target, + { files: ARTIFACT_FILES, manifest_sha256: "a".repeat(64) }, + { + writeArtifact, + verifyArtifact, + ...options, + }, + ); +} + +/** @param {string} root */ +function writeArtifact(root) { + for (const file of ARTIFACT_FILES) { + const destination = path.join(root, ...file.path.split("/")); + mkdirSync(path.dirname(destination), { recursive: true, mode: 0o755 }); + writeFileSync(destination, file.contents, { mode: file.mode }); + chmodSync(destination, file.mode); + } +} + +/** @param {string} root @param {Set} [ignoredRootEntries] */ +function verifyArtifact(root, ignoredRootEntries = new Set()) { + const expected = new Set(ARTIFACT_FILES.map((file) => file.path)); + const actual = new Set(); + walkFiles(root, "", actual, ignoredRootEntries); + assert.deepEqual(actual, expected); + for (const file of ARTIFACT_FILES) { + const destination = path.join(root, ...file.path.split("/")); + assert.equal(readFileSync(destination, "utf8"), file.contents); + if (process.platform !== "win32") { + assert.equal(lstatSync(destination).mode & 0o777, file.mode); + } + } +} + +/** @param {string} root @param {string} relative @param {Set} files @param {Set} ignoredRootEntries */ +function walkFiles(root, relative, files, ignoredRootEntries) { + const current = + relative === "" ? root : path.join(root, ...relative.split("/")); + for (const entry of readdirSync(current, { withFileTypes: true })) { + if (relative === "" && ignoredRootEntries.has(entry.name)) continue; + const child = relative === "" ? entry.name : `${relative}/${entry.name}`; + if (entry.isDirectory()) walkFiles(root, child, files, ignoredRootEntries); + else if (entry.isFile()) files.add(child); + else assert.fail(`unexpected artifact entry ${child}`); + } +} + +/** @param {import("node:test").TestContext} context */ +function temporaryDirectory(context) { + const directory = mkdtempSync(path.join(tmpdir(), "firstdraft-root-output-")); + context.after(() => rmSync(directory, { recursive: true, force: true })); + return directory; +} + +/** @param {string} cwd */ +function initializeGit(cwd) { + git(cwd, ["init", "--quiet"]); + git(cwd, ["config", "user.name", "First Draft Tests"]); + git(cwd, ["config", "user.email", "tests@firstdraft.test"]); +} + +/** @param {string} cwd @param {string[]} arguments_ */ +function git(cwd, arguments_) { + const result = spawnSync("git", arguments_, { cwd, encoding: "utf8" }); + assert.equal(result.status, 0, result.stderr); + return result.stdout; +} + +/** @param {string} cwd @param {string[]} arguments_ */ +function gitStatus(cwd, arguments_) { + return spawnSync("git", arguments_, { cwd }).status; +} From 0b8f75611e0223eeb79ed0bc60fda8c7f642eece Mon Sep 17 00:00:00 2001 From: Raghu Betina Date: Sat, 29 Aug 2026 04:54:12 -0500 Subject: [PATCH 6/9] Align output path error contract Expose the documented destination_exists reason for ordinary occupied output paths and keep the direct-output command documentation accurate for current-directory adoption. --- src/commands/compilation.js | 3 ++- src/commands/plan-compile.js | 5 +++-- src/compilation-artifact.js | 4 +++- test/compilation.test.js | 1 + 4 files changed, 9 insertions(+), 4 deletions(-) diff --git a/src/commands/compilation.js b/src/commands/compilation.js index b4ca580..b903551 100644 --- a/src/commands/compilation.js +++ b/src/commands/compilation.js @@ -384,7 +384,8 @@ export async function downloadCompilation({ /** * Start one Compilation for the exact Plan and reviewed Analysis accepted by * the current command, wait for that retained Compilation, and materialize its - * authenticated artifact into an absent destination. + * authenticated artifact into an absent destination or the eligible current + * directory. * * @param {CompileAndDownloadOptions} options */ diff --git a/src/commands/plan-compile.js b/src/commands/plan-compile.js index 1ee4196..c13258e 100644 --- a/src/commands/plan-compile.js +++ b/src/commands/plan-compile.js @@ -133,8 +133,9 @@ export async function compilePlan({ /** * Submit and analyze the exact current local Plan, then start one direct - * Compilation and materialize its verified artifact into an absent directory. - * GitHub Publication remains the no-output mode owned by compilePlan. + * Compilation and materialize its verified artifact into an absent directory + * or the eligible current directory. GitHub Publication remains the no-output + * mode owned by compilePlan. * * @param {CompilePlanToDirectoryOptions} options */ diff --git a/src/compilation-artifact.js b/src/compilation-artifact.js index 61d0ebd..a9e5bdd 100644 --- a/src/compilation-artifact.js +++ b/src/compilation-artifact.js @@ -192,13 +192,15 @@ export function resolveOutputTarget({ cwd, output }) { const target = path.resolve(cwd, output); const parent = path.dirname(target); + const targetIsAbsent = isInitiallyAbsent(target); if ( target === parent || path.basename(target).length === 0 || - !isInitiallyAbsent(target) + !targetIsAbsent ) { throw new CompilationOutputPathError( "The compilation output path must be absent.", + targetIsAbsent ? undefined : { reason: "destination_exists" }, ); } diff --git a/test/compilation.test.js b/test/compilation.test.js index c7503ac..3d08a83 100644 --- a/test/compilation.test.js +++ b/test/compilation.test.js @@ -474,6 +474,7 @@ test("compilation syntax and output preflight fail before network access", async { cwd, fetchFunction: inaccessible }, ); assertHandledFailure(invalidOutput, "invalid_output_path", 2); + assert.equal(JSON.parse(invalidOutput.stderr).reason, "destination_exists"); mkdirSync(path.join(cwd, "design")); const reservedRoot = await invoke( From c5df59aa58e081613159cfdcf573f83b70cb7f64 Mon Sep 17 00:00:00 2001 From: Raghu Betina Date: Sat, 29 Aug 2026 05:11:57 -0500 Subject: [PATCH 7/9] Close root output recovery gaps Guarantee rollback for unexpected verifier failures, make preflight and release errors stable, restore the post-analysis destination check, and document the Git object boundary before offering direct root output. --- docs/commands.md | 20 +++++++------- src/commands/compilation.js | 7 +++++ src/root-output.js | 53 ++++++++++++++++++++++++++++++++----- test/plan-compile.test.js | 37 ++++++++++++++++++++++++++ test/root-output.test.js | 11 ++++---- 5 files changed, 108 insertions(+), 20 deletions(-) diff --git a/docs/commands.md b/docs/commands.md index 5a92731..1cf86ea 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -187,11 +187,12 @@ both honored. A refusal is `invalid_output_path` with a machine-readable `reason The CLI creates `.firstdraft-root-output` with exclusive creation during the pre-push output check and holds it through analysis, Compilation, and materialization. That directory is both the single-writer lock and the owned transaction journal, so a concurrent root adoption is refused before either command sends a request. Immediately -after acquiring it, the CLI captures every other top-level entry's exact name, entry type, device, and inode. It -rechecks that set immediately before moving anything. Size, modification time, and contents are deliberately not -part of this identity: interior changes are not recursively inventoried, and a top-level directory moves intact at -the transaction boundary. A replaced, added, or removed top-level entry stops materialization. The reserved-path -precondition ignores only the transaction directory created and still held by this invocation. +after acquiring it, the CLI captures every other top-level entry's exact name, entry type, device, and inode. After +staging the artifact and any replacement Git index, it rechecks that set immediately before moving anything. Size, +modification time, and contents are deliberately not part of this identity: interior changes are not recursively +inventoried, and a top-level directory moves intact at the transaction boundary. A replaced, added, or removed +top-level entry stops materialization. The reserved-path precondition ignores only the transaction directory +created and still held by this invocation. `compilation download --output .` acquires the same lock before its first status request and holds it through artifact download and materialization. Either command removes its own transaction directory on every ordinary exit @@ -216,13 +217,14 @@ failure, not permission to copy or traverse the mount. For a Git root, the CLI first prepares a replacement index that stages each formerly tracked path at `design/` and stages every exact generated artifact path at the root. This handles overlapping names such as `README.md` and `.gitignore` without leaving the old design blob indexed at a generated path. Previously -untracked and ignored paths are never added to the index. After the worktree renames finish, the CLI installs the +untracked and ignored paths are never added to the index. Preparing that index writes the generated blobs into the +Git object database; a rollback may therefore leave unreachable blobs for ordinary Git garbage collection, while +`HEAD`, refs, configuration, and history remain unchanged. After the worktree renames finish, the CLI installs the prepared index through Git's actual index lock path and atomic lock-file commit protocol, including in a linked worktree whose index is outside the adopted root. The transaction journal retains whether an index existed plus an exact private copy, mode, and digest of its prior bytes until final verification succeeds. The preflighted ignore -protection is rechecked after the move. `HEAD`, refs, configuration, and history do not change. The caller should -inspect and commit this staged root-adoption change before using destructive worktree or index restoration commands. -A non-Git root remains non-Git and is not initialized. +protection is rechecked after the move. The caller should inspect and commit this staged root-adoption change before +using destructive worktree or index restoration commands. A non-Git root remains non-Git and is not initialized. The journal is a versioned private JSON record plus owned staging files. It records the physical root and original top-level identity set, the transaction phase, completed design and artifact renames, and, for Git, the resolved diff --git a/src/commands/compilation.js b/src/commands/compilation.js index b903551..f8bc358 100644 --- a/src/commands/compilation.js +++ b/src/commands/compilation.js @@ -12,6 +12,7 @@ import { parseCompilationArtifact, prepareCompilationOutputTarget, releaseCompilationOutputTarget, + resolveOutputTarget, } from "../compilation-artifact.js"; import { FirstDraftNetworkError, @@ -412,6 +413,12 @@ export async function compileAndDownload({ expected, fileSystem, }); + if ( + suppliedOutputTarget !== undefined && + typeof outputTarget === "string" + ) { + resolveOutputTarget({ cwd, output }); + } const deadline = now() + WAIT_TIMEOUT_MS; onProgress({ phase: "compilation", status: "waiting" }); diff --git a/src/root-output.js b/src/root-output.js index 765a47f..c887fc2 100644 --- a/src/root-output.js +++ b/src/root-output.js @@ -229,7 +229,15 @@ export function prepareRootOutput({ root, platform = process.platform }) { installSignalHandlers(target); return target; } catch (error) { - removePreparedTransaction(target); + releaseRootOutput(target); + if (error instanceof RootOutputPathError) throw error; + if (isFileSystemError(error)) { + throw new RootOutputPathError( + "The root output preflight could not be completed.", + "root_not_writable", + { cause: error }, + ); + } throw error; } } @@ -238,7 +246,21 @@ export function prepareRootOutput({ root, platform = process.platform }) { export function releaseRootOutput(target) { removeSignalHandlers(target); if (target.released || target.irreversible) return; - removePreparedTransaction(target); + try { + removePreparedTransaction(target); + } catch (error) { + if (isFileSystemError(error) && error.code === "ENOENT") { + target.released = true; + return; + } + if ( + error instanceof RootOutputMaterializationError || + isFileSystemError(error) + ) { + return; + } + throw error; + } } /** @@ -265,9 +287,9 @@ export function materializeRootOutput( chmodSync(artifactStage, DIRECTORY_MODE); writeArtifact(artifactStage); verifyArtifact(artifactStage); - recheckRootSnapshot(target); if (target.git !== null) prepareGitIndex(target, artifact.files); + recheckRootSnapshot(target); performRootTransaction(target, artifact.files, rename); verifyRootResult(target, artifact.files, verifyArtifact); @@ -769,7 +791,11 @@ function normalizeMaterializationError(error) { { cause: error }, ); } - throw error; + return new RootOutputMaterializationError( + "The root output transaction failed.", + "root_transaction_failed", + { cause: error }, + ); } /** @param {string} root @param {string} transactionPath */ @@ -1061,7 +1087,17 @@ function createPreviewEntry(root, previewWorktree, ignoredPath) { ROOT_DESIGN_NAME, ...ignoredPath.split("/"), ); - const stat = lstatSync(source); + let stat; + try { + stat = lstatSync(source); + } catch (error) { + if (!isFileSystemError(error)) throw error; + throw new RootOutputPathError( + "A currently ignored path changed during root output preflight.", + "root_ignore_not_preserved", + { cause: error }, + ); + } if (stat.isDirectory()) { mkdirSync(destination, { recursive: true, mode: DIRECTORY_MODE }); } else { @@ -1317,7 +1353,12 @@ function requiredGit(root, arguments_, options = {}) { */ function invokeGit(root, arguments_, options = {}) { /** @type {NodeJS.ProcessEnv} */ - const environment = { ...process.env, GIT_OPTIONAL_LOCKS: "0" }; + const environment = { + ...process.env, + GIT_OPTIONAL_LOCKS: "0", + LC_ALL: "C", + LANGUAGE: "", + }; delete environment.GIT_DIR; delete environment.GIT_WORK_TREE; delete environment.GIT_INDEX_FILE; diff --git a/test/plan-compile.test.js b/test/plan-compile.test.js index 83d98b6..19b529e 100644 --- a/test/plan-compile.test.js +++ b/test/plan-compile.test.js @@ -422,6 +422,43 @@ test("plan compile --output rejects an existing destination before Plan mutation assert.equal(pushes, 0); }); +test("plan compile rechecks an absent destination after analysis", async (context) => { + /** @type {string | undefined} */ + let cwd; + let compilationStarts = 0; + const server = createServer(async (request, response) => { + await readRequestBody(request); + if (request.method === "PUT" && request.url === planPath()) { + respondJson(response, 201, acceptedPlanBody(), { ETag: ETAG }); + return; + } + if (request.method === "GET" && request.url === analysisPath()) { + assert(cwd); + mkdirSync(path.join(cwd, "application")); + respondJson(response, 200, analysisBody("valid")); + return; + } + if ( + request.method === "POST" && + request.url === compilationCollectionPath() + ) { + compilationStarts += 1; + } + response.writeHead(500).end(); + }); + const apiUrl = await listen(context, server); + cwd = localDirectory(context, PLAN_SOURCE); + + const result = await invoke( + ["plan", "compile", "--output", "./application"], + { cwd, apiUrl }, + ); + + assertHandledFailure(result, "invalid_output_path", 2); + assert.equal(errorEnvelope(result.stderr).reason, "destination_exists"); + assert.equal(compilationStarts, 0); +}); + test("plan compile --output rechecks local Plan bytes before starting work", async (context) => { /** @type {string | undefined} */ let cwd; diff --git a/test/root-output.test.js b/test/root-output.test.js index fa08fec..76f3795 100644 --- a/test/root-output.test.js +++ b/test/root-output.test.js @@ -87,6 +87,10 @@ test("preflight holds one root lock and releases it before any irreversible phas assert.equal(existsSync(path.join(root, ROOT_TRANSACTION_NAME)), false); const second = prepareRootOutput({ root }); releaseRootOutput(second); + + const externallyRemoved = prepareRootOutput({ root }); + rmSync(externallyRemoved.transactionPath, { recursive: true }); + assert.doesNotThrow(() => releaseRootOutput(externallyRemoved)); }); test("rejects unsafe root shapes before mutation", (context) => { @@ -291,7 +295,7 @@ test("detects pre-move changes and rolls back a failed rename exactly", (context ); }); -test("restores exact Git index bytes when post-install verification fails", (context) => { +test("rolls back an unexpected post-install verification failure", (context) => { const root = temporaryDirectory(context); initializeGit(root); writeFileSync(path.join(root, "README.md"), "Design README\n"); @@ -316,10 +320,7 @@ test("restores exact Git index bytes when post-install verification fails", (con verifications += 1; verifyArtifact(artifactRoot, ignoredRootEntries); if (verifications === 2) { - throw new RootOutputMaterializationError( - "Injected post-install verification failure.", - "root_transaction_failed", - ); + throw new Error("Injected post-install verification failure."); } }, }, From 5cc7f2d495e26fffbb0c54b4669f97ef3d285466 Mon Sep 17 00:00:00 2001 From: Raghu Betina Date: Sat, 29 Aug 2026 05:18:27 -0500 Subject: [PATCH 8/9] Preserve external index rewrites Avoid restoring a Git index before First Draft attempted installation, and keep output-path errors truthful when Plan analysis has already used the network. --- src/cli.js | 2 +- src/root-output.js | 9 +++++++++ test/root-output.test.js | 42 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 52 insertions(+), 1 deletion(-) diff --git a/src/cli.js b/src/cli.js index 4333ab8..8dc01d8 100644 --- a/src/cli.js +++ b/src/cli.js @@ -396,7 +396,7 @@ const PLAN_COMPILE_DIRECT_ARTIFACT_INVALID_DETAIL = const PLAN_COMPILE_DIRECT_MATERIALIZATION_FAILED_DETAIL = "The retained Compilation artifact was validated but could not be materialized. Use current.compilation.id with 'firstdraft compilation download' after repairing the output path; do not start another Compilation."; const COMPILATION_INVALID_OUTPUT_PATH_DETAIL = - "The compilation output must be an absent path beneath an existing real directory or the eligible current directory. No network request was made."; + "The compilation output must be an absent path beneath an existing real directory or the eligible current directory."; const GENERATE_UUID_INVALID_ARGUMENTS_DETAIL = "Invalid arguments. Run 'firstdraft generate uuid --help' for usage."; const GENERATE_APPLICATION_KEY_INVALID_ARGUMENTS_DETAIL = diff --git a/src/root-output.js b/src/root-output.js index c887fc2..220f153 100644 --- a/src/root-output.js +++ b/src/root-output.js @@ -1263,6 +1263,15 @@ function rollbackIndex(target) { const indexJournal = /** @type {Record} */ ( target.journal.index ); + const pending = /** @type {Record | null} */ ( + target.journal.pending + ); + const installationAttempted = + indexJournal.installed === true || + target.journal.phase === "index_installed" || + pending?.kind === "install_git_index"; + if (!installationAttempted) return true; + try { const indexExists = pathExists(git.indexPath); const currentDigest = indexExists diff --git a/test/root-output.test.js b/test/root-output.test.js index 76f3795..f6ac5ac 100644 --- a/test/root-output.test.js +++ b/test/root-output.test.js @@ -295,6 +295,48 @@ test("detects pre-move changes and rolls back a failed rename exactly", (context ); }); +test("preserves a Git index changed before installation", (context) => { + const root = temporaryDirectory(context); + initializeGit(root); + writeFileSync(path.join(root, "README.md"), "Design README\n"); + git(root, ["add", "README.md"]); + git(root, ["commit", "-m", "Design application"]); + const indexPath = path.resolve( + root, + git(root, ["rev-parse", "--git-path", "index"]).trim(), + ); + const originalIndex = readFileSync(indexPath); + const target = prepareRootOutput({ root }); + /** @type {Buffer | undefined} */ + let externalIndex; + + assert.throws( + () => + materialize(target, { + rename() { + git(root, ["update-index", "--index-version=4"]); + externalIndex = readFileSync(indexPath); + assert.equal(externalIndex.equals(originalIndex), false); + const error = new Error("Injected pre-install rename failure"); + Object.assign(error, { code: "EIO" }); + throw error; + }, + }), + (error) => + error instanceof RootOutputMaterializationError && + error.reason === "root_transaction_failed", + ); + + assert(externalIndex); + assert.equal(readFileSync(indexPath).equals(externalIndex), true); + assert.equal( + readFileSync(path.join(root, "README.md"), "utf8"), + "Design README\n", + ); + assert.equal(existsSync(path.join(root, "design")), false); + assert.equal(existsSync(path.join(root, ROOT_TRANSACTION_NAME)), false); +}); + test("rolls back an unexpected post-install verification failure", (context) => { const root = temporaryDirectory(context); initializeGit(root); From e4028b20c72e3c831716c3461c744511f02f55ce Mon Sep 17 00:00:00 2001 From: Raghu Betina Date: Sat, 29 Aug 2026 05:26:56 -0500 Subject: [PATCH 9/9] Scope root output tests to POSIX --- test/compilation.test.js | 8 +++++++- test/plan-compile.test.js | 2 ++ test/root-output.test.js | 28 ++++++++++++++++++++-------- 3 files changed, 29 insertions(+), 9 deletions(-) diff --git a/test/compilation.test.js b/test/compilation.test.js index 3d08a83..19ac8b2 100644 --- a/test/compilation.test.js +++ b/test/compilation.test.js @@ -253,6 +253,7 @@ test("compilation download distinguishes Head and Plan provenance without starti }); test("compilation download adopts the current directory without starting work", async (context) => { + if (process.platform === "win32") return context.skip(); const cwd = remoteDirectory(context); writeFileSync(path.join(cwd, "product-notes.md"), "Design notes\n"); const retainedState = readFileSync( @@ -482,7 +483,12 @@ test("compilation syntax and output preflight fail before network access", async { cwd, fetchFunction: inaccessible }, ); assertHandledFailure(reservedRoot, "invalid_output_path", 2); - assert.equal(JSON.parse(reservedRoot.stderr).reason, "root_reserved_path"); + assert.equal( + JSON.parse(reservedRoot.stderr).reason, + process.platform === "win32" + ? "root_platform_unsupported" + : "root_reserved_path", + ); assert.equal(existsSync(path.join(cwd, ROOT_TRANSACTION_NAME)), false); const help = await invoke(["compilation", "download", "--help"], { diff --git a/test/plan-compile.test.js b/test/plan-compile.test.js index 19b529e..15be39d 100644 --- a/test/plan-compile.test.js +++ b/test/plan-compile.test.js @@ -304,6 +304,7 @@ test("plan compile --output starts a direct Compilation without Publication", as }); test("plan compile root output locks before push and releases after invalid analysis", async (context) => { + if (process.platform === "win32") return context.skip(); const cwd = localDirectory(context, PLAN_SOURCE); writeFileSync(path.join(cwd, "notes.md"), "design notes\n"); let compilations = 0; @@ -333,6 +334,7 @@ test("plan compile root output locks before push and releases after invalid anal }); test("plan compile root output materializes directly without Publication", async (context) => { + if (process.platform === "win32") return context.skip(); const cwd = localDirectory(context, PLAN_SOURCE, { api_url: "https://api.example.test", foundation_plan_etag: ETAG, diff --git a/test/root-output.test.js b/test/root-output.test.js index f6ac5ac..73c4995 100644 --- a/test/root-output.test.js +++ b/test/root-output.test.js @@ -34,7 +34,17 @@ const ARTIFACT_FILES = [ { path: "bin/setup", mode: 0o755, contents: "#!/bin/sh\n" }, ]; +test("refuses root adoption on Windows", () => { + assert.throws( + () => prepareRootOutput({ root: "ignored", platform: "win32" }), + (error) => + error instanceof RootOutputPathError && + error.reason === "root_platform_unsupported", + ); +}); + test("adopts an arbitrary non-Git root without traversing preserved interiors", (context) => { + if (process.platform === "win32") return context.skip(); const root = temporaryDirectory(context); mkdirSync(path.join(root, ".firstdraft")); writeFileSync( @@ -74,6 +84,7 @@ test("adopts an arbitrary non-Git root without traversing preserved interiors", }); test("preflight holds one root lock and releases it before any irreversible phase", (context) => { + if (process.platform === "win32") return context.skip(); const root = temporaryDirectory(context); writeFileSync(path.join(root, "notes.md"), "notes\n"); const first = prepareRootOutput({ root }); @@ -94,14 +105,7 @@ test("preflight holds one root lock and releases it before any irreversible phas }); test("rejects unsafe root shapes before mutation", (context) => { - const windowsRoot = temporaryDirectory(context); - assert.throws( - () => prepareRootOutput({ root: windowsRoot, platform: "win32" }), - (error) => - error instanceof RootOutputPathError && - error.reason === "root_platform_unsupported", - ); - + if (process.platform === "win32") return context.skip(); const linkedRoot = temporaryDirectory(context); writeFileSync(path.join(linkedRoot, "outside"), "outside\n"); symlinkSync("outside", path.join(linkedRoot, "linked")); @@ -124,6 +128,7 @@ test("rejects unsafe root shapes before mutation", (context) => { }); test("preserves a Git worktree and installs an exact prepared index", (context) => { + if (process.platform === "win32") return context.skip(); const root = temporaryDirectory(context); initializeGit(root); writeFileSync(path.join(root, ".gitignore"), ".env\n"); @@ -182,6 +187,7 @@ test("preserves a Git worktree and installs an exact prepared index", (context) }); test("preserves a linked Git worktree without relocating its Git file", (context) => { + if (process.platform === "win32") return context.skip(); const holder = temporaryDirectory(context); initializeGit(holder); writeFileSync(path.join(holder, "README.md"), "Design README\n"); @@ -209,6 +215,7 @@ test("preserves a linked Git worktree without relocating its Git file", (context }); test("refuses enclosing worktrees, dirty indexes, and submodule metadata", (context) => { + if (process.platform === "win32") return context.skip(); const enclosing = temporaryDirectory(context); initializeGit(enclosing); mkdirSync(path.join(enclosing, "child")); @@ -245,6 +252,7 @@ test("refuses enclosing worktrees, dirty indexes, and submodule metadata", (cont }); test("detects pre-move changes and rolls back a failed rename exactly", (context) => { + if (process.platform === "win32") return context.skip(); const changedRoot = temporaryDirectory(context); writeFileSync(path.join(changedRoot, "notes.md"), "notes\n"); const changed = prepareRootOutput({ root: changedRoot }); @@ -296,6 +304,7 @@ test("detects pre-move changes and rolls back a failed rename exactly", (context }); test("preserves a Git index changed before installation", (context) => { + if (process.platform === "win32") return context.skip(); const root = temporaryDirectory(context); initializeGit(root); writeFileSync(path.join(root, "README.md"), "Design README\n"); @@ -338,6 +347,7 @@ test("preserves a Git index changed before installation", (context) => { }); test("rolls back an unexpected post-install verification failure", (context) => { + if (process.platform === "win32") return context.skip(); const root = temporaryDirectory(context); initializeGit(root); writeFileSync(path.join(root, "README.md"), "Design README\n"); @@ -384,6 +394,7 @@ test("rolls back an unexpected post-install verification failure", (context) => }); test("refuses artifact collisions with reserved root names", (context) => { + if (process.platform === "win32") return context.skip(); const root = temporaryDirectory(context); writeFileSync(path.join(root, "notes.md"), "notes\n"); const target = prepareRootOutput({ root }); @@ -406,6 +417,7 @@ test("refuses artifact collisions with reserved root names", (context) => { }); test("retains a versioned journal only when rollback cannot complete", (context) => { + if (process.platform === "win32") return context.skip(); const root = temporaryDirectory(context); writeFileSync(path.join(root, "one"), "one\n"); writeFileSync(path.join(root, "two"), "two\n");