From 65d37cd1be49ddb68fa8912d46c5f8b54598ff87 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 30 Sep 2026 02:35:14 +0000 Subject: [PATCH] Define lowered-HFIR ABI v1 and a multi-backend conformance suite. Phase 1 records the contract and compares the interpreter, bytecode VM, Go, and JavaScript on a pure core plus capability denial. Production compilation stays on the AST. Co-authored-by: William Elias --- change_log.md | 1 + docs/hfir_execution_status.md | 6 + .../2026-09-30_lowered_hfir_abi_phase1.md | 33 +++ docs/reference/lowered_hfir_abi_v1.md | 138 ++++++++++ improvements.md | 13 +- internal/hfir/abi.go | 14 + internal/hfir/abi_v1_test.go | 119 +++++++++ internal/hfir/verifier.go | 21 +- tests/conformance/abi_v1/01_arith_if.howl | 15 ++ tests/conformance/abi_v1/02_map_keys.howl | 5 + .../abi_v1/03_map_get_absence.howl | 6 + .../abi_v1/04_map_keys_type_error.howl | 4 + .../conformance/abi_v1/05_env_capability.howl | 3 + tests/conformance/lowered_hfir_abi_v1.json | 74 ++++++ tools/difftest/conformance_test.go | 247 ++++++++++++++++++ tools/difftest/difftest.go | 219 +++++++++++----- 16 files changed, 836 insertions(+), 82 deletions(-) create mode 100644 docs/journals/2026-09-30_lowered_hfir_abi_phase1.md create mode 100644 docs/reference/lowered_hfir_abi_v1.md create mode 100644 internal/hfir/abi.go create mode 100644 internal/hfir/abi_v1_test.go create mode 100644 tests/conformance/abi_v1/01_arith_if.howl create mode 100644 tests/conformance/abi_v1/02_map_keys.howl create mode 100644 tests/conformance/abi_v1/03_map_get_absence.howl create mode 100644 tests/conformance/abi_v1/04_map_keys_type_error.howl create mode 100644 tests/conformance/abi_v1/05_env_capability.howl create mode 100644 tests/conformance/lowered_hfir_abi_v1.json create mode 100644 tools/difftest/conformance_test.go diff --git a/change_log.md b/change_log.md index cfd1bb3..fe5fb57 100644 --- a/change_log.md +++ b/change_log.md @@ -7,6 +7,7 @@ * Locked the pure-versus-store capability split in docs and tests. `map_keys` grants nothing and runs under an empty grant. `store_keys` stays `database`. A `file://` store additionally requires `filesystem`. No opcode grant changed. Journal: `docs/journals/2026-09-30_capability_surface_honesty.md`. ### Added +* Lowered-HFIR ABI v1 (`docs/reference/lowered_hfir_abi_v1.md`) and a conformance suite (`tests/conformance/lowered_hfir_abi_v1.json`) run by `tools/difftest` across the interpreter, the bytecode VM, Go, and JavaScript. The production compiler is still the AST bytecode path. Journal: `docs/journals/2026-09-30_lowered_hfir_abi_phase1.md`. * `html_escape` and `attr_escape`, with `HTML_ESCAPE` and `ATTR_ESCAPE`. Both encode `&`, `<`, `>`, `"`, and `'` the same way (`&`, `<`, `>`, `"`, `'`) and grant no capability. `attr_escape` is the attribute-context diff --git a/docs/hfir_execution_status.md b/docs/hfir_execution_status.md index 27b2025..bf53dcd 100644 --- a/docs/hfir_execution_status.md +++ b/docs/hfir_execution_status.md @@ -77,6 +77,12 @@ The existing HowlBoard backend compatibility suite passes against the baseline H Improvement #88 has a real but deliberately bounded execution destination: model-authored graphs that meet the Phase-1 schema can be verified and lowered directly to a deterministic artifact, while unsupported nodes fail closed. Before a broad adapter can replace `.howl`, HFIR still needs explicit semantic forms for functions, structured error recovery, iteration, opaque effect operations, and stronger graph/control-flow verification. Work on #88 is meaningful as constrained Phase-1 adapter design, but not as a claim that arbitrary model-authored HFIR can execute today. +## Lowered ABI v1 (improvement #90, phase 1) + +`docs/reference/lowered_hfir_abi_v1.md` versions the contract the hosts must share (`lowered-hfir-abi/v1`). The conformance suite compares the interpreter, the bytecode VM, Go, and JavaScript on a pure core and on `env` denial. That comparison runs the production AST paths through `tools/difftest`. It does not switch `-compile-bc` over to `LowerToBytecode`. + +Wasm feasibility in this revision is the closed set `exec`, `spawn_agent`, and `http_server_start` (`HFIR_TARGET_INFEASIBLE`). Control edges are still unpopulated. Calls are still not executable HFIR. Phase 2 is the execution-path move. Journal: `docs/journals/2026-09-30_lowered_hfir_abi_phase1.md`. + ## Provenance limitation HFIR-to-bytecode compile diagnostics retain the source filename, line, and column available on their semantic node. Existing runtime `VMError` values identify function, instruction offset, and opcode only; bytecode instructions do not yet carry HFIR provenance. This Phase 1 does not redesign the artifact source-map format. diff --git a/docs/journals/2026-09-30_lowered_hfir_abi_phase1.md b/docs/journals/2026-09-30_lowered_hfir_abi_phase1.md new file mode 100644 index 0000000..66fcbc8 --- /dev/null +++ b/docs/journals/2026-09-30_lowered_hfir_abi_phase1.md @@ -0,0 +1,33 @@ +# Lowered HFIR ABI, phase 1 + +## Why this slice + +Improvement #90 asks for one lowered contract so the interpreter, the bytecode VM, Go, JavaScript, and eventually Wasm mean the same thing. The production compiler does not consume HFIR. `runHFIRGate` verifies a graph, then `-compile-bc` calls `bytecode.CompileToBytecode` on the AST. `-compile-hfir-bc` is still the experimental lowerer. `LowerAST` still leaves `ControlEdges` empty. + +Shipping "HFIR owns execution" in this change would be a false Done. Phase 1 publishes the contract and proves the hosts that already run the core agree on it. + +## What landed + +`docs/reference/lowered_hfir_abi_v1.md` is version `lowered-hfir-abi/v1` (`hfir.LoweredABIV1`). It states typed values, the call shape the hosts already share, the absence of a linear-memory import, error classes, pure effects versus `env`, and the Wasm feasibility set. + +The Wasm set is exactly `exec`, `spawn_agent`, and `http_server_start`, rejected with `HFIR_TARGET_INFEASIBLE`. `isFeasible` reads `hfir.WasmInfeasibleKinds`. Other targets stay permissive. No Wasm opcode was added. + +`tests/conformance/lowered_hfir_abi_v1.json` is the suite. `tools/difftest` already compared bytecode, the interpreter, and Go, and it already knew how to run JavaScript for a `web_app`. The suite extends that harness. A `cli_app` body is rewritten to `web_app` only for the JavaScript host. Empty grants are explicit. Passing cases match normalized stdout. Rejections match one error class, exit nonzero, and print nothing. + +The core cases are exact integer arithmetic and `if`, `map_keys` including the #109 UTF-8 byte-order fixtures, `map_get` absence (#103), a dynamic `map_keys` `TYPE_ERROR` (#108), and `env` denial plus `env` granted. `map_keys` and `map_get` run with an empty grant (#107). A seeded property check generates eight arithmetic expressions and requires the same integer on every host. `defun`, `call`, and `while` stay in the existing `tests/parity` corpus for the AST hosts; `LowerToBytecode` still rejects them with `HFIR_BYTECODE_UNSUPPORTED`. + +## Phase 2 + +One lowered graph is the source for every host. + +* `-compile-bc` still compiles the AST. +* Calls and `while` are not executable HFIR. +* Control edges are still empty, so the graph is not SSA. +* Generated Go reads `env` with `os.Getenv` and does not deny a missing grant. JavaScript does not mediate `env`. The denial case therefore lists only the interpreter and the bytecode VM. +* Feasibility is still a Wasm-only set of three host effects. +* Non-exact `/` is not one rule. +* #73 and #84 stay behind this ABI. This slice does not grow them. + +## What this does not do + +No new opcode. No language surface. No module linker, HFIR module graph, or VM module opcode. No Factory or supervisor work. #102–#109 are not reopened. Wasm SSA collections are not started. diff --git a/docs/reference/lowered_hfir_abi_v1.md b/docs/reference/lowered_hfir_abi_v1.md new file mode 100644 index 0000000..73c3c25 --- /dev/null +++ b/docs/reference/lowered_hfir_abi_v1.md @@ -0,0 +1,138 @@ +# Lowered HFIR backend ABI v1 + +Version: `lowered-hfir-abi/v1` + +This is the contract a backend must meet for the deterministic core, and the shape a later lowering has to grow into. It is not a claim that HFIR owns execution. Production compilation is still the checked AST: `runHFIRGate`, then `bytecode.CompileToBytecode` (`-compile-bc`). `-compile-hfir-bc` remains the experimental research lowerer. Go, JavaScript, and the interpreter still consume the AST. + +The executable suite is `tests/conformance/lowered_hfir_abi_v1.json`, run by `tools/difftest`. The constant `hfir.LoweredABIV1` must match the manifest's `abi` field. + +## In scope for v1 + +The suite runs one fixture on every applicable host and requires the same observable outcome. + +| Outcome | What must match | +| --- | --- | +| Pass | Normalized stdout, stderr, and process exit code | +| Rejection | One error class, a nonzero exit, and empty stdout | + +Rejection exit codes do not have to be the same number. The bytecode VM exits 1. A generated Go panic exits 2. The class is the contract. + +Hosts for a `cli_app` body: + +| Host | How the suite reaches it | +| --- | --- | +| Bytecode VM | `-compile-bc` then `-run-bc`. Canonical result. | +| Interpreter | `-run` | +| Go | Default `cli_app` backend, then `go build` | +| JavaScript | The same body with the root rewritten to `web_app`, then `node`, when `node` is on `PATH` | + +A missing `node` is `BACKEND_UNSUPPORTED` for that host only. A present `node` that disagrees is a failure. + +Wasm is not an execution host in v1. Its feasibility rule is the rejection set below. This revision does not add Wasm opcodes, collections, or host imports. + +### Values + +| Value | v1 rule | +| --- | --- | +| int | Signed integer. Printed with no decimal point when the value is a small exact integer. | +| float | IEEE-754 binary64. The v1 suite does not use inexact floats. | +| string | Unicode text compared and printed as UTF-8. | +| bool | `true` or `false`. | +| list | Ordered. An empty list is empty, not nil. | +| dict | String keys. Values stay as stored. | +| nil | The store-miss sentinel. Distinct from `""`. | + +### Pure operations + +These grant nothing. The suite runs them with an empty capability grant. + +* Arithmetic `+`, `-`, `*` on integers, and `/` only when the quotient is an exact integer (`(/ 20 4)` is `5`). +* Comparisons and `if`. +* `let` and `print`. `print` writes one line. Several arguments are separated by a single space. +* `dict`, `map_get`, `map_keys`, `list`, `list_len`, `str_join`, `is_nil`. + +`(map_get dict key)` on a missing key is `""`. `is_nil` of that result is false. A present value is returned as stored. This is the #103 absence rule. `map_get` grants nothing. + +`(map_keys dict)` returns a list of strings in UTF-8 byte order (Go `sort.Strings`, Go string `<`). An empty dict is an empty list, so its length is `0`. A value that is not a dict fails at runtime with `TYPE_ERROR` and the text `map_keys expected dict`. `map_keys` grants nothing. The order, including keys outside the Basic Multilingual Plane, is the #109 rule. The suite replays `tests/fixtures/map_keys_sort_bmp.howl` and `tests/fixtures/map_keys_sort_nonbmp.howl`. Wasm does not emit `map_keys`; #73 must use this byte order if it ever does. + +Integer `/` is not one rule yet. The interpreter truncates `int64`. The bytecode VM divides `float64`. JavaScript divides IEEE numbers. Go uses the operand types it emitted. Exact quotients agree. Non-exact quotients are outside v1. Division by zero is already in `tests/parity/12_error_div_zero.howl` and normalizes to `DIVISION_BY_ZERO`. + +### Calls + +A `defun` has a name, a parameter list, optional `type_hints`, and a body. `return` leaves the function. `(call name arg ...)` passes arguments by position. The existing harness already compares that shape on the interpreter, the bytecode VM, and Go: `tests/parity/06_control_flow.howl` (`TestParityCorpus`). + +The experimental lowerer does not emit `defun`, `call`, or `while`. `LowerToBytecode` fails those graphs with `HFIR_BYTECODE_UNSUPPORTED` and no `BCProgram`. v1 does not move call execution onto HFIR. + +### Memory and runtime imports + +v1 defines no linear memory and no Wasm import table. + +Observability imports `print`, `stderr`, and `exit` grant nothing. + +Host effects are named by `capability.ForConstruct`. The v1 suite uses one of them: `(env "KEY")` requires `environment`. An empty grant denies it before the variable is read. The grant `environment` returns the value. File, network, process, and database imports stay on that same table and are not given new opcodes here. + +### Errors + +Backends may use different JSON. The suite compares the class from `tools/difftest.NormalizeError`. + +| Class | When | +| --- | --- | +| `TYPE_ERROR` | A pure operation rejects a wrong receiver, including `map_keys` on a non-dict. | +| `CAPABILITY_DENIED` | A host effect runs without its grant. | +| `DIVISION_BY_ZERO` | Division by zero. | +| `HFIR_TARGET_INFEASIBLE` | A target cannot execute a construct. Diagnostic contract `v1`. | +| `HFIR_BYTECODE_UNSUPPORTED` | The experimental lowerer is given a node outside its executable subset. No partial program. | + +`map_keys` of a list literal is a checker diagnostic (`map_keys target must be dict, got list`) and never reaches the runtime. The runtime class is locked with a dynamically typed receiver, the same shape as `internal/vm/collection_type_test.go`. + +### Effects and capability boundaries + +Pure operations declare no capability. `map_keys` and `map_get` stay pure (#107). `store_keys` stays `database`, and a `file://` store also requires `filesystem`. v1 does not change that split. + +`env` is the negative capability case. The suite binds it with `let` and prints the binding. With no grant, the interpreter and the bytecode VM both reject with `CAPABILITY_DENIED`, exit nonzero, write no stdout, and do not include the secret value. With the `environment` grant, those two hosts and the Go backend print the value. The Go backend emits `os.Getenv` for that `let` binding. + +Generated Go does not consult the capability grant. Generated JavaScript has no `env` form. Those hosts are not conformance targets for the denial case. Putting them on that case is allowed only once they reject with `CAPABILITY_DENIED` and do not reveal the value. + +### Feasibility + +`isFeasible` rejects a closed set for target `wasm`, and only that target: + +* `exec` +* `spawn_agent` +* `http_server_start` + +The diagnostic is `HFIR_TARGET_INFEASIBLE`. The same kinds are not rejected for `bytecode`, `interpreter`, `go`, `javascript`, or the empty `-validate` target. `hfir.WasmInfeasibleKinds` is that set. Adding a kind, or rejecting a v1 pure kind such as `map_keys` or `print`, changes this contract. + +Bytecode construct support stays on `hfir.VerifyConstructs` over the AST (`internal/construct`), which also emits `HFIR_TARGET_INFEASIBLE`. That scan is unchanged. + +Other targets do not yet share one feasibility table. A passing verifier result for `go` or `javascript` does not mean the effect is implemented there. + +### CFG and SSA + +A later lowering that owns meaning has to be a typed CFG in SSA: + +* Blocks end in a jump, a conditional branch, or a return. +* Each value is assigned once. +* Data edges name operand roles (`key`, `value`, `body`, and so on). +* Control edges connect blocks. +* Node kinds are constructs, not user binding names. + +`hfir.LowerAST` is not that form. It fills data edges for the semantic subset and leaves `ControlEdges` empty on every node, including the v1 arithmetic fixture. Kinds outside `lowerSemanticList` still come from the list head, so a user name can appear as a kind. v1 records that fact. It does not pretend the graph is SSA. + +## Deferred (Phase 2) + +Phase 2 is one lowered graph consumed by every host, with identical outcomes or the same feasibility rejection. + +* Production `-compile-bc` still compiles the AST. Flipping that path is Phase 2. +* `defun`, `call`, and `while` become executable HFIR, or every host rejects them with one code. Today the hosts run them and the experimental lowerer rejects them. +* `ControlEdges` are populated and the graph is SSA. +* Go and JavaScript mediate capabilities the way the interpreter and the bytecode VM do. +* One feasibility table covers every target, not only the three Wasm host effects. +* Non-exact integer division picks one rule. +* Wasm collections (#73) and `for` / `match` / `try_let` / `spawn` SSA lowering (#84) target this ABI. They are not part of v1, and this revision does not grow them. +* No bytecode module linker, no HFIR module graph, no VM module opcode (#106). +* No new Frame opcode and no new capability. + +## What the suite does not prove + +Agreement among the AST backends is not proof that HFIR is the source of that agreement. `internal/vm/hfir_equivalence_test.go` is separate evidence that the experimental lowerer matches the bytecode VM on the subset it already emits, including `map_keys` and a granted `env`. That test is not the production compiler. diff --git a/improvements.md b/improvements.md index babaf12..3f22870 100644 --- a/improvements.md +++ b/improvements.md @@ -28,7 +28,7 @@ Pending rows are ranked by a diminishing-returns score: - **Decay:** geometric halving per already-shipped item in the same theme (1.0 → 0.5 → 0.25 …). - **Effort (1–8):** roughly log-scale; 1 = minutes, 8 = weeks. -2026-09-29: the HowlFutureWorks seat poll is recorded in `docs/journals/2026-09-29_team_howlframe_backlog.md`. #102, #103, #104, #105, and #108 landed the same day. #106's design spike is Done (2026-09-30) and records do-not-build-yet; see `docs/journals/2026-09-30_bytecode_modules_spike.md`. #107 is Done (2026-09-30): `map_keys` stays pure and `store_keys` stays `database`, plus `filesystem` for `file://`. #109 is the Assurance sort-parity track. The journal's seat sequence is the agreed build order; this table still sorts by the formula above. +2026-09-29: the HowlFutureWorks seat poll is recorded in `docs/journals/2026-09-29_team_howlframe_backlog.md`. #102, #103, #104, #105, and #108 landed the same day. #106's design spike is Done (2026-09-30) and records do-not-build-yet; see `docs/journals/2026-09-30_bytecode_modules_spike.md`. #107 is Done (2026-09-30): `map_keys` stays pure and `store_keys` stays `database`, plus `filesystem` for `file://`. #109 is the Assurance sort-parity track. The journal's seat sequence is the agreed build order; this table still sorts by the formula above. #90 Phase 1 (2026-09-30) publishes the lowered-HFIR ABI and a core conformance suite. Production execution is still AST → bytecode. Journal: `docs/journals/2026-09-30_lowered_hfir_abi_phase1.md`. | # | Improvement | Status | Score (V×D÷E) | Claude model | Gemini model | OpenAI model | ROI rationale | | --- | --- | --- | --- | --- | --- | --- | --- | @@ -45,7 +45,7 @@ Pending rows are ranked by a diminishing-returns score: | 106 | [Bytecode modules: design spike only](#106-bytecode-modules-design-spike-only) | Done (2026-09-30) | 1.50 (6×1÷4) | Opus 5 | Gemini 3.1 Pro (High) | gpt-5.6-sol | Do not build a bytecode-tier linker yet. Flat multi-file programs already run on `-compile-bc` / `-run-bc` through #95. A half linker and an HFIR module patch are refused. Journal: `docs/journals/2026-09-30_bytecode_modules_spike.md`. | | 108 | [Fail-closed TYPE_ERROR on remaining dict and list ops](#108-fail-closed-type_error-on-remaining-dict-and-list-ops) | Done (2026-09-29) | 1.50 (6×1÷4) | Sonnet 5 | Gemini 3.1 Pro (High) | gpt-5.6-sol | `append`, `map_set`, `map_delete`, `map_get`, `list_get`, and `list_len` reject a wrong receiver with `TYPE_ERROR` on the VM, Go, and JS. A missing `map_get` key is still `""`. No new capability. Journal: `docs/journals/2026-09-29_type_error_collections.md`. | | 88 | [Define a provider-neutral HFIR model-adapter protocol](#88-define-a-provider-neutral-hfir-model-adapter-protocol) | Pending | 1.5 (6×1÷4) | Opus 5 | — | gpt-5.6-terra | Mask plans are provider-neutral but no adapter produces or repairs a verified semantic program artifact. A schema, constrained-decoding boundary, and delta protocol prevents lock-in and reduces regeneration cost. | -| 90 | [Define the lowered-HFIR backend ABI and conformance suite](#90-define-the-lowered-hfir-backend-abi-and-conformance-suite) | Pending | 1.5 (6×1÷4) | Opus 5 | — | gpt-5.6-sol | Go, JS, interpreter, bytecode, and Wasm currently implement overlapping semantics independently. A shared lowered contract plus deterministic conformance cases makes backend diversity trustworthy. | +| 90 | [Define the lowered-HFIR backend ABI and conformance suite](#90-define-the-lowered-hfir-backend-abi-and-conformance-suite) | Partial — Phase 1 (2026-09-30) | 1.5 (6×1÷4) | Opus 5 | — | gpt-5.6-sol | Phase 1 is the versioned ABI and the difftest conformance suite for a pure core plus capability denial. Production execution is still AST → bytecode. Wasm stays gated. Journal: `docs/journals/2026-09-30_lowered_hfir_abi_phase1.md`. | | 89 | [Add content-addressed HFIR storage and incremental compilation](#89-add-content-addressed-hfir-storage-and-incremental-compilation) | Pending | 1.4 (7×1÷5) | Opus 5 | — | gpt-5.6-sol | Stable graph identities permit small repairs, dependency-closure recompilation, reproducible artifacts, and bounded model context instead of whole-program regeneration. | | 95 | [Standalone Runtime Module Use/Export](#95-standalone-runtime-module-use-export) | Done (2026-08-08) | 1.33 (8×0.5÷3) | Sonnet 5 | Gemini 3.1 Pro (High) | — | Multi-file `cli_app` compilation now works through `-compile-bc`/`-run-bc`. `ast.ResolveModules`'s existing compile-time AST linker (improvement #93) already handled the standalone path; closed the nested-import gap with a fail-closed boundary and reclassified `use`/`export`/`module` as `CompileTimeOnly`. Journal: `docs/journals/2026-08-08_improvement_95_standalone_modules.md`. | | 109 | [Dict key sort parity across VM, Go, and JS, including non-BMP](#109-dict-key-sort-parity-across-vm-go-and-js-including-non-bmp) | Done (2026-09-30) | 1.33 (4×1÷3) | Sonnet 5 | Gemini 3.1 Pro (High) | gpt-5.6-sol | `map_keys` order is UTF-8 byte order on the VM, Go, and JS, including non-BMP keys. JavaScript no longer uses UTF-16 code-unit sort. #73 must reuse this order. Journal: `docs/journals/2026-09-30_dict_key_sort.md`. | @@ -788,13 +788,14 @@ As HowlFrame matures past transpilation into Go and JS, the ultimate objective i * **Required tests:** invalidation, hash determinism, corruption recovery, and reproducible-build integration. ### 90. Define the lowered-HFIR backend ABI and conformance suite +* **Status:** Partial — Phase 1 (2026-09-30). The contract is `docs/reference/lowered_hfir_abi_v1.md` (`lowered-hfir-abi/v1`). The suite is `tests/conformance/lowered_hfir_abi_v1.json`, executed by `tools/difftest` on the interpreter, the bytecode VM, Go, and JavaScript where that host applies. Wasm's v1 rule is `HFIR_TARGET_INFEASIBLE` for `exec`, `spawn_agent`, and `http_server_start` only. Production `-compile-bc` is still `runHFIRGate` and then `bytecode.CompileToBytecode` on the AST. `-compile-hfir-bc` stays the experimental lowerer. `ControlEdges` are still empty. `defun`, `call`, and `while` are not executable HFIR. Go and JavaScript do not mediate `env`. Phase 2 is one lowered graph for every host. Journal: `docs/journals/2026-09-30_lowered_hfir_abi_phase1.md`. * **Description:** Specify a target-independent lowered HFIR ABI for typed CFG/SSA, calls, memory/runtime imports, errors, effects, and capability boundaries. Migrate a deterministic core subset and compare interpreter, bytecode, Go, JS where applicable, and Wasm outcomes or explicit rejections. -* **Current reality (recorded 2026-08-07):** HFIR is not on the execution path at all today — it is a shadow verification pass, not the lowering source. `howlframe.go:107` calls `bytecode.CompileToBytecode(root)` directly on the **AST**, with `runHFIRGate` verifying a separately-lowered graph alongside it; no backend consumes HFIR. `hfir.LowerAST` (`internal/hfir/lowering.go`) is a near-1:1 AST mirror whose `Graph.ControlEdges` field is declared but never populated, and it derives `Kind` from the head symbol of every list, so HFIR node kinds include user binding and parameter names, not just constructs. Improvements #86/#87 are legitimately Done for what they scoped, but the blueprint's `HFIR -> lowered HFIR -> bytecode` path does not exist yet, and "HFIR owns semantic meaning" (`docs/standalone_runtime_blueprint.md`) is a goal rather than current behavior. Closing that gap is this item's real content. +* **Current reality (recorded 2026-09-30):** HFIR is still not the production execution source. `-compile-bc` verifies a lowered graph and then compiles the AST. The experimental `-compile-hfir-bc` path can emit bytecode for a semantic subset (`let`, `if`, `map_keys`, `env`, and the forms in `internal/hfir/bytecode.go`) and fails closed with `HFIR_BYTECODE_UNSUPPORTED` outside it, including `defun`, `call`, and `while`. `LowerAST` still does not populate `ControlEdges`. On 2026-08-07 that direct path did not exist; it exists now as research, not as the compiler. Go, JavaScript, and the interpreter still consume the AST. The v1 suite compares those hosts on a pure core and on capability denial. It does not flip the execution path. * **Why:** Each current backend owns overlapping semantics. More backend expansion without a contract will amplify drift. * **Dependencies:** #86 and #87; incorporates #78. #73 and #84 should target this ABI. -* **Acceptance criteria:** one lowering runs equivalently on each supported target; unsupported effects use the same feasibility contract. -* **Required tests:** differential and property-based deterministic fixtures plus negative capability tests. -* **Team backlog (2026-09-29):** stays Pending. Wasm expansion stays gated on this lowered ABI; do not grow a thin Wasm backend ahead of it. See `docs/journals/2026-09-29_team_howlframe_backlog.md`. +* **Acceptance criteria:** one lowering runs equivalently on each supported target; unsupported effects use the same feasibility contract. Phase 1 meets the second clause for the closed Wasm host-effect set and records identical outcomes for the core subset on the hosts that already implement it. The first clause, one lowering as the source for every host, is Phase 2. +* **Required tests:** differential and property-based deterministic fixtures plus negative capability tests. Phase 1 adds those on the existing difftest harness. `tests/parity` remains the broader AST corpus, including calls. +* **Team backlog (2026-09-29, updated 2026-09-30):** Phase 1 is in. Wasm expansion stays gated on Phase 2 of this ABI; do not grow a thin Wasm backend ahead of it. See `docs/journals/2026-09-29_team_howlframe_backlog.md` and `docs/journals/2026-09-30_lowered_hfir_abi_phase1.md`. ### 91. Add semantic patch deltas and bounded repair context * **Description:** Make autonomous repair operate on HFIR deltas addressed to stable IDs. Require preconditions, touched dependencies, expected invariants, and regression evidence, with compact context extracted from graph neighborhoods and diagnostics. diff --git a/internal/hfir/abi.go b/internal/hfir/abi.go new file mode 100644 index 0000000..13738f5 --- /dev/null +++ b/internal/hfir/abi.go @@ -0,0 +1,14 @@ +package hfir + +// LoweredABIV1 is the version of the lowered-HFIR backend contract. +// The prose contract is docs/reference/lowered_hfir_abi_v1.md. +const LoweredABIV1 = "lowered-hfir-abi/v1" + +// WasmInfeasibleKinds is the complete v1 host-effect rejection set for +// target "wasm". isFeasible rejects these kinds and no others. Growing +// Wasm collections or new host imports is outside this revision. +var WasmInfeasibleKinds = []string{ + "exec", + "spawn_agent", + "http_server_start", +} diff --git a/internal/hfir/abi_v1_test.go b/internal/hfir/abi_v1_test.go new file mode 100644 index 0000000..da1c439 --- /dev/null +++ b/internal/hfir/abi_v1_test.go @@ -0,0 +1,119 @@ +package hfir + +import ( + "os" + "path/filepath" + "testing" + + "github.com/howlcipher/howlframe/internal/checker" + "github.com/howlcipher/howlframe/internal/lexer" + "github.com/howlcipher/howlframe/internal/parser" +) + +func TestLoweredABIV1WasmRejectionSetIsClosed(t *testing.T) { + if LoweredABIV1 != "lowered-hfir-abi/v1" { + t.Fatalf("ABI version = %q", LoweredABIV1) + } + want := []string{"exec", "spawn_agent", "http_server_start"} + if len(WasmInfeasibleKinds) != len(want) { + t.Fatalf("wasm rejection set = %v, want %v", WasmInfeasibleKinds, want) + } + for i, kind := range want { + if WasmInfeasibleKinds[i] != kind { + t.Fatalf("wasm rejection set = %v, want %v", WasmInfeasibleKinds, want) + } + if isFeasible(kind, "wasm") { + t.Errorf("isFeasible(%q, wasm) = true, want false", kind) + } + } + + graph := NewGraph() + for _, kind := range WasmInfeasibleKinds { + graph.AddNode(&Node{Kind: kind}) + } + wasmDiags := NewVerifier(graph, "wasm").Verify() + var infeasible int + for _, diag := range wasmDiags { + if diag.Code == "HFIR_TARGET_INFEASIBLE" { + infeasible++ + if diag.ContractVersion != DiagnosticContractVersion { + t.Errorf("contract version = %q", diag.ContractVersion) + } + } + } + if infeasible != len(WasmInfeasibleKinds) { + t.Fatalf("wasm HFIR_TARGET_INFEASIBLE count = %d, diags = %#v", infeasible, wasmDiags) + } + + for _, target := range []string{"", "bytecode", "interpreter", "go", "javascript"} { + for _, diag := range NewVerifier(graph, target).Verify() { + if diag.Code == "HFIR_TARGET_INFEASIBLE" { + t.Errorf("target %q rejected a v1 wasm-only kind: %#v", target, diag) + } + } + } + + for _, kind := range []string{"map_keys", "map_get", "print", "binary", "if", "let"} { + for _, target := range []string{"wasm", "bytecode", "interpreter", "go", "javascript"} { + if !isFeasible(kind, target) { + t.Errorf("isFeasible(%q, %q) = false, want true", kind, target) + } + } + } +} + +func TestLoweredABIV1CoreFixtureHasNoControlEdges(t *testing.T) { + graph := lowerFixture(t, filepath.Join("..", "..", "tests", "conformance", "abi_v1", "01_arith_if.howl")) + if len(graph.Nodes) == 0 { + t.Fatal("arith fixture lowered no nodes") + } + for _, node := range graph.Nodes { + if len(node.ControlEdges) != 0 { + t.Fatalf("node %s kind %s has control edges %v; v1 LowerAST does not populate CFG edges", node.ID, node.Kind, node.ControlEdges) + } + } + program, diags := LowerToBytecode(graph) + if len(diags) != 0 { + t.Fatalf("experimental lowerer rejected the v1 arith fixture: %#v", diags) + } + if program == nil || len(program.Main) == 0 { + t.Fatal("experimental lowerer emitted no bytecode for the v1 arith fixture") + } +} + +func TestLoweredABIV1DefunCallAndWhileAreNotExecutable(t *testing.T) { + sources := []string{ + `(cli_app (defun add_values (a b) (type_hints (a int) (b int) (return int)) (return (+ a b))) (print 1))`, + `(cli_app (while false (print "no")))`, + } + for _, source := range sources { + root := parser.NewParser(lexer.NewLexer(source), "abi_deferred.howl").ParseExpression() + checker.Check(root) + graph, err := LowerAST(root, "abi_deferred.howl") + if err != nil { + t.Fatalf("LowerAST(%q) error = %v", source, err) + } + program, diags := LowerToBytecode(graph) + if program != nil { + t.Fatalf("LowerToBytecode(%q) returned a program; v1 must fail closed", source) + } + if len(diags) == 0 || diags[0].Code != BytecodeUnsupportedCode { + t.Fatalf("LowerToBytecode(%q) diags = %#v, want %s", source, diags, BytecodeUnsupportedCode) + } + } +} + +func lowerFixture(t *testing.T, path string) *Graph { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + root := parser.NewParser(lexer.NewLexer(string(data)), filepath.Base(path)).ParseExpression() + checker.Check(root) + graph, err := LowerAST(root, filepath.Base(path)) + if err != nil { + t.Fatalf("LowerAST: %v", err) + } + return graph +} diff --git a/internal/hfir/verifier.go b/internal/hfir/verifier.go index 3685f47..6bde567 100644 --- a/internal/hfir/verifier.go +++ b/internal/hfir/verifier.go @@ -114,18 +114,19 @@ func (v *Verifier) Verify() []Diagnostic { return v.Diagnostics } -// isFeasible has real rejection rules only for target == "wasm" today. Every -// other target identity (including ones introduced by howlframe.go's production -// wiring in improvement #87 Phase 2, e.g. "bytecode"/"interpreter"/"go"/ -// "javascript") is permissive by default - a passing result for those -// targets does not mean real per-target feasibility coverage exists yet. +// isFeasible has real rejection rules only for target == "wasm" today, and +// that set is WasmInfeasibleKinds (lowered-hfir-abi/v1). Every other target +// identity (including "bytecode", "interpreter", "go", and "javascript") is +// permissive. A passing result for those targets does not mean per-target +// feasibility coverage exists yet. Unsupported effects on those targets are +// not silently treated as feasible by this function; they are outside the +// v1 rejection set and stay a Phase 2 contract. func isFeasible(kind string, target string) bool { if target == "wasm" { - // Wasm backend doesn't support complex networking, process, or file IO natively without imports - // For the sake of the verifier, we flag them if needed, but we'll leave it permissive for now. - switch kind { - case "exec", "spawn_agent", "http_server_start": - return false + for _, rejected := range WasmInfeasibleKinds { + if kind == rejected { + return false + } } } return true diff --git a/tests/conformance/abi_v1/01_arith_if.howl b/tests/conformance/abi_v1/01_arith_if.howl new file mode 100644 index 0000000..d11f515 --- /dev/null +++ b/tests/conformance/abi_v1/01_arith_if.howl @@ -0,0 +1,15 @@ +(cli_app + (let (a 10) + (let (b 4) + (do + (print (+ a b)) + (print (- a b)) + (print (* a b)) + (print (/ 20 4)) + (if (> a b) + (print "greater") + (print "not-greater")) + (if (= a b) + (print "equal") + (print "unequal")) + (print (> a b)))))) diff --git a/tests/conformance/abi_v1/02_map_keys.howl b/tests/conformance/abi_v1/02_map_keys.howl new file mode 100644 index 0000000..674f2c3 --- /dev/null +++ b/tests/conformance/abi_v1/02_map_keys.howl @@ -0,0 +1,5 @@ +(cli_app + (let (counts (dict ("beta" "2") ("alpha" "1") ("gamma" "3"))) + (do + (print (str_join (map_keys counts) ",")) + (print (list_len (map_keys (dict))))))) diff --git a/tests/conformance/abi_v1/03_map_get_absence.howl b/tests/conformance/abi_v1/03_map_get_absence.howl new file mode 100644 index 0000000..c8af4dc --- /dev/null +++ b/tests/conformance/abi_v1/03_map_get_absence.howl @@ -0,0 +1,6 @@ +(cli_app + (let (d (dict ("present" "yes"))) + (do + (print (map_get d "present")) + (print (is_nil (map_get d "missing"))) + (print (str_join (list "x" (map_get d "missing") "y") "|"))))) diff --git a/tests/conformance/abi_v1/04_map_keys_type_error.howl b/tests/conformance/abi_v1/04_map_keys_type_error.howl new file mode 100644 index 0000000..8526874 --- /dev/null +++ b/tests/conformance/abi_v1/04_map_keys_type_error.howl @@ -0,0 +1,4 @@ +(cli_app + (let (row (dict ("s" "ada") ("xs" (list "a")) ("d" (dict ("k" "v"))))) + (let (s (map_get row "s")) + (print (map_keys s))))) diff --git a/tests/conformance/abi_v1/05_env_capability.howl b/tests/conformance/abi_v1/05_env_capability.howl new file mode 100644 index 0000000..41de839 --- /dev/null +++ b/tests/conformance/abi_v1/05_env_capability.howl @@ -0,0 +1,3 @@ +(cli_app + (let (value (env "HOWLFRAME_ABI_SECRET")) + (print value))) diff --git a/tests/conformance/lowered_hfir_abi_v1.json b/tests/conformance/lowered_hfir_abi_v1.json new file mode 100644 index 0000000..a080f27 --- /dev/null +++ b/tests/conformance/lowered_hfir_abi_v1.json @@ -0,0 +1,74 @@ +{ + "abi": "lowered-hfir-abi/v1", + "cases": [ + { + "name": "arith_if", + "fixture": "tests/conformance/abi_v1/01_arith_if.howl", + "deny_all": true, + "targets": ["bytecode", "interpreter", "go", "javascript"], + "javascript_root": "web_app", + "expect": "PASS", + "stdout": "14\n6\n40\n5\ngreater\nunequal\ntrue" + }, + { + "name": "map_keys", + "fixture": "tests/conformance/abi_v1/02_map_keys.howl", + "deny_all": true, + "targets": ["bytecode", "interpreter", "go", "javascript"], + "javascript_root": "web_app", + "expect": "PASS", + "stdout": "alpha,beta,gamma\n0" + }, + { + "name": "map_get_absence", + "fixture": "tests/conformance/abi_v1/03_map_get_absence.howl", + "deny_all": true, + "targets": ["bytecode", "interpreter", "go", "javascript"], + "javascript_root": "web_app", + "expect": "PASS", + "stdout": "yes\nfalse\nx||y" + }, + { + "name": "map_keys_sort_bmp", + "fixture": "tests/fixtures/map_keys_sort_bmp.howl", + "deny_all": true, + "targets": ["bytecode", "interpreter", "go", "javascript"], + "javascript_root": "web_app", + "expect": "PASS", + "stdout_keys": ["gamma", "beta", "alpha", "é"] + }, + { + "name": "map_keys_sort_nonbmp", + "fixture": "tests/fixtures/map_keys_sort_nonbmp.howl", + "deny_all": true, + "targets": ["bytecode", "interpreter", "go", "javascript"], + "javascript_root": "web_app", + "expect": "PASS", + "stdout_keys": ["\uD83D\uDE00", "a", "\uF000", "\uD83C\uDC00", "é"] + }, + { + "name": "map_keys_type_error", + "fixture": "tests/conformance/abi_v1/04_map_keys_type_error.howl", + "deny_all": true, + "targets": ["bytecode", "interpreter", "go", "javascript"], + "javascript_root": "web_app", + "expect": "TYPE_ERROR" + }, + { + "name": "env_denied", + "fixture": "tests/conformance/abi_v1/05_env_capability.howl", + "deny_all": true, + "targets": ["bytecode", "interpreter"], + "expect": "CAPABILITY_DENIED", + "forbid": "phase1-token" + }, + { + "name": "env_granted", + "fixture": "tests/conformance/abi_v1/05_env_capability.howl", + "allow_caps": "environment", + "targets": ["bytecode", "interpreter", "go"], + "expect": "PASS", + "stdout": "phase1-token" + } + ] +} diff --git a/tools/difftest/conformance_test.go b/tools/difftest/conformance_test.go new file mode 100644 index 0000000..9f29829 --- /dev/null +++ b/tools/difftest/conformance_test.go @@ -0,0 +1,247 @@ +package main + +import ( + "encoding/json" + "math/rand" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "testing" + + "github.com/howlcipher/howlframe/internal/hfir" +) + +type conformanceFile struct { + ABI string `json:"abi"` + Cases []conformanceCase `json:"cases"` +} + +type conformanceCase struct { + Name string `json:"name"` + Fixture string `json:"fixture"` + AllowCaps *string `json:"allow_caps"` + DenyAll bool `json:"deny_all"` + Targets []Target `json:"targets"` + JavaScriptRoot string `json:"javascript_root"` + Expect string `json:"expect"` + Stdout string `json:"stdout"` + StdoutKeys []string `json:"stdout_keys"` + Forbid string `json:"forbid"` +} + +func loadConformance(t *testing.T) (string, conformanceFile) { + t.Helper() + root, err := findRepoRoot() + if err != nil { + t.Fatalf("repo root: %v", err) + } + path := filepath.Join(root, "tests", "conformance", "lowered_hfir_abi_v1.json") + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read conformance manifest: %v", err) + } + var file conformanceFile + if err := json.Unmarshal(data, &file); err != nil { + t.Fatalf("parse conformance manifest: %v", err) + } + return root, file +} + +func (c conformanceCase) runOptions() RunOptions { + return RunOptions{ + AllowCaps: c.AllowCaps, + DenyAll: c.DenyAll, + JSRoot: c.JavaScriptRoot, + } +} + +func (c conformanceCase) wantStdout() string { + if len(c.StdoutKeys) == 0 { + return c.Stdout + } + keys := append([]string(nil), c.StdoutKeys...) + sort.Strings(keys) + return strings.Join(keys, ",") +} + +// TestLoweredHFIRABIConformance runs the v1 suite through the existing +// differential harness. Passing cases must share stdout. Rejections must +// share one error class and must not print a forbidden value. +func TestLoweredHFIRABIConformance(t *testing.T) { + t.Setenv("HOWLFRAME_ABI_SECRET", "phase1-token") + root, file := loadConformance(t) + if file.ABI != hfir.LoweredABIV1 { + t.Fatalf("manifest abi = %q, want %q", file.ABI, hfir.LoweredABIV1) + } + if len(file.Cases) == 0 { + t.Fatal("conformance manifest has no cases") + } + + for _, tc := range file.Cases { + tc := tc + t.Run(tc.Name, func(t *testing.T) { + if tc.DenyAll && tc.AllowCaps != nil { + t.Fatal("case sets both deny_all and allow_caps") + } + if tc.Name == "env_denied" { + for _, tgt := range tc.Targets { + if tgt == TargetGo || tgt == TargetJavaScript { + t.Fatalf("env_denied includes %s; those hosts do not mediate capabilities yet", tgt) + } + } + } + fixture := filepath.Join(root, tc.Fixture) + report, err := VerifyParityWithOptions(fixture, tc.Targets, tc.runOptions()) + if err != nil { + t.Fatalf("VerifyParityWithOptions: %v", err) + } + if report.OverallStatus != StatusPass { + t.Fatalf("parity: %s", strings.Join(report.Discrepancies, "\n")) + } + + wantOut := tc.wantStdout() + if tc.Expect == "PASS" { + if NormalizeOutput(report.CanonicalResult.Stdout) != NormalizeOutput(wantOut) { + t.Fatalf("canonical stdout = %q, want %q", NormalizeOutput(report.CanonicalResult.Stdout), NormalizeOutput(wantOut)) + } + if report.CanonicalResult.Status != StatusPass { + t.Fatalf("canonical status = %s, want PASS (%s)", report.CanonicalResult.Status, report.CanonicalResult.ErrorMessage) + } + } else { + if report.CanonicalResult.Status != StatusRuntimeFailure { + t.Fatalf("canonical status = %s, want runtime failure (%s)", report.CanonicalResult.Status, report.CanonicalResult.ErrorMessage) + } + if report.CanonicalResult.ErrorClass != tc.Expect { + t.Fatalf("canonical error class = %q, want %q\n%s", report.CanonicalResult.ErrorClass, tc.Expect, report.CanonicalResult.ErrorMessage) + } + } + + for _, tgt := range tc.Targets { + res := report.TargetResults[tgt] + if tgt == TargetJavaScript && res.Status == StatusBackendUnsupported && strings.Contains(res.ErrorMessage, "node runtime not available") { + t.Log("javascript skipped: node runtime not available") + continue + } + if tc.Expect != "PASS" { + if res.ErrorClass != tc.Expect { + t.Errorf("%s error class = %q, want %q\n%s", tgt, res.ErrorClass, tc.Expect, res.ErrorMessage) + } + if res.ExitCode == 0 { + t.Errorf("%s exit = 0, want rejection", tgt) + } + if strings.TrimSpace(res.Stdout) != "" { + t.Errorf("%s stdout = %q, want empty on rejection", tgt, res.Stdout) + } + } + if tc.Forbid != "" && strings.Contains(res.Stdout+res.Stderr, tc.Forbid) { + t.Errorf("%s leaked %q", tgt, tc.Forbid) + } + } + }) + } +} + +func TestCompareToCanonicalDetectsDrift(t *testing.T) { + canonical := ExecutionResult{Target: TargetBytecode, Status: StatusPass, Stdout: "14\n", ExitCode: 0} + drifted := canonical + drifted.Stdout = "15\n" + if diffs := compareToCanonical(canonical, drifted, TargetGo); len(diffs) == 0 { + t.Fatal("stdout drift was not detected") + } + + denied := ExecutionResult{Status: StatusRuntimeFailure, ErrorClass: "CAPABILITY_DENIED", ExitCode: 1} + other := ExecutionResult{Status: StatusRuntimeFailure, ErrorClass: "TYPE_ERROR", ExitCode: 1} + if diffs := compareToCanonical(denied, other, TargetInterpreter); len(diffs) == 0 { + t.Fatal("error-class drift was not detected") + } + + same := denied + if diffs := compareToCanonical(denied, same, TargetInterpreter); len(diffs) != 0 { + t.Fatalf("matching rejection reported drift: %v", diffs) + } +} + +func TestABIPropertyArithmetic(t *testing.T) { + rng := rand.New(rand.NewSource(90)) + targets := []Target{TargetBytecode, TargetInterpreter, TargetGo, TargetJavaScript} + opts := RunOptions{DenyAll: true, JSRoot: "web_app"} + var firstStdout string + var firstSource string + + for i := 0; i < 8; i++ { + node := genArith(rng, 2) + expr := formatArith(node) + want := strconv.Itoa(evalArith(node)) + dir := t.TempDir() + path := filepath.Join(dir, "arith.howl") + source := "(cli_app (print " + expr + "))\n" + if err := os.WriteFile(path, []byte(source), 0o644); err != nil { + t.Fatalf("write fixture: %v", err) + } + report, err := VerifyParityWithOptions(path, targets, opts) + if err != nil { + t.Fatalf("case %d %s: %v", i, expr, err) + } + if report.OverallStatus != StatusPass { + t.Fatalf("case %d %s parity:\n%s", i, expr, strings.Join(report.Discrepancies, "\n")) + } + if got := NormalizeOutput(report.CanonicalResult.Stdout); got != want { + t.Fatalf("case %d %s stdout = %q, want %q", i, expr, got, want) + } + if i == 0 { + firstStdout = report.CanonicalResult.Stdout + firstSource = source + again, err := VerifyParityWithOptions(path, []Target{TargetBytecode}, opts) + if err != nil { + t.Fatal(err) + } + if again.CanonicalResult.Stdout != firstStdout { + t.Fatalf("repeat of %s stdout = %q, want %q", firstSource, again.CanonicalResult.Stdout, firstStdout) + } + } + } +} + +type arithExpr struct { + op string + value int + left *arithExpr + right *arithExpr +} + +func genArith(rng *rand.Rand, depth int) *arithExpr { + if depth == 0 || rng.Intn(4) == 0 { + // Literals stay non-negative. The lexer has no negative integer token. + return &arithExpr{value: rng.Intn(13)} + } + ops := []string{"+", "-", "*"} + return &arithExpr{ + op: ops[rng.Intn(len(ops))], + left: genArith(rng, depth-1), + right: genArith(rng, depth-1), + } +} + +func evalArith(node *arithExpr) int { + switch node.op { + case "": + return node.value + case "+": + return evalArith(node.left) + evalArith(node.right) + case "-": + return evalArith(node.left) - evalArith(node.right) + case "*": + return evalArith(node.left) * evalArith(node.right) + default: + panic("unknown op " + node.op) + } +} + +func formatArith(node *arithExpr) string { + if node.op == "" { + return strconv.Itoa(node.value) + } + return "(" + node.op + " " + formatArith(node.left) + " " + formatArith(node.right) + ")" +} diff --git a/tools/difftest/difftest.go b/tools/difftest/difftest.go index 2dc4e18..2fecdd8 100644 --- a/tools/difftest/difftest.go +++ b/tools/difftest/difftest.go @@ -148,7 +148,57 @@ func NormalizeError(errStr string) string { } } +const allKnownCaps = "network,filesystem,process,environment,database" + +// RunOptions controls one differential execution. +// A nil AllowCaps keeps the historical grant of every known capability. +// DenyAll is an explicit empty grant. JSRoot, when set, rewrites the first +// cli_app root to that symbol before the JavaScript backend runs, so one +// cli_app fixture can be compared on the web_app host. +type RunOptions struct { + CLIArgs []string + Input string + AllowCaps *string + DenyAll bool + JSRoot string +} + +func (o RunOptions) capFlags() []string { + if o.DenyAll { + return nil + } + caps := allKnownCaps + if o.AllowCaps != nil { + caps = *o.AllowCaps + } + if caps == "" { + return nil + } + return []string{"-allow-caps", caps} +} + +// isStructuredRuntimeRejection reports runtime failures whose text is a +// shared error class. Checker diagnostics also use a "reason" field, so +// the interpreter path must not treat every JSON reason as a compile failure. +func isStructuredRuntimeRejection(out string) bool { + s := strings.ToLower(out) + if strings.Contains(s, "type_error:") || strings.Contains(s, `"code":"type_error"`) { + return true + } + if strings.Contains(s, "capability denied") || strings.Contains(s, `"code":"capability_denied"`) { + return true + } + if strings.Contains(s, "division by zero") || strings.Contains(s, "divide by zero") { + return true + } + return false +} + func ExecuteBytecode(filePath string, cliArgs []string, input string) ExecutionResult { + return executeBytecode(filePath, RunOptions{CLIArgs: cliArgs, Input: input}) +} + +func executeBytecode(filePath string, opts RunOptions) ExecutionResult { compiler, err := getCompiler() if err != nil { return ExecutionResult{Target: TargetBytecode, ExitCode: 1, ErrorMessage: err.Error(), Status: StatusCompileFailure} @@ -178,10 +228,12 @@ func ExecuteBytecode(filePath string, cliArgs []string, input string) ExecutionR } } - runArgs := append([]string{"-run-bc", "-allow-caps", "network,filesystem,process,environment,database", bcPath}, cliArgs...) + runArgs := append([]string{"-run-bc"}, opts.capFlags()...) + runArgs = append(runArgs, bcPath) + runArgs = append(runArgs, opts.CLIArgs...) runCmd := exec.Command(compiler, runArgs...) runCmd.Env = append(os.Environ(), "HOWLFRAME_TEST_TOKEN=expected-secret") - exitCode, stdout, stderr, runErr := runCmdWithBuffers(runCmd, input) + exitCode, stdout, stderr, runErr := runCmdWithBuffers(runCmd, opts.Input) res := ExecutionResult{ Target: TargetBytecode, @@ -194,7 +246,7 @@ func ExecuteBytecode(filePath string, cliArgs []string, input string) ExecutionR outAll := stderr + " " + stdout res.ErrorMessage = strings.TrimSpace(outAll) res.ErrorClass = NormalizeError(outAll) - if strings.Contains(outAll, `"phase":"runtime"`) || strings.Contains(outAll, "panic:") || strings.Contains(outAll, "division by zero") || strings.Contains(outAll, "VMError") { + if strings.Contains(outAll, `"phase":"runtime"`) || strings.Contains(outAll, "panic:") || strings.Contains(outAll, "division by zero") || strings.Contains(outAll, "VMError") || isStructuredRuntimeRejection(outAll) { res.Status = StatusRuntimeFailure } else if res.Stdout != "" || res.Stderr != "" { res.Status = StatusPass @@ -206,15 +258,21 @@ func ExecuteBytecode(filePath string, cliArgs []string, input string) ExecutionR } func ExecuteInterpreter(filePath string, cliArgs []string, input string) ExecutionResult { + return executeInterpreter(filePath, RunOptions{CLIArgs: cliArgs, Input: input}) +} + +func executeInterpreter(filePath string, opts RunOptions) ExecutionResult { compiler, err := getCompiler() if err != nil { return ExecutionResult{Target: TargetInterpreter, ExitCode: 1, ErrorMessage: err.Error(), Status: StatusCompileFailure} } - runArgs := append([]string{"-run", "-allow-caps", "network,filesystem,process,environment,database", filePath}, cliArgs...) + runArgs := append([]string{"-run"}, opts.capFlags()...) + runArgs = append(runArgs, filePath) + runArgs = append(runArgs, opts.CLIArgs...) runCmd := exec.Command(compiler, runArgs...) runCmd.Env = append(os.Environ(), "HOWLFRAME_TEST_TOKEN=expected-secret") - exitCode, stdout, stderr, runErr := runCmdWithBuffers(runCmd, input) + exitCode, stdout, stderr, runErr := runCmdWithBuffers(runCmd, opts.Input) outCombined := stdout + stderr if strings.Contains(outCombined, "not supported under -run") || strings.Contains(outCombined, "-run only supports cli_app") { @@ -238,7 +296,7 @@ func ExecuteInterpreter(filePath string, cliArgs []string, input string) Executi if runErr != nil && exitCode != 0 { res.ErrorMessage = strings.TrimSpace(outCombined) res.ErrorClass = NormalizeError(outCombined) - if strings.Contains(outCombined, "division by zero") || strings.Contains(outCombined, "panic:") { + if isStructuredRuntimeRejection(outCombined) || strings.Contains(outCombined, "panic:") { res.Status = StatusRuntimeFailure } else if strings.Contains(outCombined, `"reason"`) { res.Status = StatusCompileFailure @@ -312,7 +370,7 @@ func ExecuteGoBackend(filePath string, cliArgs []string, input string) Execution } else { res.ErrorMessage = strings.TrimSpace(outAll) res.ErrorClass = NormalizeError(outAll) - if strings.Contains(outAll, "panic:") || strings.Contains(outAll, "runtime error") { + if strings.Contains(outAll, "panic:") || strings.Contains(outAll, "runtime error") || isStructuredRuntimeRejection(outAll) { res.Status = StatusRuntimeFailure } else if res.Stdout != "" || res.Stderr != "" { res.Status = StatusPass @@ -325,6 +383,10 @@ func ExecuteGoBackend(filePath string, cliArgs []string, input string) Execution } func ExecuteJSBackend(filePath string, cliArgs []string, input string) ExecutionResult { + return executeJSBackend(filePath, RunOptions{CLIArgs: cliArgs, Input: input}) +} + +func executeJSBackend(filePath string, opts RunOptions) ExecutionResult { compiler, err := getCompiler() if err != nil { return ExecutionResult{Target: TargetJavaScript, ExitCode: 1, ErrorMessage: err.Error(), Status: StatusCompileFailure} @@ -345,7 +407,28 @@ func ExecuteJSBackend(filePath string, cliArgs []string, input string) Execution } defer os.RemoveAll(tmpDir) - codegenCmd := exec.Command(compiler, filePath, "-o", tmpDir) + sourcePath := filePath + if opts.JSRoot != "" { + data, err := os.ReadFile(filePath) + if err != nil { + return ExecutionResult{Target: TargetJavaScript, ExitCode: 1, ErrorMessage: err.Error(), Status: StatusCompileFailure} + } + rewritten := strings.Replace(string(data), "(cli_app", "("+opts.JSRoot, 1) + if rewritten == string(data) { + return ExecutionResult{ + Target: TargetJavaScript, + ExitCode: 1, + ErrorMessage: "fixture has no cli_app root to rewrite", + Status: StatusCompileFailure, + } + } + sourcePath = filepath.Join(tmpDir, "app.howl") + if err := os.WriteFile(sourcePath, []byte(rewritten), 0o644); err != nil { + return ExecutionResult{Target: TargetJavaScript, ExitCode: 1, ErrorMessage: err.Error(), Status: StatusCompileFailure} + } + } + + codegenCmd := exec.Command(compiler, sourcePath, "-o", tmpDir) codegenOut, codegenErr := codegenCmd.CombinedOutput() if codegenErr != nil { errMsg := strings.TrimSpace(string(codegenOut)) @@ -368,8 +451,8 @@ func ExecuteJSBackend(filePath string, cliArgs []string, input string) Execution } } - runCmd := exec.Command("node", append([]string{jsPath}, cliArgs...)...) - exitCode, stdout, stderr, runErr := runCmdWithBuffers(runCmd, input) + runCmd := exec.Command("node", append([]string{jsPath}, opts.CLIArgs...)...) + exitCode, stdout, stderr, runErr := runCmdWithBuffers(runCmd, opts.Input) res := ExecutionResult{ Target: TargetJavaScript, @@ -379,15 +462,66 @@ func ExecuteJSBackend(filePath string, cliArgs []string, input string) Execution Status: StatusPass, } if runErr != nil { - res.ErrorMessage = strings.TrimSpace(stderr) - res.ErrorClass = NormalizeError(stderr) + outAll := stderr + " " + stdout + res.ErrorMessage = strings.TrimSpace(outAll) + res.ErrorClass = NormalizeError(outAll) res.Status = StatusRuntimeFailure } return res } +// compareToCanonical returns the parity discrepancies between one target and +// the bytecode canonical result. An unsupported target has no discrepancies. +func compareToCanonical(canonical, candidate ExecutionResult, tgt Target) []string { + if candidate.Status == StatusBackendUnsupported { + return nil + } + + var discrepancies []string + if canonical.Status == StatusCompileFailure { + if candidate.Status != StatusCompileFailure { + discrepancies = append(discrepancies, fmt.Sprintf("target %s succeeded compilation but canonical failed", tgt)) + } + return discrepancies + } + + if canonical.Status == StatusRuntimeFailure { + if candidate.Status != StatusRuntimeFailure { + discrepancies = append(discrepancies, fmt.Sprintf("target %s succeeded but canonical failed runtime", tgt)) + } else if canonical.ErrorClass != "" && candidate.ErrorClass != "" && canonical.ErrorClass != candidate.ErrorClass { + discrepancies = append(discrepancies, fmt.Sprintf("target %s error class mismatch: got %s, want %s", tgt, candidate.ErrorClass, canonical.ErrorClass)) + } + return discrepancies + } + + if candidate.Status != StatusPass { + return []string{fmt.Sprintf("target %s failed (%s: %s) but canonical passed", tgt, candidate.Status, candidate.ErrorMessage)} + } + + normCanonicalOut := NormalizeOutput(canonical.Stdout) + normCandidateOut := NormalizeOutput(candidate.Stdout) + if normCanonicalOut != normCandidateOut { + discrepancies = append(discrepancies, fmt.Sprintf("target %s stdout mismatch:\n--- canonical ---\n%s\n--- target ---\n%s", tgt, normCanonicalOut, normCandidateOut)) + } + + normCanonicalErr := NormalizeOutput(canonical.Stderr) + normCandidateErr := NormalizeOutput(candidate.Stderr) + if normCanonicalErr != normCandidateErr { + discrepancies = append(discrepancies, fmt.Sprintf("target %s stderr mismatch:\n--- canonical ---\n%s\n--- target ---\n%s", tgt, normCanonicalErr, normCandidateErr)) + } + + if canonical.ExitCode != candidate.ExitCode { + discrepancies = append(discrepancies, fmt.Sprintf("target %s exit code mismatch: got %d, want %d", tgt, candidate.ExitCode, canonical.ExitCode)) + } + return discrepancies +} + func VerifyParity(filePath string, cliArgs []string, input string, targets []Target) (ParityReport, error) { - canonical := ExecuteBytecode(filePath, cliArgs, input) + return VerifyParityWithOptions(filePath, targets, RunOptions{CLIArgs: cliArgs, Input: input}) +} + +func VerifyParityWithOptions(filePath string, targets []Target, opts RunOptions) (ParityReport, error) { + canonical := executeBytecode(filePath, opts) report := ParityReport{ FixturePath: filePath, CanonicalResult: canonical, @@ -401,65 +535,18 @@ func VerifyParity(filePath string, cliArgs []string, input string, targets []Tar case TargetBytecode: candidate = canonical case TargetInterpreter: - candidate = ExecuteInterpreter(filePath, cliArgs, input) + candidate = executeInterpreter(filePath, opts) case TargetGo: - candidate = ExecuteGoBackend(filePath, cliArgs, input) + candidate = ExecuteGoBackend(filePath, opts.CLIArgs, opts.Input) case TargetJavaScript: - candidate = ExecuteJSBackend(filePath, cliArgs, input) + candidate = executeJSBackend(filePath, opts) default: return report, fmt.Errorf("unsupported target: %s", tgt) } report.TargetResults[tgt] = candidate - - // If candidate is explicitly unsupported, don't fail parity - if candidate.Status == StatusBackendUnsupported { - continue - } - - // If canonical was a compile failure, candidate must also fail compilation - if canonical.Status == StatusCompileFailure { - if candidate.Status != StatusCompileFailure { - report.Discrepancies = append(report.Discrepancies, fmt.Sprintf("target %s succeeded compilation but canonical failed", tgt)) - report.OverallStatus = StatusSemanticMismatch - } - continue - } - - // If canonical was a runtime failure, candidate must also fail runtime - if canonical.Status == StatusRuntimeFailure { - if candidate.Status != StatusRuntimeFailure { - report.Discrepancies = append(report.Discrepancies, fmt.Sprintf("target %s succeeded but canonical failed runtime", tgt)) - report.OverallStatus = StatusSemanticMismatch - } else if canonical.ErrorClass != "" && candidate.ErrorClass != "" && canonical.ErrorClass != candidate.ErrorClass { - report.Discrepancies = append(report.Discrepancies, fmt.Sprintf("target %s error class mismatch: got %s, want %s", tgt, candidate.ErrorClass, canonical.ErrorClass)) - report.OverallStatus = StatusSemanticMismatch - } - continue - } - - // Both must succeed and match stdout, stderr, and exit code - if candidate.Status != StatusPass { - report.Discrepancies = append(report.Discrepancies, fmt.Sprintf("target %s failed (%s: %s) but canonical passed", tgt, candidate.Status, candidate.ErrorMessage)) - report.OverallStatus = StatusSemanticMismatch - continue - } - - normCanonicalOut := NormalizeOutput(canonical.Stdout) - normCandidateOut := NormalizeOutput(candidate.Stdout) - if normCanonicalOut != normCandidateOut { - report.Discrepancies = append(report.Discrepancies, fmt.Sprintf("target %s stdout mismatch:\n--- canonical ---\n%s\n--- target ---\n%s", tgt, normCanonicalOut, normCandidateOut)) - report.OverallStatus = StatusSemanticMismatch - } - - normCanonicalErr := NormalizeOutput(canonical.Stderr) - normCandidateErr := NormalizeOutput(candidate.Stderr) - if normCanonicalErr != normCandidateErr { - report.Discrepancies = append(report.Discrepancies, fmt.Sprintf("target %s stderr mismatch:\n--- canonical ---\n%s\n--- target ---\n%s", tgt, normCanonicalErr, normCandidateErr)) - report.OverallStatus = StatusSemanticMismatch - } - - if canonical.ExitCode != candidate.ExitCode { - report.Discrepancies = append(report.Discrepancies, fmt.Sprintf("target %s exit code mismatch: got %d, want %d", tgt, candidate.ExitCode, canonical.ExitCode)) + discrepancies := compareToCanonical(canonical, candidate, tgt) + if len(discrepancies) > 0 { + report.Discrepancies = append(report.Discrepancies, discrepancies...) report.OverallStatus = StatusSemanticMismatch } }