HowlFrame #90 Phase 3a: experimental defun and call bytecode

change_log.md

@@ -3,6 +3,7 @@
 ## Unreleased
 
 ### Changed
+* Experimental `-compile-hfir-bc` compiles and runs `defun`, `call`, and `return` with the existing `CALL` and `RETURN` opcodes. `while` is still rejected on that path. Production `-compile-bc` is still AST bytecode after the gate. #90 stays Partial. Journal: `docs/journals/2026-09-30_lowered_hfir_abi_phase3a_defun_call.md`.
 * Generated Go and JavaScript mediate `(fetch)`. An empty, missing, or non-`network` `HOWLFRAME_ALLOW_CAPS` grant fails with `CAPABILITY_DENIED` before any HTTP request, and the denial does not include the URL. The `network` grant performs the request. The production `-compile-bc` path is unchanged. Journal: `docs/journals/2026-09-30_lowered_hfir_abi_phase2d_fetch.md`.
 * Generated Go and JavaScript mediate `(read_file)`. An empty, missing, or non-`filesystem` `HOWLFRAME_ALLOW_CAPS` grant fails with `CAPABILITY_DENIED` before any filesystem read, and the denial does not include the path. The `filesystem` grant reads the file. The production `-compile-bc` path is unchanged. Journal: `docs/journals/2026-09-30_lowered_hfir_abi_phase2c_read_file.md`.
 * Generated Go and JavaScript mediate `(exec)`. An empty, missing, or non-`process` `HOWLFRAME_ALLOW_CAPS` grant fails with `CAPABILITY_DENIED` before a subprocess starts, and the denial does not include the command. The `process` grant runs it. The production `-compile-bc` path is unchanged. Journal: `docs/journals/2026-09-30_lowered_hfir_abi_phase2b_exec.md`.

docs/hfir_execution_status.md

@@ -25,7 +25,7 @@ Semantic information added for this path includes literal kind, explicit program
 
 ## What AST still owns
 
-The parser, source expansion, module resolution, patch/context transformations, checker rules, construct-position classification, and public build integration still operate on the AST. The legacy AST bytecode compiler remains the production compiler and still owns all language constructs outside the Phase-1 subset, including function and control-frame layout.
+The parser, source expansion, module resolution, patch/context transformations, checker rules, construct-position classification, and public build integration still operate on the AST. The legacy AST bytecode compiler remains the production compiler. Phase 3a teaches the experimental lowerer `defun`, `call`, and `return` with the existing `CALL` and `RETURN` opcodes. `while` and the rest of the control-frame layout stay on the AST compiler.
 
 ## Phase-1 executable subset
 
@@ -39,12 +39,13 @@ The direct lowerer supports a deterministic `cli_app` subset:
 | Deterministic mutation | `map_set`, `map_delete`, `append` |
 | Observability | `print`, `stderr`, `exit` |
 | Capability evidence | `env`, using the existing shared capability authority |
+| Functions (Phase 3a) | `defun`, `param`, `call`, `return`. Existing `CALL` and `RETURN` opcodes. `while` is not included. |
 
 ## Unsupported HFIR nodes
 
 Every node outside the subset fails closed with one `HFIR_BYTECODE_UNSUPPORTED` error diagnostic. The diagnostic identifies the offending graph node, target `bytecode`, and available source provenance. It returns no `BCProgram`.
 
-Examples intentionally deferred include `defun`, `call`, `return`, `while`, `for`, `try_let` and `catch`, `parse_json`, `cli_args`, file/network/database/process effects, HTTP routes/lambdas, stores, and model-oriented operations. This is not a claim that those features cannot be represented in HFIR; their current graph forms do not yet preserve all needed semantic roles without AST-shaped recovery.
+Phase 3a moved `defun`, `call`, and `return` into the experimental subset. `while` stays deferred: `LowerToBytecode` still returns `HFIR_BYTECODE_UNSUPPORTED` and no `BCProgram`. Other examples that remain outside a full CFG include loops the lowerer has not been asked to treat as SSA, `try_let` and `catch` where the graph is still not the production source, HTTP routes and lambdas, stores, and model-oriented operations. `ControlEdges` are still empty. This is not a claim that `while` cannot be represented later.
 
 ## Bytecode ownership
 
@@ -71,19 +72,19 @@ HowlChangeOps needs 23 runtime constructs plus `catch`; Phase 1 does not support
 
 ## HowlBoard compatibility
 
-The existing HowlBoard backend compatibility suite passes against the baseline HowlFrame bytecode compiler, including HTTP request parsing, JSON dict/list behavior, stores, CORS, and network/database capability behavior. Its browser test is blocked locally only because Playwright Chromium is not installed. HowlBoard requires functions, routes/lambdas, HTTP response forms, request parsing, stores, and loops, so it remains outside Phase 1. The Phase-1 changes do not alter its production path.
+The existing HowlBoard backend compatibility suite passes against the baseline HowlFrame bytecode compiler, including HTTP request parsing, JSON dict/list behavior, stores, CORS, and network/database capability behavior. Its browser test is blocked locally only because Playwright Chromium is not installed. HowlBoard requires routes and lambdas, HTTP response forms, request parsing, stores, and loops, so it remains outside this experimental subset. Phase 3a covers `defun`, `call`, and `return` on `-compile-hfir-bc` only. The production path is unchanged.
 
 ## What must happen before #88
 
-Improvement #88 has a real but deliberately bounded execution destination: model-authored graphs that meet the Phase-1 schema can be verified and lowered directly to a deterministic artifact, while unsupported nodes fail closed. Before a broad adapter can replace `.howl`, HFIR still needs explicit semantic forms for functions, structured error recovery, iteration, opaque effect operations, and stronger graph/control-flow verification. Work on #88 is meaningful as constrained Phase-1 adapter design, but not as a claim that arbitrary model-authored HFIR can execute today.
+Improvement #88 has a real but deliberately bounded execution destination: model-authored graphs that meet the Phase-1 schema can be verified and lowered directly to a deterministic artifact, while unsupported nodes fail closed. Phase 3a adds source-level `defun`, `call`, and `return` on `-compile-hfir-bc`. The model-adapter transport still rejects those kinds, so this slice does not reopen #88. Structured error recovery, `while`, and control-flow edges remain. Work on #88 is still constrained Phase-1 adapter design, not a claim that arbitrary model-authored HFIR can execute today.
 
 ## Lowered ABI v1 (improvement #90, phase 1)
 
 `docs/reference/lowered_hfir_abi_v1.md` versions the contract the hosts must share (`lowered-hfir-abi/v1`). The conformance suite compares the interpreter, the bytecode VM, Go, and JavaScript on a pure core and on `env` denial. That comparison runs the production AST paths through `tools/difftest`. It does not switch `-compile-bc` over to `LowerToBytecode`.
 
 Phase 2a mediates `env` in generated Go and JavaScript. An empty `HOWLFRAME_ALLOW_CAPS` grant is `CAPABILITY_DENIED` and does not read the variable. Phase 2b mediates `exec` the same way: an empty grant, or a grant that omits `process`, is `CAPABILITY_DENIED` before any subprocess starts. Phase 2c mediates `read_file` the same way: an empty grant, or a grant that omits `filesystem`, is `CAPABILITY_DENIED` before any filesystem read. Phase 2d mediates `fetch` the same way: an empty grant, or a grant that omits `network`, is `CAPABILITY_DENIED` before any HTTP request. The production compiler is unchanged.
 
-Wasm feasibility in this revision is the closed set `exec`, `spawn_agent`, and `http_server_start` (`HFIR_TARGET_INFEASIBLE`). Control edges are still unpopulated. Calls are still not executable HFIR. The rest of Phase 2 is the execution-path move. Journals: `docs/journals/2026-09-30_lowered_hfir_abi_phase1.md`, `docs/journals/2026-09-30_lowered_hfir_abi_phase2a_env.md`, `docs/journals/2026-09-30_lowered_hfir_abi_phase2b_exec.md`, `docs/journals/2026-09-30_lowered_hfir_abi_phase2c_read_file.md`, `docs/journals/2026-09-30_lowered_hfir_abi_phase2d_fetch.md`.
+Wasm feasibility in this revision is the closed set `exec`, `spawn_agent`, and `http_server_start` (`HFIR_TARGET_INFEASIBLE`). Control edges are still unpopulated. Phase 3a makes `defun`, `call`, and `return` executable on `-compile-hfir-bc` only. `while` is still not executable HFIR. Production `-compile-bc` is still the AST. The rest of Phase 2 is one lowered graph for every host. Journals: `docs/journals/2026-09-30_lowered_hfir_abi_phase1.md`, `docs/journals/2026-09-30_lowered_hfir_abi_phase2a_env.md`, `docs/journals/2026-09-30_lowered_hfir_abi_phase2b_exec.md`, `docs/journals/2026-09-30_lowered_hfir_abi_phase2c_read_file.md`, `docs/journals/2026-09-30_lowered_hfir_abi_phase2d_fetch.md`, `docs/journals/2026-09-30_lowered_hfir_abi_phase3a_defun_call.md`.
 
 ## Provenance limitation
 

docs/journals/2026-09-30_lowered_hfir_abi_phase3a_defun_call.md

@@ -0,0 +1,38 @@
+# Lowered HFIR ABI, phase 3a: defun and call on the experimental bytecode path
+
+## Why this slice
+
+The experimental lowerer could already emit bytecode for `let`, `if`, `map_keys`, `env`, and the other forms in `internal/hfir/bytecode.go`. `defun`, `call`, and `while` still failed closed with `HFIR_BYTECODE_UNSUPPORTED`. The production hosts already run `defun` and `call`. This slice makes the experimental path run that pair, and only that pair.
+
+Phase 2 is still one lowered graph for every host. This change does not take that step. #90 stays Partial.
+
+## What landed
+
+`LowerAST` now gives a `defun` a name, ordered `param` edges, and `body` edges. `type_hint`, `type_hints`, and `type_param` are erased, and so is a return-type symbol sitting between the parameter list and the body. That matches the AST bytecode compiler, which emits nothing for those annotations. `call` stores the callee name and ordered `arg` edges. `return` has an optional `value`.
+
+`LowerToBytecode` registers a `BCFunction` and emits the existing `CALL` and `RETURN` opcodes. A definition adds no instruction to the caller. There is no new opcode and no new capability. `ControlEdges` stay empty. `while` is still outside the subset: the lowerer returns one `HFIR_BYTECODE_UNSUPPORTED` diagnostic and no `BCProgram`. A `defun` whose body contains `while` fails the same way. The model-adapter transport still rejects kind `defun`, so #88 stays closed.
+
+`tests/conformance/abi_v1/09_defun_call.howl` is the shared fixture. `tools/difftest` runs it as `defun_call`.
+
+| Host | How this case reaches it |
+| --- | --- |
+| `hfir_bytecode` | `-compile-hfir-bc`, then `-run-bc`. Experimental path. |
+| `bytecode` | `-compile-bc`, then `-run-bc`. Production AST bytecode. Canonical result. |
+| `interpreter`, `go`, `javascript` | Still the AST. The fixture is already in their subset, so they are included. JavaScript rewrites the root to `web_app`. |
+
+All of those hosts print `42` and `phase3a`. `internal/vm/hfir_equivalence_test.go` also compares the experimental artifact with AST bytecode on a forward call, a typed parameter list, recursion, and an arity rejection.
+
+Production `-compile-bc` is still `runHFIRGate` and then `bytecode.CompileToBytecode` on the AST. A `while` program still compiles and runs on that flag, and `-compile-hfir-bc` still rejects it without writing an artifact.
+
+## What is still open
+
+* `-compile-bc` still compiles the AST. One lowered graph for every host is still the rest of Phase 2.
+* `while` is still not executable HFIR. This slice is not a CFG or SSA pass.
+* `ControlEdges` are still empty.
+* Generated `write_file`, `mkdir`, and the other host effects that Phase 2 has not mediated still do not consult a grant. `env`, `exec`, `read_file`, and `fetch` do.
+* Feasibility is still a Wasm-only set. This slice adds no Wasm opcode and does not execute Wasm.
+* No module linker, no HFIR module graph, and no VM module opcode.
+
+## What this does not do
+
+No new opcode. No new capability. #102–#105 and #108 stay Done and are not reopened. #88 stays closed. #90 stays Partial. `write_file` and `mkdir` stay deferred.

docs/reference/lowered_hfir_abi_v1.md

@@ -61,7 +61,17 @@ Integer `/` is not one rule yet. The interpreter truncates `int64`. The bytecode
 
 A `defun` has a name, a parameter list, optional `type_hints`, and a body. `return` leaves the function. `(call name arg ...)` passes arguments by position. The existing harness already compares that shape on the interpreter, the bytecode VM, and Go: `tests/parity/06_control_flow.howl` (`TestParityCorpus`).
 
-The experimental lowerer does not emit `defun`, `call`, or `while`. `LowerToBytecode` fails those graphs with `HFIR_BYTECODE_UNSUPPORTED` and no `BCProgram`. v1 does not move call execution onto HFIR.
+Phase 3a makes that shape executable on the experimental lowerer only. `LowerAST` gives `defun` a name, `param` edges, and `body` edges, and erases `type_hint`, `type_hints`, and `type_param`. A return-type symbol between the parameter list and the body is erased the same way the AST bytecode compiler skips it. `call` stores the callee name and `arg` edges. `return` has an optional `value`. `LowerToBytecode` emits the existing `CALL` and `RETURN` opcodes and registers a `BCFunction`. It does not add an opcode. `while` is still `HFIR_BYTECODE_UNSUPPORTED` with no `BCProgram`. `ControlEdges` stay empty. The model-adapter transport still rejects `defun`.
+
+The conformance case `defun_call` is `tests/conformance/abi_v1/09_defun_call.howl`. Its hosts are:
+
+| Host | Path |
+| --- | --- |
+| `hfir_bytecode` | Experimental `-compile-hfir-bc`, then `-run-bc`. This is the Phase 3a host. |
+| `bytecode` | Production `-compile-bc` (AST bytecode after the gate), then `-run-bc`. Canonical result. |
+| `interpreter`, `go`, `javascript` | Still the AST. Included because this fixture is already in their executable subset. |
+
+Those hosts must print the same stdout. A program the experimental lowerer rejects, including `while`, is not a shared case: the AST hosts run it and `-compile-hfir-bc` fails closed.
 
 ### Memory and runtime imports
 
@@ -129,9 +139,9 @@ A later lowering that owns meaning has to be a typed CFG in SSA:
 
 Phase 2 is one lowered graph consumed by every host, with identical outcomes or the same feasibility rejection.
 
-* Production `-compile-bc` still compiles the AST. Flipping that path is Phase 2.
-* `defun`, `call`, and `while` become executable HFIR, or every host rejects them with one code. Today the hosts run them and the experimental lowerer rejects them.
-* `ControlEdges` are populated and the graph is SSA.
+* Production `-compile-bc` still compiles the AST. Flipping that path is still Phase 2. Phase 3a does not flip it.
+* `defun`, `call`, and `return` are executable on `-compile-hfir-bc` (Phase 3a). `while` is not. The interpreter, the production bytecode VM, Go, and JavaScript still run calls from the AST. One lowered graph for every host is still open.
+* `ControlEdges` are populated and the graph is SSA. Phase 3a does not fill them.
 * Go and JavaScript mediate `env` (Phase 2a), `exec` (Phase 2b), `read_file` (Phase 2c), and `fetch` (Phase 2d). Other generated host effects, including `write_file` and `mkdir`, still do not. One lowered graph for every host is still the rest of Phase 2.
 * One feasibility table covers every target, not only the three Wasm host effects.
 * Non-exact integer division picks one rule.
@@ -141,4 +151,4 @@ Phase 2 is one lowered graph consumed by every host, with identical outcomes or
 
 ## What the suite does not prove
 
-Agreement among the AST backends is not proof that HFIR is the source of that agreement. `internal/vm/hfir_equivalence_test.go` is separate evidence that the experimental lowerer matches the bytecode VM on the subset it already emits, including `map_keys` and a granted `env`. That test is not the production compiler.
+Agreement among the AST backends is not proof that HFIR is the source of that agreement. `internal/vm/hfir_equivalence_test.go` is separate evidence that the experimental lowerer matches the bytecode VM on the subset it already emits, including `map_keys`, a granted `env`, and Phase 3a `defun` / `call`. That test is not the production compiler. The `defun_call` conformance case compares `-compile-hfir-bc` with the AST hosts on one fixture. Matching stdout there does not mean `-compile-bc` consumes HFIR.

howlframe_test.go

@@ -1239,6 +1239,46 @@ func TestCompileBcFailsClosedOnUnsupportedConstruct(t *testing.T) {
 	}
 }
 
+// TestCompileBcStaysASTWhileHfirBcRejectsWhile locks the production flag.
+// -compile-bc still emits AST bytecode for while. -compile-hfir-bc still
+// rejects while and writes no artifact. Phase 3a does not flip that flag.
+func TestCompileBcStaysASTWhileHfirBcRejectsWhile(t *testing.T) {
+	howlframeBinary := filepath.Join(t.TempDir(), "howlframe")
+	if output, err := exec.Command("go", "build", "-o", howlframeBinary, ".").CombinedOutput(); err != nil {
+		t.Fatalf("failed to build HowlFrame binary: %v\n%s", output, err)
+	}
+
+	dir := t.TempDir()
+	source := filepath.Join(dir, "while.howl")
+	if err := os.WriteFile(source, []byte("(cli_app (let (n 0) (while (< n 1) (do (set n (+ n 1)) (print n)))))\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	astOut := filepath.Join(dir, "ast.bc.bin")
+	if output, err := exec.Command(howlframeBinary, "-compile-bc", source, "-o", astOut).CombinedOutput(); err != nil {
+		t.Fatalf("-compile-bc rejected while: %v\n%s", err, output)
+	}
+	runOut, err := exec.Command(howlframeBinary, "-run-bc", astOut).CombinedOutput()
+	if err != nil {
+		t.Fatalf("-run-bc of production while artifact: %v\n%s", err, runOut)
+	}
+	if !strings.Contains(string(runOut), "1") {
+		t.Fatalf("production while stdout = %q", runOut)
+	}
+
+	hfirOut := filepath.Join(dir, "hfir.bc.bin")
+	output, err := exec.Command(howlframeBinary, "-compile-hfir-bc", source, "-o", hfirOut).CombinedOutput()
+	if err == nil {
+		t.Fatalf("-compile-hfir-bc accepted while:\n%s", output)
+	}
+	if !strings.Contains(string(output), "while") {
+		t.Fatalf("experimental rejection = %s", output)
+	}
+	if _, statErr := os.Stat(hfirOut); !os.IsNotExist(statErr) {
+		t.Fatalf("experimental rejection wrote an artifact: %v", statErr)
+	}
+}
+
 // TestCompileBcFailsClosedCitingOwningTracker proves the diagnostic points at
 // the backlog item that owns the gap, so the failure is actionable rather than
 // just a wall.