diff --git a/Benchmarks/Kernel/AnthropicFLT/README.md b/Benchmarks/Kernel/AnthropicFLT/README.md new file mode 100644 index 000000000..c576219cb --- /dev/null +++ b/Benchmarks/Kernel/AnthropicFLT/README.md @@ -0,0 +1,192 @@ +# Focused anonymous FLT regression suite + +Use this for the inner optimization loop, before whole-Mathlib/FLT checks. +The kernel checks each selected **work item only**, trusting lazily ingressed +dependencies. This is a diagnostic benchmark, not full-corpus verification. +Declaration names and expression metadata are not needed or loaded. + +For separate diagnostics, `IX_GUARD_STACKS=1` on the subject helper emits +bounded native stacks at resource guards. `IX_DEF_EQ_NEAR_GUARD=1` emits +bounded expression-pair summaries near the depth limit. Leave both off +for paired timings. Guard fuel counters inside a speculative comparison +describe its temporary slice; the final helper report accounts for the +actual total work charged to the subject. + +For separate cache/reduction diagnostics, the subject helper prints cache +hit rates to stderr with `IX_PERF_COUNTERS=1`, and the top 20 delta/iota +addresses with `IX_REDUCE_HISTO=1`. Histogram totals include addresses beyond +the displayed top 20. These counters measure reduction events, not proof +size or unique allocations. Diagnostics leave the subject JSON unchanged; +keep both flags unset for paired timings. + +`IX_SAME_HEAD_PROFILE=1` reports actual same-head comparisons by outcome +and definition head. Inclusive fuel overlaps across nested attempts; +exclusive and root fuel do not double-count it. Window skips and rejected- +probe cache hits are separate from attempts. Failed-attempt fuel is not +necessarily all avoidable: attempts may also populate useful caches. +Per-head attribution is bounded and holds no expression graphs. Leave this +flag unset for clean paired timing runs; subject JSON stays unchanged. + +The same-head report also ranks roots separately and prints at most 32 +expensive root-pair snapshots (at least 65,536 fuel) per thread/check. +These carry expression uids, the legacy context identity, and compact shapes; +they do not canonicalize free variables or claim alpha-equivalence. +`IX_HOT_MISSES=1` prints the final member's top 25 miss shapes once when the +subject helper finishes; add `IX_HOT_MISS_CTX=1` for context keys. It does +not require the much noisier per-guard `IX_REC_FUEL_DUMP`. +The hot-miss collector retains at most 4,096 keys, with labels capped at +512 UTF-8 bytes. Its Space-Saving summary can discover late hotspots by +replacing low-frequency entries. Printed counts are lower/upper bounds +(exact before replacement); the number of retained keys is **not** the +total number of distinct misses. Exact expression/context identities, not +truncated labels, distinguish keys. No expression graphs are retained, and +the collector resets per member. Leave it off for clean timings. + +Application inference selectively caches repeated dependent prefixes using +the existing exact context-sensitive keys and separate full/infer-only result +caches. Only successful inference publishes a prefix type; a bounded two-touch +fingerprint filter merely nominates work and never supplies a type or a +validity judgment. Each spine materializes at most one extra dependent suffix +for caching, leaving the remaining telescope substitution batched. The +admission history resets per member and retains at most 32 KiB of slots. With +`IX_PERF_COUNTERS=1`, `dependent_prefix_inserts` counts these materializations. + +Same-head congruence probes use a 131,072-fuel slice for Regular definitions +and 4,096 for other hints. Nested probes inherit the remaining allowance. +Exhaustion resumes ordinary unfolding with consumed work charged to the +check; it does not establish inequality. Regular root probes back off after +33,554,432 fuel in unsuccessful attempts across the declaration. This is an +admission threshold: the last admitted attempt can cross it by up to its +allowance. There is no per-head blacklist. Successful roots do not charge +the history; +nested work is charged only as part of its unsuccessful root. Skips resume +ordinary unfolding, and `skipped_backoff` is reported separately from actual +attempts. This constant-space history resets per member; it is not a +semantic cache. +Non-Regular probes retain their startup window, measured in actual work +rather than temporarily withheld fuel. The per-constant fuel cap is unchanged. + +To resolve anonymous addresses to names without decoding expression +metadata, use the `resolve_anon_names` Rust example with `FILE.ixe` followed +by hex prefixes (8–64 digits). It prints all aliases, not a single guessed +name. This separate diagnostic reads the full file into RAM: apply an +appropriate memory limit for large artifacts. Name lookup is not checking. + +`cases.json` pins the exact existing `.ixe` by SHA-256 and byte length. It +tracks 138 target addresses: 120 fuel failures, four depth failures, five +unfinished tails, and nine passing-baseline controls. The default **25-case core** +contains all five original tails, all four depth cases, ten fuel failures, +and six positive controls. Five fuel cases were promoted +from the existing extended inventory after they formed the final tail of +the completed 100M full FLT run; no addresses were duplicated. The positive +control that needed 558 seconds in the original full sweep belongs to the +extended suite. The remaining V3 failure `fuel-6c43f78d96e1` is also promoted +with 100M fuel and a 300-second timeout, without adding a duplicate address. + +Six controls were added after early backoff V5 regressed the full FLT sweep. +The three new failures (`50cb089d71a0`, `6234fd409441`, `7bf34ca3444c`) and +V3's highest-fuel passing subject (`6303d6c017f7`, 28.385M) are in the core. +Two near-cap V5 passes (`bfeba60bbbc6`, `eac5f4ec9db3`) are extended controls. +All six have explicit 100M/300-second budgets. Their recorded observations +are full-sweep results under contention, not isolated benchmark timings. + +Initial observations came from the stopped `28fc2270` full FLT run at 40M +fuel and 64 workers, plus the earlier bounded single-subject trials. The +five new core cases are `fuel-f38456f556fd`, `fuel-f5ebaf348a49`, +`fuel-fa1b74facf9a`, `fuel-fbba56420b6b`, and `fuel-fdfcfa70a9e8`. Each +exhausted 100M fuel in `check-flt-fuel100m-v1-1` (September 6, 2026). +Their recorded full-run times are under 64-worker contention, not isolated +benchmark results. Resolved names/aliases in the manifest are descriptive +only: execution remains anonymous and addresses are authoritative. Resource +failures are not expected kernel rejections: fuel/depth limits leave checking +unresolved. +“Unfinished” does not establish a deadlock. Addresses identify declarations +in this pinned artifact; do not substitute a rebuilt corpus silently. + +## Build and run + +Build the same `check_anon_subject` example against each kernel variant, +using the same native CPU/toolchain/feature settings. Preserve both binaries +under distinct filenames before running; never rebuild while measuring. + +```sh +nix develop --offline --command cargo build -p ix-ffi --release \ + --example check_anon_subject --features parallel,net --offline +``` + +On the Linux CPU box, run inside tmux. The harness is Lean (`import Lean` +only; no project rebuild needed). It uses GNU time/timeout, systemd, and +noninteractive sudo for the per-process scopes. +The output directory must not exist. No corpus is copied or rebuilt. + +```sh +lean --run Benchmarks/Kernel/AnthropicFLT/RunSuite.lean \ + --ixe /path/to/flt-after-source-hints-1.ixe \ + --baseline /path/to/check-subject-baseline \ + --adaptive /path/to/check-subject-adaptive \ + --output /path/to/new-run-directory +``` + +Use `--case tail-04e32656609b` (repeatable) for a tiny iteration, `--rounds 3` +for repeated pairs, or `--suite all` for the extended set. The extended set +can take much longer; it is not the default inner loop. + +The preflight resolves every recorded target to its work-item primary in +**both** binaries, without checking it. Both resolutions must agree. +Members of the same mutual block are deduplicated for matching fuel budgets. +Each timed invocation has a fresh process/KEnv, one checker worker, the +manifest's fixed fuel cap, 96 GiB MemoryMax, and zero MemorySwapMax. Defaults +are 40M fuel and 120 seconds per process; the 17.1M-fuel passing control uses +20M, and the extended slow positive control gets 600 seconds. The five +newly promoted tail cases explicitly use **100M fuel and 300 seconds per +process**, still one worker and 96 GiB/no swap. The original 15 cases keep +their old budgets for comparable performance regressions; the additional +V3 failure and the six backoff controls also have explicit 100M/300-second +budgets. Raising the +kernel default does not override the manifest. Pair order +alternates across cases and rounds. Hot-miss/perf/step diagnostics are off. + +## Results and interpretation + +- `run.json`: complete manifest, selected IDs, artifact and binary hashes, + file identities, host, driver hash, and start time. `source/` preserves + the explicit prototype source files and pinned Cargo/toolchain inputs. +- `resolved.json`: target-to-primary mapping and deduplicated selected work. +- `results.jsonl`: flushed after every invocation, even if a later one times + out. Contains outcome, raw helper report, wall/CPU time and peak RSS. +- Per-invocation `.log`, `.stderr.log`, `.time.json`, and `.command.json`: + original evidence, exact argv, and the unique cgroup scope name. Both + streams are flushed as they arrive, and the harness prints a heartbeat + every 30 seconds. Caught orchestration errors clean up only that scope; + the child's independent timeout remains in force if the harness is killed. +- `summary.json`: written only after all scheduled pairs finish; compares + completed results/work counts and checker times. A completed *harness* + does not mean its subjects all passed. Read each outcome. + +Checker time excludes initial mmap/enumeration; whole-process time includes +load, checking, and teardown. Peak RSS includes the whole helper process, +not just scratch storage. A timeout is censored, not a kernel failure or an +exact checker-time measurement; no speedup ratio is manufactured from it. +Exit 137 is reported as `killed`, not automatically called OOM. Missing time +or JSON data remains missing. Compare fuel usage only under matched caps. + +The helper's `last_member_fuel` and `last_member_def_eq_peak` refer to the +last checked member. The latter is a **definitional-equality depth metric**, +not an overall peak infer/WHNF recursion-depth measurement. Do not label +these as block aggregates. Operation counts span the work item; aggregate +fuel is unavailable with perf diagnostics disabled. + +For the storage-only scratch change, completed outcomes and work/fuel counts +should agree. The driver flags mismatches without treating timeout cases as +passes. For algorithmic optimizations such as batched binder inference, +explicitly pass `--allow-work-changes`: changed work/fuel/depth counts are +still recorded, but labeled `same_outcome_changed_work` when the verdict, +error, and target count agree. Verdict/error/target changes remain mismatches +requiring review; timeouts remain incomplete. The selected policy is saved +in `run.json`; the default storage-only policy is unchanged. Keep passing +controls, kernel tests, Mathlib, and periodic full FLT sweeps as separate +correctness and whole-run performance gates. + +```sh +lean --run Benchmarks/Kernel/AnthropicFLT/RunSuite.lean --self-test +``` diff --git a/Benchmarks/Kernel/AnthropicFLT/RunSuite.lean b/Benchmarks/Kernel/AnthropicFLT/RunSuite.lean new file mode 100644 index 000000000..6831cb0e8 --- /dev/null +++ b/Benchmarks/Kernel/AnthropicFLT/RunSuite.lean @@ -0,0 +1,489 @@ +import Lean + +/-! +Bounded paired anonymous FLT subject checks, orchestrated in Lean. +No builds, downloads, corpus rewriting, or full-environment checks occur here. +The Rust subject helper trusts dependencies. See README.md for scope/metrics. +-/ + +open Lean System + +namespace Benchmarks.AnthropicFLT + +def suiteDir : FilePath := "Benchmarks/Kernel/AnthropicFLT" + +structure Case where + id : String + address : String + category : String + core : Bool + fuel : Option Nat := none + timeout_seconds : Option Nat := none + deriving FromJson, ToJson, Inhabited + +structure Defaults where + fuel : Nat + timeout_seconds : Nat + memory_gib : Nat + workers : Nat + deriving FromJson, ToJson + +structure Artifact where + filename : String + bytes : Nat + sha256 : String + deriving FromJson, ToJson + +structure Manifest where + schema : Nat + artifact : Artifact + defaults : Defaults + cases : Array Case + deriving FromJson + +structure Resolution where + requested : String + primary : String + targets : Nat + deriving FromJson, ToJson, BEq, Inhabited + +structure Group where + primary : String + targets : Nat + fuel : Nat + timeoutSeconds : Nat + caseIds : Array String + categories : Array String + requested : Array String + deriving ToJson, Inhabited + +def isHex (s : String) : Bool := + s.length == 64 && s.toList.all (fun c => c.isDigit || ('a' ≤ c && c ≤ 'f')) + +def isId (s : String) : Bool := + !s.isEmpty && s.toList.all (fun c => c.isLower || c.isDigit || c == '-') + +def need (value : Except String α) : IO α := + match value with + | .ok a => pure a + | .error e => throw (IO.userError e) + +def validate (m : Manifest) : Except String Unit := do + unless m.schema == 1 && isHex m.artifact.sha256 do throw "invalid manifest/artifact" + unless m.defaults.workers == 1 do throw "suite requires one fresh worker" + unless 0 < m.defaults.memory_gib && m.defaults.memory_gib ≤ 96 do + throw "memory budget must be between 1 and 96 GiB" + let mut ids := #[] + let mut addresses := #[] + for c in m.cases do + unless isId c.id && !ids.contains c.id do throw "invalid or duplicate case ID" + unless isHex c.address && !addresses.contains c.address do throw "invalid or duplicate address" + unless #["tail", "depth", "fuel", "control"].contains c.category do throw "unknown category" + let fuel := c.fuel.getD m.defaults.fuel + let timeout := c.timeout_seconds.getD m.defaults.timeout_seconds + unless 0 < fuel && fuel ≤ 100000000 do throw "invalid fuel budget" + unless 0 < timeout && timeout ≤ 600 do throw "invalid time budget" + ids := ids.push c.id + addresses := addresses.push c.address + +def normalize (cases : Array Case) (rows : Array Resolution) (d : Defaults) : + Except String (Array Group) := do + let mut groups : Array Group := #[] + for c in cases do + let some r := rows.find? (·.requested == c.address) | throw s!"unresolved target {c.address}" + unless isHex r.primary && r.targets > 0 do throw "invalid resolver output" + let fuel := c.fuel.getD d.fuel + let timeout := c.timeout_seconds.getD d.timeout_seconds + if let some i := groups.findIdx? (fun g => g.primary == r.primary && g.fuel == fuel) then + let g := groups[i]! + groups := groups.set! i { g with + timeoutSeconds := max g.timeoutSeconds timeout + caseIds := g.caseIds.push c.id + categories := g.categories.push c.category + requested := g.requested.push c.address } + else + groups := groups.push { + primary := r.primary, targets := r.targets, fuel, timeoutSeconds := timeout + caseIds := #[c.id], categories := #[c.category], requested := #[c.address] } + return groups + +def field (j : Json) (key : String) : Json := (j.getObjVal? key).toOption.getD .null +def str (j : Json) (key : String) : String := (j.getObjValAs? String key).toOption.getD "" + +def classify (code : Nat) (report : Json) : String := Id.run do + if code == 124 then return "timeout" + if code == 137 then return "killed" -- Not automatically an OOM diagnosis. + if code ≥ 128 then return "crash" + if str report "scope" != "subject-only" then return "harness_error" + if code == 0 && field report "passed" == toJson true then return "pass" + if code != 1 || field report "passed" != toJson false then return "harness_error" + return match str report "error" with + | "recursive fuel exhausted" => "fuel_exhausted" + | "max recursion depth exceeded" => "depth_exceeded" + | _ => "kernel_error" + +def compare (baseline adaptive : Json) (allowWorkChanges : Bool := false) : Json := Id.run do + let a := str baseline "outcome" + let b := str adaptive "outcome" + let mut result := Json.mkObj [("baseline", toJson a), ("adaptive", toJson b)] + let incomplete := #["timeout", "killed", "crash", "harness_error"] + if incomplete.contains a || incomplete.contains b then + return result.setObjVal! "comparison" (toJson "incomplete") + let ar := field baseline "report" + let br := field adaptive "report" + let keys := #["passed", "error", "targets", "last_member_fuel", "last_member_def_eq_peak", + "subst", "whnf", "def_eq", "intern", "nat_arith"] + let mismatches := keys.filter (fun k => field ar k != field br k) + result := result.setObjVal! "mismatched_fields" (toJson mismatches) + let verdictChanged := a != b || + #["passed", "error", "targets"].any (fun k => field ar k != field br k) + result := result.setObjVal! "comparison" (toJson + (if verdictChanged then "mismatch" + else if mismatches.isEmpty then "same_outcome_and_work" + else if allowWorkChanges then "same_outcome_changed_work" + else "mismatch")) + if let (.ok baseTime, .ok adaptiveTime) := + (ar.getObjValAs? Float "check_secs", br.getObjValAs? Float "check_secs") then + if baseTime > 0 then + result := result.setObjVal! "adaptive_over_baseline_check_time" (toJson (adaptiveTime / baseTime)) + return result + +def announce (message : String) : IO Unit := do + IO.println message + (← IO.getStdout).flush + +def saveJson (path : FilePath) (j : Json) : IO Unit := do + if ← path.pathExists then throw (IO.userError s!"refusing to overwrite {path}") + IO.FS.writeFile path (j.pretty ++ "\n") + +def lastJson (path : FilePath) : IO Json := do + if !(← path.pathExists) then return .null + let mut result := Json.null + for line in (← IO.FS.readFile path).splitOn "\n" do + if let .ok (.obj obj) := Json.parse line then result := .obj obj + return result + +def checkedOutput (cmd : String) (args : Array String) : IO String := do + let r ← IO.Process.output { cmd, args } + unless r.exitCode == 0 do throw (IO.userError s!"{cmd}: {r.stderr}") + return r.stdout.trimAscii.toString + +def digest (path : FilePath) : IO String := do + let output ← checkedOutput "sha256sum" #["--", path.toString] + let hash := (output.splitOn " ").head! + unless isHex hash do throw (IO.userError "invalid sha256sum output") + return hash + +def identities (paths : Array FilePath) : IO (Array String) := + paths.mapM fun p => checkedOutput "stat" #["-c", "%d:%i:%s:%y", "--", p.toString] + +def timeFormat : String := + "{\"elapsed_seconds\":%e,\"user_seconds\":%U,\"system_seconds\":%S,\"peak_rss_kib\":%M,\"exit_code\":%x}" + +def scopeArgs (unit user : String) (binary : FilePath) (args : Array String) + (timing : FilePath) (seconds fuel memory : Nat) : Array String := + #["-n", "systemd-run", "--quiet", "--scope", s!"--unit={unit}", + "-p", s!"MemoryMax={memory}G", "-p", "MemorySwapMax=0", + "sudo", "-u", user, "/usr/bin/env", "-i", "PATH=/usr/bin:/bin", "LANG=C", "LC_ALL=C", + "LEAN_NUM_THREADS=1", "RAYON_NUM_THREADS=1", s!"IX_MAX_REC_FUEL={fuel}", + "/usr/bin/time", "-f", timeFormat, "-o", timing.toString, + "/usr/bin/timeout", "--signal=TERM", "--kill-after=10s", toString seconds, + binary.toString] ++ args + +partial def pump (src dst : IO.FS.Handle) (echo : Bool) : IO Unit := do + let line ← src.getLine + unless line.isEmpty do + dst.putStr line + dst.flush + if echo then + (← IO.getStdout).putStr line + (← IO.getStdout).flush + pump src dst echo + +structure Invocation where + exitCode : Nat + report : Json + timing : Json + wrapperSeconds : Float + +def limited (binary : FilePath) (args : Array String) (out : FilePath) + (name runId user : String) (seconds fuel memory : Nat) : IO Invocation := do + let unit := s!"ix-flt-suite-{runId}-{name}" + let timing := out / s!"{name}.time.json" + let argv := scopeArgs unit user binary args timing seconds fuel memory + saveJson (out / s!"{name}.command.json") (Json.mkObj [ + ("unit", toJson unit), ("argv", toJson (#["sudo"] ++ argv))]) + announce s!"START {name}: timeout={seconds}s fuel={fuel} memory={memory}GiB" + let log ← IO.FS.Handle.mk (out / s!"{name}.log") .write + let err ← IO.FS.Handle.mk (out / s!"{name}.stderr.log") .write + let child ← IO.Process.spawn { + cmd := "sudo", args := argv, stdin := .null, stdout := .piped, stderr := .piped, setsid := true } + let stdoutTask ← IO.asTask (pump child.stdout log false) .dedicated + let stderrTask ← IO.asTask (pump child.stderr err true) .dedicated + let start ← IO.monoMsNow + let mut lastNotice := start + let code ← try + let mut status : Option UInt32 := none + while status.isNone do + status ← child.tryWait + if status.isNone then + let now ← IO.monoMsNow + if now - start > (seconds + 40) * 1000 then + throw (IO.userError s!"scope wrapper exceeded timeout for {name}") + if now - lastNotice ≥ 30000 then + announce s!"WAIT {name}: {(now - start) / 1000}s (process cap {seconds}s)" + lastNotice := now + IO.sleep 250 + pure status.get! + catch e => + let _ ← IO.Process.output { cmd := "sudo", args := #["-n", "systemctl", "kill", + "--signal=KILL", "--kill-whom=all", unit] } + child.kill + throw e + IO.ofExcept stdoutTask.get + IO.ofExcept stderrTask.get + let report ← lastJson (out / s!"{name}.log") + let measured ← lastJson timing + let finish ← IO.monoMsNow + return { + exitCode := code.toNat + report := report + timing := measured + wrapperSeconds := (finish - start).toFloat / 1000 + } + +structure Options where + ixe : String := "" + baseline : String := "" + adaptive : String := "" + output : String := "" + manifest : String := (suiteDir / "cases.json").toString + all : Bool := false + cases : Array String := #[] + rounds : Nat := 1 + selfTest : Bool := false + allowWorkChanges : Bool := false + +def parseArgs : List String → Options → Except String Options + | [], opts => .ok opts + | "--self-test" :: rest, opts => parseArgs rest { opts with selfTest := true } + | "--allow-work-changes" :: rest, opts => parseArgs rest { opts with allowWorkChanges := true } + | "--ixe" :: v :: rest, opts => parseArgs rest { opts with ixe := v } + | "--baseline" :: v :: rest, opts => parseArgs rest { opts with baseline := v } + | "--adaptive" :: v :: rest, opts => parseArgs rest { opts with adaptive := v } + | "--output" :: v :: rest, opts => parseArgs rest { opts with output := v } + | "--manifest" :: v :: rest, opts => parseArgs rest { opts with manifest := v } + | "--case" :: v :: rest, opts => parseArgs rest { opts with cases := opts.cases.push v } + | "--suite" :: v :: rest, opts => + if v == "all" || v == "core" then parseArgs rest { opts with all := v == "all" } + else .error "--suite must be core or all" + | "--rounds" :: v :: rest, opts => do + let some n := v.toNat? | throw "invalid --rounds" + unless 0 < n && n ≤ 10 do throw "--rounds must be 1..10" + parseArgs rest { opts with rounds := n } + | flag :: _, _ => .error s!"unknown or incomplete option {flag}" + +def ensure (condition : Bool) (message : String) : IO Unit := + unless condition do throw (IO.userError message) + +def selfTest (m : Manifest) : IO Unit := do + need (validate m) + ensure (m.cases.size == 138) "inventory size" + let core := m.cases.filter (·.core) + ensure (core.size == 25) "core size" + for (category, count) in #[("tail", 5), ("depth", 4), ("fuel", 10), ("control", 6)] do + ensure ((core.filter (·.category == category)).size == count) s!"core {category}" + ensure ((m.cases.filter (·.category == "fuel")).size == 120) "extended fuel count" + let lateTail := #["fuel-6c43f78d96e1", "fuel-f38456f556fd", "fuel-f5ebaf348a49", "fuel-fa1b74facf9a", + "fuel-fbba56420b6b", "fuel-fdfcfa70a9e8"] + for id in lateTail do + let some c := core.find? (·.id == id) | throw (IO.userError s!"missing late tail {id}") + ensure (c.category == "fuel" && c.fuel == some 100000000 && + c.timeout_seconds == some 300) s!"late-tail budget {id}" + let backoffControls := #["control-50cb089d71a0", "control-6234fd409441", + "control-7bf34ca3444c", "control-6303d6c017f7", "control-bfeba60bbbc6", + "control-eac5f4ec9db3"] + for id in backoffControls do + let some c := m.cases.find? (·.id == id) | throw (IO.userError s!"missing backoff control {id}") + ensure (c.category == "control" && c.fuel == some 100000000 && + c.timeout_seconds == some 300) s!"backoff-control budget {id}" + for c in core.filter (fun c => !lateTail.contains c.id && !backoffControls.contains c.id) do + ensure (c.fuel.getD m.defaults.fuel ≤ 40000000) s!"original core budget {c.id}" + for c in #[{ m.cases[0]! with fuel := some 0 }, + { m.cases[0]! with fuel := some 100000001 }, + { m.cases[0]! with timeout_seconds := some 601 }, + { m.cases[0]! with id := "../bad" }] do + ensure ((validate { m with cases := #[c] }).toOption.isNone) "manifest guard" + let a := String.ofList (List.replicate 64 'a') + let b := String.ofList (List.replicate 64 'b') + let cases : Array Case := #[ + { id := "one", address := a, category := "depth", core := true }, + { id := "two", address := b, category := "depth", core := true }] + let resolutions : Array Resolution := #[ + { requested := a, primary := a, targets := 2 }, + { requested := b, primary := a, targets := 2 }] + let groups ← need (normalize cases resolutions m.defaults) + ensure (groups.size == 1 && groups[0]!.caseIds.size == 2) "block alias dedup" + ensure ((normalize cases #[] m.defaults).toOption.isNone) "unresolved target must fail" + let different := cases.set! 1 { cases[1]! with fuel := some 20000000 } + ensure ((← need (normalize different resolutions m.defaults)).size == 2) "distinct budgets" + let fail := Json.mkObj [("scope", toJson "subject-only"), ("passed", toJson false), + ("error", toJson "recursive fuel exhausted")] + ensure (classify 1 fail == "fuel_exhausted") "fuel classification" + ensure (classify 0 fail == "harness_error") "inconsistent exit/report" + ensure (classify 0 .null == "harness_error") "missing report" + ensure (classify 124 .null == "timeout" && classify 137 .null == "killed") "resource outcomes" + ensure (classify 1 (fail.setObjVal! "error" (toJson "max recursion depth exceeded")) == + "depth_exceeded") "depth outcome" + let report := Json.mkObj [("passed", toJson true), ("last_member_fuel", toJson (7 : Nat)), + ("check_secs", toJson (2 : Nat))] + let row := Json.mkObj [("outcome", toJson "pass"), ("report", report)] + ensure (str (compare row row) "comparison" == "same_outcome_and_work") "matching work" + let changed := row.setObjVal! "report" (report.setObjVal! "last_member_fuel" (toJson (8 : Nat))) + ensure (str (compare row changed) "comparison" == "mismatch") "fuel mismatch" + ensure (str (compare row changed true) "comparison" == "same_outcome_changed_work") + "explicit algorithmic comparison retains changed work" + for changed in #[row.setObjVal! "outcome" (toJson "depth_exceeded"), + row.setObjVal! "report" (report.setObjVal! "passed" (toJson false)), + row.setObjVal! "report" (report.setObjVal! "targets" (toJson (2 : Nat))), + row.setObjVal! "report" (report.setObjVal! "error" (toJson "new error"))] do + ensure (str (compare row changed true) "comparison" == "mismatch") + "algorithmic comparison must not permit outcome/target/error changes" + let timedOut := Json.mkObj [("outcome", toJson "timeout")] + let censored := compare timedOut row + ensure (str censored "comparison" == "incomplete" && + field censored "adaptive_over_baseline_check_time" == .null) "no invented timeout speedup" + let cmd := scopeArgs "test-unit" "test-user" "/binary" #[] "/time" 120 100 96 + for flag in #["MemoryMax=96G", "MemorySwapMax=0", "-i", "RAYON_NUM_THREADS=1", + "IX_MAX_REC_FUEL=100", "--kill-after=10s", "--unit=test-unit"] do + ensure (cmd.contains flag) s!"missing guard {flag}" + announce "Lean suite self-tests passed." + +def run (opts : Options) : IO UInt32 := do + let manifestJson ← need (Json.parse (← IO.FS.readFile opts.manifest)) + let m : Manifest ← need (fromJson? manifestJson) + need (validate m) + if opts.selfTest then selfTest m; return 0 + unless [opts.ixe, opts.baseline, opts.adaptive, opts.output].all (!·.isEmpty) do + throw (IO.userError "required: --ixe FILE --baseline BIN --adaptive BIN --output NEW_DIR") + for id in opts.cases do + ensure (m.cases.any (·.id == id)) s!"unknown case {id}" + let selected := m.cases.filter fun c => + if opts.cases.isEmpty then opts.all || c.core else opts.cases.contains c.id + let paths ← #[opts.ixe, opts.baseline, opts.adaptive].mapM (IO.FS.realPath ∘ FilePath.mk) + let ixe := paths[0]! + let binaries := paths.extract 1 3 + let initial ← identities paths + ensure ((← ixe.metadata).byteSize.toNat == m.artifact.bytes) "artifact size mismatch" + let out : FilePath := opts.output + ensure (!(← out.pathExists)) s!"refusing to overwrite directory {out}" + IO.FS.createDir out + let out ← IO.FS.realPath out + announce "Fingerprinting input and executables before timed checks..." + let hashes ← paths.mapM digest + ensure (hashes[0]! == m.artifact.sha256) "artifact SHA-256 mismatch" + ensure (hashes[1]! != hashes[2]!) "baseline/adaptive binaries are identical" + ensure ((← identities paths) == initial) "input changed during fingerprinting" + let runId := s!"{← IO.Process.getPID}-{← IO.monoMsNow}" + let user ← checkedOutput "id" #["-un"] + saveJson (out / "run.json") (Json.mkObj [ + ("schema", toJson (1 : Nat)), ("run_id", toJson runId), ("manifest", manifestJson), + ("selected_case_ids", toJson (selected.map (·.id))), + ("paths", toJson (paths.map (·.toString))), ("sha256", toJson hashes), + ("file_identity", toJson initial), ("rounds", toJson opts.rounds), + ("allow_work_changes", toJson opts.allowWorkChanges), + ("host", toJson (← checkedOutput "uname" #["-a"])), + ("driver_sha256", toJson (← digest (suiteDir / "RunSuite.lean"))), + ("started_utc", toJson (← checkedOutput "date" #["-u", "+%Y-%m-%dT%H:%M:%SZ"])), + ("scope", toJson "subject-only; dependencies trusted"), ("workers", toJson (1 : Nat))]) + -- Explicit prototype source snapshot; no unrelated worktree files. + for relative in #["crates/kernel/src/env.rs", "crates/kernel/src/env/scratch.rs", + "crates/kernel/src/subst.rs", "crates/kernel/src/subst/scratch_tests.rs", + "crates/kernel/src/infer.rs", "crates/kernel/src/infer/binders.rs", + "crates/kernel/src/infer/binders/tests.rs", + "crates/kernel/src/infer/application.rs", + "crates/kernel/src/infer/application/prefix.rs", + "crates/kernel/src/infer/application/tests.rs", + "crates/kernel/src/infer/summary.rs", + "crates/kernel/src/infer/summary/tests.rs", "crates/kernel/src/perf.rs", + "crates/kernel/src/perf/same_head.rs", + "crates/kernel/src/perf/hot_misses.rs", + "crates/kernel/src/def_eq.rs", "crates/kernel/src/tc.rs", + "crates/kernel/src/tc/spine_tests.rs", "crates/kernel/src/whnf.rs", + "crates/kernel/src/def_eq/projection_tests.rs", + "crates/kernel/src/def_eq/application.rs", + "crates/kernel/src/def_eq/application/tests.rs", + "crates/kernel/src/def_eq/same_head_tests.rs", + "crates/kernel/src/def_eq/speculation.rs", + "crates/kernel/src/def_eq/binders.rs", + "crates/kernel/src/def_eq/binders/tests.rs", + "crates/ffi/examples/check_anon_subject.rs", "Cargo.toml", "Cargo.lock", + "crates/kernel/Cargo.toml", "rust-toolchain.toml", "lean-toolchain", + "flake.nix", "flake.lock", + ".cargo/config.toml", "Benchmarks/Kernel/AnthropicFLT/RunSuite.lean"] do + let src : FilePath := relative + if ← src.pathExists then + let dst := out / "source" / relative + if let some parent := dst.parent then IO.FS.createDirAll parent + IO.FS.writeBinFile dst (← IO.FS.readBinFile src) + let requested := m.cases.map (·.address) + let mut resolutions : Array Resolution := #[] + for (binary, variant) in binaries.zip #["baseline", "adaptive"] do + let result ← limited binary (#["--resolve", ixe.toString] ++ requested) out + s!"resolve-{variant}" runId user 120 m.defaults.fuel m.defaults.memory_gib + ensure (result.exitCode == 0 && str result.report "scope" == "index-only") + s!"{variant} resolution failed; see its logs" + let rows : Array Resolution ← need (result.report.getObjValAs? _ "resolutions") + ensure (rows.map (·.requested) == requested) "incomplete/reordered resolution" + if variant == "baseline" then resolutions := rows + else ensure (rows == resolutions) "variants disagree on work-item resolution" + let groups ← need (normalize selected resolutions m.defaults) + saveJson (out / "resolved.json") (Json.mkObj [ + ("all_targets", toJson resolutions), ("selected_work", toJson groups)]) + announce s!"Resolved {requested.size} targets; running {groups.size} work items × 2 variants × {opts.rounds} rounds" + let results ← IO.FS.Handle.mk (out / "results.jsonl") .write + let mut pairs : Array Json := #[] + for round in [:opts.rounds] do + for i in [:groups.size] do + let g := groups[i]! + let order := if (round + i) % 2 == 0 then #[1, 0] else #[0, 1] + let mut rows := #[Json.null, Json.null] + for v in order do + ensure ((← identities paths) == initial) "input/binary changed during suite" + let variant := if v == 0 then "baseline" else "adaptive" + let name := s!"r{round + 1}-{g.caseIds[0]!}-{variant}" + let r ← limited binaries[v]! #[ixe.toString, g.primary] out name runId user + g.timeoutSeconds g.fuel m.defaults.memory_gib + ensure ((← identities paths) == initial) "input/binary changed during check" + let mut outcome := classify r.exitCode r.report + if str r.report "scope" == "subject-only" then + if str r.report "primary" != g.primary || field r.report "targets" != toJson g.targets || + field r.report "fuel_cap_per_member" != toJson g.fuel then + outcome := "harness_error" + let row := Json.mkObj [ + ("work", toJson g), ("variant", toJson variant), ("round", toJson (round + 1)), + ("exit_code", toJson r.exitCode), ("outcome", toJson outcome), + ("report", r.report), ("time", r.timing), ("wrapper_seconds", toJson r.wrapperSeconds)] + results.putStrLn row.compress + results.flush + rows := rows.set! v row + announce s!"DONE {name}: {outcome} check={(field r.report "check_secs").compress}s peak_rss={(field r.timing "peak_rss_kib").compress}KiB" + ensure (outcome != "harness_error") s!"harness error in {name}; stopping" + let pair := (compare rows[0]! rows[1]! opts.allowWorkChanges).setObjVal! "case_ids" (toJson g.caseIds) + |>.setObjVal! "round" (toJson (round + 1)) + pairs := pairs.push pair + announce s!"PAIR {pair.compress}" + saveJson (out / "summary.json") (Json.mkObj [ + ("complete", toJson true), ("pairs", toJson pairs), + ("warning", toJson "Completed harness is not full-corpus verification; timeouts remain unresolved.")]) + return if pairs.any (fun p => str p "comparison" == "mismatch") then 1 else 0 + +end Benchmarks.AnthropicFLT + +def main (args : List String) : IO UInt32 := do + try + let opts ← Benchmarks.AnthropicFLT.need (Benchmarks.AnthropicFLT.parseArgs args {}) + Benchmarks.AnthropicFLT.run opts + catch e => + IO.eprintln s!"FLT suite: {e}" + return 2 diff --git a/Benchmarks/Kernel/AnthropicFLT/cases.json b/Benchmarks/Kernel/AnthropicFLT/cases.json new file mode 100644 index 000000000..59b907d6f --- /dev/null +++ b/Benchmarks/Kernel/AnthropicFLT/cases.json @@ -0,0 +1,1068 @@ +{ + "schema": 1, + "description": "Anonymous subject-only FLT performance regressions; dependencies are trusted, not certified.", + "baseline_kernel_commit": "28fc22702964058cd7cfe4eb012baf23b3af6e3a", + "artifact": { + "filename": "flt-after-source-hints-1.ixe", + "bytes": 30113168066, + "sha256": "5251edf00c0050d766702cd32d777b30889c167f4e0021b5ba89496abbb2f4d8" + }, + "source": "Initial inventory: check-flt-28fc2270-40m-1.log and .failures.txt, stopped 2026-09-06 with 5 unfinished work items and 124 failed target addresses. Five existing fuel cases promoted to the core from the final tail of check-flt-fuel100m-v1-1 on 2026-09-06; that full run completed with 50 fuel failures and no depth failures. Six passing V3 controls added after full-flt-same-head-backoff-v5-1 exposed three new failures and two near-cap passes, alongside V3's highest-fuel passing control.", + "defaults": { + "fuel": 40000000, + "timeout_seconds": 120, + "memory_gib": 96, + "workers": 1 + }, + "cases": [ + { + "id": "control-50cb089d71a0", + "address": "50cb089d71a05918fd793b026249f998556b8e16010becf5498e1501de6f092c", + "category": "control", + "core": true, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "V3 passed at 3748333 fuel / 8.372s; early backoff V5 exhausted 100M / 207.450s in the full sweep", + "observed_run": "full-flt-same-head-backoff-v5-1", + "observed_workers": 64 + }, + { + "id": "control-6234fd409441", + "address": "6234fd409441faeed33e68b7609594f9e7984907f421026fcc39eed4e2abf78f", + "category": "control", + "core": true, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "V3 passed at 3773953 fuel / 8.669s; early backoff V5 exhausted 100M / 221.590s in the full sweep", + "observed_run": "full-flt-same-head-backoff-v5-1", + "observed_workers": 64 + }, + { + "id": "control-7bf34ca3444c", + "address": "7bf34ca3444c8fc5baa18e74fea3ba57610cd79951470802f31a4e1edef4249b", + "category": "control", + "core": true, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "V3 passed at 2374766 fuel / 8.426s; early backoff V5 exhausted 100M / 301.760s in the full sweep", + "observed_run": "full-flt-same-head-backoff-v5-1", + "observed_workers": 64 + }, + { + "id": "control-6303d6c017f7", + "address": "6303d6c017f7b93869c735dcf4b49268aa2138d8e8b668e243d5034919081001", + "category": "control", + "core": true, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "Highest-fuel passing V3 work item: 28384918 fuel / 69.216s in the full sweep", + "observed_run": "full-flt-same-head-v3-1", + "observed_workers": 64 + }, + { + "id": "control-bfeba60bbbc6", + "address": "bfeba60bbbc6897cb90ecbff9143e211ee0bcde5ce67f164482d2eb78592ba22", + "category": "control", + "core": false, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "V3 passed at 27023380 fuel / 157.405s; early backoff V5 passed near the cap at 99998101 fuel / 198.240s", + "observed_run": "full-flt-same-head-backoff-v5-1", + "observed_workers": 64 + }, + { + "id": "control-eac5f4ec9db3", + "address": "eac5f4ec9db39307726e86dbdedb8c1c89d69918ce92c971ae7233f950031d44", + "category": "control", + "core": false, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "V3 passed at 1742241 fuel / 5.950s; early backoff V5 passed near the cap at 99999478 fuel / 158.042s", + "observed_run": "full-flt-same-head-backoff-v5-1", + "observed_workers": 64 + }, + { + "id": "tail-04e32656609b", + "address": "04e32656609b0e31aed0fa6707304432c5f27d3b6105267fd9389f8ea6cf0e93", + "category": "tail", + "core": true, + "observed": "unfinished when full run was stopped", + "observed_inflight_seconds": 2104 + }, + { + "id": "control-0139effa7dd4", + "address": "0139effa7dd49f683e998e3e5f2201878fb7c7cd4ddc7d7a76808bb7de66add4", + "category": "control", + "core": true, + "observed": "passed in full 40M run; 7.0s under 64-worker contention" + }, + { + "id": "depth-0dc6387b2eda", + "address": "0dc6387b2eda18aaefad0eda80bc5bd7a62aa8db4c2e1226de871e839dd9b631", + "category": "depth", + "core": true, + "observed": "max recursion depth exceeded" + }, + { + "id": "fuel-0072dda8396f", + "address": "0072dda8396f92d5cc944230c618d0168d168e66f6fece01862f455c3bfa3ea0", + "category": "fuel", + "core": true, + "observed": "recursive fuel exhausted" + }, + { + "id": "tail-083005c8704e", + "address": "083005c8704eb6b9890471b8af71fbe24d18a8af6dc18473b7d4078a1db44ab4", + "category": "tail", + "core": true, + "observed": "unfinished when full run was stopped", + "observed_inflight_seconds": 2092 + }, + { + "id": "control-00023a804270", + "address": "00023a804270e12c13a5f6dca9b717950d6836803716bcd8baeafd69b72f56ec", + "category": "control", + "core": true, + "fuel": 20000000, + "observed": "passed subject-only at 20M and 40M, using 17107271 fuel" + }, + { + "id": "depth-10e4f6ca8c39", + "address": "10e4f6ca8c396e268a791a3f598b25e7a9116964d328638a5dab7c600081d995", + "category": "depth", + "core": true, + "observed": "max recursion depth exceeded" + }, + { + "id": "fuel-00f8aa1e8ed4", + "address": "00f8aa1e8ed4865d7c1a8755e4e36259d425f42edd618b89d348fe38635bb79e", + "category": "fuel", + "core": true, + "observed": "recursive fuel exhausted" + }, + { + "id": "tail-0cb53e4ef3d4", + "address": "0cb53e4ef3d416681b5e41654ec3cac73179d3a0a75b1fcc5cdc71011cb28b85", + "category": "tail", + "core": true, + "observed": "unfinished when full run was stopped", + "observed_inflight_seconds": 2074 + }, + { + "id": "depth-088d4f086b58", + "address": "088d4f086b588e4aa54ac66d1d0407fbc44686a8c162749549ec97d62487d85d", + "category": "depth", + "core": true, + "observed": "max recursion depth exceeded" + }, + { + "id": "fuel-07956a57e7f0", + "address": "07956a57e7f0386045ebdc88c4288d8b8a911657b2f34001973faf8c8868b113", + "category": "fuel", + "core": true, + "observed": "recursive fuel exhausted" + }, + { + "id": "tail-b6d12915d27e", + "address": "b6d12915d27e8d6663ae3839afaa68eb760427d66966936b5ae9068a59f5f177", + "category": "tail", + "core": true, + "observed": "unfinished when full run was stopped", + "observed_inflight_seconds": 1217 + }, + { + "id": "depth-73c0037a49a3", + "address": "73c0037a49a3aa23b52e0604ebb59d96e829b600ebb334c036507ce7bbc7a5e7", + "category": "depth", + "core": true, + "observed": "max recursion depth exceeded" + }, + { + "id": "fuel-03a6d7ba729b", + "address": "03a6d7ba729bb97c8c88eb53403bc52de6e236585ecb4276140be4830daf80d9", + "category": "fuel", + "core": true, + "observed": "recursive fuel exhausted" + }, + { + "id": "tail-dfce3bf04cff", + "address": "dfce3bf04cffd1877f6d43b0749c845d1531ac879aef5d5157c4fd9ece8aaae0", + "category": "tail", + "core": true, + "observed": "unfinished when full run was stopped", + "observed_inflight_seconds": 987 + }, + { + "id": "control-f5e005d2f057", + "address": "f5e005d2f0578605572f5e516761d5b8621445062908398d1ef1db060ec83a32", + "category": "control", + "core": false, + "timeout_seconds": 600, + "observed": "passed in full 40M run after 558s; extended slow positive control" + }, + { + "id": "fuel-07d8b8833a08", + "address": "07d8b8833a087877199dec944d0844b177914c1041d0ce86d82e3099a3254c40", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-0bdaebb92c97", + "address": "0bdaebb92c97340d9d2a21b487d7194e19ee7c7f55cf6afc0ec786bf66c1a490", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-0c37e3af2549", + "address": "0c37e3af25492275f5c24d3fd79807ad3b1d48abd1cad4d67fcf56cc46bd9406", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-0e85ce4d19c8", + "address": "0e85ce4d19c8adf95d112d4d0a9d4a8bd866125574e1d7356a79cbf30d122d5d", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-0ec7030808af", + "address": "0ec7030808af282501c58304828723230d91aa110ed679e7b0c06ece4b6a14c0", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-0f70ae355991", + "address": "0f70ae3559914dbb6808e42969ad2c663348f8601ff0abf1d2fc86458c02ec38", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-11a4ac392437", + "address": "11a4ac39243731c5c45772ba2871cbe58adbd30890c9398b967b2cd85fb62d8e", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-1405b7494f3b", + "address": "1405b7494f3b77d89fb7b3b46f6b6851958a63172afb0cae019b1b25bdd3f067", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-15026274ccb6", + "address": "15026274ccb6ca020522bf2627b0dc5785e5dfc513fcb78e0308e9885872587d", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-16942012be6f", + "address": "16942012be6f4115066ed234661a5f07b30c6d9e2860afe1d13e73818eb285bb", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-17dc3c88aedf", + "address": "17dc3c88aedfae687f6c7cab4a85f674f55d64f50082d58ab6def21fab8113c4", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-1c917fd2e840", + "address": "1c917fd2e840101dadcf6ca57bc03632abc50864351224842c2d9119758f05de", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-1e1a9a86ce35", + "address": "1e1a9a86ce351b16a35be7214bdb8e37889ce935f2ba6b04233daaf1be0f0e3d", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-1e21710daa90", + "address": "1e21710daa9064806b8edc43348aa9e6f7106709f4398b4ace8a9b8e6165625c", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-20a4bd9db684", + "address": "20a4bd9db684fd46437e608e30ceb2777aff674a902cb54ca1a98eee71062be8", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-243116eed67e", + "address": "243116eed67e812348fe49f8e1a2abfee6ca789fbd826e7c7cee72c9108a9a85", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-248d9675e38f", + "address": "248d9675e38fc1f2c391bb9e14f36196b2b5c1ebff2ba16440d357800fd1ee2d", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-26b361908968", + "address": "26b3619089682cc2dce6ec3f40b52de4263a799c27d79fa4dda8f487bfff5c9b", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-28b243bcb6be", + "address": "28b243bcb6be053fa0bcd90924189c586ee7c27abf65ea19b84d84dd2f9259d8", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-2d1b0b1de8a7", + "address": "2d1b0b1de8a7aa2b8c30ea06a26522343c458292d80d3f9b7d88877abfe8d7f0", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-37c291146040", + "address": "37c2911460409e226f182dd2b31d980417b8267844a19e6475cdf29892b7ddd1", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-37d7aafd083d", + "address": "37d7aafd083d02b5884816f3ec04d0d3be6f1655c058471495d8dc026c8354c9", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-3884666acc2b", + "address": "3884666acc2b49fdbe37cd74663040047a74cfa1fe18a3211f2d965cfd2b9860", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-392e414d3364", + "address": "392e414d336442a07fecae22c8f9c65bea3569704a50921ebd75669ec5f2ee96", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-3bb38796c801", + "address": "3bb38796c80106cb5dfafce7149672e141e50819f7040491981532056aa3255f", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-3d76157b6b06", + "address": "3d76157b6b0694ec1413d12ff7a884a1eeee4c45f39a1c2f51336d53ee2be581", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-3fe3373f9792", + "address": "3fe3373f97925d6c1b569f4c4baaab57630b94499d7ef67d60ef90f314f9d758", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-453f9a7935ad", + "address": "453f9a7935ad2940d51f011e8339c4f71137d99c4053f36010af17e26588d321", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-4658e560668d", + "address": "4658e560668d97ca7faa7b0478a1646cdf8f7d1dce5938535a6ba8752a5d224a", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-47a2fc199316", + "address": "47a2fc19931658e010269db4cba32b23e168a2d1d90cbad452a89eb17388a66a", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-4a457de5c3f1", + "address": "4a457de5c3f1d92ed84179931d9870b89b1d3c0b5b66f4857a842479ce669ec6", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-4abd2603d23f", + "address": "4abd2603d23fae7a5428a320d0b5bf62a4ba696e3eeb5319ecfc491744e8d7ee", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-4c85963c570b", + "address": "4c85963c570be2aa1aaf8bb9d1680d327152ad9a1b9793230d1bca25573df4f1", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-4d91369fe012", + "address": "4d91369fe01213d6a64bf38dc9205844d320c46bda8a422f783cd983e87d2fb0", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-52ea9de8ec4a", + "address": "52ea9de8ec4a256a841956b5705d4444168ce75c72375f1f7b4e032ff824031b", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-56d43a04e16e", + "address": "56d43a04e16e99938570317e0f989070515b45ca433fa59b81b22fb0bba7f161", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-57a8b257a0ed", + "address": "57a8b257a0ed572a86ceb2b943c92af6d397aa6787b68da716ac4924bc41b8dc", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-5a43f3a65af9", + "address": "5a43f3a65af97d7f371afdd531fcbf62ce84aa63a98010c5cf63917eb8f9e895", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-5b251facf177", + "address": "5b251facf177fe81ac3c27e07ecf1181c4ad827b87b60f06458ebdaca2a53a79", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-5e22bfd63817", + "address": "5e22bfd638178e11079e7aa119e096f14fa268380e84d34584f99c556afeebe2", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-5efa2221cc6e", + "address": "5efa2221cc6e94ad8220303a881f7745934f8d9d836c553bd6b56ca25a7dfd5c", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-6c43f78d96e1", + "address": "6c43f78d96e1efe2574d5f0527de3fef23c9f67aba636f3d790a48dff29b0ab1", + "category": "fuel", + "core": true, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "sole remaining recursive fuel exhaustion at 100M", + "observed_run": "full-flt-same-head-v3-1", + "observed_workers": 64, + "observed_check_seconds": 237.671428457, + "names": [ + "P2MW.S_AutomorphicForm_exists_elliptic_family_coupled_inf_twistedCentralizer_conjAe_of_neg.HA.main" + ] + }, + { + "id": "fuel-6efa80cf44b0", + "address": "6efa80cf44b037aa38a11589943536e38ef13582425244e364ebb45aaa517602", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-6f1c9f571133", + "address": "6f1c9f5711330a6225278338ffba7a63f16d17ac7103189a23097dc204589251", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-70f279c028f0", + "address": "70f279c028f0fc3838e3e0267a937d5b79cb4a61c619775d5a42731cc2100f22", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-717e97820f76", + "address": "717e97820f764d281a0b9c30b8f667e18cc372cd8cc4b42ee22e3c5c0ea0dd15", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-734d1b55e5d0", + "address": "734d1b55e5d01ea828c6d44c40fd0036d8fd923e7e64ed7a7524bb0b051ec9b7", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-738204d710b0", + "address": "738204d710b03c925845249ac22a2b669fa7185b7c8edda1c99dfa6c8b0852a7", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-77df39414404", + "address": "77df39414404f98008a7342510e3638a00ad8486205abb8b3183c566b8f16f37", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-7a5d012eeef0", + "address": "7a5d012eeef0d74fb5bf95f2463e8c1626d485572d6d849578426f539e0e2e60", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-7c9c03fb378a", + "address": "7c9c03fb378a53f57857293be7586763788ccf0f66c95dd3ae2c688d3711a89e", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-7d8fbfacce07", + "address": "7d8fbfacce07d6f465c705757ef061659cec8d21c23f9246a194582963a10e25", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-7d9bdd35ac9c", + "address": "7d9bdd35ac9caffa4b19487377fcdaee54f5003d60a954149b00cb9a7c3268d8", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-7fc267727e58", + "address": "7fc267727e58541f8e2935f7e27a5f52d405b9aed7f760440f3dbee047dd6524", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-856bccdf8bf9", + "address": "856bccdf8bf98f658cd238e815add6624690a1b01806097321b14f0f81855d89", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-86d58c2a911b", + "address": "86d58c2a911bb4aa6b034082fab6fa98392051818aeb132ec1bdfd19bcc3bb06", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-87a00bd519e8", + "address": "87a00bd519e84e5ea2f19836f98d810075bdbe37656c67ea69d9cf487af1d246", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-87f6482adf99", + "address": "87f6482adf9941d77c8da9548daa8bc48a43d1411cbf9ee7b2109089a193ecee", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-883aee507aa5", + "address": "883aee507aa5ea8c126534ef13bac5d17429a38392db67a71d78c248ac8bef7c", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-8c3e8f09907a", + "address": "8c3e8f09907a174e824b957b6f0c1fcd90882edc5e72d5b0ca765a96dda54787", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-8fd965d4f9e4", + "address": "8fd965d4f9e4fa39e5895b687dfdc7d7908c230950d7da82a52d739a6e2fceca", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-907a79fe0ff5", + "address": "907a79fe0ff545e9d723dda9451096d92b4eea775ff985e672e623a34323cd8f", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-913fac06f1a7", + "address": "913fac06f1a72a97dc2fe180ea166dc62051d0db9e5703be937109846d9e13b6", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-921c16fe1fde", + "address": "921c16fe1fde3397dd9ccdbfedcfd1626cb7dce2342db902fada4b2048a7ec0b", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-98347e9eb5a8", + "address": "98347e9eb5a88f1860b3ce1f72050f30fffb19c0e49b4c386cd098b9147df46f", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-98fb758e8747", + "address": "98fb758e874764e23e78a3ff25cc75341a67ad3ba5eaf299344b380d9c6d495a", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-99ef9a2634d2", + "address": "99ef9a2634d2cbf28eb48c58cae257ddb47a4b9da44cd615c52f15019c1bf9b2", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-9a23b89995c7", + "address": "9a23b89995c7da21c805e6faf2ae6b410cfce5aa8babc34d2a769c0fa6165898", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-a2dd4adb20e5", + "address": "a2dd4adb20e564d0acccda9d646a013a38cd02d4e1265188cf11fb22e41b6af8", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-a6f35da719fc", + "address": "a6f35da719fc4751c751dceb2484cb08e9e3af4846713d3e1e37ec3e6c5e4eec", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-acac0fa9cc2c", + "address": "acac0fa9cc2cd1a0fcf47bcd95426859efecd68ef84a3024443ee00a6d615230", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-b277b4a39efd", + "address": "b277b4a39efdb13d15a0f73a014b19bfdc6e81be49f72dab2365937aa3a60b37", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-b306d39e0a70", + "address": "b306d39e0a70ce7c6bfb5e1d8d7084104d343b89ab73b4d854bb436dbdf26745", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-b369a9877935", + "address": "b369a9877935484b475deea6f3f49ff92be4d572dd095a898355b93bacbecf8a", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-b5f3069e3f7d", + "address": "b5f3069e3f7db33398d175498749e56c4058a34228ae46780c28cb2e5692a45a", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-b73e6f6e6d67", + "address": "b73e6f6e6d6718fdd6a83997bdcebb4394674c5ef674ce8b13901d366ac3f220", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-b7cc5cd0d80d", + "address": "b7cc5cd0d80d4ad134759092f05307917dab7de800bd4a3dc39fed0511bb7031", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-b9a7a817fca6", + "address": "b9a7a817fca69204b12da527e7a432601baec892b8bcaa74a512df8b9d600fd5", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-bc2e15748fee", + "address": "bc2e15748fee9feb1620d6dc28bfa5697404f274301e8aca0243085fb18e090f", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-bcd0b945831a", + "address": "bcd0b945831ac045a5b4fa612eb073f58f8ad46d06fa3692f83617cd05f0b35b", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-be270b7de1ca", + "address": "be270b7de1ca444e256a5fd397f31ea5b5b1ca8f5f371204ae59bf3c128cd00d", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-be9f592e20d8", + "address": "be9f592e20d8fdf5ac96782d99ca2376da2f220c6fe036745e7ea27c448bc870", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-bf72b0f4b4a0", + "address": "bf72b0f4b4a0816dec00259582e0dff373bc17f8a22f295996ecab4c9a06d7de", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-bf8dcd15bed0", + "address": "bf8dcd15bed075968122aa241cc695a2a60467b06115afad08fae8ec1833cd03", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-bffa13764333", + "address": "bffa13764333d0982f132565b2ae6600ea18576061f68c03c1615474db38466b", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-c102cb3dfae3", + "address": "c102cb3dfae3cf8a444d661b9658653966a8cfd2cffc36b78b00ab32290ed807", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-c17f4952807b", + "address": "c17f4952807bbb9c2beca9b5e207437e5153e39a0ddc63e5627faad79f5ae8be", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-c39f0bbaa298", + "address": "c39f0bbaa298f541cffe9e1496e0f776a86ae91c34dc6296db906f3023fef1d7", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-c4d4d0e1afa2", + "address": "c4d4d0e1afa2a965dc6f1244f305a62a5284bcc73688debfd8e4edaff3c06b01", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-c4ebf48fe5f1", + "address": "c4ebf48fe5f1415c82b1377f75f314eb9f65ef774a29da1fe25669b9577d3fd6", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-c5b89e205dc3", + "address": "c5b89e205dc30526a5befae5a9fd673ef6eb7ed96a190ed45ab20f80383c4605", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-c5bd67268ffa", + "address": "c5bd67268ffa182281d6b66b01dc406bf503132b424ceeed62f52213d9fd95b3", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-c94b74124c02", + "address": "c94b74124c0237baf5f51a073b1c6c94e4911bd9a9ac9b1e67a4d058b7d5a200", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-cbb8b729f984", + "address": "cbb8b729f984e651985790e8e4b5a6d1969e91841f7ae02701320e818ca5975b", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-cdef51c9c404", + "address": "cdef51c9c404cf0c29491b78758cac611faeef2915500a5b30a7a2f133d6e075", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-ce7e4b9cb918", + "address": "ce7e4b9cb918acf0314bf48ab879bd8bc258b46a5a139c2f6983dab519092d15", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-d03bdbdfaa1e", + "address": "d03bdbdfaa1eca2764a8a02bc5bf22f4c4f7b3b2254d98597968113f0c2b6158", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-d14e5a6274aa", + "address": "d14e5a6274aac1ce5f670b2c8428e1eca620ade0ca32ba624ee15cb53a0b914e", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-d22080b5eac3", + "address": "d22080b5eac3f3bdf10e4871b20781b828317a55ee1e863ba426761cc0cb349d", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-db7259a54167", + "address": "db7259a54167e37b31dd4f7909a768efbb4c610849063e6674ded5ba6a315d9b", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-de82bec0e810", + "address": "de82bec0e81006cbb6eef366a9cd5dcd00b79d1ebd7f094654da0a149a860266", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-df49324dbaeb", + "address": "df49324dbaeb64f97e0fd0ced5634de9b9eb761429db3143a7c356f313299cd0", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-dfc612609ef2", + "address": "dfc612609ef2a6dceecc24644fd4bce9a9c77ef31ba6fff7a6cf023f3cec3214", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-e05a4fb88aa2", + "address": "e05a4fb88aa2c277b7b1581da3c62edcab966dbb45b6f1bed0642b66f70aa121", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-e6afc96bd8b0", + "address": "e6afc96bd8b0a02f61e146e57d8d10ca143932f7bd1ca1bd6672c48e390c9ef0", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-eb3ed3c3c64a", + "address": "eb3ed3c3c64a507f29f7c118248a4346498f6bb383c4f40f4a9497a1bc514be9", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-eb60cc653dd9", + "address": "eb60cc653dd9521239cd73cdcaef716ce105b23e1ecfb0e2d85ef7e0be39e2c5", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-ed39b646a289", + "address": "ed39b646a2898600c75bc12cfe211c7773abd8e7f981e69f7de5bed2a9979acb", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-f1d5f1580e9c", + "address": "f1d5f1580e9cccb6b8db85fa60e974afb149895208906a205ae473a130d72e69", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-f38456f556fd", + "address": "f38456f556fd08a454d52efe87123261bc2bf1e4b3e09d0fa4fe8d2df8f848bb", + "category": "fuel", + "core": true, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "recursive fuel exhausted at 100M; one of the final five work items", + "observed_run": "check-flt-fuel100m-v1-1", + "observed_workers": 64, + "observed_check_seconds": 136.3, + "names": [ + "P2MW.S_ModularCurve_FullLevel_exists_mem_igusaNodes_over_of_levelAut_transport_linear_nodes_iff_of_eq_two_of_dvd.IgusaNodesE133.exists_transcendental_finiteDimensional_modularFunctionFieldC", + "P2MW.S_ModularCurve_FullLevel_exists_mem_igusaNodes_over_of_levelAut_transport_linear_nodes_iff_of_eq_three_of_dvd.IgusaNodesE133.exists_transcendental_finiteDimensional_modularFunctionFieldC", + "P2MW.S_ModularCurve_FullLevel_exists_mem_igusaNodes_over_of_levelAut_transport_linear_nodes_iff.IgusaNodesE133.exists_transcendental_finiteDimensional_modularFunctionFieldC" + ] + }, + { + "id": "fuel-f59ec053d54b", + "address": "f59ec053d54bc8c62d12c892740dfb505a94ff9be46db030f95aaebccbd825d6", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-f5ebaf348a49", + "address": "f5ebaf348a49a196753e9731393d8e1d31af6ba40d0454cefd5f823e37948776", + "category": "fuel", + "core": true, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "recursive fuel exhausted at 100M; one of the final five work items", + "observed_run": "check-flt-fuel100m-v1-1", + "observed_workers": 64, + "observed_check_seconds": 139.9, + "names": [ + "CerednikDrinfeld.FormalODModule.endAct_varpiEnd_endAct_varpiEnd" + ] + }, + { + "id": "fuel-fa1b74facf9a", + "address": "fa1b74facf9a8d3381d44139e410c2c51b60c76032c1a4ac7f26e74505c0b376", + "category": "fuel", + "core": true, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "recursive fuel exhausted at 100M; one of the final five work items", + "observed_run": "check-flt-fuel100m-v1-1", + "observed_workers": 64, + "observed_check_seconds": 107.9, + "names": [ + "_private.P2M.Sol.S_AutomorphicForm_isAutomorphicFnAt_pseudoEisenstein_slab.0.P2MW.S_AutomorphicForm_isAutomorphicFnAt_pseudoEisenstein_slab.AutomorphicForm.PseudoEisensteinAutomorphy.finite_and_ncard_setOf_bruhatRep_mul_le" + ] + }, + { + "id": "fuel-faf080f4c6dc", + "address": "faf080f4c6dc7b3b3faf00b93ec6c5e3b6271984c852708b5cce1e51900f5e7f", + "category": "fuel", + "core": false, + "observed": "recursive fuel exhausted" + }, + { + "id": "fuel-fbba56420b6b", + "address": "fbba56420b6b657e435567ce3b557fa6b20b8cd1143c55daec90c19854abde32", + "category": "fuel", + "core": true, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "recursive fuel exhausted at 100M; one of the final five work items", + "observed_run": "check-flt-fuel100m-v1-1", + "observed_workers": 64, + "observed_check_seconds": 125.6, + "names": [ + "P2MW.S_CerednikDrinfeld_SpecialFormalODModule_exists_fin_two_endAct_varpiEnd_eq_verschiebung_of_isAlgClosed.P2mKcUnitRootLattice.pi_pi", + "P2MW.S_CerednikDrinfeld_SpecialFormalODModule_exists_fin_two_mem_invariants_forall_existsUnique_eq_sum_smul_of_isCritical.P2mKcCritFibre.pi_pi", + "P2MW.S_CerednikDrinfeld_SpecialFormalODModule_exists_addMonoidHom_cartierModule_injective_of_isAlgClosed.P2mKcBC52Module.pi_pi" + ] + }, + { + "id": "fuel-fdfcfa70a9e8", + "address": "fdfcfa70a9e8645e8a03a0ab5034886bf0b328b796c88a11e3818a44d374741f", + "category": "fuel", + "core": true, + "fuel": 100000000, + "timeout_seconds": 300, + "observed": "recursive fuel exhausted at 100M; last work item to finish", + "observed_run": "check-flt-fuel100m-v1-1", + "observed_workers": 64, + "observed_check_seconds": 146.8, + "names": [ + "P2MW.S_ModularCurve_exists_injective_heckeEquivariant_addMonoidHom_jZero_pic0_complex.ModularCurve.K1BC.restrictAlong_liftPlace" + ] + } + ] +} diff --git a/Cargo.lock b/Cargo.lock index fe8e19385..c8c03e78f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1864,6 +1864,7 @@ dependencies = [ "quickcheck_macros", "rayon", "rustc-hash", + "smallvec", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index e728cb1da..818e14d55 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -66,6 +66,7 @@ rayon = "1" rustc-hash = "2" serde_json = "1" sha2 = "0.10" +smallvec = "1.15.1" tiny-keccak = { version = "2", features = ["keccak"] } tracing = "0.1" tracing-subscriber = { version = "0.3", features = ["env-filter"] } diff --git a/crates/compile/src/compile/aux_gen/recursor.rs b/crates/compile/src/compile/aux_gen/recursor.rs index e1b173642..0e819e216 100644 --- a/crates/compile/src/compile/aux_gen/recursor.rs +++ b/crates/compile/src/compile/aux_gen/recursor.rs @@ -2821,12 +2821,14 @@ mod tests { /// cross-reference the sibling. `all = [A, B]` on both inductives. /// No hand-written recursors — aux_gen generates them. fn build_alpha_collapse_env() -> (LeanEnv, Name, Name) { + build_alpha_collapse_env_named(n("A"), n("B")) + } + + fn build_alpha_collapse_env_named(a: Name, b: Name) -> (LeanEnv, Name, Name) { let hyg = Name::num( Name::str(Name::anon(), "a._@._internal._hyg".into()), Nat::from(0u64), ); - let a = n("A"); - let b = n("B"); let a_ctor = Name::str(a.clone(), "a".into()); let b_ctor = Name::str(b.clone(), "b".into()); let all = vec![a.clone(), b.clone()]; @@ -2906,6 +2908,8 @@ mod tests { (env, a, b) } + /// Presence-only stub for tests that call aux_gen directly. Do not schedule + /// these via compile_env: Sort 0 + no rules has no inductive dependencies. fn insert_aux_stub_rec(env: &mut LeanEnv, all: &[Name], ind: &Name) -> Name { let rec_name = Name::str(ind.clone(), "rec".into()); env.insert( @@ -2929,6 +2933,231 @@ mod tests { rec_name } + /// The uncollapsed recursors Lean generates for the Prop A/B fixture. + /// Transcribed from Lean 4.33.1's `#print A.rec` / `#print B.rec`, not + /// generated by the compiler under test. Both have two dependent Prop + /// motives, two minors, and a rule recursively calling the sibling recursor. + fn build_alpha_collapse_env_with_recursors() -> (LeanEnv, Name, Name) { + build_alpha_collapse_env_with_recursors_named(n("A"), n("B")) + } + + fn build_alpha_collapse_env_with_recursors_named( + a: Name, + b: Name, + ) -> (LeanEnv, Name, Name) { + let (mut env, a, b) = build_alpha_collapse_env_named(a, b); + let a_ctor = Name::str(a.clone(), "a".into()); + let b_ctor = Name::str(b.clone(), "b".into()); + let a_rec = Name::str(a.clone(), "rec".into()); + let b_rec = Name::str(b.clone(), "rec".into()); + let a_c = LeanExpr::cnst(a.clone(), vec![]); + let b_c = LeanExpr::cnst(b.clone(), vec![]); + insert_two_motive_recursors( + &mut env, + &[a.clone(), b.clone()], + [a_rec, b_rec], + [a_c, b_c], + [ + (a_ctor.clone(), LeanExpr::cnst(a_ctor, vec![])), + (b_ctor.clone(), LeanExpr::cnst(b_ctor, vec![])), + ], + &[], + ); + (env, a, b) + } + + /// Shared binder/rule shape of the Lean-transcribed A/B and Tree/Wrap Tree + /// fixtures. `ctor_apps` includes specialized parameters (Wrap.mk Tree). + fn insert_two_motive_recursors( + env: &mut LeanEnv, + all: &[Name], + rec_names: [Name; 2], + major_types: [LeanExpr; 2], + ctor_apps: [(Name, LeanExpr); 2], + level_params: &[Name], + ) { + let [a_rec, b_rec] = rec_names; + let [a_c, b_c] = major_types; + let [(a_ctor, a_ctor_app), (b_ctor, b_ctor_app)] = ctor_apps; + let elim_level = level_params + .first() + .map_or_else(Level::zero, |u| Level::param(u.clone())); + let motive_sort = LeanExpr::sort(elim_level); + let rec_us: Vec<_> = + level_params.iter().cloned().map(Level::param).collect(); + let bv = |i: u64| LeanExpr::bvar(Nat::from(i)); + + // Domains are scoped under the preceding prefix binders: + // {mA : A → Sort u} {mB : B → Sort u} (u = 0 for the Prop fixture) + // (a : ∀ x : B, mB x → mA (A.a x)) + // (b : ∀ x : A, mA x → mB (B.b x)). + let prefix = [ + ( + n("motive_1"), + epi(n("t"), a_c.clone(), motive_sort.clone()), + BinderInfo::Implicit, + ), + ( + n("motive_2"), + epi(n("t"), b_c.clone(), motive_sort), + BinderInfo::Implicit, + ), + ( + n("a"), + epi( + n("x"), + b_c.clone(), + epi( + n("ih"), + LeanExpr::app(bv(1), bv(0)), + LeanExpr::app(bv(3), LeanExpr::app(a_ctor_app, bv(1))), + ), + ), + BinderInfo::Default, + ), + ( + n("b"), + epi( + n("x"), + a_c.clone(), + epi( + n("ih"), + LeanExpr::app(bv(3), bv(0)), + LeanExpr::app(bv(3), LeanExpr::app(b_ctor_app, bv(1))), + ), + ), + BinderInfo::Default, + ), + ]; + + for (rec_name, ctor, major_ty, field_ty, motive_idx, minor_idx, peer) in [ + (&a_rec, &a_ctor, &a_c, &b_c, 4, 2, &b_rec), + (&b_rec, &b_ctor, &b_c, &a_c, 3, 1, &a_rec), + ] { + let typ = prefix.iter().rev().fold( + epi(n("t"), major_ty.clone(), LeanExpr::app(bv(motive_idx), bv(0))), + |body, (name, domain, info)| { + LeanExpr::all(name.clone(), domain.clone(), body, info.clone()) + }, + ); + // λ mA mB a b x, minor x (peer.rec mA mB a b x). + let recursive_call = (0..=4) + .rev() + .fold(LeanExpr::cnst(peer.clone(), rec_us.clone()), |f, i| { + LeanExpr::app(f, bv(i)) + }); + let rhs = LeanExpr::lam( + n("x"), + field_ty.clone(), + LeanExpr::app(LeanExpr::app(bv(minor_idx), bv(0)), recursive_call), + BinderInfo::Default, + ); + let rhs = prefix.iter().rev().fold(rhs, |body, (name, domain, _)| { + LeanExpr::lam(name.clone(), domain.clone(), body, BinderInfo::Default) + }); + env.insert( + rec_name.clone(), + ConstantInfo::RecInfo(RecursorVal { + cnst: ConstantVal { + name: rec_name.clone(), + level_params: level_params.to_vec(), + typ, + }, + all: all.to_vec(), + num_params: Nat::from(0u64), + num_indices: Nat::from(0u64), + num_motives: Nat::from(2u64), + num_minors: Nat::from(2u64), + rules: vec![RecursorRule { + ctor: ctor.clone(), + n_fields: Nat::from(1u64), + rhs, + }], + k: false, + is_unsafe: false, + }), + ); + } + } + + /// Lean 4.33.1: Wrap (α : Type) | mk : α → Wrap α; + /// Tree | node : Wrap Tree → Tree. The nested source peers are + /// Tree.rec, Tree.rec_1, both universe-polymorphic with two motives/minors. + fn build_nested_source_recursors() -> (LeanEnv, Name) { + let mut env = LeanEnv::default(); + let wrap = n("Wrap"); + let tree = n("Tree"); + let wrap_mk = Name::str(wrap.clone(), "mk".into()); + let node = Name::str(tree.clone(), "node".into()); + let tree_c = LeanExpr::cnst(tree.clone(), vec![]); + let wrap_c = LeanExpr::cnst(wrap.clone(), vec![]); + let wrap_tree = LeanExpr::app(wrap_c.clone(), tree_c.clone()); + let type0 = LeanExpr::sort(Level::succ(Level::zero())); + for (name, typ, ctor, params, nested) in [ + (&wrap, epi(n("α"), type0.clone(), type0.clone()), &wrap_mk, 1u64, 0u64), + (&tree, type0.clone(), &node, 0, 1), + ] { + env.insert( + name.clone(), + ConstantInfo::InductInfo(InductiveVal { + cnst: ConstantVal { name: name.clone(), level_params: vec![], typ }, + num_params: Nat::from(params), + num_indices: Nat::from(0u64), + all: vec![name.clone()], + ctors: vec![ctor.clone()], + num_nested: Nat::from(nested), + is_rec: nested != 0, + is_unsafe: false, + is_reflexive: false, + }), + ); + } + let wrap_mk_ty = LeanExpr::all( + n("α"), + type0, + epi( + n("x"), + LeanExpr::bvar(Nat::from(0u64)), + LeanExpr::app(wrap_c, LeanExpr::bvar(Nat::from(1u64))), + ), + BinderInfo::Implicit, + ); + for (name, typ, induct, params) in [ + (&wrap_mk, wrap_mk_ty, &wrap, 1u64), + (&node, epi(n("x"), wrap_tree.clone(), tree_c.clone()), &tree, 0), + ] { + env.insert( + name.clone(), + ConstantInfo::CtorInfo(ConstructorVal { + cnst: ConstantVal { name: name.clone(), level_params: vec![], typ }, + induct: induct.clone(), + cidx: Nat::from(0u64), + num_params: Nat::from(params), + num_fields: Nat::from(1u64), + is_unsafe: false, + }), + ); + } + insert_two_motive_recursors( + &mut env, + std::slice::from_ref(&tree), + [ + Name::str(tree.clone(), "rec".into()), + Name::str(tree.clone(), "rec_1".into()), + ], + [tree_c.clone(), wrap_tree], + [ + (node.clone(), LeanExpr::cnst(node, vec![])), + ( + wrap_mk.clone(), + LeanExpr::app(LeanExpr::cnst(wrap_mk, vec![]), tree_c), + ), + ], + &[n("u")], + ); + (env, tree) + } + fn insert_aux_stub_def(env: &mut LeanEnv, ind: &Name, suffix: &str) -> Name { use ix_common::env::{DefinitionSafety, DefinitionVal, ReducibilityHints}; @@ -3978,55 +4207,244 @@ mod tests { } } - /// 3h. Full compile pipeline for alpha-collapsed recursor aliases. - /// - /// Builds A/B inductives with stub recursors, runs `compile_env`, then - /// verifies their regenerated recursors share one canonical block. + fn assert_source_recursors_check( + source: &LeanEnv, + head: &Name, + expected_peers: &[Name], + ) { + use ix_kernel::{constant::KConst, ingress::lean_ingress, tc::TypeChecker}; + + let mut entries: Vec<_> = + source.iter().map(|(name, ci)| (name.clone(), (*ci).clone())).collect(); + entries.sort_by_key(|(name, _)| name.pretty()); + for reverse in [false, true] { + if reverse { + entries.reverse(); + } + let mut source = LeanEnv::default(); + for (name, ci) in &entries { + source.insert(name.clone(), ci.clone()); + } + let mut kenv = lean_ingress(&source); + let (block, _) = + kenv.consts.iter().find(|(id, _)| &id.name == head).unwrap(); + let peers: Vec<_> = kenv.blocks[block] + .iter() + .filter(|id| matches!(kenv.consts.get(id), Some(KConst::Recr { .. }))) + .map(|id| id.name.clone()) + .collect(); + assert_eq!( + peers, expected_peers, + "recursors must follow source .all, not name or insertion order", + ); + let mut ids: Vec<_> = kenv.consts.keys().cloned().collect(); + ids.sort_by_key(|id| id.name.pretty()); + for id in ids { + TypeChecker::new(&mut kenv).check_const(&id).unwrap_or_else(|e| { + panic!("source fixture {}: {e}", id.name.pretty()) + }); + } + } + } + #[test] - fn test_aux_gen_compile_roundtrip() { + fn test_lean_ingress_checks_source_recursors_in_declaration_order() { + for (a, b) in [(n("A"), n("B")), (n("Z"), n("A"))] { + let (source, _, _) = + build_alpha_collapse_env_with_recursors_named(a.clone(), b.clone()); + assert_source_recursors_check( + &source, + &a, + &[Name::str(a.clone(), "rec".into()), Name::str(b, "rec".into())], + ); + } + } + + #[test] + fn test_lean_ingress_checks_nested_source_recursors() { + let (source, tree) = build_nested_source_recursors(); + assert_source_recursors_check( + &source, + &tree, + &[ + Name::str(tree.clone(), "rec".into()), + Name::str(tree.clone(), "rec_1".into()), + ], + ); + } + + #[test] + fn test_compiled_recursors_keep_meta_and_anon_layout() { use crate::compile::env::compile_env; + use ix_kernel::{ + env::KEnv, + id::KId, + ingress::ixon_ingress, + mode::{Anon, Meta}, + tc::TypeChecker, + }; use std::sync::Arc; - let (mut env, a, b) = build_alpha_collapse_env(); + let (flat, _, _) = build_alpha_collapse_env_with_recursors(); + let (nested, _) = build_nested_source_recursors(); + for source in [flat, nested] { + let source = Arc::new(source); + let stt = compile_env(&source).unwrap(); + assert!(stt.ungrounded.is_empty(), "{:?}", stt.ungrounded); + let (mut meta_env, _intern) = ixon_ingress::(&stt.env).unwrap(); + let mut anon_env = KEnv::::new(); + for (name, _) in source.iter() { + let addr = stt.resolve_addr(name).unwrap(); + // Meta ingress stores canonical representatives, not every source + // alias. Use the actual ingressed KId for this canonical address. + let meta_id = meta_env + .consts + .keys() + .find(|id| id.addr == addr) + .unwrap_or_else(|| { + panic!("missing canonical Meta target for {}", name.pretty()) + }) + .clone(); + TypeChecker::new(&mut meta_env) + .check_const(&meta_id) + .unwrap_or_else(|e| panic!("compiled Meta {}: {e}", name.pretty())); + TypeChecker::new_with_lazy_anon(&mut anon_env, &stt.env) + .check_const(&KId::new(addr, ())) + .unwrap_or_else(|e| panic!("compiled Anon {}: {e}", name.pretty())); + } + } + } - // aux_gen only emits a regenerated `.rec` when the source env already has - // one (gate: `lean_env.get(rec_name).is_some()`). The minimal - // `build_alpha_collapse_env` doesn't add the auxiliary constants Lean - // would normally generate, so insert stub `.rec` entries here. Note: the - // stubs only have to exist for the gate; aux_gen replaces their contents - // with the regenerated value. - let all = vec![a.clone(), b.clone()]; - let _ = insert_aux_stub_rec(&mut env, &all, &a); - let _ = insert_aux_stub_rec(&mut env, &all, &b); + #[test] + fn test_lean_ingress_rejects_recursors_with_swapped_names() { + use ix_kernel::{ingress::lean_ingress, tc::TypeChecker}; + + let (mut source, a, b) = build_alpha_collapse_env_with_recursors(); + let a_rec = Name::str(a, "rec".into()); + let b_rec = Name::str(b, "rec".into()); + let mut left = source.get(&a_rec).unwrap().cloned(); + let mut right = source.get(&b_rec).unwrap().cloned(); + for (ci, name) in [(&mut left, &b_rec), (&mut right, &a_rec)] { + let ConstantInfo::RecInfo(rec) = ci else { unreachable!() }; + rec.cnst.name = name.clone(); + } + source.insert(a_rec.clone(), right); + source.insert(b_rec, left); + let mut kenv = lean_ingress(&source); + let id = kenv.consts.keys().find(|id| id.name == a_rec).unwrap().clone(); + let error = TypeChecker::new(&mut kenv).check_const(&id).unwrap_err(); + assert!(error.to_string().contains("canonical-order mismatch"), "{error}"); + } + + #[test] + fn test_lean_ingress_retains_extra_recursors_for_rejection() { + use ix_kernel::{constant::KConst, ingress::lean_ingress, tc::TypeChecker}; + + let (mut source, a, _) = build_alpha_collapse_env_with_recursors(); + let a_rec = Name::str(a, "rec".into()); + let extra_name = n("UnexpectedRecursor"); + let mut extra = source.get(&a_rec).unwrap().cloned(); + let ConstantInfo::RecInfo(rec) = &mut extra else { unreachable!() }; + rec.cnst.name = extra_name.clone(); + source.insert(extra_name.clone(), extra); + let mut kenv = lean_ingress(&source); + let id = kenv.consts.keys().find(|id| id.name == a_rec).unwrap().clone(); + let KConst::Recr { block, .. } = &kenv.consts[&id] else { unreachable!() }; + assert!(kenv.blocks[block].iter().any(|id| id.name == extra_name)); + let error = TypeChecker::new(&mut kenv).check_const(&id).unwrap_err(); + assert!( + error.to_string().contains("rec_ids/flat count mismatch"), + "{error}" + ); + } + + #[test] + fn test_alpha_collapse_source_recursors_have_structural_dependencies() { + let (env, a, b) = build_alpha_collapse_env_with_recursors(); + for (ind, peer) in [(&a, &b), (&b, &a)] { + let rec_name = Name::str(ind.clone(), "rec".into()); + let rec = env.get(&rec_name).unwrap(); + let ConstantInfo::RecInfo(val) = &*rec else { + panic!("expected a source recursor"); + }; + assert_eq!(val.num_motives, Nat::from(2u64)); + assert_eq!(val.num_minors, Nat::from(2u64)); + assert_eq!(val.rules.len(), 1); + let refs = crate::graph::get_constant_info_references(&rec); + // These must be structural edges from the type/rules, not `.all` + // metadata: the inductive block must finish before either recursor. + assert!(refs.contains(&a)); + assert!(refs.contains(&b)); + assert!(refs.contains(&val.rules[0].ctor)); + assert!(refs.contains(&Name::str(peer.clone(), "rec".into()))); + } + } + + /// 3h. Full compile pipeline for alpha-collapsed recursor aliases. + /// + /// Compile genuine uncollapsed recursors and check their canonical content, + /// not just alias equality (two compiled placeholders could compare equal). + #[test] + fn test_aux_gen_compile_roundtrip() { + use crate::compile::{CompileOptions, env::compile_env_with_options}; + use ix_kernel::{env::KEnv, id::KId, mode::Anon, tc::TypeChecker}; + use ixon::constant::ConstantInfo as IxonCI; + use std::sync::Arc; + let (env, a, b) = build_alpha_collapse_env_with_recursors(); let lean_env = Arc::new(env); + let a_rec = Name::str(a.clone(), "rec".into()); + let b_rec = Name::str(b.clone(), "rec".into()); + let mut canonical_addrs = Vec::new(); - // Compile. - let stt = compile_env(&lean_env) + for max_workers in [1, 4] { + let stt = compile_env_with_options( + &lean_env, + CompileOptions { max_workers: Some(max_workers) }, + ) .expect("compile_env should succeed for alpha-collapse inductives"); + assert!( + stt.ungrounded.is_empty(), + "compile_env must not silently return partial results: {:?}", + stt.ungrounded, + ); - // Verify A.rec was compiled. - let has_name = |n: &Name| stt.resolve_addr(n).is_some(); - let a_rec = Name::str(a.clone(), "rec".into()); - assert!(has_name(&a_rec), "A.rec should be compiled"); + let a_addr = stt.resolve_addr(&a_rec).expect("A.rec should be compiled"); + let b_addr = stt.resolve_addr(&b_rec).expect("B.rec should be compiled"); + assert_eq!(a_addr, b_addr, "alpha-equivalent recursors must alias"); + assert!(stt.aux_gen_extra_names.contains(&a_rec)); + assert!(stt.aux_gen_extra_names.contains(&b_rec)); + + // The collapsed singleton is a standalone recursor with 1+1 binders, + // not the original mutual's 2+2, nor the old stub's 0+0. + let compiled = stt.env.get_const(&a_addr).unwrap(); + let IxonCI::Recr(rec) = &compiled.info else { + panic!("expected a standalone canonical recursor"); + }; + assert_eq!( + (rec.params, rec.indices, rec.motives, rec.minors), + (0, 0, 1, 1) + ); + assert_eq!(rec.rules.len(), 1); + assert_eq!(rec.rules[0].fields, 1); + + // Validate the regenerated type and rule RHS as well as their shape. + // Check every source name's canonical target, including the inductives + // and constructors, so dependencies are checked rather than trusted. + let mut kenv = KEnv::::new(); + let mut tc = TypeChecker::new_with_lazy_anon(&mut kenv, &stt.env); + for (name, _) in lean_env.iter() { + let addr = stt.resolve_addr(name).unwrap(); + tc.check_const(&KId::new(addr, ())).unwrap_or_else(|e| { + panic!("compiled fixture {}: {e}", name.pretty()) + }); + } + canonical_addrs.push(a_addr); + } - // B.rec should also be registered (as an alias to the same canonical content). - let b_rec = Name::str(b.clone(), "rec".into()); - assert!(has_name(&b_rec), "B.rec should be compiled"); - - // Note: .below, .brecOn, .casesOn, and .recOn are only generated if the - // original Lean env contains them (same gate as `.rec`). This minimal - // test env doesn't add those, so they aren't generated. - // Full-environment tests (lake test -- rust-compile) exercise that path. - - // Verify A.rec and B.rec resolve to the same underlying Ixon block. - // Both are alpha-equivalent, so their compiled block addresses should - // be identical (they share the same RPrj/singleton block). - let a_addr = stt.resolve_addr(&a_rec).unwrap(); - let b_addr = stt.resolve_addr(&b_rec).unwrap(); assert_eq!( - a_addr, b_addr, - "A.rec and B.rec should point to the same compiled block (alpha-equivalent)" + canonical_addrs[0], canonical_addrs[1], + "canonical recursor content must not depend on worker count", ); } diff --git a/crates/ffi/examples/check_anon_subject.rs b/crates/ffi/examples/check_anon_subject.rs new file mode 100644 index 000000000..39e6eccc8 --- /dev/null +++ b/crates/ffi/examples/check_anon_subject.rs @@ -0,0 +1,308 @@ +//! Bounded-run building block for profiling ONE anonymous work item by primary +//! address. Dependencies are lazily ingressed but trusted, exactly as in one +//! work item of `ix check-rs --anon`. This is NOT corpus/closure verification. +//! +//! cargo run --release -p ix-ffi --example check_anon_subject -- FILE.ixe HEX +//! cargo run --release -p ix-ffi --example check_anon_subject -- --resolve FILE.ixe HEX... +//! +//! `--resolve` only enumerates work: map target addresses (including non-primary +//! block members) to their primary, without checking anything or loading names. +//! +//! Run under an external timeout/memory limit. IX_MAX_REC_FUEL and the existing +//! kernel diagnostic variables are honored. A fresh process gives a fresh +//! KEnv and avoids carrying worker-history caches between samples. +//! +//! Diagnostic-only runs: `IX_PERF_COUNTERS=1` prints cache hit rates; +//! `IX_REDUCE_HISTO=1` prints the top 20 delta/iota addresses and totals. +//! `IX_SAME_HEAD_PROFILE=1` reports actual same-head attempts and their fuel. +//! `IX_HOT_MISSES=1` prints miss shapes once at completion; optional +//! `IX_HOT_MISS_CTX=1` includes their context identities. +//! Collection is bounded to 4,096 keys; reported counts are intervals after +//! low-frequency entries are replaced, not exact per-key totals. +//! Reports go to stderr after checking; stdout's subject JSON is unchanged. +//! Leave these flags unset for paired benchmark timings. + +use std::{ + collections::{HashMap, HashSet}, + path::Path, + process::ExitCode, + sync::atomic::Ordering, + time::Instant, +}; + +use ix_common::address::Address; +use ix_kernel::{ + anon_work::{AnonWorkItem, build_anon_work}, + env::KEnv, + id::KId, + mode::Anon, + tc::TypeChecker, +}; +use ixon::env::Env; + +// Match the native ix executable, without calling its Lean FFI entrypoints. +#[global_allocator] +static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc; + +#[derive(Debug, PartialEq, Eq)] +struct ResolvedSubject { + requested: Address, + primary: Address, + targets: usize, +} + +fn resolve_subjects( + work: &[AnonWorkItem], + requested: &[Address], +) -> Result, String> { + let wanted: HashSet<_> = requested.iter().cloned().collect(); + let mut found = HashMap::new(); + for item in work { + for target in item.targets() { + if wanted.contains(target) { + found.insert(target.clone(), (item.primary(), item.targets().len())); + } + } + } + requested + .iter() + .map(|addr| { + let (primary, targets) = found.get(addr).ok_or_else(|| { + format!("{} is not a kernel-checkable target address", addr.hex()) + })?; + Ok(ResolvedSubject { + requested: addr.clone(), + primary: (*primary).clone(), + targets: *targets, + }) + }) + .collect() +} + +fn run(args: &[String]) -> Result { + if args.first().is_some_and(|arg| arg == "--resolve") && args.len() >= 3 { + let requested: Vec<_> = args[2..] + .iter() + .map(|arg| { + Address::from_hex(arg).ok_or_else(|| format!("invalid address: {arg}")) + }) + .collect::>()?; + let env = Env::get_anon_mmap(Path::new(&args[1]))?; + let work = build_anon_work(&env)?; + let resolved = resolve_subjects(&work, &requested)?; + let rows: Vec<_> = resolved + .iter() + .map(|row| { + serde_json::json!({ + "requested": row.requested.hex(), "primary": row.primary.hex(), + "targets": row.targets, + }) + }) + .collect(); + println!( + "{}", + serde_json::json!({ + "scope": "index-only", "resolutions": rows, + }) + ); + return Ok(true); + } + if args.len() != 2 || args[0] == "--resolve" { + return Err("usage: check_anon_subject FILE.ixe PRIMARY_HEX\n check_anon_subject --resolve FILE.ixe TARGET_HEX...\nSubject-only profiling: dependencies are trusted, not checked.".to_owned()); + } + let primary = Address::from_hex(&args[1]) + .ok_or_else(|| format!("invalid primary address: {}", args[1]))?; + check(&args[0], &primary) +} + +fn check(path: &str, primary: &Address) -> Result { + let start = Instant::now(); + let env = Env::get_anon_mmap(Path::new(path))?; + let load_secs = start.elapsed().as_secs_f64(); + let start = Instant::now(); + let work = build_anon_work(&env)?; + let item = + work.iter().find(|item| item.primary() == primary).ok_or_else(|| { + format!("{} is not a work-item primary address", primary.hex()) + })?; + let targets = item.targets().len(); + eprintln!( + "[subject] primary={} targets={targets} scope=subject-only workers=1 fuel_cap_per_member={} load={load_secs:.3}s enumerate={:.3}s", + primary.hex(), + ix_kernel::tc::max_rec_fuel(), + start.elapsed().as_secs_f64() + ); + let mut kenv = KEnv::::new(); + let _ = ix_kernel::profile::take_op_counts(); + ix_kernel::perf::same_head::reset(); + let start = Instant::now(); + let (result, last_member_fuel, peak_def_eq_depth, hot_misses) = { + let mut tc = TypeChecker::new_with_lazy_anon(&mut kenv, &env); + tc.set_debug_label(format!("#{}", primary.hex())); + let result = tc.check_const(&KId::new(primary.clone(), ())); + let fuel = tc.fuel_used(); + let peak = tc.def_eq_peak; + // TypeChecker has no Drop accounting. Flush the final member explicitly, + // after capturing its allowance and before discarding the checker. + tc.finish_constant_accounting(); + (result, fuel, peak, tc.hot_miss_summary()) + }; + let check_secs = start.elapsed().as_secs_f64(); + let ops = ix_kernel::profile::take_op_counts(); + let aggregate_fuel = ix_kernel::perf::enabled() + .then(|| kenv.perf.total_rec_fuel_used.load(Ordering::Relaxed)); + let report = serde_json::json!({ + "primary": primary.hex(), "scope": "subject-only", "targets": targets, + "passed": result.is_ok(), "error": result.as_ref().err().map(ToString::to_string), + "load_secs": load_secs, "check_secs": check_secs, + "fuel_cap_per_member": ix_kernel::tc::max_rec_fuel(), + "last_member_fuel": last_member_fuel, + // Null unless IX_PERF_COUNTERS is enabled; do not label the final member's + // budget as the total fuel of a multi-member work item. + "aggregate_fuel": aggregate_fuel, "last_member_def_eq_peak": peak_def_eq_depth, + "subst": ops.subst_nodes, "whnf": ops.whnf_calls, + "def_eq": ops.def_eq_calls, "intern": ops.intern_nodes, + "nat_arith": ops.nat_arith, + }); + println!("{report}"); + eprint!("{hot_misses}"); + eprint!("{}", ix_kernel::perf::same_head::summary()); + if ix_kernel::perf::enabled() { + // The example does not install a log backend, so KEnv's log::info! + // drop summary would otherwise be invisible. No checker work is rerun. + eprint!("{}", kenv.perf.summary()); + } + if ix_kernel::perf::reduce_histo_enabled() { + print_reductions( + "delta", + ix_kernel::perf::DELTA_HISTO + .iter() + .map(|entry| (entry.key().clone(), *entry.value())) + .collect(), + ); + print_reductions( + "iota", + ix_kernel::perf::IOTA_HISTO + .iter() + .map(|entry| (entry.key().clone(), *entry.value())) + .collect(), + ); + eprintln!( + "[reduce-histo] nat_succ_peels={}", + ix_kernel::perf::NAT_SUCC_PEELS.load(Ordering::Relaxed) + ); + } + Ok(result.is_ok()) +} + +const REDUCTION_REPORT_LIMIT: usize = 20; + +fn top_reductions( + mut entries: Vec<(Address, u64)>, +) -> (u128, usize, Vec<(Address, u64)>) { + let total = entries.iter().map(|(_, n)| u128::from(*n)).sum(); + let distinct = entries.len(); + entries.sort_unstable_by(|(a, x), (b, y)| y.cmp(x).then_with(|| a.cmp(b))); + entries.truncate(REDUCTION_REPORT_LIMIT); + (total, distinct, entries) +} + +fn print_reductions(label: &str, entries: Vec<(Address, u64)>) { + let (total, distinct, top) = top_reductions(entries); + eprintln!( + "[reduce-histo] {label}: {total} reductions across {distinct} addresses; top {}", + top.len() + ); + for (addr, count) in top { + eprintln!("[reduce-histo] {label} {count} #{}", addr.hex()); + } +} + +fn main() -> ExitCode { + let args: Vec<_> = std::env::args().skip(1).collect(); + // Match the CLI's dedicated worker stack, not the process main stack. + let worker = std::thread::Builder::new() + .name("ix-kernel-subject".to_owned()) + .stack_size(256 * 1024 * 1024) + .spawn(move || run(&args)); + match worker { + Ok(worker) => match worker.join() { + Ok(Ok(true)) => ExitCode::SUCCESS, + Ok(Ok(false)) => ExitCode::FAILURE, + Ok(Err(error)) => { + eprintln!("{error}"); + ExitCode::from(2) + }, + Err(_) => { + eprintln!("subject worker panicked"); + ExitCode::from(2) + }, + }, + Err(error) => { + eprintln!("cannot start subject worker: {error}"); + ExitCode::from(2) + }, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn reduction_report_bounds_rows_but_preserves_complete_totals() { + let entries: Vec<_> = + (0..25u64).map(|n| (Address::hash(&n.to_le_bytes()), n + 1)).collect(); + let (total, distinct, top) = top_reductions(entries); + assert_eq!(total, 325); + assert_eq!(distinct, 25); + assert_eq!(top.len(), REDUCTION_REPORT_LIMIT); + assert_eq!(top.first().unwrap().1, 25); + assert_eq!(top.last().unwrap().1, 6); + } + + #[test] + fn reduction_report_handles_ties_empty_input_and_wide_totals() { + assert_eq!(top_reductions(Vec::new()), (0, 0, Vec::new())); + let a = Address::hash(b"a"); + let b = Address::hash(b"b"); + let (total, distinct, top) = + top_reductions(vec![(b.clone(), u64::MAX), (a.clone(), u64::MAX)]); + assert_eq!(total, 2 * u128::from(u64::MAX)); + assert_eq!(distinct, 2); + assert!(top[0].0 < top[1].0); + assert_eq!(top_reductions(vec![(a, u64::MAX), (b, u64::MAX)]).2, top); + } + + #[test] + fn resolves_members_to_primary_preserving_request_order() { + let a = Address::hash(b"standalone"); + let b = Address::hash(b"primary"); + let c = Address::hash(b"member"); + let work = vec![ + AnonWorkItem::Standalone { addr: a.clone() }, + AnonWorkItem::Block { + block_addr: Address::hash(b"block"), + primary: b.clone(), + targets: vec![b.clone(), c.clone()], + }, + ]; + let rows = + resolve_subjects(&work, &[c.clone(), a.clone(), b.clone()]).unwrap(); + assert_eq!( + rows, + vec![ + ResolvedSubject { requested: c, primary: b.clone(), targets: 2 }, + ResolvedSubject { requested: a.clone(), primary: a, targets: 1 }, + ResolvedSubject { requested: b.clone(), primary: b, targets: 2 }, + ] + ); + } + + #[test] + fn missing_address_is_an_error_not_a_partial_success() { + let a = Address::hash(b"present"); + let work = vec![AnonWorkItem::Standalone { addr: a.clone() }]; + assert!(resolve_subjects(&work, &[a, Address::hash(b"absent")]).is_err()); + } +} diff --git a/crates/ffi/examples/resolve_anon_names.rs b/crates/ffi/examples/resolve_anon_names.rs new file mode 100644 index 000000000..1116a4105 --- /dev/null +++ b/crates/ffi/examples/resolve_anon_names.rs @@ -0,0 +1,37 @@ +//! Diagnostic name lookup without decoding expression metadata or checking. +//! cargo run --release -p ix-ffi --example resolve_anon_names -- FILE.ixe PREFIX... + +use std::process::ExitCode; + +fn run() -> Result<(), String> { + let args: Vec<_> = std::env::args().skip(1).collect(); + if args.len() < 2 + || args[1..].iter().any(|p| { + p.len() < 8 || p.len() > 64 || !p.bytes().all(|c| c.is_ascii_hexdigit()) + }) + { + return Err( + "usage: resolve_anon_names FILE.ixe HEX_PREFIX... (8–64 hex digits)" + .into(), + ); + } + let bytes = std::fs::read(&args[0]).map_err(|e| e.to_string())?; + let index = ixon::env::Env::parse_lazy_index(&bytes)?; + for entry in &index.named { + let hex = entry.addr.hex(); + if args[1..].iter().any(|p| hex.starts_with(&p.to_ascii_lowercase())) { + println!("{} {} {:?}", hex, entry.name, entry.hints); + } + } + Ok(()) +} + +fn main() -> ExitCode { + match run() { + Ok(()) => ExitCode::SUCCESS, + Err(e) => { + eprintln!("{e}"); + ExitCode::FAILURE + }, + } +} diff --git a/crates/ffi/src/kernel.rs b/crates/ffi/src/kernel.rs index 1786bcf52..02525767a 100644 --- a/crates/ffi/src/kernel.rs +++ b/crates/ffi/src/kernel.rs @@ -1409,6 +1409,14 @@ fn run_anon_checks_parallel( eprintln!( "[rs_kernel_check_anon] checking {work_total} work item(s) for {total} consts with {worker_count} worker(s)..." ); + // Opt-in native throughput experiment: reuse bucket capacity, never + // logical cache entries. Zero preserves the existing release policy. + let retain_capacity = env_usize("IX_KERNEL_CHECK_RETAIN_CAPACITY", 0); + if retain_capacity > 0 { + eprintln!( + "[rs_kernel_check_anon] cache reset: retaining at most {retain_capacity} entries of empty capacity per cleared table" + ); + } // Per-work-item attribution entries (addr-keyed CSV; the CLI joins // Lean names afterwards). An entry measures checking THAT item alone — // one constant, or one whole Muts block — NOT re-checking its @@ -1463,7 +1471,11 @@ fn run_anon_checks_parallel( } let item = &work[work_idx]; if checks_since_clear >= clear_every { - kenv.clear_releasing_memory(); + if retain_capacity == 0 { + kenv.clear_releasing_memory(); + } else { + kenv.clear_with_capacity_limit(retain_capacity); + } checks_since_clear = 0; } let (primary_addr, result_idxs): (Address, Vec) = match item { diff --git a/crates/kernel/Cargo.toml b/crates/kernel/Cargo.toml index edaef73dd..e61c721a4 100644 --- a/crates/kernel/Cargo.toml +++ b/crates/kernel/Cargo.toml @@ -15,6 +15,7 @@ itertools = { workspace = true } log = { workspace = true } num-bigint = { workspace = true } rustc-hash = { workspace = true } +smallvec = { workspace = true } [target.'cfg(not(target_arch = "riscv64"))'.dependencies] dashmap = { workspace = true, features = ["rayon"] } diff --git a/crates/kernel/src/check.rs b/crates/kernel/src/check.rs index dd84baa88..e6a71854c 100644 --- a/crates/kernel/src/check.rs +++ b/crates/kernel/src/check.rs @@ -1519,6 +1519,34 @@ mod tests { assert_eq!(tc.def_eq_peak, 0); } + #[test] + fn bounded_capacity_reset_matches_release_across_success_and_failure() { + for capacity in [0, 16, 4_096] { + let mut retained = KEnv::::new(); + let mut released = KEnv::::new(); + // Reuse the same fixture names after both successful and failed checks. + // Reset must not turn either a cached success or a cached error into the + // next check's result, including when empty bucket storage survives. + for name in ["id", "wrong", "Nat", "nonexistent", "wrong", "id"] { + retained.clear_with_capacity_limit(capacity); + released.clear_releasing_memory(); + for env in [&mut retained, &mut released] { + for (id, c) in test_env().iter() { + env.insert(id.clone(), c.clone()); + } + } + let run = |env: &mut KEnv| { + let mut tc = TypeChecker::new(env); + let result = tc.check_const(&mk_id(name)).map_err(|e| e.to_string()); + (result, tc.fuel_used()) + }; + let expected = run(&mut released); + assert_eq!(expected.0.is_ok(), matches!(name, "id" | "Nat")); + assert_eq!(run(&mut retained), expected); + } + } + } + // ========================================================================= // Theorem must land in Prop // ========================================================================= diff --git a/crates/kernel/src/def_eq.rs b/crates/kernel/src/def_eq.rs index 7d8f84d5f..7e04e0170 100644 --- a/crates/kernel/src/def_eq.rs +++ b/crates/kernel/src/def_eq.rs @@ -19,10 +19,18 @@ use super::level::{KUniv, univ_eq}; use super::mode::KernelMode; use super::subst::{instantiate_rev, lift}; use super::tc::{ - MAX_DEF_EQ_DEPTH, MAX_WHNF_FUEL, TypeChecker, collect_app_spine, + MAX_DEF_EQ_DEPTH, MAX_WHNF_FUEL, TypeChecker, app_head, borrow_app_spine, + collect_app_spine, }; use super::whnf::PrimFamily; +mod application; +mod binders; +mod speculation; +pub(crate) use speculation::SameHeadBackoff; +#[cfg(test)] +mod same_head_tests; + /// When set, trace every `is_def_eq` call where one side's head constant /// starts with the prefix in `IX_DEF_EQ_TRACE` (e.g. `IX_DEF_EQ_TRACE=bmod` /// to watch all `Int.bmod`-involving comparisons). Prints `[deq] a b` @@ -45,6 +53,13 @@ static IX_DEF_EQ_COUNT_LOG: crate::EnvFlag = static IX_DEF_EQ_MAX_DUMP: crate::EnvString = crate::EnvString::new(|| crate::env_var("IX_DEF_EQ_MAX_DUMP").ok()); +/// Print at most 96 cache-missing pairs near the depth guard. Identity and +/// mode/context fields distinguish a repeated state from a long descent. +static IX_DEF_EQ_NEAR_GUARD: crate::EnvFlag = + crate::EnvFlag::new(|| crate::env_var("IX_DEF_EQ_NEAR_GUARD").is_ok()); +static NEAR_GUARD_COUNT: std::sync::atomic::AtomicUsize = + std::sync::atomic::AtomicUsize::new(0); + static IX_ETA_TRACE: crate::EnvString = crate::EnvString::new(|| crate::env_var("IX_ETA_TRACE").ok()); @@ -54,12 +69,22 @@ static IX_PROJ_DELTA_TRACE: crate::EnvString = static DEF_EQ_COUNT: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); -/// Non-Regular same-head comparisons are speculative: a failed attempt must -/// not be allowed to consume the entire constant-check budget before ordinary +/// Same-head comparisons are speculative for every reducibility hint: a miss +/// must not consume the entire constant-check budget before ordinary /// delta reduction gets its turn. const SAME_HEAD_SPECULATION_ATTEMPT_FUEL: u64 = 4_096; +/// Regular congruence can profitably compare larger arguments (a Mathlib +/// regression needs about 76k fuel). Leave headroom without allowing a single +/// speculative comparison to consume the entire 100M check allowance. +const SAME_HEAD_REGULAR_ATTEMPT_FUEL: u64 = 131_072; +/// Only non-Regular probes have a per-constant startup window. Regular +/// congruence remains useful throughout a check, but each attempt is bounded. const SAME_HEAD_SPECULATION_START_FUEL: u64 = 16_384; +/// Try comparing the requested fields before comparing whole records, without +/// letting an unsuccessful probe starve the ordinary conversion algorithm. +const PROJECTION_PROBE_FUEL: u64 = 4_096; + /// Step journal (`IX_STEP_TRACE=1`): one `[deq] ~ ` line /// per `is_def_eq` entry (plus `[whnf+]` lines in whnf.rs), mirroring the /// Lean kernel's `IX_TC_STEP_TRACE` journal (`Ix.Tc` / `TcM.stepTrace`). @@ -215,6 +240,27 @@ impl TypeChecker<'_, M> { self.env.perf.record_def_eq_miss(); self.record_hot_def_eq_miss(a, b); + if *IX_DEF_EQ_NEAR_GUARD + && self.def_eq_depth >= MAX_DEF_EQ_DEPTH.saturating_sub(32) + && self.debug_label_matches_env() + && NEAR_GUARD_COUNT.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + < 96 + { + eprintln!( + "[deq near guard] depth={} local={} ctx={} cheap={} infer_only={} eager={} a={} {} b={} {}", + self.def_eq_depth, + self.depth(), + eq_ctx, + self.cheap_recursion_depth, + self.infer_only, + self.eager_reduce, + a.hash_key(), + compact_def_eq_expr(a), + b.hash_key(), + compact_def_eq_expr(b), + ); + } + // Charge recursive fuel only after the O(1) exits above. Large proof // terms can perform hundreds of thousands of pointer/equiv/cache hits; // those should not consume the same budget as an actual comparison. @@ -228,6 +274,7 @@ impl TypeChecker<'_, M> { self.def_eq_peak = self.def_eq_depth; } if self.def_eq_depth > MAX_DEF_EQ_DEPTH { + self.dump_guard_stack("def-eq-depth"); self.def_eq_depth -= 1; self.dump_def_eq_max("depth", a, b, None, None); return Err(TcError::MaxRecDepth); @@ -237,6 +284,11 @@ impl TypeChecker<'_, M> { self.def_eq_depth -= 1; let ok = result?; + // Some optional reducers treat an inner error as a miss. Do not let an + // exhausted speculative slice escape that way as a cached inequality. + if !ok && self.rec_fuel == 0 { + return Err(TcError::MaxRecFuel); + } if trace_active { log::info!( "[deq] depth={} -> {} ({})", @@ -358,18 +410,94 @@ impl TypeChecker<'_, M> { return Ok(true); } + if let Some(result) = self.def_eq_lazy_delta(a, b, &mut wa, &mut wb)? { + return Ok(result); + } + + if self.def_eq_trace_depth > 0 { + log::info!("[deq tier4 break] depth={}", self.def_eq_depth); + log::info!(" wa: {wa}"); + log::info!(" wb: {wb}"); + } + + // Tier 4b: post-delta congruence checks (lean4lean isDefEqConst/Fvar/Proj) + if self.try_structural_congruence(&wa, &wb)? { + return Ok(true); + } + + // Tier 4c: second structural pass (lean4lean:683-686, lean4 + // type_checker.cpp:1109-1110). This is deliberately `whnfCore`, not full + // `whnf`: full WHNF would delta-unfold stuck open primitives such as + // `Nat.ble` and can literally walk enormous Nat literals in their + // recursive logical models. + let wa_core = self.whnf_core(&wa)?; + let wb_core = self.whnf_core(&wb)?; + let wa_changed = + !wa_core.ptr_eq(&wa) && wa_core.hash_key() != wa.hash_key(); + let wb_changed = + !wb_core.ptr_eq(&wb) && wb_core.hash_key() != wb.hash_key(); + if wa_changed || wb_changed { + return self.is_def_eq(&wa_core, &wb_core); + } + let wa = wa_core; + let wb = wb_core; + if wa.ptr_eq(&wb) { + return Ok(true); + } + if self.quick_def_eq(&wa, &wb)? { + return Ok(true); + } + + // Tier 4d: app spine comparison (lean4lean isDefEqApp, lean4 type_checker.cpp:1115) + if self.try_def_eq_app(&wa, &wb)? { + return Ok(true); + } + + let result = self.is_def_eq_whnf(&wa, &wb); + + // Tier 5 final-fail trace: when IX_DEF_EQ_TIER5_DUMP is set and the + // pair's head names contain the configured substring, dump the + // post-whnfCore wa/wb. This is where lazy-delta + Tier 4c gave up. + if let Ok(prefix) = crate::env_var("IX_DEF_EQ_TIER5_DUMP") + && let Ok(false) = result.as_ref() + { + let a_match = head_const_name(&wa).is_some_and(|n| n.contains(&prefix)); + let b_match = head_const_name(&wb).is_some_and(|n| n.contains(&prefix)); + if prefix.is_empty() || a_match || b_match { + log::info!("[deq tier5 fail] depth={}", self.def_eq_depth); + log::info!(" wa: {wa}"); + log::info!(" wb: {wb}"); + } + } + + result + } + + /// Keep lazy-delta temporaries out of the native frame retained while + /// recursively comparing irreducible applications. In unoptimized builds + /// those temporaries otherwise occupy many KiB at EVERY application level, + /// even when neither head can unfold. Reduction order/fuel are unchanged. + #[inline] + fn def_eq_lazy_delta( + &mut self, + a: &KExpr, + b: &KExpr, + wa: &mut KExpr, + wb: &mut KExpr, + ) -> Result, TcError> { // Tier 4: iterative lazy delta (lean4lean lazyDeltaReduction) let mut fuel = MAX_WHNF_FUEL; loop { if fuel == 0 { - self.dump_def_eq_max("fuel", a, b, Some(&wa), Some(&wb)); + self.dump_guard_stack("def-eq-lazy-delta-fuel"); + self.dump_def_eq_max("fuel", a, b, Some(wa), Some(wb)); return Err(TcError::MaxRecDepth); } fuel -= 1; // M2: Nat offset reduction at top of loop (lean4lean isDefEqOffset) - if let Some(result) = self.try_def_eq_offset(&wa, &wb)? { - return Ok(result); + if let Some(result) = self.try_def_eq_offset(wa, wb)? { + return Ok(Some(result)); } // Nat primitive reduction inside lazy delta. Mirrors lean4 @@ -378,18 +506,18 @@ impl TypeChecker<'_, M> { // primitives entirely when either side has a free variable, unless // eagerReduce is active. let nat_ok = (!wa.has_fvars() && !wb.has_fvars()) || self.eager_reduce; - let fam_a = self.head_prim_family(&wa); - let fam_b = self.head_prim_family(&wb); + let fam_a = self.head_prim_family(wa); + let fam_b = self.head_prim_family(wb); if nat_ok { if fam_a == PrimFamily::Nat - && let Some(wa2) = self.try_reduce_nat(&wa)? + && let Some(wa2) = self.try_reduce_nat(wa)? { - return self.is_def_eq(&wa2, &wb); + return self.is_def_eq(&wa2, wb).map(Some); } if fam_b == PrimFamily::Nat - && let Some(wb2) = self.try_reduce_nat(&wb)? + && let Some(wb2) = self.try_reduce_nat(wb)? { - return self.is_def_eq(&wa, &wb2); + return self.is_def_eq(wa, &wb2).map(Some); } } @@ -399,29 +527,29 @@ impl TypeChecker<'_, M> { // Ix-specific `try_reduce_decidable` runs after native to keep the // reference-aligned segment tight. if fam_a == PrimFamily::Native - && let Some(wa2) = self.try_reduce_native(&wa)? + && let Some(wa2) = self.try_reduce_native(wa)? { - return self.is_def_eq(&wa2, &wb); + return self.is_def_eq(&wa2, wb).map(Some); } if fam_b == PrimFamily::Native - && let Some(wb2) = self.try_reduce_native(&wb)? + && let Some(wb2) = self.try_reduce_native(wb)? { - return self.is_def_eq(&wa, &wb2); + return self.is_def_eq(wa, &wb2).map(Some); } if fam_a == PrimFamily::Decidable - && let Some(wa2) = self.try_reduce_decidable(&wa)? + && let Some(wa2) = self.try_reduce_decidable(wa)? { - return self.is_def_eq(&wa2, &wb); + return self.is_def_eq(&wa2, wb).map(Some); } if fam_b == PrimFamily::Decidable - && let Some(wb2) = self.try_reduce_decidable(&wb)? + && let Some(wb2) = self.try_reduce_decidable(wb)? { - return self.is_def_eq(&wa, &wb2); + return self.is_def_eq(wa, &wb2).map(Some); } - let a_head = head_const_id(&wa); - let b_head = head_const_id(&wb); + let a_head = head_const_id(wa); + let b_head = head_const_id(wb); let a_delta = match &a_head { Some(h) => self.is_delta(h)?, None => false, @@ -438,15 +566,15 @@ impl TypeChecker<'_, M> { // C6: Before unfolding a definition, try reducing projection apps // on the non-definition side (lean4lean tryUnfoldProjApp). if a_delta && !b_delta { - if let Some(wb2) = self.try_unfold_proj_app(&wb)? { - wb = wb2; + if let Some(wb2) = self.try_unfold_proj_app(wb)? { + *wb = wb2; continue; } } else if b_delta && !a_delta - && let Some(wa2) = self.try_unfold_proj_app(&wa)? + && let Some(wa2) = self.try_unfold_proj_app(wa)? { - wa = wa2; + *wa = wa2; continue; } @@ -466,138 +594,86 @@ impl TypeChecker<'_, M> { }; if wa_w == wb_w { - // H2: Same-head congruence is sound for every hint. Keep Regular - // attempts unbounded; bound non-Regular speculation so a miss cannot - // starve the ordinary delta path. Cache only rejected attempts. + // H2: Same-head congruence is sound for every hint, but it is only + // a sufficient condition: unfolding may erase unequal arguments. + // Bound speculation and cache only inconclusive/rejected probes, + // never an inequality of the enclosing applications. if let (Some(ah), Some(bh)) = (&a_head, &b_head) && ah.addr == bh.addr { let (lo, hi) = canonical_pair(wa.hash_key(), wb.hash_key()); - let failure_key = (lo, hi, self.def_eq_ctx_key(&wa, &wb)); + let failure_key = (lo, hi, self.def_eq_ctx_key(wa, wb)); if !self.env.def_eq_failure.contains(&failure_key) { - let result = if self.is_regular(ah)? { - self.try_same_head_spine(&wa, &wb)? - } else { - self.try_same_head_spine_speculative(&wa, &wb)? - }; + let regular = self.is_regular(ah)?; + let result = + self.try_same_head_spine_speculative(wa, wb, ah, regular)?; if let Some(result) = result { - return Ok(result); + return Ok(Some(result)); } - // Spine comparison was attempted and failed — cache it + // This pair did not yield a useful probe (possibly skipped or + // resource-limited). The cache only skips future speculation. self.env.def_eq_failure.insert(failure_key); self.env.perf.record_def_eq_failure_insert(); } else { self.env.perf.record_def_eq_failure_hit(); + crate::perf::same_head::skip( + wa_w.0 == 1, + crate::perf::same_head::Skip::FailureCache, + ); } } // H1: Equal height — unfold BOTH sides (lean4lean:596) - let ua = self.delta_unfold_one(&wa)?; - let ub = self.delta_unfold_one(&wb)?; + let ua = self.delta_unfold_one(wa)?; + let ub = self.delta_unfold_one(wb)?; match (ua, ub) { (Some(ua), Some(ub)) => { - wa = self.whnf_no_delta_for_def_eq(&ua)?; - wb = self.whnf_no_delta_for_def_eq(&ub)?; + *wa = self.whnf_no_delta_for_def_eq(&ua)?; + *wb = self.whnf_no_delta_for_def_eq(&ub)?; }, (Some(ua), None) => { - wa = self.whnf_no_delta_for_def_eq(&ua)?; + *wa = self.whnf_no_delta_for_def_eq(&ua)?; }, (None, Some(ub)) => { - wb = self.whnf_no_delta_for_def_eq(&ub)?; + *wb = self.whnf_no_delta_for_def_eq(&ub)?; }, (None, None) => break, } } else if wa_w > wb_w { // a is heavier — unfold a first - if let Some(ua) = self.delta_unfold_one(&wa)? { - wa = self.whnf_no_delta_for_def_eq(&ua)?; + if let Some(ua) = self.delta_unfold_one(wa)? { + *wa = self.whnf_no_delta_for_def_eq(&ua)?; } else { break; } } else { // b is heavier — unfold b first - if let Some(ub) = self.delta_unfold_one(&wb)? { - wb = self.whnf_no_delta_for_def_eq(&ub)?; + if let Some(ub) = self.delta_unfold_one(wb)? { + *wb = self.whnf_no_delta_for_def_eq(&ub)?; } else { break; } } } else if a_delta { - if let Some(ua) = self.delta_unfold_one(&wa)? { - wa = self.whnf_no_delta_for_def_eq(&ua)?; + if let Some(ua) = self.delta_unfold_one(wa)? { + *wa = self.whnf_no_delta_for_def_eq(&ua)?; } else { break; } - } else if let Some(ub) = self.delta_unfold_one(&wb)? { - wb = self.whnf_no_delta_for_def_eq(&ub)?; + } else if let Some(ub) = self.delta_unfold_one(wb)? { + *wb = self.whnf_no_delta_for_def_eq(&ub)?; } else { break; } - if wa.ptr_eq(&wb) { - return Ok(true); + if wa.ptr_eq(wb) { + return Ok(Some(true)); } - if self.quick_def_eq(&wa, &wb)? { - return Ok(true); + if self.quick_def_eq(wa, wb)? { + return Ok(Some(true)); } } - if self.def_eq_trace_depth > 0 { - log::info!("[deq tier4 break] depth={}", self.def_eq_depth); - log::info!(" wa: {wa}"); - log::info!(" wb: {wb}"); - } - - // Tier 4b: post-delta congruence checks (lean4lean isDefEqConst/Fvar/Proj) - if self.try_structural_congruence(&wa, &wb)? { - return Ok(true); - } - - // Tier 4c: second structural pass (lean4lean:683-686, lean4 - // type_checker.cpp:1109-1110). This is deliberately `whnfCore`, not full - // `whnf`: full WHNF would delta-unfold stuck open primitives such as - // `Nat.ble` and can literally walk enormous Nat literals in their - // recursive logical models. - let wa_core = self.whnf_core(&wa)?; - let wb_core = self.whnf_core(&wb)?; - let wa_changed = - !wa_core.ptr_eq(&wa) && wa_core.hash_key() != wa.hash_key(); - let wb_changed = - !wb_core.ptr_eq(&wb) && wb_core.hash_key() != wb.hash_key(); - if wa_changed || wb_changed { - return self.is_def_eq(&wa_core, &wb_core); - } - let wa = wa_core; - let wb = wb_core; - if wa.ptr_eq(&wb) { - return Ok(true); - } - if self.quick_def_eq(&wa, &wb)? { - return Ok(true); - } - - // Tier 4d: app spine comparison (lean4lean isDefEqApp, lean4 type_checker.cpp:1115) - if self.try_def_eq_app(&wa, &wb)? { - return Ok(true); - } - - let result = self.is_def_eq_whnf(&wa, &wb); - - // Tier 5 final-fail trace: when IX_DEF_EQ_TIER5_DUMP is set and the - // pair's head names contain the configured substring, dump the - // post-whnfCore wa/wb. This is where lazy-delta + Tier 4c gave up. - if let Ok(prefix) = crate::env_var("IX_DEF_EQ_TIER5_DUMP") - && let Ok(false) = result.as_ref() - { - let a_match = head_const_name(&wa).is_some_and(|n| n.contains(&prefix)); - let b_match = head_const_name(&wb).is_some_and(|n| n.contains(&prefix)); - if prefix.is_empty() || a_match || b_match { - log::info!("[deq tier5 fail] depth={}", self.def_eq_depth); - log::info!(" wa: {wa}"); - log::info!(" wb: {wb}"); - } - } - - result + Ok(None) } /// Quick structural: same constructor, recursively same children (no WHNF). @@ -608,42 +684,8 @@ impl TypeChecker<'_, M> { ) -> Result> { match (a.data(), b.data()) { (ExprData::Sort(u1, _), ExprData::Sort(u2, _)) => Ok(univ_eq(u1, u2)), - ( - ExprData::Lam(name, bi, ty1, body1, _), - ExprData::Lam(_, _, ty2, body2, _), - ) - | ( - ExprData::All(name, bi, ty1, body1, _), - ExprData::All(_, _, ty2, body2, _), - ) => { - if !self.is_def_eq(ty1, ty2)? { - return Ok(false); - } - // Open both bodies with the SAME fresh fvar — the common-fvar - // trick that makes alpha-renamed bodies hash-equal under - // `instantiate_rev` and lets def-eq compare them structurally. - // Mirrors lean4lean `isDefEqBinding` - // (refs/lean4lean/Lean4Lean/TypeChecker.lean:546). - self.with_lctx_scope(|tc| { - let fv_id = tc.fresh_fvar_id(); - let fv = tc.intern(KExpr::fvar(fv_id, name.clone())); - tc.lctx.push( - fv_id, - LocalDecl::CDecl { - name: name.clone(), - bi: bi.clone(), - ty: ty1.clone(), - }, - ); - let b1_open = instantiate_rev( - &mut tc.env.intern, - body1, - std::slice::from_ref(&fv), - ); - let b2_open = instantiate_rev(&mut tc.env.intern, body2, &[fv]); - tc.is_def_eq(&b1_open, &b2_open) - }) - }, + (ExprData::Lam(..), ExprData::Lam(..)) + | (ExprData::All(..), ExprData::All(..)) => self.def_eq_binders(a, b), _ => Ok(false), } } @@ -654,8 +696,8 @@ impl TypeChecker<'_, M> { a: &KExpr, b: &KExpr, ) -> Result, TcError> { - let (a_head, a_args) = collect_app_spine(a); - let (b_head, b_args) = collect_app_spine(b); + let (a_head, a_args) = borrow_app_spine(a); + let (b_head, b_args) = borrow_app_spine(b); let (a_id, a_us) = match a_head.data() { ExprData::Const(id, us, _) => (id, us), _ => return Ok(None), @@ -672,6 +714,9 @@ impl TypeChecker<'_, M> { { return Ok(None); } + if self.try_app_congruence(a, b)? { + return Ok(Some(true)); + } for (ai, bi) in a_args.iter().zip(b_args.iter()) { if !self.is_def_eq(ai, bi)? { return Ok(None); @@ -680,7 +725,54 @@ impl TypeChecker<'_, M> { Ok(Some(true)) } - /// Give a non-Regular same-head attempt a small local slice of recursive + /// Measurement wraps only a real attempt, inside any speculative fuel slice. + /// Nested inclusive counts overlap; the diagnostic also reports exclusive + /// and root fuel. The result and all accounting of actual checking work are + /// unchanged, including errors that a speculative caller may swallow. + fn try_same_head_spine_measured( + &mut self, + a: &KExpr, + b: &KExpr, + head: &KId, + regular: bool, + ) -> Result, TcError> { + use crate::perf::same_head::{self, Outcome}; + if !same_head::enabled() { + return self.try_same_head_spine(a, b); + } + let ticket = same_head::begin(&head.addr, regular); + let before = self.rec_fuel; + let result = self.try_same_head_spine(a, b); + let outcome = match &result { + Ok(Some(true)) => Outcome::Success, + Ok(_) => Outcome::Miss, + Err(TcError::MaxRecFuel) => Outcome::FuelAbort, + Err(TcError::MaxRecDepth) => Outcome::DepthAbort, + Err(_) => Outcome::Error, + }; + let consumed = before.saturating_sub(self.rec_fuel); + same_head::finish(ticket, consumed, outcome); + if let Some(sample) = same_head::take_root_trace(ticket, consumed) { + eprintln!( + "[same-head-root] sample={sample} head=#{} regular={regular} outcome={outcome:?} fuel={consumed} pair={},{} legacy_ctx={} lbr={},{} depth={} def_eq_depth={} cheap={} infer_only={} a={} b={}", + head.addr.hex(), + a.hash_key(), + b.hash_key(), + self.ctx_id, + a.lbr(), + b.lbr(), + self.depth(), + self.def_eq_depth, + self.cheap_recursion_depth, + self.infer_only, + compact_def_eq_expr(a), + compact_def_eq_expr(b) + ); + } + result + } + + /// Give a same-head attempt a small local slice of recursive /// fuel. Nested attempts inherit the remaining slice; an exhausted slice is /// a speculative miss, after which the caller follows the ordinary delta /// path with the consumed work charged to the enclosing check. @@ -688,17 +780,45 @@ impl TypeChecker<'_, M> { &mut self, a: &KExpr, b: &KExpr, + head: &KId, + regular: bool, ) -> Result, TcError> { let saved_fuel = self.rec_fuel; + // Preserve the existing non-Regular policy inside small inherited slices. + // A Regular slice is larger: its temporarily withheld fuel is NOT work + // already performed, and must not prematurely close the startup window. let nested = saved_fuel <= SAME_HEAD_SPECULATION_ATTEMPT_FUEL; - if !nested && self.fuel_used() >= SAME_HEAD_SPECULATION_START_FUEL { + let used = self.fuel_used().saturating_sub(self.same_head_fuel_reserve); + if !regular && !nested && used >= SAME_HEAD_SPECULATION_START_FUEL { + crate::perf::same_head::skip(false, crate::perf::same_head::Skip::Window); + return Ok(None); + } + if self.same_head_backoff.should_skip(regular) { + crate::perf::same_head::skip( + regular, + crate::perf::same_head::Skip::Backoff, + ); return Ok(None); } - let local_fuel = saved_fuel.min(SAME_HEAD_SPECULATION_ATTEMPT_FUEL); + let allowance = if regular { + SAME_HEAD_REGULAR_ATTEMPT_FUEL + } else { + SAME_HEAD_SPECULATION_ATTEMPT_FUEL + }; + let local_fuel = saved_fuel.min(allowance); + let saved_reserve = self.same_head_fuel_reserve; + self.same_head_fuel_reserve += saved_fuel - local_fuel; self.rec_fuel = local_fuel; - let result = self.try_same_head_spine(a, b); + self.same_head_backoff.enter(); + let result = self.try_same_head_spine_measured(a, b, head, regular); let consumed = local_fuel.saturating_sub(self.rec_fuel); self.rec_fuel = saved_fuel.saturating_sub(consumed); + self.same_head_fuel_reserve = saved_reserve; + let unsuccessful = matches!( + result, + Ok(None | Some(false)) | Err(TcError::MaxRecDepth | TcError::MaxRecFuel) + ); + self.same_head_backoff.leave(regular, unsuccessful, consumed); match result { Err(TcError::MaxRecDepth | TcError::MaxRecFuel) => Ok(None), other => other, @@ -734,38 +854,10 @@ impl TypeChecker<'_, M> { } false }, - ( - ExprData::Lam(name, bi, ty1, body1, _), - ExprData::Lam(_, _, ty2, body2, _), - ) - | ( - ExprData::All(name, bi, ty1, body1, _), - ExprData::All(_, _, ty2, body2, _), - ) => { - if self.is_def_eq(ty1, ty2)? { - // Open both bodies with the same fresh fvar (see `quick_def_eq`). - let r = self.with_lctx_scope(|tc| { - let fv_id = tc.fresh_fvar_id(); - let fv = tc.intern(KExpr::fvar(fv_id, name.clone())); - tc.lctx.push( - fv_id, - LocalDecl::CDecl { - name: name.clone(), - bi: bi.clone(), - ty: ty1.clone(), - }, - ); - let b1_open = instantiate_rev( - &mut tc.env.intern, - body1, - std::slice::from_ref(&fv), - ); - let b2_open = instantiate_rev(&mut tc.env.intern, body2, &[fv]); - tc.is_def_eq(&b1_open, &b2_open) - })?; - if r { - return Ok(true); - } + (ExprData::Lam(..), ExprData::Lam(..)) + | (ExprData::All(..), ExprData::All(..)) => { + if self.def_eq_binders(a, b)? { + return Ok(true); } false }, @@ -873,6 +965,9 @@ impl TypeChecker<'_, M> { a: &KExpr, b: &KExpr, ) -> Result> { + if self.known_non_proof(a) { + return Ok(false); + } let a_ty = match self.with_infer_only(|tc| tc.infer(a)) { Ok(ty) => ty, Err(_) => return Ok(false), @@ -893,9 +988,9 @@ impl TypeChecker<'_, M> { /// /// On a hit this is one `FxHashMap` probe; on a miss it pays the /// existing `infer ∘ whnf` chain and stores the result. Errors from - /// the inner chain are propagated as `Ok(false)` (treating ill-typed - /// metadata as non-prop), matching the previous behaviour of - /// `try_proof_irrel`. + /// the inner chain are treated as a miss, but are NOT cached: in particular, + /// exhausting a speculative fuel slice must not permanently classify a + /// proposition as non-propositional. pub(crate) fn is_prop_type(&mut self, ty: &KExpr) -> bool { let cache_key = (ty.hash_key(), self.ctx_addr_for_lbr(ty.lbr())); if let Some(&cached) = self.env.is_prop_cache.get(&cache_key) { @@ -914,9 +1009,9 @@ impl TypeChecker<'_, M> { ExprData::Sort(u, _) => u.is_semantic_zero(), _ => false, }, - Err(_) => false, + Err(_) => return false, }, - Err(_) => false, + Err(_) => return false, }; self.env.is_prop_cache.insert(cache_key, result); result @@ -937,7 +1032,7 @@ impl TypeChecker<'_, M> { Ok(w) => w, Err(_) => return Ok(false), }; - let (a_head, _) = collect_app_spine(&a_ty_w); + let a_head = app_head(&a_ty_w); let a_ind = match a_head.data() { ExprData::Const(id, _, _) => id.clone(), _ => return Ok(false), @@ -1254,12 +1349,10 @@ impl TypeChecker<'_, M> { t: &KExpr, s: &KExpr, ) -> Result> { - use super::tc::collect_app_spine; - let t_norm = self.whnf_no_delta(t).unwrap_or_else(|_| t.clone()); // s must be a constructor application - let (s_head, s_args) = collect_app_spine(s); + let (s_head, s_args) = borrow_app_spine(s); let ctor_id = match s_head.data() { ExprData::Const(id, _, _) => id.clone(), _ => { @@ -1341,13 +1434,13 @@ impl TypeChecker<'_, M> { for i in 0..num_fields { let proj = self.intern(KExpr::prj(induct_id.clone(), i as u64, t_norm.clone())); - if !self.is_def_eq(&proj, &s_args[num_params + i])? { + if !self.is_def_eq(&proj, s_args[num_params + i])? { self.dump_eta_trace( "field-mismatch", Some(&induct_id), i, &proj, - &s_args[num_params + i], + s_args[num_params + i], ); return Ok(false); } @@ -1362,7 +1455,7 @@ impl TypeChecker<'_, M> { induct_id: &KId, num_params: usize, num_fields: usize, - args: &[KExpr], + args: &[&KExpr], ) -> Result>, TcError> { let mut base: Option> = None; for i in 0..num_fields { @@ -1396,14 +1489,17 @@ impl TypeChecker<'_, M> { { return Ok(false); } - let (a_head, a_args) = collect_app_spine(a); - let (b_head, b_args) = collect_app_spine(b); + let (a_head, a_args) = borrow_app_spine(a); + let (b_head, b_args) = borrow_app_spine(b); if a_args.len() != b_args.len() { return Ok(false); } - if !self.is_def_eq(&a_head, &b_head)? { + if !self.is_def_eq(a_head, b_head)? { return Ok(false); } + if self.try_app_congruence(a, b)? { + return Ok(true); + } for (ai, bi) in a_args.iter().zip(b_args.iter()) { if !self.is_def_eq(ai, bi)? { return Ok(false); @@ -1491,6 +1587,9 @@ impl TypeChecker<'_, M> { if id1.addr != id2.addr || f1 != f2 { return Ok(false); } + if self.try_projected_def_eq(a, b)? { + return Ok(true); + } let mut v1 = v1.clone(); let mut v2 = v2.clone(); self.lazy_delta_proj_reduction(id1, *f1, &mut v1, &mut v2) @@ -1499,6 +1598,39 @@ impl TypeChecker<'_, M> { } } + /// A positive-only shortcut: compare reduced fields before traversing the + /// other fields/arguments of their records. Failure (including local budget + /// exhaustion) leaves the ORIGINAL record-congruence path available. Never + /// turn an interrupted record comparison into a negative conversion result. + fn try_projected_def_eq( + &mut self, + a: &KExpr, + b: &KExpr, + ) -> Result> { + if self.in_projection_probe { + return Ok(false); + } + let saved_fuel = self.rec_fuel; + let local_fuel = saved_fuel.min(PROJECTION_PROBE_FUEL); + self.rec_fuel = local_fuel; + self.in_projection_probe = true; + let result = (|| { + let pa = self.whnf_core(a)?; + let pb = self.whnf_core(b)?; + if pa.hash_key() == a.hash_key() && pb.hash_key() == b.hash_key() { + return Ok(false); + } + self.is_def_eq(&pa, &pb) + })(); + self.in_projection_probe = false; + let consumed = local_fuel.saturating_sub(self.rec_fuel); + self.rec_fuel = saved_fuel.saturating_sub(consumed); + match result { + Err(TcError::MaxRecDepth | TcError::MaxRecFuel) => Ok(false), + other => other, + } + } + fn lazy_delta_proj_reduction( &mut self, struct_id: &KId, @@ -1509,6 +1641,7 @@ impl TypeChecker<'_, M> { let mut fuel = MAX_WHNF_FUEL; loop { if fuel == 0 { + self.dump_guard_stack("def-eq-projection-delta-fuel"); self.dump_def_eq_max("proj-delta-fuel", a, b, None, None); return Err(TcError::MaxRecDepth); } @@ -1595,11 +1728,9 @@ impl TypeChecker<'_, M> { } } else { if a_id.addr == b_id.addr { - let result = if self.is_regular(a_id)? { - self.try_same_head_spine(a, b)? - } else { - self.try_same_head_spine_speculative(a, b)? - }; + let regular = self.is_regular(a_id)?; + let result = + self.try_same_head_spine_speculative(a, b, a_id, regular)?; if let Some(true) = result { return Ok(LazyDeltaStep::Equal); } @@ -1670,7 +1801,7 @@ impl TypeChecker<'_, M> { &mut self, e: &KExpr, ) -> Result>, TcError> { - let (head, _) = collect_app_spine(e); + let head = app_head(e); if !matches!(head.data(), ExprData::Prj(..)) { return Ok(None); } @@ -1778,7 +1909,7 @@ fn head_const_id(e: &KExpr) -> Option> { match e.data() { ExprData::Const(id, _, _) => Some(id.clone()), ExprData::App(..) => { - let (head, _) = collect_app_spine(e); + let head = app_head(e); match head.data() { ExprData::Const(id, _, _) => Some(id.clone()), _ => None, @@ -1885,6 +2016,10 @@ impl TypeChecker<'_, M> { } } +#[cfg(test)] +#[path = "def_eq/projection_tests.rs"] +mod projection_tests; + #[cfg(test)] mod tests { @@ -1924,6 +2059,21 @@ mod tests { AE::sort(AU::succ(AU::zero())) } + #[test] + fn declaration_summary_skips_only_non_proof_irrelevance() { + let mut env = env_with_same_head_hint(ReducibilityHints::Abbrev); + let a = AE::cnst(mk_id("same_head.c"), Box::new([])); + let b = AE::cnst(mk_id("same_head.d"), Box::new([])); + let mut tc = TypeChecker::new(&mut env); + assert!(!tc.try_proof_irrel(&a, &b).unwrap()); + assert!(!tc.env.decl_summary_cache.is_empty()); + assert!(tc.env.infer_cache.is_empty()); + assert!(tc.env.infer_only_cache.is_empty()); + // Other conversion rules can still infer (e.g. structure eta). The + // summary bypasses only this probe, and never establishes equality. + assert!(!tc.is_def_eq(&a, &b).unwrap()); + } + fn env_with_id() -> KEnv { let mut env = KEnv::new(); let id_ty = AE::all((), (), sort0(), sort0()); @@ -2071,6 +2221,81 @@ mod tests { ); } + #[test] + fn same_head_profile_preserves_success_result_and_fuel() { + use crate::perf::same_head; + let run = |measured: bool| { + let mut env = env_with_same_head_hint(ReducibilityHints::Regular(7)); + let a = AE::cnst(mk_id("same_head.A"), Box::new([])); + let c = AE::cnst(mk_id("same_head.c"), Box::new([])); + let id = mk_id("same_head.head"); + let head = AE::cnst(id.clone(), Box::new([])); + let beta_arg = AE::app(AE::lam((), (), a, AE::var(0, ())), c.clone()); + let left = AE::app(head.clone(), beta_arg); + let right = AE::app(head, c); + let mut tc = TypeChecker::new(&mut env); + same_head::reset(); + let result = if measured { + tc.try_same_head_spine_measured(&left, &right, &id, true) + } else { + tc.try_same_head_spine(&left, &right) + } + .unwrap(); + if measured && same_head::enabled() { + let report = same_head::summary(); + assert!(report.contains("regular outcome=success calls=1"), "{report}"); + assert!( + report.contains(&format!("root_fuel={} ", tc.fuel_used())), + "{report}" + ); + assert!(report.contains("active=0 "), "{report}"); + assert!(report.contains("accounting_errors=0"), "{report}"); + } + (result, tc.fuel_used()) + }; + assert_eq!(run(false), run(true)); + } + + #[test] + fn same_head_profile_distinguishes_window_skip_from_fuel_abort() { + use crate::perf::same_head; + let mut env = env_with_same_head_hint(ReducibilityHints::Abbrev); + let a = AE::cnst(mk_id("same_head.A"), Box::new([])); + let c = AE::cnst(mk_id("same_head.c"), Box::new([])); + let id = mk_id("same_head.head"); + let head = AE::cnst(id.clone(), Box::new([])); + let beta_arg = AE::app(AE::lam((), (), a, AE::var(0, ())), c.clone()); + let left = AE::app(head.clone(), beta_arg); + let right = AE::app(head, c); + let mut tc = TypeChecker::new(&mut env); + tc.rec_fuel = crate::tc::max_rec_fuel() + .saturating_sub(super::SAME_HEAD_SPECULATION_START_FUEL); + let before = tc.rec_fuel; + same_head::reset(); + assert_eq!( + tc.try_same_head_spine_speculative(&left, &right, &id, false).unwrap(), + None + ); + assert_eq!(tc.rec_fuel, before); + tc.rec_fuel = 1; + assert_eq!( + tc.try_same_head_spine_speculative(&left, &right, &id, false).unwrap(), + None + ); + assert_eq!(tc.rec_fuel, 0); + if same_head::enabled() { + let report = same_head::summary(); + assert!(report.contains("non_regular skipped_window=1"), "{report}"); + assert!( + report.contains("non_regular outcome=fuel_abort calls=1"), + "{report}" + ); + assert!(report.contains("root_fuel=1 "), "{report}"); + assert!(report.contains("active=0 "), "{report}"); + assert!(report.contains("accounting_errors=0"), "{report}"); + } + } + /// Insert a `Defn` with the given reducibility hints under `name`, returning /// its `KId`. Used by `def_rank_id` ordering tests. fn insert_rank_def( diff --git a/crates/kernel/src/def_eq/application.rs b/crates/kernel/src/def_eq/application.rs new file mode 100644 index 000000000..a0682f29a --- /dev/null +++ b/crates/kernel/src/def_eq/application.rs @@ -0,0 +1,159 @@ +//! Positive-only application congruence with an explicit postorder worklist. +//! +//! Invariant: a Finish frame is reached ONLY after both child comparisons +//! succeeded in the frame's original context. Thus every published equality +//! follows from application congruence, not from a pending/visited-pair guess. +//! See Ix/Tc/Verify/DefEq/SpineArguments.lean (TrAppSpine.defEq_of_zip) for the +//! corresponding semantic rule. That theorem does not certify this Rust loop. +//! +//! No binders are opened here. All non-App pairs use ordinary conversion, +//! with recursive worklist entry disabled. A failed child abandons the probe; +//! it does NOT imply inequality of its applications (functions may ignore +//! arguments). The caller retains the original conversion path on every miss. + +use super::*; +use crate::env::CtxAddr; +use crate::equiv::EqKey; + +const APP_CONGRUENCE_FUEL: u64 = 4_096; +// Ordinary conversion is usually cheaper than speculative congruence. Use +// the worklist as a near-guard fallback, not a general structural fast path: +// a shallow trigger can spend much more work on arguments that reduction +// would erase. Reserve stack headroom for the ordinary leaf comparisons. +// This scheduling choice changes neither equality rules nor the depth cap. +const APP_CONGRUENCE_MIN_DEPTH: u32 = MAX_DEF_EQ_DEPTH - 512; + +/// Compute these in the original context, before calling any leaf reducer. +/// Only positive entries are published, so FULL may also consume cheap-mode +/// successes, just as in is_def_eq. Negative results are never published here. +struct CompletedAppKeys { + cache: (Addr, Addr, CtxAddr), + left: EqKey, + right: EqKey, + cheap: bool, +} + +enum Task { + Compare(KExpr, KExpr), + Finish(CompletedAppKeys), +} + +impl TypeChecker<'_, M> { + #[inline] + pub(super) fn try_app_congruence( + &mut self, + a: &KExpr, + b: &KExpr, + ) -> Result> { + if self.def_eq_depth < APP_CONGRUENCE_MIN_DEPTH { + return Ok(false); + } + self.app_congruence_probe(a, b) + } + + #[cold] + #[inline(never)] + fn app_congruence_probe( + &mut self, + a: &KExpr, + b: &KExpr, + ) -> Result> { + if self.in_app_congruence + || !matches!((a.data(), b.data()), (ExprData::App(..), ExprData::App(..))) + { + return Ok(false); + } + // Bound failed speculation, including leaf reductions, without changing + // the constant's global allowance. Pending frames consume O(local_fuel) + // space: each expansion is ticked before pushing at most three tasks. + let saved_fuel = self.rec_fuel; + let local_fuel = saved_fuel.min(APP_CONGRUENCE_FUEL); + self.rec_fuel = local_fuel; + self.in_app_congruence = true; + let result = self.app_congruence_worklist(a, b); + self.in_app_congruence = false; + let consumed = local_fuel.saturating_sub(self.rec_fuel); + self.rec_fuel = saved_fuel.saturating_sub(consumed); + match result { + Err(TcError::MaxRecDepth | TcError::MaxRecFuel) => Ok(false), + other => other, + } + } + + fn app_congruence_worklist( + &mut self, + a: &KExpr, + b: &KExpr, + ) -> Result> { + let mut work = vec![Task::Compare(a.clone(), b.clone())]; + while let Some(task) = work.pop() { + let (a, b) = match task { + Task::Finish(keys) => { + // Both children completed. No pending parent can take this path + // after a miss/error, because those return from the entire loop. + self.env.def_eq_cache.insert(keys.cache, true); + if keys.cheap { + self.env.def_eq_cheap_cache.insert(keys.cache, true); + } + self.equiv_manager.add_equiv(keys.left, keys.right); + continue; + }, + Task::Compare(a, b) => (a, b), + }; + if a.ptr_eq(&b) || a.hash_key() == b.hash_key() { + continue; + } + let (ExprData::App(af, aa, _), ExprData::App(bf, ba, _)) = + (a.data(), b.data()) + else { + if !self.is_def_eq(&a, &b)? { + return Ok(false); + } + continue; + }; + + crate::profile::bump_def_eq(); + let lbr = a.lbr().max(b.lbr()); + let ctx = self.def_eq_ctx_key(&a, &b); + let (lo, hi) = canonical_pair(a.hash_key(), b.hash_key()); + let keys = CompletedAppKeys { + cache: (lo, hi, ctx), + left: EqKey::new(a.hash_key(), ctx, lbr, a.lbr()), + right: EqKey::new(b.hash_key(), ctx, lbr, b.lbr()), + cheap: self.cheap_recursion_depth > 0, + }; + // Completed equality is enough to skip a shared DAG branch. In-flight + // frames are deliberately absent from both caches and the union-find. + if self.equiv_manager.is_equiv(&keys.left, &keys.right) + || self.env.def_eq_cache.get(&keys.cache) == Some(&true) + || (keys.cheap + && self.env.def_eq_cheap_cache.get(&keys.cache) == Some(&true)) + { + self.env.perf.record_def_eq_hit(); + continue; + } + // A cached inequality makes this congruence attempt unproductive. + // Abandon the probe, NOT its enclosing application comparison: a + // surrounding function can still ignore this unequal child. + if self.env.def_eq_cache.get(&keys.cache) == Some(&false) + || (keys.cheap + && self.env.def_eq_cheap_cache.get(&keys.cache) == Some(&false)) + { + self.env.perf.record_def_eq_hit(); + return Ok(false); + } + self.env.perf.record_def_eq_miss(); + self.tick()?; + // LIFO order: function first, then argument, then completion. Each + // argument is checked after its function prefix, preserving dependent + // application order. The caller-held roots keep descendants alive. + work.push(Task::Finish(keys)); + work.push(Task::Compare(aa.clone(), ba.clone())); + work.push(Task::Compare(af.clone(), bf.clone())); + } + Ok(true) + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/kernel/src/def_eq/application/tests.rs b/crates/kernel/src/def_eq/application/tests.rs new file mode 100644 index 000000000..3f99dcddf --- /dev/null +++ b/crates/kernel/src/def_eq/application/tests.rs @@ -0,0 +1,552 @@ +use super::*; +use crate::{ + constant::KConst, + env::KEnv, + mode::{Anon, Meta}, +}; +use ix_common::{ + address::Address, + env::{BinderInfo, DefinitionSafety, Name, ReducibilityHints}, +}; + +fn id(s: &str) -> KId { + KId::new( + Address::hash(s.as_bytes()), + M::meta_field(Name::str(Name::anon(), s.to_owned())), + ) +} + +fn cnst(s: &str) -> KExpr { + KExpr::cnst(id(s), Box::new([])) +} + +fn arrow(a: KExpr, b: KExpr) -> KExpr { + KExpr::all( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + a, + b, + ) +} + +fn lam(body: KExpr) -> KExpr { + KExpr::lam( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + cnst("A"), + body, + ) +} + +fn axiom(env: &mut KEnv, name: &str, ty: KExpr) { + env.insert( + id(name), + KConst::Axio { + name: M::meta_field(Name::anon()), + level_params: M::meta_field(vec![]), + is_unsafe: false, + lvls: 0, + ty, + }, + ); +} + +fn defn( + env: &mut KEnv, + name: &str, + ty: KExpr, + val: KExpr, +) { + env.insert( + id(name), + KConst::Defn { + name: M::meta_field(Name::anon()), + level_params: M::meta_field(vec![]), + kind: DefKind::Definition, + safety: DefinitionSafety::Safe, + hints: ReducibilityHints::Regular(7), + lvls: 0, + ty, + val, + lean_all: M::meta_field(vec![]), + block: id(name), + }, + ); +} + +fn setup() -> KEnv { + let mut env = KEnv::new(); + axiom(&mut env, "A", KExpr::sort(KUniv::succ(KUniv::zero()))); + axiom(&mut env, "a", cnst("A")); + axiom(&mut env, "b", cnst("A")); + axiom(&mut env, "F", arrow(cnst("A"), cnst("A"))); + axiom(&mut env, "G", arrow(cnst("A"), arrow(cnst("A"), cnst("A")))); + defn(&mut env, "alias", cnst("A"), cnst("a")); + defn( + &mut env, + "id", + arrow(cnst("A"), cnst("A")), + lam(KExpr::var(0, M::meta_field(Name::anon()))), + ); + defn(&mut env, "ignore", arrow(cnst("A"), cnst("A")), lam(cnst("a"))); + env +} + +fn app( + tc: &mut TypeChecker<'_, M>, + f: KExpr, + a: KExpr, +) -> KExpr { + tc.intern(KExpr::app(f, a)) +} + +fn pair_key( + tc: &mut TypeChecker<'_, M>, + a: &KExpr, + b: &KExpr, +) -> (Addr, Addr, CtxAddr) { + let (lo, hi) = canonical_pair(a.hash_key(), b.hash_key()); + (lo, hi, tc.def_eq_ctx_key(a, b)) +} + +fn chain( + tc: &mut TypeChecker<'_, M>, + n: usize, +) -> (KExpr, KExpr) { + let f = tc.intern(cnst("F")); + let mut a = tc.intern(cnst("alias")); + let mut b = tc.intern(cnst("a")); + for _ in 0..n { + a = app(tc, f.clone(), a); + b = app(tc, f.clone(), b); + } + (a, b) +} + +fn deep_chain() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let (a, b) = chain(&mut tc, MAX_DEF_EQ_DEPTH as usize + 100); + assert!(tc.is_def_eq(&a, &b).unwrap()); + assert!( + tc.def_eq_peak < APP_CONGRUENCE_MIN_DEPTH + 20, + "depth {}", + tc.def_eq_peak + ); + assert_eq!(tc.def_eq_depth, 0); + assert!(!tc.in_app_congruence); + + // Fresh caches and the original path: same well-typed finite chain still + // reaches the old guard. Large stack matches the real dedicated worker. + let mut old_env = setup::(); + let mut old = TypeChecker::new(&mut old_env); + old.in_app_congruence = true; + let (a, b) = chain(&mut old, MAX_DEF_EQ_DEPTH as usize + 100); + assert!(matches!(old.is_def_eq(&a, &b), Err(TcError::MaxRecDepth))); + assert_eq!(old.def_eq_depth, 0); +} + +#[test] +fn long_application_chain_uses_worklist_not_def_eq_stack() { + std::thread::Builder::new() + .stack_size(256 * 1024 * 1024) + .spawn(|| { + deep_chain::(); + deep_chain::(); + }) + .unwrap() + .join() + .unwrap(); +} + +fn partial_success() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let left_inner = app(&mut tc, cnst("F"), cnst("alias")); + let right_inner = app(&mut tc, cnst("F"), cnst("a")); + let left_fn = app(&mut tc, cnst("G"), left_inner); + let right_fn = app(&mut tc, cnst("G"), right_inner); + let a = app(&mut tc, left_fn.clone(), cnst("a")); + let b = app(&mut tc, right_fn.clone(), cnst("b")); + let root_key = pair_key(&mut tc, &a, &b); + let fn_key = pair_key(&mut tc, &left_fn, &right_fn); + assert!(!tc.app_congruence_probe(&a, &b).unwrap()); + assert_eq!(tc.env.def_eq_cache.get(&fn_key), Some(&true)); + assert!(!tc.env.def_eq_cache.contains_key(&root_key)); + assert!(!tc.equiv_manager.is_equiv( + &EqKey::new(a.hash_key(), root_key.2, 0, 0), + &EqKey::new(b.hash_key(), root_key.2, 0, 0), + )); + assert!(!tc.is_def_eq(&a, &b).unwrap()); + assert!(!tc.is_def_eq(&b, &a).unwrap()); +} + +#[test] +fn failed_child_never_completes_pending_parent() { + partial_success::(); + partial_success::(); +} + +fn non_injective() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + tc.def_eq_depth = APP_CONGRUENCE_MIN_DEPTH; + let a = app(&mut tc, cnst("ignore"), cnst("a")); + let b = app(&mut tc, cnst("ignore"), cnst("b")); + let key = pair_key(&mut tc, &a, &b); + assert!(!tc.app_congruence_probe(&a, &b).unwrap()); + assert!(!tc.env.def_eq_cache.contains_key(&key)); + assert!(tc.is_def_eq(&a, &b).unwrap()); +} + +#[test] +fn unequal_arguments_of_ignoring_function_fall_back_to_reduction() { + non_injective::(); + non_injective::(); +} + +fn diamond() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let g = tc.intern(cnst("G")); + let (mut a, mut b) = (tc.intern(cnst("alias")), tc.intern(cnst("a"))); + for _ in 0..64 { + let af = app(&mut tc, g.clone(), a.clone()); + let bf = app(&mut tc, g.clone(), b.clone()); + a = app(&mut tc, af, a); + b = app(&mut tc, bf, b); + } + tc.rec_fuel = 1_000; + assert!(tc.app_congruence_probe(&a, &b).unwrap()); + assert!(tc.rec_fuel > 700, "completed DAG branches must not be revisited"); +} + +#[test] +fn shared_diamond_uses_only_completed_equalities() { + diamond::(); + diamond::(); +} + +fn budgets() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let (a, b) = chain(&mut tc, 4); + let key = pair_key(&mut tc, &a, &b); + tc.rec_fuel = 1; + assert!(!tc.app_congruence_probe(&a, &b).unwrap()); + assert_eq!(tc.rec_fuel, 0); + assert!(!tc.in_app_congruence); + assert!(!tc.env.def_eq_cache.contains_key(&key)); + assert_eq!(tc.def_eq_depth, 0); + + tc.rec_fuel = 10_000; + tc.def_eq_depth = MAX_DEF_EQ_DEPTH; + assert!(!tc.app_congruence_probe(&a, &b).unwrap()); + assert_eq!(tc.def_eq_depth, MAX_DEF_EQ_DEPTH); + assert!(tc.rec_fuel < 10_000); + assert!(!tc.in_app_congruence); + assert!(!tc.env.def_eq_cache.contains_key(&key)); + tc.def_eq_depth = 0; + assert!(tc.is_def_eq(&a, &b).unwrap()); + + let (a, b) = + chain(&mut tc, usize::try_from(APP_CONGRUENCE_FUEL).unwrap() + 100); + let key = pair_key(&mut tc, &a, &b); + tc.rec_fuel = 20_000; + assert!(!tc.app_congruence_probe(&a, &b).unwrap()); + assert_eq!(tc.rec_fuel, 20_000 - APP_CONGRUENCE_FUEL); + assert!(!tc.in_app_congruence); + assert!(!tc.env.def_eq_cache.contains_key(&key)); +} + +#[test] +fn budget_and_depth_misses_restore_state_without_refunds() { + std::thread::Builder::new() + .stack_size(256 * 1024 * 1024) + .spawn(|| { + budgets::(); + budgets::(); + }) + .unwrap() + .join() + .unwrap(); +} + +fn malformed() { + let mut env = setup::(); + defn( + &mut env, + "bad", + cnst("A"), + KExpr::sort(KUniv::param(1, M::meta_field(Name::anon()))), + ); + let mut tc = TypeChecker::new(&mut env); + let a = + app(&mut tc, cnst("F"), KExpr::cnst(id("bad"), Box::new([KUniv::zero()]))); + let b = app(&mut tc, cnst("F"), cnst("a")); + let key = pair_key(&mut tc, &a, &b); + let fuel = tc.rec_fuel; + assert!(matches!( + tc.app_congruence_probe(&a, &b), + Err(TcError::UnivParamOutOfRange { .. }) + )); + assert!(!tc.in_app_congruence); + assert_eq!(tc.def_eq_depth, 0); + assert!(tc.rec_fuel < fuel); + assert!(!tc.env.def_eq_cache.contains_key(&key)); +} + +#[test] +fn non_budget_leaf_error_propagates_without_parent_cache_entry() { + malformed::(); + malformed::(); +} + +fn contexts() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let a = app(&mut tc, cnst("F"), KExpr::var(0, M::meta_field(Name::anon()))); + let b = app(&mut tc, cnst("F"), cnst("a")); + tc.push_let(cnst("A"), cnst("a")); + let first = pair_key(&mut tc, &a, &b); + assert!(tc.app_congruence_probe(&a, &b).unwrap()); + assert_eq!(tc.depth(), 1); + tc.pop_local(); + tc.push_let(cnst("A"), cnst("b")); + let second = pair_key(&mut tc, &a, &b); + assert_ne!(first, second); + assert!(!tc.app_congruence_probe(&a, &b).unwrap()); + assert!(!tc.env.def_eq_cache.contains_key(&second)); + assert!(!tc.is_def_eq(&a, &b).unwrap()); + assert_eq!(tc.depth(), 1); +} + +#[test] +fn completed_open_pair_is_not_reused_in_another_context() { + contexts::(); + contexts::(); +} + +fn modes() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let (a, b) = chain(&mut tc, 4); + let key = pair_key(&mut tc, &a, &b); + // A failed cheap attempt is not a full-mode inequality certificate. + tc.env.def_eq_cheap_cache.insert(key, false); + assert!(tc.app_congruence_probe(&a, &b).unwrap()); + assert_eq!(tc.env.def_eq_cache.get(&key), Some(&true)); + tc.env.clear_reduction_caches(); + tc.equiv_manager.clear(); + tc.cheap_recursion_depth = 1; + tc.infer_only = true; + tc.eager_reduce = true; + assert!(tc.app_congruence_probe(&a, &b).unwrap()); + assert_eq!(tc.env.def_eq_cheap_cache.get(&key), Some(&true)); + assert_eq!(tc.env.def_eq_cache.get(&key), Some(&true)); + assert_eq!(tc.cheap_recursion_depth, 1); + assert!(tc.infer_only && tc.eager_reduce); + assert_eq!(tc.depth(), 0); +} + +#[test] +fn mode_boundaries_and_positive_cache_promotion_are_preserved() { + modes::(); + modes::(); +} + +fn dependent() { + let mut env = setup::(); + axiom( + &mut env, + "B", + arrow(cnst("A"), KExpr::sort(KUniv::succ(KUniv::zero()))), + ); + let b_a = KExpr::app(cnst("B"), cnst("a")); + axiom(&mut env, "v", b_a); + let b_alias = KExpr::app(cnst("B"), cnst("alias")); + defn(&mut env, "vAlias", b_alias, cnst("v")); + axiom( + &mut env, + "dep", + arrow( + cnst("A"), + arrow( + KExpr::app(cnst("B"), KExpr::var(0, M::meta_field(Name::anon()))), + cnst("A"), + ), + ), + ); + let mut tc = TypeChecker::new(&mut env); + let af = app(&mut tc, cnst("dep"), cnst("alias")); + let bf = app(&mut tc, cnst("dep"), cnst("a")); + let a = app(&mut tc, af, cnst("vAlias")); + let b = app(&mut tc, bf, cnst("v")); + tc.infer(&a).unwrap(); + tc.infer(&b).unwrap(); + assert!(tc.app_congruence_probe(&a, &b).unwrap()); + assert!(tc.is_def_eq(&a, &b).unwrap()); +} + +#[test] +fn dependent_spine_compares_prefix_before_later_arguments() { + dependent::(); + dependent::(); +} + +#[test] +fn probe_does_not_nest_and_reset_clears_its_flag() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let (a, b) = chain(&mut tc, 4); + tc.in_app_congruence = true; + let fuel = tc.rec_fuel; + assert!(!tc.app_congruence_probe(&a, &b).unwrap()); + assert!(tc.in_app_congruence); + assert_eq!(tc.rec_fuel, fuel); + tc.reset(); + assert!(!tc.in_app_congruence); +} + +#[test] +fn shallow_comparisons_do_not_pay_for_speculative_work() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let (a, b) = chain(&mut tc, 4); + tc.def_eq_depth = APP_CONGRUENCE_MIN_DEPTH - 1; + let fuel = tc.rec_fuel; + assert!(!tc.try_app_congruence(&a, &b).unwrap()); + assert_eq!(tc.rec_fuel, fuel); + assert!(tc.env.def_eq_cache.is_empty()); + tc.def_eq_depth += 1; + assert!(tc.try_app_congruence(&a, &b).unwrap()); + assert_eq!(tc.def_eq_depth, APP_CONGRUENCE_MIN_DEPTH); +} + +fn below_guard_matches_original() { + for n in [1, 64, 128] { + let check = |old: bool| { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + tc.in_app_congruence = old; + let (a, b) = chain(&mut tc, n); + assert!(tc.is_def_eq(&a, &b).unwrap()); + assert!(tc.def_eq_peak < APP_CONGRUENCE_MIN_DEPTH); + (tc.fuel_used(), tc.def_eq_peak) + }; + assert_eq!(check(false), check(true), "chain length {n}"); + } +} + +#[test] +fn ordinary_comparisons_preserve_recursive_work_and_depth() { + // Depth 64 was enough to activate the earlier eager variant. These + // successful ordinary checks should do exactly the original work now. + below_guard_matches_original::(); + below_guard_matches_original::(); +} + +#[test] +fn cached_unequal_child_only_abandons_the_probe() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let a = app(&mut tc, cnst("F"), cnst("a")); + let b = app(&mut tc, cnst("F"), cnst("b")); + assert!(!tc.is_def_eq(&a, &b).unwrap()); + let left = app(&mut tc, cnst("ignore"), a); + let right = app(&mut tc, cnst("ignore"), b); + let fuel = tc.rec_fuel; + let key = pair_key(&mut tc, &left, &right); + assert!(!tc.app_congruence_probe(&left, &right).unwrap()); + assert_eq!(tc.rec_fuel, fuel - 1, "the cached child must not be expanded"); + assert!(!tc.env.def_eq_cache.contains_key(&key)); + assert!(tc.is_def_eq(&left, &right).unwrap()); +} + +fn generated( + tc: &mut TypeChecker<'_, M>, + seed: u64, + depth: u64, +) -> KExpr { + if depth == 0 { + return tc.intern(cnst(["a", "b", "alias"][(seed % 3) as usize])); + } + let child = generated(tc, seed / 3 + 1, depth - 1); + match seed % 4 { + 0 => app(tc, cnst("F"), child), + 1 => app(tc, cnst("id"), child), + 2 => app(tc, cnst("ignore"), child), + _ => { + let f = app(tc, cnst("G"), child.clone()); + app(tc, f, child) + }, + } +} + +fn differential() { + for n in 0..96 { + let check = |old: bool| { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + tc.in_app_congruence = old; + // Exercise the worklist even on these small terms. Both variants + // start at the same depth; only the old path disables the probe. + tc.def_eq_depth = APP_CONGRUENCE_MIN_DEPTH; + let a = generated(&mut tc, n * 17 + 3, 5); + let b = generated(&mut tc, n * 13 + 3, 5); + tc.is_def_eq(&a, &b).unwrap() + }; + assert_eq!(check(false), check(true), "seed {n}"); + } +} + +#[test] +fn small_well_typed_terms_match_original_recursive_conversion() { + differential::(); + differential::(); +} + +fn universes_and_arity() { + let mut env = setup::(); + // An unused universe parameter leaves both applications well typed, but + // different instantiations of an irreducible axiom are not def-eq heads. + env.insert( + id("poly"), + KConst::Axio { + name: M::meta_field(Name::anon()), + level_params: M::meta_field(vec![Name::str( + Name::anon(), + "u".to_owned(), + )]), + is_unsafe: false, + lvls: 1, + ty: arrow(cnst("A"), cnst("A")), + }, + ); + let mut tc = TypeChecker::new(&mut env); + tc.def_eq_depth = APP_CONGRUENCE_MIN_DEPTH; + let a = + app(&mut tc, KExpr::cnst(id("poly"), Box::new([KUniv::zero()])), cnst("a")); + let b = app( + &mut tc, + KExpr::cnst(id("poly"), Box::new([KUniv::succ(KUniv::zero())])), + cnst("a"), + ); + tc.infer(&a).unwrap(); + tc.infer(&b).unwrap(); + assert_eq!(tc.try_same_head_spine(&a, &b).unwrap(), None); + assert!(!tc.app_congruence_probe(&a, &b).unwrap()); + assert!(!tc.is_def_eq(&a, &b).unwrap()); + + let partial = app(&mut tc, cnst("G"), cnst("a")); + let full = app(&mut tc, partial.clone(), cnst("b")); + tc.infer(&partial).unwrap(); + tc.infer(&full).unwrap(); + assert_eq!(tc.try_same_head_spine(&partial, &full).unwrap(), None); +} + +#[test] +fn same_head_is_not_enough_without_matching_universes_and_arity() { + universes_and_arity::(); + universes_and_arity::(); +} diff --git a/crates/kernel/src/def_eq/binders.rs b/crates/kernel/src/def_eq/binders.rs new file mode 100644 index 000000000..4e7e9f00c --- /dev/null +++ b/crates/kernel/src/def_eq/binders.rs @@ -0,0 +1,191 @@ +//! Positive-only batched lambda/forall congruence. +//! +//! Domains are compared under the already-accepted prefix, before adding +//! the SAME fresh local to both sides. The terminal bodies are opened once. +//! Each accepted binder therefore follows the existing single-binder rule. +//! See Ix/Tc/Verify/DefEq/Structural.lean (quickBinder_wf) and the lamDF / +//! forallEDF rules in lean4lean. Those proofs do not certify this Rust loop. +//! +//! This is not a complete conversion procedure for binder terms: skipping +//! intermediate conversion can miss reduction/proof-irrelevance/cache wins. +//! A false or exhausted probe falls back to the original single-binder path, +//! with batching disabled throughout that fallback. No pending/suffix pair +//! is published equal, and no failed probe is published as an inequality. + +use super::*; +use ix_common::env::{BinderInfo, Name}; + +const BINDER_BATCH_MIN_LENGTH: usize = 4; +const BINDER_BATCH_FUEL: u64 = 4_096; + +struct BinderPair<'a, M: KernelMode> { + name: &'a M::MField, + bi: &'a M::MField, + left_ty: &'a KExpr, + right_ty: &'a KExpr, + left_body: &'a KExpr, + right_body: &'a KExpr, +} + +fn binder_pair<'a, M: KernelMode>( + a: &'a KExpr, + b: &'a KExpr, +) -> Option> { + match (a.data(), b.data()) { + (ExprData::Lam(name, bi, at, ab, _), ExprData::Lam(_, _, bt, bb, _)) + | (ExprData::All(name, bi, at, ab, _), ExprData::All(_, _, bt, bb, _)) => { + Some(BinderPair { + name, + bi, + left_ty: at, + right_ty: bt, + left_body: ab, + right_body: bb, + }) + }, + _ => None, + } +} + +fn has_batch_prefix<'a, M: KernelMode>( + mut a: &'a KExpr, + mut b: &'a KExpr, +) -> bool { + for _ in 0..BINDER_BATCH_MIN_LENGTH { + let Some(pair) = binder_pair(a, b) else { return false }; + a = pair.left_body; + b = pair.right_body; + } + true +} + +impl TypeChecker<'_, M> { + pub(super) fn def_eq_binders( + &mut self, + a: &KExpr, + b: &KExpr, + ) -> Result> { + if self.in_binder_batch || !has_batch_prefix(a, b) { + return self.def_eq_one_binder(a, b); + } + self.in_binder_batch = true; + let saved_fuel = self.rec_fuel; + let local_fuel = saved_fuel.min(BINDER_BATCH_FUEL); + self.rec_fuel = local_fuel; + let result = self.with_lctx_scope(|tc| tc.def_eq_binder_telescope(a, b)); + let consumed = local_fuel.saturating_sub(self.rec_fuel); + self.rec_fuel = saved_fuel.saturating_sub(consumed); + let result = match result { + Ok(false) | Err(TcError::MaxRecDepth | TcError::MaxRecFuel) => { + // Do not start another failed probe at every recursive suffix. + self.def_eq_one_binder(a, b) + }, + other => other, + }; + self.in_binder_batch = false; + result + } + + /// Original binder comparison, also the reference path for differential + /// tests. Scope cleanup occurs on success, false, and every returned error. + fn def_eq_one_binder( + &mut self, + a: &KExpr, + b: &KExpr, + ) -> Result> { + let Some(pair) = binder_pair(a, b) else { return Ok(false) }; + if !self.is_def_eq(pair.left_ty, pair.right_ty)? { + return Ok(false); + } + self.with_lctx_scope(|tc| { + let id = tc.fresh_fvar_id(); + let fv = tc.intern(KExpr::fvar(id, pair.name.clone())); + tc.lctx.push( + id, + LocalDecl::CDecl { + name: pair.name.clone(), + bi: pair.bi.clone(), + ty: pair.left_ty.clone(), + }, + ); + let left = instantiate_rev( + &mut tc.env.intern, + pair.left_body, + std::slice::from_ref(&fv), + ); + let right = instantiate_rev(&mut tc.env.intern, pair.right_body, &[fv]); + tc.is_def_eq(&left, &right) + }) + } + + fn def_eq_binder_telescope<'a>( + &mut self, + mut a: &'a KExpr, + mut b: &'a KExpr, + ) -> Result> { + let mut fvars = Vec::new(); + loop { + // Opening the same raw expression under the same prefix yields the + // same expression. This is structural identity, not an in-flight pair. + if a.ptr_eq(b) || a.hash_key() == b.hash_key() { + return Ok(true); + } + // Closed suffixes are unchanged by opening: honor existing conversion + // caches without materializing suffixes merely to look them up. + if !fvars.is_empty() + && a.lbr() == 0 + && b.lbr() == 0 + && self.has_closed_binder_result(a, b) + { + return self.is_def_eq(a, b); + } + let Some(pair) = binder_pair(a, b) else { break }; + if !fvars.is_empty() { + // Corresponds to entering another nontrivial binder comparison. + // Keeps the iterative walk fuel-bounded without growing the stack. + self.tick()?; + crate::profile::bump_def_eq(); + } + // Each substitution has its own memo: distinct prefixes must never + // share a semantic substitution result just to reuse scratch storage. + let left_ty = instantiate_rev(&mut self.env.intern, pair.left_ty, &fvars); + let right_ty = + instantiate_rev(&mut self.env.intern, pair.right_ty, &fvars); + if !self.is_def_eq(&left_ty, &right_ty)? { + return Ok(false); + } + let id = self.fresh_fvar_id(); + let fv = self.intern(KExpr::fvar(id, pair.name.clone())); + self.lctx.push( + id, + LocalDecl::CDecl { + name: pair.name.clone(), + bi: pair.bi.clone(), + ty: left_ty, + }, + ); + fvars.push(fv); + a = pair.left_body; + b = pair.right_body; + } + let left = instantiate_rev(&mut self.env.intern, a, &fvars); + let right = instantiate_rev(&mut self.env.intern, b, &fvars); + self.is_def_eq(&left, &right) + } + + fn has_closed_binder_result(&mut self, a: &KExpr, b: &KExpr) -> bool { + let ctx = self.def_eq_ctx_key(a, b); + let (lo, hi) = canonical_pair(a.hash_key(), b.hash_key()); + let key = (lo, hi, ctx); + self.env.def_eq_cache.contains_key(&key) + || (self.cheap_recursion_depth > 0 + && self.env.def_eq_cheap_cache.contains_key(&key)) + || self.equiv_manager.is_equiv( + &crate::equiv::EqKey::new(a.hash_key(), ctx, 0, 0), + &crate::equiv::EqKey::new(b.hash_key(), ctx, 0, 0), + ) + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/kernel/src/def_eq/binders/tests.rs b/crates/kernel/src/def_eq/binders/tests.rs new file mode 100644 index 000000000..9c7a81277 --- /dev/null +++ b/crates/kernel/src/def_eq/binders/tests.rs @@ -0,0 +1,516 @@ +use super::*; +use crate::env::KEnv; +use crate::level::KUniv; +use crate::mode::{Anon, Meta}; +use crate::profile::{OpCounts, take_op_counts}; +use ix_common::address::Address; +use ix_common::env::{DefinitionSafety, ReducibilityHints}; + +fn name(s: &str) -> M::MField { + M::meta_field(Name::str(Name::anon(), s.to_owned())) +} + +fn id(s: &str) -> KId { + KId::new(Address::hash(s.as_bytes()), name::(s)) +} + +fn cnst(s: &str) -> KExpr { + KExpr::cnst(id(s), Box::new([])) +} + +fn var(i: u64) -> KExpr { + KExpr::var(i, name::("original-var")) +} + +fn sort() -> KExpr { + KExpr::sort(KUniv::succ(KUniv::zero())) +} + +fn binder( + is_lam: bool, + ty: KExpr, + body: KExpr, +) -> KExpr { + let n = name::(if is_lam { "lambda" } else { "forall" }); + let bi = M::meta_field(BinderInfo::Implicit); + if is_lam { KExpr::lam(n, bi, ty, body) } else { KExpr::all(n, bi, ty, body) } +} + +fn axiom(env: &mut KEnv, s: &str, ty: KExpr) { + env.insert( + id(s), + KConst::Axio { + name: name::(s), + level_params: M::meta_field(vec![]), + is_unsafe: false, + lvls: 0, + ty, + }, + ); +} + +fn defn( + env: &mut KEnv, + s: &str, + ty: KExpr, + val: KExpr, +) { + env.insert( + id(s), + KConst::Defn { + name: name::(s), + level_params: M::meta_field(vec![]), + kind: DefKind::Definition, + safety: DefinitionSafety::Safe, + hints: ReducibilityHints::Regular(7), + lvls: 0, + ty, + val, + lean_all: M::meta_field(vec![]), + block: id(s), + }, + ); +} + +fn setup() -> KEnv { + let mut env = KEnv::new(); + axiom(&mut env, "A", sort()); + axiom(&mut env, "B", sort()); + axiom(&mut env, "a", cnst("A")); + axiom(&mut env, "b", cnst("A")); + axiom( + &mut env, + "G", + binder(false, sort(), binder(false, var(0), binder(false, var(1), var(2)))), + ); + axiom(&mut env, "P", binder(false, sort(), binder(false, var(0), sort()))); + defn(&mut env, "Alias", sort(), cnst("A")); + defn(&mut env, "alias", cnst("A"), cnst("a")); + defn( + &mut env, + "id", + binder(false, cnst("A"), cnst("A")), + binder(true, cnst("A"), var(0)), + ); + defn( + &mut env, + "ignore", + binder(false, cnst("A"), cnst("A")), + binder(true, cnst("A"), cnst("a")), + ); + env +} + +fn telescope( + n: usize, + is_lam: bool, + mut body: KExpr, +) -> KExpr { + for _ in 0..n { + body = binder(is_lam, cnst("A"), body); + } + body +} + +fn compare( + a: &KExpr, + b: &KExpr, + old: bool, +) -> (bool, u64, u32, OpCounts) { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + tc.in_binder_batch = old; + let a = tc.intern(a.clone()); + let b = tc.intern(b.clone()); + take_op_counts(); + let verdict = tc.is_def_eq(&a, &b).unwrap(); + assert!(tc.lctx.is_empty()); + assert_eq!(tc.def_eq_depth, 0); + assert_eq!(tc.in_binder_batch, old); + (verdict, tc.fuel_used(), tc.def_eq_peak, take_op_counts()) +} + +fn long_positive( + count: u64, + compare_reference: bool, + report: bool, +) { + for is_lam in [false, true] { + // (T : Type) (x1 : T) ... (x_{count-1} : T). Unlike closed suffixes, + // these domains really must be traversed by repeated single opening. + let make = |alias: bool| { + // Every value binder is used, so each single opening must revisit + // the terminal DAG; merely depending on T would only traverse once. + let g = KExpr::app(cnst("G"), var(count - 1)); + let mut body = var(0); + for i in 1..count - 1 { + body = KExpr::app(KExpr::app(g.clone(), var(i)), body); + } + if !is_lam { + body = KExpr::app(KExpr::app(cnst("P"), var(count - 1)), body); + } + if alias { + let ty = if is_lam { var(count - 1) } else { sort() }; + body = KExpr::let_(name::("alias"), ty, body, var(0), false); + } + for i in (0..count).rev() { + let ty = if i == 0 { sort() } else { var(i - 1) }; + body = binder(is_lam, ty, body); + } + body + }; + let a = make(false); + let b = make(true); + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + tc.infer(&a).unwrap(); + tc.infer(&b).unwrap(); + let new = compare::(&a, &b, false); + assert!(new.0); + assert!(new.2 < 10, "batch peak {}", new.2); + if !compare_reference { + continue; + } + let old = compare::(&a, &b, true); + if report { + println!( + "mode={} lambda={is_lam} subst={}->{} intern={}->{} fuel={}->{} peak={}->{}", + std::any::type_name::(), + old.3.subst_nodes, + new.3.subst_nodes, + old.3.intern_nodes, + new.3.intern_nodes, + old.1, + new.1, + old.2, + new.2, + ); + } + assert!(old.0); + assert!(u64::from(old.2) >= count); + assert!( + new.3.subst_nodes * 4 < old.3.subst_nodes, + "subst {} vs {}", + new.3.subst_nodes, + old.3.subst_nodes + ); + } +} + +#[test] +fn long_telescope_opens_terminal_bodies_once() { + // The deliberately recursive reference needs one native frame per + // binder. Keep this differential/work-counter fixture within the normal + // test stack; exercise the full deep case on the production path below. + long_positive::(32, true, false); + long_positive::(32, true, false); +} + +#[test] +fn deep_batched_telescope_uses_default_stack() { + // Do not spawn a large-stack worker or require RUST_MIN_STACK here. + long_positive::(128, false, false); + long_positive::(128, false, false); +} + +#[test] +#[ignore = "manual paired work-counter report; not a wall-time benchmark"] +fn report_dependent_telescope_work() { + long_positive::(32, true, true); + long_positive::(32, true, true); +} + +fn dependent() { + let mut env = setup::(); + // idPoly : (T : Type) -> T -> T + let id_ty = binder(false, sort(), binder(false, var(0), var(1))); + let id_val = binder(true, sort(), binder(true, var(0), var(0))); + defn(&mut env, "idPoly", id_ty, id_val); + for is_lam in [false, true] { + let make = |alias: bool| { + // (T : Type) (x : T) (y : T) (z : T), x / T. + let terminal = if is_lam { var(2) } else { var(3) }; + let terminal = if alias && is_lam { + KExpr::app(KExpr::app(cnst("idPoly"), var(3)), terminal) + } else { + terminal + }; + let domain_z = if alias { + // A local type alias reduces to T; both domains are well typed. + KExpr::let_(name::("D"), sort(), var(2), var(0), false) + } else { + var(2) + }; + binder( + is_lam, + sort(), + binder( + is_lam, + var(0), + binder(is_lam, var(1), binder(is_lam, domain_z, terminal)), + ), + ) + }; + let mut tc = TypeChecker::new(&mut env); + let a = tc.intern(make(false)); + let b = tc.intern(make(true)); + tc.infer(&a).unwrap(); + tc.infer(&b).unwrap(); + assert!(tc.is_def_eq(&a, &b).unwrap()); + assert!(tc.lctx.is_empty()); + assert!(!tc.in_binder_batch); + } +} + +#[test] +fn dependent_domains_and_shared_fresh_locals_are_preserved() { + dependent::(); + dependent::(); +} + +fn negative_cases() { + for n in [4, 8, 17] { + // Same domains but unequal terminal values/types. + for is_lam in [false, true] { + let (a, b) = + if is_lam { (cnst("a"), cnst("b")) } else { (cnst("A"), cnst("B")) }; + let a = telescope(n, is_lam, a); + let b = telescope(n, is_lam, b); + assert!(!compare::(&a, &b, true).0); + assert!(!compare::(&a, &b, false).0); + } + let a = telescope(n, false, binder(false, cnst("A"), cnst("A"))); + let b = telescope(n, false, binder(false, cnst("B"), cnst("A"))); + assert!(!compare::(&a, &b, true).0); + assert!(!compare::(&a, &b, false).0); + } +} + +#[test] +fn failed_domains_and_terminal_comparisons_match_original() { + negative_cases::(); + negative_cases::(); +} + +fn diverging() { + let a = telescope(4, true, binder(true, cnst("A"), var(0))); + let b = telescope(4, true, cnst("id")); + assert!(compare::(&a, &b, true).0); + assert!(compare::(&a, &b, false).0); + let a = telescope(4, false, cnst("A")); + let b = telescope(3, false, cnst("A")); + assert!(!compare::(&a, &b, true).0); + assert!(!compare::(&a, &b, false).0); +} + +#[test] +fn divergent_suffix_uses_normal_conversion_including_eta() { + diverging::(); + diverging::(); +} + +fn contexts() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + // The same loose outer Var means different values in different legacy + // let contexts. Opening four binders must shift it down by exactly four. + let a = tc.intern(telescope(4, true, var(4))); + let b = tc.intern(telescope(4, true, cnst("a"))); + tc.push_let(cnst("A"), cnst("a")); + let first = tc.def_eq_ctx_key(&a, &b); + assert!(tc.is_def_eq(&a, &b).unwrap()); + assert_eq!(tc.depth(), 1); + assert!(tc.lctx.is_empty()); + tc.pop_local(); + tc.push_let(cnst("A"), cnst("b")); + let second = tc.def_eq_ctx_key(&a, &b); + assert_ne!(first, second); + assert!(!tc.is_def_eq(&a, &b).unwrap()); + assert!(tc.lctx.is_empty()); + assert_eq!(tc.depth(), 1); +} + +#[test] +fn batched_results_are_scoped_to_the_original_outer_context() { + contexts::(); + contexts::(); +} + +#[test] +fn resource_failure_restores_scope_flag_and_does_not_cache_false() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let a = tc.intern(telescope(16, true, cnst("a"))); + let b = tc.intern(telescope(16, true, cnst("alias"))); + let (lo, hi) = canonical_pair(a.hash_key(), b.hash_key()); + let key = (lo, hi, tc.def_eq_ctx_key(&a, &b)); + let before = tc.fresh_fvar_id(); + tc.rec_fuel = 2; + assert!(matches!(tc.is_def_eq(&a, &b), Err(TcError::MaxRecFuel))); + assert_eq!(tc.rec_fuel, 0); + assert!(tc.lctx.is_empty()); + assert!(!tc.in_binder_batch); + assert_eq!(tc.def_eq_depth, 0); + assert!(!tc.env.def_eq_cache.contains_key(&key)); + let after = tc.fresh_fvar_id(); + assert_ne!(before, after); + tc.rec_fuel = 100_000; + assert!(tc.is_def_eq(&a, &b).unwrap()); +} + +#[test] +fn non_resource_errors_propagate_and_restore_scope() { + let mut env = setup::(); + defn(&mut env, "bad", cnst("A"), KExpr::sort(KUniv::param(1, ()))); + let mut tc = TypeChecker::new(&mut env); + let a = tc.intern(telescope( + 4, + true, + KExpr::cnst(id("bad"), Box::new([KUniv::zero()])), + )); + let b = tc.intern(telescope(4, true, cnst("a"))); + assert!(matches!( + tc.is_def_eq(&a, &b), + Err(TcError::UnivParamOutOfRange { .. }) + )); + assert!(tc.lctx.is_empty()); + assert!(!tc.in_binder_batch); + assert_eq!(tc.def_eq_depth, 0); +} + +#[test] +fn short_binders_keep_original_work_and_reset_clears_flag() { + for n in 1..BINDER_BATCH_MIN_LENGTH { + let a = telescope(n, true, cnst::("a")); + let b = telescope(n, true, cnst::("alias")); + let old = compare(&a, &b, true); + let new = compare(&a, &b, false); + assert_eq!((old.0, old.1, old.2), (new.0, new.1, new.2)); + assert_eq!(old.3.subst_nodes, new.3.subst_nodes); + assert_eq!(old.3.intern_nodes, new.3.intern_nodes); + } + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + tc.in_binder_batch = true; + tc.reset(); + assert!(!tc.in_binder_batch); +} + +#[test] +fn generated_telescope_pairs_match_the_original_checker() { + fn check() { + for n in 0..64 { + let body = |k| { + let a = cnst(["a", "b", "alias"][k % 3]); + if k % 2 == 0 { a } else { KExpr::app(cnst("ignore"), a) } + }; + let a = telescope(4 + n % 5, true, body(n * 7)); + let b = telescope(4 + n % 5, true, body(n * 11)); + assert_eq!( + compare::(&a, &b, true).0, + compare::(&a, &b, false).0, + "seed {n}" + ); + } + } + check::(); + check::(); +} + +#[test] +fn closed_suffix_cache_is_honored_without_reopening_it() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let left_suffix = tc.intern(telescope(12, true, cnst("a"))); + let right_suffix = tc.intern(telescope(12, true, cnst("alias"))); + assert!(tc.is_def_eq(&left_suffix, &right_suffix).unwrap()); + assert!(tc.has_closed_binder_result(&left_suffix, &right_suffix)); + let a = tc.intern(telescope(4, true, left_suffix)); + let b = tc.intern(telescope(4, true, right_suffix)); + let fuel = tc.rec_fuel; + assert!(tc.is_def_eq(&a, &b).unwrap()); + assert!(fuel - tc.rec_fuel <= 4); + assert!(tc.lctx.is_empty()); +} + +#[test] +fn cheap_negative_cache_does_not_poison_full_binder_comparison() { + fn check() { + let mut env = setup::(); + let mut tc = TypeChecker::new(&mut env); + let a = tc.intern(telescope(4, true, cnst("a"))); + let b = tc.intern(telescope(4, true, cnst("alias"))); + let (lo, hi) = canonical_pair(a.hash_key(), b.hash_key()); + let key = (lo, hi, tc.def_eq_ctx_key(&a, &b)); + tc.env.def_eq_cheap_cache.insert(key, false); + assert!(tc.is_def_eq(&a, &b).unwrap()); + assert_eq!(tc.env.def_eq_cache.get(&key), Some(&true)); + tc.env.clear_reduction_caches(); + tc.equiv_manager.clear(); + tc.cheap_recursion_depth = 1; + tc.infer_only = true; + tc.eager_reduce = true; + assert!(tc.is_def_eq(&a, &b).unwrap()); + assert_eq!(tc.env.def_eq_cheap_cache.get(&key), Some(&true)); + assert_eq!(tc.env.def_eq_cache.get(&key), Some(&true)); + assert_eq!(tc.cheap_recursion_depth, 1); + assert!(tc.infer_only && tc.eager_reduce); + assert!(!tc.in_binder_batch); + assert!(tc.lctx.is_empty()); + } + check::(); + check::(); +} + +#[test] +fn local_budget_miss_falls_back_without_refunding_work() { + let mut env = setup::(); + let mut next = cnst("a"); + // Finite beta+delta chain: productive betas exceed the batch slice. + // Bare aliases alone use the separate local delta-loop guard and do not + // necessarily consume recursive fuel once their structural forms hit. + for i in 0..3000 { + let s = format!("step{i}"); + defn(&mut env, &s, cnst("A"), KExpr::app(cnst("id"), next)); + next = cnst(&s); + } + let mut tc = TypeChecker::new(&mut env); + let a = tc.intern(telescope(4, true, next)); + let b = tc.intern(telescope(4, true, cnst("a"))); + let before = tc.fresh_fvar_id().0; + let fuel = tc.rec_fuel; + assert!(tc.is_def_eq(&a, &b).unwrap()); + let after = tc.fresh_fvar_id().0; + assert!( + after - before >= 9, + "expected batch+fallback: new fvars={}, fuel consumed={}", + after - before, + fuel - tc.rec_fuel, + ); + assert!(fuel - tc.rec_fuel > BINDER_BATCH_FUEL); + assert!(!tc.in_binder_batch); + assert!(tc.lctx.is_empty()); +} + +#[test] +fn proof_irrelevance_inside_a_telescope_remains_available() { + fn check(old: bool) { + let mut env = setup::(); + axiom(&mut env, "PropP", KExpr::sort(KUniv::zero())); + axiom(&mut env, "proof1", cnst("PropP")); + axiom(&mut env, "proof2", cnst("PropP")); + let mut tc = TypeChecker::new(&mut env); + tc.in_binder_batch = old; + let a = tc.intern(telescope(8, true, cnst("proof1"))); + let b = tc.intern(telescope(8, true, cnst("proof2"))); + tc.infer(&a).unwrap(); + tc.infer(&b).unwrap(); + assert!(tc.is_def_eq(&a, &b).unwrap()); + assert!(tc.lctx.is_empty()); + assert_eq!(tc.in_binder_batch, old); + } + for old in [false, true] { + check::(old); + check::(old); + } +} diff --git a/crates/kernel/src/def_eq/projection_tests.rs b/crates/kernel/src/def_eq/projection_tests.rs new file mode 100644 index 000000000..b0e339691 --- /dev/null +++ b/crates/kernel/src/def_eq/projection_tests.rs @@ -0,0 +1,320 @@ +//! Projection-first probes may establish equality, never assume it on a miss. + +use super::*; +use crate::env::KEnv; +use crate::mode::{Anon, Meta}; +use ix_common::address::Address; +use ix_common::env::{BinderInfo, DefinitionSafety, Name, ReducibilityHints}; + +fn id(s: &str) -> KId { + KId::new( + Address::hash(s.as_bytes()), + M::meta_field(Name::str(Name::anon(), s.to_owned())), + ) +} + +fn cnst(s: &str) -> KExpr { + KExpr::cnst(id(s), Box::new([])) +} + +fn axiom(env: &mut KEnv, s: &str, ty: KExpr) { + env.insert( + id(s), + KConst::Axio { + name: M::meta_field(Name::anon()), + level_params: M::meta_field(vec![]), + is_unsafe: false, + lvls: 0, + ty, + }, + ); +} + +fn defn( + env: &mut KEnv, + s: &str, + ty: KExpr, + val: KExpr, +) { + env.insert( + id(s), + KConst::Defn { + name: M::meta_field(Name::anon()), + level_params: M::meta_field(vec![]), + kind: DefKind::Definition, + safety: DefinitionSafety::Safe, + hints: ReducibilityHints::Regular(7), + lvls: 0, + ty, + val, + lean_all: M::meta_field(vec![]), + block: id(s), + }, + ); +} + +/// A : Type; a b : A; Box : Type; Box.mk : A → Box. +/// `pack x` either stores x or ignores x and stores a. `neutral` is an axiom. +fn setup(keep_arg: bool) -> KEnv { + let mut env = KEnv::new(); + let type0 = KExpr::sort(KUniv::succ(KUniv::zero())); + axiom(&mut env, "A", type0.clone()); + axiom(&mut env, "a", cnst("A")); + axiom(&mut env, "b", cnst("A")); + env.insert( + id("Box"), + KConst::Indc { + name: M::meta_field(Name::anon()), + level_params: M::meta_field(vec![]), + lvls: 0, + params: 0, + indices: 0, + is_unsafe: false, + block: id("Box"), + member_idx: 0, + ty: type0, + ctors: vec![id("Box.mk")], + lean_all: M::meta_field(vec![]), + }, + ); + let fun_ty = KExpr::all( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + cnst("A"), + cnst("Box"), + ); + env.insert( + id("Box.mk"), + KConst::Ctor { + name: M::meta_field(Name::anon()), + level_params: M::meta_field(vec![]), + is_unsafe: false, + lvls: 0, + induct: id("Box"), + cidx: 0, + params: 0, + fields: 1, + ty: fun_ty.clone(), + }, + ); + let field = if keep_arg { + KExpr::var(0, M::meta_field(Name::anon())) + } else { + cnst("a") + }; + let body = KExpr::lam( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + cnst("A"), + KExpr::app(cnst("Box.mk"), field), + ); + defn(&mut env, "pack", fun_ty.clone(), body); + axiom(&mut env, "neutral", fun_ty); + env +} + +fn proj(head: &str, arg: KExpr) -> KExpr { + KExpr::prj(id("Box"), 0, KExpr::app(cnst(head), arg)) +} + +fn ignored_expensive_argument() { + let make = || { + let mut env = setup::(false); + let mut expensive = cnst("a"); + for i in 0..256 { + let s = format!("expensive.{i}"); + let step = KExpr::app( + KExpr::lam( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + cnst("A"), + KExpr::var(0, M::meta_field(Name::anon())), + ), + expensive, + ); + defn(&mut env, &s, cnst("A"), step); + expensive = cnst(&s); + } + (env, expensive) + }; + let (mut env, expensive) = make(); + let mut tc = TypeChecker::new(&mut env); + tc.rec_fuel = 128; + assert!( + tc.is_def_eq(&proj("pack", expensive.clone()), &proj("pack", cnst("b"))) + .unwrap() + ); + assert!(tc.rec_fuel > 0); + assert!(!tc.in_projection_probe); + assert_eq!(tc.def_eq_depth, 0); + assert_eq!(tc.depth(), 0); + + // The original whole-record-first path exhausts the SAME total budget. + let (mut old_env, expensive) = make(); + let mut old = TypeChecker::new(&mut old_env); + old.rec_fuel = 128; + assert!(matches!( + old.lazy_delta_proj_reduction( + &id("Box"), + 0, + &mut KExpr::app(cnst("pack"), expensive), + &mut KExpr::app(cnst("pack"), cnst("b")), + ), + Err(TcError::MaxRecFuel) + )); +} + +#[test] +fn projection_probe_avoids_ignored_argument_work() { + ignored_expensive_argument::(); + ignored_expensive_argument::(); +} + +fn unequal_fields() { + let mut env = setup::(true); + let mut tc = TypeChecker::new(&mut env); + let (a, b) = (proj("pack", cnst("a")), proj("pack", cnst("b"))); + assert!(!tc.try_projected_def_eq(&a, &b).unwrap()); + assert!(!tc.is_def_eq(&a, &b).unwrap()); + assert!(!tc.is_def_eq(&b, &a).unwrap()); + assert!(!tc.in_projection_probe); +} + +#[test] +fn projection_probe_rejects_unequal_fields() { + unequal_fields::(); + unequal_fields::(); +} + +fn neutral_fallback() { + let mut env = setup::(true); + let beta = KExpr::app( + KExpr::lam( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + cnst("A"), + KExpr::var(0, M::meta_field(Name::anon())), + ), + cnst("a"), + ); + let (a, b) = (proj("neutral", beta), proj("neutral", cnst("a"))); + let mut tc = TypeChecker::new(&mut env); + assert!(!tc.try_projected_def_eq(&a, &b).unwrap()); + assert!(tc.is_def_eq(&a, &b).unwrap()); +} + +#[test] +fn projection_probe_miss_retains_record_congruence() { + neutral_fallback::(); + neutral_fallback::(); +} + +fn guard_restoration() { + let mut env = setup::(true); + let mut tc = TypeChecker::new(&mut env); + let (a, b) = (proj("pack", cnst("a")), proj("pack", cnst("b"))); + tc.rec_fuel = 1; + assert!(!tc.try_projected_def_eq(&a, &b).unwrap()); + assert_eq!(tc.rec_fuel, 0, "probe work must not be refunded"); + assert!(!tc.in_projection_probe); + assert!(tc.env.def_eq_cache.is_empty()); + assert!(tc.env.def_eq_cheap_cache.is_empty()); + + tc.rec_fuel = 10_000; + tc.def_eq_depth = MAX_DEF_EQ_DEPTH; + assert!(!tc.try_projected_def_eq(&a, &b).unwrap()); + assert_eq!(tc.def_eq_depth, MAX_DEF_EQ_DEPTH); + assert!(tc.rec_fuel < 10_000); + assert!(!tc.in_projection_probe); + assert!(tc.env.def_eq_cache.is_empty()); + tc.def_eq_depth = 0; + assert!(!tc.is_def_eq(&a, &b).unwrap()); +} + +#[test] +fn projection_probe_guards_do_not_cache_verdicts_or_refund_work() { + guard_restoration::(); + guard_restoration::(); +} + +fn malformed_field() { + let mut env = setup::(true); + // Deliberately malformed declaration: its body references universe 1, + // while the supplied substitution contains only universe 0. + defn( + &mut env, + "bad", + cnst("Box"), + KExpr::sort(KUniv::param(1, M::meta_field(Name::anon()))), + ); + let mut tc = TypeChecker::new(&mut env); + let result = tc.try_projected_def_eq( + &KExpr::prj( + id("Box"), + 0, + KExpr::cnst(id("bad"), Box::new([KUniv::zero()])), + ), + &proj("pack", cnst("b")), + ); + assert!( + matches!(result, Err(TcError::UnivParamOutOfRange { .. })), + "{result:?}" + ); + assert!(!tc.in_projection_probe); + assert!(tc.rec_fuel < crate::tc::max_rec_fuel()); +} + +#[test] +fn projection_probe_propagates_non_budget_errors() { + malformed_field::(); + malformed_field::(); +} + +#[test] +fn projection_probe_does_not_nest_or_reset_enclosing_state() { + let mut env = setup::(false); + let mut tc = TypeChecker::new(&mut env); + tc.in_projection_probe = true; + let fuel = tc.rec_fuel; + assert!( + !tc + .try_projected_def_eq(&proj("pack", cnst("a")), &proj("pack", cnst("b"))) + .unwrap() + ); + assert_eq!(tc.rec_fuel, fuel); + assert!(tc.in_projection_probe); + tc.reset(); + assert!(!tc.in_projection_probe); +} + +fn exhausted_proposition_probe() { + let mut env = setup::(true); + let type0 = KExpr::sort(KUniv::succ(KUniv::zero())); + defn( + &mut env, + "propSort", + type0.clone(), + KExpr::app( + KExpr::lam( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + type0, + KExpr::var(0, M::meta_field(Name::anon())), + ), + KExpr::sort(KUniv::zero()), + ), + ); + axiom(&mut env, "P", cnst("propSort")); + let mut tc = TypeChecker::new(&mut env); + tc.rec_fuel = 0; + assert!(!tc.is_prop_type(&cnst("P"))); + assert!(tc.env.is_prop_cache.is_empty()); + tc.rec_fuel = 10_000; + assert!(tc.is_prop_type(&cnst("P"))); +} + +#[test] +fn projection_probe_fuel_misses_do_not_poison_proposition_cache() { + exhausted_proposition_probe::(); + exhausted_proposition_probe::(); +} diff --git a/crates/kernel/src/def_eq/same_head_tests.rs b/crates/kernel/src/def_eq/same_head_tests.rs new file mode 100644 index 000000000..6d937a52e --- /dev/null +++ b/crates/kernel/src/def_eq/same_head_tests.rs @@ -0,0 +1,392 @@ +use super::*; +use crate::{ + env::KEnv, + mode::{Anon, Meta}, +}; +use ix_common::{ + address::Address, + env::{BinderInfo, DefinitionSafety, Name, ReducibilityHints}, +}; + +fn id(name: &str) -> KId { + KId::new(Address::hash(name.as_bytes()), M::meta_field(Name::anon())) +} + +fn cnst(name: &str) -> KExpr { + KExpr::cnst(id(name), Box::new([])) +} + +fn defn( + env: &mut KEnv, + name: &str, + ty: KExpr, + val: KExpr, +) { + env.insert( + id(name), + KConst::Defn { + name: M::meta_field(Name::anon()), + level_params: M::meta_field(vec![]), + kind: DefKind::Definition, + safety: DefinitionSafety::Safe, + hints: ReducibilityHints::Regular(7), + lvls: 0, + ty, + val, + lean_all: M::meta_field(vec![]), + block: id(name), + }, + ); +} + +fn setup() -> KEnv { + let mut env = KEnv::new(); + for (name, ty) in [ + ("A", KExpr::sort(KUniv::succ(KUniv::zero()))), + ("a", cnst("A")), + ("b", cnst("A")), + ] { + env.insert( + id(name), + KConst::Axio { + name: M::meta_field(Name::anon()), + level_params: M::meta_field(vec![]), + is_unsafe: false, + lvls: 0, + ty, + }, + ); + } + for (name, body) in + [("ignore", cnst("a")), ("id", KExpr::var(0, M::meta_field(Name::anon())))] + { + let ty = KExpr::all( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + cnst("A"), + cnst("A"), + ); + let val = KExpr::lam( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + cnst("A"), + body, + ); + defn(&mut env, name, ty, val); + } + env +} + +/// A well-typed, finite alias chain with beta steps per link. Bare constant +/// aliases do not consume recursive fuel in the lazy-delta loop. Each body +/// is shallow, avoiding a deep expression/drop stack or enlarged thread stack. +fn beta_alias_chain( + env: &mut KEnv, + links: u64, + betas: usize, +) -> KExpr { + let mut val = cnst("a"); + let identity = KExpr::lam( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + cnst("A"), + KExpr::var(0, M::meta_field(Name::anon())), + ); + for i in 0..links { + let name = format!("alias{i}"); + for _ in 0..betas { + val = KExpr::app(identity.clone(), val); + } + defn(env, &name, cnst("A"), val); + val = cnst(&name); + } + val +} + +fn expensive(env: &mut KEnv) -> KExpr { + // Exceed the Regular allowance without reaching the separate 10k-iteration + // lazy-delta guard or constructing a deeply nested expression body. + beta_alias_chain(env, SAME_HEAD_SPECULATION_ATTEMPT_FUEL, 64) +} + +fn assert_not_cached( + tc: &mut TypeChecker<'_, M>, + a: &KExpr, + b: &KExpr, +) { + let (lo, hi) = canonical_pair(a.hash_key(), b.hash_key()); + let key = (lo, hi, tc.def_eq_ctx_key(a, b)); + assert!(!tc.env.def_eq_cache.contains_key(&key)); + assert!(!tc.env.def_eq_cheap_cache.contains_key(&key)); +} + +fn fallback(head: &str, expected: bool) { + let mut env = setup::(); + let arg = expensive(&mut env); + let head_expr = cnst(head); + let other_arg = cnst("b"); + let a = KExpr::app(head_expr.clone(), arg.clone()); + let b = KExpr::app(head_expr.clone(), other_arg.clone()); + let mut tc = TypeChecker::new(&mut env); + let before = tc.rec_fuel; + assert_eq!( + tc.try_same_head_spine_speculative(&a, &b, &id(head), true).unwrap(), + None + ); + assert_eq!(before - tc.rec_fuel, SAME_HEAD_REGULAR_ATTEMPT_FUEL); + assert_eq!(tc.same_head_fuel_reserve, 0); + assert!(!tc.same_head_backoff.should_skip(true)); + assert_not_cached(&mut tc, &a, &b); + assert_not_cached(&mut tc, &arg, &other_arg); + assert_eq!(tc.def_eq_depth, 0); + assert_eq!(tc.is_def_eq(&a, &b).unwrap(), expected); + // Confirm delta rather than only a result rescued by congruence. + assert!(tc.env.unfold_cache.contains_key(&head_expr.hash_key())); + assert!(tc.fuel_used() > SAME_HEAD_REGULAR_ATTEMPT_FUEL); +} + +#[test] +fn expensive_regular_probe_falls_back_without_accepting_unequal_terms() { + fallback::("ignore", true); + fallback::("ignore", true); + fallback::("id", false); + fallback::("id", false); +} + +fn nested(cheap: bool) { + let mut env = setup::(); + let arg = expensive(&mut env); + let other_arg = cnst("a"); + let inner_a = KExpr::app(cnst("id"), arg.clone()); + let inner_b = KExpr::app(cnst("id"), other_arg.clone()); + let a = KExpr::app(cnst("ignore"), inner_a.clone()); + let b = KExpr::app(cnst("ignore"), inner_b.clone()); + let mut tc = TypeChecker::new(&mut env); + tc.cheap_recursion_depth = u32::from(cheap); + // Simulate an enclosing probe with less than one fresh allowance. + tc.rec_fuel = 1_000; + crate::perf::same_head::reset(); + assert_eq!( + tc.try_same_head_spine_speculative(&a, &b, &id("ignore"), true).unwrap(), + None + ); + assert_eq!(tc.rec_fuel, 0, "nested probes must not replenish fuel"); + assert_eq!(tc.same_head_fuel_reserve, 0); + assert_not_cached(&mut tc, &a, &b); + assert_not_cached(&mut tc, &inner_a, &inner_b); + assert_not_cached(&mut tc, &arg, &other_arg); + assert_eq!(tc.def_eq_depth, 0); + if crate::perf::same_head::enabled() { + let report = crate::perf::same_head::summary(); + assert!(report.contains("active=0 max_depth=2 "), "{report}"); + assert!(report.contains("root_fuel=1000 "), "{report}"); + assert!(report.contains("accounting_errors=0"), "{report}"); + } + // Resuming the same environment must not observe poisoned negative caches. + tc.rec_fuel = crate::tc::max_rec_fuel(); + tc.cheap_recursion_depth = 0; + assert!(tc.is_def_eq(&inner_a, &inner_b).unwrap()); + assert!(tc.is_def_eq(&a, &b).unwrap()); +} + +#[test] +fn nested_regular_probes_share_remaining_fuel_without_cache_poisoning() { + for cheap in [false, true] { + nested::(cheap); + nested::(cheap); + } +} + +#[test] +fn productive_regular_probe_has_more_than_the_non_regular_allowance() { + let mut env = setup::(); + let arg = beta_alias_chain(&mut env, 128, 64); + let head = cnst("id"); + let a = KExpr::app(head.clone(), arg); + let b = KExpr::app(head.clone(), cnst("a")); + let mut tc = TypeChecker::new(&mut env); + assert!(tc.is_def_eq(&a, &b).unwrap()); + assert!(tc.fuel_used() > SAME_HEAD_SPECULATION_ATTEMPT_FUEL); + assert!(tc.fuel_used() < SAME_HEAD_REGULAR_ATTEMPT_FUEL); + assert!(!tc.env.unfold_cache.contains_key(&head.hash_key())); + assert_eq!(tc.same_head_fuel_reserve, 0); +} + +#[test] +fn late_regular_probe_still_uses_successful_congruence() { + let mut env = setup::(); + defn(&mut env, "alias", cnst("A"), cnst("a")); + let head = cnst("id"); + let a = KExpr::app(head.clone(), cnst("alias")); + let b = KExpr::app(head.clone(), cnst("a")); + let mut tc = TypeChecker::new(&mut env); + tc.rec_fuel -= SAME_HEAD_SPECULATION_START_FUEL; + let before = tc.rec_fuel; + assert!(tc.is_def_eq(&a, &b).unwrap()); + assert!(before - tc.rec_fuel < SAME_HEAD_SPECULATION_ATTEMPT_FUEL); + assert!(!tc.env.unfold_cache.contains_key(&head.hash_key())); +} + +#[test] +fn regular_reservation_preserves_non_regular_startup_window() { + for used in [0, SAME_HEAD_SPECULATION_START_FUEL] { + let mut env = setup::(); + defn(&mut env, "alias", cnst("A"), cnst("a")); + let mut abbrev = env.get(&id("id")).unwrap(); + if let KConst::Defn { hints, block, .. } = &mut abbrev { + *hints = ReducibilityHints::Abbrev; + *block = id("abbrev"); + } + env.insert(id("abbrev"), abbrev); + let head = cnst("abbrev"); + let a = KExpr::app(cnst("id"), KExpr::app(head.clone(), cnst("alias"))); + let b = KExpr::app(cnst("id"), KExpr::app(head.clone(), cnst("a"))); + let mut tc = TypeChecker::new(&mut env); + tc.rec_fuel -= used; + assert_eq!( + tc.try_same_head_spine_speculative(&a, &b, &id("id"), true).unwrap(), + Some(true) + ); + // The outer Regular allowance neither closes the early window nor + // reopens a late one. Only the latter case must delta-unfold the Abbrev. + assert_eq!(tc.env.unfold_cache.contains_key(&head.hash_key()), used > 0); + assert_eq!(tc.same_head_fuel_reserve, 0); + } +} + +#[test] +fn alternate_lazy_delta_path_falls_back_after_probe_abort() { + let mut env = setup::(); + let arg = expensive(&mut env); + let mut a = KExpr::app(cnst("ignore"), arg); + let mut b = KExpr::app(cnst("ignore"), cnst("b")); + let mut tc = TypeChecker::new(&mut env); + assert!(matches!( + tc.lazy_delta_reduction_step(&mut a, &mut b).unwrap(), + LazyDeltaStep::Equal + )); + assert!(tc.fuel_used() >= SAME_HEAD_REGULAR_ATTEMPT_FUEL); +} + +#[test] +fn resource_abort_is_unknown_but_other_errors_propagate() { + let mut env = setup::(); + defn(&mut env, "alias", cnst("A"), cnst("a")); + defn(&mut env, "bad", cnst("A"), KExpr::sort(KUniv::param(1, ()))); + let a = KExpr::app(cnst("id"), cnst("alias")); + let b = KExpr::app(cnst("id"), cnst("a")); + let mut tc = TypeChecker::new(&mut env); + tc.rec_fuel = 0; + assert_eq!( + tc.try_same_head_spine_speculative(&a, &b, &id("id"), true).unwrap(), + None + ); + assert!(matches!(tc.is_def_eq(&a, &b), Err(TcError::MaxRecFuel))); + assert_not_cached(&mut tc, &a, &b); + + tc.rec_fuel = 20_000; + tc.def_eq_depth = MAX_DEF_EQ_DEPTH; + assert_eq!( + tc.try_same_head_spine_speculative(&a, &b, &id("id"), true).unwrap(), + None + ); + assert_eq!(tc.def_eq_depth, MAX_DEF_EQ_DEPTH); + assert_eq!(tc.same_head_fuel_reserve, 0); + assert!( + tc.rec_fuel < 20_000 + && tc.rec_fuel > 20_000 - SAME_HEAD_SPECULATION_ATTEMPT_FUEL + ); + tc.def_eq_depth = 0; + assert!(tc.is_def_eq(&a, &b).unwrap()); + + let bad = + KExpr::app(cnst("id"), KExpr::cnst(id("bad"), Box::new([KUniv::zero()]))); + tc.rec_fuel = 1_000_000; + let before = tc.rec_fuel; + assert!(matches!( + tc.try_same_head_spine_speculative(&bad, &b, &id("id"), true), + Err(TcError::UnivParamOutOfRange { .. }) + )); + assert!( + tc.rec_fuel < before + && tc.rec_fuel > before - SAME_HEAD_SPECULATION_ATTEMPT_FUEL + ); + assert_eq!(tc.def_eq_depth, 0); + assert_eq!(tc.same_head_fuel_reserve, 0); + assert_not_cached(&mut tc, &bad, &b); +} + +fn seed_backoff(tc: &mut TypeChecker<'_, M>) { + tc.same_head_backoff.enter(); + tc.same_head_backoff.leave(true, true, speculation::FAILED_FUEL_TOTAL); +} + +fn backoff_fallback(head: &str, expected: bool, cheap: bool) { + let mut env = setup::(); + let a = KExpr::app(cnst(head), cnst("a")); + let b = KExpr::app(cnst(head), cnst("b")); + let mut tc = TypeChecker::new(&mut env); + seed_backoff(&mut tc); + tc.cheap_recursion_depth = u32::from(cheap); + let before = tc.rec_fuel; + crate::perf::same_head::reset(); + assert_eq!( + tc.try_same_head_spine_speculative(&a, &b, &id(head), true).unwrap(), + None + ); + assert_eq!(tc.rec_fuel, before, "a skip does not spend or refund fuel"); + assert_eq!(tc.same_head_fuel_reserve, 0); + assert_not_cached(&mut tc, &a, &b); + if crate::perf::same_head::enabled() { + let report = crate::perf::same_head::summary(); + assert!(report.contains("skipped_backoff=1"), "{report}"); + assert!(report.contains("active=0 max_depth=0"), "{report}"); + } + tc.cheap_recursion_depth = 0; + assert_eq!(tc.is_def_eq(&a, &b).unwrap(), expected); +} + +#[test] +fn cumulative_backoff_never_publishes_equality_or_inequality() { + for cheap in [false, true] { + backoff_fallback::("ignore", true, cheap); + backoff_fallback::("ignore", true, cheap); + backoff_fallback::("id", false, cheap); + backoff_fallback::("id", false, cheap); + } +} + +#[test] +fn cumulative_backoff_applies_to_both_lazy_delta_paths() { + let mut env = setup::(); + let mut a = KExpr::app(cnst("ignore"), cnst("a")); + let mut b = KExpr::app(cnst("ignore"), cnst("b")); + let mut tc = TypeChecker::new(&mut env); + seed_backoff(&mut tc); + assert!(matches!( + tc.lazy_delta_reduction_step(&mut a, &mut b).unwrap(), + LazyDeltaStep::Equal + )); + assert!(tc.fuel_used() < SAME_HEAD_SPECULATION_ATTEMPT_FUEL); +} + +#[test] +fn reset_restores_productive_regular_probes_without_reusing_fvar_ids() { + let mut env = setup::(); + let arg = beta_alias_chain(&mut env, 128, 64); + let a = KExpr::app(cnst("id"), arg); + let b = KExpr::app(cnst("id"), cnst("a")); + let mut tc = TypeChecker::new(&mut env); + seed_backoff(&mut tc); + assert!(tc.same_head_backoff.should_skip(true)); + let old_fvar = tc.fresh_fvar_id(); + tc.reset(); + assert_ne!(old_fvar, tc.fresh_fvar_id()); + assert!(!tc.same_head_backoff.should_skip(true)); + assert_eq!( + tc.try_same_head_spine_speculative(&a, &b, &id("id"), true).unwrap(), + Some(true) + ); + assert!(tc.fuel_used() > SAME_HEAD_SPECULATION_ATTEMPT_FUEL); + assert!(tc.fuel_used() < SAME_HEAD_REGULAR_ATTEMPT_FUEL); +} diff --git a/crates/kernel/src/def_eq/speculation.rs b/crates/kernel/src/def_eq/speculation.rs new file mode 100644 index 000000000..48acf3022 --- /dev/null +++ b/crates/kernel/src/def_eq/speculation.rs @@ -0,0 +1,109 @@ +//! Admission backoff for repeatedly unproductive same-head comparisons. +//! +//! This is scheduling history, never a semantic equality/inequality cache. +//! Only an outermost unsuccessful Regular attempt charges the history; +//! its fuel includes its descendants, which must not charge again. Admitted +//! attempts keep their normal allowance. Nested attempts still share the +//! outer fuel slice and are not independently denied by history. + +/// Preserve productive speculation in substantial checks while reserving +/// most of the default 100M fuel for ordinary conversion. Early backoff +/// regressed FLT: unsuccessful earlier pairs do not tell us whether +/// later comparisons of the same definition will be useful. +pub(super) const FAILED_FUEL_TOTAL: u64 = 33_554_432; + +/// The threshold stops *new* root admissions. The last admitted attempt may +/// cross it by at most its allowance; it is never refunded or +/// prematurely truncated just because its eventual result might be a miss. +#[derive(Default)] +pub(crate) struct SameHeadBackoff { + active: usize, + failed_fuel: u64, +} + +impl SameHeadBackoff { + #[inline] + pub(super) fn should_skip(&self, regular: bool) -> bool { + self.active == 0 && regular && self.failed_fuel >= FAILED_FUEL_TOTAL + } + + #[inline] + pub(super) fn enter(&mut self) { + self.active += 1; + } + + pub(super) fn leave( + &mut self, + regular: bool, + unsuccessful: bool, + consumed: u64, + ) { + debug_assert!(self.active > 0); + self.active -= 1; + if self.active != 0 || !regular || !unsuccessful || consumed == 0 { + return; + } + self.failed_fuel = self.failed_fuel.saturating_add(consumed); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn charge(s: &mut SameHeadBackoff, fuel: u64) { + s.enter(); + s.leave(true, true, fuel); + } + + #[test] + fn backoff_only_blocks_new_regular_roots_after_the_threshold() { + let mut s = SameHeadBackoff::default(); + charge(&mut s, FAILED_FUEL_TOTAL - 1); + assert!(!s.should_skip(true)); + charge(&mut s, 1); + assert!(s.should_skip(true)); + assert!(!s.should_skip(false)); + s.enter(); + assert!(!s.should_skip(true)); + s.leave(true, false, 100); + assert_eq!(s.failed_fuel, FAILED_FUEL_TOTAL); + } + + #[test] + fn nested_cost_is_charged_once_and_successes_do_not_charge() { + let mut s = SameHeadBackoff::default(); + s.enter(); + s.enter(); + s.leave(true, true, 90); + assert_eq!(s.failed_fuel, 0); + s.leave(true, true, 100); + assert_eq!(s.failed_fuel, 100); + s.enter(); + s.enter(); + s.leave(true, true, 90); + s.leave(true, false, 100); + assert_eq!(s.failed_fuel, 100); + assert_eq!(s.active, 0); + } + + #[test] + fn a_complete_admitted_attempt_can_cross_the_threshold_without_refund() { + let mut s = SameHeadBackoff::default(); + charge(&mut s, FAILED_FUEL_TOTAL - 1); + assert!(!s.should_skip(true)); + charge(&mut s, 131_072); + assert_eq!(s.failed_fuel, FAILED_FUEL_TOTAL + 131_071); + assert!(s.should_skip(true)); + } + + #[test] + fn non_regular_and_zero_cost_attempts_leave_no_history() { + let mut s = SameHeadBackoff::default(); + s.enter(); + s.leave(false, true, 1_000); + charge(&mut s, 0); + assert_eq!(s.failed_fuel, 0); + assert_eq!(s.active, 0); + } +} diff --git a/crates/kernel/src/env.rs b/crates/kernel/src/env.rs index 0f5c91349..4768cdf78 100644 --- a/crates/kernel/src/env.rs +++ b/crates/kernel/src/env.rs @@ -7,11 +7,13 @@ //! and move parallelism above the kernel state boundary. use std::collections::BTreeSet; +use std::collections::hash_map::Entry; use rustc_hash::{FxHashMap, FxHashSet}; use std::cell::OnceCell; use ix_common::address::Address; +use ix_common::env::{BinderInfo, Name}; use super::constant::{KConst, RecRule}; use super::error::TcError; @@ -22,6 +24,9 @@ use super::mode::KernelMode; use super::perf::PerfCounters; use super::primitive::Primitives; +mod scratch; +use scratch::ScratchMap; + /// Canonical identity of an expression or universe node: the /// intern-assigned uid. Plain `u64`, allocated from a process-global /// counter (`expr.rs::fresh_uid`) and NEVER reused, so uid equality @@ -109,15 +114,15 @@ pub struct InternTable { /// meaningful. pub(crate) canon_exprs: FxHashSet, pub(crate) canon_univs: FxHashSet, - /// Scratch buffer for `subst` / `simul_subst` per-call memoization, - /// keyed by `(addr, depth)`. Cleared on entry. Owned here so the - /// allocation persists across calls. - pub(crate) subst_scratch: FxHashMap<(Addr, u64), KExpr>, + /// Scratch buffer for `subst` / `simul_subst` / binder opening and closing, + /// keyed by `(addr, depth)`. Cleared on entry; retained allocation adapts + /// to recent occupancy without sharing logical entries between calls. + pub(crate) subst_scratch: ScratchMap<(Addr, u64), KExpr>, /// Scratch buffer for `lift` per-call memoization, keyed by /// `(addr, cutoff)`. Cleared on entry. Separate from `subst_scratch` /// because `lift` is invoked from inside `subst_cached`, and the two /// caches have different semantics, so they must not share entries. - pub(crate) lift_scratch: FxHashMap<(Addr, u64), KExpr>, + pub(crate) lift_scratch: ScratchMap<(Addr, u64), KExpr>, /// Pool of scratch maps for `clo_subst` per-call memoization, keyed by /// `(addr, depth)`. A pool rather than a single buffer because /// `clo_subst` re-enters itself through `clo_readback` of environment @@ -198,8 +203,8 @@ impl InternTable { exprs: FxHashMap::default(), canon_exprs: FxHashSet::default(), canon_univs: FxHashSet::default(), - subst_scratch: FxHashMap::default(), - lift_scratch: FxHashMap::default(), + subst_scratch: ScratchMap::default(), + lift_scratch: ScratchMap::default(), clo_scratch_pool: Vec::new(), } } @@ -218,6 +223,56 @@ impl InternTable { self.exprs.get(key).cloned() } + /// Construct an application only if its canonical parent is absent. + /// Equivalent to `intern_expr(KExpr::app(f, a))`, including child traversal + /// order and first-insert-wins metadata. Inputs need not be canonical. + pub(crate) fn intern_app(&mut self, f: &KExpr, a: &KExpr) -> KExpr { + crate::profile::bump_intern_nodes(); + let mut memo = InternMemo::default(); + // Keep both input roots alive until the call-local pointer memo is gone. + let cf = self.intern_expr_cached(f, &mut memo); + let ca = self.intern_expr_cached(a, &mut memo); + self.intern_expr_with(ExprKey::App(*cf.addr(), *ca.addr()), || { + KExpr::app(cf, ca) + }) + } + + /// Allocate-on-miss counterpart of `intern_expr(KExpr::all(...))`. + pub(crate) fn intern_all( + &mut self, + name: M::MField, + bi: M::MField, + ty: &KExpr, + body: &KExpr, + ) -> KExpr { + crate::profile::bump_intern_nodes(); + let mut memo = InternMemo::default(); + let ct = self.intern_expr_cached(ty, &mut memo); + let cb = self.intern_expr_cached(body, &mut memo); + self.intern_expr_with(ExprKey::All(*ct.addr(), *cb.addr()), || { + KExpr::all(name, bi, ct, cb) + }) + } + + /// Private constructor gate: `key` MUST describe `make()` exactly, with + /// children already canonical in this table. Never expose arbitrary keys + /// to callers. Entry lookup avoids a second parent probe on misses. + fn intern_expr_with( + &mut self, + key: ExprKey, + make: impl FnOnce() -> KExpr, + ) -> KExpr { + match self.exprs.entry(key) { + Entry::Occupied(entry) => entry.get().clone(), + Entry::Vacant(entry) => { + let e = make(); + debug_assert_eq!(entry.key(), &expr_key(&e)); + self.canon_exprs.insert(*e.addr()); + entry.insert(e).clone() + }, + } + } + /// Intern a universe: returns the canonical value for its structural /// identity, recursively canonicalizing children as needed so the /// shallow key is meaningful. @@ -360,17 +415,30 @@ impl InternTable { } }, ExprData::Const(id, us, _) => { - let cus: Box<[KUniv]> = - us.iter().map(|un| self.intern_univ_cached(un, memo)).collect(); - if cus.iter().zip(us.iter()).all(|(a, b)| a.ptr_eq(b)) { - input.clone() - } else { + // Most generated constants already have canonical levels. Delay the + // replacement buffer until the first changed child, without changing + // traversal order or re-interning the unchanged prefix. + let mut changed: Option>> = None; + for (i, un) in us.iter().enumerate() { + let cu = self.intern_univ_cached(un, memo); + if let Some(cus) = &mut changed { + cus.push(cu); + } else if !cu.ptr_eq(un) { + let mut cus = Vec::with_capacity(us.len()); + cus.extend(us[..i].iter().cloned()); + cus.push(cu); + changed = Some(cus); + } + } + if let Some(cus) = changed { KExpr::cnst_full( id.clone(), - cus, + cus.into_boxed_slice(), input.mdata().clone(), input.univ_decor().clone(), ) + } else { + input.clone() } }, ExprData::App(f, a, _) => { @@ -501,6 +569,7 @@ pub struct KEnvCacheSizes { pub unfold: usize, pub ingress: usize, pub is_prop: usize, + pub decl_summary: usize, pub is_rec: usize, pub recursor: usize, pub rec_majors: usize, @@ -531,6 +600,7 @@ impl KEnvCacheSizes { self.unfold, self.ingress, self.is_prop, + self.decl_summary, self.is_rec, self.recursor, self.rec_majors, @@ -547,7 +617,7 @@ impl std::fmt::Display for KEnvCacheSizes { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { write!( f, - "consts={} intern_exprs={} intern_univs={} whnf={}/{}/{}/{}/{} infer={}/{} def_eq={}/{}/{} unfold={} ingress={} is_prop={}", + "consts={} intern_exprs={} intern_univs={} whnf={}/{}/{}/{}/{} infer={}/{} def_eq={}/{}/{} unfold={} ingress={} is_prop={} decl_summary={}", self.consts, self.intern_exprs, self.intern_univs, @@ -564,6 +634,7 @@ impl std::fmt::Display for KEnvCacheSizes { self.unfold, self.ingress, self.is_prop, + self.decl_summary, ) } } @@ -671,6 +742,14 @@ pub struct KEnv { /// is the dominant cost on mathlib proof-heavy blocks, where the same /// propositions are tested for equality thousands of times. pub is_prop_cache: FxHashMap<(Addr, CtxAddr), bool>, + /// Conservative proof-eligibility summaries keyed by the exact Const + /// expression UID (including its instantiated universes). Types in this + /// declaration environment are the same assumptions ordinary inference + /// consults; no argument or declaration validation is replaced by a hit. + /// Clear on environment resets and declaration replacement, including + /// replacements of dependencies consulted while constructing a summary. + pub(crate) decl_summary_cache: + FxHashMap, /// Computed `is_rec` per inductive, keyed by content address pub is_rec_cache: FxHashMap, /// Generated recursors, keyed by inductive Muts block id. @@ -764,6 +843,7 @@ impl KEnv { nat_succ_stuck: FxHashSet::default(), ingress_cache: FxHashMap::default(), is_prop_cache: FxHashMap::default(), + decl_summary_cache: FxHashMap::default(), is_rec_cache: FxHashMap::default(), recursor_cache: FxHashMap::default(), recursor_aux_order, @@ -824,7 +904,9 @@ impl KEnv { id.addr.hex() ); } - self.consts.insert(id, c); + if self.consts.insert(id, c).is_some() { + self.decl_summary_cache.clear(); + } } pub fn len(&self) -> usize { @@ -884,6 +966,7 @@ impl KEnv { self.nat_succ_stuck.clear(); self.ingress_cache.clear(); self.is_prop_cache.clear(); + self.decl_summary_cache.clear(); self.recursor_cache.clear(); self.rec_majors_cache.clear(); self.block_peer_agreement_cache.clear(); @@ -914,6 +997,7 @@ impl KEnv { unfold: self.unfold_cache.len(), ingress: self.ingress_cache.len(), is_prop: self.is_prop_cache.len(), + decl_summary: self.decl_summary_cache.len(), is_rec: self.is_rec_cache.len(), recursor: self.recursor_cache.len(), rec_majors: self.rec_majors_cache.len(), @@ -948,6 +1032,7 @@ impl KEnv { self.nat_succ_stuck = FxHashSet::default(); self.ingress_cache = FxHashMap::default(); self.is_prop_cache = FxHashMap::default(); + self.decl_summary_cache = FxHashMap::default(); self.recursor_cache = FxHashMap::default(); self.rec_majors_cache = FxHashMap::default(); self.block_peer_agreement_cache = FxHashSet::default(); @@ -956,6 +1041,60 @@ impl KEnv { self.next_fvar_id = 0; } + /// Clear the same logical state as [`Self::clear`], retaining only modest + /// backing tables for the next scheduled check. + /// + /// `max_capacity` is an entry-capacity bound on EACH cleared collection, + /// not a byte budget for the environment or process. Oversized allocations + /// are discarded, not shrunk/reallocated. Expression references, canonical + /// uid sets, scope-sensitive memo entries and block results are all cleared + /// before free-variable ids can be reused. The existing address-keyed + /// `is_rec_cache`, profile sink and configuration survive just as they do + /// with `clear` and `clear_releasing_memory`. + pub fn clear_with_capacity_limit(&mut self, max_capacity: usize) { + // Keep logical reset centralized: capacity reuse must never accidentally + // retain an entry when the ordinary reset gains another cache. + self.clear(); + macro_rules! release_oversized { + ($($table:expr),+ $(,)?) => { + $(if $table.capacity() > max_capacity { + $table = Default::default(); + })+ + }; + } + release_oversized!( + self.consts, + self.blocks, + self.intern.univs, + self.intern.exprs, + self.intern.canon_exprs, + self.intern.canon_univs, + self.intern.subst_scratch, + self.intern.lift_scratch, + self.intern.clo_scratch_pool, + self.whnf_cache, + self.whnf_no_delta_cache, + self.whnf_no_delta_cheap_cache, + self.whnf_core_cache, + self.whnf_core_cheap_cache, + self.infer_cache, + self.infer_only_cache, + self.def_eq_cache, + self.def_eq_cheap_cache, + self.def_eq_failure, + self.unfold_cache, + self.nat_succ_stuck, + self.ingress_cache, + self.is_prop_cache, + self.decl_summary_cache, + self.recursor_cache, + self.rec_majors_cache, + self.block_peer_agreement_cache, + self.block_check_results, + self.prim_family_cache, + ); + } + /// Clear only the reduction-memo caches (whnf / infer / def-eq / unfold / /// is-prop). Structural caches (`consts`, `blocks`, `intern`, recursor /// caches, `block_check_results`) and the profile sink are preserved. @@ -966,6 +1105,7 @@ impl KEnv { /// the in-circuit cost, which has no cross-constant memoization. Clearing a /// pure memo never affects correctness — only performance. pub fn clear_reduction_caches(&mut self) { + self.decl_summary_cache.clear(); self.whnf_cache.clear(); self.whnf_no_delta_cache.clear(); self.whnf_no_delta_cheap_cache.clear(); @@ -1046,6 +1186,89 @@ mod tests { assert!(env.get(&mk_id("missing")).is_none()); } + #[test] + fn bounded_clear_reuses_small_tables_but_not_logical_entries() { + let mut env = KEnv::::new(); + let id = mk_id("old"); + let ctx = blake3::hash(b"old context"); + let old = env.intern.intern_expr(KExpr::var(0, ())); + let key = (old.hash_key(), ctx); + env.insert(id.clone(), mk_axio("old")); + env.blocks.insert(id.clone(), vec![id.clone()]); + env.whnf_cache.insert(key, old.clone()); + env.infer_cache.insert(key, old.clone()); + env.infer_only_cache.insert(key, old.clone()); + env.def_eq_cache.insert((key.0, key.0, ctx), true); + env.block_check_results.insert(id.clone(), Ok(())); + env + .intern + .subst_scratch + .restore_after_call(FxHashMap::from_iter([((key.0, 0), old.clone())])); + env + .intern + .lift_scratch + .restore_after_call(FxHashMap::from_iter([((key.0, 0), old.clone())])); + env + .intern + .clo_scratch_pool + .push(FxHashMap::from_iter([((key.0, 0), old.clone())])); + assert_eq!(env.fresh_fvar_id(), FVarId(0)); + assert_eq!(env.fresh_fvar_id(), FVarId(1)); + let capacities = ( + env.consts.capacity(), + env.intern.exprs.capacity(), + env.whnf_cache.capacity(), + ); + + env.clear_with_capacity_limit(128); + + assert_eq!(env.cache_sizes().max(), 0); + assert!(env.intern.canon_exprs.is_empty()); + assert!(env.intern.canon_univs.is_empty()); + assert!(env.intern.subst_scratch.is_empty()); + assert!(env.intern.lift_scratch.is_empty()); + assert!(env.intern.clo_scratch_pool.is_empty()); + assert_eq!(env.fresh_fvar_id(), FVarId(0)); + assert_eq!( + capacities, + ( + env.consts.capacity(), + env.intern.exprs.capacity(), + env.whnf_cache.capacity(), + ) + ); + let new = env.intern.intern_expr(KExpr::var(0, ())); + assert!(!old.ptr_eq(&new), "reset must release the old canonical entry"); + } + + #[test] + fn bounded_clear_discards_oversized_allocations() { + let mut env = KEnv::::new(); + env.consts.reserve(256); + env.intern.exprs.reserve(256); + env.intern.canon_exprs.reserve(256); + env.whnf_cache.reserve(256); + let mut scratch = env.intern.subst_scratch.take_for_call(); + scratch.reserve(256); + env.intern.subst_scratch.restore_after_call(scratch); + env.intern.clo_scratch_pool.reserve(256); + env.infer_cache.reserve(16); + let small_capacity = env.infer_cache.capacity(); + + env.clear_with_capacity_limit(128); + + assert_eq!(env.consts.capacity(), 0); + assert_eq!(env.intern.exprs.capacity(), 0); + assert_eq!(env.intern.canon_exprs.capacity(), 0); + assert_eq!(env.whnf_cache.capacity(), 0); + assert_eq!(env.intern.subst_scratch.capacity(), 0); + assert_eq!(env.intern.clo_scratch_pool.capacity(), 0); + assert_eq!(env.infer_cache.capacity(), small_capacity); + + env.clear_with_capacity_limit(0); + assert_eq!(env.infer_cache.capacity(), 0); + } + #[test] fn get_by_id_works() { let mut env = KEnv::::new(); diff --git a/crates/kernel/src/env/scratch.rs b/crates/kernel/src/env/scratch.rs new file mode 100644 index 000000000..ca26e3d54 --- /dev/null +++ b/crates/kernel/src/env/scratch.rs @@ -0,0 +1,190 @@ +//! Reuse allocation, never logical memo entries, between independent calls. + +use rustc_hash::FxHashMap; + +// Entry-capacity floor, not a bound on a live traversal's memory. These are +// storage heuristics only: a call may grow its memo without limit as before. +const SMALL_CAPACITY: usize = 4_096; +const SPARSE_RATIO: usize = 16; +const SPARSE_USES_BEFORE_RELEASE: u8 = 2; + +/// A per-call memo whose backing allocation follows recent occupancy. +/// +/// A bulk traversal can grow a table that subsequent tiny calls spend most +/// of their time clearing: HashMap::clear scans control bytes across its +/// capacity, even when few entries remain. Release an oversized allocation +/// after two consecutive uses below 1/16 occupancy. Keeping one sparse use +/// tolerates alternating large/small calls without repeatedly regrowing the +/// large table. Small and well-used tables retain their allocation. +/// +/// Entries from the previous call are retained solely to observe occupancy +/// and are ALWAYS removed before handing the map to the next call. Neither +/// keys nor values nor within-call memoization semantics depend on sizing. +pub(crate) struct ScratchMap { + map: FxHashMap, + sparse_uses: u8, +} + +impl Default for ScratchMap { + fn default() -> Self { + Self { map: FxHashMap::default(), sparse_uses: 0 } + } +} + +impl ScratchMap { + /// Borrow an EMPTY memo for one traversal. The owner keeps only sizing + /// history and an empty placeholder until `restore_after_call`. + pub(crate) fn take_for_call(&mut self) -> FxHashMap { + let capacity = self.map.capacity(); + if capacity > SMALL_CAPACITY && self.map.len() < capacity / SPARSE_RATIO { + self.sparse_uses += 1; + if self.sparse_uses == SPARSE_USES_BEFORE_RELEASE { + // Drop directly: clearing first would scan the oversized allocation + // once more just to discard it. A fresh map allocates only on insert. + self.map = FxHashMap::default(); + self.sparse_uses = 0; + } + } else { + self.sparse_uses = 0; + } + let mut map = std::mem::take(&mut self.map); + map.clear(); + map + } + + pub(crate) fn restore_after_call(&mut self, map: FxHashMap) { + debug_assert_eq!(self.map.capacity(), 0, "scratch is already restored"); + self.map = map; + } + + /// Ordinary environment reset preserves capacity but not sizing history. + pub(crate) fn clear(&mut self) { + self.map.clear(); + self.sparse_uses = 0; + } + + pub(crate) fn capacity(&self) -> usize { + self.map.capacity() + } + + #[cfg(test)] + pub(crate) fn is_empty(&self) -> bool { + self.map.is_empty() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Arc; + + fn large_scratch() -> ScratchMap { + let mut scratch = ScratchMap::default(); + scratch.map.reserve(SMALL_CAPACITY * 4); + scratch + } + + fn fill(scratch: &mut ScratchMap, len: usize) { + assert!(scratch.map.is_empty()); + scratch.map.extend((0..len).map(|i| (i, i))); + } + + #[test] + fn scratch_discards_only_after_two_sparse_uses() { + let mut scratch = large_scratch(); + let large = scratch.capacity(); + fill(&mut scratch, 1); + let mut map = scratch.take_for_call(); + assert!(map.is_empty()); + assert_eq!(map.capacity(), large, "one sparse use retains capacity"); + map.insert(2, 3); + scratch.restore_after_call(map); + let map = scratch.take_for_call(); + assert!(map.is_empty()); + assert_eq!(map.capacity(), 0, "sustained sparse use releases capacity"); + assert_eq!(scratch.sparse_uses, 0); + } + + #[test] + fn scratch_retains_small_tables() { + let mut scratch = ScratchMap::::default(); + scratch.map.reserve(128); + let capacity = scratch.capacity(); + assert!(capacity <= SMALL_CAPACITY); + for _ in 0..8 { + fill(&mut scratch, 1); + let map = scratch.take_for_call(); + assert!(map.is_empty()); + assert_eq!(map.capacity(), capacity); + scratch.restore_after_call(map); + } + } + + #[test] + fn scratch_retains_well_used_large_tables_at_ratio_boundary() { + let mut scratch = large_scratch(); + let capacity = scratch.capacity(); + for _ in 0..8 { + fill(&mut scratch, capacity / SPARSE_RATIO); + let map = scratch.take_for_call(); + assert!(map.is_empty()); + assert_eq!(map.capacity(), capacity); + assert_eq!(scratch.sparse_uses, 0); + scratch.restore_after_call(map); + } + } + + #[test] + fn scratch_alternating_large_small_uses_do_not_churn() { + let mut scratch = large_scratch(); + let capacity = scratch.capacity(); + for _ in 0..8 { + for len in [1, capacity / 2] { + fill(&mut scratch, len); + let map = scratch.take_for_call(); + assert!(map.is_empty()); + assert_eq!(map.capacity(), capacity); + scratch.restore_after_call(map); + } + } + } + + #[test] + fn scratch_empty_large_table_also_releases() { + let mut scratch = large_scratch(); + let map = scratch.take_for_call(); + scratch.restore_after_call(map); + assert_eq!(scratch.take_for_call().capacity(), 0); + } + + #[test] + fn scratch_clear_resets_history_and_releases_values() { + let mut scratch = large_scratch(); + let capacity = scratch.capacity(); + fill(&mut scratch, 1); + let mut map = scratch.take_for_call(); + map.insert(1, 2); + scratch.restore_after_call(map); + assert_eq!(scratch.sparse_uses, 1); + scratch.clear(); + assert!(scratch.is_empty()); + assert_eq!(scratch.sparse_uses, 0); + assert_eq!(scratch.capacity(), capacity); + assert_eq!(scratch.take_for_call().capacity(), capacity); + } + + #[test] + fn scratch_never_returns_old_entries_or_retains_their_values() { + let mut scratch = ScratchMap::>::default(); + scratch.map.reserve(SMALL_CAPACITY * 4); + for key in 0..4 { + let value = Arc::new(key); + let weak = Arc::downgrade(&value); + scratch.map.insert(key, value); + let map = scratch.take_for_call(); + assert!(map.is_empty()); + assert!(weak.upgrade().is_none(), "old memo value survived reset"); + scratch.restore_after_call(map); + } + } +} diff --git a/crates/kernel/src/infer.rs b/crates/kernel/src/infer.rs index af3302807..291d91934 100644 --- a/crates/kernel/src/infer.rs +++ b/crates/kernel/src/infer.rs @@ -10,6 +10,11 @@ use super::mode::KernelMode; use super::subst::{abstract_fvars, cheap_beta_reduce, instantiate_rev, subst}; use super::tc::{TypeChecker, collect_app_spine}; +mod application; +pub(crate) use application::PrefixAdmission; +mod binders; +pub(crate) mod summary; + /// Emit detailed `[app diff]` trace when `infer`'s App path rejects an /// argument via `AppTypeMismatch`. Off by default — every rejection in a /// kernel-check pass would print multiple whnf dumps per failing constant, @@ -107,6 +112,10 @@ impl TypeChecker<'_, M> { self.instantiate_univ_params(&ty, &us_vec)? }, + ExprData::App(f, _, _) if matches!(f.data(), ExprData::App(..)) => { + self.infer_app_spine(e)? + }, + ExprData::App(f, a, _) => { let f_ty = self.infer(f)?; let (dom, cod) = self.ensure_forall(&f_ty).inspect_err(|_err| { @@ -135,68 +144,24 @@ impl TypeChecker<'_, M> { } } })?; - if !infer_only { - let a_ty = self.infer(a)?; - let is_eager = self.is_eager_reduce(a); - if is_eager { - self.eager_reduce = true; - } - let eq = self.is_def_eq(&a_ty, &dom)?; - if is_eager { - self.eager_reduce = false; - } - if !eq { - if *IX_APP_DIFF && self.debug_label_matches_env() { - // WHNF both sides so we can see where reduction actually - // terminates. The raw `a_ty` / `dom` are already in the - // error — what's useful here is the post-whnf forms and - // whether they converge under `is_def_eq`'s lazy unfold - // strategy. - // - // stderr, not the log facade: no log backend is installed - // in the CLI or test binaries, so `log::info!` dumps are - // silently dropped (same fix as the inductive.rs - // canonicity dumps). - let a_whnf = self.whnf(&a_ty); - let d_whnf = self.whnf(&dom); - let depth = crate::env_var("IX_APP_DIFF_DEPTH") - .ok() - .and_then(|s| s.parse::().ok()) - .unwrap_or(2); - eprintln!( - "[app diff] AppTypeMismatch at depth={} in {}", - self.ctx.len(), - self.debug_label.as_deref().unwrap_or("") - ); - eprintln!(" f: {}", compact_expr(f)); - eprintln!(" a: {}", compact_expr(a)); - eprintln!(" a_ty: {}", compact_expr_deep(&a_ty, depth)); - eprintln!(" dom: {}", compact_expr_deep(&dom, depth)); - eprintln!(" a_ty data: {:?}", a_ty.data()); - eprintln!(" dom data: {:?}", dom.data()); - match &a_whnf { - Ok(w) => { - eprintln!(" a_ty whnf: {}", compact_expr_deep(w, depth)) - }, - Err(e) => eprintln!(" a_ty whnf: ERR {e}"), - } - match &d_whnf { - Ok(w) => { - eprintln!(" dom whnf: {}", compact_expr_deep(w, depth)) - }, - Err(e) => eprintln!(" dom whnf: ERR {e}"), - } - } - return Err(TcError::AppTypeMismatch { - a_ty, - dom, - depth: self.ctx.len(), - }); - } - } + self.check_app_argument(f, a, &dom)?; subst(&mut self.env.intern, &cod, a, 0) }, + // Avoid telescope-vector allocation on the existing single-binder + // path. Telescopes can open their terminal body once per batch. + ExprData::Lam(_, _, _, body, _) + if matches!(body.data(), ExprData::Lam(..)) => + { + self.infer_lambda_telescope(e)? + }, + + ExprData::All(_, _, _, body, _) + if matches!(body.data(), ExprData::All(..)) => + { + self.infer_forall_telescope(e)? + }, + ExprData::Lam(name, bi, ty, body, _) => { if !infer_only { let t = self.infer(ty)?; diff --git a/crates/kernel/src/infer/application.rs b/crates/kernel/src/infer/application.rs new file mode 100644 index 000000000..c6579b4e6 --- /dev/null +++ b/crates/kernel/src/infer/application.rs @@ -0,0 +1,197 @@ +//! Application inference with delayed telescope substitution. +//! +//! At each step, `ty` under `pending` denotes exactly the type obtained by +//! sequential App inference. Check the next instantiated domain, then peel +//! its raw Pi body. Materialize the residual type at a non-Pi boundary or at +//! the end, plus at most one repeated prefix selected for caching. Arguments +//! live in the caller's context, NOT under the +//! peeled binders, so simultaneous substitution must lift them under nested +//! binders; `instantiate_rev`'s FVar-only shortcut is not appropriate here. + +use smallvec::SmallVec; + +use super::*; +use crate::subst::simul_subst; + +mod prefix; +pub(crate) use prefix::PrefixAdmission; + +impl TypeChecker<'_, M> { + pub(super) fn infer_app_spine( + &mut self, + e: &KExpr, + ) -> Result, TcError> { + // Borrow original prefix nodes (no reconstruction just to probe a cache). + // Stop at the nearest cached prefix in the caller's inference mode. + let mut prefixes: SmallVec<[&KExpr; 8]> = SmallVec::new(); + let mut args: SmallVec<[KExpr; 8]> = SmallVec::new(); + let mut hot_prefix = None; + let mut head = e; + let cached = loop { + let ExprData::App(f, a, _) = head.data() else { break None }; + if !args.is_empty() { + let key = self.infer_key(head); + if let Some(ty) = self.env.infer_cache.get(&key) { + self.env.perf.record_infer_hit(); + break Some(ty.clone()); + } + self.env.perf.record_infer_miss(); + if self.infer_only + && let Some(ty) = self.env.infer_only_cache.get(&key) + { + self.env.perf.record_infer_only_hit(); + break Some(ty.clone()); + } + if self.infer_only { + self.env.perf.record_infer_only_miss(); + self.record_hot_miss("infer-only", head); + } else { + self.record_hot_miss("infer", head); + } + // Prefer the longest repeated prefix. At most ONE dependent suffix + // may be materialized solely for caching in this spine inference. + // Merely observing a miss never publishes a type or validates a term. + if self.prefix_admission.observe(key, self.infer_only) + && hot_prefix.is_none() + { + hot_prefix = Some(prefixes.len()); + } + } + prefixes.push(head); + args.push(a.clone()); + head = f; + }; + let mut ty = match cached { + Some(ty) => ty, + None => self.infer(head)?, + }; + + // args is innermost-de-Bruijn-first (reverse application order). At + // position i, args[i+1..end] contains the already-consumed arguments + // since the last materialization. Slicing avoids reversing/copying a + // growing argument vector for every dependent domain. + let mut end = args.len(); + for i in (0..args.len()).rev() { + let ExprData::App(f, _, _) = prefixes[i].data() else { + unreachable!("only application prefixes were collected") + }; + let (dom, cod) = if let ExprData::All(_, _, dom, cod, _) = ty.data() { + let dom = + instantiate_pending(&mut self.env.intern, dom, &args[i + 1..end]); + (dom, cod.clone()) + } else { + // A reducible type/let or a substituted type variable hides the + // next Pi. Flush in the ambient context BEFORE normalization. + ty = instantiate_pending(&mut self.env.intern, &ty, &args[i + 1..end]); + end = i + 1; + let result = self.ensure_forall(&ty); + if result.is_err() + && *IX_INFER_APP_FORALL_DUMP + && self.debug_label_matches_env() + { + log::info!( + "[infer App batch] ensure_forall FAILED: f={f}, f_ty={ty}, a={}", + args[i] + ); + } + result? + }; + self.check_app_argument(f, &args[i], &dom)?; + ty = cod; + + // Every preceding argument has now succeeded in the caller's mode. + // Preserve cheap results and, for ONE repeated prefix, materialize the + // residual type with the SAME ambient-context substitution as the final + // result. Store it only under the existing exact key/mode contract. + // Keep `ty` and `end` delayed for dependent results: caching must not + // turn the rest of the spine back into sequential suffix substitution. + if i > 0 && (ty.lbr() == 0 || hot_prefix == Some(i)) { + let cached_ty = if ty.lbr() == 0 { + ty.clone() + } else { + self.env.perf.record_dependent_prefix_insert(); + instantiate_pending(&mut self.env.intern, &ty, &args[i..end]) + }; + let key = self.infer_key(prefixes[i]); + if self.infer_only { + self.env.infer_only_cache.insert(key, cached_ty); + } else { + self.env.infer_cache.insert(key, cached_ty); + } + if ty.lbr() == 0 { + end = i; + } + } + } + Ok(instantiate_pending(&mut self.env.intern, &ty, &args[..end])) + } + + /// Shared by the single-App and batched paths. Validation is never elided + /// by batching; infer-only continues to have a separate cache contract. + pub(super) fn check_app_argument( + &mut self, + f: &KExpr, + a: &KExpr, + dom: &KExpr, + ) -> Result<(), TcError> { + if self.infer_only { + return Ok(()); + } + let a_ty = self.infer(a)?; + let saved_eager = self.eager_reduce; + self.eager_reduce |= self.is_eager_reduce(a); + let eq = self.is_def_eq(&a_ty, dom); + // Restore even on errors and preserve an enclosing eager scope. + self.eager_reduce = saved_eager; + if eq? { + return Ok(()); + } + if *IX_APP_DIFF && self.debug_label_matches_env() { + let a_whnf = self.whnf(&a_ty); + let d_whnf = self.whnf(dom); + let depth = crate::env_var("IX_APP_DIFF_DEPTH") + .ok() + .and_then(|s| s.parse::().ok()) + .unwrap_or(2); + eprintln!( + "[app diff] AppTypeMismatch at depth={} in {}", + self.ctx.len(), + self.debug_label.as_deref().unwrap_or("") + ); + eprintln!(" f: {}", compact_expr(f)); + eprintln!(" a: {}", compact_expr(a)); + eprintln!(" a_ty: {}", compact_expr_deep(&a_ty, depth)); + eprintln!(" dom: {}", compact_expr_deep(dom, depth)); + eprintln!(" a_ty data: {:?}", a_ty.data()); + eprintln!(" dom data: {:?}", dom.data()); + match &a_whnf { + Ok(w) => eprintln!(" a_ty whnf: {}", compact_expr_deep(w, depth)), + Err(e) => eprintln!(" a_ty whnf: ERR {e}"), + } + match &d_whnf { + Ok(w) => eprintln!(" dom whnf: {}", compact_expr_deep(w, depth)), + Err(e) => eprintln!(" dom whnf: ERR {e}"), + } + } + Err(TcError::AppTypeMismatch { + a_ty, + dom: dom.clone(), + depth: self.ctx.len(), + }) + } +} + +fn instantiate_pending( + intern: &mut crate::env::InternTable, + ty: &KExpr, + args: &[KExpr], +) -> KExpr { + match args { + [] => ty.clone(), + [arg] => subst(intern, ty, arg, 0), + _ => simul_subst(intern, ty, args, 0), + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/kernel/src/infer/application/prefix.rs b/crates/kernel/src/infer/application/prefix.rs new file mode 100644 index 000000000..f1cd35dd1 --- /dev/null +++ b/crates/kernel/src/infer/application/prefix.rs @@ -0,0 +1,98 @@ +//! Bounded admission history, NOT a cache of types or validity judgments. +//! +//! A repeated fingerprint nominates a prefix for ordinary inference and +//! materialization. Collisions can only nominate extra work (or forget a hot +//! prefix); actual results always use the exact, mode-separated infer caches. + +use std::hash::{Hash, Hasher}; + +use crate::env::{Addr, CtxAddr}; + +const INITIAL_SLOTS: usize = 64; +const MAX_SLOTS: usize = 4096; + +/// A lazily allocated, direct-mapped two-touch filter. It starts at 512 bytes; +/// collision-heavy checks grow up to 32 KiB. No expressions, +/// contexts, or type results are retained. Reset between checked members. +#[derive(Default)] +pub(crate) struct PrefixAdmission { + seen: Vec, + collisions: usize, +} + +impl PrefixAdmission { + pub(crate) fn observe( + &mut self, + key: (Addr, CtxAddr), + infer_only: bool, + ) -> bool { + let mut hasher = rustc_hash::FxHasher::default(); + (key, infer_only).hash(&mut hasher); + // Zero denotes an empty slot. The extremely rare remapping collision is + // harmless: this is only an allocation/work admission heuristic. + self.observe_fingerprint(hasher.finish().max(1)) + } + + fn slot(fingerprint: u64, len: usize) -> usize { + usize::try_from(fingerprint & (len as u64 - 1)) + .expect("masked to at most MAX_SLOTS - 1") + } + + fn observe_fingerprint(&mut self, fingerprint: u64) -> bool { + if self.seen.is_empty() { + self.seen.resize(INITIAL_SLOTS, 0); + } + let mut slot = Self::slot(fingerprint, self.seen.len()); + if self.seen[slot] == fingerprint { + return true; + } + if self.seen[slot] != 0 && self.seen.len() < MAX_SLOTS { + self.collisions += 1; + if self.collisions == self.seen.len() { + let next_len = self.seen.len() * 2; + let old = std::mem::replace(&mut self.seen, vec![0; next_len]); + for saved in old.into_iter().filter(|&f| f != 0) { + let i = Self::slot(saved, self.seen.len()); + self.seen[i] = saved; + } + self.collisions = 0; + slot = Self::slot(fingerprint, self.seen.len()); + } + } + self.seen[slot] = fingerprint; + false + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn repeated_keys_are_mode_and_context_specific() { + let mut history = PrefixAdmission::default(); + let a = (1, blake3::hash(b"a")); + let b = (1, blake3::hash(b"b")); + assert!(history.seen.is_empty()); + assert!(!history.observe(a, false)); + assert!(history.observe(a, false)); + assert!(!history.observe(a, true)); + assert!(history.observe(a, true)); + assert!(!history.observe(b, true)); + assert!(history.observe(b, true)); + } + + #[test] + fn collisions_forget_history_and_growth_is_bounded() { + let mut history = PrefixAdmission::default(); + for i in 1..100_000 { + // All fingerprints collide, even at the maximum size. + let f = i * MAX_SLOTS as u64; + assert!(!history.observe_fingerprint(f)); + assert!(history.observe_fingerprint(f)); + assert!(history.seen.len() <= MAX_SLOTS); + } + assert_eq!(history.seen.len(), MAX_SLOTS); + assert!(!history.observe_fingerprint(MAX_SLOTS as u64)); + } +} diff --git a/crates/kernel/src/infer/application/tests.rs b/crates/kernel/src/infer/application/tests.rs new file mode 100644 index 000000000..393cce16d --- /dev/null +++ b/crates/kernel/src/infer/application/tests.rs @@ -0,0 +1,724 @@ +//! Differential tests against sequential, one-App-at-a-time inference. + +use super::*; +use crate::env::{InternTable, KEnv}; +use crate::mode::{Anon, Meta}; +use crate::profile::take_op_counts; +use ix_common::address::Address; +use ix_common::env::{BinderInfo, DataValue, Name}; + +fn name(s: &str) -> M::MField { + M::meta_field(Name::str(Name::anon(), s.to_owned())) +} +fn var(i: u64) -> KExpr { + KExpr::var(i, name::("variable")) +} +fn sort(n: u64) -> KExpr { + let mut u = KUniv::zero(); + for _ in 0..n { + u = KUniv::succ(u); + } + KExpr::sort(u) +} +fn all(dom: KExpr, cod: KExpr) -> KExpr { + KExpr::all(name::("binder"), M::meta_field(BinderInfo::Implicit), dom, cod) +} +fn app(f: KExpr, a: KExpr) -> KExpr { + KExpr::app_mdata( + f, + a, + M::meta_field(vec![vec![( + Name::str(Name::anon(), "tag".to_owned()), + DataValue::OfString("application differential".to_owned()), + )]]), + ) +} +fn axiom(env: &mut KEnv, s: &str, ty: KExpr) -> KExpr { + let id = KId::new(Address::hash(s.as_bytes()), name::(s)); + env.insert( + id.clone(), + KConst::Axio { + name: name::(s), + level_params: M::meta_field(vec![]), + is_unsafe: false, + lvls: 0, + ty, + }, + ); + KExpr::cnst(id, Box::new([])) +} + +// Independent copy of the former recursive App rule. Other expression +// constructors use the normal checker, so this isolates the changed rule. +fn reference( + tc: &mut TypeChecker<'_, M>, + e: &KExpr, +) -> Result, TcError> { + let key = tc.infer_key(e); + if let Some(ty) = tc.env.infer_cache.get(&key) { + return Ok(ty.clone()); + } + if tc.infer_only + && let Some(ty) = tc.env.infer_only_cache.get(&key) + { + return Ok(ty.clone()); + } + let ExprData::App(f, a, _) = e.data() else { return tc.infer(e) }; + let f_ty = reference(tc, f)?; + let (dom, cod) = tc.ensure_forall(&f_ty)?; + if !tc.infer_only { + let a_ty = reference(tc, a)?; + let eager = tc.eager_reduce; + tc.eager_reduce |= tc.is_eager_reduce(a); + let eq = tc.is_def_eq(&a_ty, &dom); + tc.eager_reduce = eager; + if !eq? { + return Err(TcError::AppTypeMismatch { a_ty, dom, depth: tc.ctx.len() }); + } + } + let ty = subst(&mut tc.env.intern, &cod, a, 0); + if tc.infer_only { + tc.env.infer_only_cache.insert(key, ty.clone()); + } else { + tc.env.infer_cache.insert(key, ty.clone()); + } + Ok(ty) +} + +fn same_type(a: KExpr, b: KExpr) { + let mut pending = vec![(&a, &b)]; + let mut seen = rustc_hash::FxHashSet::default(); + while let Some((a, b)) = pending.pop() { + if !seen.insert((*a.addr(), *b.addr())) { + continue; + } + assert_eq!(a.mdata(), b.mdata()); + assert_eq!(a.univ_decor(), b.univ_decor()); + assert_eq!(a.lbr(), b.lbr()); + match (a.data(), b.data()) { + (ExprData::App(f, x, _), ExprData::App(g, y, _)) => { + pending.extend([(f, g), (x, y)]) + }, + (ExprData::All(n, i, d, c, _), ExprData::All(m, j, e, r, _)) + | (ExprData::Lam(n, i, d, c, _), ExprData::Lam(m, j, e, r, _)) => { + assert_eq!(n, m); + assert_eq!(i, j); + pending.extend([(d, e), (c, r)]); + }, + (ExprData::Var(i, n, _), ExprData::Var(j, m, _)) => { + assert_eq!(i, j); + assert_eq!(n, m); + }, + (ExprData::Let(n, t, v, r, nd, _), ExprData::Let(m, u, w, s, md, _)) => { + assert_eq!(n, m); + assert_eq!(nd, md); + pending.extend([(t, u), (v, w), (r, s)]); + }, + (ExprData::Prj(_, _, v, _), ExprData::Prj(_, _, w, _)) => { + pending.push((v, w)) + }, + _ => {}, + } + } + let mut intern = InternTable::new(); + let a = intern.intern_expr(a); + let b = intern.intern_expr(b); + assert!(a.ptr_eq(&b), "different inferred types: {a:?}\n{b:?}"); +} + +fn fixture(env: &mut KEnv) -> (KExpr, Vec>) { + let a = axiom(env, "A", sort(1)); + let b = axiom(env, "B", all(a.clone(), sort(1))); + let x = axiom(env, "x", a.clone()); + let y = axiom(env, "y", app(b.clone(), x.clone())); + // f : (A : Type) -> (B : A -> Type) -> (x : A) -> B x -> B x + let ty = all( + sort(1), + all( + all(var(0), sort(1)), + all(var(1), all(app(var(1), var(0)), app(var(2), var(1)))), + ), + ); + (axiom(env, "f", ty), vec![a, b, x, y]) +} + +fn dependent() { + for infer_only in [false, true] { + for count in 1..=4 { + let mut env_a = KEnv::::new(); + let mut env_b = KEnv::::new(); + let (mut a, args_a) = fixture(&mut env_a); + let (mut b, args_b) = fixture(&mut env_b); + for i in 0..count { + a = app(a, args_a[i].clone()); + b = app(b, args_b[i].clone()); + } + let mut tc_a = TypeChecker::new(&mut env_a); + let mut tc_b = TypeChecker::new(&mut env_b); + tc_a.infer_only = infer_only; + tc_b.infer_only = infer_only; + same_type(reference(&mut tc_a, &a).unwrap(), tc_b.infer(&b).unwrap()); + assert!(tc_b.lctx.is_empty() && tc_b.ctx.is_empty()); + } + } +} + +#[test] +fn dependent_partial_and_full_applications() { + dependent::(); + dependent::(); +} + +fn boundaries() { + for infer_only in [false, true] { + for hidden in [false, true] { + let mut a = KEnv::new(); + let mut b = KEnv::new(); + let mk = |env: &mut KEnv| { + // An applied type variable, or a let, exposes a new Pi only AFTER + // substituting the first argument into the function's codomain. + let result = if hidden { + KExpr::let_(name::("T"), sort(1), var(0), var(0), false) + } else { + var(0) + }; + let f = axiom(env, "dependentFunction", all(sort(1), result)); + app(app(f, all(sort(0), sort(0))), axiom(env, "P", sort(0))) + }; + let ea = mk(&mut a); + let eb = mk(&mut b); + let mut a = TypeChecker::new(&mut a); + let mut b = TypeChecker::new(&mut b); + a.infer_only = infer_only; + b.infer_only = infer_only; + same_type(reference(&mut a, &ea).unwrap(), b.infer(&eb).unwrap()); + } + } +} + +#[test] +fn flushes_pending_arguments_before_revealing_hidden_pi() { + boundaries::(); + boundaries::(); +} + +fn open_args() { + for infer_only in [false, true] { + let mut a = KEnv::new(); + let mut b = KEnv::new(); + let mk = |env: &mut KEnv| { + // f : (A : Type) -> A -> ((z : A) -> A). Return a function so the + // open replacement A must be lifted underneath the residual binder. + let f = axiom( + env, + "openFunction", + all(sort(1), all(var(0), all(var(1), var(2)))), + ); + app(app(f, var(1)), var(0)) + }; + let ea = mk(&mut a); + let eb = mk(&mut b); + let mut a = TypeChecker::new(&mut a); + let mut b = TypeChecker::new(&mut b); + a.infer_only = infer_only; + b.infer_only = infer_only; + a.push_local(sort(1)); + a.push_local(var(0)); + b.push_local(sort(1)); + b.push_local(var(0)); + same_type(reference(&mut a, &ea).unwrap(), b.infer(&eb).unwrap()); + assert_eq!(b.ctx.len(), 2); + } +} + +#[test] +fn lifts_ambient_arguments_under_residual_binders() { + open_args::(); + open_args::(); +} + +#[test] +fn checks_every_argument_and_keeps_infer_only_cache_separate() { + for bad_index in 0..4 { + let mut env = KEnv::::new(); + let (mut f, mut args) = fixture(&mut env); + args[bad_index] = sort(4); + for a in args { + f = app(f, a); + } + let mut tc = TypeChecker::new(&mut env); + let _ = tc.with_infer_only(|tc| tc.infer(&f)); + assert!(tc.env.infer_cache.is_empty()); + let key = tc.infer_key(&f); + assert!(matches!(tc.infer(&f), Err(TcError::AppTypeMismatch { .. }))); + assert!(!tc.env.infer_cache.contains_key(&key)); + } +} + +#[test] +fn overapplication_and_unknown_heads_are_rejected() { + let mut env = KEnv::::new(); + let (mut f, args) = fixture(&mut env); + for a in args { + f = app(f, a); + } + let f = app(f, sort(0)); + let mut tc = TypeChecker::new(&mut env); + assert!(matches!(tc.infer(&f), Err(TcError::FunExpected { .. }))); + let missing = + KExpr::cnst(KId::new(Address::hash(b"absent"), ()), Box::new([])); + assert!(matches!( + tc.infer(&app(app(missing, sort(0)), sort(0))), + Err(TcError::UnknownConst(_)) + )); +} + +#[test] +fn batched_heads_still_validate_universe_arity_and_scope() { + let mut env = KEnv::::new(); + let id = KId::new(Address::hash(b"universeFunction"), ()); + env.insert( + id.clone(), + KConst::Axio { + name: (), + level_params: (), + is_unsafe: false, + lvls: 1, + ty: all(KExpr::sort(KUniv::param(0, ())), all(var(0), var(1))), + }, + ); + let mut tc = TypeChecker::new(&mut env); + let wrong_arity = + app(app(KExpr::cnst(id.clone(), Box::new([])), sort(0)), sort(0)); + for infer_only in [false, true] { + tc.infer_only = infer_only; + assert!(matches!( + tc.infer(&wrong_arity), + Err(TcError::UnivParamMismatch { .. }) + )); + } + tc.env.insert( + id.clone(), + KConst::Axio { + name: (), + level_params: (), + is_unsafe: false, + lvls: 1, + ty: all(KExpr::sort(KUniv::param(1, ())), all(var(0), var(1))), + }, + ); + let bad_scope = + app(app(KExpr::cnst(id, Box::new([KUniv::zero()])), sort(0)), sort(0)); + for infer_only in [false, true] { + tc.infer_only = infer_only; + assert!(matches!( + tc.infer(&bad_scope), + Err(TcError::UnivParamOutOfRange { .. }) + )); + } +} + +#[test] +fn application_cache_does_not_cross_local_let_contexts() { + let mut env = KEnv::::new(); + let a = axiom(&mut env, "A", sort(1)); + let b = axiom(&mut env, "B", sort(1)); + let x = axiom(&mut env, "x", a.clone()); + let f = axiom(&mut env, "id", all(sort(1), all(var(0), var(1)))); + let e = app(app(f, var(1)), var(0)); + let mut tc = TypeChecker::new(&mut env); + tc.push_let(sort(1), a.clone()); + tc.push_let(a.clone(), x.clone()); + let key = tc.infer_key(&e); + tc.infer(&e).unwrap(); + tc.pop_local(); + tc.pop_local(); + tc.push_let(sort(1), b); + tc.push_let(a, x); + assert_ne!(key, tc.infer_key(&e)); + assert!(matches!(tc.infer(&e), Err(TcError::AppTypeMismatch { .. }))); +} + +fn fvar_arguments() { + let mut env = KEnv::::new(); + let f = axiom(&mut env, "id", all(sort(1), all(var(0), all(var(1), var(2))))); + let mut tc = TypeChecker::new(&mut env); + tc.with_lctx_scope(|tc| { + let aid = tc.fresh_fvar_id(); + let a = tc.intern(KExpr::fvar(aid, name::("A"))); + tc.lctx.push( + aid, + LocalDecl::CDecl { + name: name::("A"), + bi: M::meta_field(BinderInfo::Default), + ty: sort(1), + }, + ); + let xid = tc.fresh_fvar_id(); + let x = tc.intern(KExpr::fvar(xid, name::("x"))); + tc.lctx.push( + xid, + LocalDecl::CDecl { + name: name::("x"), + bi: M::meta_field(BinderInfo::Default), + ty: a.clone(), + }, + ); + let e = app(app(f, a), x); + for infer_only in [false, true] { + tc.infer_only = infer_only; + tc.env.clear_reduction_caches(); + let expected = reference(tc, &e)?; + tc.env.clear_reduction_caches(); + tc.prefix_admission = Default::default(); + let root = tc.infer_key(&e); + let ExprData::App(prefix, _, _) = e.data() else { unreachable!() }; + let key = tc.infer_key(prefix); + for _ in 0..2 { + tc.env.infer_cache.remove(&root); + tc.env.infer_only_cache.remove(&root); + tc.prefix_admission.observe(key, infer_only); + same_type(expected.clone(), tc.infer(&e)?); + } + assert!( + tc.env.infer_cache.contains_key(&key) + || tc.env.infer_only_cache.contains_key(&key) + ); + } + Ok::<(), TcError>(()) + }) + .unwrap(); + assert!(tc.lctx.is_empty()); +} + +#[test] +fn fvar_arguments_match_sequential_substitution_in_both_modes() { + fvar_arguments::(); + fvar_arguments::(); +} + +#[test] +fn honors_cached_dependent_prefixes() { + let mut env = KEnv::::new(); + let (f, args) = fixture(&mut env); + let prefix = app(app(f, args[0].clone()), args[1].clone()); + let mut tc = TypeChecker::new(&mut env); + let ty = tc.infer(&prefix).unwrap(); + let key = tc.infer_key(&prefix); + assert!(tc.env.infer_cache.get(&key).unwrap().ptr_eq(&ty)); + let tail = app(app(prefix, args[2].clone()), args[3].clone()); + // A valid cached prefix should be enough; no inference of the head is + // needed. Keep the declaration environment intact, remove all other + // synthesis results, then verify that the prefix remains reusable. + tc.env.infer_cache.retain(|k, _| k == &key); + let actual = tc.infer(&tail).unwrap(); + same_type(actual, app(args[1].clone(), args[2].clone())); + assert!(tc.env.infer_cache.get(&key).unwrap().ptr_eq(&ty)); +} + +#[test] +fn eager_scope_is_restored_on_errors_and_success() { + let mut env = KEnv::::new(); + let a = axiom(&mut env, "someValue", sort(0)); + let mut tc = TypeChecker::new(&mut env); + for saved in [false, true] { + tc.eager_reduce = saved; + tc.check_app_argument(&a, &a, &sort(0)).unwrap(); + assert_eq!(tc.eager_reduce, saved); + assert!(tc.check_app_argument(&a, &a, &sort(3)).is_err()); + assert_eq!(tc.eager_reduce, saved); + tc.rec_fuel = 0; + assert!(tc.check_app_argument(&a, &a, &sort(5)).is_err()); + assert_eq!(tc.eager_reduce, saved); + tc.rec_fuel = crate::tc::max_rec_fuel(); + } +} + +fn long_application(env: &mut KEnv, count: u64) -> KExpr { + let mut pack_ty = sort(1); + for _ in 0..count { + pack_ty = all(sort(1), pack_ty); + } + let mut ty = axiom(env, "Pack", pack_ty); + // Every argument occurs in the result, so sequential inference revisits + // a growingly-instantiated Pack application at every telescope step. + for i in (0..count).rev() { + ty = app(ty, var(i)); + } + for _ in 0..count { + ty = all(sort(1), ty); + } + let mut f = axiom(env, "longFunction", ty); + for _ in 0..count { + f = app(f, sort(0)); + } + f +} + +#[test] +fn long_telescope_avoids_quadratic_codomain_construction() { + let mut a = KEnv::new(); + let mut b = KEnv::new(); + let ea = long_application(&mut a, 96); + let eb = long_application(&mut b, 96); + let mut a = TypeChecker::new(&mut a); + let mut b = TypeChecker::new(&mut b); + take_op_counts(); + let expected = reference(&mut a, &ea).unwrap(); + let old = take_op_counts(); + let actual = b.infer(&eb).unwrap(); + let new = take_op_counts(); + eprintln!("application work: sequential={old:?}, batched={new:?}"); + assert!(new.intern_nodes * 4 < old.intern_nodes); + same_type(actual, expected); +} + +fn hot_prefixes() { + for infer_only in [false, true] { + let mut env = KEnv::::new(); + let (f, args) = fixture(&mut env); + let result = app(args[1].clone(), args[2].clone()); + let y2 = axiom(&mut env, "anotherY", result.clone()); + let y3 = axiom(&mut env, "thirdY", result.clone()); + let prefix = + app(app(app(f, args[0].clone()), args[1].clone()), args[2].clone()); + let first = app(prefix.clone(), args[3].clone()); + let mut reference_env = KEnv::::new(); + for (id, declaration) in env.iter() { + reference_env.insert(id, declaration); + } + let mut reference_tc = TypeChecker::new(&mut reference_env); + reference_tc.infer_only = infer_only; + let expected = reference(&mut reference_tc, &first).unwrap(); + let expected_prefix = reference(&mut reference_tc, &prefix).unwrap(); + let mut tc = TypeChecker::new(&mut env); + tc.infer_only = infer_only; + let key = tc.infer_key(&prefix); + same_type(tc.infer(&first).unwrap(), expected.clone()); + assert!(!tc.env.infer_cache.contains_key(&key)); + assert!(!tc.env.infer_only_cache.contains_key(&key)); + // The bounded filter is allowed to forget intervening collisions. Seed + // this nomination immediately so the cache-contract test is UID-order + // independent, including under parallel unit-test scheduling. + tc.prefix_admission.observe(key, infer_only); + same_type(tc.infer(&app(prefix.clone(), y2)).unwrap(), expected.clone()); + let cache = + if infer_only { &tc.env.infer_only_cache } else { &tc.env.infer_cache }; + same_type(cache[&key].clone(), expected_prefix); + // A third use needs no synthesis of earlier prefixes or the head. + tc.env.infer_cache.retain(|k, _| k == &key); + tc.env.infer_only_cache.retain(|k, _| k == &key); + same_type(tc.infer(&app(prefix, y3)).unwrap(), expected); + if infer_only { + assert!(tc.env.infer_cache.is_empty()); + } + } +} + +#[test] +fn repeated_dependent_prefixes_are_published_in_the_correct_mode() { + hot_prefixes::(); + hot_prefixes::(); +} + +fn invalid_hot_prefixes() { + for bad_index in 0..4 { + let mut env = KEnv::::new(); + let (mut f, mut args) = fixture(&mut env); + args[bad_index] = sort(4); + let mut prefixes = vec![]; + for arg in args { + f = app(f, arg); + prefixes.push(f.clone()); + } + let mut tc = TypeChecker::new(&mut env); + // Warm unchecked results/admission history first. They must never grant + // full-mode validity, even when the same UID is seen repeatedly. + for _ in 0..3 { + let root = tc.infer_key(&f); + tc.env.infer_only_cache.remove(&root); + let _ = tc.with_infer_only(|tc| tc.infer(&f)); + } + for _ in 0..3 { + assert!(matches!(tc.infer(&f), Err(TcError::AppTypeMismatch { .. }))); + for invalid in &prefixes[bad_index..] { + let key = tc.infer_key(invalid); + assert!(!tc.env.infer_cache.contains_key(&key)); + } + } + } +} + +#[test] +fn repeated_invalid_prefixes_and_infer_only_results_never_validate_arguments() { + invalid_hot_prefixes::(); + invalid_hot_prefixes::(); +} + +fn warm_open_prefix() { + for infer_only in [false, true] { + let mut env = KEnv::::new(); + let a = axiom(&mut env, "A", sort(1)); + let b = axiom(&mut env, "B", sort(1)); + let x = axiom(&mut env, "x", a.clone()); + let f = + axiom(&mut env, "openId", all(sort(1), all(var(0), all(var(1), var(2))))); + let prefix = app(f, var(1)); + let e = app(prefix.clone(), var(0)); + let mut tc = TypeChecker::new(&mut env); + tc.infer_only = infer_only; + tc.push_let(sort(1), a.clone()); + tc.push_let(a.clone(), x.clone()); + let root = tc.infer_key(&e); + let key_a = tc.infer_key(&prefix); + let expected = reference(&mut tc, &e).unwrap(); + let expected_prefix = reference(&mut tc, &prefix).unwrap(); + tc.env.clear_reduction_caches(); + for _ in 0..2 { + tc.env.infer_cache.remove(&root); + tc.env.infer_only_cache.remove(&root); + tc.prefix_admission.observe(key_a, infer_only); + same_type(tc.infer(&e).unwrap(), expected.clone()); + } + // The cached residual binder contains correctly lifted ambient Vars. + let cache = + if infer_only { &tc.env.infer_only_cache } else { &tc.env.infer_cache }; + same_type(cache[&key_a].clone(), expected_prefix); + tc.pop_local(); + tc.pop_local(); + tc.push_let(sort(1), b); + tc.push_let(a, x); + let key_b = tc.infer_key(&prefix); + assert_ne!(key_a, key_b); + assert!(!tc.env.infer_cache.contains_key(&key_b)); + assert!(!tc.env.infer_only_cache.contains_key(&key_b)); + if !infer_only { + assert!(matches!(tc.infer(&e), Err(TcError::AppTypeMismatch { .. }))); + } + } +} + +#[test] +fn warmed_dependent_prefixes_lift_ambient_vars_and_respect_let_contexts() { + warm_open_prefix::(); + warm_open_prefix::(); +} + +#[test] +fn hot_long_telescope_materializes_at_most_one_dependent_suffix() { + let mut env = KEnv::new(); + let e = long_application(&mut env, 96); + let mut tc = TypeChecker::new(&mut env); + // Warm every prefix's admission history WITHOUT precomputing types. This + // is the adversarial case for accidentally restoring quadratic batching. + let mut prefixes = vec![]; + let ExprData::App(p, _, _) = e.data() else { unreachable!() }; + let mut p = p; + while let ExprData::App(f, _, _) = p.data() { + let key = tc.infer_key(p); + tc.prefix_admission.observe(key, false); + prefixes.push(p); + p = f; + } + let longest = tc.infer_key(prefixes[0]); + tc.prefix_admission.observe(longest, false); + take_op_counts(); + let expected = tc.infer(&e).unwrap(); + let work = take_op_counts(); + let published = prefixes + .iter() + .filter(|p| { + let key = tc.infer_key(p); + tc.env.infer_cache.contains_key(&key) + }) + .count(); + assert_eq!(published, 1); + assert!(work.intern_nodes < 2000, "suffixes must stay batched: {work:?}"); + tc.env.clear_reduction_caches(); + tc.prefix_admission = Default::default(); + same_type(tc.infer(&e).unwrap(), expected); +} + +#[test] +fn reset_forgets_admission_and_full_prefixes_remain_usable_by_infer_only() { + let mut env = KEnv::::new(); + let (f, args) = fixture(&mut env); + let prefix = + app(app(app(f, args[0].clone()), args[1].clone()), args[2].clone()); + let e = app(prefix.clone(), args[3].clone()); + let mut tc = TypeChecker::new(&mut env); + let key = tc.infer_key(&prefix); + assert!(!tc.prefix_admission.observe(key, false)); + tc.reset(); + assert!(!tc.prefix_admission.observe(key, false)); + let expected = tc.infer(&e).unwrap(); + assert!(tc.env.infer_cache.contains_key(&key)); + let root = tc.infer_key(&e); + tc.env.infer_cache.remove(&root); + tc.env.infer_only_cache.clear(); + tc.env.infer_cache.retain(|k, _| k == &key); + same_type(tc.with_infer_only(|tc| tc.infer(&e)).unwrap(), expected); + assert!(tc.env.infer_only_cache.contains_key(&root)); + assert!(!tc.env.infer_only_cache.contains_key(&key)); + tc.env.clear_reduction_caches(); + assert!(!tc.env.infer_cache.contains_key(&key)); + assert!(!tc.env.infer_only_cache.contains_key(&key)); +} + +#[test] +fn warm_hidden_pi_boundaries_still_flush_pending_arguments() { + for infer_only in [false, true] { + let mut env = KEnv::::new(); + let ty = all(sort(1), var(0)); + let f = axiom(&mut env, "hiddenPrefix", ty); + let p = axiom(&mut env, "P", sort(0)); + let h = app(f, all(sort(0), all(sort(0), sort(0)))); + let e = app(app(h, p.clone()), p); + let mut tc = TypeChecker::new(&mut env); + tc.infer_only = infer_only; + let expected = reference(&mut tc, &e).unwrap(); + tc.env.clear_reduction_caches(); + let root = tc.infer_key(&e); + for _ in 0..3 { + tc.env.infer_cache.remove(&root); + tc.env.infer_only_cache.remove(&root); + same_type(tc.infer(&e).unwrap(), expected.clone()); + } + } +} + +#[test] +fn hot_prefix_reuse_reduces_checked_argument_work() { + let run = |admit: bool| { + let mut env = KEnv::::new(); + let application = long_application(&mut env, 32); + let ExprData::App(prefix, _, _) = application.data() else { + unreachable!() + }; + let tails: Vec<_> = (0..32) + .map(|i| { + let arg = axiom(&mut env, &format!("tailType{i}"), sort(1)); + app(prefix.clone(), arg) + }) + .collect(); + let mut tc = TypeChecker::new(&mut env); + let key = tc.infer_key(prefix); + take_op_counts(); + for tail in tails { + if !admit { + // A cold two-touch filter never nominates a prefix, reproducing the + // previous batched result-cache policy without a production toggle. + tc.prefix_admission = Default::default(); + } else { + tc.prefix_admission.observe(key, false); + } + tc.infer(&tail).unwrap(); + } + take_op_counts() + }; + let cold = run(false); + let hot = run(true); + eprintln!("reused prefix work: cold={cold:?}, hot={hot:?}"); + assert!(hot.def_eq_calls * 4 < cold.def_eq_calls); + assert!(hot.intern_nodes * 2 < cold.intern_nodes); +} diff --git a/crates/kernel/src/infer/binders.rs b/crates/kernel/src/infer/binders.rs new file mode 100644 index 000000000..0b591c350 --- /dev/null +++ b/crates/kernel/src/infer/binders.rs @@ -0,0 +1,141 @@ +//! Batched opening of consecutive lambdas/foralls during inference. +//! +//! Walk the original telescope, opening each domain under its own prefix of +//! fresh locals, then instantiate the terminal body once. Substitution memos +//! remain call-local: different prefixes MUST NOT share logical entries. +//! The outer `infer` call still owns cache lookup/publication; domains and the +//! terminal body use ordinary, mode-separated cached inference. We omit +//! partially opened suffix entries rather than constructing them just to +//! cache them. An unchanged closed suffix can still use an existing result. + +use super::*; + +impl TypeChecker<'_, M> { + /// A suffix with no loose bvars is unaffected by the accumulated opening. + /// Probe only that case: building other suffixes to probe their cache would + /// reintroduce the repeated DAG traversals this optimization removes. + fn has_closed_infer_result(&mut self, e: &KExpr) -> bool { + if e.lbr() != 0 { + return false; + } + let key = self.infer_key(e); + self.env.infer_cache.contains_key(&key) + || (self.infer_only && self.env.infer_only_cache.contains_key(&key)) + } + + pub(super) fn infer_lambda_telescope( + &mut self, + e: &KExpr, + ) -> Result, TcError> { + self.with_lctx_scope(|tc| { + let mut fvars = Vec::new(); + let mut ids = Vec::new(); + let mut domains = Vec::new(); + let mut body = e; + while let ExprData::Lam(name, bi, ty, rest, _) = body.data() { + if !fvars.is_empty() && tc.has_closed_infer_result(body) { + break; + } + // Check in dependency order, before introducing this binder. In + // infer-only mode, preserve the existing skipped-domain validation. + let domain = instantiate_rev(&mut tc.env.intern, ty, &fvars); + if !tc.infer_only { + let domain_ty = tc.infer(&domain)?; + tc.ensure_sort(&domain_ty)?; + } + let id = tc.fresh_fvar_id(); + let fv = tc.intern(KExpr::fvar(id, name.clone())); + tc.lctx.push( + id, + LocalDecl::CDecl { + name: name.clone(), + bi: bi.clone(), + ty: domain.clone(), + }, + ); + fvars.push(fv); + ids.push(id); + domains.push(domain); + body = rest; + } + + let opened = instantiate_rev(&mut tc.env.intern, body, &fvars); + let body_ty = tc.infer(&opened)?; + // In the recursive implementation only the innermost call can see a + // head beta redex; every outer call sees the newly constructed All. + let body_ty = cheap_beta_reduce(&mut tc.env.intern, &body_ty); + let mut result = abstract_fvars(&mut tc.env.intern, &body_ty, &ids); + for (i, domain) in domains.iter().enumerate().rev() { + // Close a domain over ONLY its earlier binders. Do not reuse the + // original raw domain: opening/closing canonicalizes variable names + // and affected metadata just as the single-binder implementation does. + let domain = abstract_fvars(&mut tc.env.intern, domain, &ids[..i]); + // Inferred Pis deliberately use anonymous/default binder metadata, + // matching the single-binder path and recursor synthesis exactly. + result = tc.env.intern.intern_all( + M::meta_field(ix_common::env::Name::anon()), + M::meta_field(ix_common::env::BinderInfo::Default), + &domain, + &result, + ); + } + Ok(result) + }) + } + + pub(super) fn infer_forall_telescope( + &mut self, + e: &KExpr, + ) -> Result, TcError> { + self.with_lctx_scope(|tc| { + let mut fvars = Vec::new(); + let mut levels = Vec::new(); + let mut body = e; + while let ExprData::All(name, bi, ty, rest, _) = body.data() { + if !fvars.is_empty() && tc.has_closed_infer_result(body) { + break; + } + let domain = instantiate_rev(&mut tc.env.intern, ty, &fvars); + // Foralls validate domain sorts even in infer-only mode. + let domain_ty = tc.infer(&domain)?; + levels.push(tc.ensure_sort(&domain_ty)?); + let id = tc.fresh_fvar_id(); + let fv = tc.intern(KExpr::fvar(id, name.clone())); + if crate::env_var("IX_FVAR_TRACE").is_ok() { + log::info!( + "[fvar All batch push] fv={id} ty.addr={:?} ty.lbr={} ctx_len_before_push={} batch_prefix={}", + domain.addr(), + domain.lbr(), + tc.ctx.len(), + fvars.len(), + ); + log::info!(" ty data: {:?}", domain.data()); + } + tc.lctx.push( + id, + LocalDecl::CDecl { + name: name.clone(), + bi: bi.clone(), + ty: domain, + }, + ); + fvars.push(fv); + body = rest; + } + + let opened = instantiate_rev(&mut tc.env.intern, body, &fvars); + let mut result = tc.infer(&opened)?; + for domain_level in levels.into_iter().rev() { + let body_level = tc.ensure_sort(&result)?; + // Preserve the right-associated imax and normalize each intermediate + // sort as before. Prop codomains and symbolic universes need imax, + // not a max over all the domains. + result = tc.intern(KExpr::sort(KUniv::imax(domain_level, body_level))); + } + Ok(result) + }) + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/kernel/src/infer/binders/tests.rs b/crates/kernel/src/infer/binders/tests.rs new file mode 100644 index 000000000..51aebb265 --- /dev/null +++ b/crates/kernel/src/infer/binders/tests.rs @@ -0,0 +1,508 @@ +//! Differential tests against the former one-binder-at-a-time inference. + +use super::*; +use crate::env::{InternTable, KEnv}; +use crate::expr::FVarId; +use crate::mode::{Anon, Meta}; +use crate::profile::{OpCounts, take_op_counts}; +use ix_common::address::Address; +use ix_common::env::{BinderInfo, DataValue, Name}; + +fn name(s: &str) -> M::MField { + M::meta_field(Name::str(Name::anon(), s.to_owned())) +} + +fn var(i: u64) -> KExpr { + KExpr::var(i, name::("source-variable")) +} + +fn sort(n: u64) -> KExpr { + let mut u = KUniv::zero(); + for _ in 0..n { + u = KUniv::succ(u); + } + KExpr::sort(u) +} + +fn lam(ty: KExpr, body: KExpr) -> KExpr { + KExpr::lam(name::("lambda"), M::meta_field(BinderInfo::Implicit), ty, body) +} + +fn all(ty: KExpr, body: KExpr) -> KExpr { + KExpr::all( + name::("forall"), + M::meta_field(BinderInfo::InstImplicit), + ty, + body, + ) +} + +fn app(f: KExpr, a: KExpr) -> KExpr { + KExpr::app_mdata( + f, + a, + M::meta_field(vec![vec![( + Name::str(Name::anon(), "tag".to_owned()), + DataValue::OfString("binder differential".to_owned()), + )]]), + ) +} + +// A test-only copy of the former Lam/All branches, including cache behavior, +// dependent domain validation, one-at-a-time opening/closing, and scope +// restoration. Non-binder operations use the normal checker. This is an +// oracle for telescope handling, not a second implementation of the kernel. +fn reference( + tc: &mut TypeChecker<'_, M>, + e: &KExpr, +) -> Result, TcError> { + let key = tc.infer_key(e); + if let Some(ty) = tc.env.infer_cache.get(&key) { + return Ok(ty.clone()); + } + if tc.infer_only + && let Some(ty) = tc.env.infer_only_cache.get(&key) + { + return Ok(ty.clone()); + } + let result = match e.data() { + ExprData::Lam(name, bi, ty, body, _) => { + if !tc.infer_only { + let domain_ty = reference(tc, ty)?; + tc.ensure_sort(&domain_ty)?; + } + tc.with_lctx_scope(|tc| { + let id = tc.fresh_fvar_id(); + let fv = tc.intern(KExpr::fvar(id, name.clone())); + tc.lctx.push( + id, + LocalDecl::CDecl { + name: name.clone(), + bi: bi.clone(), + ty: ty.clone(), + }, + ); + let opened = instantiate_rev(&mut tc.env.intern, body, &[fv]); + let body_ty = reference(tc, &opened)?; + let body_ty = cheap_beta_reduce(&mut tc.env.intern, &body_ty); + let closed = abstract_fvars(&mut tc.env.intern, &body_ty, &[id]); + Ok(tc.intern(KExpr::all( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + ty.clone(), + closed, + ))) + })? + }, + ExprData::All(name, bi, ty, body, _) => { + let domain_ty = reference(tc, ty)?; + let u1 = tc.ensure_sort(&domain_ty)?; + tc.with_lctx_scope(|tc| { + let id = tc.fresh_fvar_id(); + let fv = tc.intern(KExpr::fvar(id, name.clone())); + tc.lctx.push( + id, + LocalDecl::CDecl { + name: name.clone(), + bi: bi.clone(), + ty: ty.clone(), + }, + ); + let opened = instantiate_rev(&mut tc.env.intern, body, &[fv]); + let body_ty = reference(tc, &opened)?; + let u2 = tc.ensure_sort(&body_ty)?; + Ok(tc.intern(KExpr::sort(KUniv::imax(u1, u2)))) + })? + }, + _ => return tc.infer(e), + }; + if tc.infer_only { + tc.env.infer_only_cache.insert(key, result.clone()); + } else { + tc.env.infer_cache.insert(key, result.clone()); + } + Ok(result) +} + +fn same_shape(a: KExpr, b: KExpr) { + // Check occurrence metadata BEFORE interning together: first-insert-wins + // interning intentionally ignores names, binder info, and mdata. + let mut pending = vec![(&a, &b)]; + let mut seen = rustc_hash::FxHashSet::default(); + while let Some((a, b)) = pending.pop() { + if !seen.insert((*a.addr(), *b.addr())) { + continue; + } + assert_eq!(a.mdata(), b.mdata(), "expression metadata differs"); + assert_eq!(a.univ_decor(), b.univ_decor(), "universe spelling differs"); + assert_eq!(a.lbr(), b.lbr()); + assert_eq!(a.count_0(), b.count_0()); + assert_eq!(a.has_fvars(), b.has_fvars()); + match (a.data(), b.data()) { + (ExprData::Var(i, n, _), ExprData::Var(j, m, _)) => { + assert_eq!(i, j); + assert_eq!(n, m); + }, + (ExprData::FVar(i, n, _), ExprData::FVar(j, m, _)) => { + assert_eq!(i, j); + assert_eq!(n, m); + }, + (ExprData::App(f, x, _), ExprData::App(g, y, _)) => { + pending.extend([(f, g), (x, y)]); + }, + (ExprData::Lam(n, bi, t, r, _), ExprData::Lam(m, bj, u, s, _)) + | (ExprData::All(n, bi, t, r, _), ExprData::All(m, bj, u, s, _)) => { + assert_eq!(n, m); + assert_eq!(bi, bj); + pending.extend([(t, u), (r, s)]); + }, + (ExprData::Let(n, t, v, r, nd, _), ExprData::Let(m, u, w, s, md, _)) => { + assert_eq!(n, m); + assert_eq!(nd, md); + pending.extend([(t, u), (v, w), (r, s)]); + }, + (ExprData::Prj(i, f, v, _), ExprData::Prj(j, g, w, _)) => { + assert_eq!(i, j); + assert_eq!(f, g); + pending.push((v, w)); + }, + (ExprData::Sort(..), ExprData::Sort(..)) + | (ExprData::Const(..), ExprData::Const(..)) + | (ExprData::Nat(..), ExprData::Nat(..)) + | (ExprData::Str(..), ExprData::Str(..)) => {}, + _ => panic!("different expression constructors"), + } + } + // The common interner additionally checks semantic payloads and complete + // universe structure/spelling (without depending on separate-env UIDs). + let mut intern = InternTable::new(); + let a = intern.intern_expr(a); + let b = intern.intern_expr(b); + assert!(a.ptr_eq(&b), "different inferred shapes:\n{a:?}\n{b:?}"); +} + +fn differential(e: &KExpr) { + for infer_only in [false, true] { + let mut baseline = KEnv::new(); + let mut batched = KEnv::new(); + let mut a = TypeChecker::new(&mut baseline); + let mut b = TypeChecker::new(&mut batched); + a.infer_only = infer_only; + b.infer_only = infer_only; + let ea = a.intern(e.clone()); + let eb = b.intern(e.clone()); + let expected = reference(&mut a, &ea).unwrap(); + let actual = b.infer(&eb).unwrap(); + assert!(a.lctx.is_empty() && b.lctx.is_empty()); + assert!(!actual.has_fvars(), "batch-local FVars escaped inference"); + same_shape(actual, expected); + } +} + +fn dependent_examples() { + // (A : Type) (B : A -> Type) (x : A) (y : B x), with deliberately + // nonempty binder/Var/App metadata. In particular the returned dependent + // domains must have the same metadata normalization as sequential opening. + let domains = [sort(1), all(var(0), sort(1)), var(1), app(var(1), var(0))]; + let mut lambda = var::(0); + let mut forall = app::(var(2), var(1)); + for domain in domains.into_iter().rev() { + lambda = lam(domain.clone(), lambda); + forall = all(domain, forall); + } + differential(&lambda); + differential(&forall); + + // A let interrupts the lambda telescope; its existing zeta/beta handling + // must still close the inferred type without leaving a Let or local FVar. + let body = KExpr::let_(name::("z"), var::(1), var(0), var(0), false); + differential(&lam(sort(1), lam(var(0), body))); + + // A lambda's inferred body type can be a head beta redex. Full validation + // must accept the dependent annotation and closing must still cheap-beta. + let redex = app(lam(sort(1), var(0)), var(0)); + differential(&lam(sort(1), lam(redex, var::(0)))); +} + +#[test] +fn batch_dependent_domains_and_metadata_match_sequential() { + dependent_examples::(); + dependent_examples::(); +} + +fn universes() { + let p = KUniv::param(0, name::("u")); + let q = KUniv::param(1, name::("v")); + for domain in [sort(0), sort(1), KExpr::sort(p.clone())] { + for inner in [sort(0), sort(2), KExpr::sort(q.clone())] { + differential(&all::(domain.clone(), all(inner, sort(0)))); + } + } + // (P : Prop) -> (A : Sort u) -> P lives in Prop regardless of u. + let e = all(sort(0), all(KExpr::sort(p), var::(1))); + differential(&e); + let mut env = KEnv::new(); + let mut tc = TypeChecker::new(&mut env); + let ty = tc.infer(&e).unwrap(); + assert!(matches!(ty.data(), ExprData::Sort(u, _) if u.is_zero())); +} + +#[test] +fn batch_forall_preserves_imax_prop_and_symbolic_universes() { + universes::(); + universes::(); +} + +fn outer_contexts() { + let mut baseline = KEnv::new(); + let mut batched = KEnv::new(); + let mut a = TypeChecker::new(&mut baseline); + let mut b = TypeChecker::new(&mut batched); + // Preserve both kinds of enclosing local context while opening a batch. + for tc in [&mut a, &mut b] { + tc.push_local(sort(1)); + let id = tc.fresh_fvar_id(); + assert_eq!(id, FVarId(0)); + tc.lctx.push( + id, + LocalDecl::CDecl { + name: name::("outer"), + bi: M::meta_field(BinderInfo::Default), + ty: sort(1), + }, + ); + } + let legacy = lam(var(0), lam(var(1), var::(0))); + let outer = KExpr::fvar(FVarId(0), name::("outer")); + let free = lam(outer.clone(), lam(outer, var(0))); + for e in [legacy, free] { + let ea = a.intern(e.clone()); + let eb = b.intern(e); + same_shape(reference(&mut a, &ea).unwrap(), b.infer(&eb).unwrap()); + assert_eq!(a.lctx.len(), 1); + assert_eq!(b.lctx.len(), 1); + assert_eq!(b.ctx.len(), 1); + assert!(b.lctx.find(FVarId(0)).is_some()); + } +} + +#[test] +fn batch_preserves_outer_fvars_and_legacy_context() { + outer_contexts::(); + outer_contexts::(); +} + +fn missing(s: &str) -> KExpr { + KExpr::cnst(KId::new(Address::hash(s.as_bytes()), name::(s)), Box::new([])) +} + +fn errors() { + // Every domain, not just the first, must be checked before the body. + let good = [sort::(1), sort(0), var(1)]; + for bad_index in 0..=good.len() { + for forall in [false, true] { + let mut e = missing("bad-body"); + for (i, ty) in good.iter().enumerate().rev() { + let domain = + if i == bad_index { missing("bad-domain") } else { ty.clone() }; + e = if forall { all(domain, e) } else { lam(domain, e) }; + } + for infer_only in [false, true] { + let mut baseline = KEnv::new(); + let mut batched = KEnv::new(); + let mut a = TypeChecker::new(&mut baseline); + let mut b = TypeChecker::new(&mut batched); + a.infer_only = infer_only; + b.infer_only = infer_only; + let expected = reference(&mut a, &e).unwrap_err(); + let actual = b.infer(&e).unwrap_err(); + assert_eq!(expected.to_string(), actual.to_string()); + assert!(b.lctx.is_empty()); + let key = b.infer_key(&e); + assert!(!b.env.infer_cache.contains_key(&key)); + assert!(!b.env.infer_only_cache.contains_key(&key)); + } + } + } + // A domain whose inferred type is not a sort, and an ill-scoped body. + for e in [ + lam(sort(1), lam(var(0), lam(var(0), sort::(0)))), + all(sort(1), all(var(0), all(var(0), sort(0)))), + lam(sort(1), lam(sort(1), var(5))), + all(sort(1), all(sort(1), var(5))), + ] { + let mut baseline = KEnv::new(); + let mut batched = KEnv::new(); + let expected = + reference(&mut TypeChecker::new(&mut baseline), &e).unwrap_err(); + let mut tc = TypeChecker::new(&mut batched); + let actual = tc.infer(&e).unwrap_err(); + assert_eq!(expected.to_string(), actual.to_string()); + assert!(tc.lctx.is_empty()); + } +} + +#[test] +fn batch_errors_preserve_validation_order_and_restore_scope() { + errors::(); + errors::(); +} + +fn caches() { + let mut env = KEnv::new(); + let mut tc = TypeChecker::new(&mut env); + let suffix = tc.intern(lam(sort(1), lam(sort(1), sort::(0)))); + let suffix_ty = tc.infer(&suffix).unwrap(); + let e = tc.intern(lam(sort(1), suffix)); + let before = tc.fresh_fvar_id().0; + let ty = tc.infer(&e).unwrap(); + let after = tc.fresh_fvar_id().0; + assert_eq!(after - before, 2, "cached suffix was opened again"); + let expected = tc.env.intern.intern_all( + M::meta_field(Name::anon()), + M::meta_field(BinderInfo::Default), + &sort(1), + &suffix_ty, + ); + same_shape(ty.clone(), expected); + let before = tc.fresh_fvar_id().0; + let entries = tc.env.infer_cache.len(); + assert!(tc.infer(&e).unwrap().ptr_eq(&ty)); + assert_eq!(tc.fresh_fvar_id().0 - before, 1); + assert_eq!(tc.env.infer_cache.len(), entries); + + // An infer-only suffix must not be accepted as a validated full-mode + // result. This application passes synthesis but has an invalid argument. + let bad_app = app(lam(sort(1), var(0)), sort(1)); + let bad_suffix = tc.intern(lam(sort(1), lam(sort(1), bad_app))); + assert!(tc.with_infer_only(|tc| tc.infer(&bad_suffix)).is_ok()); + let bad_outer = tc.intern(lam(sort(1), bad_suffix)); + assert!(matches!(tc.infer(&bad_outer), Err(TcError::AppTypeMismatch { .. }))); + assert!(tc.lctx.is_empty()); + let key = tc.infer_key(&bad_outer); + assert!(!tc.env.infer_cache.contains_key(&key)); + + // Conversely, a full result is available to infer-only callers too. + let before = tc.fresh_fvar_id().0; + assert!(tc.with_infer_only(|tc| tc.infer(&e)).unwrap().ptr_eq(&ty)); + assert_eq!(tc.fresh_fvar_id().0 - before, 1); +} + +#[test] +fn batch_keeps_outer_and_closed_suffix_cache_hits_mode_safe() { + caches::(); + caches::(); +} + +#[test] +fn batch_forall_reuses_closed_suffix_without_reopening() { + let mut env = KEnv::::new(); + let mut tc = TypeChecker::new(&mut env); + let suffix = tc.intern(all(sort(1), all(sort(2), sort(0)))); + let suffix_ty = tc.infer(&suffix).unwrap(); + let e = tc.intern(all(sort(1), suffix)); + let before = tc.fresh_fvar_id().0; + let ty = tc.infer(&e).unwrap(); + assert_eq!(tc.fresh_fvar_id().0 - before, 2); + let body_level = tc.ensure_sort(&suffix_ty).unwrap(); + let domain_level = tc.ensure_sort(&sort(2)).unwrap(); + let expected = tc.intern(KExpr::sort(KUniv::imax(domain_level, body_level))); + same_shape(ty, expected); +} + +#[test] +fn batch_error_preserves_existing_local_and_does_not_recycle_ids() { + let mut env = KEnv::::new(); + let mut tc = TypeChecker::new(&mut env); + let (outer_id, outer) = tc.push_fvar_decl_anon(sort(1)); + let e = lam(outer.clone(), lam(outer.clone(), missing("bad-body"))); + assert!(tc.infer(&e).is_err()); + assert_eq!(tc.lctx.len(), 1); + assert!(tc.lctx.find(outer_id).is_some()); + assert_eq!(tc.fresh_fvar_id().0, outer_id.0 + 3); + let ty = tc.infer(&outer).unwrap(); + same_shape(ty, sort(1)); +} + +// A well-typed DAG under a long telescope: +// (A : Type) (x_0 ... x_n : A) (f : A -> A -> A) => balanced f-tree. +// Distinct leaves keep the term wide without making recursive descent deep. +fn wide_telescope(n: u64) -> KExpr { + assert!(n.is_power_of_two()); + let mut domains = vec![sort(1)]; + domains.extend((0..n).map(var)); + domains.push(all(var(n), all(var(n + 1), var(n + 2)))); + let f = var::(0); + let mut layer: Vec<_> = (1..=n).map(var).collect(); + while layer.len() > 1 { + layer = layer + .as_chunks::<2>() + .0 + .iter() + .map(|pair| app(app(f.clone(), pair[0].clone()), pair[1].clone())) + .collect(); + } + let mut body = layer.pop().unwrap(); + for domain in domains.into_iter().rev() { + body = lam(domain, body); + } + body +} + +struct Measurement { + ty: KExpr, + ops: OpCounts, + elapsed: std::time::Duration, + entries: usize, +} + +fn measure(e: &KExpr, batched: bool) -> Measurement { + let mut env = KEnv::new(); + let mut tc = TypeChecker::new(&mut env); + let e = tc.intern(e.clone()); + take_op_counts(); + let start = std::time::Instant::now(); + let ty = if batched { tc.infer(&e) } else { reference(&mut tc, &e) }.unwrap(); + let elapsed = start.elapsed(); + Measurement { + ty, + ops: take_op_counts(), + elapsed, + entries: tc.env.infer_cache.len(), + } +} + +fn work_reduction() { + let e = wide_telescope::(64); + let a = measure(&e, false); + let b = measure(&e, true); + same_shape(a.ty, b.ty); + assert!( + b.ops.subst_nodes * 4 < a.ops.subst_nodes, + "batch did not eliminate repeated opening: {} vs {} visits", + b.ops.subst_nodes, + a.ops.subst_nodes, + ); + assert!(b.entries < a.entries, "intermediate suffixes were still cached"); +} + +#[test] +fn batch_reduces_telescope_node_visits_without_changing_type() { + work_reduction::(); + work_reduction::(); +} + +#[test] +#[ignore = "manual paired release microbenchmark; timings are not a test gate"] +fn batch_binder_microbenchmark() { + let e = wide_telescope::(128); + for round in 0..5 { + for batched in if round % 2 == 0 { [false, true] } else { [true, false] } { + let m = measure(&e, batched); + eprintln!( + "round={round} batched={batched} elapsed={:?} subst={} intern={} infer_entries={}", + m.elapsed, m.ops.subst_nodes, m.ops.intern_nodes, m.entries, + ); + } + } +} diff --git a/crates/kernel/src/infer/summary.rs b/crates/kernel/src/infer/summary.rs new file mode 100644 index 000000000..e8c7df77a --- /dev/null +++ b/crates/kernel/src/infer/summary.rs @@ -0,0 +1,232 @@ +//! Conservative declaration summaries for proof-irrelevance eligibility. +//! +//! For a syntactic telescope `c : (x1 : A1) ... (xn : An) -> T`, the +//! classifying sort of T determines whether c, and each of its first n +//! partial applications, can be a proof: `imax a b` is zero iff b is zero. +//! We recognize T's sort ONLY from Sort nodes or syntactic types of its +//! constant/bound-variable head. No delta, reduction probes, fresh locals, +//! or argument validation occur here. Unknown means use ordinary inference. +//! +//! The substitution lemma justifies transferring a known nonzero sort to +//! well-typed arguments. This is NOT a typechecking shortcut: all arguments, +//! declaration types/bodies and universe scopes still undergo normal checks. + +use super::*; +use crate::level::UnivData; + +const MAX_SUMMARY_ARITY: usize = 64; +const MAX_LEVEL_VISITS: usize = 256; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum ProofEligibility { + Unknown, + /// The result's type has sort Prop; still check BOTH proposition types + /// with the existing proof-irrelevance procedure, never accept from here. + ProofEligible, + NonProof, +} + +#[derive(Clone, Copy, Debug)] +pub(crate) struct DeclarationSummary { + pub(crate) arity: usize, + pub(crate) result: ProofEligibility, +} + +const UNKNOWN: DeclarationSummary = + DeclarationSummary { arity: 0, result: ProofEligibility::Unknown }; + +/// A bounded syntactic walk; deeper spines use ordinary inference. +fn head_and_arity( + mut e: &KExpr, +) -> Option<(&KExpr, usize)> { + let mut arity = 0; + while let ExprData::App(f, _, _) = e.data() { + if arity == MAX_SUMMARY_ARITY { + return None; + } + arity += 1; + e = f; + } + Some((e, arity)) +} + +impl TypeChecker<'_, M> { + /// This predicate can only SKIP an inapplicable proof-irrelevance attempt. + /// False includes unknown, malformed, overapplied, and propositional cases. + pub(crate) fn known_non_proof(&mut self, e: &KExpr) -> bool { + let Some((head, arity)) = head_and_arity(e) else { return false }; + if !matches!(head.data(), ExprData::Const(..)) { + return false; + } + let key = head.hash_key(); + let summary = if let Some(summary) = self.env.decl_summary_cache.get(&key) { + self.env.perf.record_decl_summary_hit(); + *summary + } else { + self.env.perf.record_decl_summary_miss(); + // Dependency lookup/arity/substitution errors are not memoized as + // negative facts. A later call with the dependency loaded can retry. + let Ok(summary) = self.summarize_declaration(head) else { return false }; + self.env.decl_summary_cache.insert(key, summary); + summary + }; + let skip = + arity <= summary.arity && summary.result == ProofEligibility::NonProof; + if skip { + self.env.perf.record_non_proof_skip(); + } + skip + } + + fn summarize_declaration( + &mut self, + head: &KExpr, + ) -> Result> { + let ExprData::Const(id, us, _) = head.data() else { return Ok(UNKNOWN) }; + let c = self.get_const(id)?; + if u64_to_usize::(c.lvls())? != us.len() { + return Err(TcError::UnivParamMismatch { + expected: c.lvls(), + got: us.len(), + }); + } + // Ordinary declaration checking enforces these. The summary must also + // decline malformed hand-built/temporarily loaded declarations. + if c.ty().lbr() != 0 || c.ty().has_fvars() { + return Ok(UNKNOWN); + } + let mut domains: smallvec::SmallVec<[&KExpr; 8]> = + smallvec::SmallVec::new(); + let mut ty = c.ty(); + while let ExprData::All(_, _, dom, cod, _) = ty.data() { + if domains.len() == MAX_SUMMARY_ARITY { + return Ok(UNKNOWN); + } + domains.push(dom); + ty = cod; + } + let Some(result) = self.summary_type_classification(ty, &domains, us)? + else { + return Ok(UNKNOWN); + }; + Ok(DeclarationSummary { arity: domains.len(), result }) + } + + /// Infer only a syntactically evident sort of a telescope's terminal + /// type. For a constant/application, its head type must expose exactly the + /// required Pis followed by Sort. Bound-variable heads use their original + /// domain; no lifting is necessary for the final universe-only result. + fn summary_type_classification( + &mut self, + ty: &KExpr, + domains: &[&KExpr], + outer_us: &[KUniv], + ) -> Result, TcError> { + let mut visits = MAX_LEVEL_VISITS; + if let ExprData::Sort(u, _) = ty.data() { + // Sort u : Sort (u+1), never Prop. Still validate the universe + // substitution within the same bounded walk before recording a fact. + return Ok( + classify_sort(u, &[outer_us], &mut visits)? + .map(|_| ProofEligibility::NonProof), + ); + } + let Some((head, args)) = head_and_arity(ty) else { return Ok(None) }; + let (head_ty, levels) = match head.data() { + ExprData::Var(i, _, _) => { + let Some(pos) = usize::try_from(*i) + .ok() + .and_then(|i| i.checked_add(1)) + .and_then(|n| domains.len().checked_sub(n)) + else { + return Ok(None); + }; + (domains[pos].clone(), None) + }, + ExprData::Const(id, us, _) => { + let c = self.get_const(id)?; + if u64_to_usize::(c.lvls())? != us.len() { + return Err(TcError::UnivParamMismatch { + expected: c.lvls(), + got: us.len(), + }); + } + if c.ty().lbr() != 0 || c.ty().has_fvars() { + return Ok(None); + } + (c.ty().clone(), Some(us)) + }, + _ => return Ok(None), + }; + let mut result = &head_ty; + for _ in 0..args { + let ExprData::All(_, _, _, cod, _) = result.data() else { + return Ok(None); + }; + result = cod; + } + let ExprData::Sort(u, _) = result.data() else { return Ok(None) }; + // Type-head universes are in this declaration's formal parameters. + // Apply the inner substitution BEFORE the outer one, lazily, without + // building levels or invoking unbounded universe-normalization helpers. + match levels { + Some(us) => classify_sort(u, &[us, outer_us], &mut visits), + None => classify_sort(u, &[outer_us], &mut visits), + } + } +} + +/// Interpret chained universe substitutions without constructing a level. +/// A parameter consumes ONE substitution layer, so an actual universe's +/// parameters cannot accidentally be captured by its own substitution. +/// Validate both branches, even when one determines nonzeroness, so errors +/// are not memoized as facts. None means the bounded analysis ran out of work. +/// A symbolic parameter with no substitution left is Unknown, NOT NonProof. +fn classify_sort( + u: &KUniv, + substitutions: &[&[KUniv]], + remaining: &mut usize, +) -> Result, TcError> { + use ProofEligibility::{NonProof, ProofEligible, Unknown}; + if *remaining == 0 { + return Ok(None); + } + *remaining -= 1; + let result = match u.data() { + UnivData::Zero(_) => Some(ProofEligible), + UnivData::Succ(inner, _) => { + classify_sort(inner, substitutions, remaining)?.map(|_| NonProof) + }, + UnivData::Param(i, _, _) => { + let Some((us, rest)) = substitutions.split_first() else { + return Ok(Some(Unknown)); + }; + let Some(actual) = usize::try_from(*i).ok().and_then(|i| us.get(i)) + else { + return Err(TcError::UnivParamOutOfRange { idx: *i, bound: us.len() }); + }; + return classify_sort(actual, rest, remaining); + }, + UnivData::IMax(a, b, _) | UnivData::Max(a, b, _) => { + let Some(a) = classify_sort(a, substitutions, remaining)? else { + return Ok(None); + }; + let Some(b) = classify_sort(b, substitutions, remaining)? else { + return Ok(None); + }; + Some(if matches!(u.data(), UnivData::IMax(..)) { + b + } else { + match (a, b) { + (NonProof, _) | (_, NonProof) => NonProof, + (ProofEligible, ProofEligible) => ProofEligible, + _ => Unknown, + } + }) + }, + }; + Ok(result) +} + +#[cfg(test)] +mod tests; diff --git a/crates/kernel/src/infer/summary/tests.rs b/crates/kernel/src/infer/summary/tests.rs new file mode 100644 index 000000000..b347c2636 --- /dev/null +++ b/crates/kernel/src/infer/summary/tests.rs @@ -0,0 +1,294 @@ +use super::*; +use crate::env::KEnv; +use crate::mode::{Anon, Meta}; +use ix_common::address::Address; +use ix_common::env::{BinderInfo, Name}; + +fn name(s: &str) -> M::MField { + M::meta_field(Name::str(Name::anon(), s.to_owned())) +} +fn sort(n: u64) -> KExpr { + let mut u = KUniv::zero(); + for _ in 0..n { + u = KUniv::succ(u); + } + KExpr::sort(u) +} +fn var(i: u64) -> KExpr { + KExpr::var(i, name::("var")) +} +fn all(dom: KExpr, cod: KExpr) -> KExpr { + KExpr::all(name::("binder"), M::meta_field(BinderInfo::Implicit), dom, cod) +} +fn axiom( + env: &mut KEnv, + s: &str, + lvls: u64, + ty: KExpr, +) -> KId { + let id = KId::new(Address::hash(s.as_bytes()), name::(s)); + env.insert( + id.clone(), + KConst::Axio { + name: name::(s), + level_params: M::meta_field( + (0..lvls).map(|i| Name::str(Name::anon(), format!("u{i}"))).collect(), + ), + is_unsafe: false, + lvls, + ty, + }, + ); + id +} +fn cnst(id: &KId, us: &[KUniv]) -> KExpr { + KExpr::cnst(id.clone(), us.to_vec().into_boxed_slice()) +} +fn app(f: KExpr, a: KExpr) -> KExpr { + KExpr::app(f, a) +} + +fn propositions_and_data() { + let mut env = KEnv::::new(); + let prop = axiom(&mut env, "P", 0, sort(0)); + let p = cnst(&prop, &[]); + let proof1 = axiom(&mut env, "p1", 0, p.clone()); + let proof2 = axiom(&mut env, "p2", 0, p.clone()); + let data = axiom(&mut env, "A", 0, sort(1)); + let a = cnst(&data, &[]); + let data1 = axiom(&mut env, "x1", 0, a.clone()); + let data2 = axiom(&mut env, "x2", 0, a.clone()); + let mut tc = TypeChecker::new(&mut env); + assert!(tc.known_non_proof(&p), "a proposition is not itself a proof"); + assert!(tc.known_non_proof(&a)); + assert!(tc.known_non_proof(&cnst(&data1, &[]))); + assert!(!tc.known_non_proof(&cnst(&proof1, &[]))); + assert!(tc.is_def_eq(&cnst(&proof1, &[]), &cnst(&proof2, &[])).unwrap()); + assert!(!tc.is_def_eq(&cnst(&data1, &[]), &cnst(&data2, &[])).unwrap()); +} + +#[test] +fn distinguishes_propositions_proofs_and_data_in_both_modes() { + propositions_and_data::(); + propositions_and_data::(); +} + +fn universes() { + let mut env = KEnv::::new(); + let u = KUniv::param(0, name::("u")); + // id.{u} : (A : Sort u) -> A -> A. For u=0 it is a proof even + // before application, because the remaining Pis are impredicative. + let id = + axiom(&mut env, "identity", 1, all(KExpr::sort(u), all(var(0), var(1)))); + let mut tc = TypeChecker::new(&mut env); + for (u, expected) in [ + (KUniv::zero(), false), + (KUniv::succ(KUniv::zero()), true), + (KUniv::param(7, name::("symbolic")), false), + (KUniv::imax(KUniv::succ(KUniv::zero()), KUniv::zero()), false), + ( + KUniv::max(KUniv::param(2, name::("v")), KUniv::succ(KUniv::zero())), + true, + ), + ] { + let head = cnst(&id, &[u]); + let summary = tc.summarize_declaration(&head).unwrap(); + assert_eq!(summary.arity, 2); + assert_eq!(summary.result == ProofEligibility::NonProof, expected); + let mut term = head; + for _ in 0..=2 { + assert_eq!(tc.known_non_proof(&term), expected); + // Eligibility is independent of argument values; this does NOT assert + // that these deliberately arbitrary arguments are well-typed. + term = app(term, sort(0)); + } + assert!(!tc.known_non_proof(&term), "overapplication is not summarized"); + } + assert_eq!(tc.env.decl_summary_cache.len(), 5); + assert!(tc.env.infer_cache.is_empty()); + assert!(tc.env.infer_only_cache.is_empty()); + assert!(tc.ctx.is_empty() && tc.lctx.is_empty()); +} + +#[test] +fn universe_instantiations_and_partial_applications_stay_separate() { + universes::(); + universes::(); +} + +fn partial_proofs() { + let mut env = KEnv::::new(); + let p_id = axiom(&mut env, "P", 0, sort(0)); + let p = cnst(&p_id, &[]); + let h_id = axiom(&mut env, "hP", 0, p.clone()); + let h = cnst(&h_id, &[]); + let ty = + all(KExpr::sort(KUniv::param(0, name::("u"))), all(var(0), var(1))); + let f = axiom(&mut env, "f", 1, ty.clone()); + let g = axiom(&mut env, "g", 1, ty); + for (u, args, non_proof) in [ + (KUniv::zero(), [p.clone(), h], false), + (KUniv::succ(KUniv::zero()), [sort(0), p], true), + ] { + let mut a = cnst(&f, std::slice::from_ref(&u)); + let mut b = cnst(&g, &[u]); + let mut tc = TypeChecker::new(&mut env); + for applied in 0..=2 { + tc.infer(&a).unwrap(); + tc.infer(&b).unwrap(); + assert_eq!(tc.known_non_proof(&a), non_proof); + assert_eq!(tc.is_def_eq(&a, &b).unwrap(), !non_proof); + if let Some(arg) = args.get(applied) { + a = app(a, arg.clone()); + b = app(b, arg.clone()); + } + } + } +} + +#[test] +fn impredicative_partial_applications_preserve_proof_irrelevance() { + partial_proofs::(); + partial_proofs::(); +} + +fn families() { + let mut env = KEnv::::new(); + let u = KUniv::param(0, name::("u")); + // F.{u} : Type -> Sort u. g.{u} : (A : Type) -> F.{u} A. + let family = axiom(&mut env, "F", 1, all(sort(1), KExpr::sort(u.clone()))); + let result = app(cnst(&family, std::slice::from_ref(&u)), var(0)); + let g = axiom(&mut env, "g", 1, all(sort(1), result)); + // h : (A : Type) -> (B : A -> Type) -> (x : A) -> B x + let h = axiom( + &mut env, + "h", + 0, + all(sort(1), all(all(var(0), sort(1)), all(var(1), app(var(1), var(0))))), + ); + let h = cnst(&h, &[]); + let mut tc = TypeChecker::new(&mut env); + assert!(!tc.known_non_proof(&cnst(&g, &[KUniv::zero()]))); + assert!(tc.known_non_proof(&cnst(&g, &[KUniv::succ(KUniv::zero())]))); + assert!(tc.known_non_proof(&h)); + // The summary does not create/open FVars or depend on the caller's local + // context. Warm queries must remain free of type-inference/fuel work. + tc.push_local(sort(1)); + tc.rec_fuel = 0; + assert!(tc.known_non_proof(&h)); + assert_eq!(tc.rec_fuel, 0); + assert_eq!(tc.ctx.len(), 1); + assert!(tc.lctx.is_empty()); +} + +#[test] +fn composes_universe_substitutions_and_handles_bound_type_families() { + families::(); + families::(); +} + +#[test] +fn unknown_shapes_fall_back_without_unfolding() { + let mut env = KEnv::::new(); + let hidden = KExpr::let_((), sort(1), sort(0), var(0), false); + let f = axiom(&mut env, "hidden", 0, hidden); + let mut deep = sort(1); + for _ in 0..65 { + deep = all(sort(1), deep); + } + let deep = axiom(&mut env, "deep", 0, deep); + let malformed = axiom(&mut env, "loose", 0, var(0)); + let mut tc = TypeChecker::new(&mut env); + for id in [&f, &deep, &malformed] { + assert!(!tc.known_non_proof(&cnst(id, &[]))); + } + assert!(!tc.known_non_proof(&app(var(0), sort(0)))); + assert!(tc.env.whnf_cache.is_empty() && tc.env.unfold_cache.is_empty()); + assert!(tc.env.infer_cache.is_empty() && tc.env.infer_only_cache.is_empty()); + assert_eq!(tc.fuel_used(), 0); +} + +#[test] +fn arity_missing_dependencies_and_universe_errors_are_not_cached() { + let mut env = KEnv::::new(); + let id = axiom(&mut env, "poly", 1, KExpr::sort(KUniv::param(0, ()))); + let missing = KId::new(Address::hash(b"missing"), ()); + let f = axiom(&mut env, "f", 0, cnst(&missing, &[])); + let broken = axiom(&mut env, "broken", 0, KExpr::sort(KUniv::param(1, ()))); + let mut tc = TypeChecker::new(&mut env); + for e in + [cnst(&id, &[]), cnst(&f, &[]), cnst(&broken, &[]), cnst(&missing, &[])] + { + assert!(!tc.known_non_proof(&e)); + } + assert!(tc.env.decl_summary_cache.is_empty()); + let inserted = axiom(tc.env, "missing", 0, sort(1)); + assert_eq!(inserted, missing); + assert!(tc.known_non_proof(&cnst(&f, &[]))); +} + +#[test] +fn declaration_replacement_and_all_reset_paths_invalidate_summaries() { + let mut env = KEnv::::new(); + let t = axiom(&mut env, "T", 0, sort(1)); + let f = axiom(&mut env, "f", 0, cnst(&t, &[])); + let e = cnst(&f, &[]); + assert!(TypeChecker::new(&mut env).known_non_proof(&e)); + assert_eq!(env.cache_sizes().decl_summary, 1); + // Change a dependency, not just f itself. + axiom(&mut env, "T", 0, sort(0)); + assert!(env.decl_summary_cache.is_empty()); + assert!(!TypeChecker::new(&mut env).known_non_proof(&e)); + for reset in 0..4 { + let t = axiom(&mut env, "T", 0, sort(1)); + let f = axiom(&mut env, "f", 0, cnst(&t, &[])); + assert!(TypeChecker::new(&mut env).known_non_proof(&cnst(&f, &[]))); + assert!(!env.decl_summary_cache.is_empty()); + match reset { + 0 => env.clear_reduction_caches(), + 1 => env.clear(), + 2 => env.clear_with_capacity_limit(0), + _ => env.clear_releasing_memory(), + } + assert!(env.decl_summary_cache.is_empty()); + } +} + +#[test] +fn bounded_level_classification_handles_imax_and_symbolic_zero() { + use ProofEligibility::{NonProof, ProofEligible, Unknown}; + let u: KUniv = KUniv::param(0, ()); + for (level, expected) in [ + (u.clone(), Unknown), + (KUniv::zero(), ProofEligible), + (KUniv::succ(u.clone()), NonProof), + (KUniv::imax(KUniv::succ(u.clone()), u.clone()), Unknown), + (KUniv::imax(u.clone(), KUniv::zero()), ProofEligible), + (KUniv::max(u, KUniv::succ(KUniv::zero())), NonProof), + ] { + assert_eq!(classify_sort(&level, &[], &mut 256).unwrap(), Some(expected)); + assert_eq!(classify_sort(&level, &[], &mut 0).unwrap(), None); + } +} + +#[test] +fn universe_analysis_is_bounded_and_does_not_capture_actual_parameters() { + let mut u: KUniv = KUniv::zero(); + for _ in 0..MAX_LEVEL_VISITS + 1 { + u = KUniv::succ(u); + } + let mut visits = MAX_LEVEL_VISITS; + assert_eq!(classify_sort(&u, &[], &mut visits).unwrap(), None); + // Substituting u0 -> u0 leaves a symbolic parameter, not a cycle. + let p = KUniv::::param(0, ()); + assert_eq!( + classify_sort(&p, &[std::slice::from_ref(&p)], &mut 8).unwrap(), + Some(ProofEligibility::Unknown) + ); + let inner = [p.clone()]; + let outer = [KUniv::zero()]; + assert_eq!( + classify_sort(&p, &[&inner, &outer], &mut 8).unwrap(), + Some(ProofEligibility::ProofEligible) + ); +} diff --git a/crates/kernel/src/ingress.rs b/crates/kernel/src/ingress.rs index 1f8b7eba2..76ddbc869 100644 --- a/crates/kernel/src/ingress.rs +++ b/crates/kernel/src/ingress.rs @@ -3102,6 +3102,47 @@ fn lean_const_to_kconst( } } +/// Lean emits original recursors in `.all` declaration order, followed by +/// nested auxiliaries named `.rec_N` in numeric source-walk order. +/// This is loader layout metadata, not evidence that a recursor is valid: +/// the kernel still checks every peer's full header, type, and rules. +#[cfg(not(target_arch = "riscv64"))] +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +enum LeanRecursorOrder { + Original(usize), + Nested(usize), + Unrecognized, +} + +#[cfg(not(target_arch = "riscv64"))] +fn lean_recursor_order( + name: &Name, + block_name: &Name, + original_positions: &FxHashMap, +) -> LeanRecursorOrder { + use ix_common::env::NameData; + + let NameData::Str(parent, suffix, _) = name.as_data() else { + return LeanRecursorOrder::Unrecognized; + }; + if suffix == "rec" { + return original_positions + .get(parent) + .map_or(LeanRecursorOrder::Unrecognized, |&i| { + LeanRecursorOrder::Original(i) + }); + } + if parent == block_name + && let Some(index) = suffix.strip_prefix("rec_") + && !index.starts_with('0') + && index.bytes().all(|b| b.is_ascii_digit()) + && let Ok(index) = index.parse::() + { + return LeanRecursorOrder::Nested(index); + } + LeanRecursorOrder::Unrecognized +} + /// Direct ingress: build a `KEnv` from a Lean `Env` without going /// through Ixon compilation. Used by the `kernel-lean-roundtrip` /// diagnostic test and by `compile_env` to produce the `orig_kenv` @@ -3188,7 +3229,7 @@ pub fn lean_ingress(lean_env: &LeanEnv) -> KEnv { // the block, rule RHS construction returns None and the stored // rules can't be verified). // - // **Order matters for inductives.** `discover_block_inductives` + // **Order matters for inductives and recursors.** `discover_block_inductives` // filters the block's member list down to `KConst::Indc` entries // and the resulting order drives `build_flat_block` → `build_rec_type` // → motive-binder emission in `generate_block_recursors`. That @@ -3211,9 +3252,9 @@ pub fn lean_ingress(lean_env: &LeanEnv) -> KEnv { // `lean_env` directly to push each constant's `self_kid` gave // random (FxHashMap iteration) order; we now seed each block with // its `all` list the first time any member is observed, then - // append ctors and recursors in a second pass. Ctors/recursors - // land at the tail — the block's inductive-prefix carries the - // declaration order that `discover_block_inductives` consumes. + // append ctors and source-ordered recursors in a second pass. The + // inductive and recursor subsequences must align positionally with the + // flat block (original `.all` members, then source-walk nested auxes). // // `ixon_ingress` builds an analogous list for `kctx.kenv`, but // there the ordering comes from `sort_consts`' equivalence-class @@ -3232,7 +3273,7 @@ pub fn lean_ingress(lean_env: &LeanEnv) -> KEnv { }; // Phase A: seed each block's initial member list from the constant's - // `all` list (canonical order), exactly once per block. Constants + // `all` list (source order), exactly once per block. Constants // without `all` (axioms, quotients, ctors) seed a singleton block // under their own KId. let t = Instant::now(); @@ -3255,12 +3296,12 @@ pub fn lean_ingress(lean_env: &LeanEnv) -> KEnv { ); } - // Phase B: append constructors (for each inductive in the block) and - // recursors (which aren't in `all` — `all` lists inductives even for - // RecInfo). Order within ctors/recs doesn't affect kernel correctness - // because consumer lookups go by KId (ctors) or major-inductive match - // (`find_peer_recursors` for recs). + // Phase B: append constructors and collect recursors separately. `.all` + // lists inductives even for RecInfo; environment iteration is not the + // recursor order required by positional peer matching in the kernel. let t = Instant::now(); + let mut recursors: FxHashMap, Vec>> = + FxHashMap::default(); for (name, ci) in lean_env.iter() { match &*ci { LeanCI::InductInfo(v) => { @@ -3274,7 +3315,7 @@ pub fn lean_ingress(lean_env: &LeanEnv) -> KEnv { LeanCI::RecInfo(_) => { let block_id = block_rep(name, &ci); let self_kid = KId::new(leon_addr_of(name, &n2a), name.clone()); - kenv.blocks.entry(block_id).or_default().push(self_kid); + recursors.entry(block_id).or_default().push(self_kid); }, // Inductives and Defns/Thms/Opaques are already in the Phase-A // seed via their `all` list; axioms, quotients, and ctors are @@ -3282,6 +3323,21 @@ pub fn lean_ingress(lean_env: &LeanEnv) -> KEnv { _ => {}, } } + for (block_id, mut peers) in recursors { + let members = kenv.blocks.entry(block_id.clone()).or_default(); + let original_positions: FxHashMap = members + .iter() + .filter(|id| matches!(kenv.consts.get(id), Some(KConst::Indc { .. }))) + .enumerate() + .map(|(i, id)| (id.name.clone(), i)) + .collect(); + peers.sort_by_cached_key(|id| { + lean_recursor_order(&id.name, &block_id.name, &original_positions) + }); + // Keep unrecognized names too. Dropping malformed/extra peers here + // could hide a recursor-count or type mismatch from kernel validation. + members.extend(peers); + } if !quiet { log::info!( "[lean_ingress] phase B (ctor/rec append): {:.2}s", @@ -4817,6 +4873,62 @@ mod tests { Nat::from(x) } + #[cfg(not(target_arch = "riscv64"))] + #[test] + fn lean_recursor_layout_uses_source_then_numeric_order() { + // Layout-only test: the source mutual may split into structural SCCs, + // but direct ingress must keep all three originals in `.all` order. + let head = mk_name("Z"); + let positions = FxHashMap::from_iter([ + (head.clone(), 0), + (mk_name("A"), 1), + (mk_name("C"), 2), + ]); + let mut peers = + ["Z.rec_10", "A.rec", "Z.rec_2", "C.rec", "Z.rec_1", "Z.rec"] + .map(mk_name); + peers + .sort_by_cached_key(|name| lean_recursor_order(name, &head, &positions)); + assert_eq!( + peers.map(|name| name.pretty()), + ["Z.rec", "A.rec", "C.rec", "Z.rec_1", "Z.rec_2", "Z.rec_10"], + ); + } + + #[cfg(not(target_arch = "riscv64"))] + #[test] + fn lean_recursor_layout_does_not_recognize_foreign_or_malformed_names() { + let head = mk_name("Z"); + let positions = + FxHashMap::from_iter([(head.clone(), 0), (mk_name("A"), 1)]); + for name in [ + "Foreign.rec", + "Foreign.rec_1", + "A.rec_1", + "Z.rec_0", + "Z.rec_01", + "Z.rec_", + "Z.rec_+1", + "Z.rec_-1", + "Z.rec_١", + "Z.rec_9999999999999999999999999999999999999999", + ] { + assert_eq!( + lean_recursor_order(&mk_name(name), &head, &positions), + LeanRecursorOrder::Unrecognized, + "{name} must not claim a valid source slot", + ); + } + assert_eq!( + lean_recursor_order( + &Name::num(head.clone(), n_lit(1)), + &head, + &positions + ), + LeanRecursorOrder::Unrecognized, + ); + } + // ---- lean_level_to_kuniv ---- #[test] diff --git a/crates/kernel/src/intern_tests.rs b/crates/kernel/src/intern_tests.rs index 288581133..9f3d7a651 100644 --- a/crates/kernel/src/intern_tests.rs +++ b/crates/kernel/src/intern_tests.rs @@ -226,6 +226,174 @@ fn interning_matches_reference_in_both_modes_and_seeded_tables() { differential::(); } +fn smart_constructor_differential() { + let mut old = KEnv::::new(); + let mut new = KEnv::::new(); + // Reusing old inputs after clearing must not reuse stale canonical keys. + let inputs = fixtures::("input"); + for seeded in [false, true, false] { + old.clear_releasing_memory(); + new.clear_releasing_memory(); + if seeded { + for seed in fixtures::("first") { + old.intern.intern_expr_reference(seed.clone()); + new.intern.intern_expr(seed); + } + } + for pair in inputs.windows(2) { + let (a, b) = (&pair[0], &pair[1]); + // Seed parent occurrences too: their metadata must win on a hit, + // including when the new constructor supplies different binder info. + if seeded { + for root in [ + KExpr::app_mdata( + a.clone(), + b.clone(), + M::meta_field(metadata("parent")), + ), + KExpr::all_mdata( + M::meta_field(name("parent")), + M::meta_field(BinderInfo::Implicit), + a.clone(), + b.clone(), + M::meta_field(metadata("parent")), + ), + ] { + old.intern.intern_expr_reference(root.clone()); + new.intern.intern_expr(root); + } + } + for _ in 0..2 { + let expected = + old.intern.intern_expr_reference(KExpr::app(a.clone(), b.clone())); + let actual = new.intern.intern_app(a, b); + assert_same_expr(&actual, &expected); + assert!(new.intern.intern_expr(actual.clone()).ptr_eq(&actual)); + let n = M::meta_field(name("second")); + let bi = M::meta_field(BinderInfo::InstImplicit); + let expected = old.intern.intern_expr_reference(KExpr::all( + n.clone(), + bi.clone(), + a.clone(), + b.clone(), + )); + let actual = new.intern.intern_all(n, bi, a, b); + assert_same_expr(&actual, &expected); + } + } + assert_eq!(new.intern.exprs.len(), old.intern.exprs.len()); + assert_eq!(new.intern.univs.len(), old.intern.univs.len()); + } +} + +#[test] +fn smart_constructors_match_reference_with_metadata_and_resets() { + smart_constructor_differential::(); + smart_constructor_differential::(); +} + +#[test] +fn smart_constructor_hit_does_not_invoke_node_factory() { + let mut intern = InternTable::::new(); + let a = intern.intern_expr(KExpr::var(0, ())); + let b = intern.intern_expr(KExpr::var(1, ())); + let key = ExprKey::App(*a.addr(), *b.addr()); + let app = intern.intern_expr_with(key.clone(), || KExpr::app(a, b)); + let hit = intern.intern_expr_with(key, || panic!("allocated on hit")); + assert!(app.ptr_eq(&hit)); + assert!(intern.canon_exprs.contains(app.addr())); +} + +#[test] +fn smart_constructor_canonicalizes_shared_noncanonical_children_once() { + let mut intern = InternTable::::new(); + intern.intern_expr(KExpr::sort(KUniv::zero())); + let input = expr_dag(KExpr::sort(KUniv::zero()), 60); + take_op_counts(); + let app = intern.intern_app(&input, &input); + assert_eq!(take_op_counts().intern_nodes, 124); + let ExprData::App(a, b, _) = app.data() else { panic!("app") }; + assert!(a.ptr_eq(b)); + assert!(!intern.canon_exprs.contains(input.addr())); +} + +#[test] +fn smart_constructor_preserves_same_uid_occurrence_metadata() { + let a = KExpr::::var(0, name("a")); + let mut info = a.info().clone(); + info.mdata = metadata("b"); + let b = KExpr::new(ExprData::Var(0, name("b"), info)); + let expected = + InternTable::new().intern_expr_reference(KExpr::app(a.clone(), b.clone())); + let actual = InternTable::new().intern_app(&a, &b); + assert_same_expr(&actual, &expected); + let ExprData::App(_, rhs, _) = actual.data() else { panic!("app") }; + assert!(rhs.ptr_eq(&b)); +} + +#[test] +fn constant_level_buffer_preserves_unchanged_prefix_and_decorations() { + for first_change in 0..=4 { + let mut old = InternTable::::new(); + let mut new = InternTable::::new(); + let mut levels = Vec::new(); + for i in 0..4 { + let seed = KUniv::param(i, name("first")); + old.intern_univ_reference(seed.clone()); + new.intern_univ(seed.clone()); + levels.push(if i < first_change { + seed + } else { + KUniv::param(i, name("second")) + }); + } + let root = KExpr::cnst_full( + KId::new(Address::hash(b"C"), name("C")), + levels.into(), + metadata("constant"), + Some(UnivDecor::Const(vec![Univ::zero(); 4].into())), + ); + let actual = new.intern_expr(root.clone()); + let expected = old.intern_expr_reference(root.clone()); + assert_same_expr(&actual, &expected); + assert_eq!(actual.ptr_eq(&root), first_change == 4); + } +} + +#[test] +#[ignore = "manual release benchmark; no wall-clock assertions"] +fn benchmark_smart_constructors() { + use std::{hint::black_box, time::Instant}; + for hits in [true, false] { + let count: usize = if hits { 200_000 } else { 30_000 }; + for smart in [false, true, true, false] { + let mut table = InternTable::::new(); + let head = table.intern_expr(KExpr::var(0, ())); + let inputs: Vec<_> = (0..if hits { 1 } else { count }) + .map(|i| { + table.intern_expr(KExpr::var(u64::try_from(i).unwrap() + 1, ())) + }) + .collect(); + if hits { + table.intern_app(&head, &inputs[0]); + } + let start = Instant::now(); + for i in 0..count { + let arg = inputs[if hits { 0 } else { i }].clone(); + black_box(if smart { + table.intern_app(&head, &arg) + } else { + table.intern_expr(KExpr::app(head.clone(), arg)) + }); + } + eprintln!( + "smart={smart} hits={hits} count={count} elapsed={:?}", + start.elapsed() + ); + } + } +} + #[test] fn shared_expression_dag_visits_edges_not_expanded_tree() { let depth = 60; @@ -329,6 +497,10 @@ fn input_pointer_keys_preserve_same_uid_spelling_twins() { old.intern_univ_reference(KUniv::zero()); new.intern_univ(KUniv::zero()); let expected = old.intern_expr_reference(root.clone()); + let mut smart = InternTable::new(); + smart.intern_univ(KUniv::zero()); + let ExprData::App(f, a, _) = root.data() else { panic!("app") }; + assert_same_expr(&smart.intern_app(f, a), &expected); let result = new.intern_expr(root); assert_same_expr(&result, &expected); let ExprData::App(a, rest, _) = result.data() else { panic!("app") }; diff --git a/crates/kernel/src/perf.rs b/crates/kernel/src/perf.rs index 033a77615..42868d892 100644 --- a/crates/kernel/src/perf.rs +++ b/crates/kernel/src/perf.rs @@ -1,6 +1,7 @@ //! Performance counters for cache hit-rate and fuel-consumption analysis. //! -//! All counters are gated behind the `IX_PERF_COUNTERS=1` environment variable. +//! The environment counters below use `IX_PERF_COUNTERS=1`; the separate +//! [`same_head`] diagnostic uses `IX_SAME_HEAD_PROFILE=1`. //! When the variable is unset (production default), every recording call is a //! single inlined branch on a `LazyLock` and skips the atomic increment //! entirely. When set, the counters track: @@ -29,6 +30,9 @@ use std::fmt; use std::sync::atomic::{AtomicU64, Ordering}; +pub(crate) mod hot_misses; +pub mod same_head; + static PERF_ENABLED: crate::EnvFlag = crate::EnvFlag::new(|| crate::env_var_os("IX_PERF_COUNTERS").is_some()); @@ -56,6 +60,8 @@ pub struct PerfCounters { pub infer_cache_misses: AtomicU64, pub infer_only_cache_hits: AtomicU64, pub infer_only_cache_misses: AtomicU64, + /// Selectively materialized dependent application-prefix types. + pub dependent_prefix_inserts: AtomicU64, // -- Def-eq caches -- pub def_eq_cache_hits: AtomicU64, @@ -71,6 +77,11 @@ pub struct PerfCounters { pub is_prop_cache_hits: AtomicU64, pub is_prop_cache_misses: AtomicU64, + // -- Conservative declaration summaries -- + pub decl_summary_hits: AtomicU64, + pub decl_summary_misses: AtomicU64, + pub non_proof_skips: AtomicU64, + // -- Recursive fuel -- /// Running max of fuel actually consumed by any single constant check. pub peak_rec_fuel_used: AtomicU64, @@ -139,6 +150,10 @@ impl PerfCounters { bump(&self.infer_only_cache_misses); } + pub fn record_dependent_prefix_insert(&self) { + bump(&self.dependent_prefix_inserts); + } + // ----------------------------------------------------------------------- // Def-eq caches // ----------------------------------------------------------------------- @@ -183,6 +198,16 @@ impl PerfCounters { bump(&self.is_prop_cache_misses); } + pub fn record_decl_summary_hit(&self) { + bump(&self.decl_summary_hits); + } + pub fn record_decl_summary_miss(&self) { + bump(&self.decl_summary_misses); + } + pub fn record_non_proof_skip(&self) { + bump(&self.non_proof_skips); + } + // ----------------------------------------------------------------------- // Recursive fuel // ----------------------------------------------------------------------- @@ -262,6 +287,11 @@ impl PerfCounters { &self.infer_only_cache_hits, &self.infer_only_cache_misses, )?; + writeln!( + out, + " dependent_prefix_inserts={}", + self.dependent_prefix_inserts.load(Ordering::Relaxed) + )?; write_rate( out, " def_eq_cache ", @@ -282,6 +312,17 @@ impl PerfCounters { )?; let fail_hits = self.def_eq_failure_hits.load(Ordering::Relaxed); + write_rate( + out, + " decl_summary ", + &self.decl_summary_hits, + &self.decl_summary_misses, + )?; + writeln!( + out, + " non_proof_skips {}", + self.non_proof_skips.load(Ordering::Relaxed) + )?; let fail_inserts = self.def_eq_failure_inserts.load(Ordering::Relaxed); writeln!( out, diff --git a/crates/kernel/src/perf/hot_misses.rs b/crates/kernel/src/perf/hot_misses.rs new file mode 100644 index 000000000..247809e33 --- /dev/null +++ b/crates/kernel/src/perf/hot_misses.rs @@ -0,0 +1,282 @@ +//! Bounded, diagnostic-only heavy hitters. No expression graphs are retained. +//! +//! Space-Saving counters replace the least frequent tracked key when full. +//! A replacement inherits that counter as its error bound. For each retained +//! key, the actual event count lies in [count - error, count]; it is exact +//! before any eviction. Unlike keeping just the first N keys, late hotspots +//! can enter the report. An indexed min-heap bounds storage AND update work +//! (O(log N)); a lazy heap would accumulate stale entries without bound. + +use std::fmt::{self, Write}; + +use rustc_hash::FxHashMap; + +use crate::env::{Addr, CtxAddr}; + +const MAX_ENTRIES: usize = 4096; +const MAX_LABEL_BYTES: usize = 512; + +/// Exact diagnostic identity, independent of the truncated display label. +/// `context` is present only for IX_HOT_MISS_CTX. FVars remain distinguished +/// by their expression UID, just as in the previous string-keyed report. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub(crate) struct MissKey { + pub(crate) phase: &'static str, + pub(crate) a: Addr, + pub(crate) b: Option, + pub(crate) context: Option<(CtxAddr, u64)>, +} + +struct Entry { + key: MissKey, + label: String, + count: u64, + error: u64, +} + +#[derive(Default)] +pub(crate) struct HotMisses { + positions: FxHashMap, + heap: Vec, + events: u64, + replacements: u64, +} + +impl HotMisses { + pub(crate) fn record( + &mut self, + key: MissKey, + label: impl FnOnce(&mut Label) -> fmt::Result, + ) { + self.record_with_capacity(key, label, MAX_ENTRIES); + } + + fn record_with_capacity( + &mut self, + key: MissKey, + label: impl FnOnce(&mut Label) -> fmt::Result, + capacity: usize, + ) { + self.events = self.events.saturating_add(1); + if let Some(&i) = self.positions.get(&key) { + self.heap[i].count = self.heap[i].count.saturating_add(1); + self.sift_down(i); + return; + } + // Format ONLY on admission, not on every hit. Label storage is bounded + // even for enormous metadata names. Keys hold only UIDs/context hashes. + let mut text = Label(String::with_capacity(MAX_LABEL_BYTES)); + let _ = label(&mut text); + if self.heap.len() < capacity { + let i = self.heap.len(); + self.heap.push(Entry { key, label: text.0, count: 1, error: 0 }); + self.positions.insert(key, i); + let mut i = i; + while i > 0 { + let parent = (i - 1) / 2; + if self.heap[parent].count <= self.heap[i].count { + break; + } + self.swap(i, parent); + i = parent; + } + } else { + let min = &self.heap[0]; + let error = min.count; + self.positions.remove(&min.key); + self.heap[0] = + Entry { key, label: text.0, count: error.saturating_add(1), error }; + self.positions.insert(key, 0); + self.replacements = self.replacements.saturating_add(1); + self.sift_down(0); + } + } + + fn swap(&mut self, a: usize, b: usize) { + self.heap.swap(a, b); + *self.positions.get_mut(&self.heap[a].key).unwrap() = a; + *self.positions.get_mut(&self.heap[b].key).unwrap() = b; + } + + fn sift_down(&mut self, mut i: usize) { + loop { + let left = 2 * i + 1; + if left >= self.heap.len() { + return; + } + let right = left + 1; + let child = if right < self.heap.len() + && self.heap[right].count < self.heap[left].count + { + right + } else { + left + }; + if self.heap[i].count <= self.heap[child].count { + return; + } + self.swap(i, child); + i = child; + } + } + + pub(crate) fn clear(&mut self) { + self.positions.clear(); + self.heap.clear(); + self.events = 0; + self.replacements = 0; + } + + pub(crate) fn summary(&self) -> String { + if self.heap.is_empty() { + return String::new(); + } + let mut entries: Vec<_> = self.heap.iter().collect(); + entries.sort_unstable_by(|a, b| { + b.count.cmp(&a.count).then_with(|| a.label.cmp(&b.label)) + }); + let mut out = format!( + "[hot misses] events={} tracked={}/{} replacements={}; top {}; counts are [lower, upper] bounds, not exact after replacement:\n", + self.events, + entries.len(), + MAX_ENTRIES, + self.replacements, + entries.len().min(25) + ); + if self.events == u64::MAX { + out.push_str(" counters saturated: upper bounds may be truncated\n"); + } + for entry in entries.into_iter().take(25) { + let _ = writeln!( + out, + " [{:>8}, {:>8}] {}", + entry.count - entry.error, + entry.count, + entry.label + ); + } + out + } +} + +/// A UTF-8-safe bounded formatter. Returning Err stops Display writers before +/// they accumulate an arbitrarily large diagnostic string. The ellipsis is +/// included in the bound. Labels are never used as identities. +pub(crate) struct Label(String); + +impl Write for Label { + fn write_str(&mut self, s: &str) -> fmt::Result { + let remaining = MAX_LABEL_BYTES - self.0.len(); + if s.len() <= remaining { + self.0.push_str(s); + return Ok(()); + } + let prefix = self.0.len().min(MAX_LABEL_BYTES - 3); + self.0.truncate(self.0.floor_char_boundary(prefix)); + let end = s.floor_char_boundary(MAX_LABEL_BYTES - 3 - self.0.len()); + self.0.push_str(&s[..end]); + self.0.push_str("..."); + Err(fmt::Error) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn key(a: u64) -> MissKey { + MissKey { phase: "infer", a, b: None, context: None } + } + + fn record(misses: &mut HotMisses, id: u64, capacity: usize) { + misses.record_with_capacity(key(id), |s| write!(s, "uid{id}"), capacity); + } + + fn invariant(misses: &HotMisses, exact: &FxHashMap) { + assert_eq!(misses.heap.len(), misses.positions.len()); + assert_eq!(misses.events, exact.values().sum()); + assert_eq!(misses.events, misses.heap.iter().map(|e| e.count).sum()); + for (i, e) in misses.heap.iter().enumerate() { + assert_eq!(misses.positions[&e.key], i); + assert!(e.count - e.error <= exact[&e.key.a]); + assert!(exact[&e.key.a] <= e.count); + if i > 0 { + assert!(misses.heap[(i - 1) / 2].count <= e.count); + } + } + } + + #[test] + fn exact_until_full_and_labels_are_only_formatted_on_misses() { + let mut misses = HotMisses::default(); + record(&mut misses, 1, 4); + misses.record(key(1), |_| panic!("do not format an existing key")); + record(&mut misses, 2, 4); + assert_eq!(misses.replacements, 0); + assert_eq!(misses.heap[misses.positions[&key(1)]].count, 2); + assert!(misses.heap.iter().all(|e| e.error == 0)); + } + + #[test] + fn bounded_heap_and_error_intervals_match_exact_counts() { + for capacity in [1, 2, 7, 32] { + let mut misses = HotMisses::default(); + let mut exact = FxHashMap::default(); + let mut random = 1u64; + for i in 0..10_000 { + random = random.wrapping_mul(6364136223846793005).wrapping_add(1); + let id = if i % 3 == 0 { i % 5 } else { random >> 48 }; + record(&mut misses, id, capacity); + *exact.entry(id).or_default() += 1; + assert!(misses.heap.len() <= capacity); + invariant(&misses, &exact); + } + } + } + + #[test] + fn late_hotspots_displace_cold_entries_and_reset_clears_counts() { + let mut misses = HotMisses::default(); + for i in 0..100_000 { + record(&mut misses, i, MAX_ENTRIES); + } + for _ in 0..10_000 { + record(&mut misses, 100_001, MAX_ENTRIES); + } + assert_eq!(misses.heap.len(), MAX_ENTRIES); + assert_eq!(misses.positions.len(), MAX_ENTRIES); + let hot = &misses.heap[misses.positions[&key(100_001)]]; + assert_eq!(hot.count - hot.error, 10_000); + assert!(misses.summary().contains("uid100001")); + assert_eq!(misses.summary().lines().count(), 26); + misses.clear(); + assert!(misses.summary().is_empty()); + assert!(misses.positions.is_empty()); + assert_eq!(misses.events, 0); + assert_eq!(misses.replacements, 0); + record(&mut misses, 1, MAX_ENTRIES); + assert_eq!(misses.heap[0].count, 1); + assert_eq!(misses.heap[0].error, 0); + } + + #[test] + fn labels_are_bounded_utf8_and_never_merge_distinct_keys() { + let mut misses = HotMisses::default(); + for id in 0..2 { + misses.record(key(id), |s| write!(s, "{}", "λ".repeat(100_000))); + } + assert_eq!(misses.heap.len(), 2); + for e in &misses.heap { + assert!(e.label.len() <= MAX_LABEL_BYTES); + assert!(e.label.ends_with("...")); + } + let mut label = Label("λ".repeat(256)); + assert!(label.write_str("λ").is_err()); + assert!(label.0.len() <= MAX_LABEL_BYTES); + let context = Some((blake3::hash(b"ctx"), 3)); + misses.record(MissKey { context, ..key(0) }, |s| write!(s, "context")); + misses.record(MissKey { b: Some(0), ..key(0) }, |s| write!(s, "pair")); + misses.record(MissKey { phase: "whnf", ..key(0) }, |s| write!(s, "phase")); + assert_eq!(misses.heap.len(), 5); + } +} diff --git a/crates/kernel/src/perf/same_head.rs b/crates/kernel/src/perf/same_head.rs new file mode 100644 index 000000000..da9aa71bf --- /dev/null +++ b/crates/kernel/src/perf/same_head.rs @@ -0,0 +1,383 @@ +//! Diagnostic-only accounting of actual same-head conversion attempts. +//! +//! `IX_SAME_HEAD_PROFILE=1` records completed attempts by outcome and head. +//! Inclusive fuel overlaps for nested attempts; exclusive fuel subtracts +//! nested attempts, and root fuel counts each charged tick at most once. +//! Failed-attempt fuel is not necessarily all avoidable: even an abandoned +//! comparison can populate useful completed-result caches. Skipped probes +//! are counted separately and never reported as attempted comparisons. +//! No expression graphs are retained and no checking policy is changed. + +use ix_common::address::Address; + +static ENABLED: crate::EnvFlag = + crate::EnvFlag::new(|| crate::env_var_os("IX_SAME_HEAD_PROFILE").is_some()); + +#[inline] +pub(crate) fn enabled() -> bool { + *ENABLED +} + +#[derive(Clone, Copy, Debug)] +pub(crate) enum Outcome { + Success, + Miss, + FuelAbort, + DepthAbort, + Error, +} + +#[cfg(not(target_os = "zkvm"))] +const OUTCOMES: [&str; 5] = + ["success", "miss", "fuel_abort", "depth_abort", "error"]; + +#[derive(Clone, Copy)] +pub(crate) enum Skip { + Window, + FailureCache, + Backoff, +} + +#[cfg(not(target_os = "zkvm"))] +mod native { + use super::*; + use rustc_hash::FxHashMap; + use std::{cell::RefCell, fmt::Write}; + + const MAX_HEADS: usize = 1024; + + #[derive(Default, Clone, Copy)] + struct Counts { + calls: u64, + inclusive: u64, + exclusive: u64, + roots: u64, + root_fuel: u64, + max: u64, + ge4096: u64, + ge65536: u64, + ge1000000: u64, + } + + impl Counts { + fn record(&mut self, fuel: u64, child_fuel: u64, root: bool) { + self.calls += 1; + self.inclusive = self.inclusive.saturating_add(fuel); + self.exclusive = + self.exclusive.saturating_add(fuel.saturating_sub(child_fuel)); + if root { + self.roots += 1; + self.root_fuel = self.root_fuel.saturating_add(fuel); + } + self.max = self.max.max(fuel); + self.ge4096 += u64::from(fuel >= 4096); + self.ge65536 += u64::from(fuel >= 65536); + self.ge1000000 += u64::from(fuel >= 1_000_000); + } + + fn line(&self, out: &mut String, label: &str, outcome: &str) { + if self.calls == 0 { + return; + } + let _ = writeln!( + out, + " {label} outcome={outcome} calls={} inclusive_fuel={} exclusive_fuel={} roots={} root_fuel={} max_fuel={} ge4096={} ge65536={} ge1000000={}", + self.calls, + self.inclusive, + self.exclusive, + self.roots, + self.root_fuel, + self.max, + self.ge4096, + self.ge65536, + self.ge1000000 + ); + } + } + + struct Frame { + head: Address, + regular: bool, + children: u64, + } + + #[derive(Default)] + pub(super) struct State { + stack: Vec, + totals: [[Counts; OUTCOMES.len()]; 2], + heads: FxHashMap<(Address, bool), [Counts; OUTCOMES.len()]>, + overflow: [Counts; OUTCOMES.len()], + skips: [[u64; 3]; 2], + max_depth: usize, + accounting_errors: u64, + root_traces: usize, + } + + fn class(regular: bool) -> &'static str { + if regular { "regular" } else { "non_regular" } + } + + impl State { + pub(super) fn begin(&mut self, head: &Address, regular: bool) -> usize { + let ticket = self.stack.len(); + self.stack.push(Frame { head: head.clone(), regular, children: 0 }); + self.max_depth = self.max_depth.max(self.stack.len()); + ticket + } + + pub(super) fn finish( + &mut self, + ticket: usize, + fuel: u64, + outcome: Outcome, + ) { + let frame = self.stack.pop().expect("paired diagnostic begin/finish"); + assert_eq!(ticket, self.stack.len(), "same-head diagnostic stack order"); + let root = self.stack.is_empty(); + self.accounting_errors += u64::from(frame.children > fuel); + if let Some(parent) = self.stack.last_mut() { + parent.children = parent.children.saturating_add(fuel); + } + let outcome = outcome as usize; + self.totals[usize::from(frame.regular)][outcome].record( + fuel, + frame.children, + root, + ); + let key = (frame.head, frame.regular); + if self.heads.len() < MAX_HEADS || self.heads.contains_key(&key) { + self.heads.entry(key).or_default()[outcome].record( + fuel, + frame.children, + root, + ); + } else { + // Keep global accounting exact even when per-head attribution fills. + self.overflow[outcome].record(fuel, frame.children, root); + } + } + + pub(super) fn skip(&mut self, regular: bool, reason: Skip) { + self.skips[usize::from(regular)][reason as usize] += 1; + } + + pub(super) fn take_root_trace( + &mut self, + ticket: Option, + fuel: u64, + ) -> Option { + if ticket != Some(0) || fuel < 65_536 || self.root_traces >= 32 { + return None; + } + self.root_traces += 1; + Some(self.root_traces) + } + + pub(super) fn summary(&self) -> String { + let mut out = format!( + "[same-head-profile] thread-local; inclusive fuel overlaps; root/exclusive fuel do not; active={} max_depth={} tracked_heads={} accounting_errors={}\n", + self.stack.len(), + self.max_depth, + self.heads.len(), + self.accounting_errors + ); + for regular in [true, false] { + let cls = class(regular); + let _ = writeln!( + out, + " {cls} skipped_window={} skipped_failure_cache={} skipped_backoff={}", + self.skips[usize::from(regular)][0], + self.skips[usize::from(regular)][1], + self.skips[usize::from(regular)][2] + ); + for (outcome, c) in + OUTCOMES.iter().zip(self.totals[usize::from(regular)]) + { + c.line(&mut out, cls, outcome); + } + } + let mut heads: Vec<_> = self.heads.iter().collect(); + heads.sort_unstable_by(|(ak, a), (bk, b)| { + let cost = |cs: &[Counts; OUTCOMES.len()]| -> u128 { + cs.iter().map(|c| u128::from(c.exclusive)).sum() + }; + cost(b).cmp(&cost(a)).then_with(|| ak.cmp(bk)) + }); + for ((head, regular), counts) in heads.iter().take(20).copied() { + let label = format!("head=#{} {}", head.hex(), class(*regular)); + for (outcome, c) in OUTCOMES.iter().zip(counts) { + c.line(&mut out, &label, outcome); + } + } + // A mostly nested head can have little exclusive cost but be the root + // that repeatedly admits a large subtree. Show that attribution too. + heads.sort_unstable_by(|(ak, a), (bk, b)| { + let cost = |cs: &[Counts; OUTCOMES.len()]| -> u128 { + cs.iter().map(|c| u128::from(c.root_fuel)).sum() + }; + cost(b).cmp(&cost(a)).then_with(|| ak.cmp(bk)) + }); + for ((head, regular), counts) in heads.into_iter().take(20) { + for (outcome, c) in OUTCOMES.iter().zip(counts) { + if c.roots > 0 { + let _ = writeln!( + out, + " root_head=#{} {} outcome={outcome} roots={} root_fuel={}", + head.hex(), + class(*regular), + c.roots, + c.root_fuel + ); + } + } + } + for (outcome, c) in OUTCOMES.iter().zip(self.overflow) { + c.line(&mut out, "untracked_heads", outcome); + } + out + } + } + + thread_local! { + pub(super) static STATE: RefCell = RefCell::new(State::default()); + } + + #[cfg(test)] + mod tests { + use super::*; + + #[test] + fn nested_fuel_is_not_double_counted_and_skips_are_not_attempts() { + let mut s = State::default(); + let head = Address::hash(b"head"); + let root = s.begin(&head, true); + let child = s.begin(&head, false); + s.finish(child, 70, Outcome::Success); + s.skip(false, Skip::Window); + s.finish(root, 100, Outcome::Miss); + let child = s.totals[0][Outcome::Success as usize]; + let parent = s.totals[1][Outcome::Miss as usize]; + assert_eq!(child.exclusive + parent.exclusive, 100); + assert_eq!(child.root_fuel + parent.root_fuel, 100); + assert_eq!(child.inclusive + parent.inclusive, 170); + assert_eq!(s.skips[0][0], 1); + assert_eq!(child.calls + parent.calls, 2); + assert!(s.stack.is_empty()); + assert_eq!(s.accounting_errors, 0); + } + + #[test] + fn root_cost_is_reported_even_when_exclusive_cost_is_zero() { + let mut s = State::default(); + let head = Address::hash(b"root with nested cost"); + let root = s.begin(&head, true); + let child = s.begin(&Address::hash(b"child"), false); + s.finish(child, 100_000, Outcome::Miss); + s.finish(root, 100_000, Outcome::FuelAbort); + for i in 0u64..21 { + let t = s.begin(&Address::hash(&i.to_le_bytes()), true); + s.finish(t, 1, Outcome::Success); + } + let report = s.summary(); + assert!(!report.contains(&format!("\n head=#{} ", head.hex()))); + assert!(report.contains(&format!( + "root_head=#{} regular outcome=fuel_abort roots=1 root_fuel=100000", + head.hex() + ))); + } + + #[test] + fn root_trace_is_bounded_and_never_reports_nested_or_disabled_probes() { + let mut s = State::default(); + assert_eq!(s.take_root_trace(None, 100_000), None); + assert_eq!(s.take_root_trace(Some(1), 100_000), None); + assert_eq!(s.take_root_trace(Some(0), 65_535), None); + for i in 1..=32 { + assert_eq!(s.take_root_trace(Some(0), 65_536), Some(i)); + } + assert_eq!(s.take_root_trace(Some(0), 100_000), None); + } + + #[test] + fn aborts_have_cost_but_skips_do_not_and_head_storage_is_bounded() { + let mut s = State::default(); + for i in 0..MAX_HEADS + 1 { + let t = s.begin(&Address::hash(&i.to_le_bytes()), true); + s.finish(t, 4096, Outcome::FuelAbort); + } + let t = s.begin(&Address::hash(b"depth"), false); + s.finish(t, 19, Outcome::DepthAbort); + s.skip(true, Skip::FailureCache); + assert_eq!(s.heads.len(), MAX_HEADS); + assert_eq!(s.overflow[Outcome::FuelAbort as usize].calls, 1); + let totals = s.totals[1][Outcome::FuelAbort as usize]; + assert_eq!(totals.root_fuel, 4096 * (MAX_HEADS as u64 + 1)); + assert_eq!(totals.ge4096, MAX_HEADS as u64 + 1); + assert_eq!(totals.ge65536, 0); + assert!(s.summary().contains("outcome=depth_abort calls=1")); + } + } +} + +pub(crate) fn begin(head: &Address, regular: bool) -> Option { + if !enabled() { + return None; + } + #[cfg(not(target_os = "zkvm"))] + return Some(native::STATE.with(|s| s.borrow_mut().begin(head, regular))); + #[cfg(target_os = "zkvm")] + { + let _ = (head, regular); + None + } +} + +pub(crate) fn finish(ticket: Option, fuel: u64, outcome: Outcome) { + #[cfg(not(target_os = "zkvm"))] + if let Some(ticket) = ticket { + native::STATE.with(|s| s.borrow_mut().finish(ticket, fuel, outcome)); + } + #[cfg(target_os = "zkvm")] + let _ = (ticket, fuel, outcome); +} + +pub(crate) fn skip(regular: bool, reason: Skip) { + if enabled() { + #[cfg(not(target_os = "zkvm"))] + native::STATE.with(|s| s.borrow_mut().skip(regular, reason)); + } + #[cfg(target_os = "zkvm")] + let _ = (regular, reason); +} + +/// Admit at most 32 expensive root snapshots per thread-local check. Callers +/// only format the expressions after admission; no expression is retained. +pub(crate) fn take_root_trace( + ticket: Option, + fuel: u64, +) -> Option { + #[cfg(not(target_os = "zkvm"))] + return native::STATE.with(|s| s.borrow_mut().take_root_trace(ticket, fuel)); + #[cfg(target_os = "zkvm")] + { + let _ = (ticket, fuel); + None + } +} + +/// Clear only this thread's diagnostic state, between subject checks. +pub fn reset() { + #[cfg(not(target_os = "zkvm"))] + native::STATE.with(|s| *s.borrow_mut() = native::State::default()); +} + +/// Report only this thread; no effect on checking or the subject JSON. +pub fn summary() -> String { + if !enabled() { + return String::new(); + } + #[cfg(not(target_os = "zkvm"))] + return native::STATE.with(|s| s.borrow().summary()); + #[cfg(target_os = "zkvm")] + String::new() +} diff --git a/crates/kernel/src/subst.rs b/crates/kernel/src/subst.rs index 9779c76ba..271a7b331 100644 --- a/crates/kernel/src/subst.rs +++ b/crates/kernel/src/subst.rs @@ -20,6 +20,12 @@ use super::env::{Addr, InternTable}; use super::expr::{ExprData, FVarId, KExpr}; use super::mode::KernelMode; +#[cfg(test)] +mod menv_tests; + +#[cfg(test)] +mod scratch_tests; + /// When set, log every 100K `subst` (top-level) entries. Substitution is /// called once per `App` in `infer` (plus other sites in whnf / def_eq), /// and each call recursively rebuilds the body; a check that spends @@ -40,8 +46,9 @@ static SUBST_COUNT: std::sync::atomic::AtomicUsize = /// shared sub-expressions within `body` are walked once per depth. /// /// Memoization scratch is borrowed from `env.subst_scratch` to avoid -/// allocating a fresh `FxHashMap` per call. We `mem::take` it out -/// (replacing with an empty placeholder) so the borrow checker lets us +/// allocating a fresh `FxHashMap` per call. `take_for_call` clears entries +/// and adaptively releases persistently sparse, oversized allocations. It +/// leaves an empty placeholder so the borrow checker lets us /// thread `&mut env` and `&mut scratch` separately into `subst_cached`, /// then put it back on the way out. `subst_cached` does not call back /// into `subst`, so there is no risk of recursive scratch use. @@ -62,10 +69,9 @@ pub fn subst( if body.lbr() <= depth { return body.clone(); } - let mut cache = std::mem::take(&mut env.subst_scratch); - cache.clear(); + let mut cache = env.subst_scratch.take_for_call(); let result = subst_cached(env, body, arg, depth, &mut cache); - env.subst_scratch = cache; + env.subst_scratch.restore_after_call(cache); result } @@ -182,7 +188,9 @@ fn subst_cached( ExprData::App(f, x, _) => { let f2 = subst_cached(env, f, arg, depth, cache); let x2 = subst_cached(env, x, arg, depth, cache); - KExpr::app(f2, x2) + let r = env.intern_app(&f2, &x2); + cache.insert(key, r.clone()); + return r; }, ExprData::Lam(name, bi, ty, inner, _) => { @@ -194,7 +202,9 @@ fn subst_cached( ExprData::All(name, bi, ty, inner, _) => { let ty2 = subst_cached(env, ty, arg, depth, cache); let inner2 = subst_cached(env, inner, arg, depth + 1, cache); - KExpr::all(name.clone(), bi.clone(), ty2, inner2) + let r = env.intern_all(name.clone(), bi.clone(), &ty2, &inner2); + cache.insert(key, r.clone()); + return r; }, ExprData::Let(name, ty, val, inner, nd, _) => { @@ -244,13 +254,12 @@ pub fn simul_subst( if body.lbr() <= depth { return body.clone(); } - // See `subst` for the mem::take/restore pattern. `simul_subst_cached` + // See `subst` for the take/restore pattern. `simul_subst_cached` // does not call into `subst`/`simul_subst`, so it is safe to share the // single `subst_scratch` between them. - let mut cache = std::mem::take(&mut env.subst_scratch); - cache.clear(); + let mut cache = env.subst_scratch.take_for_call(); let result = simul_subst_cached(env, body, substs, depth, &mut cache); - env.subst_scratch = cache; + env.subst_scratch.restore_after_call(cache); result } @@ -293,7 +302,9 @@ fn simul_subst_cached( ExprData::App(f, x, _) => { let f2 = simul_subst_cached(env, f, substs, depth, cache); let x2 = simul_subst_cached(env, x, substs, depth, cache); - KExpr::app(f2, x2) + let r = env.intern_app(&f2, &x2); + cache.insert(key, r.clone()); + return r; }, ExprData::Lam(name, bi, ty, inner, _) => { @@ -305,7 +316,9 @@ fn simul_subst_cached( ExprData::All(name, bi, ty, inner, _) => { let ty2 = simul_subst_cached(env, ty, substs, depth, cache); let inner2 = simul_subst_cached(env, inner, substs, depth + 1, cache); - KExpr::all(name.clone(), bi.clone(), ty2, inner2) + let r = env.intern_all(name.clone(), bi.clone(), &ty2, &inner2); + cache.insert(key, r.clone()); + return r; }, ExprData::Let(name, ty, val, inner, nd, _) => { @@ -355,10 +368,9 @@ pub fn lift( // buffer keeps both available simultaneously. `lift_cached` does not // call back into `lift`/`subst`/`simul_subst`, so the scratch is safe // to share across calls without nested-borrow risk. - let mut cache = std::mem::take(&mut env.lift_scratch); - cache.clear(); + let mut cache = env.lift_scratch.take_for_call(); let result = lift_cached(env, e, shift, cutoff, &mut cache); - env.lift_scratch = cache; + env.lift_scratch.restore_after_call(cache); result } @@ -448,7 +460,9 @@ fn lift_cached( ExprData::App(f, x, _) => { let f2 = lift_cached(env, f, shift, cutoff, cache); let x2 = lift_cached(env, x, shift, cutoff, cache); - KExpr::app(f2, x2) + let r = env.intern_app(&f2, &x2); + cache.insert(key, r.clone()); + return r; }, ExprData::Lam(name, bi, ty, body, _) => { @@ -460,7 +474,9 @@ fn lift_cached( ExprData::All(name, bi, ty, body, _) => { let ty2 = lift_cached(env, ty, shift, cutoff, cache); let body2 = lift_cached(env, body, shift, cutoff + 1, cache); - KExpr::all(name.clone(), bi.clone(), ty2, body2) + let r = env.intern_all(name.clone(), bi.clone(), &ty2, &body2); + cache.insert(key, r.clone()); + return r; }, ExprData::Let(name, ty, val, body, nd, _) => { @@ -526,13 +542,27 @@ impl Clo { struct MEnvNode { head: Arc>, - tail: MEnv, + tail: Option>>, + /// The ancestor `jump_len` ordinary tail steps away, or `None` when + /// those steps reach the empty environment. Spans have size 2^k - 1. + jump: Option>>, + jump_len: u64, } -/// Persistent cons-list environment: O(1) push with structural sharing -/// across the closures captured at each binder. `len` is carried on the -/// handle — recomputing it per suffix was a measured cost on the IxVM -/// port of this machine. +/// Persistent skew-binary random-access environment: O(1) push and +/// O(log n) lookup, with the most recent binding still O(1). +/// +/// Ordinary tails retain the original cons-list meaning. A jump skips a +/// complete preorder block of size 2^k - 1; following jumps partitions the +/// list into increasing blocks, with only the first two allowed equal. +/// Prepending merges two equal first blocks with the new head, or adds a +/// singleton block. Both cases need one node and constant work, even when +/// branching from an old snapshot. Lookup skips whole blocks or takes an +/// ordinary tail to descend into one, without materializing any closures. +/// +/// `len` lives on the handle, not on each tail. Compared with the original +/// cons node this adds one pointer-sized field on 64-bit hosts, rather than +/// a separate allocation or a logarithmic jump table at every binder. pub(crate) struct MEnv { node: Option>>, len: u64, @@ -555,20 +585,50 @@ impl MEnv { } pub(crate) fn push(&self, c: Arc>) -> Self { + let len = self.len.checked_add(1).expect("MEnv length overflow"); + let (jump, jump_len) = if let Some(first) = &self.node + && let Some(second) = &first.jump + && first.jump_len == second.jump_len + { + // New head + two equal blocks. The span cannot overflow: both + // blocks belong to `self`, whose length was checked above. + (second.jump.clone(), 1 + 2 * first.jump_len) + } else { + (self.node.clone(), 1) + }; MEnv { - node: Some(Arc::new(MEnvNode { head: c, tail: self.clone() })), - len: self.len + 1, + node: Some(Arc::new(MEnvNode { + head: c, + tail: self.node.clone(), + jump, + jump_len, + })), + len, } } - /// O(i) cons-list walk; `i` must be `< self.len()`. Machine variable - /// lookups are typically near the front (recently pushed args). + /// Return the same closure as `i` ordinary tail steps, in O(log n). + /// `i` must be `< self.len()`; no closure is forced or copied here. pub(crate) fn get(&self, i: u64) -> &Arc> { let mut node = self.node.as_ref().expect("MEnv::get out of range"); + if i == 0 { + return &node.head; + } let mut i = i; + // Recent bindings are common. Once the remaining offset is tiny, + // ordinary tails avoid a jump-size branch at every visited node. + while i >= 8 { + if node.jump_len <= i { + i -= node.jump_len; + node = node.jump.as_ref().expect("MEnv::get out of range"); + } else { + i -= 1; + node = node.tail.as_ref().expect("MEnv::get out of range"); + } + } while i > 0 { - node = node.tail.node.as_ref().expect("MEnv::get out of range"); i -= 1; + node = node.tail.as_ref().expect("MEnv::get out of range"); } &node.head } @@ -658,7 +718,9 @@ fn clo_subst_cached( ExprData::App(f, x, _) => { let f2 = clo_subst_cached(intern, f, env, depth, cache); let x2 = clo_subst_cached(intern, x, env, depth, cache); - KExpr::app(f2, x2) + let r = intern.intern_app(&f2, &x2); + cache.insert(key, r.clone()); + return r; }, ExprData::Lam(name, bi, ty, inner, _) => { @@ -670,7 +732,9 @@ fn clo_subst_cached( ExprData::All(name, bi, ty, inner, _) => { let ty2 = clo_subst_cached(intern, ty, env, depth, cache); let inner2 = clo_subst_cached(intern, inner, env, depth + 1, cache); - KExpr::all(name.clone(), bi.clone(), ty2, inner2) + let r = intern.intern_all(name.clone(), bi.clone(), &ty2, &inner2); + cache.insert(key, r.clone()); + return r; }, ExprData::Let(name, ty, val, inner, nd, _) => { @@ -775,7 +839,7 @@ pub fn cheap_beta_reduce( if head.lbr() == 0 { let mut result = head; for arg in &args[i..] { - result = env.intern_expr(KExpr::app(result, arg.clone())); + result = env.intern_app(&result, arg); } return result; } @@ -791,7 +855,7 @@ pub fn cheap_beta_reduce( let chosen_idx = i - (k as usize) - 1; let mut result = args[chosen_idx].clone(); for arg in &args[i..] { - result = env.intern_expr(KExpr::app(result, arg.clone())); + result = env.intern_app(&result, arg); } return result; } @@ -832,10 +896,9 @@ pub fn instantiate_rev( // `subst`/`simul_subst`). `instantiate_rev_cached` does not call back // into subst/simul_subst/lift, so the scratch is safe to share across // top-level calls without nested-borrow risk. - let mut cache = std::mem::take(&mut env.subst_scratch); - cache.clear(); + let mut cache = env.subst_scratch.take_for_call(); let result = instantiate_rev_cached(env, body, fvars, 0, &mut cache); - env.subst_scratch = cache; + env.subst_scratch.restore_after_call(cache); result } @@ -889,7 +952,9 @@ fn instantiate_rev_cached( ExprData::App(f, x, _) => { let f2 = instantiate_rev_cached(env, f, fvars, depth, cache); let x2 = instantiate_rev_cached(env, x, fvars, depth, cache); - KExpr::app(f2, x2) + let r = env.intern_app(&f2, &x2); + cache.insert(key, r.clone()); + return r; }, ExprData::Lam(name, bi, ty, inner, _) => { @@ -901,7 +966,9 @@ fn instantiate_rev_cached( ExprData::All(name, bi, ty, inner, _) => { let ty2 = instantiate_rev_cached(env, ty, fvars, depth, cache); let inner2 = instantiate_rev_cached(env, inner, fvars, depth + 1, cache); - KExpr::all(name.clone(), bi.clone(), ty2, inner2) + let r = env.intern_all(name.clone(), bi.clone(), &ty2, &inner2); + cache.insert(key, r.clone()); + return r; }, ExprData::Let(name, ty, val, inner, nd, _) => { @@ -964,11 +1031,10 @@ pub fn abstract_fvars( pos.insert(*fv, (fvars.len() - 1 - i) as u64); } - let mut cache = std::mem::take(&mut env.subst_scratch); - cache.clear(); + let mut cache = env.subst_scratch.take_for_call(); let n = fvars.len() as u64; let result = abstract_fvars_cached(env, body, &pos, n, 0, &mut cache); - env.subst_scratch = cache; + env.subst_scratch.restore_after_call(cache); result } @@ -1024,7 +1090,9 @@ fn abstract_fvars_cached( ExprData::App(f, x, _) => { let f2 = abstract_fvars_cached(env, f, pos, n, depth, cache); let x2 = abstract_fvars_cached(env, x, pos, n, depth, cache); - KExpr::app(f2, x2) + let r = env.intern_app(&f2, &x2); + cache.insert(key, r.clone()); + return r; }, ExprData::Lam(name, bi, ty, inner, _) => { @@ -1036,7 +1104,9 @@ fn abstract_fvars_cached( ExprData::All(name, bi, ty, inner, _) => { let ty2 = abstract_fvars_cached(env, ty, pos, n, depth, cache); let inner2 = abstract_fvars_cached(env, inner, pos, n, depth + 1, cache); - KExpr::all(name.clone(), bi.clone(), ty2, inner2) + let r = env.intern_all(name.clone(), bi.clone(), &ty2, &inner2); + cache.insert(key, r.clone()); + return r; }, ExprData::Let(name, ty, val, inner, nd, _) => { diff --git a/crates/kernel/src/subst/menv_tests.rs b/crates/kernel/src/subst/menv_tests.rs new file mode 100644 index 000000000..44763f193 --- /dev/null +++ b/crates/kernel/src/subst/menv_tests.rs @@ -0,0 +1,389 @@ +//! Persistent-environment regressions and an opt-in paired microbenchmark. + +// Match the production closure API, including its worker-private Arc use; +// replacing these with Rc would change what the reference benchmark measures. +#![allow(clippy::arc_with_non_send_sync)] + +use std::hint::black_box; +use std::mem::size_of; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use super::{Clo, MEnv, MEnvNode, clo_readback}; +use crate::env::{InternTable, KEnv}; +use crate::expr::KExpr; +use crate::level::KUniv; +use crate::mode::Anon; +use crate::tc::TypeChecker; + +type Closure = Arc>; + +/// The original representation, retained only as a benchmark/reference. +#[derive(Clone)] +struct LinearEnv { + node: Option>, + len: u64, +} + +struct LinearNode { + head: Closure, + tail: LinearEnv, +} + +trait TestEnv: Clone { + fn empty() -> Self; + fn push(&self, c: Closure) -> Self; + fn get(&self, i: u64) -> &Closure; +} + +impl TestEnv for LinearEnv { + fn empty() -> Self { + Self { node: None, len: 0 } + } + + fn push(&self, c: Closure) -> Self { + Self { + node: Some(Arc::new(LinearNode { head: c, tail: self.clone() })), + len: self.len + 1, + } + } + + fn get(&self, mut i: u64) -> &Closure { + let mut node = self.node.as_ref().expect("linear lookup out of range"); + while i > 0 { + node = node.tail.node.as_ref().expect("linear lookup out of range"); + i -= 1; + } + &node.head + } +} + +impl TestEnv for MEnv { + fn empty() -> Self { + Self::empty() + } + + fn push(&self, c: Closure) -> Self { + self.push(c) + } + + fn get(&self, i: u64) -> &Closure { + self.get(i) + } +} + +fn closure(tag: u64) -> Closure { + Arc::new(Clo::closed(KExpr::var(tag, ()))) +} + +#[test] +fn menv_lookup_preserves_every_snapshot() { + let entries: Vec<_> = (0..256).map(closure).collect(); + let mut versions = vec![MEnv::empty()]; + for entry in &entries { + let next = versions.last().unwrap().push(entry.clone()); + versions.push(next); + } + for (len, env) in versions.iter().enumerate() { + assert_eq!(env.len(), u64::try_from(len).unwrap()); + for (i, expected) in entries[..len].iter().rev().enumerate() { + assert!(Arc::ptr_eq(env.get(u64::try_from(i).unwrap()), expected)); + } + } + // Release newest first so even the linear reference has shallow drops. + while versions.pop().is_some() {} +} + +#[test] +fn menv_persistent_branches_match_vector_model() { + let mut versions = vec![(MEnv::empty(), Vec::::new())]; + let mut rng = 1u64; + for tag in 0..2_000 { + rng = rng.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1); + let parent = + usize::try_from(rng % u64::try_from(versions.len()).unwrap()).unwrap(); + let (env, entries) = &versions[parent]; + let value = closure(tag); + let next = env.push(value.clone()); + let mut expected = entries.clone(); + expected.push(value); + for (i, entry) in expected.iter().rev().enumerate() { + assert!(Arc::ptr_eq(next.get(u64::try_from(i).unwrap()), entry)); + } + versions.push((next, expected)); + } + while versions.pop().is_some() {} +} + +#[test] +fn menv_jump_spans_match_ordinary_ancestors() { + let value = closure(0); + let mut versions = build_versions::>(4_096, &value); + for (len, env) in versions.iter().enumerate().skip(1) { + let node = env.node.as_ref().unwrap(); + let span = usize::try_from(node.jump_len).unwrap(); + assert!((span + 1).is_power_of_two()); + assert!(span <= len); + match (&node.jump, &versions[len - span].node) { + (Some(actual), Some(expected)) => assert!(Arc::ptr_eq(actual, expected)), + (None, None) => {}, + _ => panic!("jump must reach the same ancestor as its span"), + } + let mut block = node; + let mut position = 0; + while let Some(next) = &block.jump { + assert!(block.jump_len <= next.jump_len); + if block.jump_len == next.jump_len { + assert_eq!(position, 0, "only the first two blocks may be equal"); + } + block = next; + position += 1; + } + } + while versions.pop().is_some() {} +} + +#[test] +fn menv_deep_lookups_match_linear_model_with_logarithmic_hops() { + let entries: Vec<_> = (0..16_385).map(closure).collect(); + let mut versions = vec![MEnv::empty()]; + for value in &entries { + versions.push(versions.last().unwrap().push(value.clone())); + } + for len in [63usize, 64, 65, 1_023, 1_024, 1_025, 16_383, 16_384, 16_385] { + let env = &versions[len]; + for (index, expected) in entries[..len].iter().rev().enumerate() { + let index = u64::try_from(index).unwrap(); + assert!(Arc::ptr_eq(env.get(index), expected)); + // Check the structural work bound without adding instrumentation + // to the production lookup hot path. + let mut node = env.node.as_ref().unwrap(); + let mut remaining = index; + let mut hops = 0; + while remaining >= 8 { + if node.jump_len <= remaining { + remaining -= node.jump_len; + node = node.jump.as_ref().unwrap(); + } else { + remaining -= 1; + node = node.tail.as_ref().unwrap(); + } + hops += 1; + } + while remaining > 0 { + remaining -= 1; + node = node.tail.as_ref().unwrap(); + hops += 1; + } + assert!(Arc::ptr_eq(&node.head, expected)); + assert!(hops <= 2 * (env.len().ilog2() + 1)); + } + } + while versions.pop().is_some() {} +} + +#[test] +fn menv_readback_preserves_captured_environments_and_lifting() { + let mut env = MEnv::empty().push(closure(3)); + let mut versions = vec![env.clone()]; + for _ in 0..127 { + // Each new entry denotes the previous entry in its captured snapshot, + // not Var(0) in the eventual caller's environment. + env = env.push(Arc::new(Clo::new(KExpr::var(0, ()), env.clone()))); + versions.push(env.clone()); + } + let ty = KExpr::sort(KUniv::zero()); + let body = KExpr::lam( + (), + (), + ty.clone(), + KExpr::app( + KExpr::var(1, ()), + KExpr::app(KExpr::var(env.len(), ()), KExpr::var(env.len() + 5, ())), + ), + ); + let c = Clo::new(body, env); + let expected = KExpr::lam( + (), + (), + ty, + KExpr::app( + KExpr::var(4, ()), + KExpr::app(KExpr::var(4, ()), KExpr::var(5, ())), + ), + ); + let mut intern = InternTable::new(); + let result = clo_readback(&mut intern, &c); + assert_eq!(result, expected); + assert!(result.ptr_eq(&clo_readback(&mut intern, &c))); + drop(c); + while versions.pop().is_some() {} +} + +#[test] +fn menv_deep_beta_machine_selects_original_arguments() { + for depth in [7u64, 8, 15, 63, 64, 65, 127, 128, 129] { + for selected in [0, depth / 2, depth - 1] { + let mut env = KEnv::::new(); + let ty = KExpr::sort(KUniv::zero()); + // Open arguments let us distinguish every position and also verify + // that ambient variables are not captured by the machine binders. + let args: Vec<_> = (0..depth).map(|i| KExpr::var(i + 10, ())).collect(); + let mut expr = KExpr::var(depth - 1 - selected, ()); + for _ in 0..depth { + expr = KExpr::lam((), (), ty.clone(), expr); + } + for arg in &args { + expr = KExpr::app(expr, arg.clone()); + } + let mut tc = TypeChecker::new(&mut env); + let result = tc.whnf(&expr).unwrap(); + assert_eq!(result, args[usize::try_from(selected).unwrap()]); + } + } +} + +#[test] +#[should_panic(expected = "MEnv::get out of range")] +fn menv_empty_lookup_panics() { + let _ = MEnv::::empty().get(0); +} + +#[test] +#[should_panic(expected = "MEnv::get out of range")] +fn menv_len_lookup_panics() { + let env = MEnv::empty().push(closure(0)); + let _ = env.get(env.len()); +} + +#[test] +#[should_panic(expected = "MEnv::get out of range")] +fn menv_large_out_of_range_lookup_panics() { + let env = MEnv::empty().push(closure(0)); + let _ = env.get(u64::MAX); +} + +fn build_versions(depth: u64, value: &Closure) -> Vec { + let mut versions = Vec::with_capacity(usize::try_from(depth + 1).unwrap()); + versions.push(E::empty()); + for _ in 0..depth { + let next = versions.last().unwrap().push(value.clone()); + versions.push(next); + } + versions +} + +fn measure_lookup(env: &E, indices: &[u64], count: u64) -> f64 { + let start = Instant::now(); + let mut indices = indices.iter().cycle(); + for _ in 0..count { + black_box(black_box(env).get(black_box(*indices.next().unwrap()))); + } + start.elapsed().as_secs_f64() * 1e9 / f64::from(u32::try_from(count).unwrap()) +} + +fn median(mut samples: Vec) -> f64 { + samples.sort_by(f64::total_cmp); + samples[samples.len() / 2] +} + +fn build_sample( + depth: u64, + value: &Closure, +) -> (Duration, Duration) { + let start = Instant::now(); + let mut versions = black_box(build_versions::(depth, value)); + let build = start.elapsed(); + let start = Instant::now(); + while versions.pop().is_some() {} + (build, start.elapsed()) +} + +/// No timing assertions: performance results depend on host load/allocator. +/// Construction includes retaining each version; destruction releases newest +/// first. Closure allocation is excluded, to isolate the environment itself. +#[test] +#[ignore = "release microbenchmark: --release menv_lookup_benchmark -- --ignored --nocapture"] +fn menv_lookup_benchmark() { + eprintln!( + "MEnv node payload={} B; linear={} B; handle={} B; one node allocation/push (excludes Arc header/allocator rounding)", + size_of::>(), + size_of::(), + size_of::>(), + ); + let value = closure(0); + for depth in [8u64, 64, 1_024, 16_384] { + let mut current = build_versions::>(depth, &value); + let mut linear = build_versions::(depth, &value); + let uniform: Vec<_> = (0..256u64) + .map(|i| i.wrapping_mul(6_364_136_223_846_793_005) % depth) + .collect(); + for (pattern, indices) in [ + ("front", vec![0]), + ("near", (0..8).collect()), + ("oldest", vec![depth - 1]), + ("uniform", uniform), + ] { + let count = if pattern == "front" || pattern == "near" { + 500_000 + } else { + (4_000_000 / depth).clamp(4_096, 500_000) + }; + let mut current_samples = Vec::new(); + let mut linear_samples = Vec::new(); + for round in 0..7 { + if round % 2 == 0 { + current_samples.push(measure_lookup( + current.last().unwrap(), + &indices, + count, + )); + linear_samples.push(measure_lookup( + linear.last().unwrap(), + &indices, + count, + )); + } else { + linear_samples.push(measure_lookup( + linear.last().unwrap(), + &indices, + count, + )); + current_samples.push(measure_lookup( + current.last().unwrap(), + &indices, + count, + )); + } + } + let current_ns = median(current_samples); + let linear_ns = median(linear_samples); + eprintln!( + "depth={depth:5} {pattern:7}: current={current_ns:10.2} ns/lookup linear={linear_ns:10.2} ns/lookup ratio={:.3}", + current_ns / linear_ns + ); + } + while current.pop().is_some() {} + while linear.pop().is_some() {} + } + for depth in [1_024u32, 16_384] { + let mut build = [Vec::new(), Vec::new()]; + let mut drop = [Vec::new(), Vec::new()]; + for round in 0..7 { + for index in [round % 2, 1 - round % 2] { + let (built, dropped) = if index == 0 { + build_sample::>(u64::from(depth), &value) + } else { + build_sample::(u64::from(depth), &value) + }; + build[index].push(built.as_secs_f64() * 1e9 / f64::from(depth)); + drop[index].push(dropped.as_secs_f64() * 1e9 / f64::from(depth)); + } + } + let [current_build, linear_build] = build.map(median); + let [current_drop, linear_drop] = drop.map(median); + eprintln!( + "depth={depth:5} build: current={current_build:.2} linear={linear_build:.2} ns/binding; drop: current={current_drop:.2} linear={linear_drop:.2} ns/binding" + ); + } +} diff --git a/crates/kernel/src/subst/scratch_tests.rs b/crates/kernel/src/subst/scratch_tests.rs new file mode 100644 index 000000000..db0194bc0 --- /dev/null +++ b/crates/kernel/src/subst/scratch_tests.rs @@ -0,0 +1,364 @@ +//! Scratch allocation regressions: results remain call-local while capacity +//! adapts to a large binder-opening traversal followed by tiny substitutions. + +use super::*; +use crate::env::KEnv; +use crate::id::KId; +use crate::level::KUniv; +use crate::mode::{Anon, Meta}; +use crate::tc::TypeChecker; +use ix_common::address::Address; +use ix_common::env::{BinderInfo, DataValue, Name}; + +fn meta_name(s: &str) -> M::MField { + M::meta_field(Name::str(Name::anon(), s.to_owned())) +} + +// A balanced tree has many distinct open nodes without deep recursion in +// construction, substitution, or destruction. Every leaf must be visited. +fn wide_open_tree( + intern: &mut InternTable, + leaves: u64, +) -> KExpr { + assert!(leaves.is_power_of_two()); + let mut layer: Vec<_> = (0..leaves) + .map(|i| intern.intern_expr(KExpr::var(i, meta_name::("v")))) + .collect(); + while layer.len() > 1 { + layer = layer + .as_chunks::<2>() + .0 + .iter() + .map(|pair| intern.intern_app(&pair[0], &pair[1])) + .collect(); + } + layer.pop().unwrap() +} + +fn large_then_small() { + let mut intern = InternTable::::new(); + let body = wide_open_tree(&mut intern, 8_192); + let fv = intern.intern_expr(KExpr::fvar(FVarId(0), meta_name::("bulk"))); + let _ = instantiate_rev(&mut intern, &body, &[fv]); + let high_water = intern.subst_scratch.capacity(); + assert!(high_water > 4_096); + + let var = intern.intern_expr(KExpr::var(0, meta_name::("v"))); + for i in 1..=5 { + let arg = + intern.intern_expr(KExpr::fvar(FVarId(i), meta_name::("small"))); + let result = subst(&mut intern, &var, &arg, 0); + assert!(result.ptr_eq(&arg), "memo entries leaked between calls"); + } + assert!( + intern.subst_scratch.capacity() <= 4_096, + "tiny substitutions retained the bulk table: high_water={high_water}, current={}", + intern.subst_scratch.capacity() + ); +} + +#[test] +fn scratch_releases_bulk_capacity_after_tiny_substitutions() { + large_then_small::(); + large_then_small::(); +} + +fn lift_large_then_small() { + let mut intern = InternTable::::new(); + let body = wide_open_tree(&mut intern, 8_192); + let _ = lift(&mut intern, &body, 1, 0); + assert!(intern.lift_scratch.capacity() > 4_096); + let var = intern.intern_expr(KExpr::var(0, meta_name::("v"))); + for shift in 1..=5 { + let result = lift(&mut intern, &var, shift, 0); + assert!(matches!(result.data(), ExprData::Var(i, ..) if *i == shift)); + } + assert!(intern.lift_scratch.capacity() <= 4_096); +} + +#[test] +fn scratch_lift_adapts_independently() { + lift_large_then_small::(); + lift_large_then_small::(); +} + +// Small, shared expressions containing every binder/child position touched +// by substitution. Metadata is deliberately nonempty in Meta mode. +fn expressions(intern: &mut InternTable) -> Vec> { + let mut nodes: Vec<_> = (0..4) + .map(|i| intern.intern_expr(KExpr::var(i, meta_name::("var")))) + .collect(); + for i in 0..4 { + nodes + .push(intern.intern_expr(KExpr::fvar(FVarId(i), meta_name::("free")))); + } + let metadata = M::meta_field(vec![vec![( + Name::str(Name::anon(), "tag".to_owned()), + DataValue::OfString("scratch differential".to_owned()), + )]]); + let ty = + intern.intern_expr(KExpr::sort_mdata(KUniv::zero(), metadata.clone())); + nodes.push(ty.clone()); + for i in 0..40 { + let a = nodes[i % nodes.len()].clone(); + let b = nodes[(i * 7 + 3) % nodes.len()].clone(); + let node = match i % 5 { + 0 => KExpr::app_mdata(a.clone(), a, metadata.clone()), + 1 => KExpr::lam( + meta_name::("lambda"), + M::meta_field(BinderInfo::Implicit), + a, + b, + ), + 2 => KExpr::all( + meta_name::("forall"), + M::meta_field(BinderInfo::InstImplicit), + a, + b, + ), + 3 => KExpr::let_(meta_name::("let"), ty.clone(), a, b, i % 2 == 0), + _ => KExpr::prj( + KId::new(Address::hash(b"scratch struct"), meta_name::("S")), + 0, + a, + ), + }; + nodes.push(intern.intern_expr(node)); + } + nodes +} + +fn differential() { + let mut intern = InternTable::::new(); + let bulk = wide_open_tree(&mut intern, 8_192); + let _ = lift(&mut intern, &bulk, 1, 0); + let fv = intern.intern_expr(KExpr::fvar(FVarId(0), meta_name::("free"))); + let _ = instantiate_rev(&mut intern, &bulk, &[fv]); + let nodes = expressions(&mut intern); + + for round in 0..4u64 { + let fvars: Vec<_> = (0..3) + .map(|i| { + intern.intern_expr(KExpr::fvar( + FVarId((i + round) % 4), + meta_name::("free"), + )) + }) + .collect(); + // Open replacements force nested lift calls while subst owns its memo. + let replacements = + [nodes[usize::try_from(round).unwrap()].clone(), fvars[0].clone()]; + let ids: Vec<_> = (0..3).map(|i| FVarId((i + round) % 4)).collect(); + let pos: FxHashMap<_, _> = ids + .iter() + .rev() + .enumerate() + .map(|(i, id)| (*id, u64::try_from(i).unwrap())) + .collect(); + for body in &nodes { + for op in 0..5 { + // Same trusted traversal with an independent, empty memo: no old + // result, capacity, replacement list, binder depth, or shift can leak + // from the adaptive cache into this reference result. + let mut fresh = FxHashMap::default(); + let expected = match op { + 0 => { + subst_cached(&mut intern, body, &replacements[0], round, &mut fresh) + }, + 1 => simul_subst_cached( + &mut intern, + body, + &replacements, + round, + &mut fresh, + ), + 2 => lift_cached(&mut intern, body, round + 1, round, &mut fresh), + 3 => instantiate_rev_cached(&mut intern, body, &fvars, 0, &mut fresh), + _ => abstract_fvars_cached(&mut intern, body, &pos, 3, 0, &mut fresh), + }; + let actual = match op { + 0 => subst(&mut intern, body, &replacements[0], round), + 1 => simul_subst(&mut intern, body, &replacements, round), + 2 => lift(&mut intern, body, round + 1, round), + 3 => instantiate_rev(&mut intern, body, &fvars), + _ => abstract_fvars(&mut intern, body, &ids), + }; + // Pointer equality in the SAME live interner also checks metadata + // and annotations, unlike KExpr's structural PartialEq alone. + assert!(actual.ptr_eq(&expected), "op={op} round={round}"); + } + } + } + assert!(intern.subst_scratch.capacity() <= 4_096); + assert!(intern.lift_scratch.capacity() <= 4_096); +} + +#[test] +fn scratch_mixed_operations_match_fresh_memos() { + differential::(); + differential::(); +} + +fn checked_application(polluted: bool) -> u64 { + let mut env = KEnv::::new(); + if polluted { + let bulk = wide_open_tree(&mut env.intern, 8_192); + let _ = lift(&mut env.intern, &bulk, 1, 0); + let fv = + env.intern.intern_expr(KExpr::fvar(FVarId(100), meta_name::("seed"))); + let _ = instantiate_rev(&mut env.intern, &bulk, &[fv]); + } + // (fun (A : Sort 1) => A) (Sort 0) is a closed, well-typed application. + let sort0 = KExpr::sort(KUniv::zero()); + let sort1 = KExpr::sort(KUniv::succ(KUniv::zero())); + let identity = KExpr::lam( + meta_name::("A"), + M::meta_field(BinderInfo::Default), + sort1.clone(), + KExpr::var(0, meta_name::("A")), + ); + let app = KExpr::app(identity, sort0.clone()); + let mut tc = TypeChecker::new(&mut env); + let ty = tc.infer(&app).unwrap(); + assert_eq!(ty, sort1); + let reduced = tc.whnf(&app).unwrap(); + assert_eq!(reduced, sort0); + tc.fuel_used() +} + +#[test] +fn scratch_history_does_not_change_kernel_results_or_fuel() { + assert_eq!( + checked_application::(false), + checked_application::(true) + ); + assert_eq!( + checked_application::(false), + checked_application::(true) + ); +} + +// Pre-adaptive take/clear/restore policy, kept only for a paired benchmark. +// The traversal, interner, operand shapes, and call-local memo keys are the +// same. All benchmark arguments are closed, so nested lift is a no-op. +fn legacy_subst( + intern: &mut InternTable, + slot: &mut FxHashMap<(Addr, u64), KExpr>, + body: &KExpr, + arg: &KExpr, +) -> KExpr { + // Match the disabled production diagnostic branch, without timing logs. + assert!(!*IX_SUBST_COUNT_LOG); + if body.lbr() == 0 { + return body.clone(); + } + let mut cache = std::mem::take(slot); + cache.clear(); + let result = subst_cached(intern, body, arg, 0, &mut cache); + *slot = cache; + result +} + +#[derive(Clone, Copy, Debug)] +enum Workload { + Small, + LargeThenSmall, + Dense, + Alternating, +} + +fn scratch_sample( + workload: Workload, + adaptive: bool, +) -> (std::time::Duration, usize, usize) { + use std::hint::black_box; + use std::time::Instant; + + let mut intern = InternTable::::new(); + let leaves = match workload { + Workload::Small => 32, + Workload::LargeThenSmall => 131_072, + Workload::Dense | Workload::Alternating => 4_096, + }; + let large = wide_open_tree(&mut intern, leaves); + let small = intern.intern_expr(KExpr::var(0, ())); + let arg = intern.intern_expr(KExpr::fvar(FVarId(0), ())); + let mut legacy = FxHashMap::default(); + // Simulate the real preceding bulk binder-opening operation, outside the + // timed loop. Both variants retain all of its canonical nodes identically. + if adaptive { + black_box(instantiate_rev(&mut intern, &large, std::slice::from_ref(&arg))); + } else { + black_box(instantiate_rev_cached( + &mut intern, + &large, + std::slice::from_ref(&arg), + 0, + &mut legacy, + )); + } + let before = + if adaptive { intern.subst_scratch.capacity() } else { legacy.capacity() }; + let iterations = match workload { + Workload::Small => 200_000, + Workload::LargeThenSmall => 5_000, + Workload::Dense => 100, + Workload::Alternating => 200, + }; + let start = Instant::now(); + for i in 0..iterations { + let body = match workload { + Workload::Dense => &large, + Workload::Alternating if i % 2 == 0 => &large, + _ => &small, + }; + let result = if adaptive { + subst(&mut intern, black_box(body), black_box(&arg), 0) + } else { + legacy_subst(&mut intern, &mut legacy, black_box(body), black_box(&arg)) + }; + black_box(result); + } + let elapsed = start.elapsed(); + let after = + if adaptive { intern.subst_scratch.capacity() } else { legacy.capacity() }; + if adaptive && matches!(workload, Workload::LargeThenSmall) { + assert!(after <= 4_096); + } else { + assert_eq!(before, after, "unexpected capacity churn in {workload:?}"); + } + (elapsed, before, after) +} + +#[test] +#[ignore = "manual paired release microbenchmark; no wall-clock assertions"] +fn benchmark_adaptive_scratch() { + for workload in [ + Workload::Small, + Workload::LargeThenSmall, + Workload::Dense, + Workload::Alternating, + ] { + let mut samples = [Vec::new(), Vec::new()]; + let mut capacities = [(0, 0); 2]; + for round in 0..5 { + // Alternate order to reduce warmup/thermal bias. Each sample has a + // fresh interner and scratch; construction and teardown are excluded. + for variant in [round % 2, 1 - round % 2] { + let (elapsed, before, after) = scratch_sample(workload, variant == 1); + samples[variant].push(elapsed); + capacities[variant] = (before, after); + } + } + let [old, new] = samples.map(|mut times| { + times.sort_unstable(); + times[times.len() / 2].as_secs_f64() * 1_000.0 + }); + eprintln!( + "{workload:?}: legacy={old:.3}ms adaptive={new:.3}ms ratio={:.3}; capacities legacy={:?} adaptive={:?}", + new / old, + capacities[0], + capacities[1] + ); + } +} diff --git a/crates/kernel/src/tc.rs b/crates/kernel/src/tc.rs index b0ed4df08..acbff3560 100644 --- a/crates/kernel/src/tc.rs +++ b/crates/kernel/src/tc.rs @@ -25,6 +25,7 @@ use super::ingress::{ use super::lctx::LocalDecl; use super::level::{KUniv, UnivData}; use super::mode::KernelMode; +use super::perf::hot_misses::{HotMisses, MissKey}; use super::primitive::Primitives; use super::subst::{instantiate_rev, lift}; @@ -44,18 +45,16 @@ pub const MAX_DEF_EQ_DEPTH: u32 = 2_000; /// Shared recursive fuel budget, consumed by recursive whnf/infer/isDefEq /// entries and by productive structural-WHNF beta/zeta/iota transitions. -/// lean4lean uses 10,000 with step-indexed recursion; the lean4 C++ kernel -/// uses ~200,000 heartbeats. We use a higher budget than both because this -/// kernel lacks compiled native reduction and checks some large proof terms -/// by interpreting their full expression trees. In particular, BVDecide's -/// generated mutual proofs can legitimately exceed one million recursive -/// kernel steps even after cache hits stop consuming fuel. +/// This is cumulative work per declaration, not a recursion-depth limit; +/// its units are not directly comparable to lean4lean's step-indexed depth +/// allowance or the Lean C++ kernel's heartbeats. /// -/// Mathlib-scale category/algebra proof terms also exceed the old 1.5M budget -/// without hitting the actual `MAX_DEF_EQ_DEPTH` guard. Keep this high enough -/// for legitimate large proofs while retaining the `IX_MAX_REC_FUEL` override -/// for bisecting suspected loops. -pub const MAX_REC_FUEL: u64 = 10_000_000; +/// Large FLT proofs can finish successfully after roughly 54M counted steps +/// with shallow def-eq recursion. Leave headroom for these finite checks; +/// retain the independent depth/WHNF guards and the `IX_MAX_REC_FUEL` +/// override for bounded experiments and bisecting suspected loops. This +/// budget does not replace external time or memory limits. +pub const MAX_REC_FUEL: u64 = 100_000_000; static IX_MAX_REC_FUEL: crate::EnvOptU64 = crate::EnvOptU64::new(|| { crate::env_var("IX_MAX_REC_FUEL").ok().and_then(|s| s.parse().ok()) @@ -64,6 +63,13 @@ static IX_MAX_REC_FUEL: crate::EnvOptU64 = crate::EnvOptU64::new(|| { static IX_HOT_MISSES: crate::EnvFlag = crate::EnvFlag::new(|| crate::env_var("IX_HOT_MISSES").is_ok()); +/// Opt-in, bounded native call stacks at resource guards. Useful with the +/// one-subject helper: ordinary CLI runs do not install a `log` subscriber. +static IX_GUARD_STACKS: crate::EnvFlag = + crate::EnvFlag::new(|| crate::env_var("IX_GUARD_STACKS").is_ok()); +static GUARD_STACK_COUNT: std::sync::atomic::AtomicUsize = + std::sync::atomic::AtomicUsize::new(0); + static IX_HOT_MISS_CTX: crate::EnvFlag = crate::EnvFlag::new(|| crate::env_var("IX_HOT_MISS_CTX").is_ok()); @@ -151,6 +157,14 @@ pub struct TypeChecker<'a, M: KernelMode> { /// cache while projected values are reduced structurally instead of through /// full WHNF. pub cheap_recursion_depth: u32, + /// Avoid recursively starting speculative projection-first comparisons. + pub(crate) in_projection_probe: bool, + /// Leaf conversion inside an application worklist must not start another + /// worklist. The original conversion algorithm remains its fallback. + pub(crate) in_app_congruence: bool, + /// Disable nested binder-batch probes, including throughout a failed + /// probe's ordinary recursive fallback. + pub(crate) in_binder_batch: bool, /// When true, the Bool.true fast-path in is_def_eq fires even on open terms. pub eager_reduce: bool, /// Current def-eq recursion depth. @@ -162,6 +176,14 @@ pub struct TypeChecker<'a, M: KernelMode> { pub def_eq_peak: u32, /// Shared recursive fuel remaining for this constant check. pub rec_fuel: u64, + /// Unspent outer fuel temporarily withheld by same-head probes. Only used + /// to preserve the non-Regular startup window inside a larger Regular + /// slice; it never increases the fuel available to a nested computation. + pub(crate) same_head_fuel_reserve: u64, + /// Per-declaration admission history, not cached conversion facts. + pub(crate) same_head_backoff: super::def_eq::SameHeadBackoff, + /// Bounded non-semantic history for selective dependent-prefix caching. + pub(crate) prefix_admission: super::infer::PrefixAdmission, /// Optional diagnostic label for the current top-level constant. pub debug_label: Option, @@ -173,9 +195,9 @@ pub struct TypeChecker<'a, M: KernelMode> { /// Addresses of constants whose bodies were delta-unfolded during the current /// constant's check. Drained per constant by `record_current_fuel_used`. pub(crate) delta_targets: FxHashSet
, - /// Gated miss sampler for fuel-exhaustion diagnostics. Populated only when - /// `IX_HOT_MISSES=1`, keyed by a compact phase/head/lbr shape. - hot_misses: FxHashMap, + /// Gated, bounded heavy-hitter sampler for fuel-exhaustion diagnostics. + /// Populated only when `IX_HOT_MISSES=1`; never retains expressions. + hot_misses: HotMisses, /// Memoization cache for [`Self::ctx_addr_for_lbr`]. /// @@ -215,15 +237,21 @@ impl<'a, M: KernelMode> TypeChecker<'a, M> { infer_only: false, in_native_reduce: false, cheap_recursion_depth: 0, + in_projection_probe: false, + in_app_congruence: false, + in_binder_batch: false, eager_reduce: false, def_eq_depth: 0, def_eq_trace_depth: 0, def_eq_peak: 0, rec_fuel: max_rec_fuel(), + same_head_fuel_reserve: 0, + same_head_backoff: Default::default(), + prefix_admission: Default::default(), debug_label: None, cur_const: None, delta_targets: FxHashSet::default(), - hot_misses: FxHashMap::default(), + hot_misses: HotMisses::default(), ctx_addr_cache: FxHashMap::default(), lctx: super::lctx::LocalContext::new(), } @@ -746,7 +774,9 @@ impl<'a, M: KernelMode> TypeChecker<'a, M> { ExprData::App(f, a, _) => { let f2 = self.inst_univ_inner(f, us, cache)?; let a2 = self.inst_univ_inner(a, us, cache)?; - KExpr::app(f2, a2) + let r = self.env.intern.intern_app(&f2, &a2); + cache.insert(key, r.clone()); + return Ok(r); }, ExprData::Lam(name, bi, ty, body, _) => { @@ -758,7 +788,10 @@ impl<'a, M: KernelMode> TypeChecker<'a, M> { ExprData::All(name, bi, ty, body, _) => { let ty2 = self.inst_univ_inner(ty, us, cache)?; let body2 = self.inst_univ_inner(body, us, cache)?; - KExpr::all(name.clone(), bi.clone(), ty2, body2) + let r = + self.env.intern.intern_all(name.clone(), bi.clone(), &ty2, &body2); + cache.insert(key, r.clone()); + return Ok(r); }, ExprData::Let(name, ty, val, body, nd, _) => { @@ -835,6 +868,9 @@ impl<'a, M: KernelMode> TypeChecker<'a, M> { self.infer_only = false; self.in_native_reduce = false; self.cheap_recursion_depth = 0; + self.in_projection_probe = false; + self.in_app_congruence = false; + self.in_binder_batch = false; self.eager_reduce = false; self.def_eq_depth = 0; self.def_eq_peak = 0; @@ -848,6 +884,9 @@ impl<'a, M: KernelMode> TypeChecker<'a, M> { self.env.clear_reduction_caches(); } self.rec_fuel = max_rec_fuel(); + self.same_head_fuel_reserve = 0; + self.same_head_backoff = Default::default(); + self.prefix_admission = Default::default(); self.hot_misses.clear(); // Reset the local context (it must always be empty between constants). // The fvar id counter lives on KEnv and is intentionally not reset here: @@ -874,6 +913,7 @@ impl<'a, M: KernelMode> TypeChecker<'a, M> { #[inline] pub fn tick(&mut self) -> Result<(), TcError> { if self.rec_fuel == 0 { + self.dump_guard_stack("recursive-fuel"); if crate::env_var("IX_REC_FUEL_DUMP").is_ok() && self.debug_label_matches_env() { @@ -898,6 +938,30 @@ impl<'a, M: KernelMode> TypeChecker<'a, M> { Ok(()) } + /// Diagnostics only: at most four stacks per process, 100 lines each. + /// Capture only at a guard, never on the checking hot path; do not retain + /// expressions, change limits, or turn an exhausted check into a verdict. + pub(crate) fn dump_guard_stack(&self, guard: &str) { + if !*IX_GUARD_STACKS || !self.debug_label_matches_env() { + return; + } + if GUARD_STACK_COUNT.fetch_add(1, std::sync::atomic::Ordering::Relaxed) >= 4 + { + return; + } + eprintln!( + "[guard stack] {guard} const={} local_depth={} def_eq_depth={} fuel_used={}", + self.debug_label.as_deref().unwrap_or(""), + self.depth(), + self.def_eq_depth, + self.fuel_used(), + ); + let trace = std::backtrace::Backtrace::force_capture().to_string(); + for line in trace.lines().take(100) { + eprintln!("[guard stack] {line}"); + } + } + /// Starting fuel for the current check. Used by diagnostics that want /// to report fuel consumed at a given point. pub fn fuel_used(&self) -> u64 { @@ -982,14 +1046,10 @@ impl<'a, M: KernelMode> TypeChecker<'a, M> { /// Check if expression is of the form `eagerReduce _ _` (2 args applied to the eagerReduce const). pub fn is_eager_reduce(&self, e: &KExpr) -> bool { - let (head, args) = collect_app_spine(e); - if args.len() != 2 { - return false; - } - match head.data() { - ExprData::Const(id, _, _) => id.addr == self.prims.eager_reduce.addr, - _ => false, - } + let ExprData::App(f, _, _) = e.data() else { return false }; + let ExprData::App(head, _, _) = f.data() else { return false }; + matches!(head.data(), ExprData::Const(id, _, _) + if id.addr == self.prims.eager_reduce.addr) } /// Intern an expression through the mutable intern environment. @@ -1006,45 +1066,47 @@ impl<'a, M: KernelMode> TypeChecker<'a, M> { if !*IX_HOT_MISSES { return; } - let mut key = format!("{} {}", phase, hot_expr_shape(e)); - if *IX_HOT_MISS_CTX { - let ctx = self.ctx_addr_for_lbr(e.lbr()); - key.push_str(&format!( - " ctx={} depth={}", - short_ctx_addr(&ctx), - self.depth() - )); - } - *self.hot_misses.entry(key).or_insert(0) += 1; + let context = (*IX_HOT_MISS_CTX) + .then(|| (self.ctx_addr_for_lbr(e.lbr()), self.depth())); + let key = MissKey { phase, a: *e.addr(), b: None, context }; + self.hot_misses.record(key, |out| { + use std::fmt::Write; + write!(out, "{phase} ")?; + write_hot_expr_shape(out, e)?; + write_hot_context(out, context) + }); } pub fn record_hot_def_eq_miss(&mut self, a: &KExpr, b: &KExpr) { if !*IX_HOT_MISSES { return; } - let mut key = - format!("defeq {} =?= {}", hot_expr_shape(a), hot_expr_shape(b)); - if *IX_HOT_MISS_CTX { - let ctx = self.def_eq_ctx_key(a, b); - key.push_str(&format!( - " ctx={} depth={}", - short_ctx_addr(&ctx), - self.depth() - )); - } - *self.hot_misses.entry(key).or_insert(0) += 1; + let context = + (*IX_HOT_MISS_CTX).then(|| (self.def_eq_ctx_key(a, b), self.depth())); + let key = + MissKey { phase: "defeq", a: *a.addr(), b: Some(*b.addr()), context }; + self.hot_misses.record(key, |out| { + use std::fmt::Write; + write!(out, "defeq ")?; + write_hot_expr_shape(out, a)?; + write!(out, " =?= ")?; + write_hot_expr_shape(out, b)?; + write_hot_context(out, context) + }); } fn dump_hot_misses(&self) { - if !*IX_HOT_MISSES || self.hot_misses.is_empty() { - return; - } - let mut entries: Vec<_> = self.hot_misses.iter().collect(); - entries.sort_unstable_by(|a, b| b.1.cmp(a.1).then_with(|| a.0.cmp(b.0))); - eprintln!("[hot misses] top {}:", entries.len().min(25)); - for (key, count) in entries.into_iter().take(25) { - eprintln!(" {count:>8} {key}"); + eprint!("{}", self.hot_miss_summary()); + } + + /// Snapshot opt-in miss counters without rerunning work or retaining terms. + /// The subject helper can report once at completion instead of dumping at + /// every exhausted speculative slice via `IX_REC_FUEL_DUMP`. + pub fn hot_miss_summary(&self) -> String { + if !*IX_HOT_MISSES { + return String::new(); } + self.hot_misses.summary() } } @@ -1088,11 +1150,50 @@ impl<'a> TypeChecker<'a, super::mode::Anon> { /// Check whether an expression mentions a constant with the given address. /// Iterative (stack-based) — immune to stack overflow on deeply nested input. pub fn expr_mentions_addr(e: &KExpr, addr: &Address) -> bool { + expr_mentions_any_addr(e, std::slice::from_ref(addr)) +} + +/// Check syntactic occurrences in one traversal for the entire address set. +/// No unfolding, context-dependent memoization or dependency traversal. +pub fn expr_mentions_any_addr( + e: &KExpr, + addrs: &[Address], +) -> bool { + expr_mentions_any_addr_impl(e, addrs, || {}) +} + +// Small trees are cheaper to inspect directly than to allocate a visited set. +// This is a fixed prefix budget, not an input-dependent heuristic: after it +// is spent the remaining walk is DAG-aware, so worst-case work stays linear +// in unique nodes/edges plus this constant prefix and its pending edges. +const OCCURRENCE_TREE_BUDGET: usize = 32; + +fn expr_mentions_any_addr_impl( + e: &KExpr, + addrs: &[Address], + mut on_visit: impl FnMut(), +) -> bool { + if addrs.is_empty() { + return false; + } + // Exact allocation identity, scoped to one fixed target set. Borrowing + // the root keeps every descendant alive, so pointer reuse is impossible. + // A shared subtree has the same occurrences at every binder depth. + let mut seen = FxHashSet::default(); + let mut tree_budget = OCCURRENCE_TREE_BUDGET; let mut stack: Vec<&KExpr> = vec![e]; while let Some(e) = stack.pop() { + // Zero-cost callback in production; tests bound attempted node visits + // so a regression cannot hang on an exponentially expanded diamond. + on_visit(); + if tree_budget > 0 { + tree_budget -= 1; + } else if !seen.insert(std::ptr::from_ref(e.data()).addr()) { + continue; + } match e.data() { ExprData::Const(id, _, _) => { - if id.addr == *addr { + if addrs.contains(&id.addr) { return true; } }, @@ -1110,7 +1211,7 @@ pub fn expr_mentions_addr(e: &KExpr, addr: &Address) -> bool { stack.push(body); }, ExprData::Prj(id, _, val, _) => { - if id.addr == *addr { + if addrs.contains(&id.addr) { return true; } stack.push(val); @@ -1125,12 +1226,28 @@ pub fn expr_mentions_addr(e: &KExpr, addr: &Address) -> bool { false } -/// Check whether an expression mentions any constant from a set of addresses. -pub fn expr_mentions_any_addr( - e: &KExpr, - addrs: &[Address], -) -> bool { - addrs.iter().any(|a| expr_mentions_addr(e, a)) +/// Borrow the head of an application spine without collecting arguments or +/// cloning any Arcs. Does not reduce the expression. +pub(crate) fn app_head(mut e: &KExpr) -> &KExpr { + while let ExprData::App(f, _, _) = e.data() { + e = f; + } + e +} + +/// Borrow an application head and its arguments in source application order. +/// The caller-held root owns every returned node; nothing is borrowed from a +/// mutable interner/cache. Short spines need neither allocation nor Arc clones. +pub(crate) fn borrow_app_spine( + mut e: &KExpr, +) -> (&KExpr, smallvec::SmallVec<[&KExpr; 8]>) { + let mut args = smallvec::SmallVec::new(); + while let ExprData::App(f, a, _) = e.data() { + args.push(a); + e = f; + } + args.reverse(); + (e, args) } /// Collect the application spine: `App(App(f, a1), a2)` → `(f, [a1, a2])`. @@ -1156,41 +1273,97 @@ pub fn collect_app_spine( return (e.clone(), Vec::new()); } let mut args = Vec::with_capacity(count); - let mut cur = e.clone(); + let mut cur = e; while let ExprData::App(f, a, _) = cur.data() { args.push(a.clone()); - cur = f.clone(); + cur = f; } args.reverse(); - (cur, args) + (cur.clone(), args) } -fn hot_expr_shape(e: &KExpr) -> String { - let (head, args) = collect_app_spine(e); - let head = match head.data() { - ExprData::Var(i, _, _) => format!("#{i}"), - ExprData::FVar(id, _, _) => format!("{id}"), - ExprData::Sort(u, _) => format!("Sort({u})"), - ExprData::Const(id, us, _) => format!("{id}.{{{}}}", us.len()), - ExprData::App(..) => "app".to_string(), - ExprData::Lam(..) => "lam".to_string(), - ExprData::All(..) => "forall".to_string(), - ExprData::Let(..) => "let".to_string(), - ExprData::Prj(id, field, _, _) => format!("Prj({id}.{field})"), - ExprData::Nat(v, _, _) => format!("Nat({})", v.0), - ExprData::Str(v, _, _) => format!("Str(len={})", v.len()), - }; - format!("{head}/{} lbr={} @{}", args.len(), e.lbr(), short_addr(e.addr())) +#[cfg(test)] +mod spine_tests; + +fn write_hot_expr_shape( + out: &mut impl std::fmt::Write, + e: &KExpr, +) -> std::fmt::Result { + let mut head = e; + let mut arity = 0usize; + while let ExprData::App(f, _, _) = head.data() { + head = f; + arity += 1; + } + match head.data() { + ExprData::Var(i, _, _) => write!(out, "#{i}"), + ExprData::FVar(id, _, _) => write!(out, "{id}"), + // Do not render an arbitrarily large universe DAG or Nat literal. + ExprData::Sort(u, _) => write!(out, "Sort(uid{})", u.addr()), + ExprData::Const(id, us, _) => { + write_hot_id(out, id)?; + write!(out, ".{{{}}}", us.len()) + }, + ExprData::App(..) => unreachable!("peeled all applications"), + ExprData::Lam(..) => write!(out, "lam"), + ExprData::All(..) => write!(out, "forall"), + ExprData::Let(..) => write!(out, "let"), + ExprData::Prj(id, field, _, _) => { + write!(out, "Prj(")?; + write_hot_id(out, id)?; + write!(out, ".{field})") + }, + ExprData::Nat(..) => write!(out, "Nat(uid{})", head.addr()), + ExprData::Str(v, _, _) => write!(out, "Str(len={})", v.len()), + }?; + write!(out, "/{arity} lbr={} @uid{}", e.lbr(), e.addr()) } -fn short_addr(addr: &Addr) -> String { - format!("uid{addr}") +fn write_hot_id( + out: &mut impl std::fmt::Write, + id: &KId, +) -> std::fmt::Result { + // Name::Display builds its whole `pretty()` string before passing it to + // the writer. A bounded output writer alone cannot limit that temporary. + // Check at most 64 borrowed components/256 bytes before invoking Display; + // oversized names fall back to their fixed-size declaration address. + let compact = M::meta_get(&id.name).is_none_or(|name| { + use ix_common::env::NameData; + let mut name = name; + let mut bytes = 256usize; + for _ in 0..64 { + let (parent, cost) = match name.as_data() { + NameData::Anonymous(_) => return true, + NameData::Str(parent, s, _) => (parent, s.len().saturating_add(1)), + NameData::Num(parent, n, _) => { + if n.to_u64().is_none() { + return false; + } + (parent, 21) + }, + }; + let Some(remaining) = bytes.checked_sub(cost) else { return false }; + bytes = remaining; + name = parent; + } + false + }); + if compact { write!(out, "{id}") } else { write!(out, "#{}", id.addr.hex()) } } -fn short_ctx_addr(addr: &CtxAddr) -> String { - addr.to_hex().chars().take(12).collect() +fn write_hot_context( + out: &mut impl std::fmt::Write, + context: Option<(CtxAddr, u64)>, +) -> std::fmt::Result { + if let Some((ctx, depth)) = context { + write!(out, " ctx={} depth={depth}", &ctx.to_hex()[..12])?; + } + Ok(()) } +#[cfg(test)] +mod scan_tests; + #[cfg(test)] mod tests { use super::super::testing::{ @@ -1205,6 +1378,24 @@ mod tests { TypeChecker::new(env) } + #[test] + fn hot_miss_labels_do_not_pretty_print_unbounded_names() { + use ix_common::env::Name; + for name in [ + Name::str(Name::anon(), "λ".repeat(100_000)), + (0..128).fold(Name::anon(), |n, _| Name::str(n, "x".to_owned())), + ] { + let id = KId::::new(Address::hash(b"large-name"), name); + let mut out = String::new(); + write_hot_id(&mut out, &id).unwrap(); + assert_eq!(out, format!("#{}", id.addr.hex())); + } + let id = mk_id("List.below"); + let mut out = String::new(); + write_hot_id(&mut out, &id).unwrap(); + assert_eq!(out, format!("{id}")); + } + // ---- Context push/pop ---- #[test] @@ -1528,6 +1719,25 @@ mod tests { // ---- tick / fuel ---- + #[test] + fn configured_fuel_initializes_and_resets_checks() { + assert_eq!(MAX_REC_FUEL, 100_000_000); + let expected = crate::env_var("IX_MAX_REC_FUEL") + .ok() + .and_then(|s| s.parse::().ok()) + .unwrap_or(MAX_REC_FUEL); + assert_eq!(max_rec_fuel(), expected); + + let mut tc = new_tc(); + assert_eq!(tc.rec_fuel, expected); + tc.rec_fuel = 0; + tc.reset(); + assert_eq!(tc.rec_fuel, expected); + tc.rec_fuel = 0; + tc.finish_constant_accounting(); + assert_eq!(tc.rec_fuel, expected); + } + #[test] fn tick_consumes_fuel() { let mut tc = new_tc(); diff --git a/crates/kernel/src/tc/scan_tests.rs b/crates/kernel/src/tc/scan_tests.rs new file mode 100644 index 000000000..caa845737 --- /dev/null +++ b/crates/kernel/src/tc/scan_tests.rs @@ -0,0 +1,239 @@ +//! Differential and work-bound tests for syntactic expression inspection. + +use super::*; +use crate::expr::FVarId; +use crate::mode::{Anon, Meta}; +use bignat::Nat; +use ix_common::env::{BinderInfo, Name}; + +// Frozen tree walker: keep it independent of the DAG-aware implementation. +fn mentions_reference(e: &KExpr, addr: &Address) -> bool { + let mut stack = vec![e]; + while let Some(e) = stack.pop() { + match e.data() { + ExprData::Const(id, _, _) => { + if &id.addr == addr { + return true; + } + }, + ExprData::App(f, a, _) => stack.extend([f, a]), + ExprData::Lam(_, _, t, b, _) | ExprData::All(_, _, t, b, _) => { + stack.extend([t, b]); + }, + ExprData::Let(_, t, v, b, _, _) => stack.extend([t, v, b]), + ExprData::Prj(id, _, v, _) => { + if &id.addr == addr { + return true; + } + stack.push(v); + }, + ExprData::Var(..) + | ExprData::FVar(..) + | ExprData::Sort(..) + | ExprData::Nat(..) + | ExprData::Str(..) => {}, + } + } + false +} + +fn differential() { + let a = Address::hash(b"A"); + let b = Address::hash(b"B"); + let missing = Address::hash(b"absent"); + let name = || M::meta_field(Name::anon()); + let bi = || M::meta_field(BinderInfo::Default); + let mut nodes: Vec> = vec![ + KExpr::var(0, name()), + KExpr::fvar(FVarId(0), name()), + KExpr::sort(KUniv::zero()), + KExpr::cnst(KId::new(a.clone(), name()), Box::new([])), + KExpr::cnst(KId::new(b.clone(), name()), Box::new([])), + KExpr::nat(Nat::from(17u64), Address::hash(b"17")), + KExpr::str("hello".to_owned(), Address::hash(b"hello")), + ]; + for i in 0..24 { + let x = nodes[i].clone(); + let y = nodes[(i * 7 + 3) % nodes.len()].clone(); + let z = nodes[(i + 4) % nodes.len()].clone(); + nodes.push(match i % 5 { + 0 => KExpr::app(x, y), + 1 => KExpr::lam(name(), bi(), x, y), + 2 => KExpr::all(name(), bi(), x, y), + 3 => KExpr::let_(name(), x, y, z, i % 2 == 0), + _ => KExpr::prj(KId::new(a.clone(), name()), 2, x), + }); + } + for node in &nodes { + for targets in [ + vec![], + vec![missing.clone()], + vec![a.clone()], + vec![b.clone()], + vec![missing.clone(), b.clone()], + vec![a.clone(), a.clone(), b.clone()], + ] { + let expected = targets.iter().any(|a| mentions_reference(node, a)); + assert_eq!(expr_mentions_any_addr(node, &targets), expected); + for addr in &targets { + assert_eq!( + expr_mentions_addr(node, addr), + mentions_reference(node, addr) + ); + } + } + } +} + +#[test] +fn occurrence_queries_match_tree_reference_in_both_modes() { + differential::(); + differential::(); +} + +#[test] +fn occurrence_queries_visit_diamond_edges_linearly() { + let target = Address::hash(b"present"); + let mut root = + KExpr::::cnst(KId::new(target.clone(), ()), Box::new([])); + let depth = 60; + for _ in 0..depth { + root = KExpr::app(root.clone(), root); + } + let mut visits = 0; + assert!(!expr_mentions_any_addr_impl( + &root, + &[Address::hash(b"missing"), Address::hash(b"also missing")], + || { + visits += 1; + assert!( + visits <= 2 * OCCURRENCE_TREE_BUDGET + 2 * depth + 1, + "expanded shared DAG as a tree" + ); + }, + )); + assert!(visits > 2 * depth); + // Neither the visited set nor a negative result survives a query. + assert!(expr_mentions_addr(&root, &target)); + assert!(!expr_mentions_any_addr_impl(&root, &[], || panic!( + "empty query walked" + ))); +} + +#[test] +fn occurrence_query_checks_every_binder_and_projection_position() { + let id = KId::::new(Address::hash(b"target"), ()); + let hit = KExpr::cnst(id.clone(), Box::new([])); + let miss = KExpr::var(0, ()); + for e in [ + KExpr::lam((), (), hit.clone(), miss.clone()), + KExpr::lam((), (), miss.clone(), hit.clone()), + KExpr::all((), (), hit.clone(), miss.clone()), + KExpr::all((), (), miss.clone(), hit.clone()), + KExpr::let_((), hit.clone(), miss.clone(), miss.clone(), false), + KExpr::let_((), miss.clone(), hit.clone(), miss.clone(), false), + KExpr::let_((), miss.clone(), miss.clone(), hit.clone(), true), + KExpr::prj(id.clone(), 0, miss), + KExpr::prj(KId::new(Address::hash(b"other"), ()), 0, hit), + ] { + assert!(expr_mentions_addr(&e, &id.addr)); + } +} + +#[test] +fn borrowed_app_head_matches_owned_collector_by_pointer() { + for head in [ + KExpr::::cnst(KId::new(Address::hash(b"f"), ()), Box::new([])), + KExpr::var(0, ()), + KExpr::prj(KId::new(Address::hash(b"S"), ()), 0, KExpr::var(1, ())), + ] { + let mut e = head.clone(); + // Holding each prefix also keeps final cleanup from recursively dropping + // a deep unique spine on the test runner's small stack. + let mut prefixes = Vec::new(); + for n in 0..1024 { + if [0, 1, 2, 3, 8, 1023].contains(&n) { + let (owned_head, args) = collect_app_spine(&e); + assert!(app_head(&e).ptr_eq(&owned_head)); + assert!(app_head(&e).ptr_eq(&head)); + assert_eq!(args.len(), n); + } + prefixes.push(e.clone()); + e = KExpr::app(e, KExpr::var(0, ())); + } + drop(e); + while prefixes.pop().is_some() {} + } +} + +#[test] +fn eager_reduce_requires_exactly_two_arguments_and_correct_head() { + let mut env = KEnv::::new(); + let tc = TypeChecker::new(&mut env); + for id in [ + tc.prims.eager_reduce.clone(), + KId::new(Address::hash(b"other"), Name::anon()), + ] { + let mut e = KExpr::cnst(id.clone(), Box::new([])); + for arity in 0..5 { + assert_eq!( + tc.is_eager_reduce(&e), + arity == 2 && id == tc.prims.eager_reduce + ); + e = KExpr::app(e, KExpr::var(0, Name::anon())); + } + } + let non_const = KExpr::app( + KExpr::app(KExpr::var(0, Name::anon()), KExpr::var(1, Name::anon())), + KExpr::var(2, Name::anon()), + ); + assert!(!tc.is_eager_reduce(&non_const)); +} + +#[test] +#[ignore = "manual release benchmark; no wall-clock assertions"] +fn benchmark_occurrence_and_head_walks() { + use std::{hint::black_box, time::Instant}; + let absent = Address::hash(b"absent"); + for depth in [0, 4, 16, 20] { + let mut e = KExpr::::var(0, ()); + for _ in 0..depth { + e = KExpr::app(e.clone(), e); + } + let repeats = if depth <= 4 { 100_000 } else { 4 }; + for dag in [false, true, true, false] { + let start = Instant::now(); + for _ in 0..repeats { + black_box(if dag { + expr_mentions_addr(&e, &absent) + } else { + mentions_reference(&e, &absent) + }); + } + eprintln!( + "occurrence dag={dag} depth={depth} repeats={repeats} elapsed={:?}", + start.elapsed() + ); + } + } + for arity in [0, 2, 8, 64] { + let mut e = KExpr::::var(0, ()); + for _ in 0..arity { + e = KExpr::app(e, KExpr::var(1, ())); + } + for borrowed in [false, true, true, false] { + let start = Instant::now(); + for _ in 0..100_000 { + if borrowed { + black_box(app_head(&e)); + } else { + black_box(collect_app_spine(&e)); + } + } + eprintln!( + "head borrowed={borrowed} arity={arity} elapsed={:?}", + start.elapsed() + ); + } + } +} diff --git a/crates/kernel/src/tc/spine_tests.rs b/crates/kernel/src/tc/spine_tests.rs new file mode 100644 index 000000000..9a051199a --- /dev/null +++ b/crates/kernel/src/tc/spine_tests.rs @@ -0,0 +1,74 @@ +use super::*; +use crate::mode::{Anon, Meta}; +use ix_common::env::{BinderInfo, Name}; + +fn spine_identity(arity: usize) { + let head = KExpr::::var(3, M::meta_field(Name::anon())); + let mut root = head.clone(); + let mut expected = Vec::new(); + for i in 0..arity { + // Named lambdas make preserving the original occurrences observable + // in Meta mode too. No interning/canonicalization is part of a spine walk. + let arg = KExpr::lam( + M::meta_field(Name::str(Name::anon(), format!("arg{i}"))), + M::meta_field(BinderInfo::Implicit), + KExpr::sort(KUniv::zero()), + KExpr::var(i as u64, M::meta_field(Name::anon())), + ); + expected.push(arg.clone()); + root = KExpr::app(root, arg); + } + let (borrowed_head, borrowed) = borrow_app_spine(&root); + assert!(borrowed_head.ptr_eq(&head)); + assert_eq!(borrowed.len(), arity); + assert_eq!(borrowed.spilled(), arity > 8); + for (a, b) in borrowed.iter().zip(&expected) { + assert!(a.ptr_eq(b)); + } + let (owned_head, owned) = collect_app_spine(&root); + assert!(owned_head.ptr_eq(borrowed_head)); + for (a, b) in owned.iter().zip(&borrowed) { + assert!(a.ptr_eq(b)); + } + + // Safe while caller-owned roots survive even if the interner is mutated + // or discarded; the borrow does not reach into that table's storage. + let mut env = KEnv::::new(); + env.intern.intern_expr(root.clone()); + drop(env); + assert!(borrowed[arity - 1].ptr_eq(&expected[arity - 1])); +} + +#[test] +fn borrowed_spines_preserve_order_identity_and_inline_boundary() { + for n in [1, 2, 7, 8, 9, 16, 33] { + spine_identity::(n); + spine_identity::(n); + } +} + +#[test] +fn borrowed_empty_spine_preserves_head_without_spilling() { + fn check() { + let e = KExpr::::sort(KUniv::zero()); + let (head, args) = borrow_app_spine(&e); + assert!(std::ptr::eq(head, &e)); + assert!(args.is_empty()); + assert!(!args.spilled()); + } + check::(); + check::(); +} + +#[test] +fn deep_borrowed_spines_are_iterative() { + std::thread::Builder::new() + .stack_size(256 * 1024 * 1024) + .spawn(|| { + spine_identity::(4096); + spine_identity::(4096); + }) + .unwrap() + .join() + .unwrap(); +} diff --git a/crates/kernel/src/whnf.rs b/crates/kernel/src/whnf.rs index 38557a953..0a04f2aeb 100644 --- a/crates/kernel/src/whnf.rs +++ b/crates/kernel/src/whnf.rs @@ -78,7 +78,9 @@ use super::mode::KernelMode; use super::subst::{ Clo, MEnv, clo_readback, clo_subst, subst, subst_no_intern, }; -use super::tc::{IotaInfo, MAX_WHNF_FUEL, TypeChecker, collect_app_spine}; +use super::tc::{ + IotaInfo, MAX_WHNF_FUEL, TypeChecker, app_head, collect_app_spine, +}; use bignat::Nat; @@ -701,8 +703,8 @@ impl TypeChecker<'_, M> { } // App: collect spine, whnf_core head, try beta/iota - let (f0, args) = collect_app_spine(&cur); - let f = self.whnf_core_with_flags(&f0, flags)?; + let (f0, args) = super::tc::borrow_app_spine(&cur); + let f = self.whnf_core_with_flags(f0, flags)?; // Beta: enter the environment machine. Subsequent betas/zetas are // O(1) environment pushes; substitution materializes only at the @@ -712,16 +714,19 @@ impl TypeChecker<'_, M> { // beta firing — Const-headed terms (e.g. literal recursor loops) // never pay the closure-wrap + readback overhead. if matches!(f.data(), ExprData::Lam(..)) { - cur = self.machine_whnf(f, &args, flags)?; + let reduced = self.machine_whnf(f, &args, flags)?; + drop(args); + cur = reduced; continue; } // If head reduced, rebuild and try iota - if !f.ptr_eq(&f0) { + if !f.ptr_eq(f0) { let mut rebuilt = f; for arg in &args { - rebuilt = self.intern(KExpr::app(rebuilt, arg.clone())); + rebuilt = self.intern(KExpr::app(rebuilt, (*arg).clone())); } + drop(args); if let Some(reduced) = self.try_iota_with_flags(&rebuilt, flags)? { cur = reduced; continue; @@ -730,6 +735,7 @@ impl TypeChecker<'_, M> { } // Try iota on original + drop(args); if let Some(reduced) = self.try_iota_with_flags(&cur, flags)? { cur = reduced; continue; @@ -773,13 +779,13 @@ impl TypeChecker<'_, M> { fn machine_whnf( &mut self, head: KExpr, - args: &[KExpr], + args: &[&KExpr], flags: WhnfFlags, ) -> Result, TcError> { let mut head = head; let mut env: MEnv = MEnv::empty(); let mut spine: Vec>> = - args.iter().rev().map(|a| Arc::new(Clo::closed(a.clone()))).collect(); + args.iter().rev().map(|a| Arc::new(Clo::closed((*a).clone()))).collect(); loop { match head.data() { @@ -1328,7 +1334,7 @@ impl TypeChecker<'_, M> { e: &KExpr, flags: WhnfFlags, ) -> Result>, TcError> { - let (head, spine) = collect_app_spine(e); + let (head, spine) = super::tc::borrow_app_spine(e); let (rec_id, rec_us) = match head.data() { ExprData::Const(id, us, _) => (id.clone(), us.clone()), @@ -1366,6 +1372,10 @@ impl TypeChecker<'_, M> { _ => return Ok(None), }; + // Only an actual recursor needs ownership for reduction/reconstruction. + // The common non-recursor rejection above only inspected borrowed nodes. + let spine: Vec<_> = spine.into_iter().cloned().collect(); + // K-like recursor: try to synthesize a nullary constructor before WHNF. // This handles cases like `Eq.rec motive minor major` where major isn't // a constructor but its type matches the inductive — we build `Eq.refl params...`. @@ -2303,7 +2313,7 @@ impl TypeChecker<'_, M> { let w = self.whnf(&ty)?; match w.data() { ExprData::All(_, _, dom, body, _) => { - let (head, _) = collect_app_spine(dom); + let head = app_head(dom); if let ExprData::Const(id, _, _) = head.data() { // Only accept if the head resolves to an inductive. if matches!(self.try_get_const(id)?, Some(KConst::Indc { .. })) { @@ -2866,7 +2876,7 @@ impl TypeChecker<'_, M> { } fn is_stuck_nat_predicate_probe(&self, e: &KExpr) -> bool { - let (head, _) = collect_app_spine(e); + let head = app_head(e); match head.data() { ExprData::Const(id, _, _) => { self.is_nat_bin_pred_addr(&id.addr) @@ -2876,7 +2886,7 @@ impl TypeChecker<'_, M> { if id.addr == self.prims.fin.addr { return true; } - let (val_head, _) = collect_app_spine(val); + let val_head = app_head(val); matches!( val_head.data(), ExprData::Const(val_id, _, _) @@ -3406,7 +3416,7 @@ impl TypeChecker<'_, M> { // constant function 1, but its body recurses on an open unit variable. // Reduce this primitive singleton case directly. if id.addr == self.prims.size_of_size_of.addr && args.len() == 3 { - let (ty_head, _) = collect_app_spine(&args[0]); + let ty_head = app_head(&args[0]); if let ExprData::Const(ty_id, _, _) = ty_head.data() && (ty_id.addr == self.prims.unit.addr || ty_id.addr == self.prims.punit.addr) @@ -5635,7 +5645,7 @@ mod tests { // The shared per-constant budget guards against unbounded expansion of Nat // literals into Nat.succ chains when the same recursor peels consecutive // predecessors for thousands of steps. Give this adversarial unit a small - // explicit budget: production's 10M allowance is intentionally large for + // explicit budget: production's 100M allowance is intentionally large for // real certificate computations and would make the termination test slow. // =========================================================================