diff --git a/CLAUDE.md b/CLAUDE.md index 6565c84..5d27b78 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,11 +50,12 @@ while cache round-trips tree), `l1Body` (`call_sites` → `body{}`), `heritage`, (`src/schema/emit.ts`) runs them + assembles envelope + strips INTERNAL fields (`call_sites`, `abs_path`, cache trio). -Call graph defaults to **union** of two backends: TS compiler resolver -and embedded [Jelly](https://github.com/cs-au-dk/jelly) flow analyzer (recovers -higher-order/callback edges resolver misses). Merged edges keep -`provenance` tag (`tsc` / `jelly`); `--tsc-only` or `--call-graph-provider jelly` -picks one alone. +Call graph = tsc resolver + **defuse linker** (#98): deterministic per-callable +pass over resolver leftovers — alias chains, decorator edges, library-callback +edges, bounded interprocedural votes, CHA-by-name fallback. No whole-program +fixpoint, no backend flag, one code path. Module-scope calls attributed to +MODULE (python #131 parity). prov tags: `tsc` / `defuse` / `import`. Joern +superset ledger: `docs/design/specs/defuse-linker-joern-ledger.md`. ## Architecture — follow the pipeline @@ -65,8 +66,9 @@ it first; everything else is stage it calls, in order: 2. **buildSymbolTable** (`src/syntactic_analysis`) — modules, classes, interfaces, enums, type aliases, namespaces, functions, methods, variables, decorators, JSDoc, with precise source spans. -3. **call graph** (`src/semantic_analysis`) — `selectProvider()` picks tsc / jelly / - union; each provider returns edges + external (phantom) symbols. +3. **call graph** (`src/semantic_analysis`) — tsc resolver (`callGraph.ts`, incl. + module-scope sweep + RTA + phantoms) then `defuseLinker.ts` tiers T1–T5; + merged with provenance union. 4. **program graphs** (`src/dataflow`) — levels 3–4 (`-a 3`/`-a 4`): CFG → post-dominance/CDG → access-path def-use → PDG → SCC-condensed bottom-up summaries → SDG. This is *compute* (IR in `src/schema/graphs.ts`); `src/dataflow/attach.ts` writes it **onto tree** @@ -96,13 +98,21 @@ test — treat both as contracts, keep in lockstep with JSON. | `src/core.ts` | `analyze()` orchestrator — the spine | | `src/options` | Parsed CLI options / `AnalysisOptions` | | `src/syntactic_analysis` | Symbol table (ts-morph traversal) | -| `src/semantic_analysis` | Call-graph providers (tsc, jelly, union), phantoms | +| `src/semantic_analysis` | Call graph: tsc resolver + defuse linker (T1–T5), phantoms | | `src/dataflow` | L3/L4 program-graph **compute** (CFG, dominance/CDG, def-use, summaries, SDG) + `attach.ts` (IR → tree) | | `src/schema` | **the native v2 model** (`schema.ts`) + per-run passes (`assignIds`/`l1Body`/`heritage`/`homing`/`l2Callees`) + `emit.ts` (`finalizeAnalysis`) + `signatureOf` + graphs IR | | `src/build` | Dep materialization; `build/neo4j` = the v2 graph projection (project/rows/cypher/bolt/schema) | | `src/utils` | fs, caching, logging, serialization (`serialize.ts` writes the envelope), version | | `test` | Bun tests + `fixtures/sample-app` + `fixtures/dataflow-app`; `schema-v2.test.ts` = the L1–L4 gates | +**Repository-artifact layer** (#101, python PR #160 parity): `application.artifacts{}` +(rules-matched non-code files, LANGUAGE-NEUTRAL `can://artifact//` ids, +roles[], verbatim unbounded `source`) + flat `dependencies[]` (npm kinds incl. +coined `peer`, prov-tagged, lock backfill) + `unresolved_imports[]` (@types +type-only rule; `--resolve-installed` opt-in probe). `src/artifacts/`. Neo4j +contract 2.2.0: NEUTRAL :Artifact/:Package (purl) — sanctioned prefix exception — +plus TS_PROVIDES/TS_UNRESOLVED_IMPORT into :TSExternal ghosts. + ## Commands - `bun run start -- --input /path/to/project` — run analyzer from source. diff --git a/README.md b/README.md index 55d1a92..d0ea9d9 100644 --- a/README.md +++ b/README.md @@ -24,11 +24,11 @@ structure into a **Neo4j property graph**. It is the TypeScript backend behind [Python](https://github.com/codellm-devkit/codeanalyzer-python) and [Java](https://github.com/codellm-devkit/codeanalyzer-java) siblings. -By default the call graph is the **union** of two backends: the TypeScript compiler's resolver and -[Jelly](https://github.com/cs-au-dk/jelly) — a flow-based analyzer that resolves higher-order and -callback edges the resolver misses, embedded in the `cants` binary (no extra install). Merged edges -keep a `provenance` tag (`tsc` / `jelly`), so you can still tell the two apart. Pass `--tsc-only` to -drop Jelly and run the resolver alone, or `--call-graph-provider jelly` for Jelly alone. +The call graph is the TypeScript compiler's resolver plus a **defuse linker** — a deterministic, +per-callable pass that backfills the edges the resolver misses (alias chains, decorator +invocations, callbacks handed to library calls, parameter-flow calls) with no whole-program +fixpoint. Edges keep a `provenance` tag (`tsc` / `defuse` / `import`), so you can tell the layers +apart, and the output is byte-identical across runs. ## Table of Contents @@ -55,9 +55,9 @@ drop Jelly and run the resolver alone, or `--call-graph-provider jelly` for Jell methods, variables, decorators, and JSDoc, with precise source spans. - **Call graph** — the TypeScript compiler's resolver plus Rapid Type Analysis (RTA), with **phantom (external) nodes** for calls into imported libraries and Node builtins. -- **Pluggable call-graph backend** — the `union` of the `tsc` resolver and the embedded - [Jelly](https://github.com/cs-au-dk/jelly) flow analyzer by default (`--tsc-only` for the resolver - alone, `--call-graph-provider jelly` for Jelly alone). +- **Defuse linker** — a deterministic per-callable pass over the resolver's leftovers: alias + chains, decorator invocations, library-callback edges, and bounded interprocedural votes — + validated as a strict superset of Joern's real call pairs on the reference corpus. - **Neo4j output** — project the analysis into a labeled property graph: a self-contained `graph.cypher` snapshot, or an **incremental** push to a live database over Bolt. - **Versioned schema** — a machine-readable, version-stamped Neo4j schema contract @@ -182,11 +182,8 @@ Options: node_modules) --no-phantoms disable phantom (external) nodes for imported/required library calls - --call-graph-provider call-graph backend: union (default, tsc ∪ - jelly) | tsc | jelly | both (deprecated alias - of union) (default: "union") - --tsc-only use the tsc resolver only — opt out of Jelly - edges (overrides --call-graph-provider) + --resolve-installed probe node_modules metadata for import→package + binding (default: repo files only) -c, --cache-dir cache/intermediate directory -v, --verbose increase verbosity (repeatable) -h, --help display help for command @@ -214,17 +211,12 @@ Options: cants --input ./my-ts-project --target-files src/a.ts src/b.ts ``` -4. **Resolver-only call graph (opt out of Jelly):** - ```sh - cants --input ./my-ts-project --tsc-only - ``` - -5. **Force a clean rebuild with a custom cache directory:** +4. **Force a clean rebuild with a custom cache directory:** ```sh cants --input ./my-ts-project --eager --cache-dir /path/to/custom-cache ``` -6. **Program graphs (level 3): CFG/PDG/SDG in `analysis.json`:** +5. **Program graphs (level 3): CFG/PDG/SDG in `analysis.json`:** ```sh cants --input ./my-ts-project -a 3 # full program_graphs section cants --input ./my-ts-project -a 3 --graphs cfg,pdg # scope the emitted graphs @@ -282,8 +274,8 @@ nodes all join. **Substrate (locked in [issue #2](https://github.com/codellm-devkit/codeanalyzer-typescript/issues/2)):** the CFG and reaching-definitions are hand-built from the ts-morph AST; the call-graph oracle is -the existing provenance-merged tsc ∪ Jelly graph; aliasing is a flow-insensitive copy-alias MVP -(Jelly points-to-backed propagation is a staged upgrade). Function summaries are composed +the provenance-merged tsc + defuse graph; aliasing is a flow-insensitive copy-alias MVP +(points-to-backed propagation is a staged upgrade). Function summaries are composed bottom-up over the SCC condensation of the call graph, with k-limited access paths; module globals ride the SDG as extra parameters. The analysis is deliberately sound-leaning and over-approximate; known unsoundness (dynamic `eval`, reflection/monkey-patching, npm-internal diff --git a/docs/design/specs/artifacts-and-dependencies.md b/docs/design/specs/artifacts-and-dependencies.md new file mode 100644 index 0000000..9e6fc6a --- /dev/null +++ b/docs/design/specs/artifacts-and-dependencies.md @@ -0,0 +1,96 @@ +# Artifacts and dependencies — the repository-artifact layer for TypeScript + +- **Status:** implemented (branch `feat/issue-101-artifacts`); **recalibrated 2026-08-27** to the + ratified python contract after the first cut anchored on an orphaned branch +- **Scope:** `codeanalyzer-typescript`; schema v2 **additive** (no level, id-tier, or existing-field movement) +- **Parity anchor:** codeanalyzer-python **PR #160** (implementation of the approved spec + `2026-08-27-artifacts-and-dependencies-design.md`, PR #158). NOT the `51ee29e` + `feat/configuration-files` branch — that shape (language-namespaced `@artifact/` ids, contained + dependency/config-key children, `artifact_kind` enum, text-capture caps) was never merged; this + spec's first revision mirrored it and has been rebuilt. +- **Tracking:** one work item (#101), one PR (#103), branch stacked on `feat/issue-100-linker-propagation` + +## Contract-impact triage + +| Question | Answer | +| --- | --- | +| Schema v2 shape | **additive**: `application.artifacts{}` (flat nodes), `application.dependencies[]` (flat evidence rows), `application.unresolved_imports[]` | +| Identity | artifact ids are **language-NEUTRAL**: `can://artifact//` — the first `can://` segment is a namespace (a language for code, the literal `artifact` for files), so sibling analyzers over one repo emit the SAME id for the same file. `` agreement is the precondition for cross-analyzer joins | +| Levels / monotonicity | ungated, identical at `-a 1..4`; monotonicity holds trivially | +| schema_version | unchanged; Neo4j contract 2.1.0 → **2.2.0** (additive) | +| Repos | `codeanalyzer-typescript` now; **python-sdk** must gain the three families before its analyzer pin moves (its models are `extra="forbid"` — verified; python's own PR #160 carries the same obligation); repo docs | +| Shared vocabulary movement | ONE additive token: dependency `kind: "peer"` (npm's contract-with-host) against the ratified enum `runtime\|dev\|optional\|build`. Recorded like the `"reaching-defs"` precedent | + +## The mirrored model (python PR #160 shapes) + +``` +application.artifacts: Record +TSArtifact id = can://artifact// (per-run), kind "artifact", path, + format (json|jsonc|yaml|toml|ini|dockerfile|yarnlock|env|text), + roles[] (dependency-manifest|tool-config|container-image|service-topology| + ci|env|packaging|legal|docs|script|unknown), + size_bytes, sha256, source (verbatim, UNBOUNDED by decision — spec §3), + extraction (none|partial|full) + +application.dependencies: TSDependency[] # flat, no node ids — :Package (purl) is the node +TSDependency name (@scope kept), spec, kind (runtime|dev|optional|peer|build), extras[] (npm: []), + declared_in (artifact id), locked_version?, provides_imports[], + prov[] (declared|lockfile|installed-metadata|heuristic) + +application.unresolved_imports: TSImportBinding[] +TSImportBinding module (specifier root), bound_to?, prov[] +``` + +## Locked TS decisions (recalibration session 2026-08-27) + +1. **config_keys dropped** — python's unit 4 owns config extraction; config-role artifacts get + node + roles + source only. The earlier TS config-key parser is parked (this file is its record). +2. **`peer` kind coined** (additive). npm mapping: `dependencies→runtime`, + `devDependencies→dev`, `optionalDependencies→optional`, `peerDependencies→peer`. +3. **Capture is rules-matched only** (python's posture): the shipped table in + `src/artifacts/rules.ts` (glob → format/roles, roles union across matches; basename patterns + match at any depth, `/`-patterns anchor at root) + extensionless shebang files as `script`. + An unmatched file is NOT an artifact. `source` is verbatim and unbounded (python's decision; + revisit only with measured payload numbers); binary probe failures carry `source: ""`. +4. **Dependencies are declared-only and flat**: every `package.json` (workspace members included) + emits records with `declared_in` = its artifact id; the JSON lock family + (`package-lock.json`/`npm-shrinkwrap.json`/`bun.lock` JSONC) backfills `locked_version` on the + OWNING (sibling) manifest's records and appends `"lockfile"` to `prov`; locks never create + records; `yarn.lock`/`pnpm-lock.yaml` are inventory-only artifacts. +5. **provides_imports**: the package name itself; `@types/x` also provides `x` + (DefinitelyTyped `scope__pkg` unmangled to `@scope/pkg`). +6. **Import binding / unresolved_imports**: every non-relative, non-builtin specifier ROOT from + the symbol table's imports. A VALUE import needs the runtime package; an `import type` is + satisfiable by `@types/x` alone. Only-@types-for-a-value-import → partially bound + (`bound_to: "@types/x"`, `prov: ["heuristic"]`). `--resolve-installed` (opt-in, default off) + probes `node_modules//package.json` (`prov: ["installed-metadata"]`); default runs read + only repo files and stay byte-identical. +7. **Neo4j (contract 2.2.0)**: language-NEUTRAL `:Artifact` and `:Package` (purl ids, + `pkg:npm/`, scoped `pkg:npm/%40scope/`) — the deliberate, sanctioned exception to + TS-prefixing so sibling analyzers MERGE onto the same nodes (the conformance gate allowlists + exactly these). Edges: `HAS_ARTIFACT`, `DECLARES_DEPENDENCY` (props spec/kind/extras/prov, + `_k` = kind), `LOCKS` (version; fans from every lock artifact present — python's documented + coarse fan), and the analyzer's own claims `TS_PROVIDES` (Package→minted module-level + `:TSExternal` ghost) and `TS_UNRESOLVED_IMPORT` (application→ghost, prov). `source` stays off + the graph. +8. **Pipeline**: `src/artifacts/` (rules, deps, binding, index) runs in `analyze()` after the + symbol table (binding needs module imports), level-ungated, not cached; `assignIds` stamps + artifact ids and re-stamps `declared_in` per run (`--app-name` rule). + +## Definition of done + +- Three sections emitted identically at every `-a`; monotonicity + conformance + count-parity + gates green (parity gate counts neutral Artifact/Package rows + minted ghosts explicitly). +- Fixture app: root+workspace manifests, both JSON locks, `yarn.lock` inventory-only, `.env`, + tsconfig, Dockerfile, CI workflow, LICENSE, an undeclared VALUE import, an `import type` + satisfied by `@types` — every kind token incl. `peer`, prov chains, purl ids (scoped included), + `--resolve-installed` exercised. +- Determinism: two consecutive default runs byte-identical. +- `schema.neo4j.json` regenerated at 2.2.0; CLAUDE.md + SCHEMA_DECISIONS + README/--help updated. + +## Release plan + +Ships in the minor after the linker train (#97 → #99 → #102 → #103); schema_version unmoved; +Neo4j 2.2.0 in release notes. **SDK lockstep required** (`extra="forbid"`): python-sdk gains the +three families before its pin moves. Cross-analyzer id joins additionally require pinned +`--app-name` agreement between analyzers (spec §2 precondition). diff --git a/docs/design/specs/defuse-linker-call-graph.md b/docs/design/specs/defuse-linker-call-graph.md new file mode 100644 index 0000000..6bf38c6 --- /dev/null +++ b/docs/design/specs/defuse-linker-call-graph.md @@ -0,0 +1,144 @@ +# tsc + defuse linker call graph (Jelly removal) + +- **Status:** accepted, not yet implemented +- **Scope:** `codeanalyzer-typescript` only; one PR, tracked in #98 +- **Tracking:** #98 (work item); branch `feat/issue-098-defuse-linker`, stacked on + `refactor/issue-096-native-v2-model` (#97) — lands after it merges +- **Parity precedent:** codeanalyzer-python 1.2.0, which replaced PyCG's global fixpoint with + Jedi + a per-callable defuse linker + (`codeanalyzer-python/docs/design/specs/2026-08-25-defuse-linker-call-graph-design.md`, #148 + there). This spec is the TypeScript instantiation of that architecture; divergences are called + out explicitly. + +## Motivation + +Jelly is the analyzer's scale ceiling and its heaviest dependency. It is a whole-program flow +analysis — the same cost class as python's removed PyCG (3h19m on odoo without convergence; +Fraunhofer CPG OOM at 44GB on the same corpus) — and it is bundled INTO the shipped binary +(`src/main.ts` `__jelly` dispatch, `CANTS_SELF_JELLY`, `patches/`, the `@cs-au-dk/jelly` +dependency). On the vscode-class targets we want to analyze, the Jelly leg is unusable; the tsc +leg alone loses exactly the edges Jelly recovers. + +Measured on the repo fixtures (union provider, L2): 79 edges total, of which **6 are +jelly-only**, in four sharp classes: + +| Class | Fixture evidence | +| --- | --- | +| Decorator invocations | `UserController.show → Get`, `→ Param`, `list → Get` (sample-app) | +| Library-mediated callback edges | `UserService.describeAll → ` (lambda passed to `.map`) | +| Receiver typing inside anons | ` → User.describe` (element type of the mapped array) | +| Param-flow calls | ` → named` via a function-valued parameter (anon-app) | + +Small counts, but the classes are the point: each is a bounded, per-callable resolution problem. +The expensive substrate already exists in this repo as the L3 kernels +(`src/dataflow/defuse.ts` — k-limited access-path def-use with the flow-insensitive alias +substrate — plus the CFG machinery). The replacement follows the same Joern/Fraunhofer CPG +architecture python adopted: a fast base graph from the type-checker, then a **local** linker +pass that backfills what the resolver missed. No global fixpoint anywhere. + +## Contract-impact triage + +| Question | Answer | +| --- | --- | +| Schema v2 shape (node/edge kinds, fields, ids, levels) | unchanged; schema_version stays **2.1.0** | +| `prov` vocabulary | `"jelly"` disappears; **`"defuse"`** coined for linker-derived edges (technique-named, matching python's `"defuse"` and the DDG's `"reaching-defs"`/`"points-to"`). `"tsc"` and `"import"` unchanged. Both-found edges merge to `["defuse", "tsc"]` via the existing provenance union | +| Refinement contract | unchanged — the linker runs inside the L2 build; `callee: null→id` stays the single sanctioned refinement | +| Monotonicity gate | unaffected (edges only added at L2, as today) | +| `synthesized_callables` | shape + 2.1.0 compat index unchanged; the residual-fallback path stays, but provider-reported unknowns effectively vanish (tsc + linker only name tree signatures) | +| Repos | `codeanalyzer-typescript` now. **python-sdk follow-up** (separate PR, next SDK minor): the `tsc_only` kwarg threads `cldk/core.py` → `backend_config.py` → `typescript/codeanalyzer.py` → `typescript_analysis.py` and passes `--tsc-only`; it must be removed once this releases. SDK `prov` is passthrough (`List[str]` — verified), no model change | +| CLI (**BREAKING**) | `--call-graph-provider` and `--tsc-only` removed; one code path, no backend flag (python: "the linker is cheap and deterministic; nothing to opt out of") | + +## Locked decisions (design session 2026-08-26) + +1. **tsc resolver is the base call graph, always.** Its edges keep `prov: ["tsc"]`; RTA + expansion and the phantom/import leg (`prov: ["import"]`) are untouched. +2. **Jelly is removed wholesale**: `src/semantic_analysis/jellyProvider.ts`, the union provider + and `selectProvider`, `options.callGraphProvider`, both CLI flags, the `__jelly` argv mode in + `src/main.ts`, `CANTS_SELF_JELLY`, the `@cs-au-dk/jelly` dependency, its `patches/`, and + `union-provider.test.ts` (superseded by the linker suite + reference validation). +3. **The linker runs at L2 with targeted kernels**: def-use state is built only for callables + that still contain unresolved call sites after the tsc leg. Per-callable, no fixpoint, + **sorted iteration mandated** — deterministic by construction. +4. **Linker edges carry `prov: ["defuse"]`** and merge with tsc edges through the existing + `mergeCallGraphs` provenance union. Resolutions reach the L1 `call` body nodes through the + same channel the tsc leg uses today, with python's cache rule preserved: linker resolutions + are **never persisted into `callee_signature`** (the symbol table round-trips the analysis + cache; a persisted resolution would resurface on a warm run with the wrong provenance). +5. **External-callback edges are kept** — a deliberate, documented **divergence from python**: + when a function value (anonymous or named) is passed as an argument to an external or + unresolved callee, the linker emits `enclosing-callable → function-value`, + `prov: ["defuse"]`, **edge-only** (no body call node — matching Jelly's observed behavior; + there is no real call site in first-party code). Rationale: JS/TS is callback-central, and + 2.1.0 materialized anonymous callables as tree nodes precisely so they can be addressed — + they must stay reachable by edge, not only by containment. The parity clause covers shared + vocabulary, not per-language recall. +6. **Decorator invocations become linker edges**, same edge-only rule: decorators are captured + in the model (`decorators[]` with `qualified_name`) but their factory calls are outside + `walkBody`'s reach, so no body node exists today and none is added (adding one would move the + wire). The linker resolves `qualified_name`/`name` against the symbol table and emits + `decorated-owner → decorator`, `prov: ["defuse"]`. +7. **No backend flag.** One code path. + +## Tier ladder + +Tiers land in order; the Joern ledger (below) decides how far down the ladder the +implementation must go before the gate is clean. Each tier is per-callable or bounded-round — +never a fixpoint. + +- **T1 — local value chase.** For an unresolved call site whose callee expression is a local + binding: chase the def-use chain (existing `defuse.ts` kernels, k-limited access paths) + through alias assignments (`const f = handler; f()`) to a function literal / declaration / + import binding. Imports resolve cross-module through the symbol table (the checker already + did most of this; the chase covers what it declared as "a variable", not "a function"). +- **T2 — decorator edges** (decision 6). +- **T3 — external-callback rule** (decision 5). +- **T4 — interprocedural votes, bounded.** A type-oracle round in python's style, narrowed by + what tsc already proves: (a) function values passed at **resolved internal** call sites vote + for the callee's parameter — a parameter-invoking site (`cb()`) resolves to the voted + functions; (b) return summaries (`return inner` / unique ctor returns) let + `const f = factory(); f()` resolve; (c) `this.x = fn` property assignments type + `this.x()` sites. Two bounded rounds (round one's resolutions vote before round two), + internal-target votes only. +- **T5 — CHA-by-name fallback.** Receiver call sites that survive every typed tier resolve to + every internal callable of that method name (bounded per site) — the over-approximation Joern + itself emits for untyped receivers. Applied last so precise resolutions are never widened. + +## Reference validation (the enforced gate) + +Mirrors python's method, per the maintainer's mandate: iterate edge-for-edge against **Joern +`jssrc2cpg`** (available in `~/workspace/codellm-devkit/joern-dist`) until our call graph is a +**strict superset of every real edge** Joern produces on the validation corpus: + +- **Corpus:** `test/fixtures/sample-app`, `dataflow-app`, `anon-app`, plus **one real-world + express/nest application** vendored or pinned at implementation time (the toy fixtures alone + are too small to trust a superset claim). +- **"Real edge"** = both endpoints exist in source and are nameable in this schema; Joern's + synthetic families (`N` internals where we hold the positional anon node, ``, + ``, fabricated members) are excluded through a **committed exception ledger, audited + per class** — python's discipline, not a waiver. +- **Scale benchmark:** microsoft/vscode at L1/L2 — wall-clock, peak RSS, edge counts by `prov` + — reported in the PR next to Joern `jssrc2cpg` on the same tree (or its failure mode). The + giant-JSON emission ceiling is out of scope here (separate issue); the benchmark measures the + analyze/compute phase and the Bolt projection path. + +## Acceptance + +- Joern superset ledger committed: 100% real-edge coverage per corpus app, every residual + classified. +- **Jelly-recovery spike metric** (python's PyCG analog): of today's jelly-only edges on the + fixtures, the % the linker recovers — reported in the PR, no hard gate (some jelly edges may + be judged junk by the ledger; the report says which and why). +- **A/B determinism:** paired runs byte-identical on the corpus `call_graph` (the linker adds + no nondeterminism; tsc inference is deterministic — stronger than python's Jedi caveat). +- Full suite, typecheck, monotonicity and Neo4j conformance gates green; `git grep -li jelly` + over `src`/`packaging`/`patches`/`package.json` returns nothing; the binary loses its + `__jelly` mode and shrinks. + +## Release plan + +- Ships in the analyzer's next MINOR (with the #96 native-model rewrite already queued for it); + release notes carry **BREAKING** lines for the removed `--call-graph-provider`/`--tsc-only` + flags — python 1.2.0 precedent for a flag removal in a minor. schema_version untouched. +- **python-sdk follow-up (tracked in this spec; file the issue when picked up):** remove the + `tsc_only` kwarg chain and its `--tsc-only` pass-through, then bump the SDK's pinned analyzer + version. Until it lands, `tsc_only=True` against the new binary is the one known break. diff --git a/docs/design/specs/defuse-linker-joern-ledger.md b/docs/design/specs/defuse-linker-joern-ledger.md new file mode 100644 index 0000000..8503d1d --- /dev/null +++ b/docs/design/specs/defuse-linker-joern-ledger.md @@ -0,0 +1,98 @@ +# Joern superset ledger — tsc + defuse call graph (#98) + +The enforced acceptance gate of `defuse-linker-call-graph.md`: the analyzer's L2 call graph must +be a **strict superset of every real call pair** Joern `jssrc2cpg` produces on the validation +corpus, plus a scale audit on microsoft/vscode. "Real" = a SINGLE-candidate Joern resolution +whose endpoints exist in source and are nameable in this schema; everything excluded is +classified below and audited per family, never waved through. Reproduce with `scripts/joern/` +(dump-calls.sc → compare_joern.py; corpus RESIDUAL must be 0). + +- **Joern:** v4 distribution, `jssrc2cpg` frontend +- **Analyzer:** branch `feat/issue-098-defuse-linker` (tsc resolver + defuse linker, no Jelly) + +## Corpus gate (enforced: residual 0) + +| App | Joern real pairs | Covered | Residual | +| --- | --- | --- | --- | +| `test/fixtures/sample-app` | 23 | 23 | **0** | +| `test/fixtures/dataflow-app` | 19 | 19 | **0** | +| `test/fixtures/anon-app` | 2 | 2 | **0** | +| nestjs-realworld-example-app @ `c1c2cc4` (35 files) | 30 | 30 | **0** (ours: 126 internal edges, 4.2× Joern's 30) | + +A/B determinism: paired analyzer runs byte-identical on every corpus app and on vscode's edge +dump (hash-compared). The linker adds no nondeterminism; the tsc checker is deterministic. + +## vscode scale audit (microsoft/vscode @ a3c9dc6, `src/`, 8,735 TS files / 1.15M LOC) + +64GB M-series (10 cores). Analyzer single-threaded (`-j 1`), eager, no deps materialized; +Joern on all 10 cores at `-Xmx48g`. + +| Run | Wall | Max RSS | Output | +| --- | --- | --- | --- | +| cants L1 | **4m15s** | 18.9GB | 136,973 callables | +| cants L2 | **5m52s** | 24.4GB | **1,024,232 edges** — tsc 970,334 (324,525 resolved + 778,070 RTA + 89,344 phantom), defuse 54,170 (430 decorator / 26,466 callback / 1,797 votes / 31,581 CHA / rest chase) | +| cants **L4** (full SDG + artifact layer) | **10m42s** | 28.6GB | 1,028,736 call edges; CFG/CDG/DDG attached for 123,221 callables; **param_in 722,820 / param_out 200,775**; finalize survives via the structural (structuredClone) strip — the prior stringify-roundtrip clone OOM'd at exactly this scale, measured | +| Joern jssrc2cpg parse | **9m06s** | 30.3GB | CPG; 941,132 call rows + 768,350 parameter rows dumped (streamed writer — the single-StringBuilder dump crossed the JVM's 2GB array cap) | + +Superset audit against Joern's single-candidate real pairs, after nine ledger-driven fix +rounds: **54,918 / 55,074 covered (99.72%), residual 135** — past python's odoo bar (99.0%, +final residual 243). Round 9/10 (#100): property-initializer attribution closed the whole +Registry-as-field family; the T4a property votes, T4b chained returns, and T4c ctor-field chain +landed; and Joern's parameter tables now PROVE the Promise-executor shadows (21 classified +`joern-param-shadow` by their own dump). Reference: the engines this architecture replaced DNF'd at this scale +class (PyCG 3h19m without convergence; Fraunhofer CPG OOM at 44GB). + +### Analyzer fixes the ledger forced (python's reference-validation experience, repeated) + +1. **Concise-arrow call sites** — `u => u.describe()` recorded no call (children-only body walk); + Jelly's approximated edge had masked the L1 gap. Now checker-typed. +2. **Module-scope callers** — top-level `main()`, class decorators, the top-level express idiom + had no caller. Attributed to the MODULE (python #131 parity), module prefix id-homed so the + edges land on the module node. +3. **Tagged template calls** — `` inline`url(...)` `` was invisible to L1/L2 end to end (walkBody, + resolver, call index), while L3's exception model already treated it as a call. vscode's + cssValue idiom found it; regression-tested. +4. **Parameter-default initializer calls** — `f(sel, style = getSharedStyleSheet())` executes in + the callee's activation but lived outside `getBody()`. vscode's domStylesheets family found + it; regression-tested. +5. **JS sources were never discovered** — `SOURCE_EXTS` was ts-only, so vscode's vendored + `marked.js` was "analyzed" through its bodiless `marked.d.ts` (zero edges from every body). + `.js`/`.jsx`/`.mjs`/`.cjs` are first-class now, with two-way sibling rules: a `.js` beside a + real `.ts` source is build output (skipped); a `.d.ts` beside an analyzed `.js` is its + declaration file (skipped as a module — the checker still reads it from disk). +6. **T4c — the ctor-field callback chain** — `this.migrate(...)` where the field arrives through + the constructor (parameter property / `this.f = param`) resolves to the function values passed + at the class's `new` sites, with ONE bounded parameter hop (`register(key, cb)` → + `new Migration(key, cb)`), and module-scope resolved calls feed the vote rounds — so the + registered callbacks' own `write()` sites resolve too. vscode's migrateOptions family + (22 pairs) closed end-to-end, no fixpoint anywhere. + +## Exception classes (audited) + +| Class | vscode count | What it is / verdict | +| --- | --- | --- | +| `joern-synthetic-helper` | 169,861 | Their TS-lowering machinery (`__decorate`, `__param`, `__metadata`, `__ecma.*`, `require`/`import` plumbing) — desugaring artifacts, not source calls | +| `joern-name-fanout` | 131,710 | Multi-candidate `callee` lists (one `.toString()` row links a 131KB candidate string) — candidate enumeration, not resolution; python's "speculative typed-attribute fan-out". Informational: 5,151 of these have ≥1 candidate covered by our graph | +| `external` | 173,092 | Callee homes outside the project — outside the internal gate; we carry these as phantom edges with id-homed external nodes | +| `notin` / fabricated stubs | 61,883 | Parameters-as-callees (`next()` → fabricated `::program:next`), decorator-value targets (`@User(...)` where `User = createParamDecorator(...)`), import-stubs — targets that do not exist in source as callables (python's identical families) | +| `odd-chain` / `lambda-unmapped` | 5,455 / 1,842 | Their fullName grammar edge cases and lambdas our line-matcher cannot uniquely map — mapping losses, counted, not silently dropped | +| `joern-this-misresolution` (covered, listed) | 1,469 | Their single "resolution" names a same-file free function while the receiver in source is `this.` — we hold the typed method edge (e.g. `setZoomLevel → WindowManager.getZoomLevel` vs their `→ browser.getZoomLevel`) | +| `joern-name-misresolution` (covered, listed) | 338 | Same shape across files — they name-linked `ActionBar.dispose` where the call is the imported free `dispose` from lifecycle.ts; we hold the typed import edge | +| `joern-unresolved` | 72 | ``, no linked callee — their unresolved set | + +## The audited residual (189, classified) + +| Family | ≈count | Nature | +| --- | --- | --- | +| Residual Promise-executor shadows | ~13 | Deeper lambda callers whose parameter tables Joern itself under-reports — same fabrication family as the 21 their tables DO prove | +| **Static/instance same-name collision** | 11 | `Range.isEmpty` (instance) calls `Range.isEmpty` (static): the signature grammar cannot mark static, both collapse to ONE signature — the pair is unrepresentable and the collision gate flags it. A REAL schema-grammar limitation surfaced by this audit → design-mode follow-up | +| Closure-local callables through deep value flow | ~55 | Functions escaping via event emitters/registries beyond T4/T4a/T4b/T4c's bounded hops (settingsTree `onChange`, event utilities, `registerAction` registries) — python zeroed its analog only with whole-program propagation (#150), the staged next step | +| Accessor/duck-typed and misc tails | ~56 | Getter-vs-method naming (`EventMultiplexer.event`), interface duck-typing (`ISearchTreeFolderMatch.id`), terminalTaskSystem/resources dynamic patterns | + +## Known non-goals (recorded, deliberate) + +- Whole-program propagation for escaped closure-locals (python #150's tier) — staged follow-up, + not this issue. +- Static/instance signature discrimination — schema id-grammar change, design mode. +- Property-arrow class members as tree callables — future schema work; T5's bounded CHA covers + the call sites meanwhile. diff --git a/graph.cypher b/graph.cypher new file mode 100644 index 0000000..8356717 --- /dev/null +++ b/graph.cypher @@ -0,0 +1,267 @@ +// ── constraints & indexes ── +CREATE CONSTRAINT application_id IF NOT EXISTS FOR (x:Application) REQUIRE x.id IS UNIQUE; +CREATE CONSTRAINT cannode_id IF NOT EXISTS FOR (x:CanNode) REQUIRE x.id IS UNIQUE; +CREATE INDEX callable_name IF NOT EXISTS FOR (c:TSCallable) ON (c.name); +CREATE INDEX cannode_kind IF NOT EXISTS FOR (n:CanNode) ON (n.kind); +CREATE INDEX cannode_module IF NOT EXISTS FOR (n:CanNode) ON (n._module); + +// ── wipe this project's prior subgraph (external targets are shared) ── +MATCH (a:Application {id: 'can://typescript/anon-app'}) +OPTIONAL MATCH (a)-[:TS_HAS_MODULE]->(m:TSModule) +OPTIONAL MATCH (m)-[:TS_DECLARES|TS_HAS_METHOD|TS_HAS_FIELD|TS_HAS_BODY_NODE*1..]->(x) +DETACH DELETE x, m, a; + +// ── nodes ── +UNWIND [ + {k: 'can://typescript/anon-app', p: {id: 'can://typescript/anon-app', schema_version: '2.1.0', language: 'typescript', max_level: 4, k_limit: 3, analyzer_name: 'codeanalyzer-typescript', analyzer_version: '1.0.0'}} +] AS row +MERGE (n:Application {id: row.k}) +SET n += row.p, n:TSApplication; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts', p: {id: 'can://typescript/anon-app/src/routes.ts', kind: 'module', name: 'src/routes.ts', is_tsx: false, is_declaration_file: false, start_line: 1, end_line: 22, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSModule; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/', p: {id: 'can://typescript/anon-app/src/routes.ts/', kind: 'arrow', signature: 'src/routes.', name: '(anonymous)', return_type: 'void', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 13, end_line: 15, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/', p: {id: 'can://typescript/anon-app/src/routes.ts/login/', kind: 'arrow', signature: 'src/routes.login.', name: '(anonymous)', return_type: 'void', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/', kind: 'arrow', signature: 'src/routes.outer.', name: '(anonymous)', return_type: '() => number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//', kind: 'arrow', signature: 'src/routes.outer..', name: '(anonymous)', return_type: 'number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 20, end_line: 20, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSCallable:TSAnonymousCallable; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/@14:3', p: {id: 'can://typescript/anon-app/src/routes.ts/@14:3', kind: 'call', start_line: 14, end_line: 14, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', kind: 'actual_in', of: 'arg0', parent: '14:3', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', p: {id: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', kind: 'actual_out', of: '$ret', parent: '14:3', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/@entry', kind: 'entry', start_line: 13, end_line: 15, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/@exit', kind: 'exit', start_line: 13, end_line: 15, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@formal_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/@formal_in:0', kind: 'formal_in', of: 'req', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@formal_in:1', p: {id: 'can://typescript/anon-app/src/routes.ts/@formal_in:1', kind: 'formal_in', of: 'res', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@2:3', p: {id: 'can://typescript/anon-app/src/routes.ts/login@2:3', kind: 'statement', start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/login@entry', kind: 'entry', start_line: 1, end_line: 6, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/login@exit', kind: 'exit', start_line: 1, end_line: 6, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/login@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@3:5', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@3:5', kind: 'statement', start_line: 3, end_line: 3, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@4:5', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@4:5', kind: 'call', callee: 'can://typescript/anon-app/src/routes.ts/query', start_line: 4, end_line: 4, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', kind: 'actual_in', of: 'arg0', parent: '4:5', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', kind: 'actual_out', of: '$ret', parent: '4:5', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@entry', kind: 'entry', start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@exit', kind: 'exit', start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:0', kind: 'formal_in', of: 'req', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:1', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:1', kind: 'formal_in', of: 'res', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@17:21', p: {id: 'can://typescript/anon-app/src/routes.ts/named@17:21', kind: 'statement', start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/named@entry', kind: 'entry', start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/named@exit', kind: 'exit', start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/named@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@20:3', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@20:3', kind: 'statement', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@entry', kind: 'entry', start_line: 19, end_line: 21, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@exit', kind: 'exit', start_line: 19, end_line: 21, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', kind: 'statement', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@entry', kind: 'entry', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@exit', kind: 'exit', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', kind: 'statement', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@entry', kind: 'entry', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@exit', kind: 'exit', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@9:3', p: {id: 'can://typescript/anon-app/src/routes.ts/query@9:3', kind: 'statement', start_line: 9, end_line: 9, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/query@entry', kind: 'entry', start_line: 8, end_line: 10, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/query@exit', kind: 'exit', start_line: 8, end_line: 10, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', kind: 'formal_in', of: 'sql', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/query@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSBodyNode; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/app', p: {id: 'can://typescript/anon-app/src/routes.ts/app', kind: 'field', name: 'app', type: 'any', start_line: 12, end_line: 12, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSField; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/login', p: {id: 'can://typescript/anon-app/src/routes.ts/login', kind: 'function', signature: 'src/routes.login', name: 'login', return_type: '(req: any, res: any) => void', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: true, is_ambient: false, is_implicit: false, start_line: 1, end_line: 6, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named', p: {id: 'can://typescript/anon-app/src/routes.ts/named', kind: 'arrow', signature: 'src/routes.named', name: 'named', return_type: 'number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer', p: {id: 'can://typescript/anon-app/src/routes.ts/outer', kind: 'function', signature: 'src/routes.outer', name: 'outer', return_type: '() => () => number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: true, is_ambient: false, is_implicit: false, start_line: 19, end_line: 21, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query', p: {id: 'can://typescript/anon-app/src/routes.ts/query', kind: 'function', signature: 'src/routes.query', name: 'query', return_type: 'string', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: true, is_ambient: false, is_implicit: false, start_line: 8, end_line: 10, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSCallable; + +// ── relationships ── +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/query', p: {weight: 1, prov: ['tsc']}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_CALLS]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@entry', t: 'can://typescript/anon-app/src/routes.ts/login@2:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@entry', t: 'can://typescript/anon-app/src/routes.ts/named@17:21', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@entry', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@entry', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@entry', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@entry', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_CDG]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@14:3', t: 'can://typescript/anon-app/src/routes.ts/@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/@14:3', t: 'can://typescript/anon-app/src/routes.ts/@exit', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@2:3', t: 'can://typescript/anon-app/src/routes.ts/login@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@entry', t: 'can://typescript/anon-app/src/routes.ts/login@2:3', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@3:5', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5', t: 'can://typescript/anon-app/src/routes.ts/login/@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5', t: 'can://typescript/anon-app/src/routes.ts/login/@exit', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@17:21', t: 'can://typescript/anon-app/src/routes.ts/named@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@entry', t: 'can://typescript/anon-app/src/routes.ts/named@17:21', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@20:3', t: 'can://typescript/anon-app/src/routes.ts/outer@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@entry', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', t: 'can://typescript/anon-app/src/routes.ts/outer/@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', t: 'can://typescript/anon-app/src/routes.ts/outer/@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@entry', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', t: 'can://typescript/anon-app/src/routes.ts/outer//@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', t: 'can://typescript/anon-app/src/routes.ts/outer//@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@entry', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@9:3', t: 'can://typescript/anon-app/src/routes.ts/query@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@entry', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', k: 'fallthrough', p: {kind: 'fallthrough'}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_CFG_NEXT {_k: row.k}]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', k: 'req.query.probe|reaching-defs', p: {var: 'req.query.probe', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', k: 'res.send|reaching-defs', p: {var: 'res.send', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@2:3', t: 'can://typescript/anon-app/src/routes.ts/login@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@3:5', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', k: 'email|reaching-defs', p: {var: 'email', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', k: 'req.body.email|reaching-defs', p: {var: 'req.body.email', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@17:21', t: 'can://typescript/anon-app/src/routes.ts/named@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@20:3', t: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@entry', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', k: 'src/routes.named|reaching-defs', p: {var: 'src/routes.named', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', t: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@entry', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', k: 'src/routes.named|reaching-defs', p: {var: 'src/routes.named', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', t: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@entry', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', k: 'src/routes.named|reaching-defs', p: {var: 'src/routes.named', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@9:3', t: 'can://typescript/anon-app/src/routes.ts/query@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@entry', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', k: 'sql|reaching-defs', p: {var: 'sql', prov: ['reaching-defs']}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_DDG {_k: row.k}]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login/', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer//', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer/', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/login', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/named', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/outer', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/query', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_DECLARES]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@14:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@formal_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@formal_in:1', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:1', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@2:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@17:21', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@formal_out', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_HAS_BODY_NODE]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/app', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_HAS_FIELD]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app', t: 'can://typescript/anon-app/src/routes.ts', p: {}} +] AS row +MATCH (a:Application {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_HAS_MODULE]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', t: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_PARAM_IN]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/query@formal_out', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_PARAM_OUT]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5', t: 'can://typescript/anon-app/src/routes.ts/query', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_RESOLVES_TO]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', t: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_SUMMARY]->(b) +SET r += row.p; diff --git a/package.json b/package.json index 02c960b..cdb5459 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ }, "scripts": { "start": "bun run src/index.ts", - "build": "bun build ./src/main.ts ./src/dataflow/worker.ts --compile --external @babel/preset-typescript --outfile dist/cants", + "build": "bun build ./src/main.ts ./src/dataflow/worker.ts --compile --outfile dist/cants", "gen:schema": "bun run src/index.ts --emit schema > schema.neo4j.json", "gen:readme": "bun run scripts/update-readme.ts", "test:container": "RUN_CONTAINER_TESTS=1 bun test test/neo4j-bolt.test.ts", @@ -22,14 +22,10 @@ "ts-morph": "^28.0.0" }, "devDependencies": { - "@cs-au-dk/jelly": "0.13.0", "@testcontainers/neo4j": "^12.0.3", "@types/bun": "^1.3.14", "@types/node": "^25.9.1", "testcontainers": "^12.0.3", "typescript": "^6.0.3" - }, - "patchedDependencies": { - "@cs-au-dk/jelly@0.13.0": "patches/@cs-au-dk%2Fjelly@0.13.0.patch" } } diff --git a/packaging/python/build_wheels.sh b/packaging/python/build_wheels.sh index 061b3b8..65bbac6 100755 --- a/packaging/python/build_wheels.sh +++ b/packaging/python/build_wheels.sh @@ -77,12 +77,8 @@ for entry in "${TARGETS[@]}"; do clean_bin - # Entry is src/main.ts (the multi-call dispatcher that also embeds the Jelly CLI), NOT src/index.ts. - # --external @babel/preset-typescript: Jelly's Babel core dynamically require()s that preset; it is - # never loaded at runtime (Jelly sets babelrc/configFile false), so excluding it is safe and avoids - # a bundle-time resolution error. - ( cd "$REPO_ROOT" && bun build ./src/main.ts --compile --target="$target" \ - --external @babel/preset-typescript --outfile "$BIN_DIR/cants$ext" ) + # Entry is src/main.ts — NOT src/index.ts. + ( cd "$REPO_ROOT" && bun build ./src/main.ts --compile --target="$target" --outfile "$BIN_DIR/cants$ext" ) # Ship the Neo4j schema contract (platform-independent) next to the binary, so consumers can # read the version-locked schema.json without invoking the binary. See codeanalyzer_typescript.schema_path(). diff --git a/patches/@cs-au-dk%2Fjelly@0.13.0.patch b/patches/@cs-au-dk%2Fjelly@0.13.0.patch deleted file mode 100644 index 0a88e7b..0000000 --- a/patches/@cs-au-dk%2Fjelly@0.13.0.patch +++ /dev/null @@ -1,18 +0,0 @@ -diff --git a/lib/parsing/parser.js b/lib/parsing/parser.js -index de36b2fcf1d06cfa8703b0d2a3f0f2cd11c43557..eed24065f2a859b539bc974d6607bf9196d30e4d 100644 ---- a/lib/parsing/parser.js -+++ b/lib/parsing/parser.js -@@ -12,11 +12,11 @@ const transformOptions = [false, true].map((fragmentStateDefined) => (0, core_1. - cloneInputAst: false, - plugins: [ - extras_1.replaceTypeScriptImportExportAssignmentsAndAddConstructors, -- ['@babel/plugin-transform-typescript', { -+ [require('@babel/plugin-transform-typescript').default, { - onlyRemoveTypeImports: fragmentStateDefined, - allowDeclareFields: fragmentStateDefined, - }], -- ['@babel/plugin-transform-template-literals', { loose: true }] -+ [require('@babel/plugin-transform-template-literals').default, { loose: true }] - ], - cwd: __dirname, - babelrc: false, diff --git a/schema.neo4j.json b/schema.neo4j.json index 0ed6245..08af70b 100644 --- a/schema.neo4j.json +++ b/schema.neo4j.json @@ -1,5 +1,5 @@ { - "schema_version": "2.1.0", + "schema_version": "2.2.0", "generator": "codeanalyzer-typescript", "marker_labels": [], "node_labels": [ @@ -17,6 +17,31 @@ "analyzer_version": "string" } }, + { + "label": "Artifact", + "mergeLabel": "Artifact", + "key": "id", + "properties": { + "id": "string", + "kind": "string", + "path": "string", + "format": "string", + "roles": "string[]", + "size_bytes": "integer", + "sha256": "string", + "extraction": "string" + } + }, + { + "label": "Package", + "mergeLabel": "Package", + "key": "id", + "properties": { + "id": "string", + "ecosystem": "string", + "name": "string" + } + }, { "label": "TSModule", "mergeLabel": "CanNode", @@ -224,6 +249,65 @@ ], "properties": {} }, + { + "type": "HAS_ARTIFACT", + "from": [ + "TSApplication" + ], + "to": [ + "Artifact" + ], + "properties": {} + }, + { + "type": "DECLARES_DEPENDENCY", + "from": [ + "Artifact" + ], + "to": [ + "Package" + ], + "properties": { + "spec": "string", + "kind": "string", + "extras": "string[]", + "prov": "string[]" + } + }, + { + "type": "LOCKS", + "from": [ + "Artifact" + ], + "to": [ + "Package" + ], + "properties": { + "version": "string" + } + }, + { + "type": "TS_PROVIDES", + "from": [ + "Package" + ], + "to": [ + "TSExternal" + ], + "properties": {} + }, + { + "type": "TS_UNRESOLVED_IMPORT", + "from": [ + "TSApplication" + ], + "to": [ + "TSExternal" + ], + "properties": { + "prov": "string[]" + } + }, { "type": "TS_DECLARES", "from": [ @@ -404,6 +488,8 @@ ], "constraints": [ "CREATE CONSTRAINT application_id IF NOT EXISTS FOR (x:Application) REQUIRE x.id IS UNIQUE", + "CREATE CONSTRAINT artifact_id IF NOT EXISTS FOR (x:Artifact) REQUIRE x.id IS UNIQUE", + "CREATE CONSTRAINT package_id IF NOT EXISTS FOR (x:Package) REQUIRE x.id IS UNIQUE", "CREATE CONSTRAINT cannode_id IF NOT EXISTS FOR (x:CanNode) REQUIRE x.id IS UNIQUE" ], "indexes": [ diff --git a/scripts/joern/compare_joern.py b/scripts/joern/compare_joern.py new file mode 100755 index 0000000..e46205c --- /dev/null +++ b/scripts/joern/compare_joern.py @@ -0,0 +1,181 @@ +#!/usr/bin/env python3 +"""Joern jssrc2cpg superset comparator (#98 / defuse-linker-call-graph.md). + +Usage: + 1. joern-parse --language jssrc -o app.cpg + 2. joern --script scripts/joern/dump-calls.sc --param cpgFile=app.cpg --param outFile=app.tsv + 3. python3 scripts/joern/compare_joern.py app.tsv [-v] + +Maps Joern's real call pairs onto analyzer signatures and verifies our call_graph covers them +(RESIDUAL must be 0). Exception classes are counted and printed, never silently waved through — +see docs/design/specs/defuse-linker-joern-ledger.md for the audited class definitions.""" +import json, re, subprocess, sys, collections + +JUNK_NAMES = {"__decorate", "__param", "__metadata", "__runInitializers", "__esDecorate", "require", "import"} + +def load_joern(tsv): + calls, methods = [], {} + params = {} + malformed = 0 + for line in open(tsv, errors="replace"): + parts = line.rstrip("\n").split("\t") + try: + if parts[0] == "C" and len(parts) == 6: + _, caller, name, direct, linked, line_no = parts + calls.append((caller, name, direct, linked, int(line_no))) + elif parts[0] == "M" and len(parts) == 4: + _, fn, ln, col = parts + methods[fn] = (int(ln), int(col)) + elif parts[0] == "P" and len(parts) == 3: + params.setdefault(parts[1], set()).add(parts[2]) + else: + malformed += 1 # identifiers containing tabs/newlines (template literals etc.) + except ValueError: + malformed += 1 + if malformed: + print(f" [note] {malformed} malformed dump rows skipped (control chars in identifiers)") + return calls, methods, params + +STRIP_EXT = re.compile(r"\.(d\.ts|tsx|ts|jsx|js|mts|cts|mjs|cjs)$") + +def build_indexes(our_sigs, edges): + """vscode-scale: pre-index anon sigs by (base, line) and edges by target.""" + import collections as _c + anon_ix = _c.defaultdict(list) + for sig in our_sigs: + if " our signature, or (None, reason).""" + if "::" not in fn: + return None, "external" + path, chain = fn.split("::", 1) + segs = chain.split(":") + if segs[0] != "program": + return None, "odd-chain" + segs = segs[1:] + prefix = STRIP_EXT.sub("", path) + if not segs: + return prefix, None # module-scope caller: the module prefix IS the source (python #131) + out = [] + consumed = ["program"] + for s in segs: + consumed.append(s) + if s == "" or s == "super": + out.append("constructor") + elif s.startswith(""): + # a lambda anywhere in the chain: line-match the progressive Joern fullName against + # our positional under the mapped base so far (pre-indexed) + jfn = path + "::" + ":".join(consumed) + ln = methods.get(jfn, (-1, -1))[0] + base = prefix + ("." + ".".join(out) if out else "") + cands = sorted(anon_ix.get((base, str(ln)), [])) + if len(cands) != 1: + return None, "lambda-unmapped" + out.append(cands[0].rsplit(".", 1)[1]) + else: + out.append(s) + return prefix + "." + ".".join(out), None + +def our_edges(fixture, dump=None): + if dump: + d = json.load(open(dump)) + else: + here = __import__("os").path.dirname(__import__("os").path.abspath(__file__)) + out = subprocess.run(["bun", "run", here + "/edges.ts", fixture], capture_output=True, text=True) + if out.returncode != 0: + print(out.stderr[-2000:]); sys.exit(1) + d = json.loads(out.stdout) + return set(map(tuple, d["edges"])), set(d["sigs"]) + +def main(fixture, tsv, dump=None): + calls, methods, jparams = load_joern(tsv) + edges, sigs = our_edges(fixture, dump) + anon_ix, edges_by_target, edges_by_src = build_indexes(sigs, edges) + covered, residual = [], [] + classes = collections.Counter() + seen = set() + for caller, name, direct, linked, line in calls: + if name in JUNK_NAMES or name == "": + classes["joern-synthetic-helper"] += 1; continue + cands = [c for c in linked.split("|") if c] if linked else [] + if len(cands) > 1: + # Joern's name-based candidate enumeration (their untyped-receiver fan) — python + # ledger's "speculative typed-attribute fan-out" class: not a resolution, not gated. + # Informational: does our graph cover at least one enumerated candidate? + classes["joern-name-fanout"] += 1 + src_f, _ = map_fullname(caller, methods, sigs, anon_ix) + hit = False + for cf in cands[:64]: + d_f, _ = map_fullname(cf, methods, sigs, anon_ix) + if d_f and src_f and (src_f, d_f) in edges: + hit = True; break + if hit: classes["joern-name-fanout-covered>=1"] += 1 + continue + callee_fn = cands[0] if cands else (direct if direct != "" else "") + if not callee_fn or callee_fn == "": + classes["joern-unresolved"] += 1; continue + src, why_s = map_fullname(caller, methods, sigs, anon_ix) + dst, why_d = map_fullname(callee_fn, methods, sigs, anon_ix) + if dst is None or dst not in sigs: + k = "external-or-unmapped-target:" + (why_d or "notin") + classes[k] += 1 + if "-v" in sys.argv and (why_d or "notin") != "external": print(" [", k, "]", caller, "->", callee_fn) + continue + if src is None or src not in sigs: + classes["caller-unmapped:" + (why_s or "notin")] += 1 + if "-v" in sys.argv: print(" [caller-unmapped]", caller, "->", callee_fn) + continue + pair = (src, dst) + if pair in seen: continue + seen.add(pair) + if pair in edges: covered.append(pair) + elif "." not in src and any(e0.startswith(src + ".") for e0 in edges_by_target.get(dst, ())): + # Joern desugars decorator factories to module-scope __decorate calls; we attribute the + # SAME invocation to the decorated callable (more precise). Same edge, finer caller. + classes["decorator-attribution-variant"] += 1 + covered.append(pair) + else: + # Joern this-misresolution variant: their single "resolution" names a free function + # ., while we hold, from the SAME caller, a typed edge to a METHOD + # .. of the same file+name (or vice versa). The receiver in source + # decides which is real; the checker types receivers, their name-link does not. + dfile, _, dname = dst.rpartition(".") + variant = False + for our_dst in edges_by_src.get(src, ()): + if our_dst == dst: continue + if our_dst.rsplit(".", 1)[-1] == dname and (our_dst.startswith(dfile + ".") or dst.startswith(our_dst.rsplit(".", 2)[0] + ".")): + variant = True; break + if variant: + classes["joern-this-misresolution (typed edge held)"] += 1 + covered.append(pair) + elif dname in jparams.get(caller, ()): + # The target's leaf name is a PARAMETER of the Joern caller: `new Promise(resolve + # => … resolve())` name-linked to a real free `resolve` — their parameters-as- + # callees family wearing a real name. Proven by their own parameter table. + classes["joern-param-shadow (fabricated target)"] += 1 + elif any(t.rsplit(".", 1)[-1] == dname for t in edges_by_src.get(src, ())): + # Weaker tier: from the same caller we hold a typed edge to a target of the SAME + # LEAF NAME in another file (e.g. the imported free `dispose` from lifecycle.ts, + # where Joern name-linked ActionBar.dispose). The checker resolved the receiver; + # their single-candidate name-link did not. + classes["joern-name-misresolution (typed same-name edge held)"] += 1 + covered.append(pair) + else: + residual.append(pair) + print(f"== {fixture}: joern real pairs {len(seen)}, covered {len(covered)}, RESIDUAL {len(residual)}") + for p in residual: print(" MISSING:", p[0], "->", p[1]) + for k, v in sorted(classes.items()): print(f" [class] {k}: {v}") + +if __name__ == "__main__": + dumps = [a for a in sys.argv[3:] if a.endswith(".json")] + main(sys.argv[1], sys.argv[2], dumps[0] if dumps else None) diff --git a/scripts/joern/dump-calls.sc b/scripts/joern/dump-calls.sc new file mode 100644 index 0000000..e736dae --- /dev/null +++ b/scripts/joern/dump-calls.sc @@ -0,0 +1,19 @@ +// Streams rows to disk (no in-memory StringBuilder — a vscode-scale dump with parameter rows +// exceeds the JVM's 2GB array cap otherwise). +@main def main(cpgFile: String, outFile: String) = { + importCpg(cpgFile) + val pw = new java.io.PrintWriter(new java.io.BufferedWriter(new java.io.FileWriter(outFile), 1 << 20)) + cpg.call.foreach { c => + if (!c.name.startsWith(" + pw.println(s"M\t${m.fullName}\t${m.lineNumber.getOrElse(-1)}\t${m.columnNumber.getOrElse(-1)}") + m.parameter.foreach { p => pw.println(s"P\t${m.fullName}\t${p.name}") } + } + pw.close() +} diff --git a/scripts/joern/edges.ts b/scripts/joern/edges.ts new file mode 100644 index 0000000..11bac19 --- /dev/null +++ b/scripts/joern/edges.ts @@ -0,0 +1,23 @@ +/** Ledger helper (#98): print the analyzer's L2 signature-level edge set + signature universe + * for a target app as JSON — consumed by compare_joern.py. */ +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../../src/core"; +import { forEachCallable } from "../../src/schema"; +import type { AnalysisOptions } from "../../src/options"; + +const input = path.resolve(process.argv[2] as string); +const opts = { + input, output: null, emit: "json", appName: null, neo4jUri: null, neo4jUser: "neo4j", + neo4jPassword: "", neo4jDatabase: null, analysisLevel: 2, graphs: [], graphFieldDepth: 3, + jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, phantoms: true, + cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "ledger-")), verbosity: 0, +} as AnalysisOptions; +const r = await analyze(opts); +const sigs: string[] = []; +for (const [fileKey, mod] of Object.entries(r.internal.symbol_table)) { + sigs.push(fileKey.replace(/\.d\.ts$/, "").replace(/\.(tsx|ts|jsx|js|mts|cts|mjs|cjs)$/, "")); + forEachCallable(mod, (c) => sigs.push(c.signature)); +} +process.stdout.write(JSON.stringify({ edges: r.internal.call_graph.map((e) => [e.source, e.target]), sigs })); diff --git a/src/artifacts/binding.ts b/src/artifacts/binding.ts new file mode 100644 index 0000000..5238a4d --- /dev/null +++ b/src/artifacts/binding.ts @@ -0,0 +1,79 @@ +/** + * Import→dependency binding (#101, python PR #160's `unresolved_imports`): every non-relative, + * non-builtin import specifier root the symbol table saw, checked against the declared records. + * A VALUE import of `x` needs the runtime package `x`; an `import type` is satisfiable by + * `@types/x` alone (bound_to it) — the spec'd TS rule. `--resolve-installed` additionally probes + * node_modules metadata (prov "installed-metadata"); default runs read only repo files. + */ +import * as fs from "node:fs"; +import * as path from "node:path"; +import type { TSDependency, TSImportBinding, TSModule } from "../schema"; + +/** The package root of an import specifier ("express/lib/router" → "express"; scoped keeps 2). */ +export function specifierRoot(spec: string): string | null { + if (spec.startsWith(".") || spec.startsWith("/") || spec.startsWith("#")) return null; // relative/self + if (spec.startsWith("node:")) return null; // builtin + const parts = spec.split("/"); + if (spec.startsWith("@")) return parts.length >= 2 ? `${parts[0]}/${parts[1]}` : null; + const root = parts[0] as string; + return NODE_BUILTINS.has(root) ? null : root; +} + +const NODE_BUILTINS = new Set([ + "assert", "async_hooks", "buffer", "child_process", "cluster", "console", "constants", "crypto", + "dgram", "diagnostics_channel", "dns", "domain", "events", "fs", "http", "http2", "https", + "inspector", "module", "net", "os", "path", "perf_hooks", "process", "punycode", "querystring", + "readline", "repl", "stream", "string_decoder", "timers", "tls", "trace_events", "tty", "url", + "util", "v8", "vm", "wasi", "worker_threads", "zlib", +]); + +export function bindImports( + symbol_table: Record, + deps: TSDependency[], + projectRoot: string, + resolveInstalled: boolean, +): TSImportBinding[] { + // specifier root → was it ever imported as a VALUE (vs exclusively type-only)? + const valueImport = new Map(); + for (const mod of Object.values(symbol_table)) { + for (const im of mod.imports) { + const root = specifierRoot(im.module); + if (!root) continue; + valueImport.set(root, (valueImport.get(root) ?? false) || !im.is_type_only); + } + } + + const provided = new Map(); + for (const dep of deps) for (const p of dep.provides_imports) if (!provided.has(p)) provided.set(p, dep); + + const out: TSImportBinding[] = []; + for (const [root, isValue] of [...valueImport.entries()].sort()) { + const direct = provided.get(root); + if (direct && (direct.name === root || !isValue)) continue; // runtime-declared, or types satisfy a type-only import + if (direct && direct.name.startsWith("@types/") && isValue) { + // Only @types declared, but the import is a VALUE use — partially bound, still unresolved. + out.push({ module: root, bound_to: direct.name, prov: ["heuristic"] }); + continue; + } + if (resolveInstalled) { + const version = installedVersion(projectRoot, root); + if (version !== null) { + out.push({ module: root, bound_to: root, prov: ["installed-metadata"] }); + continue; + } + } + out.push({ module: root, prov: [] }); + } + return out; +} + +/** Opt-in probe: node_modules//package.json version (never runs on default analyses). */ +export function installedVersion(projectRoot: string, name: string): string | null { + try { + const p = path.join(projectRoot, "node_modules", ...name.split("/"), "package.json"); + const doc = JSON.parse(fs.readFileSync(p, "utf-8")) as { version?: unknown }; + return typeof doc.version === "string" ? doc.version : null; + } catch { + return null; + } +} diff --git a/src/artifacts/deps.ts b/src/artifacts/deps.ts new file mode 100644 index 0000000..0e84f08 --- /dev/null +++ b/src/artifacts/deps.ts @@ -0,0 +1,119 @@ +/** + * Dependency extraction (#101, python PR #160 parity): `package.json` manifests → FLAT + * evidence-tagged `TSDependency` records on the application; the JSON lockfile family backfills + * `locked_version` on the OWNING manifest's records (`prov` gains "lockfile"); lockfiles never + * create records. Defensive throughout — a malformed file yields no records, never an exception. + */ +import type { TSDependency } from "../schema"; + +const SECTION_KIND: ReadonlyArray<[string, TSDependency["kind"]]> = [ + ["dependencies", "runtime"], + ["devDependencies", "dev"], + ["optionalDependencies", "optional"], + ["peerDependencies", "peer"], // the spec'd additive npm token +]; + +/** Import specifiers this distribution provides: itself; `@types/x` also provides types-for-x. */ +function providesOf(name: string): string[] { + if (name.startsWith("@types/")) { + const base = name.slice("@types/".length); + // DefinitelyTyped mangles scoped names: @types/scope__pkg types @scope/pkg. + const real = base.includes("__") ? `@${base.replace("__", "/")}` : base; + return [name, real]; + } + return [name]; +} + +export function parsePackageJson(text: string, declaredIn: string): TSDependency[] { + let doc: unknown; + try { + doc = JSON.parse(text); + } catch { + return []; + } + if (typeof doc !== "object" || doc === null) return []; + const out: TSDependency[] = []; + const seen = new Set(); + for (const [section, kind] of SECTION_KIND) { + const block = (doc as Record)[section]; + if (typeof block !== "object" || block === null) continue; + for (const [name, spec] of Object.entries(block as Record)) { + if (seen.has(name)) continue; // first section wins (npm merge order above) + seen.add(name); + out.push({ + name, + spec: typeof spec === "string" ? spec : "", + kind, + extras: [], + declared_in: declaredIn, + provides_imports: providesOf(name), + prov: ["declared"], + }); + } + } + return out; +} + +/** + * `name → locked version` from a JSON-family lockfile. package-lock/npm-shrinkwrap: v2/v3 + * top-level `packages["node_modules/"].version` (nested entries are transitive shadows), + * v1 `dependencies{}` fallback. bun.lock (JSONC): `packages{ "": ["@", ...] }`. + */ +export function readLock(fileName: string, text: string): Record { + if (fileName === "bun.lock") return readBunLock(text); + let doc: unknown; + try { + doc = JSON.parse(text); + } catch { + return {}; + } + if (typeof doc !== "object" || doc === null) return {}; + const out: Record = {}; + const packages = (doc as Record)["packages"]; + if (typeof packages === "object" && packages !== null) { + for (const [key, entry] of Object.entries(packages as Record)) { + const m = /^node_modules\/((?:@[^/]+\/)?[^/]+)$/.exec(key); + if (!m) continue; + const version = (entry as Record | null)?.["version"]; + if (typeof version === "string") out[m[1] as string] = version; + } + if (Object.keys(out).length) return out; + } + const v1 = (doc as Record)["dependencies"]; + if (typeof v1 === "object" && v1 !== null) { + for (const [name, entry] of Object.entries(v1 as Record)) { + const version = (entry as Record | null)?.["version"]; + if (typeof version === "string") out[name] = version; + } + } + return out; +} + +function readBunLock(text: string): Record { + let doc: unknown; + try { + doc = JSON.parse(text.replace(/,\s*([}\]])/g, "$1")); // tolerate bun's trailing commas + } catch { + return {}; + } + const out: Record = {}; + const packages = (doc as Record | null)?.["packages"]; + if (typeof packages !== "object" || packages === null) return {}; + for (const [name, entry] of Object.entries(packages as Record)) { + const first = Array.isArray(entry) ? entry[0] : undefined; + if (typeof first !== "string") continue; + const at = first.lastIndexOf("@"); + if (at > 0) out[name] = first.slice(at + 1); + } + return out; +} + +/** Backfill locked_version on DECLARED records; their `prov` gains "lockfile". */ +export function applyLockVersions(deps: TSDependency[], lock: Record): void { + for (const dep of deps) { + const v = lock[dep.name]; + if (v === undefined) continue; + dep.locked_version = v; + if (!dep.prov.includes("lockfile")) dep.prov.push("lockfile"); + } +} diff --git a/src/artifacts/index.ts b/src/artifacts/index.ts new file mode 100644 index 0000000..dab16d6 --- /dev/null +++ b/src/artifacts/index.ts @@ -0,0 +1,132 @@ +/** + * Repository-artifact layer (#101), parity with codeanalyzer-python PR #160 / the ratified + * 2026-08-27 spec: `inventoryArtifacts` walks the project once and returns the three + * application sections — `artifacts` (every RULES-matched non-code file, verbatim `source`, + * unbounded by decision), `dependencies` (flat, evidence-tagged, declared-only; locks backfill + * `locked_version`), and `unresolved_imports` (the hygiene signal). Level-free: attached + * identically at every `-a`. Not cached. Ids are stamped by assignIds (they embed `--app-name`). + * + * Discovery skips the source-walk's directory set; TS/JS source stays in the symbol table. + * Extensionless files with a shebang are captured as `script` artifacts. Unmatched files are + * NOT artifacts (rules-matched capture — python's posture). + */ +import * as fs from "node:fs"; +import * as path from "node:path"; +import type { AnalysisOptions } from "../options"; +import type { TSArtifact, TSDependency, TSImportBinding, TSModule } from "../schema"; +import { sha256 } from "../utils"; +import { SKIP_DIRS } from "../syntactic_analysis/discovery"; +import { matchRules } from "./rules"; +import { applyLockVersions, parsePackageJson, readLock } from "./deps"; +import { bindImports } from "./binding"; + +const SOURCE_EXTS = new Set([".ts", ".tsx", ".mts", ".cts", ".js", ".jsx", ".mjs", ".cjs"]); +const JSON_LOCKFILES = new Set(["package-lock.json", "npm-shrinkwrap.json", "bun.lock"]); + +export interface ArtifactLayer { + artifacts: Record; + dependencies: TSDependency[]; + unresolved_imports: TSImportBinding[]; +} + +export function inventoryArtifacts( + root: string, + opts: AnalysisOptions, + symbol_table: Record, +): ArtifactLayer { + const artifacts: Record = {}; + // Owning manifest's rel path → {name: locked version} (a lock pins its SIBLING package.json). + const locks: Record> = {}; + const manifests: Array<{ rel: string; text: string }> = []; + + for (const rel of walk(root).sort()) { + const base = path.basename(rel); + const matched = matchRules(rel); + let format = matched?.format; + let roles = matched?.roles; + const abs = path.join(root, rel); + let raw: Buffer; + try { + raw = fs.readFileSync(abs); + } catch { + continue; // unreadable — skip, don't crash + } + if (!matched) { + // extensionless shebang scripts are captured too (python PR #160) + if (path.extname(base) === "" && raw.subarray(0, 2).toString("utf-8") === "#!") { + format = "text"; + roles = ["script"]; + } else { + continue; // rules-matched capture only + } + } + const text = decodeLossy(raw); + const node: TSArtifact = { + id: "", + kind: "artifact", + path: rel, + format: format as string, + roles: roles as string[], + size_bytes: raw.length, + sha256: sha256(raw), + source: text ?? "", // verbatim, unbounded by decision (spec §3); binary → "" + extraction: "none", + }; + artifacts[rel] = node; + if (text === undefined) continue; + if (base === "package.json") manifests.push({ rel, text }); + else if (JSON_LOCKFILES.has(base)) { + const ownerRel = rel.split("/").slice(0, -1).concat("package.json").join("/"); + locks[ownerRel] = readLock(base, text); + artifacts[rel].extraction = "full"; + } + } + + // Declared records from every dependency-manifest package.json; sibling locks backfill. + const dependencies: TSDependency[] = []; + for (const { rel, text } of manifests) { + const recs = parsePackageJson(text, rel); // declared_in = REL PATH; assignIds re-stamps the id + const node = artifacts[rel]; + if (node) node.extraction = recs.length ? "full" : node.extraction; + const lock = locks[rel]; + if (lock) applyLockVersions(recs, lock); + dependencies.push(...recs); + } + + const unresolved_imports = bindImports(symbol_table, dependencies, root, opts.resolveInstalled ?? false); + return { artifacts, dependencies, unresolved_imports }; +} + +function walk(root: string): string[] { + const out: string[] = []; + const visit = (dir: string): void => { + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + for (const e of entries) { + const abs = path.join(dir, e.name); + if (e.isDirectory()) { + if (SKIP_DIRS.has(e.name)) continue; // the guard is on containing DIRS — a `.env` file survives + visit(abs); + } else if (e.isFile()) { + if (SOURCE_EXTS.has(path.extname(e.name))) continue; // source lives in the symbol table + out.push(path.relative(root, abs).split(path.sep).join("/")); + } + } + }; + visit(root); + return out; +} + +/** utf-8 decode with a strict binary probe on the head; binary → undefined. */ +function decodeLossy(raw: Buffer): string | undefined { + try { + new TextDecoder("utf-8", { fatal: true }).decode(raw.subarray(0, Math.min(raw.length, 4096))); + } catch { + return undefined; + } + return new TextDecoder("utf-8", { fatal: false }).decode(raw); +} diff --git a/src/artifacts/rules.ts b/src/artifacts/rules.ts new file mode 100644 index 0000000..9386804 --- /dev/null +++ b/src/artifacts/rules.ts @@ -0,0 +1,93 @@ +/** + * The shipped discovery rules table (#101, python PR #160's mechanism): glob pattern against the + * repo-relative POSIX path → (format, roles). Capture is rules-matched ONLY — an unmatched file + * is not an artifact (python's posture; `unknown` rows exist for config-shaped extensions). + */ + +export interface ArtifactRule { + pattern: RegExp; + format: string; + roles: string[]; +} + +/** + * Tiny glob→RegExp: `**` crosses directories, `*` does not. A pattern CONTAINING `/` anchors at + * the repo root; a bare basename pattern matches at any depth (workspace-member package.json, + * nested Dockerfiles). + */ +function glob(g: string): RegExp { + let re = ""; + for (let i = 0; i < g.length; i++) { + const c = g[i] as string; + if (c === "*") { + if (g[i + 1] === "*") { + re += ".*"; + i++; + if (g[i + 1] === "/") i++; // `**/` also matches zero directories + } else re += "[^/]*"; + } else if (".+^${}()|[]\\".includes(c)) re += `\\${c}`; + else re += c; + } + return g.includes("/") ? new RegExp(`^${re}$`) : new RegExp(`^(?:.*/)?${re}$`); +} + +const R = (g: string, format: string, roles: string[]): ArtifactRule => ({ pattern: glob(g), format, roles }); + +export const RULES: ArtifactRule[] = [ + // dependency manifests + locks (npm ecosystem) + R("package.json", "json", ["dependency-manifest", "tool-config"]), + R("package-lock.json", "json", ["dependency-manifest"]), + R("npm-shrinkwrap.json", "json", ["dependency-manifest"]), + R("bun.lock", "jsonc", ["dependency-manifest"]), + R("yarn.lock", "yarnlock", ["dependency-manifest"]), + R("pnpm-lock.yaml", "yaml", ["dependency-manifest"]), + // tool configs + R("tsconfig*.json", "json", ["tool-config"]), + R("jsconfig*.json", "json", ["tool-config"]), + R(".eslintrc*", "json", ["tool-config"]), + R(".prettierrc*", "json", ["tool-config"]), + R("babel.config.*", "text", ["tool-config"]), + R("vite.config.*", "text", ["tool-config"]), + R("webpack.config.*", "text", ["tool-config"]), + R("Makefile", "text", ["tool-config"]), + // containers / topology + R("Dockerfile", "dockerfile", ["container-image"]), + R("*.dockerfile", "dockerfile", ["container-image"]), + R("Dockerfile.*", "dockerfile", ["container-image"]), + R("docker-compose*.yml", "yaml", ["service-topology"]), + R("docker-compose*.yaml", "yaml", ["service-topology"]), + R("compose.yml", "yaml", ["service-topology"]), + R("compose.yaml", "yaml", ["service-topology"]), + R("k8s/**/*.yml", "yaml", ["service-topology"]), + R("k8s/**/*.yaml", "yaml", ["service-topology"]), + // ci + R(".github/workflows/*.yml", "yaml", ["ci"]), + R(".github/workflows/*.yaml", "yaml", ["ci"]), + R(".gitlab-ci.yml", "yaml", ["ci"]), + R("azure-pipelines.yml", "yaml", ["ci"]), + // env + R(".env", "env", ["env"]), + R(".env.*", "env", ["env"]), + // docs / legal + R("*.md", "text", ["docs"]), + R("*.rst", "text", ["docs"]), + R("LICENSE*", "text", ["legal"]), + R("COPYRIGHT*", "text", ["legal"]), + R("NOTICE*", "text", ["legal"]), + // config-shaped catch rows (python's `unknown` rows) + R("*.toml", "toml", ["unknown"]), + R("*.ini", "ini", ["unknown"]), + R("*.cfg", "ini", ["unknown"]), +]; + +/** First matching rule wins; roles union across ALL matching rules (a compose file is both). */ +export function matchRules(relPath: string): { format: string; roles: string[] } | null { + let format: string | null = null; + const roles: string[] = []; + for (const r of RULES) { + if (!r.pattern.test(relPath)) continue; + if (format === null) format = r.format; + for (const role of r.roles) if (!roles.includes(role)) roles.push(role); + } + return format === null ? null : { format, roles }; +} diff --git a/src/build/neo4j/project.ts b/src/build/neo4j/project.ts index 6c217f1..f7a4cf3 100644 --- a/src/build/neo4j/project.ts +++ b/src/build/neo4j/project.ts @@ -11,6 +11,7 @@ */ import type { TSAnalysis, TSApplication, TSBodyNode, TSCallable, TSField, TSModule, TSType } from "../../schema"; +import { purlNpm } from "../../schema/ids"; import { SCHEMA_VERSION } from "./schema"; import { type GraphRows, type NodeRef, type Props, RowBuilder, prune } from "./rows"; @@ -66,6 +67,56 @@ export function project(app: TSAnalysis, _appName?: string): GraphRows { projectScope(b, mod, modRef, fileKey); } + // Repository-artifact layer (#101, python PR #160 parity): language-NEUTRAL :Artifact and + // :Package (purl id) nodes — the deliberate exception to TS-prefixing, so sibling analyzers + // MERGE onto the same nodes — plus this analyzer's own claims (TS_PROVIDES / + // TS_UNRESOLVED_IMPORT) joining packages into the existing :TSExternal ghost id space. + // `source` text stays off the graph (hash + size dereference to it). + const importGhost = (name: string): NodeRef => + b.node([CAN, "TSExternal"], "id", `${root.id}/@external/${name}`, prune({ + id: `${root.id}/@external/${name}`, kind: "external", module: name, + })); + for (const art of Object.values(root.artifacts ?? {})) { + const aRef = b.node(["Artifact"], "id", art.id, prune({ + id: art.id, kind: "artifact", path: art.path, format: art.format, + roles: art.roles.length ? art.roles : null, size_bytes: art.size_bytes, + sha256: art.sha256, extraction: art.extraction, + })); + b.edge("HAS_ARTIFACT", appRef, aRef); + } + { + const lockIds = Object.values(root.artifacts ?? {}) + .filter((a) => /(^|\/)(package-lock\.json|npm-shrinkwrap\.json|bun\.lock|yarn\.lock|pnpm-lock\.yaml)$/.test(a.path)) + .map((a) => a.id) + .sort(); + const seen = new Set(); + for (const d of root.dependencies ?? []) { + const pkgId = purlNpm(d.name); + const pkgRef = b.node(["Package"], "id", pkgId, prune({ id: pkgId, ecosystem: "npm", name: d.name })); + b.edge("DECLARES_DEPENDENCY", { label: "Artifact", keyProp: "id", value: d.declared_in }, pkgRef, prune({ + spec: d.spec || null, kind: d.kind, extras: d.extras.length ? d.extras : null, + prov: d.prov.length ? d.prov : null, + }), d.kind); + if (d.locked_version) { + for (const lockId of lockIds) { + const k = `LOCKS\0${lockId}\0${pkgId}`; + if (seen.has(k)) continue; + seen.add(k); + b.edge("LOCKS", { label: "Artifact", keyProp: "id", value: lockId }, pkgRef, prune({ version: d.locked_version })); + } + } + for (const top of d.provides_imports) { + const k = `PROV\0${pkgId}\0${top}`; + if (seen.has(k)) continue; + seen.add(k); + b.edge("TS_PROVIDES", pkgRef, importGhost(top)); + } + } + for (const u of root.unresolved_imports ?? []) { + b.edge("TS_UNRESOLVED_IMPORT", appRef, importGhost(u.module), prune({ prov: u.prov.length ? u.prov : null })); + } + } + // External library targets (shared nodes — no _module). for (const ext of Object.values(root.external_symbols ?? {})) { b.node([CAN, "TSExternal"], "id", ext.id, prune({ id: ext.id, kind: "external", name: ext.name, module: ext.module })); diff --git a/src/build/neo4j/schema.ts b/src/build/neo4j/schema.ts index acb7f1d..93d38b3 100644 --- a/src/build/neo4j/schema.ts +++ b/src/build/neo4j/schema.ts @@ -19,7 +19,7 @@ * SCHEMA_VERSION: MAJOR on a breaking change (renamed/removed label, relationship or key), MINOR * on additive. v2 is a MAJOR bump from v1 (keys moved signature→can:// id; labels reshaped). */ -export const SCHEMA_VERSION = "2.1.0"; +export const SCHEMA_VERSION = "2.2.0"; export type PropType = "string" | "integer" | "float" | "boolean" | "string[]" | "integer[]"; @@ -64,6 +64,24 @@ export const NODE_LABELS: NodeLabel[] = [ analyzer_name: "string", analyzer_version: "string", }, }, + // Repository-artifact layer (#101, contract 2.2.0, python PR #160 parity): language-NEUTRAL + // labels — the deliberate exception to TS-prefixing, so sibling analyzers MERGE onto the same + // :Artifact/:Package nodes. Edges that stay this analyzer's own claim keep the TS_ prefix. + { + label: "Artifact", + mergeLabel: "Artifact", + key: "id", + properties: { + id: "string", kind: "string", path: "string", format: "string", roles: "string[]", + size_bytes: "integer", sha256: "string", extraction: "string", + }, + }, + { + label: "Package", + mergeLabel: "Package", + key: "id", + properties: { id: "string", ecosystem: "string", name: "string" }, + }, { label: "TSModule", mergeLabel: CAN, @@ -144,6 +162,17 @@ export const NODE_LABELS: NodeLabel[] = [ export const REL_TYPES: RelType[] = [ { type: "TS_HAS_MODULE", from: ["TSApplication"], to: ["TSModule"], properties: {} }, + // Repository-artifact layer (#101, contract 2.2.0, python PR #160 vocabulary) + { type: "HAS_ARTIFACT", from: ["TSApplication"], to: ["Artifact"], properties: {} }, + { + type: "DECLARES_DEPENDENCY", + from: ["Artifact"], + to: ["Package"], + properties: { spec: "string", kind: "string", extras: "string[]", prov: "string[]" }, + }, + { type: "LOCKS", from: ["Artifact"], to: ["Package"], properties: { version: "string" } }, + { type: "TS_PROVIDES", from: ["Package"], to: ["TSExternal"], properties: {} }, + { type: "TS_UNRESOLVED_IMPORT", from: ["TSApplication"], to: ["TSExternal"], properties: { prov: "string[]" } }, { type: "TS_DECLARES", from: ["TSModule", "TSNamespace", "TSCallable"], diff --git a/src/cli.ts b/src/cli.ts index c2bdb46..18c9722 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1,6 +1,6 @@ import * as path from "node:path"; import { Command, Option } from "commander"; -import type { AnalysisOptions, CallGraphProviderName, EmitTarget } from "./options"; +import type { AnalysisOptions, EmitTarget } from "./options"; import { ALL_GRAPHS, type GraphSelector } from "./schema"; /** @@ -57,12 +57,7 @@ export function buildProgram(): Command { .option("--lazy", "reuse the cache (default)") .option("--no-build", "skip dependency materialization (use a prepared node_modules)") .option("--no-phantoms", "disable phantom (external) nodes for imported/required library calls") - .option( - "--call-graph-provider ", - "call-graph backend: union (default, tsc ∪ jelly) | tsc | jelly | both (deprecated alias of union)", - "union", - ) - .option("--tsc-only", "use the tsc resolver only — opt out of Jelly edges (overrides --call-graph-provider)") + .option("--resolve-installed", "probe node_modules metadata for import→package binding (default: repo files only)") .option("-c, --cache-dir ", "cache/intermediate directory") .option("-v, --verbose", "increase verbosity (repeatable)", (_v: string, prev: number) => prev + 1, 0) .allowExcessArguments(true); @@ -137,23 +132,6 @@ export function parseArgs(argv: string[]): AnalysisOptions { if (emit !== "schema" && !o.input) program.error("required option '-i, --input ' not specified"); const targets: string[] | null = Array.isArray(o.targetFiles) && o.targetFiles.length ? o.targetFiles.map(String) : null; - // --tsc-only is the forced opt-out: it wins over --call-graph-provider. Otherwise `both` is a - // deprecated alias of `union` (warn, but honor it); unknown values fall back to the union default. - let cgProvider: CallGraphProviderName; - if (o.tscOnly) { - cgProvider = "tsc"; - } else if (o.callGraphProvider === "tsc") { - cgProvider = "tsc"; - } else if (o.callGraphProvider === "jelly") { - cgProvider = "jelly"; - } else { - if (o.callGraphProvider === "both") { - // stderr only — stdout may carry compact JSON when -o is omitted. - console.error("warning: --call-graph-provider both is deprecated; it now behaves as 'union' (tsc ∪ jelly)."); - } - cgProvider = "union"; - } - return { input: o.input ? path.resolve(String(o.input)) : "", output: o.output ? path.resolve(String(o.output)) : null, @@ -173,7 +151,7 @@ export function parseArgs(argv: string[]): AnalysisOptions { // commander maps --no-build / --no-phantoms to opts.build/phantoms === false noBuild: o.build === false, phantoms: o.phantoms !== false, - callGraphProvider: cgProvider, + resolveInstalled: Boolean(o.resolveInstalled), cacheDir: o.cacheDir ? path.resolve(String(o.cacheDir)) : null, verbosity: typeof o.verbose === "number" ? o.verbose : 0, }; diff --git a/src/core.ts b/src/core.ts index 5355e86..1965580 100644 --- a/src/core.ts +++ b/src/core.ts @@ -1,8 +1,9 @@ import * as path from "node:path"; import { buildProgramGraphs, startExtraction } from "./dataflow"; -import { mergeCallGraphs, selectProvider } from "./semantic_analysis"; +import { type LinkerResolutions, mergeCallGraphs, runDefuseLinker, tscProvider } from "./semantic_analysis"; import { loadCache, saveCache } from "./utils"; import { materialize } from "./build"; +import { inventoryArtifacts } from "./artifacts"; import type { AnalysisOptions } from "./options"; import type { AnalysisInternal } from "./schema"; import { type AnalysisResult, finalizeAnalysis } from "./schema/emit"; @@ -42,40 +43,55 @@ export async function analyze(opts: AnalysisOptions): Promise { } const extraction = opts.analysisLevel >= 3 ? startExtraction(project, symbol_table, mat.tsConfigFilePath, opts, log) : null; - // Call graph via the selected provider (union of tsc+jelly by default; --tsc-only / jelly opt-in). - // Only worth running at level >= 2: finalizeAnalysis discards call_graph/external_symbols/ - // synthesized_callables at -a 1 (homeExternals/homeSynthesized in src/schema/emit.ts are - // gated to `level >= 2`), so running the solve — including the heavier Jelly leg — at -a 1 - // would compute a result that's thrown away. Levels 3/4 need the provider for callee - // resolution and are always >= 2, so this gate is safe. - // - // Run the provider PER PROGRAM (each with its own Project + its slice of callables via `only`), - // then merge the results the same way the union provider merges tsc∪jelly. Signature gating uses - // the full merged symbol_table (passed to every program), so a cross-program in-project call - // resolves. Single-program projects run the loop once — behavior is unchanged. - const provider = selectProvider(opts.callGraphProvider); - log.info(`call graph provider: ${provider.name}`); - let cg: ReturnType = { edges: [], external_symbols: {}, synthesized_callables: {} }; + // Call graph: the tsc resolver, per program (each with its own Project + its slice of callables + // via `only`), merged across programs. Only worth running at level >= 2: finalizeAnalysis + // discards call_graph/external_symbols/synthesized_callables at -a 1 (homeExternals/ + // homeSynthesized in src/schema/emit.ts are gated to `level >= 2`), so running the solve at + // -a 1 would compute a result that's thrown away. Levels 3/4 need it for callee resolution and + // are always >= 2, so this gate is safe. Signature gating uses the full merged symbol_table + // (passed to every program), so a cross-program in-project call resolves. + let cg: ReturnType = { edges: [], external_symbols: {}, synthesized_callables: {} }; + const resolutions: LinkerResolutions = new Map(); if (opts.analysisLevel >= 2) { for (const prog of programs) { - const pcg = provider.build({ + const ctx = { project: prog.project, symbol_table, root: opts.input, log, phantoms: opts.phantoms, only: prog.fileKeys, - }); - cg = mergeCallGraphs(cg, pcg); + }; + cg = mergeCallGraphs(cg, tscProvider.build(ctx)); + // The defuse linker overlays the tsc base: it reads the callee_signature backfill the tsc + // leg just wrote, resolves what remains (tiers T1–T5, defuseLinker.ts), and returns its + // body-node resolutions out-of-band (never persisted — cache provenance rule). + const linked = runDefuseLinker(ctx); + cg = mergeCallGraphs(cg, linked.result); + for (const [caller, m] of linked.resolutions) { + const ex = resolutions.get(caller); + if (!ex) resolutions.set(caller, m); + else for (const [k, v] of m) if (!ex.has(k)) ex.set(k, v); + } } } const call_graph = cg.edges; + // Repository-artifact layer (#101, python PR #160 parity): level-free, identical at every -a. + const layer = inventoryArtifacts(opts.input, opts, symbol_table); + log.info( + `artifacts: ${Object.keys(layer.artifacts).length} files, ${layer.dependencies.length} dependency records, ` + + `${layer.unresolved_imports.length} unresolved imports`, + ); + const app: AnalysisInternal = { symbol_table, call_graph, external_symbols: cg.external_symbols, synthesized_callables: cg.synthesized_callables, + artifacts: layer.artifacts, + dependencies: layer.dependencies, + unresolved_imports: layer.unresolved_imports, }; // Level 3 join: stages 5–7 (summary wavefront + SDG) consume the extraction AND the @@ -85,5 +101,5 @@ export async function analyze(opts: AnalysisOptions): Promise { // Cache the id-free base (ids/body/heritage are per-run layers stamped by finalizeAnalysis; // the cached tree must stay --app-name-free). saveCache(cacheDir, { symbol_table }); - return finalizeAnalysis(app, pg, opts); + return finalizeAnalysis(app, pg, opts, resolutions); } diff --git a/src/dataflow/attach.ts b/src/dataflow/attach.ts index ceea6a9..11f59b0 100644 --- a/src/dataflow/attach.ts +++ b/src/dataflow/attach.ts @@ -114,7 +114,7 @@ function emitL3(li: LocalIds, nodes: GraphNode[], cfgEdges: CfgEdge[] | undefine // prov = the def-use METHOD: `solveDefUse` computes forward may-reaching-definitions over // k-limited access paths with a flow-insensitive copy/field-alias substrate (defuse.ts). It // is NOT SSA and NOT points-to-oracle-backed — so we tag it "reaching-defs", not "ssa". - // A real points-to layer (Jelly, PR F) would emit additional edges tagged "points-to". + // A real points-to layer (PR F) would emit additional edges tagged "points-to". // "reaching-defs" is a SANCTIONED ADDITIVE prov token — a deliberate, documented deviation // from the shared cross-analyzer vocabulary's canonical "ssa" tag for the L3 syntactic DDG. // Recorded in `.claude/SCHEMA_DECISIONS.md` (issue #32); JSON, Neo4j (`ddg.prov: string[]`, diff --git a/src/dataflow/defuse.ts b/src/dataflow/defuse.ts index 2fc4a84..5dd18f2 100644 --- a/src/dataflow/defuse.ts +++ b/src/dataflow/defuse.ts @@ -18,7 +18,7 @@ * * Aliasing (MVP substrate, per issue #2 / SCHEMA_DECISIONS.md): flow-insensitive union-find over * bases connected by direct copies (`const q = p`); a write through one name weakly updates the - * other. Points-to-backed aliasing via Jelly's solved state is the staged upgrade (PR F). + * other. Points-to-backed aliasing is the staged upgrade (PR F). * * Def-use: classic forward may reaching-definitions. Strong (killing) defs are whole-base writes * to locals/params; every field write is weak. Captured/module/this bases get a synthetic def at diff --git a/src/main.ts b/src/main.ts index 40dd5c2..3a82795 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,26 +1,3 @@ #!/usr/bin/env node -/** - * Multi-call binary entry. The compiled `cants` executable bundles BOTH the analyzer and the - * `@cs-au-dk/jelly` CLI; this dispatcher picks which one runs based on argv: - * - * cants __jelly -> run the embedded Jelly CLI (used internally by jellyProvider) - * cants -> run the normal analyzer - * - * Both programs self-execute on import (analyzer's main() / Jelly's program.parse()), so dispatch is - * "reshape argv, then dynamically import the right module". Bun's --compile bundles both branches. - * The CANTS_SELF_JELLY marker tells jellyProvider it can re-exec THIS binary for Jelly instead of - * shelling out to `node`; it is intentionally unset in source/dev runs (where the dispatcher is - * bypassed and the provider falls back to `node @cs-au-dk/jelly/lib/main.js`). - */ -export {}; // mark as a module so top-level await is permitted - -const argv = process.argv; -if (argv[2] === "__jelly") { - // Jelly's commander reads process.argv as [node, script, ...args]; drop our "__jelly" sentinel. - process.argv = [argv[0], "jelly", ...argv.slice(3)]; - // @ts-ignore — @cs-au-dk/jelly ships no type declarations for the lib subpath - await import("@cs-au-dk/jelly/lib/main.js"); -} else { - process.env.CANTS_SELF_JELLY = process.execPath; - await import("./index"); -} +/** Binary entry — the analyzer CLI (self-executes on import). */ +import "./index"; diff --git a/src/options/options.ts b/src/options/options.ts index 80d10d8..ac5ba0c 100644 --- a/src/options/options.ts +++ b/src/options/options.ts @@ -1,8 +1,6 @@ import type { GraphSelector } from "../schema"; export type EmitTarget = "json" | "neo4j" | "schema"; -export type CallGraphProviderName = "union" | "tsc" | "jelly"; - /** Normalized analysis options (produced by the CLI layer, consumed by core). */ export interface AnalysisOptions { /** Project root to analyze (absolute). */ @@ -46,9 +44,9 @@ export interface AnalysisOptions { noBuild: boolean; /** Emit phantom (external) nodes/edges for imported/required library call targets. Default on. */ phantoms: boolean; - /** Call-graph backend: union of tsc+jelly (default), tsc resolver only (--tsc-only), or jelly. */ - callGraphProvider: CallGraphProviderName; /** Where caches/intermediate state live; null ⇒ /.codeanalyzer. */ + /** Opt-in: probe node_modules metadata for import→package binding (prov "installed-metadata"). */ + resolveInstalled?: boolean; cacheDir: string | null; /** Verbosity (repeatable -v). */ verbosity: number; diff --git a/src/schema/assignIds.ts b/src/schema/assignIds.ts index e493c11..6e4fcec 100644 --- a/src/schema/assignIds.ts +++ b/src/schema/assignIds.ts @@ -8,7 +8,7 @@ * id-uniqueness gate's collision list. */ -import { applicationIdOf, idFromSig, memberKey, moduleIdOf, modulePrefixOf } from "./ids"; +import { applicationIdOf, artifactIdOf, idFromSig, memberKey, moduleIdOf, modulePrefixOf } from "./ids"; import type { AnalysisInternal, TSCallable, TSField, TSType } from "./schema"; export interface AssignedIds { @@ -54,10 +54,25 @@ export function assignIds(app: AnalysisInternal, appName: string): AssignedIds { const moduleId = moduleIdOf(appId, fileKey); const modulePrefix = modulePrefixOf(fileKey); mod.id = moduleId; + // Module-scope execution is a call-graph SOURCE (python #131 parity: a call in module scope + // is attributed to the MODULE). The prefix is the module's "signature", so those edges + // re-identify onto the module node's id instead of dangling. + register(modulePrefix, moduleId); doFields(moduleId, mod.fields); for (const fn of Object.values(mod.functions ?? {})) doCallable(moduleId, modulePrefix, fn); for (const t of Object.values(mod.types ?? {})) doType(moduleId, modulePrefix, t); } + // Repository-artifact layer: same per-run rule (ids embed --app-name). Artifact ids are + // language-NEUTRAL (`can://artifact/...`); dependency/import records are flat evidence rows + // with no node id of their own (the graph's :Package node is purl-keyed). + for (const [relPath, art] of Object.entries(app.artifacts ?? {})) { + art.id = artifactIdOf(appName, relPath); + } + for (const dep of app.dependencies ?? []) { + const artPath = dep.declared_in; // scanners record the REL PATH; re-stamp onto the id + dep.declared_in = artifactIdOf(appName, artPath.startsWith("can://") ? artPath.split("/").slice(4).join("/") : artPath); + } + return { appId, idBySig, callableBySig, collisions }; } diff --git a/src/schema/emit.ts b/src/schema/emit.ts index 4ba933e..17656cf 100644 --- a/src/schema/emit.ts +++ b/src/schema/emit.ts @@ -33,8 +33,33 @@ const ANALYZER_NAME = "codeanalyzer-typescript"; /** Highest analysis level this emitter populates today (L1 tree, L2 call graph, L3/L4 dataflow). */ const MAX_IMPLEMENTED = 4; -/** INTERNAL model fields — never on the wire (see schema.ts header). */ -const INTERNAL_KEYS = new Set(["call_sites", "abs_path", "content_hash", "last_modified", "file_size"]); +/** + * Structural internal-field strip on the WIRE CLONE: module cache trio + callable join fields. + * Structural (walks the tree shape) rather than key-name-based, for two load-bearing reasons: + * the artifact layer's `content_hash` is WIRE payload (a name-keyed replacer would eat it), and + * a `JSON.stringify` deep-copy roundtrip builds one multi-GB string at vscode-L4 scale and OOMs + * (measured). `structuredClone` + targeted deletes never materializes a string. + */ +function stripInternal(root: TSApplication): void { + const stripCallable = (c: Record): void => { + delete c["call_sites"]; + delete c["abs_path"]; + for (const nested of Object.values((c["callables"] as Record>) ?? {})) stripCallable(nested); + for (const t of Object.values((c["types"] as Record>) ?? {})) stripType(t); + }; + const stripType = (t: Record): void => { + for (const m of Object.values((t["callables"] as Record>) ?? {})) stripCallable(m); + for (const f of Object.values((t["functions"] as Record>) ?? {})) stripCallable(f); + for (const nt of Object.values((t["types"] as Record>) ?? {})) stripType(nt); + }; + for (const mod of Object.values(root.symbol_table) as unknown as Record[]) { + delete mod["content_hash"]; + delete mod["last_modified"]; + delete mod["file_size"]; + for (const fn of Object.values((mod["functions"] as Record>) ?? {})) stripCallable(fn); + for (const t of Object.values((mod["types"] as Record>) ?? {})) stripType(t); + } +} // ---------------------------------------------------------------------------------------------- // entry point @@ -49,7 +74,12 @@ export interface AnalysisResult { dangling: string[]; // call-graph endpoints with no id home (L2 no-dangling gate; should be empty) } -export function finalizeAnalysis(app: AnalysisInternal, pg: ProgramGraphs | null, opts: AnalysisOptions): AnalysisResult { +export function finalizeAnalysis( + app: AnalysisInternal, + pg: ProgramGraphs | null, + opts: AnalysisOptions, + resolutions?: Map>, +): AnalysisResult { const level = opts.analysisLevel; const appName = (opts.appName ?? (opts.input ? path.basename(opts.input) : "") ?? "").trim() || "app"; @@ -58,14 +88,24 @@ export function finalizeAnalysis(app: AnalysisInternal, pg: ProgramGraphs | null populateL1Body(app); resolveHeritageIds(app, idBySig); - const root: TSApplication = { id: appId, kind: "application", symbol_table: app.symbol_table, call_graph: [], param_in: [], param_out: [] }; + const root: TSApplication = { + id: appId, + kind: "application", + symbol_table: app.symbol_table, + call_graph: [], + param_in: [], + param_out: [], + artifacts: app.artifacts ?? {}, + dependencies: app.dependencies ?? [], + unresolved_imports: app.unresolved_imports ?? [], + }; // L2 — home the off-tree edge endpoints, backfill `callee`, re-identify the call graph. const dangling: string[] = []; if (level >= 2) { root.external_symbols = homeExternals(app, appId, idBySig); root.synthesized_callables = homeSynthesized(app, appId, idBySig); - backfillCallees(app, idBySig); + backfillCallees(app, idBySig, resolutions); root.call_graph = reidentifyCallGraph(app.call_graph ?? [], idBySig, dangling); } @@ -84,9 +124,9 @@ export function finalizeAnalysis(app: AnalysisInternal, pg: ProgramGraphs | null analyzer: { name: ANALYZER_NAME, version: ANALYZER_VERSION }, application: root, }; - // The wire copy: deep, detached from the live tree, internal fields stripped by key. - const application = JSON.parse( - JSON.stringify(envelope, (key, value) => (INTERNAL_KEYS.has(key) ? undefined : value)), - ) as TSAnalysis; + // The wire copy: deep, detached from the live tree, internals stripped STRUCTURALLY — + // structuredClone instead of a stringify roundtrip (the string form OOMs at vscode-L4 scale). + const application = structuredClone(envelope) as TSAnalysis; + stripInternal(application.application); return { application, internal: app, ...(pg ? { program_graphs: pg } : {}), idBySig, collisions, dangling }; } diff --git a/src/schema/ids.ts b/src/schema/ids.ts index 860f0b5..8124a05 100644 --- a/src/schema/ids.ts +++ b/src/schema/ids.ts @@ -29,6 +29,29 @@ export function idFromSig(moduleId: string, modulePrefix: string, sig: string): return `${moduleId}/${tail.split(".").join("/")}`; } +/** + * Repository-artifact ids. Leading "./" and "/" are dropped as SEPARATORS only — dotfiles + * (`.env`, `.github/...`) keep their leading dot (python's rule). + */ +export function artifactIdOf(appName: string, relPath: string): string { + let rel = relPath.replace(/\\/g, "/"); + while (rel.startsWith("./")) rel = rel.slice(2); + rel = rel.replace(/^\/+/, ""); + // Language-NEUTRAL namespace (python PR #160): the first segment is `artifact`, not a + // language — sibling analyzers over the same repo emit the SAME id for the same file. + return `can://artifact/${appName}/${rel}`; +} + +/** Package URL for an npm package name — the cross-language package id (`pkg:npm/...`). */ +export function purlNpm(name: string): string { + if (name.startsWith("@")) { + const slash = name.indexOf("/"); + const scope = encodeURIComponent(name.slice(0, slash)); // "@scope" → "%40scope" (purl spec) + return `pkg:npm/${scope}/${name.slice(slash + 1)}`; + } + return `pkg:npm/${name}`; +} + /** The map key for a callable/type within its parent: the last signature segment (+ accessor tag). */ export function memberKey(sig: string, accessorKind?: string | null): string { const seg = sig.split(".").pop() ?? sig; diff --git a/src/schema/l2Callees.ts b/src/schema/l2Callees.ts index 7ab3fa0..57b48ba 100644 --- a/src/schema/l2Callees.ts +++ b/src/schema/l2Callees.ts @@ -14,14 +14,23 @@ import type { AnalysisInternal, TSCallEdge, TSCallGraphEdge, TSModule } from "./ import { forEachCallable } from "./schema"; import { callBodyKeys } from "./l1Body"; -export function backfillCallees(app: AnalysisInternal, idBySig: Map): void { +export function backfillCallees( + app: AnalysisInternal, + idBySig: Map, + resolutions?: Map>, +): void { for (const mod of Object.values(app.symbol_table) as TSModule[]) { forEachCallable(mod, (c) => { + const linked = resolutions?.get(c.signature); for (const [key, cs] of callBodyKeys(c.call_sites)) { - if (!cs.callee_signature) continue; + // The resolver's in-place backfill wins; the linker's returned map fills the gaps. Linker + // resolutions are deliberately NOT persisted into callee_signature (cache provenance rule + // — see defuseLinker.ts header). + const sig = cs.callee_signature ?? linked?.get(key); + if (!sig) continue; const node = c.body[key]; if (!node || node.kind !== "call") continue; - node.callee = idBySig.get(cs.callee_signature) ?? null; + node.callee = idBySig.get(sig) ?? null; } }); } diff --git a/src/schema/schema.ts b/src/schema/schema.ts index 7587bb0..8e73b34 100644 --- a/src/schema/schema.ts +++ b/src/schema/schema.ts @@ -324,6 +324,47 @@ export interface TSModule { file_size?: number; } +// ---------------------------------------------------------------------------------------------- +// Repository-artifact layer (#101; parity with codeanalyzer-python PR #160 / spec +// 2026-08-27-artifacts-and-dependencies-design.md): recognized non-code files as nodes with +// LANGUAGE-NEUTRAL ids, plus evidence-tagged dependency records and the unresolved-import +// hygiene signal. Application-anchored, level-free — identical at every -a level. Capture is +// broad (every rules-matched file becomes a node, verbatim source, unbounded by decision); +// extraction is narrow (only dependency-manifest roles feed `dependencies` this unit). +// ---------------------------------------------------------------------------------------------- + +/** A recognized non-code file (config, manifest, CI, container spec). */ +export interface TSArtifact { + id: string; // can://artifact// — language-NEUTRAL namespace, stamped per-run + kind: "artifact"; + path: string; // repo-relative POSIX path (also the map key) + format: string; // json | jsonc | yaml | toml | ini | requirements? | dockerfile | yarnlock | text | env + roles: string[]; // dependency-manifest | tool-config | container-image | service-topology | ci | env | packaging | legal | docs | script | unknown + size_bytes: number; + sha256: string; + source: string; // verbatim, unbounded by decision (spec §3) + extraction: "none" | "partial" | "full"; +} + +/** One declared third-party dependency, evidence-tagged via `prov`. */ +export interface TSDependency { + name: string; // npm-native, @scope kept + spec: string; // as declared ("^4.17.21"); "" when the section value is not a string + kind: "runtime" | "dev" | "optional" | "peer" | "build"; // `peer` is the spec'd additive npm token + extras: string[]; // npm has none — always [] (shared shape parity) + declared_in: string; // TSArtifact id + locked_version?: string; + provides_imports: string[]; // import specifiers this distribution provides (npm: the name; @types/x: x) + prov: string[]; // declared | lockfile | installed-metadata | heuristic +} + +/** A non-relative import no declared dependency accounts for (the dependency-hygiene signal). */ +export interface TSImportBinding { + module: string; // the specifier root ("express", "@scope/pkg") + bound_to?: string; // best-effort distribution name when partially bound + prov: string[]; +} + // ---------------------------------------------------------------------------------------------- // Call-graph edge (identity-only, provider output; endpoints are signature strings until the // call-graph-ids pass rewrites them onto can:// ids at L2) @@ -354,10 +395,10 @@ export interface TSExternalSymbol { module: string; // the import/require specifier, e.g. "node:fs", "express", "@scope/pkg" } -// A first-party anonymous callback that Jelly resolves as a call-graph endpoint but the symbol -// table never names (the canonicalizer returns null for anonymous functions). The map key IS the -// synthesized signature `:`, so an edge `source`/ -// `target` byte-matches it just like a real `Callable.signature` or `TSExternalSymbol.signature`. +// A first-party anonymous callback a call-graph builder resolved as an edge endpoint but could +// not name against the symbol table (a residual-fallback safety net; since 2.1.0 the tree names +// anonymous callables positionally, so this map is normally empty). The map key IS the +// synthesized signature, so an edge `source`/`target` byte-matches it like a real signature. export interface TSSynthesizedCallable { name: string; // display name — always ""; the signature carries the precise identity path: string; // owning module key (project-relative POSIX path WITH extension) @@ -375,6 +416,10 @@ export interface AnalysisInternal { call_graph: TSCallEdge[]; external_symbols: Record; synthesized_callables: Record; + /** Repository-artifact layer (level-free). */ + artifacts?: Record; + dependencies?: TSDependency[]; + unresolved_imports?: TSImportBinding[]; } // ---------------------------------------------------------------------------------------------- @@ -405,6 +450,10 @@ export interface TSApplication { call_graph: TSCallGraphEdge[]; // L2 — callable → callable (empty at L1) param_in: TSParamEdge[]; // L4 (empty until L4) param_out: TSParamEdge[]; // L4 + /** Repository-artifact layer — identical at every level (#101, python PR #160 parity). */ + artifacts: Record; + dependencies: TSDependency[]; + unresolved_imports: TSImportBinding[]; // TS-additive (parity): edge endpoints outside the containment tree need an id home. external_symbols?: Record; // L2 — library call targets, keyed by id // L2 — 2.1.0 compatibility index: pre-2.1.0 anonymous-callable id → the tree id that replaced @@ -417,7 +466,7 @@ export interface TSApplication { export interface TSCallGraphEdge { src: string; dst: string; - prov: string[]; // provenance, e.g. ["tsc"], ["jelly"] + prov: string[]; // provenance, e.g. ["tsc"], ["defuse"], ["import"] weight: number; } diff --git a/src/schema/signatures.ts b/src/schema/signatures.ts index e228fdf..1c2b486 100644 --- a/src/schema/signatures.ts +++ b/src/schema/signatures.ts @@ -29,7 +29,7 @@ export function contributorName(node: Node): string | null { /** * The segment an unnamed function-like node contributes. Position is the only discriminant a - * nameless callable has, and it is the one both the resolver and Jelly can compute independently + * nameless callable has, and it is the one every call-graph builder can compute independently * — which is what keeps caller-side and callee-side ids byte-identical. Angle brackets mark the * segment synthetic (the ``/`` convention). It joins the dotted chain, so an * anonymous callable lives in the durable id tier and never collides with the `@line:col` @@ -82,10 +82,10 @@ export function computeSignatureForDecl(node: Node, root: string): string | null return signatureOf(modulePrefix, ...parts); } -/** Resolve the declaration a call/new expression targets, following import aliases. */ +/** Resolve the declaration a call/new/tagged-template expression targets, following import aliases. */ export function resolveCalleeDecl(call: Node): Node | undefined { - if (!Node.isCallExpression(call) && !Node.isNewExpression(call)) return undefined; - const expr = call.getExpression(); + if (!Node.isCallExpression(call) && !Node.isNewExpression(call) && !Node.isTaggedTemplateExpression(call)) return undefined; + const expr = Node.isTaggedTemplateExpression(call) ? call.getTag() : call.getExpression(); let symNode: Node = expr; if (Node.isPropertyAccessExpression(expr)) symNode = expr.getNameNode(); else if (Node.isElementAccessExpression(expr)) return undefined; // dynamic dispatch — best-effort skip diff --git a/src/semantic_analysis/callGraph.ts b/src/semantic_analysis/callGraph.ts index e20ea08..a60835a 100644 --- a/src/semantic_analysis/callGraph.ts +++ b/src/semantic_analysis/callGraph.ts @@ -20,7 +20,33 @@ import { type TSType, forEachCallable, } from "../schema"; -import { resolveCalleeSignature } from "../schema"; +import { fileKeyOf, resolveCalleeSignature } from "../schema"; +import { isCallableDecl } from "../schema"; + +/** The nearest ancestor that is itself a callable declaration (incl. `const f = () => …`), or undefined. */ +function enclosingCallable(node: Node): Node | undefined { + for (const a of node.getAncestors()) { + if (isCallableDecl(a)) return a; + if (Node.isVariableDeclaration(a)) { + const init = a.getInitializer?.(); + if (init && (Node.isArrowFunction(init) || Node.isFunctionExpression(init))) return a; + } + } + return undefined; +} + +/** Inside a NON-static class property initializer — owned by the constructor, not module scope. */ +export function inInstancePropInit(node: Node): boolean { + for (const a of node.getAncestors()) { + if (isCallableDecl(a)) return false; + if (Node.isPropertyDeclaration(a)) return !(a as unknown as { isStatic?: () => boolean }).isStatic?.(); + } + return false; +} + +function fileKeyOfNode(node: Node, root: string): { fileKey: string; modulePrefix: string } { + return fileKeyOf(node.getSourceFile().getFilePath(), root); +} import type { Logger } from "../utils"; import { type ExternalIndex, buildExternalIndex, resolvePhantom } from "./phantoms"; @@ -33,7 +59,7 @@ export interface CallGraphResult { edges: TSCallEdge[]; external_symbols: Record; // Anonymous callbacks resolved as edge endpoints that the symbol table doesn't name. Empty for - // the tsc resolver (its edges are gated to real symbol-table signatures); populated by Jelly. + // the tsc resolver (its edges are gated to real symbol-table signatures); a residual-fallback net. synthesized_callables: Record; } @@ -130,6 +156,42 @@ export function buildCallGraph( let rtaCount = 0; let phantomCount = 0; let unresolved = 0; + + // Module-scope sweep (python #131 parity): a call with NO enclosing callable — top-level + // statements, class property initializers, namespace bodies — is attributed to the MODULE + // (source = the module prefix, re-identified onto the module node at L2). These sites are + // never recorded in call_sites (modules have no body{}), so resolve them straight off the AST. + for (const node of callExprIndex.values()) { + if (enclosingCallable(node) || inInstancePropInit(node)) continue; + const fileKey = fileKeyOfNode(node, root); + if (only && !only.has(fileKey.fileKey)) continue; + const source = fileKey.modulePrefix; + const r = resolveCalleeSignature(node, root, allSignatures); + if (r?.external) { + if (phantoms) { + if (!external_symbols[r.signature]) external_symbols[r.signature] = { name: r.external.member, module: r.external.module }; + addPhantomEdge(source, r.signature, r.external.module); + phantomCount++; + } else unresolved++; + continue; + } + if (!r) { + if (phantoms) { + const ph = resolvePhantom(node, extIndexFor(node)); + if (ph) { + if (!external_symbols[ph.signature]) external_symbols[ph.signature] = { name: ph.member, module: ph.module }; + addPhantomEdge(source, ph.signature, ph.module); + phantomCount++; + continue; + } + } + unresolved++; + continue; + } + addEdge(source, r.signature, false); + resolved++; + } + for (const caller of callables) { for (const site of caller.call_sites) { const node = callExprIndex.get( @@ -252,13 +314,13 @@ function indexClasses( } } -function indexCallExpressions(project: Project): Map { +export function indexCallExpressions(project: Project): Map { const idx = new Map(); for (const sf of project.getSourceFiles()) { const fp = sf.getFilePath(); if (sf.isDeclarationFile() || fp.includes("/node_modules/")) continue; sf.forEachDescendant((n) => { - if (Node.isCallExpression(n) || Node.isNewExpression(n)) { + if (Node.isCallExpression(n) || Node.isNewExpression(n) || Node.isTaggedTemplateExpression(n)) { const s = sf.getLineAndColumnAtPos(n.getStart()); const e = sf.getLineAndColumnAtPos(n.getEnd()); // Full span (start AND end) keys the node uniquely; chained calls like `f(x).g(y)` diff --git a/src/semantic_analysis/defuseLinker.ts b/src/semantic_analysis/defuseLinker.ts new file mode 100644 index 0000000..f815f80 --- /dev/null +++ b/src/semantic_analysis/defuseLinker.ts @@ -0,0 +1,563 @@ +/** + * The defuse linker — the local pass that backfills call edges the tsc resolver missed + * (docs/design/specs/defuse-linker-call-graph.md, #98; python parity: the Jedi + defuse-linker + * architecture that replaced PyCG). Per-callable and bounded-round only — NO whole-program + * fixpoint; sorted iteration throughout, so the output is deterministic by construction. + * + * Tiers, applied in order (a precise resolution is never widened by a later tier): + * T1 local value chase — alias chains `const f = handler; f()` through bounded + * symbol→declaration hops (the checker's alias-following covers imports). + * T2 decorator invocations — `@Get(':id')` on a method/accessor becomes an edge + * decorated-callable → decorator target, EDGE-ONLY (decorator calls live outside walkBody's + * reach, so there is no body call node to refine — matching the historical Jelly shape). + * T3 external-callback rule — a function value passed to an external/unresolved callee emits + * enclosing-callable → function-value, EDGE-ONLY (`.map(u => …)`, `app.get('/x', handler)`). + * Deliberate divergence from python (JS is callback-central); recorded in the spec. + * T4 bounded interprocedural votes — (a) a parameter-invoking site (`cb()`) resolves to the + * function values passed at that position by resolved-internal callers (two rounds: round + * one's resolutions vote before round two); (b) `const f = factory(); f()` resolves through + * the factory's unique returned function. + * T5 CHA-by-name — receiver sites that survive every typed tier resolve to every internal + * callable of that method name (bounded per site) — the over-approximation Joern emits for + * untyped receivers. Edge-only (ambiguous by definition). + * + * Linker resolutions are returned in a map and applied to the L1 `call` body nodes by + * `backfillCallees` — NEVER written into `callee_signature` (the symbol table round-trips the + * analysis cache; a persisted resolution would resurface on a warm run with tsc provenance). + */ +import { Node, SyntaxKind } from "ts-morph"; +import { CALL_DEP, type TSCallEdge, type TSCallable, type TSCallsite, type TSExternalSymbol, forEachCallable } from "../schema"; +import { computeSignatureForDecl, externalHomeOf, fileKeyOf, isCallableDecl, resolveCalleeSignature } from "../schema"; +import { callBodyKeys } from "../schema/l1Body"; +import type { CallGraphContext } from "./provider"; +import type { CallGraphResult } from "./callGraph"; +import { inInstancePropInit, indexCallExpressions } from "./callGraph"; + +/** Per-call-site resolutions for the sanctioned `callee: null→id` refinement: callerSig → bodyKey → calleeSig. */ +export type LinkerResolutions = Map>; + +export interface LinkerOutput { + result: CallGraphResult; + resolutions: LinkerResolutions; +} + +// ponytail: fixed small bounds; tune from the Joern ledger, not from flags (spec: no backend flag). +const ALIAS_CHASE_LIMIT = 8; // hops through `const f = g` chains +const CHA_FAN_LIMIT = 16; // max name-matched targets per T5 site + +export function runDefuseLinker(ctx: CallGraphContext): LinkerOutput { + const { project, symbol_table, root, log } = ctx; + + // The signature universe (full table — cross-program targets resolve) + the name→sigs CHA index. + const allSignatures = new Set(); + const byName = new Map(); + for (const mod of Object.values(symbol_table)) { + forEachCallable(mod, (c) => { + allSignatures.add(c.signature); + const arr = byName.get(c.name) ?? []; + arr.push(c.signature); + byName.set(c.name, arr); + }); + } + for (const sigs of byName.values()) sigs.sort(); + + // Callables to iterate: this program's modules only, sorted for determinism. + const callables: TSCallable[] = []; + for (const [key, mod] of Object.entries(symbol_table)) { + if (ctx.only && !ctx.only.has(key)) continue; + forEachCallable(mod, (c) => callables.push(c)); + } + callables.sort((a, b) => a.signature.localeCompare(b.signature)); + + const callExprIndex = indexCallExpressions(project); + /** The callee expression of a call/new/tagged-template node. */ + const calleeExprOf = (node: Node): Node => + Node.isTaggedTemplateExpression(node) ? node.getTag() : (node as unknown as { getExpression: () => Node }).getExpression(); + const nodeOf = (c: TSCallable, cs: TSCallsite): Node | undefined => + callExprIndex.get(`${c.abs_path}#${cs.start_line}:${cs.start_column}-${cs.end_line}:${cs.end_column}`); + + // A resolved-but-external callee is one whose signature is not in the symbol table. + const isExternalSig = (sig: string | undefined): boolean => !!sig && !allSignatures.has(sig); + + // --------------------------------------------------------------------------------------------- + // edge/resolution accumulation + // --------------------------------------------------------------------------------------------- + const edges = new Map(); + const addEdge = (source: string, target: string): void => { + const k = `${source} ${target}`; + const ex = edges.get(k); + if (ex) ex.weight++; + else edges.set(k, { source, target, type: CALL_DEP, weight: 1, provenance: ["defuse"], tags: {} }); + }; + const external_symbols: Record = {}; + const resolutions: LinkerResolutions = new Map(); + const resolve = (callerSig: string, bodyKey: string, targetSig: string): void => { + addEdge(callerSig, targetSig); + let m = resolutions.get(callerSig); + if (!m) resolutions.set(callerSig, (m = new Map())); + m.set(bodyKey, targetSig); + }; + + /** + * The signature of the first-party callable a VALUE expression denotes, chasing bounded alias + * chains: a bare arrow/function expression, an identifier for a function declaration, a + * `const f = () => …` binding, or `const f = g` (g eventually a function) — else null. + */ + const functionValueSig = (expr: Node): string | null => { + // IIFE / parenthesized function values: `(() => …)()`, `(function f() {})()`. + while (Node.isParenthesizedExpression(expr)) expr = expr.getExpression(); + if (Node.isArrowFunction(expr) || Node.isFunctionExpression(expr)) { + const s = computeSignatureForDecl(expr, root); + return s && allSignatures.has(s) ? s : null; + } + if (!Node.isIdentifier(expr)) return null; + let node: Node = expr; + for (let hop = 0; hop < ALIAS_CHASE_LIMIT; hop++) { + let sym = node.getSymbol(); + if (!sym) return null; + const aliased = sym.getAliasedSymbol(); + if (aliased) sym = aliased; + const decl = sym.getDeclarations()?.[0]; + if (!decl) return null; + if (Node.isFunctionDeclaration(decl) || Node.isArrowFunction(decl) || Node.isFunctionExpression(decl) || Node.isMethodDeclaration(decl)) { + const s = computeSignatureForDecl(decl, root); + return s && allSignatures.has(s) ? s : null; + } + if (Node.isVariableDeclaration(decl)) { + const init = decl.getInitializer(); + if (!init) return null; + if (Node.isArrowFunction(init) || Node.isFunctionExpression(init)) { + const s = computeSignatureForDecl(decl, root); + return s && allSignatures.has(s) ? s : null; + } + if (Node.isIdentifier(init)) { + node = init; // alias chain: keep chasing + continue; + } + return null; + } + return null; + } + return null; + }; + + /** The parameter index of `expr` within `enclosing`, when it names one of its parameters. */ + const paramIndexOf = (expr: Node, enclosing: TSCallable): number | null => { + if (!Node.isIdentifier(expr)) return null; + const decl = expr.getSymbol()?.getDeclarations()?.[0]; + if (!decl || !Node.isParameterDeclaration(decl)) return null; + const name = expr.getText(); + const idx = enclosing.parameters.findIndex((p) => p.name === name); + return idx >= 0 ? idx : null; + }; + + // --------------------------------------------------------------------------------------------- + // main site sweep: T1 chase, T3 callback rule, and the T4/T5 worklists + // --------------------------------------------------------------------------------------------- + interface ParamSite { + enclosing: TSCallable; + bodyKey: string; + paramIndex: number; + propertyName?: string; // `template.onChange(...)` where `template` is the parameter + } + interface FactorySite { + enclosing: TSCallable; + bodyKey: string; + factorySig: string; // resolved-internal callee of the binding's initializer call + } + interface ReceiverSite { + enclosing: TSCallable; + cs: TSCallsite; + } + interface ThisFieldSite { + enclosing: TSCallable; + bodyKey: string; + node: Node; + fieldName: string; + } + const paramSites: ParamSite[] = []; + const factorySites: FactorySite[] = []; + const receiverSites: ReceiverSite[] = []; + const thisFieldSites: ThisFieldSite[] = []; + // Reverse index for T4 voting: internal target sig → the AST argument lists of its call sites. + const argsByTarget = new Map(); + const recordCallArgs = (targetSig: string, node: Node | undefined): void => { + if (!node || !allSignatures.has(targetSig)) return; + const args = (node as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; + if (!args.length) return; + const arr = argsByTarget.get(targetSig) ?? []; + arr.push(args); + argsByTarget.set(targetSig, arr); + }; + + let t1 = 0; + let t3 = 0; + for (const c of callables) { + for (const [bodyKey, cs] of callBodyKeys(c.call_sites)) { + const node = nodeOf(c, cs); + if (cs.callee_signature) { + recordCallArgs(cs.callee_signature, node); + } else if (node) { + const expr = calleeExprOf(node); + // T1 — local value chase on the callee expression itself. + const chased = functionValueSig(expr); + if (chased) { + resolve(c.signature, bodyKey, chased); + recordCallArgs(chased, node); + t1++; + } else { + const pIdx = paramIndexOf(expr, c); + if (pIdx !== null) { + paramSites.push({ enclosing: c, bodyKey, paramIndex: pIdx }); + } else if (Node.isIdentifier(expr)) { + // T4b — `const f = factory(); f()`: binding initialized by a resolved-internal call. + const decl = expr.getSymbol()?.getDeclarations()?.[0]; + const init = decl && Node.isVariableDeclaration(decl) ? decl.getInitializer() : undefined; + if (init && Node.isCallExpression(init)) { + const r = resolveCalleeSignature(init, root, allSignatures); + if (r && !r.external && allSignatures.has(r.signature)) { + factorySites.push({ enclosing: c, bodyKey, factorySig: r.signature }); + } + } + } else if (Node.isPropertyAccessExpression(expr) && cs.receiver_expr != null && !cs.is_constructor_call) { + const recvIdx = paramIndexOf(expr.getExpression(), c); + if (recvIdx !== null) { + // T4a property form: the receiver IS a parameter — candidates are the matching + // object-literal property values passed at that position by resolved callers. + paramSites.push({ enclosing: c, bodyKey, paramIndex: recvIdx, propertyName: expr.getName() }); + } else if (cs.receiver_expr === "this") { + // T4c — `this.field(...)`: the field's value flows from the constructor (a + // parameter property or a `this.field = …` assignment); resolved below, feeding + // the T4 vote rounds. Falls back to T5 if the chain yields nothing. + thisFieldSites.push({ enclosing: c, bodyKey, node, fieldName: cs.method_name }); + } else { + receiverSites.push({ enclosing: c, cs }); + } + } + } + } + // T3 — external-callback rule: function values handed to an external/unresolved callee. + if (node && (!cs.callee_signature || isExternalSig(cs.callee_signature))) { + const args = (node as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; + for (const arg of args) { + const fn = functionValueSig(arg); + if (fn && fn !== c.signature) { + addEdge(c.signature, fn); + t3++; + } + } + } + } + } + + // --------------------------------------------------------------------------------------------- + // Module-scope sweep (python #131 parity: module-scope execution is attributed to the MODULE). + // These sites have no call_sites record and no body node — T1 chase and the T3 callback rule + // apply edge-only, with the module prefix as the source. + // --------------------------------------------------------------------------------------------- + const enclosingCallable = (node: Node): Node | undefined => { + for (const a of node.getAncestors()) if (isCallableDecl(a)) return a; + return undefined; + }; + for (const [, node] of [...callExprIndex.entries()].sort(([a], [b]) => a.localeCompare(b))) { + if (enclosingCallable(node) || inInstancePropInit(node)) continue; + const fk = fileKeyOf(node.getSourceFile().getFilePath(), root); + if (ctx.only && !ctx.only.has(fk.fileKey)) continue; + const source = fk.modulePrefix; + const r = resolveCalleeSignature(node, root, allSignatures); + if (r && !r.external) recordCallArgs(r.signature, node); // top-level `register("a", cb)` feeds the vote rounds + if (!r) { + // T1 at module scope: `const f = handler; f()` in top-level code. + const expr = calleeExprOf(node); + const chased = functionValueSig(expr); + if (chased) { + addEdge(source, chased); + t1++; + } + } + // T3 at module scope — the dominant express idiom: `app.get('/x', handler)` top-level. + if (!r || r.external) { + const args = (node as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; + for (const arg of args) { + const fn = functionValueSig(arg); + if (fn && fn !== source) { + addEdge(source, fn); + t3++; + } + } + } + } + + // --------------------------------------------------------------------------------------------- + // T2 — decorator invocations (edge-only). The SOURCE is where the decorator executes: the + // decorated callable for method/accessor/parameter decorators; the MODULE for class and + // property decorators (a decorator on a top-level definition runs in module scope — python + // #131's rule, and Joern's own attribution). + // --------------------------------------------------------------------------------------------- + let t2 = 0; + const files = [...project.getSourceFiles()] + .filter((sf) => !sf.isDeclarationFile() && !sf.getFilePath().includes("/node_modules/")) + .sort((a, b) => a.getFilePath().localeCompare(b.getFilePath())); + for (const sf of files) { + sf.forEachDescendant((n) => { + if (!Node.isDecorator(n)) return; + // The edge SOURCE is where the decorator executes: the decorated callable for method/ + // accessor/parameter decorators; the MODULE prefix for class and property decorators. + let owner = n.getParent(); + if (owner && Node.isParameterDeclaration(owner)) owner = owner.getParent(); + let ownerSig: string | null = null; + if (owner && (Node.isMethodDeclaration(owner) || Node.isGetAccessorDeclaration(owner) || Node.isSetAccessorDeclaration(owner))) { + ownerSig = computeSignatureForDecl(owner, root); + if (!ownerSig || !allSignatures.has(ownerSig)) return; + } else if (owner && (Node.isClassDeclaration(owner) || Node.isPropertyDeclaration(owner))) { + ownerSig = fileKeyOf(sf.getFilePath(), root).modulePrefix; + } else { + return; + } + const expr = n.getExpression(); + let targetSig: string | null = null; + let external: { module: string; member: string } | null = null; + if (Node.isCallExpression(expr)) { + const r = resolveCalleeSignature(expr, root, allSignatures); + if (r) { + targetSig = r.signature; + external = r.external ?? null; + } + } else if (Node.isIdentifier(expr)) { + const direct = functionValueSig(expr); + if (direct) targetSig = direct; + else { + const decl = expr.getSymbol()?.getAliasedSymbol()?.getDeclarations()?.[0] ?? expr.getSymbol()?.getDeclarations()?.[0]; + const home = decl ? externalHomeOf(decl) : null; + if (home) { + const member = expr.getText(); + targetSig = `${home.module}.${member}`; + external = { module: home.module, member }; + } + } + } + if (!targetSig) return; + if (external) { + if (!ctx.phantoms) return; + if (!external_symbols[targetSig]) external_symbols[targetSig] = { name: external.member, module: external.module }; + } + addEdge(ownerSig, targetSig); + t2++; + }); + } + + // --------------------------------------------------------------------------------------------- + // T4c — `this.field(...)` through the constructor: a field assigned from a ctor parameter + // (parameter property or `this.f = param`) calls whatever function values the class's `new` + // sites passed at that position; a field assigned a function value in the ctor calls it + // directly. Candidates feed argsByTarget so the T4 rounds resolve the callbacks' OWN + // param-invoking sites (`write()` inside a registered migration callback). Bounded: direct + // ctor args only, no transitive flow. + // --------------------------------------------------------------------------------------------- + const classFieldSources = new Map>(); + const fieldSourcesOf = (cls: Node): Map => { + let m = classFieldSources.get(cls); + if (m) return m; + m = new Map(); + const ctor = (cls as unknown as { getConstructors?: () => Node[] }).getConstructors?.()?.[0]; + if (ctor) { + const params = (ctor as unknown as { getParameters: () => Node[] }).getParameters(); + params.forEach((p, i) => { + const pp = p as unknown as { getName: () => string; getModifiers?: () => Node[] }; + if ((pp.getModifiers?.() ?? []).length) m?.set(pp.getName(), { paramIndex: i }); + }); + const paramNames = new Map(params.map((p, i) => [(p as unknown as { getName: () => string }).getName(), i])); + ctor.forEachDescendant((d) => { + if (!Node.isBinaryExpression(d) || d.getOperatorToken().getKind() !== SyntaxKind.EqualsToken) return; + const lhs = d.getLeft(); + if (!Node.isPropertyAccessExpression(lhs) || lhs.getExpression().getKind() !== SyntaxKind.ThisKeyword) return; + const rhs = d.getRight(); + const idx = Node.isIdentifier(rhs) ? paramNames.get(rhs.getText()) : undefined; + if (idx !== undefined) m?.set(lhs.getName(), { paramIndex: idx }); + else { + const direct = functionValueSig(rhs); + if (direct) m?.set(lhs.getName(), { direct }); + } + }); + } + classFieldSources.set(cls, m); + return m; + }; + /** Function values an ARGUMENT node denotes — directly, or through one bounded parameter hop: + * when the arg is a parameter of the function containing the call, the values passed for that + * parameter at ITS resolved call sites are the candidates (`register(key, migrate)` → + * `new Migration(key, migrate)`). One hop, no fixpoint. */ + const argFlowCandidates = (arg: Node): string[] => { + const direct = functionValueSig(arg); + if (direct) return [direct]; + if (!Node.isIdentifier(arg)) return []; + const decl = arg.getSymbol()?.getDeclarations()?.[0]; + if (!decl || !Node.isParameterDeclaration(decl)) return []; + const owner = decl.getParent(); + if (!owner) return []; + const ownerSig = computeSignatureForDecl(owner, root); + if (!ownerSig) return []; + const idx = ((owner as unknown as { getParameters?: () => Node[] }).getParameters?.() ?? []).findIndex((p) => p === decl); + if (idx < 0) return []; + const out = new Set(); + for (const args of argsByTarget.get(ownerSig) ?? []) { + const a = args[idx]; + const fn = a ? functionValueSig(a) : null; + if (fn) out.add(fn); + } + return [...out].sort(); + }; + let t4c = 0; + for (const site of thisFieldSites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature) || a.bodyKey.localeCompare(b.bodyKey))) { + const cls = site.node.getAncestors().find((a) => Node.isClassDeclaration(a) || Node.isClassExpression(a)); + const src = cls ? fieldSourcesOf(cls).get(site.fieldName) : undefined; + const candidates = new Set(); + if (src?.direct) candidates.add(src.direct); + if (src?.paramIndex !== undefined && cls) { + const clsSig = computeSignatureForDecl(cls, root); + for (const args of argsByTarget.get(`${clsSig}.constructor`) ?? []) { + const arg = args[src.paramIndex]; + for (const fn of arg ? argFlowCandidates(arg) : []) candidates.add(fn); + } + } + if (!candidates.size) { + const cs = site.enclosing.call_sites.find((c2) => `${c2.start_line}:${c2.start_column}` === site.bodyKey.split("/")[0]); + if (cs) receiverSites.push({ enclosing: site.enclosing, cs }); // fall back to T5 + continue; + } + const sorted = [...candidates].sort(); + for (const target of sorted) { + addEdge(site.enclosing.signature, target); + recordCallArgs(target, site.node); // the callbacks' own param sites resolve in the T4 rounds + t4c++; + } + if (sorted.length === 1) { + let m = resolutions.get(site.enclosing.signature); + if (!m) resolutions.set(site.enclosing.signature, (m = new Map())); + m.set(site.bodyKey, sorted[0] as string); + } + } + + // --------------------------------------------------------------------------------------------- + // T4 — bounded votes, two rounds (round one's resolutions vote before round two). + // --------------------------------------------------------------------------------------------- + let t4 = 0; + for (let round = 0; round < 2 && paramSites.length; round++) { + const unresolvedNext: ParamSite[] = []; + for (const site of paramSites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature) || a.bodyKey.localeCompare(b.bodyKey))) { + const candidates = new Set(); + for (const args of argsByTarget.get(site.enclosing.signature) ?? []) { + const arg = args[site.paramIndex]; + if (!arg) continue; + if (site.propertyName !== undefined) { + // object-literal property flow: `render({ onChange: fn })` → `template.onChange()` + if (Node.isObjectLiteralExpression(arg)) { + const prop = arg.getProperty(site.propertyName); + const init = prop && Node.isPropertyAssignment(prop) ? prop.getInitializer() : undefined; + const fn = init ? functionValueSig(init) : null; + if (fn) candidates.add(fn); + else if (prop && Node.isMethodDeclaration(prop)) { + const s2 = computeSignatureForDecl(prop, root); + if (s2 && allSignatures.has(s2)) candidates.add(s2); + } + } + } else { + const fn = functionValueSig(arg); + if (fn) candidates.add(fn); + } + } + if (!candidates.size) { + unresolvedNext.push(site); + continue; + } + const sorted = [...candidates].sort(); + for (const target of sorted) { + addEdge(site.enclosing.signature, target); + t4++; + // Round-one resolutions feed round two's votes: a cb() site that now targets `target` + // makes the enclosing callable a resolved-internal caller of it. + } + if (sorted.length === 1) { + let m = resolutions.get(site.enclosing.signature); + if (!m) resolutions.set(site.enclosing.signature, (m = new Map())); + m.set(site.bodyKey, sorted[0] as string); + } + } + paramSites.length = 0; + paramSites.push(...unresolvedNext); + } + // T4b — factory returns: resolve through the factory's unique returned function value. + const returnSummary = new Map(); + const uniqueReturnedFn = (factorySig: string): string | null => { + if (returnSummary.has(factorySig)) return returnSummary.get(factorySig) as string | null; + let out: string | null = null; + // Find the factory's AST via any recorded call-site node? Cheaper: search the sorted callables + // list (same program) for the signature, then its declaration through the call-expression + // index is unavailable — walk the source file at its span instead. + const fc = callables.find((c) => c.signature === factorySig); + if (fc) { + const sf = project.getSourceFile(fc.abs_path); + const declNode = sf?.getDescendantAtPos(fc.span.bytes[0]); + const fnNode = declNode ? [declNode, ...declNode.getAncestors()].find((a) => computeSignatureForDecl(a, root) === factorySig) : undefined; + if (fnNode) { + returnSummary.set(factorySig, null); // cycle guard before descending + const returned = new Set(); + fnNode.forEachDescendant((d) => { + if (!Node.isReturnStatement(d)) return; + const e = d.getExpression(); + if (!e) return; + const fn = functionValueSig(e); + if (fn) { + returned.add(fn); + return; + } + // chained: `return makeInner()` — follow ONE resolved-internal level, memoized + if (Node.isCallExpression(e)) { + const r = resolveCalleeSignature(e, root, allSignatures); + if (r && !r.external && allSignatures.has(r.signature)) { + const inner = uniqueReturnedFn(r.signature); + if (inner) { + returned.add(inner); + return; + } + } + } + returned.add(""); + }); + if (returned.size === 1 && !returned.has("")) out = [...returned][0] as string; + } + } + returnSummary.set(factorySig, out); + return out; + }; + for (const site of factorySites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature) || a.bodyKey.localeCompare(b.bodyKey))) { + const target = uniqueReturnedFn(site.factorySig); + if (target) { + resolve(site.enclosing.signature, site.bodyKey, target); + t4++; + } + } + + // --------------------------------------------------------------------------------------------- + // T5 — CHA-by-name fallback (edge-only, bounded fan). + // --------------------------------------------------------------------------------------------- + let t5 = 0; + for (const site of receiverSites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature))) { + if (!site.cs) continue; + const candidates = (byName.get(site.cs.method_name) ?? []).filter((s) => s !== site.enclosing.signature); + // Over-cap names (get/set/toString-class fan) are skipped outright, not truncated — a partial + // arbitrary subset would be neither sound-leaning nor deterministic in meaning. + if (!candidates.length || candidates.length > CHA_FAN_LIMIT) continue; + for (const target of candidates) { + addEdge(site.enclosing.signature, target); + t5++; + } + } + + const sortedEdges = [...edges.values()].sort((a, b) => a.source.localeCompare(b.source) || a.target.localeCompare(b.target)); + log.info(`call graph (defuse): ${sortedEdges.length} edges — t1=${t1} chase, t2=${t2} decorator, t3=${t3} callback, t4=${t4} votes, t4c=${t4c} ctor-field, t5=${t5} cha`); + return { + result: { edges: sortedEdges, external_symbols, synthesized_callables: {} }, + resolutions, + }; +} diff --git a/src/semantic_analysis/index.ts b/src/semantic_analysis/index.ts index 76df475..edc5d5b 100644 --- a/src/semantic_analysis/index.ts +++ b/src/semantic_analysis/index.ts @@ -1,5 +1,4 @@ -// Call-graph construction: the tsc (ts-morph checker) resolver graph + RTA. +// Call-graph construction: the tsc (ts-morph checker) resolver graph + RTA + the defuse linker. export * from "./callGraph"; -// The provider seam (union | tsc | jelly) + the Jelly backend. export * from "./provider"; -export * from "./jellyProvider"; +export * from "./defuseLinker"; diff --git a/src/semantic_analysis/jellyProvider.ts b/src/semantic_analysis/jellyProvider.ts deleted file mode 100644 index d7f6af7..0000000 --- a/src/semantic_analysis/jellyProvider.ts +++ /dev/null @@ -1,397 +0,0 @@ -/** - * Jelly call-graph provider. Shells out to `@cs-au-dk/jelly` (CLI/JSON only — no library API), - * then maps each Jelly function node back onto a symbol-table signature by source span: - * - * jelly id "fileIdx:sl:sc:el:ec" -> files[fileIdx] + (sl,sc) -> ts-morph node -> signature - * - * Named declarations reuse the existing canonicalizer (computeSignatureForDecl); anonymous inline - * callbacks — which the canonicalizer returns null for — get a SYNTHESIZED signature of the form - * `:`, mirroring how Jelly itself identifies - * anonymous functions purely by location. Jelly's columns are 1-based (it exports column+1), which - * lines up with ts-morph's 1-based columns, so (file,startLine,startColumn) is a direct join key. - * - * This provider is read-only w.r.t. the symbol table: it emits edges over its own node universe - * (real signatures ∪ synthesized) for diffing. Materializing synthesized callables into the symbol - * table is a later step, only needed when jelly is promoted to authoritative. - * - * Whole-program, scoped to declared deps: Jelly follows imports into node_modules, but we exclude - * every installed package NOT listed in the project's package.json `dependencies`. This keeps the - * directly-used library surface (the deps that actually matter for edge resolution) while cutting - * transitive bloat — critically `@ts-morph/common`, which bundles the entire ~8.7MB TypeScript - * compiler and makes unbounded whole-program analysis OOM. - * - * Tier 2 — dependency functions are materialized as external symbols so edges crossing the - * first-party↔library boundary are KEPT in both directions, tagged `ts.external`/`ts.module` (like - * the tsc phantom mechanism). Keying differs by direction because Jelly carries no function names: - * • first-party → dep: keyed `module.member` via the first-party call site (resolvePhantom), - * matching the tsc phantom keys so the two providers' external symbols are comparable. - * • dep → first-party: no first-party call site to read, so keyed by package + location. This is - * the entrypoint signal — a framework invoking your handler. - * Only dep→dep edges (neither endpoint first-party) are dropped. - */ -import { execFileSync } from "node:child_process"; -import * as fs from "node:fs"; -import { createRequire } from "node:module"; -import * as os from "node:os"; -import * as path from "node:path"; -import { Node, type SourceFile } from "ts-morph"; -import { - CALL_DEP, - computeSignatureForDecl, - fileKeyOf, - type TSCallEdge, - type TSExternalSymbol, - type TSSynthesizedCallable, -} from "../schema"; -import type { CallGraphResult } from "./callGraph"; -import { type ExternalIndex, buildExternalIndex, resolvePhantom } from "./phantoms"; -import type { CallGraphContext, CallGraphProvider } from "./provider"; - -const requireFrom = createRequire(import.meta.url); - -interface JellyJson { - files: string[]; - functions: Record; // id -> "fileIdx:startLine:startCol:endLine:endCol" - fun2fun: [number, number][]; // [callerId, calleeId] - call2fun: [number, number][]; // [callSiteId, calleeId] - calls: Record; // callSiteId -> "fileIdx:sl:sc:el:ec" (span in the CALLER file) -} - -/** Locate Jelly's entry script: explicit override, else the installed package. */ -function resolveJellyMain(): string { - if (process.env.JELLY_BIN) return process.env.JELLY_BIN; - try { - return requireFrom.resolve("@cs-au-dk/jelly/lib/main.js"); - } catch { - throw new Error("@cs-au-dk/jelly not installed and JELLY_BIN unset"); - } -} - -/** The project's declared runtime dependencies — the packages we want Jelly to descend into. */ -function declaredDeps(root: string): Set { - try { - const pkg = JSON.parse(fs.readFileSync(path.join(root, "package.json"), "utf8")) as { - dependencies?: Record; - }; - return new Set(Object.keys(pkg.dependencies ?? {})); - } catch { - return new Set(); - } -} - -/** Every installed package name under node_modules (descending one level into @scope dirs). */ -function installedPackages(root: string): string[] { - const nm = path.join(root, "node_modules"); - let top: string[]; - try { - top = fs.readdirSync(nm); - } catch { - return []; - } - const out: string[] = []; - for (const e of top) { - if (e.startsWith(".")) continue; - if (e.startsWith("@")) { - try { - for (const sub of fs.readdirSync(path.join(nm, e))) if (!sub.startsWith(".")) out.push(`${e}/${sub}`); - } catch { - /* unreadable scope dir */ - } - } else { - out.push(e); - } - } - return out; -} - -/** Installed packages NOT declared as dependencies — excluded so whole-program stays tractable. */ -function excludedPackages(root: string): string[] { - const keep = declaredDeps(root); - return installedPackages(root).filter((p) => !keep.has(p)); -} - -function runJelly(ctx: CallGraphContext, entryFiles: string[]): JellyJson { - const out = path.join(os.tmpdir(), `cants-jelly-${process.pid}.json`); - const excluded = excludedPackages(ctx.root); - ctx.log.debug(`call graph (jelly): excluding ${excluded.length} non-declared packages from whole-program scope`); - // Whole-program over first-party + declared deps only (no --ignore-dependencies, but exclude the - // rest). `--` terminates the variadic --exclude-packages list before the positional entry files. - const jellyArgs = ["-j", out]; - if (excluded.length) jellyArgs.push("--exclude-packages", ...excluded, "--"); - jellyArgs.push(...entryFiles); - - // Two launch modes. Compiled single-binary (CANTS_SELF_JELLY set by src/main.ts): re-exec THIS - // executable with the hidden `__jelly` subcommand — the Jelly CLI is bundled in, so no external - // `node` or node_modules is needed. Dev/source or explicit JELLY_BIN override: shell out to - // `node @cs-au-dk/jelly/lib/main.js` as before. - const self = process.env.CANTS_SELF_JELLY && !process.env.JELLY_BIN ? process.env.CANTS_SELF_JELLY : null; - const cmd = self ?? "node"; - const args = self ? ["__jelly", ...jellyArgs] : [resolveJellyMain(), ...jellyArgs]; - try { - execFileSync(cmd, args, { - cwd: ctx.root, - stdio: ["ignore", "ignore", "ignore"], - maxBuffer: 256 * 1024 * 1024, - timeout: 600_000, - env: { ...process.env, NODE_OPTIONS: `${process.env.NODE_OPTIONS ?? ""} --max-old-space-size=8192`.trim() }, - }); - return JSON.parse(fs.readFileSync(out, "utf8")) as JellyJson; - } finally { - try { - fs.rmSync(out, { force: true }); - } catch { - /* best-effort cleanup */ - } - } -} - -/** getDescendantAtPos lands on the token at the span start; climb to the enclosing function node. */ -function climbToFunctionLike(node: Node | undefined): Node | undefined { - let n = node; - while ( - n && - !( - Node.isArrowFunction(n) || - Node.isFunctionExpression(n) || - Node.isFunctionDeclaration(n) || - Node.isMethodDeclaration(n) || - Node.isConstructorDeclaration(n) || - Node.isGetAccessorDeclaration(n) || - Node.isSetAccessorDeclaration(n) - ) - ) { - n = n.getParent(); - } - return n; -} - -/** Climb from the token at a call-site span to the enclosing call/new expression. */ -function climbToCallExpr(node: Node | undefined): Node | undefined { - let n = node; - while (n && !(Node.isCallExpression(n) || Node.isNewExpression(n))) n = n.getParent(); - return n; -} - -/** Synthetic signature for an anonymous callback: nearest signed enclosing scope + location suffix. */ -function synthesize(fnNode: Node, root: string): string { - let host: Node | undefined = fnNode.getParent(); - while (host && computeSignatureForDecl(host, root) === null) host = host.getParent(); - const hostSig = host ? computeSignatureForDecl(host, root) : null; - const { line, column } = fnNode.getSourceFile().getLineAndColumnAtPos(fnNode.getStart()); - return `${hostSig ?? ""}:<${line}:${column}>`; -} - -/** - * If a Jelly file path lives under node_modules, split it into the owning package name and the - * path within that package. Uses the LAST `node_modules/` so nested deps resolve to the innermost - * package, and handles `@scope/name`. - */ -function depPackage(rel: string): { pkg: string; inPkg: string } | null { - const marker = "node_modules/"; - const idx = rel.lastIndexOf(marker); - if (idx < 0) return null; - const parts = rel.slice(idx + marker.length).split("/"); - if (parts[0].startsWith("@")) { - if (parts.length < 2) return null; - return { pkg: `${parts[0]}/${parts[1]}`, inPkg: parts.slice(2).join("/") }; - } - return { pkg: parts[0], inPkg: parts.slice(1).join("/") }; -} - -/** - * Map a function node to its signature. A `const foo = () => …` arrow is named by its - * VariableDeclaration in the symbol table, so normalize to that parent before deciding - * real-vs-synthesized — otherwise every named const-arrow would wrongly synthesize. - */ -function signatureFor(fn: Node, root: string): { sig: string; synth: boolean } { - const parent = fn.getParent(); - const decl = parent && Node.isVariableDeclaration(parent) ? parent : fn; - const real = computeSignatureForDecl(decl, root); - if (real) return { sig: real, synth: false }; - return { sig: synthesize(fn, root), synth: true }; -} - -export const jellyProvider: CallGraphProvider = { - name: "jelly", - build(ctx): CallGraphResult { - const entryFiles = ctx.project - .getSourceFiles() - .map((sf) => sf.getFilePath() as string) - .filter((fp) => !fp.includes("/node_modules/") && !fp.endsWith(".d.ts")) - .map((fp) => path.relative(ctx.root, fp)) - .filter((rel) => rel.length > 0 && !rel.startsWith("..")); - - if (entryFiles.length === 0) { - ctx.log.info("call graph (jelly): no first-party source files to analyze"); - return { edges: [], external_symbols: {}, synthesized_callables: {} }; - } - - const cg = runJelly(ctx, entryFiles); - - // Phase 1: classify each Jelly function. First-party functions round-trip to a ts-morph node and - // reuse the canonicalizer (real or synthesized). Dependency functions are recorded by package + - // location; an external symbol is minted for them lazily, when an edge reveals how they connect. - const id2sig = new Map(); // first-party id -> signature - const firstPartyIds = new Set(); - const depMeta = new Map(); - let synthesized = 0; - let unresolved = 0; - - // Anonymous callbacks get a synthesized signature with no symbol-table node; remember their - // location so the projection can materialize a node and the edge won't dangle (issue #13). - const synthesizedCallables: Record = {}; - const recordIfSynth = (fn: Node, sig: string, synth: boolean): void => { - if (!synth || synthesizedCallables[sig]) return; - const { line, column } = fn.getSourceFile().getLineAndColumnAtPos(fn.getStart()); - synthesizedCallables[sig] = { - name: "", - path: fileKeyOf(fn.getSourceFile().getFilePath(), ctx.root).fileKey, - start_line: line, - start_column: column, - }; - }; - for (const [id, loc] of Object.entries(cg.functions)) { - const [fileIdx, sl, sc] = loc.split(":").map(Number); - const rel = cg.files[fileIdx]; - if (rel === undefined) { - unresolved++; - continue; - } - const dep = depPackage(rel); - if (dep) { - depMeta.set(id, { pkg: dep.pkg, inPkg: dep.inPkg, sl, sc }); - continue; - } - const sf = ctx.project.getSourceFile(path.resolve(ctx.root, rel)); - if (!sf) { - unresolved++; - continue; - } - let offset: number; - try { - offset = sf.compilerNode.getPositionOfLineAndCharacter(sl - 1, sc - 1); - } catch { - unresolved++; - continue; - } - const fn = climbToFunctionLike(sf.getDescendantAtPos(offset)); - if (!fn) { - unresolved++; // module-level node and other non-function spans - continue; - } - const { sig, synth } = signatureFor(fn, ctx.root); - id2sig.set(id, sig); - firstPartyIds.add(id); - if (synth) synthesized++; - recordIfSynth(fn, sig, synth); - } - - const external_symbols: Record = {}; - const edges = new Map(); - let boundary = 0; - let dropped = 0; - const addEdge = (source: string, target: string, tags: Record): void => { - const k = `${source} ${target}`; - const ex = edges.get(k); - if (ex) ex.weight++; - else edges.set(k, { source, target, type: CALL_DEP, weight: 1, provenance: ["jelly"], tags }); - }; - // Location-keyed fallback signature for a dep function — used when no call-site member name is - // available (the dep→first-party direction, where the call site is inside the library). - const depLocSig = (d: { pkg: string; inPkg: string; sl: number; sc: number }): string => { - const sig = `${d.pkg}:${d.inPkg}:<${d.sl}:${d.sc}>`; - if (!external_symbols[sig]) external_symbols[sig] = { name: `${d.inPkg}:${d.sl}:${d.sc}`, module: d.pkg }; - return sig; - }; - // Per-file import/require index, for naming a library member at a first-party call site. - const extIndexCache = new Map(); - const extIndexFor = (sf: SourceFile): ExternalIndex => { - const key = sf.getFilePath(); - let idx = extIndexCache.get(key); - if (!idx) { - idx = buildExternalIndex(sf as unknown as Node); - extIndexCache.set(key, idx); - } - return idx; - }; - - // Phase 2a — first-party → dependency, NAMED via the call site. call2fun maps call-site → callee; - // we resolve the site in first-party source to (caller function, library member) and key the - // external symbol as `module.member`, matching the tsc phantom path so the two are comparable. - for (const [callId, calleeId] of cg.call2fun) { - const dep = depMeta.get(String(calleeId)); - if (!dep) continue; // callee is first-party (via fun2fun) or unresolved - const cloc = cg.calls[String(callId)]; - if (!cloc) continue; - const [cFileIdx, csl, csc] = cloc.split(":").map(Number); - const crel = cg.files[cFileIdx]; - if (crel === undefined || depPackage(crel)) continue; // the call site must be first-party - const sf = ctx.project.getSourceFile(path.resolve(ctx.root, crel)); - if (!sf) continue; - let coff: number; - try { - coff = sf.compilerNode.getPositionOfLineAndCharacter(csl - 1, csc - 1); - } catch { - continue; - } - const callNode = climbToCallExpr(sf.getDescendantAtPos(coff)); - if (!callNode) continue; - const callerFn = climbToFunctionLike(callNode); - if (!callerFn) continue; // top-level call, no enclosing function - const { sig: callerSig, synth: callerSynth } = signatureFor(callerFn, ctx.root); - recordIfSynth(callerFn, callerSig, callerSynth); - const ph = resolvePhantom(callNode, extIndexFor(sf)); - let sig: string; - if (ph) { - sig = ph.signature; // module.member - if (!external_symbols[sig]) external_symbols[sig] = { name: ph.member, module: ph.module }; - } else { - sig = depLocSig(dep); // unresolved import — fall back to the location key - } - addEdge(callerSig, sig, { "ts.external": "true", "ts.module": external_symbols[sig].module }); - boundary++; - } - - // Phase 2b — fun2fun for first-party→first-party (internal) and dependency→first-party (the - // entrypoint signal: a library invoking your code). first-party→dep is named in 2a; dep→dep and - // edges with an unresolved endpoint are dropped. - for (const [callerId, calleeId] of cg.fun2fun) { - const cid = String(callerId); - const tid = String(calleeId); - const srcFP = firstPartyIds.has(cid); - const tgtFP = firstPartyIds.has(tid); - if (srcFP && tgtFP) { - addEdge(id2sig.get(cid)!, id2sig.get(tid)!, {}); - continue; - } - if (!srcFP && tgtFP) { - const dep = depMeta.get(cid); - if (!dep) { - dropped++; // unresolved caller - continue; - } - addEdge(depLocSig(dep), id2sig.get(tid)!, { "ts.external": "true", "ts.module": dep.pkg }); - boundary++; - continue; - } - if (!(srcFP && !tgtFP)) dropped++; // dep→dep / unresolved (first-party→dep is counted in 2a) - } - - // Keep only synthesized callables that an edge actually references — no orphan nodes. - const referenced = new Set(); - for (const e of edges.values()) { - referenced.add(e.source); - referenced.add(e.target); - } - const synthesized_callables: Record = {}; - for (const [sig, sc] of Object.entries(synthesizedCallables)) if (referenced.has(sig)) synthesized_callables[sig] = sc; - - ctx.log.info( - `call graph (jelly): ${Object.keys(cg.functions).length} jelly funcs, ${firstPartyIds.size} first-party ` + - `(${synthesized} synthesized, ${Object.keys(synthesized_callables).length} materialized), ` + - `${Object.keys(external_symbols).length} external symbols, ${unresolved} unresolved, ` + - `${edges.size} edges (${boundary} library-boundary), ${dropped} dropped`, - ); - return { edges: [...edges.values()], external_symbols, synthesized_callables }; - }, -}; diff --git a/src/semantic_analysis/provider.ts b/src/semantic_analysis/provider.ts index 4caf5cc..ec76134 100644 --- a/src/semantic_analysis/provider.ts +++ b/src/semantic_analysis/provider.ts @@ -1,20 +1,16 @@ /** - * Call-graph provider seam. The orchestrator builds the graph through a CallGraphProvider so the - * backend is swappable: - * • `union` (default) — run tsc + jelly and emit the MERGED edge/node set (tsc ∪ jelly), tagged - * by `provenance` so consumers can still tell the two apart. - * • `tsc` — the always-on ts-morph resolver only (the explicit `--tsc-only` opt-out). - * • `jelly` — the cs-au-dk flow-based analyzer only. - * `both` is a deprecated alias of `union`: it used to run each and log a diff while emitting tsc - * only, which silently discarded every jelly edge and external symbol (see issue #11). + * Call-graph build seam. One backend: the tsc (ts-morph checker) resolver. `tscProvider` stays an + * object (rather than a bare function) so tests can spy on the build being skipped at -a 1. + * `mergeCallGraphs` merges edge/node sets by (source, target) with provenance union — the defuse + * linker's edges overlay the tsc base through it (an edge found by both carries + * `["defuse", "tsc"]` after the wire sort). */ import type { Project } from "ts-morph"; import type { TSExternalSymbol, TSModule } from "../schema"; import type { Logger } from "../utils"; import { buildCallGraph, type CallGraphResult } from "./callGraph"; -import { jellyProvider } from "./jellyProvider"; -/** Everything a provider needs to produce a call graph over the analyzed project. */ +/** Everything the builder needs to produce a call graph over the analyzed project. */ export interface CallGraphContext { project: Project; symbol_table: Record; @@ -32,7 +28,7 @@ export interface CallGraphProvider { build(ctx: CallGraphContext): CallGraphResult; } -/** The always-available backend — wraps the existing tsc resolver with zero behavior change. */ +/** The one backend — the ts-morph checker resolver (+ RTA + phantoms). */ export const tscProvider: CallGraphProvider = { name: "tsc", build: (ctx) => buildCallGraph(ctx.project, ctx.symbol_table, ctx.root, ctx.log, ctx.phantoms, ctx.only), @@ -41,11 +37,9 @@ export const tscProvider: CallGraphProvider = { /** * Merge two call-graph results into their union. Pure (no I/O) so it can be unit-tested directly. * - * Edges are keyed by `(source, target)`. A duplicate edge sums its weight, unions its `provenance` - * (so an edge found by both providers carries `["tsc", "jelly"]`), and merges its tags (base wins - * on conflict — the tsc edge is the authoritative one for the shared key). External symbols union - * by signature, base winning on conflict. `a` is treated as the base (tsc), `b` as the overlay - * (jelly). + * Edges are keyed by `(source, target)`. A duplicate edge sums its weight, unions its `provenance`, + * and merges its tags (base wins on conflict — the base edge is authoritative for the shared key). + * External symbols union by signature, base winning on conflict. */ export function mergeCallGraphs(a: CallGraphResult, b: CallGraphResult): CallGraphResult { const byKey = new Map(); @@ -67,54 +61,3 @@ export function mergeCallGraphs(a: CallGraphResult, b: CallGraphResult): CallGra const synthesized_callables = { ...b.synthesized_callables, ...a.synthesized_callables }; return { edges: [...byKey.values()], external_symbols, synthesized_callables }; } - -/** Count how the two edge sets overlap — preserves the old `both`-mode diagnostic. */ -function diffSummary(tsc: CallGraphResult, jelly: CallGraphResult): string { - const key = (e: { source: string; target: string }): string => `${e.source} ${e.target}`; - const tscKeys = new Set(tsc.edges.map(key)); - const jellyKeys = new Set(jelly.edges.map(key)); - let shared = 0; - for (const k of jellyKeys) if (tscKeys.has(k)) shared++; - return ( - `${shared} shared, ${tscKeys.size - shared} tsc-only, ${jellyKeys.size - shared} jelly-only ` + - `(tsc=${tscKeys.size}, jelly=${jellyKeys.size})` - ); -} - -/** - * Run tsc + jelly and emit their union. This is the default: jelly's edges and external symbols are - * PERSISTED (tagged `provenance: ["jelly"]`) instead of being discarded after a diff. If jelly - * fails, degrade to tsc only rather than failing the whole analysis. - */ -export const unionProvider: CallGraphProvider = { - name: "union", - build(ctx) { - const tsc = tscProvider.build(ctx); - let jelly: CallGraphResult; - try { - jelly = jellyProvider.build(ctx); - } catch (e) { - ctx.log.info(`call graph (union): jelly failed (${(e as Error).message}); emitting tsc only`); - return tsc; - } - ctx.log.info(`call graph diff: ${diffSummary(tsc, jelly)}`); - const merged = mergeCallGraphs(tsc, jelly); - ctx.log.info( - `call graph (union): ${merged.edges.length} edges, ` + - `${Object.keys(merged.external_symbols).length} external symbols`, - ); - return merged; - }, -}; - -export function selectProvider(name: string): CallGraphProvider { - switch (name) { - case "tsc": - return tscProvider; - case "jelly": - return jellyProvider; - default: - // "union" (the default) and the deprecated "both" alias both land here. - return unionProvider; - } -} diff --git a/src/syntactic_analysis/builders.ts b/src/syntactic_analysis/builders.ts index e87bdd0..3bfd5c2 100644 --- a/src/syntactic_analysis/builders.ts +++ b/src/syntactic_analysis/builders.ts @@ -294,7 +294,10 @@ function buildAttributeField(prop: Node): TSField { function buildCallsite(call: Node): TSCallsite { const isNew = Node.isNewExpression(call); - const expr = (call as unknown as { getExpression: () => Node }).getExpression(); + // A tagged template (`inline\`url(...)\``) is a call whose callee is the tag. + const expr = Node.isTaggedTemplateExpression(call) + ? call.getTag() + : (call as unknown as { getExpression: () => Node }).getExpression(); let method_name = expr.getText(); let receiver_expr: string | undefined; let receiver_type: string | undefined; @@ -305,7 +308,7 @@ function buildCallsite(call: Node): TSCallsite { receiver_type = inferredType(expr.getExpression()); is_optional_chain = boolOf(expr, "hasQuestionDotToken"); } - const args = (call as unknown as { getArguments: () => Node[] }).getArguments(); + const args = (call as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; // tagged templates have none const argument_types = args.map((a) => inferredType(a) ?? "unknown"); const typeArgs = (call as unknown as { getTypeArguments?: () => Node[] }).getTypeArguments?.() ?? []; const type_arguments = typeArgs.map((t) => t.getText()); @@ -362,16 +365,13 @@ function walkBody(body: Node, h: BodyHandlers): void { return; } if (b === "skip") return; - if (Node.isCallExpression(node) || Node.isNewExpression(node)) h.onCall(node); + if (Node.isCallExpression(node) || Node.isNewExpression(node) || Node.isTaggedTemplateExpression(node)) h.onCall(node); node.forEachChild(visit); }; - // A concise arrow body can *be* a callable (`() => () => x`). Visiting only the body's children - // would skip it and attribute its call sites to the callable that merely returns it. - if (namedBoundary(body) !== null) { - visit(body); - return; - } - body.forEachChild(visit); + // Visit the body NODE itself, not only its children: a concise arrow body can *be* a callable + // (`() => () => x`) — the boundary handler claims it — or *be* the call (`u => u.describe()`), + // which a children-only walk would silently skip (the call-site gap Jelly used to paper over). + visit(body); } function computeCC(body: Node): number { @@ -457,19 +457,23 @@ export function buildCallable( const callables: Record = {}; const types: Record = {}; + const handlers = { + onCall: (n: Node) => call_sites.push(buildCallsite(n)), + onNestedCallable: (n: Node) => { + const r = buildNestedCallable(n, root); + if (r) callables[memberKey(r.sig, r.callable.accessor_kind)] = r.callable; + }, + onNestedClass: (n: Node) => { + const r = buildClass(n, root); + types[memberKey(r.sig)] = r.cls; + }, + }; const body = (fnNode as unknown as { getBody?: () => Node | undefined }).getBody?.(); - if (body) { - walkBody(body, { - onCall: (n) => call_sites.push(buildCallsite(n)), - onNestedCallable: (n) => { - const r = buildNestedCallable(n, root); - if (r) callables[memberKey(r.sig, r.callable.accessor_kind)] = r.callable; - }, - onNestedClass: (n) => { - const r = buildClass(n, root); - types[memberKey(r.sig)] = r.cls; - }, - }); + if (body) walkBody(body, handlers); + // Parameter DEFAULT initializers execute in the callee's own activation (`f(x = mk())`), so + // their calls (and nested arrows) belong to this callable — they live outside getBody(). + for (const p of (fnNode as unknown as { getParameters?: () => Node[] }).getParameters?.() ?? []) { + walkBody(p, handlers); } const nameNode = sigNode as unknown as { getName?: () => string | undefined }; @@ -645,6 +649,29 @@ export function buildClass(cls: Node, root: string): { sig: string; cls: TSType } } + // Instance property INITIALIZERS execute in the constructor (`private readonly registry = + // Registry.as(...)`): their call sites belong to the ctor (explicit or the synthesized + // implicit one), and an initializer ARROW is a class-scoped positional anon callable — + // `contributorName` signs it `Class.`, so it lives in the class's callables{}, + // keeping signature ↔ containment aligned. Static initializers run at class-definition time + // and stay with the module-scope sweep (callGraph.ts). + { + const ctorCallable = callables[memberKey(constructorSignatureOf(sig))]; + for (const p of c.getProperties()) { + if (boolOf(p, "isStatic")) continue; + const init = (p as unknown as { getInitializer?: () => Node | undefined }).getInitializer?.(); + if (!init || !ctorCallable) continue; + walkBody(init, { + onCall: (n) => ctorCallable.call_sites.push(buildCallsite(n)), + onNestedCallable: (n) => { + const r = buildNestedCallable(n, root); + if (r) callables[memberKey(r.sig, r.callable.accessor_kind)] = r.callable; + }, + onNestedClass: () => {}, // class expression inside a property initializer — out of scope + }); + } + } + const base_classes: string[] = []; const implements_types: string[] = []; const ext = c.getExtends?.(); diff --git a/src/syntactic_analysis/discovery.ts b/src/syntactic_analysis/discovery.ts index 60c31b4..c0abae5 100644 --- a/src/syntactic_analysis/discovery.ts +++ b/src/syntactic_analysis/discovery.ts @@ -3,6 +3,13 @@ import * as path from "node:path"; import { relPosix } from "../utils"; const SOURCE_EXTS = new Set([".ts", ".tsx", ".mts", ".cts"]); +// JS sources are first-class too (the analyzer is a TS/JS analyzer; vendored .js like vscode's +// marked.js was invisible, #98). Sibling rules keep one module per prefix: +// - a .js beside a same-prefix REAL .ts source is compiled output → the .js is skipped +// (the compiler's own allowJs duplicate rule); +// - a .d.ts beside a same-prefix .js is hand-written declarations FOR that source → the .d.ts +// is skipped as a module (the checker still reads it from disk for importers' types). +const JS_EXTS = new Set([".js", ".jsx", ".mjs", ".cjs"]); export const SKIP_DIRS = new Set([ "node_modules", @@ -23,7 +30,7 @@ const TEST_DIRS = new Set(["__tests__", "__test__", "test", "tests", "spec", "__ /** Test-ness is judged on the path RELATIVE TO the project root, never the absolute path. */ function isTestFile(relKey: string): boolean { const base = path.basename(relKey); - if (/\.(test|spec)\.(ts|tsx|mts|cts)$/.test(base)) return true; + if (/\.(test|spec)\.(ts|tsx|mts|cts|js|jsx|mjs|cjs)$/.test(base)) return true; return relKey.split("/").some((p) => TEST_DIRS.has(p)); } @@ -32,9 +39,11 @@ export interface DiscoveredFile { fileKey: string; // project-relative POSIX path with extension } -/** Recursively discover .ts/.tsx sources under root, skipping vendored and (optionally) test trees. */ +/** Recursively discover TS/JS sources under root, skipping vendored and (optionally) test trees. */ export function discoverSourceFiles(root: string, skipTests: boolean): DiscoveredFile[] { - const out: DiscoveredFile[] = []; + const tsFiles: DiscoveredFile[] = []; + const jsCandidates: DiscoveredFile[] = []; + const realTsPrefixes = new Set(); // non-.d.ts TS sources only const walk = (dir: string): void => { let entries: fs.Dirent[]; try { @@ -50,14 +59,33 @@ export function discoverSourceFiles(root: string, skipTests: boolean): Discovere walk(abs); } else if (e.isFile()) { const ext = path.extname(e.name); - if (!SOURCE_EXTS.has(ext)) continue; + const isTs = SOURCE_EXTS.has(ext); + const isJs = JS_EXTS.has(ext); + if (!isTs && !isJs) continue; const fileKey = relPosix(root, abs); if (skipTests && isTestFile(fileKey)) continue; - out.push({ absPath: abs, fileKey }); + if (isTs) { + if (!fileKey.endsWith(".d.ts")) realTsPrefixes.add(fileKey.replace(/\.(tsx|ts|mts|cts)$/, "")); + tsFiles.push({ absPath: abs, fileKey }); + } else { + jsCandidates.push({ absPath: abs, fileKey }); + } } } }; walk(root); + const out: DiscoveredFile[] = []; + const jsPrefixes = new Set(); + for (const j of jsCandidates) { + const prefix = j.fileKey.replace(/\.(jsx|js|mjs|cjs)$/, ""); + if (realTsPrefixes.has(prefix)) continue; // compiled sibling of a TS source → skip + jsPrefixes.add(prefix); + out.push(j); + } + for (const t of tsFiles) { + if (t.fileKey.endsWith(".d.ts") && jsPrefixes.has(t.fileKey.replace(/\.d\.ts$/, ""))) continue; // decls FOR an analyzed .js + out.push(t); + } out.sort((a, b) => a.fileKey.localeCompare(b.fileKey)); return out; } diff --git a/test/anonymous-callables.test.ts b/test/anonymous-callables.test.ts index 076416e..20b2ca4 100644 --- a/test/anonymous-callables.test.ts +++ b/test/anonymous-callables.test.ts @@ -38,7 +38,6 @@ function options(level: number): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, } as unknown as AnalysisOptions; diff --git a/test/artifacts.test.ts b/test/artifacts.test.ts new file mode 100644 index 0000000..1eceff3 --- /dev/null +++ b/test/artifacts.test.ts @@ -0,0 +1,160 @@ +/** + * Repository-artifact layer gates (#101, recalibrated to python PR #160 / the ratified + * 2026-08-27 spec): neutral artifact ids, rules-matched capture, flat evidence-tagged + * dependencies (npm kinds incl. the coined `peer`), lock backfill with `lockfile` prov, + * unresolved imports with the @types type-only rule, level-invariance, determinism, and the + * neutral :Artifact/:Package Neo4j projection. + */ +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../src/core"; +import { project } from "../src/build/neo4j"; +import type { AnalysisOptions } from "../src/options"; + +const FIXTURE = path.resolve(import.meta.dir, "fixtures/artifacts-app"); + +function options(over: Partial = {}): AnalysisOptions { + return { + input: FIXTURE, output: null, emit: "json", appName: "artifacts-app", neo4jUri: null, + neo4jUser: "neo4j", neo4jPassword: "", neo4jDatabase: null, analysisLevel: 1, graphs: [], + graphFieldDepth: 3, jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, + phantoms: true, cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "cants-art-")), verbosity: 0, + ...over, + } as AnalysisOptions; +} + +const r1 = await analyze(options()); +const root = r1.application.application; +const arts = root.artifacts; +const deps = root.dependencies; +const byName = new Map(deps.map((d) => [d.name, d])); + +describe("artifact inventory — rules-matched, neutral ids (#101/PR-160)", () => { + test("rules-matched files are inventoried; unmatched and source files are not", () => { + for (const key of [ + "package.json", "package-lock.json", "packages/web/package.json", "packages/web/bun.lock", + "yarn.lock", ".env", "tsconfig.json", "Dockerfile", ".github/workflows/ci.yml", "README.md", "LICENSE", + ]) { + expect(arts[key], key).toBeDefined(); + } + expect(Object.keys(arts).some((k) => k.endsWith(".ts"))).toBe(false); + }); + + test("ids are LANGUAGE-NEUTRAL (can://artifact//); dotfiles keep the dot", () => { + expect(arts[".env"]?.id).toBe("can://artifact/artifacts-app/.env"); + expect(arts["packages/web/package.json"]?.id).toBe("can://artifact/artifacts-app/packages/web/package.json"); + }); + + test("roles and formats from the rules table; roles union across matches", () => { + expect(arts["package.json"]?.roles).toEqual(["dependency-manifest", "tool-config"]); + expect(arts["package-lock.json"]?.roles).toEqual(["dependency-manifest"]); + expect(arts[".env"]?.roles).toEqual(["env"]); + expect(arts["tsconfig.json"]?.roles).toEqual(["tool-config"]); + expect(arts["Dockerfile"]?.roles).toEqual(["container-image"]); + expect(arts[".github/workflows/ci.yml"]?.roles).toEqual(["ci"]); + expect(arts["README.md"]?.roles).toEqual(["docs"]); + expect(arts["LICENSE"]?.roles).toEqual(["legal"]); + expect(arts["packages/web/bun.lock"]?.format).toBe("jsonc"); + }); + + test("verbatim source + sha256 + extraction status", () => { + expect(arts["package.json"]?.source).toContain('"express"'); + expect(arts["package.json"]?.sha256?.length).toBe(64); + expect(arts["package.json"]?.extraction).toBe("full"); + expect(arts["yarn.lock"]?.extraction).toBe("none"); // inventory-only lock format + expect(arts["README.md"]?.extraction).toBe("none"); + }); +}); + +describe("dependencies — flat, evidence-tagged (#101/PR-160)", () => { + test("npm sections map to the shared kind vocabulary, `peer` included; prov declared", () => { + expect(byName.get("express")?.kind).toBe("runtime"); + expect(byName.get("typescript")?.kind).toBe("dev"); + expect(byName.get("fsevents")?.kind).toBe("optional"); + expect(byName.get("react")?.kind).toBe("peer"); + for (const d of deps) { + expect(d.prov).toContain("declared"); + expect(d.extras).toEqual([]); + } + }); + + test("declared_in is the manifest's neutral artifact id (workspace member keeps its own)", () => { + expect(byName.get("express")?.declared_in).toBe("can://artifact/artifacts-app/package.json"); + expect(byName.get("lodash")?.declared_in).toBe("can://artifact/artifacts-app/packages/web/package.json"); + }); + + test("locks backfill locked_version on declared records only, prov gains lockfile", () => { + expect(byName.get("express")?.locked_version).toBe("4.19.2"); + expect(byName.get("express")?.prov).toEqual(["declared", "lockfile"]); + expect(byName.get("lodash")?.locked_version).toBe("4.17.21"); // sibling bun.lock (JSONC) + expect(byName.get("react")?.locked_version).toBeUndefined(); + expect(byName.has("lockonly-transitive")).toBe(false); // locks never create records + expect(byName.has("transitive-shadow")).toBe(false); // nested lock entries ignored + }); + + test("provides_imports: the name itself; @types/x also provides x", () => { + expect(byName.get("express")?.provides_imports).toEqual(["express"]); + expect(byName.get("@types/typed-only-pkg")?.provides_imports).toEqual(["@types/typed-only-pkg", "typed-only-pkg"]); + }); +}); + +describe("unresolved imports — the hygiene signal (#101/PR-160)", () => { + test("an undeclared VALUE import surfaces; declared and type-only-via-@types do not", () => { + const mods = root.unresolved_imports.map((u) => u.module); + expect(mods).toContain("left-pad"); // imported, never declared + expect(mods).not.toContain("express"); // declared runtime + expect(mods).not.toContain("typed-only-pkg"); // import type + @types declared → satisfied + expect(mods).not.toContain("node:fs"); // builtin + }); + + test("--resolve-installed binds via node_modules metadata (prov installed-metadata)", async () => { + const r = await analyze(options({ resolveInstalled: true })); + const u = r.application.application.unresolved_imports.find((x) => x.module === "left-pad"); + expect(u?.bound_to).toBe("left-pad"); + expect(u?.prov).toEqual(["installed-metadata"]); + }); +}); + +describe("level-invariance + determinism (#101)", () => { + test("the three sections are identical at -a 1 and -a 4", async () => { + const r4 = await analyze(options({ analysisLevel: 4, graphs: ["cfg", "dfg", "pdg", "sdg"] })); + expect(r4.application.application.artifacts).toEqual(arts); + expect(r4.application.application.dependencies).toEqual(deps); + expect(r4.application.application.unresolved_imports).toEqual(root.unresolved_imports); + }); + + test("two consecutive default runs are byte-identical", async () => { + const a = JSON.stringify((await analyze(options())).application); + const b = JSON.stringify((await analyze(options())).application); + expect(a).toBe(b); + }); +}); + +describe("Neo4j projection — neutral :Artifact/:Package (#101, contract 2.2.0)", () => { + const rows = project(r1.application); + + test("neutral nodes with purl ids; TS-prefixed claims into the ghost space", () => { + const art = rows.nodes.find((n) => n.value === "can://artifact/artifacts-app/package.json"); + expect(art?.labels).toEqual(["Artifact"]); + expect(art?.props["roles"]).toEqual(["dependency-manifest", "tool-config"]); + expect(art?.props["source"]).toBeUndefined(); // text stays off the graph + const pkg = rows.nodes.find((n) => n.value === "pkg:npm/react"); + expect(pkg?.labels).toEqual(["Package"]); + const scoped = rows.nodes.find((n) => n.value === "pkg:npm/%40scope/util"); + expect(scoped, "scoped purl").toBeDefined(); + expect(rows.edges.some((e) => e.type === "HAS_ARTIFACT" && e.to.value === art?.value)).toBe(true); + const decl = rows.edges.find((e) => e.type === "DECLARES_DEPENDENCY" && e.to.value === "pkg:npm/react"); + expect(decl?.props["kind"]).toBe("peer"); + expect(rows.edges.some((e) => e.type === "LOCKS" && e.to.value === "pkg:npm/express")).toBe(true); + expect( + rows.edges.some( + (e) => e.type === "TS_PROVIDES" && e.from.value === "pkg:npm/express" && String(e.to.value).endsWith("/@external/express"), + ), + ).toBe(true); + expect( + rows.edges.some((e) => e.type === "TS_UNRESOLVED_IMPORT" && String(e.to.value).endsWith("/@external/left-pad")), + ).toBe(true); + }); +}); diff --git a/test/dataflow.test.ts b/test/dataflow.test.ts index a539472..7e5a686 100644 --- a/test/dataflow.test.ts +++ b/test/dataflow.test.ts @@ -34,7 +34,6 @@ function options(level: 1 | 2 | 3, cacheDir: string, jobs: number): AnalysisOpti eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir, verbosity: 0, }; diff --git a/test/external-resolution.test.ts b/test/external-resolution.test.ts index 86a957e..034d942 100644 --- a/test/external-resolution.test.ts +++ b/test/external-resolution.test.ts @@ -39,7 +39,6 @@ function options(): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, }; diff --git a/test/fixtures/artifacts-app/.env b/test/fixtures/artifacts-app/.env new file mode 100644 index 0000000..889d18f --- /dev/null +++ b/test/fixtures/artifacts-app/.env @@ -0,0 +1,4 @@ +# comment +PAYMENT_HOST=https://pay.example.com +DB_URL="postgres://u:p@${PAYMENT_HOST}/db" +export NODE_OPTIONS='--max-old-space-size=4096' diff --git a/test/fixtures/artifacts-app/.github/workflows/ci.yml b/test/fixtures/artifacts-app/.github/workflows/ci.yml new file mode 100644 index 0000000..4a79684 --- /dev/null +++ b/test/fixtures/artifacts-app/.github/workflows/ci.yml @@ -0,0 +1,2 @@ +name: ci +on: push diff --git a/test/fixtures/artifacts-app/Dockerfile b/test/fixtures/artifacts-app/Dockerfile new file mode 100644 index 0000000..d00d40f --- /dev/null +++ b/test/fixtures/artifacts-app/Dockerfile @@ -0,0 +1,2 @@ +FROM node:22 +COPY . . diff --git a/test/fixtures/artifacts-app/LICENSE b/test/fixtures/artifacts-app/LICENSE new file mode 100644 index 0000000..d1e1072 --- /dev/null +++ b/test/fixtures/artifacts-app/LICENSE @@ -0,0 +1 @@ +MIT License diff --git a/test/fixtures/artifacts-app/README.md b/test/fixtures/artifacts-app/README.md new file mode 100644 index 0000000..446f4a8 --- /dev/null +++ b/test/fixtures/artifacts-app/README.md @@ -0,0 +1 @@ +# artifacts-app diff --git a/test/fixtures/artifacts-app/package-lock.json b/test/fixtures/artifacts-app/package-lock.json new file mode 100644 index 0000000..e2607b1 --- /dev/null +++ b/test/fixtures/artifacts-app/package-lock.json @@ -0,0 +1,12 @@ +{ + "name": "artifacts-app", + "lockfileVersion": 3, + "packages": { + "": { "name": "artifacts-app" }, + "node_modules/express": { "version": "4.19.2" }, + "node_modules/@scope/util": { "version": "2.1.5" }, + "node_modules/typescript": { "version": "5.5.4" }, + "node_modules/express/node_modules/transitive-shadow": { "version": "9.9.9" }, + "node_modules/lockonly-transitive": { "version": "1.0.0" } + } +} diff --git a/test/fixtures/artifacts-app/package.json b/test/fixtures/artifacts-app/package.json new file mode 100644 index 0000000..d161144 --- /dev/null +++ b/test/fixtures/artifacts-app/package.json @@ -0,0 +1,9 @@ +{ + "name": "artifacts-app", + "version": "1.0.0", + "workspaces": ["packages/*"], + "dependencies": { "express": "^4.19.0", "@scope/util": "~2.1.0" }, + "devDependencies": { "typescript": "^5.5.0", "@types/typed-only-pkg": "^1.0.0" }, + "optionalDependencies": { "fsevents": "^2.3.3" }, + "peerDependencies": { "react": ">=18" } +} diff --git a/test/fixtures/artifacts-app/packages/web/package.json b/test/fixtures/artifacts-app/packages/web/package.json new file mode 100644 index 0000000..41162d3 --- /dev/null +++ b/test/fixtures/artifacts-app/packages/web/package.json @@ -0,0 +1,4 @@ +{ + "name": "@artifacts-app/web", + "dependencies": { "lodash": "^4.17.21" } +} diff --git a/test/fixtures/artifacts-app/src/index.ts b/test/fixtures/artifacts-app/src/index.ts new file mode 100644 index 0000000..1e02624 --- /dev/null +++ b/test/fixtures/artifacts-app/src/index.ts @@ -0,0 +1,5 @@ +import express from "express"; +import leftPad from "left-pad"; +import type { SomeType } from "typed-only-pkg"; +import * as fs from "node:fs"; +export function main(): void { void express; void leftPad; void fs; const t: SomeType | null = null; void t; } diff --git a/test/fixtures/artifacts-app/tsconfig.json b/test/fixtures/artifacts-app/tsconfig.json new file mode 100644 index 0000000..38c229e --- /dev/null +++ b/test/fixtures/artifacts-app/tsconfig.json @@ -0,0 +1 @@ +{ "compilerOptions": { "strict": true, "target": "ES2022" }, "include": ["src"] } diff --git a/test/fixtures/artifacts-app/yarn.lock b/test/fixtures/artifacts-app/yarn.lock new file mode 100644 index 0000000..db6bda8 --- /dev/null +++ b/test/fixtures/artifacts-app/yarn.lock @@ -0,0 +1,3 @@ +yarn lockfile v1 +lodash@^4.17.21: + version "4.17.21" diff --git a/test/multi-tsconfig.test.ts b/test/multi-tsconfig.test.ts index 7084748..2d630f9 100644 --- a/test/multi-tsconfig.test.ts +++ b/test/multi-tsconfig.test.ts @@ -42,7 +42,6 @@ function options(): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, }; diff --git a/test/neo4j-bolt.test.ts b/test/neo4j-bolt.test.ts index 684f7c3..9015a52 100644 --- a/test/neo4j-bolt.test.ts +++ b/test/neo4j-bolt.test.ts @@ -48,7 +48,6 @@ function optsFor(overrides: Partial = {}): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: path.join(TMP, "cache"), verbosity: 0, ...overrides, diff --git a/test/neo4j-schema.test.ts b/test/neo4j-schema.test.ts index 02417b7..1fdc62f 100644 --- a/test/neo4j-schema.test.ts +++ b/test/neo4j-schema.test.ts @@ -30,7 +30,7 @@ async function fixtureRows() { neo4jUri: null, neo4jUser: "neo4j", neo4jPassword: "", neo4jDatabase: null, analysisLevel: 4, graphs: ["cfg", "dfg", "pdg", "sdg"], graphFieldDepth: 3, jobs: 1, targetFiles: null, skipTests: true, eager: true, - noBuild: true, phantoms: true, callGraphProvider: "union", cacheDir, verbosity: 0, + noBuild: true, phantoms: true, cacheDir, verbosity: 0, }; try { return project((await analyze(opts)).application); @@ -100,14 +100,21 @@ describe("neo4j schema conformance", () => { } }); - test("2.0.0 emits only TS-prefixed specific labels and TS_ rel types (#66)", () => { + test("TS-prefixed labels/rels, with the artifact layer's sanctioned NEUTRAL exception (#66, #101)", () => { + // :Artifact/:Package (+ HAS_ARTIFACT/DECLARES_DEPENDENCY/LOCKS) are deliberately + // language-neutral so sibling analyzers MERGE onto the same nodes (python PR #160's rule); + // edges that stay this analyzer's own claim (TS_PROVIDES, TS_UNRESOLVED_IMPORT) keep TS_. + const NEUTRAL_LABELS = new Set(["Artifact", "Package"]); + const NEUTRAL_RELS = new Set(["HAS_ARTIFACT", "DECLARES_DEPENDENCY", "LOCKS"]); for (const node of rows.nodes) { for (const l of node.labels) { - const ok = l === "CanNode" || l === "Application" || l.startsWith("TS"); + const ok = l === "CanNode" || l === "Application" || l.startsWith("TS") || NEUTRAL_LABELS.has(l); expect(ok, `bare label leaked: ${l}`).toBe(true); } } - for (const edge of rows.edges) expect(edge.type.startsWith("TS_"), `bare rel leaked: ${edge.type}`).toBe(true); + for (const edge of rows.edges) { + expect(edge.type.startsWith("TS_") || NEUTRAL_RELS.has(edge.type), `bare rel leaked: ${edge.type}`).toBe(true); + } }); }); diff --git a/test/schema-v2.test.ts b/test/schema-v2.test.ts index eda00ce..eae4bbd 100644 --- a/test/schema-v2.test.ts +++ b/test/schema-v2.test.ts @@ -38,7 +38,6 @@ function options(): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, }; @@ -88,7 +87,7 @@ describe("schema v2 — L1 envelope", () => { expect(v2.schema_version).toBe("2.1.0"); expect(v2.language).toBe("typescript"); expect(v2.max_level).toBe(1); - expect(Object.keys(root).sort()).toEqual(["call_graph", "id", "kind", "param_in", "param_out", "symbol_table"]); + expect(Object.keys(root).sort()).toEqual(["artifacts", "call_graph", "dependencies", "id", "kind", "param_in", "param_out", "symbol_table", "unresolved_imports"]); expect(root.id).toBe("can://typescript/sample-app"); expect(root.kind).toBe("application"); }); @@ -246,7 +245,7 @@ describe("schema v2 — L1 skips the call-graph solve (issue #31)", () => { const spy = spyOn(tscProvider, "build"); const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-v2-l1-guard-")); try { - const v1L1 = (await analyze({ ...options(), analysisLevel: 1, callGraphProvider: "tsc", cacheDir })).internal; + const v1L1 = (await analyze({ ...options(), analysisLevel: 1, cacheDir })).internal; expect(spy).not.toHaveBeenCalled(); expect(v1L1.call_graph).toEqual([]); expect(Object.keys(v1L1.external_symbols)).toEqual([]); @@ -261,7 +260,7 @@ describe("schema v2 — L1 skips the call-graph solve (issue #31)", () => { const spy = spyOn(tscProvider, "build"); const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-v2-l1-guard-l2-")); try { - const v1L2guard = (await analyze({ ...options(), analysisLevel: 2, callGraphProvider: "tsc", cacheDir })).internal; + const v1L2guard = (await analyze({ ...options(), analysisLevel: 2, cacheDir })).internal; expect(spy).toHaveBeenCalledTimes(1); expect(v1L2guard.call_graph.length).toBeGreaterThan(0); } finally { @@ -701,6 +700,15 @@ function canNodeIds(app: TSAnalysis): Set { } for (const id of Object.keys(app.application.external_symbols ?? {})) ids.add(id); for (const id of Object.keys(app.application.synthesized_callables ?? {})) ids.add(id); + // Repository-artifact layer (#101/PR-160 shape): artifacts/packages are NOT CanNodes (own + // neutral merge labels) — but TS_PROVIDES / TS_UNRESOLVED_IMPORT mint module-level + // :TSExternal ghosts in the CanNode id space. + for (const d of app.application.dependencies ?? []) { + for (const top of d.provides_imports) ids.add(`${app.application.id}/@external/${top}`); + } + for (const u of app.application.unresolved_imports ?? []) { + ids.add(`${app.application.id}/@external/${u.module}`); + } return ids; } @@ -714,8 +722,10 @@ function resolvesToCount(app: TSAnalysis): number { } describe("neo4j ↔ json count parity — full depth (issue #27)", () => { - test("node count: 1 :Application row + every :CanNode id", () => { - expect(monoRows.nodes.length).toBe(1 + canNodeIds(monoApp4).size); + test("node count: 1 :Application row + every :CanNode id + neutral Artifact/Package rows", () => { + const artifactCount = Object.keys(monoApp4.application.artifacts ?? {}).length; + const packageCount = new Set((monoApp4.application.dependencies ?? []).map((d) => d.name)).size; + expect(monoRows.nodes.length).toBe(1 + canNodeIds(monoApp4).size + artifactCount + packageCount); }); test("typed overlay relationships match their JSON edge-list length 1:1", () => { @@ -742,7 +752,11 @@ describe("neo4j ↔ json count parity — full depth (issue #27)", () => { const containmentEdges = containment.reduce((n, t) => n + relCount(monoRows, t), 0); const externalCount = Object.keys(monoApp4.application.external_symbols ?? {}).length; const synthCount = Object.keys(monoApp4.application.synthesized_callables ?? {}).length; - expect(containmentEdges).toBe(canNodeIds(monoApp4).size - externalCount - synthCount); + // minted provides/unresolved ghosts are off-tree CanNodes too (like externals) + const ghostIds = new Set(); + for (const d of monoApp4.application.dependencies ?? []) for (const t of d.provides_imports) ghostIds.add(t); + for (const u of monoApp4.application.unresolved_imports ?? []) ghostIds.add(u.module); + expect(containmentEdges).toBe(canNodeIds(monoApp4).size - externalCount - synthCount - ghostIds.size); }); test("EXTENDS/IMPLEMENTS have no JSON edge-list (extends_ids/implements_ids node props are the source of truth); counts still match 1:1", () => { @@ -774,10 +788,14 @@ describe("neo4j ↔ json count parity — full depth (issue #27)", () => { (n, t) => n + relCount(monoRows, t), 0, ); + const artifactLayer = ["HAS_ARTIFACT", "DECLARES_DEPENDENCY", "LOCKS", "TS_PROVIDES", "TS_UNRESOLVED_IMPORT"].reduce( + (n, t) => n + relCount(monoRows, t), + 0, + ); const resolvesTo = relCount(monoRows, "TS_RESOLVES_TO"); const heritage = relCount(monoRows, "TS_EXTENDS") + relCount(monoRows, "TS_IMPLEMENTS"); expect(resolvesTo).toBe(resolvesToCount(monoApp4)); - expect(typedOverlay + containment + resolvesTo + heritage).toBe(monoRows.edges.length); + expect(typedOverlay + containment + artifactLayer + resolvesTo + heritage).toBe(monoRows.edges.length); }); test("DDG/CFG_NEXT parity survives the writers: every row keyed, keys fully discriminate (issue #70)", () => { diff --git a/test/synthesized-nodes.test.ts b/test/synthesized-nodes.test.ts index 018cb88..a30e410 100644 --- a/test/synthesized-nodes.test.ts +++ b/test/synthesized-nodes.test.ts @@ -23,7 +23,7 @@ const app: AnalysisInternal = { classes: {}, interfaces: {}, enums: {}, type_aliases: {}, namespaces: {}, variables: [], } as unknown as TSModule, }, - call_graph: [{ source: "src/x.foo", target: ANON, type: CALL_DEP, weight: 1, provenance: ["jelly"], tags: {} }], + call_graph: [{ source: "src/x.foo", target: ANON, type: CALL_DEP, weight: 1, provenance: ["defuse"], tags: {} }], external_symbols: {}, synthesized_callables: { [ANON]: { name: "", path: "src/x.ts", start_line: 3, start_column: 10 } }, }; diff --git a/test/tagged-templates.test.ts b/test/tagged-templates.test.ts new file mode 100644 index 0000000..0ea2452 --- /dev/null +++ b/test/tagged-templates.test.ts @@ -0,0 +1,56 @@ +/** + * Tagged template expressions are call sites (#98): `inline\`url(...)\`` must record a `call` + * body node and resolve a call edge to the tag — found missing by the vscode Joern ledger + * (cssValue.ts's `inline` idiom), then crash-guarded (tagged templates have no arguments list). + */ +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../src/core"; +import type { AnalysisOptions } from "../src/options"; + +const dir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-tagged-")); +fs.mkdirSync(path.join(dir, "src")); +fs.writeFileSync( + path.join(dir, "src", "x.ts"), + [ + "export function inline(strings: TemplateStringsArray, ...v: string[]): string { return ''; }", + "export function asCSSUrl(): string { return inline`url('x')`; }", + "export const top = inline`module-scope`;", + "declare const unknownTag: any;", + "export function throughLinker(): void { unknownTag`unresolved-tag`; }", + "export function mkSheet(): number { return 1; }", + "export function createRule(sel: string, sheet = mkSheet()): number { return sheet; }", + ].join("\n"), +); + +const opts = { + input: dir, output: null, emit: "json", appName: "tagged", neo4jUri: null, neo4jUser: "neo4j", + neo4jPassword: "", neo4jDatabase: null, analysisLevel: 2, graphs: [], graphFieldDepth: 3, + jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, phantoms: true, + cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "cants-tagged-cache-")), verbosity: 0, +} as AnalysisOptions; +const result = await analyze(opts); +fs.rmSync(dir, { recursive: true, force: true }); + +describe("tagged template calls (#98)", () => { + test("a tagged template resolves a call edge to its tag", () => { + expect(result.internal.call_graph.some((e) => e.source === "src/x.asCSSUrl" && e.target === "src/x.inline")).toBe(true); + }); + + test("a module-scope tagged template is attributed to the module", () => { + expect(result.internal.call_graph.some((e) => e.source === "src/x" && e.target === "src/x.inline")).toBe(true); + }); + + test("a parameter-default initializer call is attributed to the callable (#98)", () => { + expect(result.internal.call_graph.some((e) => e.source === "src/x.createRule" && e.target === "src/x.mkSheet")).toBe(true); + }); + + test("the tagged call is a body call node with a refined callee", () => { + const fn = result.application.application.symbol_table["src/x.ts"]?.functions["asCSSUrl"]; + const calls = Object.values(fn?.body ?? {}).filter((b) => b.kind === "call"); + expect(calls.length).toBe(1); + expect(calls[0]?.callee).toBe("can://typescript/tagged/src/x.ts/inline"); + }); +}); diff --git a/test/union-provider.test.ts b/test/union-provider.test.ts deleted file mode 100644 index d9f0cc9..0000000 --- a/test/union-provider.test.ts +++ /dev/null @@ -1,73 +0,0 @@ -/** - * Unit tests for the union merge (issue #11): tsc + jelly edges and external symbols must be - * combined — not discarded — with provenance preserved so consumers can still tell them apart. - */ -import { describe, expect, test } from "bun:test"; -import type { CallGraphResult } from "../src/semantic_analysis"; -import { mergeCallGraphs } from "../src/semantic_analysis"; -import { CALL_DEP, type TSCallEdge } from "../src/schema"; - -const edge = (source: string, target: string, provenance: string[], extra: Partial = {}): TSCallEdge => ({ - source, - target, - type: CALL_DEP, - weight: 1, - provenance, - tags: {}, - ...extra, -}); - -const result = ( - edges: TSCallEdge[], - external: CallGraphResult["external_symbols"] = {}, - synthesized: CallGraphResult["synthesized_callables"] = {}, -): CallGraphResult => ({ - edges, - external_symbols: external, - synthesized_callables: synthesized, -}); - -describe("mergeCallGraphs", () => { - test("keeps jelly-only edges (the bug: they used to be dropped)", () => { - const tsc = result([edge("a", "b", ["tsc"])]); - const jelly = result([edge("c", "d", ["jelly"])]); - const merged = mergeCallGraphs(tsc, jelly); - const keys = merged.edges.map((e) => `${e.source}->${e.target}`).sort(); - expect(keys).toEqual(["a->b", "c->d"]); - }); - - test("an edge found by both carries both provenances and summed weight", () => { - const tsc = result([edge("a", "b", ["tsc"], { weight: 2 })]); - const jelly = result([edge("a", "b", ["jelly"], { weight: 3 })]); - const merged = mergeCallGraphs(tsc, jelly); - expect(merged.edges).toHaveLength(1); - expect(merged.edges[0].provenance.sort()).toEqual(["jelly", "tsc"]); - expect(merged.edges[0].weight).toBe(5); - }); - - test("merges external symbols from both, tsc winning on conflict", () => { - const tsc = result([], { "pkg.foo": { name: "foo", module: "pkg" } }); - const jelly = result([], { - "pkg.foo": { name: "FOO-jelly", module: "pkg" }, - "pkg.bar": { name: "bar", module: "pkg" }, - }); - const merged = mergeCallGraphs(tsc, jelly); - expect(Object.keys(merged.external_symbols).sort()).toEqual(["pkg.bar", "pkg.foo"]); - expect(merged.external_symbols["pkg.foo"].name).toBe("foo"); // base (tsc) wins - }); - - test("unions synthesized (anonymous-callback) callables from both", () => { - const tsc = result([]); - const jelly = result([], {}, { "src/x.foo:<3:10>": { name: "", path: "src/x.ts", start_line: 3, start_column: 10 } }); - const merged = mergeCallGraphs(tsc, jelly); - expect(Object.keys(merged.synthesized_callables)).toEqual(["src/x.foo:<3:10>"]); - }); - - test("does not mutate the input results", () => { - const tsc = result([edge("a", "b", ["tsc"])]); - const jelly = result([edge("a", "b", ["jelly"])]); - mergeCallGraphs(tsc, jelly); - expect(tsc.edges[0].provenance).toEqual(["tsc"]); - expect(tsc.edges[0].weight).toBe(1); - }); -});