diff --git a/CLAUDE.md b/CLAUDE.md index 6565c84..398eb88 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,11 +50,12 @@ while cache round-trips tree), `l1Body` (`call_sites` → `body{}`), `heritage`, (`src/schema/emit.ts`) runs them + assembles envelope + strips INTERNAL fields (`call_sites`, `abs_path`, cache trio). -Call graph defaults to **union** of two backends: TS compiler resolver -and embedded [Jelly](https://github.com/cs-au-dk/jelly) flow analyzer (recovers -higher-order/callback edges resolver misses). Merged edges keep -`provenance` tag (`tsc` / `jelly`); `--tsc-only` or `--call-graph-provider jelly` -picks one alone. +Call graph = tsc resolver + **defuse linker** (#98): deterministic per-callable +pass over resolver leftovers — alias chains, decorator edges, library-callback +edges, bounded interprocedural votes, CHA-by-name fallback. No whole-program +fixpoint, no backend flag, one code path. Module-scope calls attributed to +MODULE (python #131 parity). prov tags: `tsc` / `defuse` / `import`. Joern +superset ledger: `docs/design/specs/defuse-linker-joern-ledger.md`. ## Architecture — follow the pipeline @@ -65,8 +66,9 @@ it first; everything else is stage it calls, in order: 2. **buildSymbolTable** (`src/syntactic_analysis`) — modules, classes, interfaces, enums, type aliases, namespaces, functions, methods, variables, decorators, JSDoc, with precise source spans. -3. **call graph** (`src/semantic_analysis`) — `selectProvider()` picks tsc / jelly / - union; each provider returns edges + external (phantom) symbols. +3. **call graph** (`src/semantic_analysis`) — tsc resolver (`callGraph.ts`, incl. + module-scope sweep + RTA + phantoms) then `defuseLinker.ts` tiers T1–T5; + merged with provenance union. 4. **program graphs** (`src/dataflow`) — levels 3–4 (`-a 3`/`-a 4`): CFG → post-dominance/CDG → access-path def-use → PDG → SCC-condensed bottom-up summaries → SDG. This is *compute* (IR in `src/schema/graphs.ts`); `src/dataflow/attach.ts` writes it **onto tree** @@ -96,7 +98,7 @@ test — treat both as contracts, keep in lockstep with JSON. | `src/core.ts` | `analyze()` orchestrator — the spine | | `src/options` | Parsed CLI options / `AnalysisOptions` | | `src/syntactic_analysis` | Symbol table (ts-morph traversal) | -| `src/semantic_analysis` | Call-graph providers (tsc, jelly, union), phantoms | +| `src/semantic_analysis` | Call graph: tsc resolver + defuse linker (T1–T5), phantoms | | `src/dataflow` | L3/L4 program-graph **compute** (CFG, dominance/CDG, def-use, summaries, SDG) + `attach.ts` (IR → tree) | | `src/schema` | **the native v2 model** (`schema.ts`) + per-run passes (`assignIds`/`l1Body`/`heritage`/`homing`/`l2Callees`) + `emit.ts` (`finalizeAnalysis`) + `signatureOf` + graphs IR | | `src/build` | Dep materialization; `build/neo4j` = the v2 graph projection (project/rows/cypher/bolt/schema) | diff --git a/README.md b/README.md index 55d1a92..6131c94 100644 --- a/README.md +++ b/README.md @@ -24,11 +24,11 @@ structure into a **Neo4j property graph**. It is the TypeScript backend behind [Python](https://github.com/codellm-devkit/codeanalyzer-python) and [Java](https://github.com/codellm-devkit/codeanalyzer-java) siblings. -By default the call graph is the **union** of two backends: the TypeScript compiler's resolver and -[Jelly](https://github.com/cs-au-dk/jelly) — a flow-based analyzer that resolves higher-order and -callback edges the resolver misses, embedded in the `cants` binary (no extra install). Merged edges -keep a `provenance` tag (`tsc` / `jelly`), so you can still tell the two apart. Pass `--tsc-only` to -drop Jelly and run the resolver alone, or `--call-graph-provider jelly` for Jelly alone. +The call graph is the TypeScript compiler's resolver plus a **defuse linker** — a deterministic, +per-callable pass that backfills the edges the resolver misses (alias chains, decorator +invocations, callbacks handed to library calls, parameter-flow calls) with no whole-program +fixpoint. Edges keep a `provenance` tag (`tsc` / `defuse` / `import`), so you can tell the layers +apart, and the output is byte-identical across runs. ## Table of Contents @@ -55,9 +55,9 @@ drop Jelly and run the resolver alone, or `--call-graph-provider jelly` for Jell methods, variables, decorators, and JSDoc, with precise source spans. - **Call graph** — the TypeScript compiler's resolver plus Rapid Type Analysis (RTA), with **phantom (external) nodes** for calls into imported libraries and Node builtins. -- **Pluggable call-graph backend** — the `union` of the `tsc` resolver and the embedded - [Jelly](https://github.com/cs-au-dk/jelly) flow analyzer by default (`--tsc-only` for the resolver - alone, `--call-graph-provider jelly` for Jelly alone). +- **Defuse linker** — a deterministic per-callable pass over the resolver's leftovers: alias + chains, decorator invocations, library-callback edges, and bounded interprocedural votes — + validated as a strict superset of Joern's real call pairs on the reference corpus. - **Neo4j output** — project the analysis into a labeled property graph: a self-contained `graph.cypher` snapshot, or an **incremental** push to a live database over Bolt. - **Versioned schema** — a machine-readable, version-stamped Neo4j schema contract @@ -182,11 +182,6 @@ Options: node_modules) --no-phantoms disable phantom (external) nodes for imported/required library calls - --call-graph-provider call-graph backend: union (default, tsc ∪ - jelly) | tsc | jelly | both (deprecated alias - of union) (default: "union") - --tsc-only use the tsc resolver only — opt out of Jelly - edges (overrides --call-graph-provider) -c, --cache-dir cache/intermediate directory -v, --verbose increase verbosity (repeatable) -h, --help display help for command @@ -214,17 +209,12 @@ Options: cants --input ./my-ts-project --target-files src/a.ts src/b.ts ``` -4. **Resolver-only call graph (opt out of Jelly):** - ```sh - cants --input ./my-ts-project --tsc-only - ``` - -5. **Force a clean rebuild with a custom cache directory:** +4. **Force a clean rebuild with a custom cache directory:** ```sh cants --input ./my-ts-project --eager --cache-dir /path/to/custom-cache ``` -6. **Program graphs (level 3): CFG/PDG/SDG in `analysis.json`:** +5. **Program graphs (level 3): CFG/PDG/SDG in `analysis.json`:** ```sh cants --input ./my-ts-project -a 3 # full program_graphs section cants --input ./my-ts-project -a 3 --graphs cfg,pdg # scope the emitted graphs @@ -282,8 +272,8 @@ nodes all join. **Substrate (locked in [issue #2](https://github.com/codellm-devkit/codeanalyzer-typescript/issues/2)):** the CFG and reaching-definitions are hand-built from the ts-morph AST; the call-graph oracle is -the existing provenance-merged tsc ∪ Jelly graph; aliasing is a flow-insensitive copy-alias MVP -(Jelly points-to-backed propagation is a staged upgrade). Function summaries are composed +the provenance-merged tsc + defuse graph; aliasing is a flow-insensitive copy-alias MVP +(points-to-backed propagation is a staged upgrade). Function summaries are composed bottom-up over the SCC condensation of the call graph, with k-limited access paths; module globals ride the SDG as extra parameters. The analysis is deliberately sound-leaning and over-approximate; known unsoundness (dynamic `eval`, reflection/monkey-patching, npm-internal diff --git a/docs/design/specs/defuse-linker-call-graph.md b/docs/design/specs/defuse-linker-call-graph.md new file mode 100644 index 0000000..6bf38c6 --- /dev/null +++ b/docs/design/specs/defuse-linker-call-graph.md @@ -0,0 +1,144 @@ +# tsc + defuse linker call graph (Jelly removal) + +- **Status:** accepted, not yet implemented +- **Scope:** `codeanalyzer-typescript` only; one PR, tracked in #98 +- **Tracking:** #98 (work item); branch `feat/issue-098-defuse-linker`, stacked on + `refactor/issue-096-native-v2-model` (#97) — lands after it merges +- **Parity precedent:** codeanalyzer-python 1.2.0, which replaced PyCG's global fixpoint with + Jedi + a per-callable defuse linker + (`codeanalyzer-python/docs/design/specs/2026-08-25-defuse-linker-call-graph-design.md`, #148 + there). This spec is the TypeScript instantiation of that architecture; divergences are called + out explicitly. + +## Motivation + +Jelly is the analyzer's scale ceiling and its heaviest dependency. It is a whole-program flow +analysis — the same cost class as python's removed PyCG (3h19m on odoo without convergence; +Fraunhofer CPG OOM at 44GB on the same corpus) — and it is bundled INTO the shipped binary +(`src/main.ts` `__jelly` dispatch, `CANTS_SELF_JELLY`, `patches/`, the `@cs-au-dk/jelly` +dependency). On the vscode-class targets we want to analyze, the Jelly leg is unusable; the tsc +leg alone loses exactly the edges Jelly recovers. + +Measured on the repo fixtures (union provider, L2): 79 edges total, of which **6 are +jelly-only**, in four sharp classes: + +| Class | Fixture evidence | +| --- | --- | +| Decorator invocations | `UserController.show → Get`, `→ Param`, `list → Get` (sample-app) | +| Library-mediated callback edges | `UserService.describeAll → ` (lambda passed to `.map`) | +| Receiver typing inside anons | ` → User.describe` (element type of the mapped array) | +| Param-flow calls | ` → named` via a function-valued parameter (anon-app) | + +Small counts, but the classes are the point: each is a bounded, per-callable resolution problem. +The expensive substrate already exists in this repo as the L3 kernels +(`src/dataflow/defuse.ts` — k-limited access-path def-use with the flow-insensitive alias +substrate — plus the CFG machinery). The replacement follows the same Joern/Fraunhofer CPG +architecture python adopted: a fast base graph from the type-checker, then a **local** linker +pass that backfills what the resolver missed. No global fixpoint anywhere. + +## Contract-impact triage + +| Question | Answer | +| --- | --- | +| Schema v2 shape (node/edge kinds, fields, ids, levels) | unchanged; schema_version stays **2.1.0** | +| `prov` vocabulary | `"jelly"` disappears; **`"defuse"`** coined for linker-derived edges (technique-named, matching python's `"defuse"` and the DDG's `"reaching-defs"`/`"points-to"`). `"tsc"` and `"import"` unchanged. Both-found edges merge to `["defuse", "tsc"]` via the existing provenance union | +| Refinement contract | unchanged — the linker runs inside the L2 build; `callee: null→id` stays the single sanctioned refinement | +| Monotonicity gate | unaffected (edges only added at L2, as today) | +| `synthesized_callables` | shape + 2.1.0 compat index unchanged; the residual-fallback path stays, but provider-reported unknowns effectively vanish (tsc + linker only name tree signatures) | +| Repos | `codeanalyzer-typescript` now. **python-sdk follow-up** (separate PR, next SDK minor): the `tsc_only` kwarg threads `cldk/core.py` → `backend_config.py` → `typescript/codeanalyzer.py` → `typescript_analysis.py` and passes `--tsc-only`; it must be removed once this releases. SDK `prov` is passthrough (`List[str]` — verified), no model change | +| CLI (**BREAKING**) | `--call-graph-provider` and `--tsc-only` removed; one code path, no backend flag (python: "the linker is cheap and deterministic; nothing to opt out of") | + +## Locked decisions (design session 2026-08-26) + +1. **tsc resolver is the base call graph, always.** Its edges keep `prov: ["tsc"]`; RTA + expansion and the phantom/import leg (`prov: ["import"]`) are untouched. +2. **Jelly is removed wholesale**: `src/semantic_analysis/jellyProvider.ts`, the union provider + and `selectProvider`, `options.callGraphProvider`, both CLI flags, the `__jelly` argv mode in + `src/main.ts`, `CANTS_SELF_JELLY`, the `@cs-au-dk/jelly` dependency, its `patches/`, and + `union-provider.test.ts` (superseded by the linker suite + reference validation). +3. **The linker runs at L2 with targeted kernels**: def-use state is built only for callables + that still contain unresolved call sites after the tsc leg. Per-callable, no fixpoint, + **sorted iteration mandated** — deterministic by construction. +4. **Linker edges carry `prov: ["defuse"]`** and merge with tsc edges through the existing + `mergeCallGraphs` provenance union. Resolutions reach the L1 `call` body nodes through the + same channel the tsc leg uses today, with python's cache rule preserved: linker resolutions + are **never persisted into `callee_signature`** (the symbol table round-trips the analysis + cache; a persisted resolution would resurface on a warm run with the wrong provenance). +5. **External-callback edges are kept** — a deliberate, documented **divergence from python**: + when a function value (anonymous or named) is passed as an argument to an external or + unresolved callee, the linker emits `enclosing-callable → function-value`, + `prov: ["defuse"]`, **edge-only** (no body call node — matching Jelly's observed behavior; + there is no real call site in first-party code). Rationale: JS/TS is callback-central, and + 2.1.0 materialized anonymous callables as tree nodes precisely so they can be addressed — + they must stay reachable by edge, not only by containment. The parity clause covers shared + vocabulary, not per-language recall. +6. **Decorator invocations become linker edges**, same edge-only rule: decorators are captured + in the model (`decorators[]` with `qualified_name`) but their factory calls are outside + `walkBody`'s reach, so no body node exists today and none is added (adding one would move the + wire). The linker resolves `qualified_name`/`name` against the symbol table and emits + `decorated-owner → decorator`, `prov: ["defuse"]`. +7. **No backend flag.** One code path. + +## Tier ladder + +Tiers land in order; the Joern ledger (below) decides how far down the ladder the +implementation must go before the gate is clean. Each tier is per-callable or bounded-round — +never a fixpoint. + +- **T1 — local value chase.** For an unresolved call site whose callee expression is a local + binding: chase the def-use chain (existing `defuse.ts` kernels, k-limited access paths) + through alias assignments (`const f = handler; f()`) to a function literal / declaration / + import binding. Imports resolve cross-module through the symbol table (the checker already + did most of this; the chase covers what it declared as "a variable", not "a function"). +- **T2 — decorator edges** (decision 6). +- **T3 — external-callback rule** (decision 5). +- **T4 — interprocedural votes, bounded.** A type-oracle round in python's style, narrowed by + what tsc already proves: (a) function values passed at **resolved internal** call sites vote + for the callee's parameter — a parameter-invoking site (`cb()`) resolves to the voted + functions; (b) return summaries (`return inner` / unique ctor returns) let + `const f = factory(); f()` resolve; (c) `this.x = fn` property assignments type + `this.x()` sites. Two bounded rounds (round one's resolutions vote before round two), + internal-target votes only. +- **T5 — CHA-by-name fallback.** Receiver call sites that survive every typed tier resolve to + every internal callable of that method name (bounded per site) — the over-approximation Joern + itself emits for untyped receivers. Applied last so precise resolutions are never widened. + +## Reference validation (the enforced gate) + +Mirrors python's method, per the maintainer's mandate: iterate edge-for-edge against **Joern +`jssrc2cpg`** (available in `~/workspace/codellm-devkit/joern-dist`) until our call graph is a +**strict superset of every real edge** Joern produces on the validation corpus: + +- **Corpus:** `test/fixtures/sample-app`, `dataflow-app`, `anon-app`, plus **one real-world + express/nest application** vendored or pinned at implementation time (the toy fixtures alone + are too small to trust a superset claim). +- **"Real edge"** = both endpoints exist in source and are nameable in this schema; Joern's + synthetic families (`N` internals where we hold the positional anon node, ``, + ``, fabricated members) are excluded through a **committed exception ledger, audited + per class** — python's discipline, not a waiver. +- **Scale benchmark:** microsoft/vscode at L1/L2 — wall-clock, peak RSS, edge counts by `prov` + — reported in the PR next to Joern `jssrc2cpg` on the same tree (or its failure mode). The + giant-JSON emission ceiling is out of scope here (separate issue); the benchmark measures the + analyze/compute phase and the Bolt projection path. + +## Acceptance + +- Joern superset ledger committed: 100% real-edge coverage per corpus app, every residual + classified. +- **Jelly-recovery spike metric** (python's PyCG analog): of today's jelly-only edges on the + fixtures, the % the linker recovers — reported in the PR, no hard gate (some jelly edges may + be judged junk by the ledger; the report says which and why). +- **A/B determinism:** paired runs byte-identical on the corpus `call_graph` (the linker adds + no nondeterminism; tsc inference is deterministic — stronger than python's Jedi caveat). +- Full suite, typecheck, monotonicity and Neo4j conformance gates green; `git grep -li jelly` + over `src`/`packaging`/`patches`/`package.json` returns nothing; the binary loses its + `__jelly` mode and shrinks. + +## Release plan + +- Ships in the analyzer's next MINOR (with the #96 native-model rewrite already queued for it); + release notes carry **BREAKING** lines for the removed `--call-graph-provider`/`--tsc-only` + flags — python 1.2.0 precedent for a flag removal in a minor. schema_version untouched. +- **python-sdk follow-up (tracked in this spec; file the issue when picked up):** remove the + `tsc_only` kwarg chain and its `--tsc-only` pass-through, then bump the SDK's pinned analyzer + version. Until it lands, `tsc_only=True` against the new binary is the one known break. diff --git a/docs/design/specs/defuse-linker-joern-ledger.md b/docs/design/specs/defuse-linker-joern-ledger.md new file mode 100644 index 0000000..e474651 --- /dev/null +++ b/docs/design/specs/defuse-linker-joern-ledger.md @@ -0,0 +1,95 @@ +# Joern superset ledger — tsc + defuse call graph (#98) + +The enforced acceptance gate of `defuse-linker-call-graph.md`: the analyzer's L2 call graph must +be a **strict superset of every real call pair** Joern `jssrc2cpg` produces on the validation +corpus, plus a scale audit on microsoft/vscode. "Real" = a SINGLE-candidate Joern resolution +whose endpoints exist in source and are nameable in this schema; everything excluded is +classified below and audited per family, never waved through. Reproduce with `scripts/joern/` +(dump-calls.sc → compare_joern.py; corpus RESIDUAL must be 0). + +- **Joern:** v4 distribution, `jssrc2cpg` frontend +- **Analyzer:** branch `feat/issue-098-defuse-linker` (tsc resolver + defuse linker, no Jelly) + +## Corpus gate (enforced: residual 0) + +| App | Joern real pairs | Covered | Residual | +| --- | --- | --- | --- | +| `test/fixtures/sample-app` | 23 | 23 | **0** | +| `test/fixtures/dataflow-app` | 19 | 19 | **0** | +| `test/fixtures/anon-app` | 2 | 2 | **0** | +| nestjs-realworld-example-app @ `c1c2cc4` (35 files) | 30 | 30 | **0** (ours: 126 internal edges, 4.2× Joern's 30) | + +A/B determinism: paired analyzer runs byte-identical on every corpus app and on vscode's edge +dump (hash-compared). The linker adds no nondeterminism; the tsc checker is deterministic. + +## vscode scale audit (microsoft/vscode @ a3c9dc6, `src/`, 8,735 TS files / 1.15M LOC) + +64GB M-series (10 cores). Analyzer single-threaded (`-j 1`), eager, no deps materialized; +Joern on all 10 cores at `-Xmx48g`. + +| Run | Wall | Max RSS | Output | +| --- | --- | --- | --- | +| cants L1 | **4m15s** | 18.9GB | 136,973 callables | +| cants L2 | **5m52s** | 24.4GB | **1,024,232 edges** — tsc 970,334 (324,525 resolved + 778,070 RTA + 89,344 phantom), defuse 54,170 (430 decorator / 26,466 callback / 1,797 votes / 31,581 CHA / rest chase) | +| Joern jssrc2cpg parse | **9m06s** | 30.3GB | CPG; 941,132 call rows dumped | + +Superset audit against Joern's single-candidate real pairs, after seven ledger-driven fix +rounds: **54,885 / 55,074 covered (99.66%), residual 189** — past python's odoo bar (99.0%, +final residual 243). Reference: the engines this architecture replaced DNF'd at this scale +class (PyCG 3h19m without convergence; Fraunhofer CPG OOM at 44GB). + +### Analyzer fixes the ledger forced (python's reference-validation experience, repeated) + +1. **Concise-arrow call sites** — `u => u.describe()` recorded no call (children-only body walk); + Jelly's approximated edge had masked the L1 gap. Now checker-typed. +2. **Module-scope callers** — top-level `main()`, class decorators, the top-level express idiom + had no caller. Attributed to the MODULE (python #131 parity), module prefix id-homed so the + edges land on the module node. +3. **Tagged template calls** — `` inline`url(...)` `` was invisible to L1/L2 end to end (walkBody, + resolver, call index), while L3's exception model already treated it as a call. vscode's + cssValue idiom found it; regression-tested. +4. **Parameter-default initializer calls** — `f(sel, style = getSharedStyleSheet())` executes in + the callee's activation but lived outside `getBody()`. vscode's domStylesheets family found + it; regression-tested. +5. **JS sources were never discovered** — `SOURCE_EXTS` was ts-only, so vscode's vendored + `marked.js` was "analyzed" through its bodiless `marked.d.ts` (zero edges from every body). + `.js`/`.jsx`/`.mjs`/`.cjs` are first-class now, with two-way sibling rules: a `.js` beside a + real `.ts` source is build output (skipped); a `.d.ts` beside an analyzed `.js` is its + declaration file (skipped as a module — the checker still reads it from disk). +6. **T4c — the ctor-field callback chain** — `this.migrate(...)` where the field arrives through + the constructor (parameter property / `this.f = param`) resolves to the function values passed + at the class's `new` sites, with ONE bounded parameter hop (`register(key, cb)` → + `new Migration(key, cb)`), and module-scope resolved calls feed the vote rounds — so the + registered callbacks' own `write()` sites resolve too. vscode's migrateOptions family + (22 pairs) closed end-to-end, no fixpoint anywhere. + +## Exception classes (audited) + +| Class | vscode count | What it is / verdict | +| --- | --- | --- | +| `joern-synthetic-helper` | 169,861 | Their TS-lowering machinery (`__decorate`, `__param`, `__metadata`, `__ecma.*`, `require`/`import` plumbing) — desugaring artifacts, not source calls | +| `joern-name-fanout` | 131,710 | Multi-candidate `callee` lists (one `.toString()` row links a 131KB candidate string) — candidate enumeration, not resolution; python's "speculative typed-attribute fan-out". Informational: 5,151 of these have ≥1 candidate covered by our graph | +| `external` | 173,092 | Callee homes outside the project — outside the internal gate; we carry these as phantom edges with id-homed external nodes | +| `notin` / fabricated stubs | 61,883 | Parameters-as-callees (`next()` → fabricated `::program:next`), decorator-value targets (`@User(...)` where `User = createParamDecorator(...)`), import-stubs — targets that do not exist in source as callables (python's identical families) | +| `odd-chain` / `lambda-unmapped` | 5,455 / 1,842 | Their fullName grammar edge cases and lambdas our line-matcher cannot uniquely map — mapping losses, counted, not silently dropped | +| `joern-this-misresolution` (covered, listed) | 1,469 | Their single "resolution" names a same-file free function while the receiver in source is `this.` — we hold the typed method edge (e.g. `setZoomLevel → WindowManager.getZoomLevel` vs their `→ browser.getZoomLevel`) | +| `joern-name-misresolution` (covered, listed) | 338 | Same shape across files — they name-linked `ActionBar.dispose` where the call is the imported free `dispose` from lifecycle.ts; we hold the typed import edge | +| `joern-unresolved` | 72 | ``, no linked callee — their unresolved set | + +## The audited residual (189, classified) + +| Family | ≈count | Nature | +| --- | --- | --- | +| Registry-pattern generics | ~16 | `Registry.as(Extensions.X)` through re-exported const + type args — the shape resolves in isolation; the vscode instantiation defeats the checker without deps materialized | +| Promise-executor params named like real functions | ~14 | `new Promise(resolve => … resolve())` where the file also declares a real `resolve` — Joern name-links the free function; the true target is the parameter (their parameters-as-callees family wearing a real name) | +| **Static/instance same-name collision** | 11 | `Range.isEmpty` (instance) calls `Range.isEmpty` (static): the signature grammar cannot mark static, both collapse to ONE signature — the pair is unrepresentable and the collision gate flags it. A REAL schema-grammar limitation surfaced by this audit → design-mode follow-up | +| Closure-local callables through deep value flow | ~60 | Functions escaping via event emitters/registries beyond T4/T4c's bounded hops (settingsTree `onChange`, event utilities) — python zeroed its analog only with whole-program propagation (#150), the staged next step | +| Accessor/duck-typed and misc tails | ~88 | Getter-vs-method naming (`EventMultiplexer.event`), interface duck-typing (`ISearchTreeFolderMatch.id`), terminalTaskSystem dynamic patterns | + +## Known non-goals (recorded, deliberate) + +- Whole-program propagation for escaped closure-locals (python #150's tier) — staged follow-up, + not this issue. +- Static/instance signature discrimination — schema id-grammar change, design mode. +- Property-arrow class members as tree callables — future schema work; T5's bounded CHA covers + the call sites meanwhile. diff --git a/graph.cypher b/graph.cypher new file mode 100644 index 0000000..8356717 --- /dev/null +++ b/graph.cypher @@ -0,0 +1,267 @@ +// ── constraints & indexes ── +CREATE CONSTRAINT application_id IF NOT EXISTS FOR (x:Application) REQUIRE x.id IS UNIQUE; +CREATE CONSTRAINT cannode_id IF NOT EXISTS FOR (x:CanNode) REQUIRE x.id IS UNIQUE; +CREATE INDEX callable_name IF NOT EXISTS FOR (c:TSCallable) ON (c.name); +CREATE INDEX cannode_kind IF NOT EXISTS FOR (n:CanNode) ON (n.kind); +CREATE INDEX cannode_module IF NOT EXISTS FOR (n:CanNode) ON (n._module); + +// ── wipe this project's prior subgraph (external targets are shared) ── +MATCH (a:Application {id: 'can://typescript/anon-app'}) +OPTIONAL MATCH (a)-[:TS_HAS_MODULE]->(m:TSModule) +OPTIONAL MATCH (m)-[:TS_DECLARES|TS_HAS_METHOD|TS_HAS_FIELD|TS_HAS_BODY_NODE*1..]->(x) +DETACH DELETE x, m, a; + +// ── nodes ── +UNWIND [ + {k: 'can://typescript/anon-app', p: {id: 'can://typescript/anon-app', schema_version: '2.1.0', language: 'typescript', max_level: 4, k_limit: 3, analyzer_name: 'codeanalyzer-typescript', analyzer_version: '1.0.0'}} +] AS row +MERGE (n:Application {id: row.k}) +SET n += row.p, n:TSApplication; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts', p: {id: 'can://typescript/anon-app/src/routes.ts', kind: 'module', name: 'src/routes.ts', is_tsx: false, is_declaration_file: false, start_line: 1, end_line: 22, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSModule; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/', p: {id: 'can://typescript/anon-app/src/routes.ts/', kind: 'arrow', signature: 'src/routes.', name: '(anonymous)', return_type: 'void', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 13, end_line: 15, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/', p: {id: 'can://typescript/anon-app/src/routes.ts/login/', kind: 'arrow', signature: 'src/routes.login.', name: '(anonymous)', return_type: 'void', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/', kind: 'arrow', signature: 'src/routes.outer.', name: '(anonymous)', return_type: '() => number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//', kind: 'arrow', signature: 'src/routes.outer..', name: '(anonymous)', return_type: 'number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 20, end_line: 20, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSCallable:TSAnonymousCallable; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/@14:3', p: {id: 'can://typescript/anon-app/src/routes.ts/@14:3', kind: 'call', start_line: 14, end_line: 14, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', kind: 'actual_in', of: 'arg0', parent: '14:3', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', p: {id: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', kind: 'actual_out', of: '$ret', parent: '14:3', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/@entry', kind: 'entry', start_line: 13, end_line: 15, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/@exit', kind: 'exit', start_line: 13, end_line: 15, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@formal_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/@formal_in:0', kind: 'formal_in', of: 'req', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@formal_in:1', p: {id: 'can://typescript/anon-app/src/routes.ts/@formal_in:1', kind: 'formal_in', of: 'res', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@2:3', p: {id: 'can://typescript/anon-app/src/routes.ts/login@2:3', kind: 'statement', start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/login@entry', kind: 'entry', start_line: 1, end_line: 6, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/login@exit', kind: 'exit', start_line: 1, end_line: 6, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/login@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@3:5', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@3:5', kind: 'statement', start_line: 3, end_line: 3, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@4:5', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@4:5', kind: 'call', callee: 'can://typescript/anon-app/src/routes.ts/query', start_line: 4, end_line: 4, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', kind: 'actual_in', of: 'arg0', parent: '4:5', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', kind: 'actual_out', of: '$ret', parent: '4:5', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@entry', kind: 'entry', start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@exit', kind: 'exit', start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:0', kind: 'formal_in', of: 'req', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:1', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:1', kind: 'formal_in', of: 'res', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@17:21', p: {id: 'can://typescript/anon-app/src/routes.ts/named@17:21', kind: 'statement', start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/named@entry', kind: 'entry', start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/named@exit', kind: 'exit', start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/named@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@20:3', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@20:3', kind: 'statement', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@entry', kind: 'entry', start_line: 19, end_line: 21, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@exit', kind: 'exit', start_line: 19, end_line: 21, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', kind: 'statement', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@entry', kind: 'entry', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@exit', kind: 'exit', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', kind: 'statement', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@entry', kind: 'entry', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@exit', kind: 'exit', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@9:3', p: {id: 'can://typescript/anon-app/src/routes.ts/query@9:3', kind: 'statement', start_line: 9, end_line: 9, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/query@entry', kind: 'entry', start_line: 8, end_line: 10, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/query@exit', kind: 'exit', start_line: 8, end_line: 10, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', kind: 'formal_in', of: 'sql', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/query@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSBodyNode; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/app', p: {id: 'can://typescript/anon-app/src/routes.ts/app', kind: 'field', name: 'app', type: 'any', start_line: 12, end_line: 12, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSField; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/login', p: {id: 'can://typescript/anon-app/src/routes.ts/login', kind: 'function', signature: 'src/routes.login', name: 'login', return_type: '(req: any, res: any) => void', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: true, is_ambient: false, is_implicit: false, start_line: 1, end_line: 6, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named', p: {id: 'can://typescript/anon-app/src/routes.ts/named', kind: 'arrow', signature: 'src/routes.named', name: 'named', return_type: 'number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer', p: {id: 'can://typescript/anon-app/src/routes.ts/outer', kind: 'function', signature: 'src/routes.outer', name: 'outer', return_type: '() => () => number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: true, is_ambient: false, is_implicit: false, start_line: 19, end_line: 21, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query', p: {id: 'can://typescript/anon-app/src/routes.ts/query', kind: 'function', signature: 'src/routes.query', name: 'query', return_type: 'string', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: true, is_ambient: false, is_implicit: false, start_line: 8, end_line: 10, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSCallable; + +// ── relationships ── +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/query', p: {weight: 1, prov: ['tsc']}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_CALLS]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@entry', t: 'can://typescript/anon-app/src/routes.ts/login@2:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@entry', t: 'can://typescript/anon-app/src/routes.ts/named@17:21', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@entry', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@entry', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@entry', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@entry', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_CDG]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@14:3', t: 'can://typescript/anon-app/src/routes.ts/@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/@14:3', t: 'can://typescript/anon-app/src/routes.ts/@exit', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@2:3', t: 'can://typescript/anon-app/src/routes.ts/login@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@entry', t: 'can://typescript/anon-app/src/routes.ts/login@2:3', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@3:5', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5', t: 'can://typescript/anon-app/src/routes.ts/login/@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5', t: 'can://typescript/anon-app/src/routes.ts/login/@exit', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@17:21', t: 'can://typescript/anon-app/src/routes.ts/named@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@entry', t: 'can://typescript/anon-app/src/routes.ts/named@17:21', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@20:3', t: 'can://typescript/anon-app/src/routes.ts/outer@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@entry', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', t: 'can://typescript/anon-app/src/routes.ts/outer/@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', t: 'can://typescript/anon-app/src/routes.ts/outer/@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@entry', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', t: 'can://typescript/anon-app/src/routes.ts/outer//@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', t: 'can://typescript/anon-app/src/routes.ts/outer//@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@entry', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@9:3', t: 'can://typescript/anon-app/src/routes.ts/query@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@entry', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', k: 'fallthrough', p: {kind: 'fallthrough'}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_CFG_NEXT {_k: row.k}]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', k: 'req.query.probe|reaching-defs', p: {var: 'req.query.probe', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', k: 'res.send|reaching-defs', p: {var: 'res.send', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@2:3', t: 'can://typescript/anon-app/src/routes.ts/login@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@3:5', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', k: 'email|reaching-defs', p: {var: 'email', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', k: 'req.body.email|reaching-defs', p: {var: 'req.body.email', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@17:21', t: 'can://typescript/anon-app/src/routes.ts/named@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@20:3', t: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@entry', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', k: 'src/routes.named|reaching-defs', p: {var: 'src/routes.named', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', t: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@entry', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', k: 'src/routes.named|reaching-defs', p: {var: 'src/routes.named', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', t: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@entry', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', k: 'src/routes.named|reaching-defs', p: {var: 'src/routes.named', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@9:3', t: 'can://typescript/anon-app/src/routes.ts/query@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@entry', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', k: 'sql|reaching-defs', p: {var: 'sql', prov: ['reaching-defs']}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_DDG {_k: row.k}]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login/', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer//', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer/', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/login', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/named', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/outer', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/query', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_DECLARES]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@14:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@formal_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@formal_in:1', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:1', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@2:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@17:21', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@formal_out', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_HAS_BODY_NODE]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/app', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_HAS_FIELD]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app', t: 'can://typescript/anon-app/src/routes.ts', p: {}} +] AS row +MATCH (a:Application {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_HAS_MODULE]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', t: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_PARAM_IN]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/query@formal_out', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_PARAM_OUT]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5', t: 'can://typescript/anon-app/src/routes.ts/query', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_RESOLVES_TO]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', t: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_SUMMARY]->(b) +SET r += row.p; diff --git a/package.json b/package.json index 02c960b..cdb5459 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ }, "scripts": { "start": "bun run src/index.ts", - "build": "bun build ./src/main.ts ./src/dataflow/worker.ts --compile --external @babel/preset-typescript --outfile dist/cants", + "build": "bun build ./src/main.ts ./src/dataflow/worker.ts --compile --outfile dist/cants", "gen:schema": "bun run src/index.ts --emit schema > schema.neo4j.json", "gen:readme": "bun run scripts/update-readme.ts", "test:container": "RUN_CONTAINER_TESTS=1 bun test test/neo4j-bolt.test.ts", @@ -22,14 +22,10 @@ "ts-morph": "^28.0.0" }, "devDependencies": { - "@cs-au-dk/jelly": "0.13.0", "@testcontainers/neo4j": "^12.0.3", "@types/bun": "^1.3.14", "@types/node": "^25.9.1", "testcontainers": "^12.0.3", "typescript": "^6.0.3" - }, - "patchedDependencies": { - "@cs-au-dk/jelly@0.13.0": "patches/@cs-au-dk%2Fjelly@0.13.0.patch" } } diff --git a/packaging/python/build_wheels.sh b/packaging/python/build_wheels.sh index 061b3b8..65bbac6 100755 --- a/packaging/python/build_wheels.sh +++ b/packaging/python/build_wheels.sh @@ -77,12 +77,8 @@ for entry in "${TARGETS[@]}"; do clean_bin - # Entry is src/main.ts (the multi-call dispatcher that also embeds the Jelly CLI), NOT src/index.ts. - # --external @babel/preset-typescript: Jelly's Babel core dynamically require()s that preset; it is - # never loaded at runtime (Jelly sets babelrc/configFile false), so excluding it is safe and avoids - # a bundle-time resolution error. - ( cd "$REPO_ROOT" && bun build ./src/main.ts --compile --target="$target" \ - --external @babel/preset-typescript --outfile "$BIN_DIR/cants$ext" ) + # Entry is src/main.ts — NOT src/index.ts. + ( cd "$REPO_ROOT" && bun build ./src/main.ts --compile --target="$target" --outfile "$BIN_DIR/cants$ext" ) # Ship the Neo4j schema contract (platform-independent) next to the binary, so consumers can # read the version-locked schema.json without invoking the binary. See codeanalyzer_typescript.schema_path(). diff --git a/patches/@cs-au-dk%2Fjelly@0.13.0.patch b/patches/@cs-au-dk%2Fjelly@0.13.0.patch deleted file mode 100644 index 0a88e7b..0000000 --- a/patches/@cs-au-dk%2Fjelly@0.13.0.patch +++ /dev/null @@ -1,18 +0,0 @@ -diff --git a/lib/parsing/parser.js b/lib/parsing/parser.js -index de36b2fcf1d06cfa8703b0d2a3f0f2cd11c43557..eed24065f2a859b539bc974d6607bf9196d30e4d 100644 ---- a/lib/parsing/parser.js -+++ b/lib/parsing/parser.js -@@ -12,11 +12,11 @@ const transformOptions = [false, true].map((fragmentStateDefined) => (0, core_1. - cloneInputAst: false, - plugins: [ - extras_1.replaceTypeScriptImportExportAssignmentsAndAddConstructors, -- ['@babel/plugin-transform-typescript', { -+ [require('@babel/plugin-transform-typescript').default, { - onlyRemoveTypeImports: fragmentStateDefined, - allowDeclareFields: fragmentStateDefined, - }], -- ['@babel/plugin-transform-template-literals', { loose: true }] -+ [require('@babel/plugin-transform-template-literals').default, { loose: true }] - ], - cwd: __dirname, - babelrc: false, diff --git a/scripts/joern/compare_joern.py b/scripts/joern/compare_joern.py new file mode 100755 index 0000000..7dc44c3 --- /dev/null +++ b/scripts/joern/compare_joern.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Joern jssrc2cpg superset comparator (#98 / defuse-linker-call-graph.md). + +Usage: + 1. joern-parse --language jssrc -o app.cpg + 2. joern --script scripts/joern/dump-calls.sc --param cpgFile=app.cpg --param outFile=app.tsv + 3. python3 scripts/joern/compare_joern.py app.tsv [-v] + +Maps Joern's real call pairs onto analyzer signatures and verifies our call_graph covers them +(RESIDUAL must be 0). Exception classes are counted and printed, never silently waved through — +see docs/design/specs/defuse-linker-joern-ledger.md for the audited class definitions.""" +import json, re, subprocess, sys, collections + +JUNK_NAMES = {"__decorate", "__param", "__metadata", "__runInitializers", "__esDecorate", "require", "import"} + +def load_joern(tsv): + calls, methods = [], {} + malformed = 0 + for line in open(tsv, errors="replace"): + parts = line.rstrip("\n").split("\t") + try: + if parts[0] == "C" and len(parts) == 6: + _, caller, name, direct, linked, line_no = parts + calls.append((caller, name, direct, linked, int(line_no))) + elif parts[0] == "M" and len(parts) == 4: + _, fn, ln, col = parts + methods[fn] = (int(ln), int(col)) + else: + malformed += 1 # identifiers containing tabs/newlines (template literals etc.) + except ValueError: + malformed += 1 + if malformed: + print(f" [note] {malformed} malformed dump rows skipped (control chars in identifiers)") + return calls, methods + +STRIP_EXT = re.compile(r"\.(d\.ts|tsx|ts|jsx|js|mts|cts|mjs|cjs)$") + +def build_indexes(our_sigs, edges): + """vscode-scale: pre-index anon sigs by (base, line) and edges by target.""" + import collections as _c + anon_ix = _c.defaultdict(list) + for sig in our_sigs: + if " our signature, or (None, reason).""" + if "::" not in fn: + return None, "external" + path, chain = fn.split("::", 1) + segs = chain.split(":") + if segs[0] != "program": + return None, "odd-chain" + segs = segs[1:] + prefix = STRIP_EXT.sub("", path) + if not segs: + return prefix, None # module-scope caller: the module prefix IS the source (python #131) + out = [] + consumed = ["program"] + for s in segs: + consumed.append(s) + if s == "" or s == "super": + out.append("constructor") + elif s.startswith(""): + # a lambda anywhere in the chain: line-match the progressive Joern fullName against + # our positional under the mapped base so far (pre-indexed) + jfn = path + "::" + ":".join(consumed) + ln = methods.get(jfn, (-1, -1))[0] + base = prefix + ("." + ".".join(out) if out else "") + cands = sorted(anon_ix.get((base, str(ln)), [])) + if len(cands) != 1: + return None, "lambda-unmapped" + out.append(cands[0].rsplit(".", 1)[1]) + else: + out.append(s) + return prefix + "." + ".".join(out), None + +def our_edges(fixture, dump=None): + if dump: + d = json.load(open(dump)) + else: + here = __import__("os").path.dirname(__import__("os").path.abspath(__file__)) + out = subprocess.run(["bun", "run", here + "/edges.ts", fixture], capture_output=True, text=True) + if out.returncode != 0: + print(out.stderr[-2000:]); sys.exit(1) + d = json.loads(out.stdout) + return set(map(tuple, d["edges"])), set(d["sigs"]) + +def main(fixture, tsv, dump=None): + calls, methods = load_joern(tsv) + edges, sigs = our_edges(fixture, dump) + anon_ix, edges_by_target, edges_by_src = build_indexes(sigs, edges) + covered, residual = [], [] + classes = collections.Counter() + seen = set() + for caller, name, direct, linked, line in calls: + if name in JUNK_NAMES or name == "": + classes["joern-synthetic-helper"] += 1; continue + cands = [c for c in linked.split("|") if c] if linked else [] + if len(cands) > 1: + # Joern's name-based candidate enumeration (their untyped-receiver fan) — python + # ledger's "speculative typed-attribute fan-out" class: not a resolution, not gated. + # Informational: does our graph cover at least one enumerated candidate? + classes["joern-name-fanout"] += 1 + src_f, _ = map_fullname(caller, methods, sigs, anon_ix) + hit = False + for cf in cands[:64]: + d_f, _ = map_fullname(cf, methods, sigs, anon_ix) + if d_f and src_f and (src_f, d_f) in edges: + hit = True; break + if hit: classes["joern-name-fanout-covered>=1"] += 1 + continue + callee_fn = cands[0] if cands else (direct if direct != "" else "") + if not callee_fn or callee_fn == "": + classes["joern-unresolved"] += 1; continue + src, why_s = map_fullname(caller, methods, sigs, anon_ix) + dst, why_d = map_fullname(callee_fn, methods, sigs, anon_ix) + if dst is None or dst not in sigs: + k = "external-or-unmapped-target:" + (why_d or "notin") + classes[k] += 1 + if "-v" in sys.argv and (why_d or "notin") != "external": print(" [", k, "]", caller, "->", callee_fn) + continue + if src is None or src not in sigs: + classes["caller-unmapped:" + (why_s or "notin")] += 1 + if "-v" in sys.argv: print(" [caller-unmapped]", caller, "->", callee_fn) + continue + pair = (src, dst) + if pair in seen: continue + seen.add(pair) + if pair in edges: covered.append(pair) + elif "." not in src and any(e0.startswith(src + ".") for e0 in edges_by_target.get(dst, ())): + # Joern desugars decorator factories to module-scope __decorate calls; we attribute the + # SAME invocation to the decorated callable (more precise). Same edge, finer caller. + classes["decorator-attribution-variant"] += 1 + covered.append(pair) + else: + # Joern this-misresolution variant: their single "resolution" names a free function + # ., while we hold, from the SAME caller, a typed edge to a METHOD + # .. of the same file+name (or vice versa). The receiver in source + # decides which is real; the checker types receivers, their name-link does not. + dfile, _, dname = dst.rpartition(".") + variant = False + for our_dst in edges_by_src.get(src, ()): + if our_dst == dst: continue + if our_dst.rsplit(".", 1)[-1] == dname and (our_dst.startswith(dfile + ".") or dst.startswith(our_dst.rsplit(".", 2)[0] + ".")): + variant = True; break + if variant: + classes["joern-this-misresolution (typed edge held)"] += 1 + covered.append(pair) + elif any(t.rsplit(".", 1)[-1] == dname for t in edges_by_src.get(src, ())): + # Weaker tier: from the same caller we hold a typed edge to a target of the SAME + # LEAF NAME in another file (e.g. the imported free `dispose` from lifecycle.ts, + # where Joern name-linked ActionBar.dispose). The checker resolved the receiver; + # their single-candidate name-link did not. + classes["joern-name-misresolution (typed same-name edge held)"] += 1 + covered.append(pair) + else: + residual.append(pair) + print(f"== {fixture}: joern real pairs {len(seen)}, covered {len(covered)}, RESIDUAL {len(residual)}") + for p in residual: print(" MISSING:", p[0], "->", p[1]) + for k, v in sorted(classes.items()): print(f" [class] {k}: {v}") + +if __name__ == "__main__": + dumps = [a for a in sys.argv[3:] if a.endswith(".json")] + main(sys.argv[1], sys.argv[2], dumps[0] if dumps else None) diff --git a/scripts/joern/dump-calls.sc b/scripts/joern/dump-calls.sc new file mode 100644 index 0000000..0fa57cc --- /dev/null +++ b/scripts/joern/dump-calls.sc @@ -0,0 +1,16 @@ +@main def main(cpgFile: String, outFile: String) = { + importCpg(cpgFile) + val sb = new StringBuilder + cpg.call.foreach { c => + if (!c.name.startsWith(" + sb.append(s"M\t${m.fullName}\t${m.lineNumber.getOrElse(-1)}\t${m.columnNumber.getOrElse(-1)}\n") + } + val pw = new java.io.PrintWriter(outFile); pw.write(sb.toString); pw.close() +} diff --git a/scripts/joern/edges.ts b/scripts/joern/edges.ts new file mode 100644 index 0000000..11bac19 --- /dev/null +++ b/scripts/joern/edges.ts @@ -0,0 +1,23 @@ +/** Ledger helper (#98): print the analyzer's L2 signature-level edge set + signature universe + * for a target app as JSON — consumed by compare_joern.py. */ +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../../src/core"; +import { forEachCallable } from "../../src/schema"; +import type { AnalysisOptions } from "../../src/options"; + +const input = path.resolve(process.argv[2] as string); +const opts = { + input, output: null, emit: "json", appName: null, neo4jUri: null, neo4jUser: "neo4j", + neo4jPassword: "", neo4jDatabase: null, analysisLevel: 2, graphs: [], graphFieldDepth: 3, + jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, phantoms: true, + cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "ledger-")), verbosity: 0, +} as AnalysisOptions; +const r = await analyze(opts); +const sigs: string[] = []; +for (const [fileKey, mod] of Object.entries(r.internal.symbol_table)) { + sigs.push(fileKey.replace(/\.d\.ts$/, "").replace(/\.(tsx|ts|jsx|js|mts|cts|mjs|cjs)$/, "")); + forEachCallable(mod, (c) => sigs.push(c.signature)); +} +process.stdout.write(JSON.stringify({ edges: r.internal.call_graph.map((e) => [e.source, e.target]), sigs })); diff --git a/src/cli.ts b/src/cli.ts index c2bdb46..71a3400 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1,6 +1,6 @@ import * as path from "node:path"; import { Command, Option } from "commander"; -import type { AnalysisOptions, CallGraphProviderName, EmitTarget } from "./options"; +import type { AnalysisOptions, EmitTarget } from "./options"; import { ALL_GRAPHS, type GraphSelector } from "./schema"; /** @@ -57,12 +57,6 @@ export function buildProgram(): Command { .option("--lazy", "reuse the cache (default)") .option("--no-build", "skip dependency materialization (use a prepared node_modules)") .option("--no-phantoms", "disable phantom (external) nodes for imported/required library calls") - .option( - "--call-graph-provider ", - "call-graph backend: union (default, tsc ∪ jelly) | tsc | jelly | both (deprecated alias of union)", - "union", - ) - .option("--tsc-only", "use the tsc resolver only — opt out of Jelly edges (overrides --call-graph-provider)") .option("-c, --cache-dir ", "cache/intermediate directory") .option("-v, --verbose", "increase verbosity (repeatable)", (_v: string, prev: number) => prev + 1, 0) .allowExcessArguments(true); @@ -137,23 +131,6 @@ export function parseArgs(argv: string[]): AnalysisOptions { if (emit !== "schema" && !o.input) program.error("required option '-i, --input ' not specified"); const targets: string[] | null = Array.isArray(o.targetFiles) && o.targetFiles.length ? o.targetFiles.map(String) : null; - // --tsc-only is the forced opt-out: it wins over --call-graph-provider. Otherwise `both` is a - // deprecated alias of `union` (warn, but honor it); unknown values fall back to the union default. - let cgProvider: CallGraphProviderName; - if (o.tscOnly) { - cgProvider = "tsc"; - } else if (o.callGraphProvider === "tsc") { - cgProvider = "tsc"; - } else if (o.callGraphProvider === "jelly") { - cgProvider = "jelly"; - } else { - if (o.callGraphProvider === "both") { - // stderr only — stdout may carry compact JSON when -o is omitted. - console.error("warning: --call-graph-provider both is deprecated; it now behaves as 'union' (tsc ∪ jelly)."); - } - cgProvider = "union"; - } - return { input: o.input ? path.resolve(String(o.input)) : "", output: o.output ? path.resolve(String(o.output)) : null, @@ -173,7 +150,6 @@ export function parseArgs(argv: string[]): AnalysisOptions { // commander maps --no-build / --no-phantoms to opts.build/phantoms === false noBuild: o.build === false, phantoms: o.phantoms !== false, - callGraphProvider: cgProvider, cacheDir: o.cacheDir ? path.resolve(String(o.cacheDir)) : null, verbosity: typeof o.verbose === "number" ? o.verbose : 0, }; diff --git a/src/core.ts b/src/core.ts index 5355e86..8a781c7 100644 --- a/src/core.ts +++ b/src/core.ts @@ -1,6 +1,6 @@ import * as path from "node:path"; import { buildProgramGraphs, startExtraction } from "./dataflow"; -import { mergeCallGraphs, selectProvider } from "./semantic_analysis"; +import { type LinkerResolutions, mergeCallGraphs, runDefuseLinker, tscProvider } from "./semantic_analysis"; import { loadCache, saveCache } from "./utils"; import { materialize } from "./build"; import type { AnalysisOptions } from "./options"; @@ -42,31 +42,36 @@ export async function analyze(opts: AnalysisOptions): Promise { } const extraction = opts.analysisLevel >= 3 ? startExtraction(project, symbol_table, mat.tsConfigFilePath, opts, log) : null; - // Call graph via the selected provider (union of tsc+jelly by default; --tsc-only / jelly opt-in). - // Only worth running at level >= 2: finalizeAnalysis discards call_graph/external_symbols/ - // synthesized_callables at -a 1 (homeExternals/homeSynthesized in src/schema/emit.ts are - // gated to `level >= 2`), so running the solve — including the heavier Jelly leg — at -a 1 - // would compute a result that's thrown away. Levels 3/4 need the provider for callee - // resolution and are always >= 2, so this gate is safe. - // - // Run the provider PER PROGRAM (each with its own Project + its slice of callables via `only`), - // then merge the results the same way the union provider merges tsc∪jelly. Signature gating uses - // the full merged symbol_table (passed to every program), so a cross-program in-project call - // resolves. Single-program projects run the loop once — behavior is unchanged. - const provider = selectProvider(opts.callGraphProvider); - log.info(`call graph provider: ${provider.name}`); - let cg: ReturnType = { edges: [], external_symbols: {}, synthesized_callables: {} }; + // Call graph: the tsc resolver, per program (each with its own Project + its slice of callables + // via `only`), merged across programs. Only worth running at level >= 2: finalizeAnalysis + // discards call_graph/external_symbols/synthesized_callables at -a 1 (homeExternals/ + // homeSynthesized in src/schema/emit.ts are gated to `level >= 2`), so running the solve at + // -a 1 would compute a result that's thrown away. Levels 3/4 need it for callee resolution and + // are always >= 2, so this gate is safe. Signature gating uses the full merged symbol_table + // (passed to every program), so a cross-program in-project call resolves. + let cg: ReturnType = { edges: [], external_symbols: {}, synthesized_callables: {} }; + const resolutions: LinkerResolutions = new Map(); if (opts.analysisLevel >= 2) { for (const prog of programs) { - const pcg = provider.build({ + const ctx = { project: prog.project, symbol_table, root: opts.input, log, phantoms: opts.phantoms, only: prog.fileKeys, - }); - cg = mergeCallGraphs(cg, pcg); + }; + cg = mergeCallGraphs(cg, tscProvider.build(ctx)); + // The defuse linker overlays the tsc base: it reads the callee_signature backfill the tsc + // leg just wrote, resolves what remains (tiers T1–T5, defuseLinker.ts), and returns its + // body-node resolutions out-of-band (never persisted — cache provenance rule). + const linked = runDefuseLinker(ctx); + cg = mergeCallGraphs(cg, linked.result); + for (const [caller, m] of linked.resolutions) { + const ex = resolutions.get(caller); + if (!ex) resolutions.set(caller, m); + else for (const [k, v] of m) if (!ex.has(k)) ex.set(k, v); + } } } const call_graph = cg.edges; @@ -85,5 +90,5 @@ export async function analyze(opts: AnalysisOptions): Promise { // Cache the id-free base (ids/body/heritage are per-run layers stamped by finalizeAnalysis; // the cached tree must stay --app-name-free). saveCache(cacheDir, { symbol_table }); - return finalizeAnalysis(app, pg, opts); + return finalizeAnalysis(app, pg, opts, resolutions); } diff --git a/src/dataflow/attach.ts b/src/dataflow/attach.ts index ceea6a9..11f59b0 100644 --- a/src/dataflow/attach.ts +++ b/src/dataflow/attach.ts @@ -114,7 +114,7 @@ function emitL3(li: LocalIds, nodes: GraphNode[], cfgEdges: CfgEdge[] | undefine // prov = the def-use METHOD: `solveDefUse` computes forward may-reaching-definitions over // k-limited access paths with a flow-insensitive copy/field-alias substrate (defuse.ts). It // is NOT SSA and NOT points-to-oracle-backed — so we tag it "reaching-defs", not "ssa". - // A real points-to layer (Jelly, PR F) would emit additional edges tagged "points-to". + // A real points-to layer (PR F) would emit additional edges tagged "points-to". // "reaching-defs" is a SANCTIONED ADDITIVE prov token — a deliberate, documented deviation // from the shared cross-analyzer vocabulary's canonical "ssa" tag for the L3 syntactic DDG. // Recorded in `.claude/SCHEMA_DECISIONS.md` (issue #32); JSON, Neo4j (`ddg.prov: string[]`, diff --git a/src/dataflow/defuse.ts b/src/dataflow/defuse.ts index 2fc4a84..5dd18f2 100644 --- a/src/dataflow/defuse.ts +++ b/src/dataflow/defuse.ts @@ -18,7 +18,7 @@ * * Aliasing (MVP substrate, per issue #2 / SCHEMA_DECISIONS.md): flow-insensitive union-find over * bases connected by direct copies (`const q = p`); a write through one name weakly updates the - * other. Points-to-backed aliasing via Jelly's solved state is the staged upgrade (PR F). + * other. Points-to-backed aliasing is the staged upgrade (PR F). * * Def-use: classic forward may reaching-definitions. Strong (killing) defs are whole-base writes * to locals/params; every field write is weak. Captured/module/this bases get a synthetic def at diff --git a/src/main.ts b/src/main.ts index 40dd5c2..3a82795 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,26 +1,3 @@ #!/usr/bin/env node -/** - * Multi-call binary entry. The compiled `cants` executable bundles BOTH the analyzer and the - * `@cs-au-dk/jelly` CLI; this dispatcher picks which one runs based on argv: - * - * cants __jelly -> run the embedded Jelly CLI (used internally by jellyProvider) - * cants -> run the normal analyzer - * - * Both programs self-execute on import (analyzer's main() / Jelly's program.parse()), so dispatch is - * "reshape argv, then dynamically import the right module". Bun's --compile bundles both branches. - * The CANTS_SELF_JELLY marker tells jellyProvider it can re-exec THIS binary for Jelly instead of - * shelling out to `node`; it is intentionally unset in source/dev runs (where the dispatcher is - * bypassed and the provider falls back to `node @cs-au-dk/jelly/lib/main.js`). - */ -export {}; // mark as a module so top-level await is permitted - -const argv = process.argv; -if (argv[2] === "__jelly") { - // Jelly's commander reads process.argv as [node, script, ...args]; drop our "__jelly" sentinel. - process.argv = [argv[0], "jelly", ...argv.slice(3)]; - // @ts-ignore — @cs-au-dk/jelly ships no type declarations for the lib subpath - await import("@cs-au-dk/jelly/lib/main.js"); -} else { - process.env.CANTS_SELF_JELLY = process.execPath; - await import("./index"); -} +/** Binary entry — the analyzer CLI (self-executes on import). */ +import "./index"; diff --git a/src/options/options.ts b/src/options/options.ts index 80d10d8..6c3b350 100644 --- a/src/options/options.ts +++ b/src/options/options.ts @@ -1,8 +1,6 @@ import type { GraphSelector } from "../schema"; export type EmitTarget = "json" | "neo4j" | "schema"; -export type CallGraphProviderName = "union" | "tsc" | "jelly"; - /** Normalized analysis options (produced by the CLI layer, consumed by core). */ export interface AnalysisOptions { /** Project root to analyze (absolute). */ @@ -46,8 +44,6 @@ export interface AnalysisOptions { noBuild: boolean; /** Emit phantom (external) nodes/edges for imported/required library call targets. Default on. */ phantoms: boolean; - /** Call-graph backend: union of tsc+jelly (default), tsc resolver only (--tsc-only), or jelly. */ - callGraphProvider: CallGraphProviderName; /** Where caches/intermediate state live; null ⇒ /.codeanalyzer. */ cacheDir: string | null; /** Verbosity (repeatable -v). */ diff --git a/src/schema/assignIds.ts b/src/schema/assignIds.ts index e493c11..ced9379 100644 --- a/src/schema/assignIds.ts +++ b/src/schema/assignIds.ts @@ -54,6 +54,10 @@ export function assignIds(app: AnalysisInternal, appName: string): AssignedIds { const moduleId = moduleIdOf(appId, fileKey); const modulePrefix = modulePrefixOf(fileKey); mod.id = moduleId; + // Module-scope execution is a call-graph SOURCE (python #131 parity: a call in module scope + // is attributed to the MODULE). The prefix is the module's "signature", so those edges + // re-identify onto the module node's id instead of dangling. + register(modulePrefix, moduleId); doFields(moduleId, mod.fields); for (const fn of Object.values(mod.functions ?? {})) doCallable(moduleId, modulePrefix, fn); for (const t of Object.values(mod.types ?? {})) doType(moduleId, modulePrefix, t); diff --git a/src/schema/emit.ts b/src/schema/emit.ts index 4ba933e..84e7dfb 100644 --- a/src/schema/emit.ts +++ b/src/schema/emit.ts @@ -49,7 +49,12 @@ export interface AnalysisResult { dangling: string[]; // call-graph endpoints with no id home (L2 no-dangling gate; should be empty) } -export function finalizeAnalysis(app: AnalysisInternal, pg: ProgramGraphs | null, opts: AnalysisOptions): AnalysisResult { +export function finalizeAnalysis( + app: AnalysisInternal, + pg: ProgramGraphs | null, + opts: AnalysisOptions, + resolutions?: Map>, +): AnalysisResult { const level = opts.analysisLevel; const appName = (opts.appName ?? (opts.input ? path.basename(opts.input) : "") ?? "").trim() || "app"; @@ -65,7 +70,7 @@ export function finalizeAnalysis(app: AnalysisInternal, pg: ProgramGraphs | null if (level >= 2) { root.external_symbols = homeExternals(app, appId, idBySig); root.synthesized_callables = homeSynthesized(app, appId, idBySig); - backfillCallees(app, idBySig); + backfillCallees(app, idBySig, resolutions); root.call_graph = reidentifyCallGraph(app.call_graph ?? [], idBySig, dangling); } diff --git a/src/schema/l2Callees.ts b/src/schema/l2Callees.ts index 7ab3fa0..57b48ba 100644 --- a/src/schema/l2Callees.ts +++ b/src/schema/l2Callees.ts @@ -14,14 +14,23 @@ import type { AnalysisInternal, TSCallEdge, TSCallGraphEdge, TSModule } from "./ import { forEachCallable } from "./schema"; import { callBodyKeys } from "./l1Body"; -export function backfillCallees(app: AnalysisInternal, idBySig: Map): void { +export function backfillCallees( + app: AnalysisInternal, + idBySig: Map, + resolutions?: Map>, +): void { for (const mod of Object.values(app.symbol_table) as TSModule[]) { forEachCallable(mod, (c) => { + const linked = resolutions?.get(c.signature); for (const [key, cs] of callBodyKeys(c.call_sites)) { - if (!cs.callee_signature) continue; + // The resolver's in-place backfill wins; the linker's returned map fills the gaps. Linker + // resolutions are deliberately NOT persisted into callee_signature (cache provenance rule + // — see defuseLinker.ts header). + const sig = cs.callee_signature ?? linked?.get(key); + if (!sig) continue; const node = c.body[key]; if (!node || node.kind !== "call") continue; - node.callee = idBySig.get(cs.callee_signature) ?? null; + node.callee = idBySig.get(sig) ?? null; } }); } diff --git a/src/schema/schema.ts b/src/schema/schema.ts index 7587bb0..e6f32aa 100644 --- a/src/schema/schema.ts +++ b/src/schema/schema.ts @@ -354,10 +354,10 @@ export interface TSExternalSymbol { module: string; // the import/require specifier, e.g. "node:fs", "express", "@scope/pkg" } -// A first-party anonymous callback that Jelly resolves as a call-graph endpoint but the symbol -// table never names (the canonicalizer returns null for anonymous functions). The map key IS the -// synthesized signature `:`, so an edge `source`/ -// `target` byte-matches it just like a real `Callable.signature` or `TSExternalSymbol.signature`. +// A first-party anonymous callback a call-graph builder resolved as an edge endpoint but could +// not name against the symbol table (a residual-fallback safety net; since 2.1.0 the tree names +// anonymous callables positionally, so this map is normally empty). The map key IS the +// synthesized signature, so an edge `source`/`target` byte-matches it like a real signature. export interface TSSynthesizedCallable { name: string; // display name — always ""; the signature carries the precise identity path: string; // owning module key (project-relative POSIX path WITH extension) @@ -417,7 +417,7 @@ export interface TSApplication { export interface TSCallGraphEdge { src: string; dst: string; - prov: string[]; // provenance, e.g. ["tsc"], ["jelly"] + prov: string[]; // provenance, e.g. ["tsc"], ["defuse"], ["import"] weight: number; } diff --git a/src/schema/signatures.ts b/src/schema/signatures.ts index e228fdf..1c2b486 100644 --- a/src/schema/signatures.ts +++ b/src/schema/signatures.ts @@ -29,7 +29,7 @@ export function contributorName(node: Node): string | null { /** * The segment an unnamed function-like node contributes. Position is the only discriminant a - * nameless callable has, and it is the one both the resolver and Jelly can compute independently + * nameless callable has, and it is the one every call-graph builder can compute independently * — which is what keeps caller-side and callee-side ids byte-identical. Angle brackets mark the * segment synthetic (the ``/`` convention). It joins the dotted chain, so an * anonymous callable lives in the durable id tier and never collides with the `@line:col` @@ -82,10 +82,10 @@ export function computeSignatureForDecl(node: Node, root: string): string | null return signatureOf(modulePrefix, ...parts); } -/** Resolve the declaration a call/new expression targets, following import aliases. */ +/** Resolve the declaration a call/new/tagged-template expression targets, following import aliases. */ export function resolveCalleeDecl(call: Node): Node | undefined { - if (!Node.isCallExpression(call) && !Node.isNewExpression(call)) return undefined; - const expr = call.getExpression(); + if (!Node.isCallExpression(call) && !Node.isNewExpression(call) && !Node.isTaggedTemplateExpression(call)) return undefined; + const expr = Node.isTaggedTemplateExpression(call) ? call.getTag() : call.getExpression(); let symNode: Node = expr; if (Node.isPropertyAccessExpression(expr)) symNode = expr.getNameNode(); else if (Node.isElementAccessExpression(expr)) return undefined; // dynamic dispatch — best-effort skip diff --git a/src/semantic_analysis/callGraph.ts b/src/semantic_analysis/callGraph.ts index e20ea08..dd2a37f 100644 --- a/src/semantic_analysis/callGraph.ts +++ b/src/semantic_analysis/callGraph.ts @@ -20,7 +20,24 @@ import { type TSType, forEachCallable, } from "../schema"; -import { resolveCalleeSignature } from "../schema"; +import { fileKeyOf, resolveCalleeSignature } from "../schema"; +import { isCallableDecl } from "../schema"; + +/** The nearest ancestor that is itself a callable declaration (incl. `const f = () => …`), or undefined. */ +function enclosingCallable(node: Node): Node | undefined { + for (const a of node.getAncestors()) { + if (isCallableDecl(a)) return a; + if (Node.isVariableDeclaration(a)) { + const init = a.getInitializer?.(); + if (init && (Node.isArrowFunction(init) || Node.isFunctionExpression(init))) return a; + } + } + return undefined; +} + +function fileKeyOfNode(node: Node, root: string): { fileKey: string; modulePrefix: string } { + return fileKeyOf(node.getSourceFile().getFilePath(), root); +} import type { Logger } from "../utils"; import { type ExternalIndex, buildExternalIndex, resolvePhantom } from "./phantoms"; @@ -33,7 +50,7 @@ export interface CallGraphResult { edges: TSCallEdge[]; external_symbols: Record; // Anonymous callbacks resolved as edge endpoints that the symbol table doesn't name. Empty for - // the tsc resolver (its edges are gated to real symbol-table signatures); populated by Jelly. + // the tsc resolver (its edges are gated to real symbol-table signatures); a residual-fallback net. synthesized_callables: Record; } @@ -130,6 +147,42 @@ export function buildCallGraph( let rtaCount = 0; let phantomCount = 0; let unresolved = 0; + + // Module-scope sweep (python #131 parity): a call with NO enclosing callable — top-level + // statements, class property initializers, namespace bodies — is attributed to the MODULE + // (source = the module prefix, re-identified onto the module node at L2). These sites are + // never recorded in call_sites (modules have no body{}), so resolve them straight off the AST. + for (const node of callExprIndex.values()) { + if (enclosingCallable(node)) continue; + const fileKey = fileKeyOfNode(node, root); + if (only && !only.has(fileKey.fileKey)) continue; + const source = fileKey.modulePrefix; + const r = resolveCalleeSignature(node, root, allSignatures); + if (r?.external) { + if (phantoms) { + if (!external_symbols[r.signature]) external_symbols[r.signature] = { name: r.external.member, module: r.external.module }; + addPhantomEdge(source, r.signature, r.external.module); + phantomCount++; + } else unresolved++; + continue; + } + if (!r) { + if (phantoms) { + const ph = resolvePhantom(node, extIndexFor(node)); + if (ph) { + if (!external_symbols[ph.signature]) external_symbols[ph.signature] = { name: ph.member, module: ph.module }; + addPhantomEdge(source, ph.signature, ph.module); + phantomCount++; + continue; + } + } + unresolved++; + continue; + } + addEdge(source, r.signature, false); + resolved++; + } + for (const caller of callables) { for (const site of caller.call_sites) { const node = callExprIndex.get( @@ -252,13 +305,13 @@ function indexClasses( } } -function indexCallExpressions(project: Project): Map { +export function indexCallExpressions(project: Project): Map { const idx = new Map(); for (const sf of project.getSourceFiles()) { const fp = sf.getFilePath(); if (sf.isDeclarationFile() || fp.includes("/node_modules/")) continue; sf.forEachDescendant((n) => { - if (Node.isCallExpression(n) || Node.isNewExpression(n)) { + if (Node.isCallExpression(n) || Node.isNewExpression(n) || Node.isTaggedTemplateExpression(n)) { const s = sf.getLineAndColumnAtPos(n.getStart()); const e = sf.getLineAndColumnAtPos(n.getEnd()); // Full span (start AND end) keys the node uniquely; chained calls like `f(x).g(y)` diff --git a/src/semantic_analysis/defuseLinker.ts b/src/semantic_analysis/defuseLinker.ts new file mode 100644 index 0000000..28295a3 --- /dev/null +++ b/src/semantic_analysis/defuseLinker.ts @@ -0,0 +1,528 @@ +/** + * The defuse linker — the local pass that backfills call edges the tsc resolver missed + * (docs/design/specs/defuse-linker-call-graph.md, #98; python parity: the Jedi + defuse-linker + * architecture that replaced PyCG). Per-callable and bounded-round only — NO whole-program + * fixpoint; sorted iteration throughout, so the output is deterministic by construction. + * + * Tiers, applied in order (a precise resolution is never widened by a later tier): + * T1 local value chase — alias chains `const f = handler; f()` through bounded + * symbol→declaration hops (the checker's alias-following covers imports). + * T2 decorator invocations — `@Get(':id')` on a method/accessor becomes an edge + * decorated-callable → decorator target, EDGE-ONLY (decorator calls live outside walkBody's + * reach, so there is no body call node to refine — matching the historical Jelly shape). + * T3 external-callback rule — a function value passed to an external/unresolved callee emits + * enclosing-callable → function-value, EDGE-ONLY (`.map(u => …)`, `app.get('/x', handler)`). + * Deliberate divergence from python (JS is callback-central); recorded in the spec. + * T4 bounded interprocedural votes — (a) a parameter-invoking site (`cb()`) resolves to the + * function values passed at that position by resolved-internal callers (two rounds: round + * one's resolutions vote before round two); (b) `const f = factory(); f()` resolves through + * the factory's unique returned function. + * T5 CHA-by-name — receiver sites that survive every typed tier resolve to every internal + * callable of that method name (bounded per site) — the over-approximation Joern emits for + * untyped receivers. Edge-only (ambiguous by definition). + * + * Linker resolutions are returned in a map and applied to the L1 `call` body nodes by + * `backfillCallees` — NEVER written into `callee_signature` (the symbol table round-trips the + * analysis cache; a persisted resolution would resurface on a warm run with tsc provenance). + */ +import { Node, SyntaxKind } from "ts-morph"; +import { CALL_DEP, type TSCallEdge, type TSCallable, type TSCallsite, type TSExternalSymbol, forEachCallable } from "../schema"; +import { computeSignatureForDecl, externalHomeOf, fileKeyOf, isCallableDecl, resolveCalleeSignature } from "../schema"; +import { callBodyKeys } from "../schema/l1Body"; +import type { CallGraphContext } from "./provider"; +import type { CallGraphResult } from "./callGraph"; +import { indexCallExpressions } from "./callGraph"; + +/** Per-call-site resolutions for the sanctioned `callee: null→id` refinement: callerSig → bodyKey → calleeSig. */ +export type LinkerResolutions = Map>; + +export interface LinkerOutput { + result: CallGraphResult; + resolutions: LinkerResolutions; +} + +// ponytail: fixed small bounds; tune from the Joern ledger, not from flags (spec: no backend flag). +const ALIAS_CHASE_LIMIT = 8; // hops through `const f = g` chains +const CHA_FAN_LIMIT = 16; // max name-matched targets per T5 site + +export function runDefuseLinker(ctx: CallGraphContext): LinkerOutput { + const { project, symbol_table, root, log } = ctx; + + // The signature universe (full table — cross-program targets resolve) + the name→sigs CHA index. + const allSignatures = new Set(); + const byName = new Map(); + for (const mod of Object.values(symbol_table)) { + forEachCallable(mod, (c) => { + allSignatures.add(c.signature); + const arr = byName.get(c.name) ?? []; + arr.push(c.signature); + byName.set(c.name, arr); + }); + } + for (const sigs of byName.values()) sigs.sort(); + + // Callables to iterate: this program's modules only, sorted for determinism. + const callables: TSCallable[] = []; + for (const [key, mod] of Object.entries(symbol_table)) { + if (ctx.only && !ctx.only.has(key)) continue; + forEachCallable(mod, (c) => callables.push(c)); + } + callables.sort((a, b) => a.signature.localeCompare(b.signature)); + + const callExprIndex = indexCallExpressions(project); + /** The callee expression of a call/new/tagged-template node. */ + const calleeExprOf = (node: Node): Node => + Node.isTaggedTemplateExpression(node) ? node.getTag() : (node as unknown as { getExpression: () => Node }).getExpression(); + const nodeOf = (c: TSCallable, cs: TSCallsite): Node | undefined => + callExprIndex.get(`${c.abs_path}#${cs.start_line}:${cs.start_column}-${cs.end_line}:${cs.end_column}`); + + // A resolved-but-external callee is one whose signature is not in the symbol table. + const isExternalSig = (sig: string | undefined): boolean => !!sig && !allSignatures.has(sig); + + // --------------------------------------------------------------------------------------------- + // edge/resolution accumulation + // --------------------------------------------------------------------------------------------- + const edges = new Map(); + const addEdge = (source: string, target: string): void => { + const k = `${source} ${target}`; + const ex = edges.get(k); + if (ex) ex.weight++; + else edges.set(k, { source, target, type: CALL_DEP, weight: 1, provenance: ["defuse"], tags: {} }); + }; + const external_symbols: Record = {}; + const resolutions: LinkerResolutions = new Map(); + const resolve = (callerSig: string, bodyKey: string, targetSig: string): void => { + addEdge(callerSig, targetSig); + let m = resolutions.get(callerSig); + if (!m) resolutions.set(callerSig, (m = new Map())); + m.set(bodyKey, targetSig); + }; + + /** + * The signature of the first-party callable a VALUE expression denotes, chasing bounded alias + * chains: a bare arrow/function expression, an identifier for a function declaration, a + * `const f = () => …` binding, or `const f = g` (g eventually a function) — else null. + */ + const functionValueSig = (expr: Node): string | null => { + // IIFE / parenthesized function values: `(() => …)()`, `(function f() {})()`. + while (Node.isParenthesizedExpression(expr)) expr = expr.getExpression(); + if (Node.isArrowFunction(expr) || Node.isFunctionExpression(expr)) { + const s = computeSignatureForDecl(expr, root); + return s && allSignatures.has(s) ? s : null; + } + if (!Node.isIdentifier(expr)) return null; + let node: Node = expr; + for (let hop = 0; hop < ALIAS_CHASE_LIMIT; hop++) { + let sym = node.getSymbol(); + if (!sym) return null; + const aliased = sym.getAliasedSymbol(); + if (aliased) sym = aliased; + const decl = sym.getDeclarations()?.[0]; + if (!decl) return null; + if (Node.isFunctionDeclaration(decl) || Node.isArrowFunction(decl) || Node.isFunctionExpression(decl) || Node.isMethodDeclaration(decl)) { + const s = computeSignatureForDecl(decl, root); + return s && allSignatures.has(s) ? s : null; + } + if (Node.isVariableDeclaration(decl)) { + const init = decl.getInitializer(); + if (!init) return null; + if (Node.isArrowFunction(init) || Node.isFunctionExpression(init)) { + const s = computeSignatureForDecl(decl, root); + return s && allSignatures.has(s) ? s : null; + } + if (Node.isIdentifier(init)) { + node = init; // alias chain: keep chasing + continue; + } + return null; + } + return null; + } + return null; + }; + + /** The parameter index of `expr` within `enclosing`, when it names one of its parameters. */ + const paramIndexOf = (expr: Node, enclosing: TSCallable): number | null => { + if (!Node.isIdentifier(expr)) return null; + const decl = expr.getSymbol()?.getDeclarations()?.[0]; + if (!decl || !Node.isParameterDeclaration(decl)) return null; + const name = expr.getText(); + const idx = enclosing.parameters.findIndex((p) => p.name === name); + return idx >= 0 ? idx : null; + }; + + // --------------------------------------------------------------------------------------------- + // main site sweep: T1 chase, T3 callback rule, and the T4/T5 worklists + // --------------------------------------------------------------------------------------------- + interface ParamSite { + enclosing: TSCallable; + bodyKey: string; + paramIndex: number; + } + interface FactorySite { + enclosing: TSCallable; + bodyKey: string; + factorySig: string; // resolved-internal callee of the binding's initializer call + } + interface ReceiverSite { + enclosing: TSCallable; + cs: TSCallsite; + } + interface ThisFieldSite { + enclosing: TSCallable; + bodyKey: string; + node: Node; + fieldName: string; + } + const paramSites: ParamSite[] = []; + const factorySites: FactorySite[] = []; + const receiverSites: ReceiverSite[] = []; + const thisFieldSites: ThisFieldSite[] = []; + // Reverse index for T4 voting: internal target sig → the AST argument lists of its call sites. + const argsByTarget = new Map(); + const recordCallArgs = (targetSig: string, node: Node | undefined): void => { + if (!node || !allSignatures.has(targetSig)) return; + const args = (node as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; + if (!args.length) return; + const arr = argsByTarget.get(targetSig) ?? []; + arr.push(args); + argsByTarget.set(targetSig, arr); + }; + + let t1 = 0; + let t3 = 0; + for (const c of callables) { + for (const [bodyKey, cs] of callBodyKeys(c.call_sites)) { + const node = nodeOf(c, cs); + if (cs.callee_signature) { + recordCallArgs(cs.callee_signature, node); + } else if (node) { + const expr = calleeExprOf(node); + // T1 — local value chase on the callee expression itself. + const chased = functionValueSig(expr); + if (chased) { + resolve(c.signature, bodyKey, chased); + recordCallArgs(chased, node); + t1++; + } else { + const pIdx = paramIndexOf(expr, c); + if (pIdx !== null) { + paramSites.push({ enclosing: c, bodyKey, paramIndex: pIdx }); + } else if (Node.isIdentifier(expr)) { + // T4b — `const f = factory(); f()`: binding initialized by a resolved-internal call. + const decl = expr.getSymbol()?.getDeclarations()?.[0]; + const init = decl && Node.isVariableDeclaration(decl) ? decl.getInitializer() : undefined; + if (init && Node.isCallExpression(init)) { + const r = resolveCalleeSignature(init, root, allSignatures); + if (r && !r.external && allSignatures.has(r.signature)) { + factorySites.push({ enclosing: c, bodyKey, factorySig: r.signature }); + } + } + } else if (Node.isPropertyAccessExpression(expr) && cs.receiver_expr != null && !cs.is_constructor_call) { + if (cs.receiver_expr === "this") { + // T4c — `this.field(...)`: the field's value flows from the constructor (a + // parameter property or a `this.field = …` assignment); resolved below, feeding + // the T4 vote rounds. Falls back to T5 if the chain yields nothing. + thisFieldSites.push({ enclosing: c, bodyKey, node, fieldName: cs.method_name }); + } else { + receiverSites.push({ enclosing: c, cs }); + } + } + } + } + // T3 — external-callback rule: function values handed to an external/unresolved callee. + if (node && (!cs.callee_signature || isExternalSig(cs.callee_signature))) { + const args = (node as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; + for (const arg of args) { + const fn = functionValueSig(arg); + if (fn && fn !== c.signature) { + addEdge(c.signature, fn); + t3++; + } + } + } + } + } + + // --------------------------------------------------------------------------------------------- + // Module-scope sweep (python #131 parity: module-scope execution is attributed to the MODULE). + // These sites have no call_sites record and no body node — T1 chase and the T3 callback rule + // apply edge-only, with the module prefix as the source. + // --------------------------------------------------------------------------------------------- + const enclosingCallable = (node: Node): Node | undefined => { + for (const a of node.getAncestors()) if (isCallableDecl(a)) return a; + return undefined; + }; + for (const [, node] of [...callExprIndex.entries()].sort(([a], [b]) => a.localeCompare(b))) { + if (enclosingCallable(node)) continue; + const fk = fileKeyOf(node.getSourceFile().getFilePath(), root); + if (ctx.only && !ctx.only.has(fk.fileKey)) continue; + const source = fk.modulePrefix; + const r = resolveCalleeSignature(node, root, allSignatures); + if (r && !r.external) recordCallArgs(r.signature, node); // top-level `register("a", cb)` feeds the vote rounds + if (!r) { + // T1 at module scope: `const f = handler; f()` in top-level code. + const expr = calleeExprOf(node); + const chased = functionValueSig(expr); + if (chased) { + addEdge(source, chased); + t1++; + } + } + // T3 at module scope — the dominant express idiom: `app.get('/x', handler)` top-level. + if (!r || r.external) { + const args = (node as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; + for (const arg of args) { + const fn = functionValueSig(arg); + if (fn && fn !== source) { + addEdge(source, fn); + t3++; + } + } + } + } + + // --------------------------------------------------------------------------------------------- + // T2 — decorator invocations (edge-only). The SOURCE is where the decorator executes: the + // decorated callable for method/accessor/parameter decorators; the MODULE for class and + // property decorators (a decorator on a top-level definition runs in module scope — python + // #131's rule, and Joern's own attribution). + // --------------------------------------------------------------------------------------------- + let t2 = 0; + const files = [...project.getSourceFiles()] + .filter((sf) => !sf.isDeclarationFile() && !sf.getFilePath().includes("/node_modules/")) + .sort((a, b) => a.getFilePath().localeCompare(b.getFilePath())); + for (const sf of files) { + sf.forEachDescendant((n) => { + if (!Node.isDecorator(n)) return; + // The edge SOURCE is where the decorator executes: the decorated callable for method/ + // accessor/parameter decorators; the MODULE prefix for class and property decorators. + let owner = n.getParent(); + if (owner && Node.isParameterDeclaration(owner)) owner = owner.getParent(); + let ownerSig: string | null = null; + if (owner && (Node.isMethodDeclaration(owner) || Node.isGetAccessorDeclaration(owner) || Node.isSetAccessorDeclaration(owner))) { + ownerSig = computeSignatureForDecl(owner, root); + if (!ownerSig || !allSignatures.has(ownerSig)) return; + } else if (owner && (Node.isClassDeclaration(owner) || Node.isPropertyDeclaration(owner))) { + ownerSig = fileKeyOf(sf.getFilePath(), root).modulePrefix; + } else { + return; + } + const expr = n.getExpression(); + let targetSig: string | null = null; + let external: { module: string; member: string } | null = null; + if (Node.isCallExpression(expr)) { + const r = resolveCalleeSignature(expr, root, allSignatures); + if (r) { + targetSig = r.signature; + external = r.external ?? null; + } + } else if (Node.isIdentifier(expr)) { + const direct = functionValueSig(expr); + if (direct) targetSig = direct; + else { + const decl = expr.getSymbol()?.getAliasedSymbol()?.getDeclarations()?.[0] ?? expr.getSymbol()?.getDeclarations()?.[0]; + const home = decl ? externalHomeOf(decl) : null; + if (home) { + const member = expr.getText(); + targetSig = `${home.module}.${member}`; + external = { module: home.module, member }; + } + } + } + if (!targetSig) return; + if (external) { + if (!ctx.phantoms) return; + if (!external_symbols[targetSig]) external_symbols[targetSig] = { name: external.member, module: external.module }; + } + addEdge(ownerSig, targetSig); + t2++; + }); + } + + // --------------------------------------------------------------------------------------------- + // T4c — `this.field(...)` through the constructor: a field assigned from a ctor parameter + // (parameter property or `this.f = param`) calls whatever function values the class's `new` + // sites passed at that position; a field assigned a function value in the ctor calls it + // directly. Candidates feed argsByTarget so the T4 rounds resolve the callbacks' OWN + // param-invoking sites (`write()` inside a registered migration callback). Bounded: direct + // ctor args only, no transitive flow. + // --------------------------------------------------------------------------------------------- + const classFieldSources = new Map>(); + const fieldSourcesOf = (cls: Node): Map => { + let m = classFieldSources.get(cls); + if (m) return m; + m = new Map(); + const ctor = (cls as unknown as { getConstructors?: () => Node[] }).getConstructors?.()?.[0]; + if (ctor) { + const params = (ctor as unknown as { getParameters: () => Node[] }).getParameters(); + params.forEach((p, i) => { + const pp = p as unknown as { getName: () => string; getModifiers?: () => Node[] }; + if ((pp.getModifiers?.() ?? []).length) m?.set(pp.getName(), { paramIndex: i }); + }); + const paramNames = new Map(params.map((p, i) => [(p as unknown as { getName: () => string }).getName(), i])); + ctor.forEachDescendant((d) => { + if (!Node.isBinaryExpression(d) || d.getOperatorToken().getKind() !== SyntaxKind.EqualsToken) return; + const lhs = d.getLeft(); + if (!Node.isPropertyAccessExpression(lhs) || lhs.getExpression().getKind() !== SyntaxKind.ThisKeyword) return; + const rhs = d.getRight(); + const idx = Node.isIdentifier(rhs) ? paramNames.get(rhs.getText()) : undefined; + if (idx !== undefined) m?.set(lhs.getName(), { paramIndex: idx }); + else { + const direct = functionValueSig(rhs); + if (direct) m?.set(lhs.getName(), { direct }); + } + }); + } + classFieldSources.set(cls, m); + return m; + }; + /** Function values an ARGUMENT node denotes — directly, or through one bounded parameter hop: + * when the arg is a parameter of the function containing the call, the values passed for that + * parameter at ITS resolved call sites are the candidates (`register(key, migrate)` → + * `new Migration(key, migrate)`). One hop, no fixpoint. */ + const argFlowCandidates = (arg: Node): string[] => { + const direct = functionValueSig(arg); + if (direct) return [direct]; + if (!Node.isIdentifier(arg)) return []; + const decl = arg.getSymbol()?.getDeclarations()?.[0]; + if (!decl || !Node.isParameterDeclaration(decl)) return []; + const owner = decl.getParent(); + if (!owner) return []; + const ownerSig = computeSignatureForDecl(owner, root); + if (!ownerSig) return []; + const idx = ((owner as unknown as { getParameters?: () => Node[] }).getParameters?.() ?? []).findIndex((p) => p === decl); + if (idx < 0) return []; + const out = new Set(); + for (const args of argsByTarget.get(ownerSig) ?? []) { + const a = args[idx]; + const fn = a ? functionValueSig(a) : null; + if (fn) out.add(fn); + } + return [...out].sort(); + }; + let t4c = 0; + for (const site of thisFieldSites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature) || a.bodyKey.localeCompare(b.bodyKey))) { + const cls = site.node.getAncestors().find((a) => Node.isClassDeclaration(a) || Node.isClassExpression(a)); + const src = cls ? fieldSourcesOf(cls).get(site.fieldName) : undefined; + const candidates = new Set(); + if (src?.direct) candidates.add(src.direct); + if (src?.paramIndex !== undefined && cls) { + const clsSig = computeSignatureForDecl(cls, root); + for (const args of argsByTarget.get(`${clsSig}.constructor`) ?? []) { + const arg = args[src.paramIndex]; + for (const fn of arg ? argFlowCandidates(arg) : []) candidates.add(fn); + } + } + if (!candidates.size) { + const cs = site.enclosing.call_sites.find((c2) => `${c2.start_line}:${c2.start_column}` === site.bodyKey.split("/")[0]); + if (cs) receiverSites.push({ enclosing: site.enclosing, cs }); // fall back to T5 + continue; + } + const sorted = [...candidates].sort(); + for (const target of sorted) { + addEdge(site.enclosing.signature, target); + recordCallArgs(target, site.node); // the callbacks' own param sites resolve in the T4 rounds + t4c++; + } + if (sorted.length === 1) { + let m = resolutions.get(site.enclosing.signature); + if (!m) resolutions.set(site.enclosing.signature, (m = new Map())); + m.set(site.bodyKey, sorted[0] as string); + } + } + + // --------------------------------------------------------------------------------------------- + // T4 — bounded votes, two rounds (round one's resolutions vote before round two). + // --------------------------------------------------------------------------------------------- + let t4 = 0; + for (let round = 0; round < 2 && paramSites.length; round++) { + const unresolvedNext: ParamSite[] = []; + for (const site of paramSites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature) || a.bodyKey.localeCompare(b.bodyKey))) { + const candidates = new Set(); + for (const args of argsByTarget.get(site.enclosing.signature) ?? []) { + const arg = args[site.paramIndex]; + if (!arg) continue; + const fn = functionValueSig(arg); + if (fn) candidates.add(fn); + } + if (!candidates.size) { + unresolvedNext.push(site); + continue; + } + const sorted = [...candidates].sort(); + for (const target of sorted) { + addEdge(site.enclosing.signature, target); + t4++; + // Round-one resolutions feed round two's votes: a cb() site that now targets `target` + // makes the enclosing callable a resolved-internal caller of it. + } + if (sorted.length === 1) { + let m = resolutions.get(site.enclosing.signature); + if (!m) resolutions.set(site.enclosing.signature, (m = new Map())); + m.set(site.bodyKey, sorted[0] as string); + } + } + paramSites.length = 0; + paramSites.push(...unresolvedNext); + } + // T4b — factory returns: resolve through the factory's unique returned function value. + const returnSummary = new Map(); + const uniqueReturnedFn = (factorySig: string): string | null => { + if (returnSummary.has(factorySig)) return returnSummary.get(factorySig) as string | null; + let out: string | null = null; + // Find the factory's AST via any recorded call-site node? Cheaper: search the sorted callables + // list (same program) for the signature, then its declaration through the call-expression + // index is unavailable — walk the source file at its span instead. + const fc = callables.find((c) => c.signature === factorySig); + if (fc) { + const sf = project.getSourceFile(fc.abs_path); + const declNode = sf?.getDescendantAtPos(fc.span.bytes[0]); + const fnNode = declNode ? [declNode, ...declNode.getAncestors()].find((a) => computeSignatureForDecl(a, root) === factorySig) : undefined; + if (fnNode) { + const returned = new Set(); + fnNode.forEachDescendant((d) => { + if (!Node.isReturnStatement(d)) return; + const e = d.getExpression(); + if (!e) return; + const fn = functionValueSig(e); + if (fn) returned.add(fn); + else returned.add(""); + }); + if (returned.size === 1 && !returned.has("")) out = [...returned][0] as string; + } + } + returnSummary.set(factorySig, out); + return out; + }; + for (const site of factorySites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature) || a.bodyKey.localeCompare(b.bodyKey))) { + const target = uniqueReturnedFn(site.factorySig); + if (target) { + resolve(site.enclosing.signature, site.bodyKey, target); + t4++; + } + } + + // --------------------------------------------------------------------------------------------- + // T5 — CHA-by-name fallback (edge-only, bounded fan). + // --------------------------------------------------------------------------------------------- + let t5 = 0; + for (const site of receiverSites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature))) { + if (!site.cs) continue; + const candidates = (byName.get(site.cs.method_name) ?? []).filter((s) => s !== site.enclosing.signature); + // Over-cap names (get/set/toString-class fan) are skipped outright, not truncated — a partial + // arbitrary subset would be neither sound-leaning nor deterministic in meaning. + if (!candidates.length || candidates.length > CHA_FAN_LIMIT) continue; + for (const target of candidates) { + addEdge(site.enclosing.signature, target); + t5++; + } + } + + const sortedEdges = [...edges.values()].sort((a, b) => a.source.localeCompare(b.source) || a.target.localeCompare(b.target)); + log.info(`call graph (defuse): ${sortedEdges.length} edges — t1=${t1} chase, t2=${t2} decorator, t3=${t3} callback, t4=${t4} votes, t4c=${t4c} ctor-field, t5=${t5} cha`); + return { + result: { edges: sortedEdges, external_symbols, synthesized_callables: {} }, + resolutions, + }; +} diff --git a/src/semantic_analysis/index.ts b/src/semantic_analysis/index.ts index 76df475..edc5d5b 100644 --- a/src/semantic_analysis/index.ts +++ b/src/semantic_analysis/index.ts @@ -1,5 +1,4 @@ -// Call-graph construction: the tsc (ts-morph checker) resolver graph + RTA. +// Call-graph construction: the tsc (ts-morph checker) resolver graph + RTA + the defuse linker. export * from "./callGraph"; -// The provider seam (union | tsc | jelly) + the Jelly backend. export * from "./provider"; -export * from "./jellyProvider"; +export * from "./defuseLinker"; diff --git a/src/semantic_analysis/jellyProvider.ts b/src/semantic_analysis/jellyProvider.ts deleted file mode 100644 index d7f6af7..0000000 --- a/src/semantic_analysis/jellyProvider.ts +++ /dev/null @@ -1,397 +0,0 @@ -/** - * Jelly call-graph provider. Shells out to `@cs-au-dk/jelly` (CLI/JSON only — no library API), - * then maps each Jelly function node back onto a symbol-table signature by source span: - * - * jelly id "fileIdx:sl:sc:el:ec" -> files[fileIdx] + (sl,sc) -> ts-morph node -> signature - * - * Named declarations reuse the existing canonicalizer (computeSignatureForDecl); anonymous inline - * callbacks — which the canonicalizer returns null for — get a SYNTHESIZED signature of the form - * `:`, mirroring how Jelly itself identifies - * anonymous functions purely by location. Jelly's columns are 1-based (it exports column+1), which - * lines up with ts-morph's 1-based columns, so (file,startLine,startColumn) is a direct join key. - * - * This provider is read-only w.r.t. the symbol table: it emits edges over its own node universe - * (real signatures ∪ synthesized) for diffing. Materializing synthesized callables into the symbol - * table is a later step, only needed when jelly is promoted to authoritative. - * - * Whole-program, scoped to declared deps: Jelly follows imports into node_modules, but we exclude - * every installed package NOT listed in the project's package.json `dependencies`. This keeps the - * directly-used library surface (the deps that actually matter for edge resolution) while cutting - * transitive bloat — critically `@ts-morph/common`, which bundles the entire ~8.7MB TypeScript - * compiler and makes unbounded whole-program analysis OOM. - * - * Tier 2 — dependency functions are materialized as external symbols so edges crossing the - * first-party↔library boundary are KEPT in both directions, tagged `ts.external`/`ts.module` (like - * the tsc phantom mechanism). Keying differs by direction because Jelly carries no function names: - * • first-party → dep: keyed `module.member` via the first-party call site (resolvePhantom), - * matching the tsc phantom keys so the two providers' external symbols are comparable. - * • dep → first-party: no first-party call site to read, so keyed by package + location. This is - * the entrypoint signal — a framework invoking your handler. - * Only dep→dep edges (neither endpoint first-party) are dropped. - */ -import { execFileSync } from "node:child_process"; -import * as fs from "node:fs"; -import { createRequire } from "node:module"; -import * as os from "node:os"; -import * as path from "node:path"; -import { Node, type SourceFile } from "ts-morph"; -import { - CALL_DEP, - computeSignatureForDecl, - fileKeyOf, - type TSCallEdge, - type TSExternalSymbol, - type TSSynthesizedCallable, -} from "../schema"; -import type { CallGraphResult } from "./callGraph"; -import { type ExternalIndex, buildExternalIndex, resolvePhantom } from "./phantoms"; -import type { CallGraphContext, CallGraphProvider } from "./provider"; - -const requireFrom = createRequire(import.meta.url); - -interface JellyJson { - files: string[]; - functions: Record; // id -> "fileIdx:startLine:startCol:endLine:endCol" - fun2fun: [number, number][]; // [callerId, calleeId] - call2fun: [number, number][]; // [callSiteId, calleeId] - calls: Record; // callSiteId -> "fileIdx:sl:sc:el:ec" (span in the CALLER file) -} - -/** Locate Jelly's entry script: explicit override, else the installed package. */ -function resolveJellyMain(): string { - if (process.env.JELLY_BIN) return process.env.JELLY_BIN; - try { - return requireFrom.resolve("@cs-au-dk/jelly/lib/main.js"); - } catch { - throw new Error("@cs-au-dk/jelly not installed and JELLY_BIN unset"); - } -} - -/** The project's declared runtime dependencies — the packages we want Jelly to descend into. */ -function declaredDeps(root: string): Set { - try { - const pkg = JSON.parse(fs.readFileSync(path.join(root, "package.json"), "utf8")) as { - dependencies?: Record; - }; - return new Set(Object.keys(pkg.dependencies ?? {})); - } catch { - return new Set(); - } -} - -/** Every installed package name under node_modules (descending one level into @scope dirs). */ -function installedPackages(root: string): string[] { - const nm = path.join(root, "node_modules"); - let top: string[]; - try { - top = fs.readdirSync(nm); - } catch { - return []; - } - const out: string[] = []; - for (const e of top) { - if (e.startsWith(".")) continue; - if (e.startsWith("@")) { - try { - for (const sub of fs.readdirSync(path.join(nm, e))) if (!sub.startsWith(".")) out.push(`${e}/${sub}`); - } catch { - /* unreadable scope dir */ - } - } else { - out.push(e); - } - } - return out; -} - -/** Installed packages NOT declared as dependencies — excluded so whole-program stays tractable. */ -function excludedPackages(root: string): string[] { - const keep = declaredDeps(root); - return installedPackages(root).filter((p) => !keep.has(p)); -} - -function runJelly(ctx: CallGraphContext, entryFiles: string[]): JellyJson { - const out = path.join(os.tmpdir(), `cants-jelly-${process.pid}.json`); - const excluded = excludedPackages(ctx.root); - ctx.log.debug(`call graph (jelly): excluding ${excluded.length} non-declared packages from whole-program scope`); - // Whole-program over first-party + declared deps only (no --ignore-dependencies, but exclude the - // rest). `--` terminates the variadic --exclude-packages list before the positional entry files. - const jellyArgs = ["-j", out]; - if (excluded.length) jellyArgs.push("--exclude-packages", ...excluded, "--"); - jellyArgs.push(...entryFiles); - - // Two launch modes. Compiled single-binary (CANTS_SELF_JELLY set by src/main.ts): re-exec THIS - // executable with the hidden `__jelly` subcommand — the Jelly CLI is bundled in, so no external - // `node` or node_modules is needed. Dev/source or explicit JELLY_BIN override: shell out to - // `node @cs-au-dk/jelly/lib/main.js` as before. - const self = process.env.CANTS_SELF_JELLY && !process.env.JELLY_BIN ? process.env.CANTS_SELF_JELLY : null; - const cmd = self ?? "node"; - const args = self ? ["__jelly", ...jellyArgs] : [resolveJellyMain(), ...jellyArgs]; - try { - execFileSync(cmd, args, { - cwd: ctx.root, - stdio: ["ignore", "ignore", "ignore"], - maxBuffer: 256 * 1024 * 1024, - timeout: 600_000, - env: { ...process.env, NODE_OPTIONS: `${process.env.NODE_OPTIONS ?? ""} --max-old-space-size=8192`.trim() }, - }); - return JSON.parse(fs.readFileSync(out, "utf8")) as JellyJson; - } finally { - try { - fs.rmSync(out, { force: true }); - } catch { - /* best-effort cleanup */ - } - } -} - -/** getDescendantAtPos lands on the token at the span start; climb to the enclosing function node. */ -function climbToFunctionLike(node: Node | undefined): Node | undefined { - let n = node; - while ( - n && - !( - Node.isArrowFunction(n) || - Node.isFunctionExpression(n) || - Node.isFunctionDeclaration(n) || - Node.isMethodDeclaration(n) || - Node.isConstructorDeclaration(n) || - Node.isGetAccessorDeclaration(n) || - Node.isSetAccessorDeclaration(n) - ) - ) { - n = n.getParent(); - } - return n; -} - -/** Climb from the token at a call-site span to the enclosing call/new expression. */ -function climbToCallExpr(node: Node | undefined): Node | undefined { - let n = node; - while (n && !(Node.isCallExpression(n) || Node.isNewExpression(n))) n = n.getParent(); - return n; -} - -/** Synthetic signature for an anonymous callback: nearest signed enclosing scope + location suffix. */ -function synthesize(fnNode: Node, root: string): string { - let host: Node | undefined = fnNode.getParent(); - while (host && computeSignatureForDecl(host, root) === null) host = host.getParent(); - const hostSig = host ? computeSignatureForDecl(host, root) : null; - const { line, column } = fnNode.getSourceFile().getLineAndColumnAtPos(fnNode.getStart()); - return `${hostSig ?? ""}:<${line}:${column}>`; -} - -/** - * If a Jelly file path lives under node_modules, split it into the owning package name and the - * path within that package. Uses the LAST `node_modules/` so nested deps resolve to the innermost - * package, and handles `@scope/name`. - */ -function depPackage(rel: string): { pkg: string; inPkg: string } | null { - const marker = "node_modules/"; - const idx = rel.lastIndexOf(marker); - if (idx < 0) return null; - const parts = rel.slice(idx + marker.length).split("/"); - if (parts[0].startsWith("@")) { - if (parts.length < 2) return null; - return { pkg: `${parts[0]}/${parts[1]}`, inPkg: parts.slice(2).join("/") }; - } - return { pkg: parts[0], inPkg: parts.slice(1).join("/") }; -} - -/** - * Map a function node to its signature. A `const foo = () => …` arrow is named by its - * VariableDeclaration in the symbol table, so normalize to that parent before deciding - * real-vs-synthesized — otherwise every named const-arrow would wrongly synthesize. - */ -function signatureFor(fn: Node, root: string): { sig: string; synth: boolean } { - const parent = fn.getParent(); - const decl = parent && Node.isVariableDeclaration(parent) ? parent : fn; - const real = computeSignatureForDecl(decl, root); - if (real) return { sig: real, synth: false }; - return { sig: synthesize(fn, root), synth: true }; -} - -export const jellyProvider: CallGraphProvider = { - name: "jelly", - build(ctx): CallGraphResult { - const entryFiles = ctx.project - .getSourceFiles() - .map((sf) => sf.getFilePath() as string) - .filter((fp) => !fp.includes("/node_modules/") && !fp.endsWith(".d.ts")) - .map((fp) => path.relative(ctx.root, fp)) - .filter((rel) => rel.length > 0 && !rel.startsWith("..")); - - if (entryFiles.length === 0) { - ctx.log.info("call graph (jelly): no first-party source files to analyze"); - return { edges: [], external_symbols: {}, synthesized_callables: {} }; - } - - const cg = runJelly(ctx, entryFiles); - - // Phase 1: classify each Jelly function. First-party functions round-trip to a ts-morph node and - // reuse the canonicalizer (real or synthesized). Dependency functions are recorded by package + - // location; an external symbol is minted for them lazily, when an edge reveals how they connect. - const id2sig = new Map(); // first-party id -> signature - const firstPartyIds = new Set(); - const depMeta = new Map(); - let synthesized = 0; - let unresolved = 0; - - // Anonymous callbacks get a synthesized signature with no symbol-table node; remember their - // location so the projection can materialize a node and the edge won't dangle (issue #13). - const synthesizedCallables: Record = {}; - const recordIfSynth = (fn: Node, sig: string, synth: boolean): void => { - if (!synth || synthesizedCallables[sig]) return; - const { line, column } = fn.getSourceFile().getLineAndColumnAtPos(fn.getStart()); - synthesizedCallables[sig] = { - name: "", - path: fileKeyOf(fn.getSourceFile().getFilePath(), ctx.root).fileKey, - start_line: line, - start_column: column, - }; - }; - for (const [id, loc] of Object.entries(cg.functions)) { - const [fileIdx, sl, sc] = loc.split(":").map(Number); - const rel = cg.files[fileIdx]; - if (rel === undefined) { - unresolved++; - continue; - } - const dep = depPackage(rel); - if (dep) { - depMeta.set(id, { pkg: dep.pkg, inPkg: dep.inPkg, sl, sc }); - continue; - } - const sf = ctx.project.getSourceFile(path.resolve(ctx.root, rel)); - if (!sf) { - unresolved++; - continue; - } - let offset: number; - try { - offset = sf.compilerNode.getPositionOfLineAndCharacter(sl - 1, sc - 1); - } catch { - unresolved++; - continue; - } - const fn = climbToFunctionLike(sf.getDescendantAtPos(offset)); - if (!fn) { - unresolved++; // module-level node and other non-function spans - continue; - } - const { sig, synth } = signatureFor(fn, ctx.root); - id2sig.set(id, sig); - firstPartyIds.add(id); - if (synth) synthesized++; - recordIfSynth(fn, sig, synth); - } - - const external_symbols: Record = {}; - const edges = new Map(); - let boundary = 0; - let dropped = 0; - const addEdge = (source: string, target: string, tags: Record): void => { - const k = `${source} ${target}`; - const ex = edges.get(k); - if (ex) ex.weight++; - else edges.set(k, { source, target, type: CALL_DEP, weight: 1, provenance: ["jelly"], tags }); - }; - // Location-keyed fallback signature for a dep function — used when no call-site member name is - // available (the dep→first-party direction, where the call site is inside the library). - const depLocSig = (d: { pkg: string; inPkg: string; sl: number; sc: number }): string => { - const sig = `${d.pkg}:${d.inPkg}:<${d.sl}:${d.sc}>`; - if (!external_symbols[sig]) external_symbols[sig] = { name: `${d.inPkg}:${d.sl}:${d.sc}`, module: d.pkg }; - return sig; - }; - // Per-file import/require index, for naming a library member at a first-party call site. - const extIndexCache = new Map(); - const extIndexFor = (sf: SourceFile): ExternalIndex => { - const key = sf.getFilePath(); - let idx = extIndexCache.get(key); - if (!idx) { - idx = buildExternalIndex(sf as unknown as Node); - extIndexCache.set(key, idx); - } - return idx; - }; - - // Phase 2a — first-party → dependency, NAMED via the call site. call2fun maps call-site → callee; - // we resolve the site in first-party source to (caller function, library member) and key the - // external symbol as `module.member`, matching the tsc phantom path so the two are comparable. - for (const [callId, calleeId] of cg.call2fun) { - const dep = depMeta.get(String(calleeId)); - if (!dep) continue; // callee is first-party (via fun2fun) or unresolved - const cloc = cg.calls[String(callId)]; - if (!cloc) continue; - const [cFileIdx, csl, csc] = cloc.split(":").map(Number); - const crel = cg.files[cFileIdx]; - if (crel === undefined || depPackage(crel)) continue; // the call site must be first-party - const sf = ctx.project.getSourceFile(path.resolve(ctx.root, crel)); - if (!sf) continue; - let coff: number; - try { - coff = sf.compilerNode.getPositionOfLineAndCharacter(csl - 1, csc - 1); - } catch { - continue; - } - const callNode = climbToCallExpr(sf.getDescendantAtPos(coff)); - if (!callNode) continue; - const callerFn = climbToFunctionLike(callNode); - if (!callerFn) continue; // top-level call, no enclosing function - const { sig: callerSig, synth: callerSynth } = signatureFor(callerFn, ctx.root); - recordIfSynth(callerFn, callerSig, callerSynth); - const ph = resolvePhantom(callNode, extIndexFor(sf)); - let sig: string; - if (ph) { - sig = ph.signature; // module.member - if (!external_symbols[sig]) external_symbols[sig] = { name: ph.member, module: ph.module }; - } else { - sig = depLocSig(dep); // unresolved import — fall back to the location key - } - addEdge(callerSig, sig, { "ts.external": "true", "ts.module": external_symbols[sig].module }); - boundary++; - } - - // Phase 2b — fun2fun for first-party→first-party (internal) and dependency→first-party (the - // entrypoint signal: a library invoking your code). first-party→dep is named in 2a; dep→dep and - // edges with an unresolved endpoint are dropped. - for (const [callerId, calleeId] of cg.fun2fun) { - const cid = String(callerId); - const tid = String(calleeId); - const srcFP = firstPartyIds.has(cid); - const tgtFP = firstPartyIds.has(tid); - if (srcFP && tgtFP) { - addEdge(id2sig.get(cid)!, id2sig.get(tid)!, {}); - continue; - } - if (!srcFP && tgtFP) { - const dep = depMeta.get(cid); - if (!dep) { - dropped++; // unresolved caller - continue; - } - addEdge(depLocSig(dep), id2sig.get(tid)!, { "ts.external": "true", "ts.module": dep.pkg }); - boundary++; - continue; - } - if (!(srcFP && !tgtFP)) dropped++; // dep→dep / unresolved (first-party→dep is counted in 2a) - } - - // Keep only synthesized callables that an edge actually references — no orphan nodes. - const referenced = new Set(); - for (const e of edges.values()) { - referenced.add(e.source); - referenced.add(e.target); - } - const synthesized_callables: Record = {}; - for (const [sig, sc] of Object.entries(synthesizedCallables)) if (referenced.has(sig)) synthesized_callables[sig] = sc; - - ctx.log.info( - `call graph (jelly): ${Object.keys(cg.functions).length} jelly funcs, ${firstPartyIds.size} first-party ` + - `(${synthesized} synthesized, ${Object.keys(synthesized_callables).length} materialized), ` + - `${Object.keys(external_symbols).length} external symbols, ${unresolved} unresolved, ` + - `${edges.size} edges (${boundary} library-boundary), ${dropped} dropped`, - ); - return { edges: [...edges.values()], external_symbols, synthesized_callables }; - }, -}; diff --git a/src/semantic_analysis/provider.ts b/src/semantic_analysis/provider.ts index 4caf5cc..ec76134 100644 --- a/src/semantic_analysis/provider.ts +++ b/src/semantic_analysis/provider.ts @@ -1,20 +1,16 @@ /** - * Call-graph provider seam. The orchestrator builds the graph through a CallGraphProvider so the - * backend is swappable: - * • `union` (default) — run tsc + jelly and emit the MERGED edge/node set (tsc ∪ jelly), tagged - * by `provenance` so consumers can still tell the two apart. - * • `tsc` — the always-on ts-morph resolver only (the explicit `--tsc-only` opt-out). - * • `jelly` — the cs-au-dk flow-based analyzer only. - * `both` is a deprecated alias of `union`: it used to run each and log a diff while emitting tsc - * only, which silently discarded every jelly edge and external symbol (see issue #11). + * Call-graph build seam. One backend: the tsc (ts-morph checker) resolver. `tscProvider` stays an + * object (rather than a bare function) so tests can spy on the build being skipped at -a 1. + * `mergeCallGraphs` merges edge/node sets by (source, target) with provenance union — the defuse + * linker's edges overlay the tsc base through it (an edge found by both carries + * `["defuse", "tsc"]` after the wire sort). */ import type { Project } from "ts-morph"; import type { TSExternalSymbol, TSModule } from "../schema"; import type { Logger } from "../utils"; import { buildCallGraph, type CallGraphResult } from "./callGraph"; -import { jellyProvider } from "./jellyProvider"; -/** Everything a provider needs to produce a call graph over the analyzed project. */ +/** Everything the builder needs to produce a call graph over the analyzed project. */ export interface CallGraphContext { project: Project; symbol_table: Record; @@ -32,7 +28,7 @@ export interface CallGraphProvider { build(ctx: CallGraphContext): CallGraphResult; } -/** The always-available backend — wraps the existing tsc resolver with zero behavior change. */ +/** The one backend — the ts-morph checker resolver (+ RTA + phantoms). */ export const tscProvider: CallGraphProvider = { name: "tsc", build: (ctx) => buildCallGraph(ctx.project, ctx.symbol_table, ctx.root, ctx.log, ctx.phantoms, ctx.only), @@ -41,11 +37,9 @@ export const tscProvider: CallGraphProvider = { /** * Merge two call-graph results into their union. Pure (no I/O) so it can be unit-tested directly. * - * Edges are keyed by `(source, target)`. A duplicate edge sums its weight, unions its `provenance` - * (so an edge found by both providers carries `["tsc", "jelly"]`), and merges its tags (base wins - * on conflict — the tsc edge is the authoritative one for the shared key). External symbols union - * by signature, base winning on conflict. `a` is treated as the base (tsc), `b` as the overlay - * (jelly). + * Edges are keyed by `(source, target)`. A duplicate edge sums its weight, unions its `provenance`, + * and merges its tags (base wins on conflict — the base edge is authoritative for the shared key). + * External symbols union by signature, base winning on conflict. */ export function mergeCallGraphs(a: CallGraphResult, b: CallGraphResult): CallGraphResult { const byKey = new Map(); @@ -67,54 +61,3 @@ export function mergeCallGraphs(a: CallGraphResult, b: CallGraphResult): CallGra const synthesized_callables = { ...b.synthesized_callables, ...a.synthesized_callables }; return { edges: [...byKey.values()], external_symbols, synthesized_callables }; } - -/** Count how the two edge sets overlap — preserves the old `both`-mode diagnostic. */ -function diffSummary(tsc: CallGraphResult, jelly: CallGraphResult): string { - const key = (e: { source: string; target: string }): string => `${e.source} ${e.target}`; - const tscKeys = new Set(tsc.edges.map(key)); - const jellyKeys = new Set(jelly.edges.map(key)); - let shared = 0; - for (const k of jellyKeys) if (tscKeys.has(k)) shared++; - return ( - `${shared} shared, ${tscKeys.size - shared} tsc-only, ${jellyKeys.size - shared} jelly-only ` + - `(tsc=${tscKeys.size}, jelly=${jellyKeys.size})` - ); -} - -/** - * Run tsc + jelly and emit their union. This is the default: jelly's edges and external symbols are - * PERSISTED (tagged `provenance: ["jelly"]`) instead of being discarded after a diff. If jelly - * fails, degrade to tsc only rather than failing the whole analysis. - */ -export const unionProvider: CallGraphProvider = { - name: "union", - build(ctx) { - const tsc = tscProvider.build(ctx); - let jelly: CallGraphResult; - try { - jelly = jellyProvider.build(ctx); - } catch (e) { - ctx.log.info(`call graph (union): jelly failed (${(e as Error).message}); emitting tsc only`); - return tsc; - } - ctx.log.info(`call graph diff: ${diffSummary(tsc, jelly)}`); - const merged = mergeCallGraphs(tsc, jelly); - ctx.log.info( - `call graph (union): ${merged.edges.length} edges, ` + - `${Object.keys(merged.external_symbols).length} external symbols`, - ); - return merged; - }, -}; - -export function selectProvider(name: string): CallGraphProvider { - switch (name) { - case "tsc": - return tscProvider; - case "jelly": - return jellyProvider; - default: - // "union" (the default) and the deprecated "both" alias both land here. - return unionProvider; - } -} diff --git a/src/syntactic_analysis/builders.ts b/src/syntactic_analysis/builders.ts index e87bdd0..16a1477 100644 --- a/src/syntactic_analysis/builders.ts +++ b/src/syntactic_analysis/builders.ts @@ -294,7 +294,10 @@ function buildAttributeField(prop: Node): TSField { function buildCallsite(call: Node): TSCallsite { const isNew = Node.isNewExpression(call); - const expr = (call as unknown as { getExpression: () => Node }).getExpression(); + // A tagged template (`inline\`url(...)\``) is a call whose callee is the tag. + const expr = Node.isTaggedTemplateExpression(call) + ? call.getTag() + : (call as unknown as { getExpression: () => Node }).getExpression(); let method_name = expr.getText(); let receiver_expr: string | undefined; let receiver_type: string | undefined; @@ -305,7 +308,7 @@ function buildCallsite(call: Node): TSCallsite { receiver_type = inferredType(expr.getExpression()); is_optional_chain = boolOf(expr, "hasQuestionDotToken"); } - const args = (call as unknown as { getArguments: () => Node[] }).getArguments(); + const args = (call as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; // tagged templates have none const argument_types = args.map((a) => inferredType(a) ?? "unknown"); const typeArgs = (call as unknown as { getTypeArguments?: () => Node[] }).getTypeArguments?.() ?? []; const type_arguments = typeArgs.map((t) => t.getText()); @@ -362,16 +365,13 @@ function walkBody(body: Node, h: BodyHandlers): void { return; } if (b === "skip") return; - if (Node.isCallExpression(node) || Node.isNewExpression(node)) h.onCall(node); + if (Node.isCallExpression(node) || Node.isNewExpression(node) || Node.isTaggedTemplateExpression(node)) h.onCall(node); node.forEachChild(visit); }; - // A concise arrow body can *be* a callable (`() => () => x`). Visiting only the body's children - // would skip it and attribute its call sites to the callable that merely returns it. - if (namedBoundary(body) !== null) { - visit(body); - return; - } - body.forEachChild(visit); + // Visit the body NODE itself, not only its children: a concise arrow body can *be* a callable + // (`() => () => x`) — the boundary handler claims it — or *be* the call (`u => u.describe()`), + // which a children-only walk would silently skip (the call-site gap Jelly used to paper over). + visit(body); } function computeCC(body: Node): number { @@ -457,19 +457,23 @@ export function buildCallable( const callables: Record = {}; const types: Record = {}; + const handlers = { + onCall: (n: Node) => call_sites.push(buildCallsite(n)), + onNestedCallable: (n: Node) => { + const r = buildNestedCallable(n, root); + if (r) callables[memberKey(r.sig, r.callable.accessor_kind)] = r.callable; + }, + onNestedClass: (n: Node) => { + const r = buildClass(n, root); + types[memberKey(r.sig)] = r.cls; + }, + }; const body = (fnNode as unknown as { getBody?: () => Node | undefined }).getBody?.(); - if (body) { - walkBody(body, { - onCall: (n) => call_sites.push(buildCallsite(n)), - onNestedCallable: (n) => { - const r = buildNestedCallable(n, root); - if (r) callables[memberKey(r.sig, r.callable.accessor_kind)] = r.callable; - }, - onNestedClass: (n) => { - const r = buildClass(n, root); - types[memberKey(r.sig)] = r.cls; - }, - }); + if (body) walkBody(body, handlers); + // Parameter DEFAULT initializers execute in the callee's own activation (`f(x = mk())`), so + // their calls (and nested arrows) belong to this callable — they live outside getBody(). + for (const p of (fnNode as unknown as { getParameters?: () => Node[] }).getParameters?.() ?? []) { + walkBody(p, handlers); } const nameNode = sigNode as unknown as { getName?: () => string | undefined }; diff --git a/src/syntactic_analysis/discovery.ts b/src/syntactic_analysis/discovery.ts index 60c31b4..c0abae5 100644 --- a/src/syntactic_analysis/discovery.ts +++ b/src/syntactic_analysis/discovery.ts @@ -3,6 +3,13 @@ import * as path from "node:path"; import { relPosix } from "../utils"; const SOURCE_EXTS = new Set([".ts", ".tsx", ".mts", ".cts"]); +// JS sources are first-class too (the analyzer is a TS/JS analyzer; vendored .js like vscode's +// marked.js was invisible, #98). Sibling rules keep one module per prefix: +// - a .js beside a same-prefix REAL .ts source is compiled output → the .js is skipped +// (the compiler's own allowJs duplicate rule); +// - a .d.ts beside a same-prefix .js is hand-written declarations FOR that source → the .d.ts +// is skipped as a module (the checker still reads it from disk for importers' types). +const JS_EXTS = new Set([".js", ".jsx", ".mjs", ".cjs"]); export const SKIP_DIRS = new Set([ "node_modules", @@ -23,7 +30,7 @@ const TEST_DIRS = new Set(["__tests__", "__test__", "test", "tests", "spec", "__ /** Test-ness is judged on the path RELATIVE TO the project root, never the absolute path. */ function isTestFile(relKey: string): boolean { const base = path.basename(relKey); - if (/\.(test|spec)\.(ts|tsx|mts|cts)$/.test(base)) return true; + if (/\.(test|spec)\.(ts|tsx|mts|cts|js|jsx|mjs|cjs)$/.test(base)) return true; return relKey.split("/").some((p) => TEST_DIRS.has(p)); } @@ -32,9 +39,11 @@ export interface DiscoveredFile { fileKey: string; // project-relative POSIX path with extension } -/** Recursively discover .ts/.tsx sources under root, skipping vendored and (optionally) test trees. */ +/** Recursively discover TS/JS sources under root, skipping vendored and (optionally) test trees. */ export function discoverSourceFiles(root: string, skipTests: boolean): DiscoveredFile[] { - const out: DiscoveredFile[] = []; + const tsFiles: DiscoveredFile[] = []; + const jsCandidates: DiscoveredFile[] = []; + const realTsPrefixes = new Set(); // non-.d.ts TS sources only const walk = (dir: string): void => { let entries: fs.Dirent[]; try { @@ -50,14 +59,33 @@ export function discoverSourceFiles(root: string, skipTests: boolean): Discovere walk(abs); } else if (e.isFile()) { const ext = path.extname(e.name); - if (!SOURCE_EXTS.has(ext)) continue; + const isTs = SOURCE_EXTS.has(ext); + const isJs = JS_EXTS.has(ext); + if (!isTs && !isJs) continue; const fileKey = relPosix(root, abs); if (skipTests && isTestFile(fileKey)) continue; - out.push({ absPath: abs, fileKey }); + if (isTs) { + if (!fileKey.endsWith(".d.ts")) realTsPrefixes.add(fileKey.replace(/\.(tsx|ts|mts|cts)$/, "")); + tsFiles.push({ absPath: abs, fileKey }); + } else { + jsCandidates.push({ absPath: abs, fileKey }); + } } } }; walk(root); + const out: DiscoveredFile[] = []; + const jsPrefixes = new Set(); + for (const j of jsCandidates) { + const prefix = j.fileKey.replace(/\.(jsx|js|mjs|cjs)$/, ""); + if (realTsPrefixes.has(prefix)) continue; // compiled sibling of a TS source → skip + jsPrefixes.add(prefix); + out.push(j); + } + for (const t of tsFiles) { + if (t.fileKey.endsWith(".d.ts") && jsPrefixes.has(t.fileKey.replace(/\.d\.ts$/, ""))) continue; // decls FOR an analyzed .js + out.push(t); + } out.sort((a, b) => a.fileKey.localeCompare(b.fileKey)); return out; } diff --git a/test/anonymous-callables.test.ts b/test/anonymous-callables.test.ts index 076416e..20b2ca4 100644 --- a/test/anonymous-callables.test.ts +++ b/test/anonymous-callables.test.ts @@ -38,7 +38,6 @@ function options(level: number): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, } as unknown as AnalysisOptions; diff --git a/test/dataflow.test.ts b/test/dataflow.test.ts index a539472..7e5a686 100644 --- a/test/dataflow.test.ts +++ b/test/dataflow.test.ts @@ -34,7 +34,6 @@ function options(level: 1 | 2 | 3, cacheDir: string, jobs: number): AnalysisOpti eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir, verbosity: 0, }; diff --git a/test/external-resolution.test.ts b/test/external-resolution.test.ts index 86a957e..034d942 100644 --- a/test/external-resolution.test.ts +++ b/test/external-resolution.test.ts @@ -39,7 +39,6 @@ function options(): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, }; diff --git a/test/multi-tsconfig.test.ts b/test/multi-tsconfig.test.ts index 7084748..2d630f9 100644 --- a/test/multi-tsconfig.test.ts +++ b/test/multi-tsconfig.test.ts @@ -42,7 +42,6 @@ function options(): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, }; diff --git a/test/neo4j-bolt.test.ts b/test/neo4j-bolt.test.ts index 684f7c3..9015a52 100644 --- a/test/neo4j-bolt.test.ts +++ b/test/neo4j-bolt.test.ts @@ -48,7 +48,6 @@ function optsFor(overrides: Partial = {}): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: path.join(TMP, "cache"), verbosity: 0, ...overrides, diff --git a/test/neo4j-schema.test.ts b/test/neo4j-schema.test.ts index 02417b7..4812598 100644 --- a/test/neo4j-schema.test.ts +++ b/test/neo4j-schema.test.ts @@ -30,7 +30,7 @@ async function fixtureRows() { neo4jUri: null, neo4jUser: "neo4j", neo4jPassword: "", neo4jDatabase: null, analysisLevel: 4, graphs: ["cfg", "dfg", "pdg", "sdg"], graphFieldDepth: 3, jobs: 1, targetFiles: null, skipTests: true, eager: true, - noBuild: true, phantoms: true, callGraphProvider: "union", cacheDir, verbosity: 0, + noBuild: true, phantoms: true, cacheDir, verbosity: 0, }; try { return project((await analyze(opts)).application); diff --git a/test/schema-v2.test.ts b/test/schema-v2.test.ts index eda00ce..0f02b27 100644 --- a/test/schema-v2.test.ts +++ b/test/schema-v2.test.ts @@ -38,7 +38,6 @@ function options(): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, }; @@ -246,7 +245,7 @@ describe("schema v2 — L1 skips the call-graph solve (issue #31)", () => { const spy = spyOn(tscProvider, "build"); const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-v2-l1-guard-")); try { - const v1L1 = (await analyze({ ...options(), analysisLevel: 1, callGraphProvider: "tsc", cacheDir })).internal; + const v1L1 = (await analyze({ ...options(), analysisLevel: 1, cacheDir })).internal; expect(spy).not.toHaveBeenCalled(); expect(v1L1.call_graph).toEqual([]); expect(Object.keys(v1L1.external_symbols)).toEqual([]); @@ -261,7 +260,7 @@ describe("schema v2 — L1 skips the call-graph solve (issue #31)", () => { const spy = spyOn(tscProvider, "build"); const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-v2-l1-guard-l2-")); try { - const v1L2guard = (await analyze({ ...options(), analysisLevel: 2, callGraphProvider: "tsc", cacheDir })).internal; + const v1L2guard = (await analyze({ ...options(), analysisLevel: 2, cacheDir })).internal; expect(spy).toHaveBeenCalledTimes(1); expect(v1L2guard.call_graph.length).toBeGreaterThan(0); } finally { diff --git a/test/synthesized-nodes.test.ts b/test/synthesized-nodes.test.ts index 018cb88..a30e410 100644 --- a/test/synthesized-nodes.test.ts +++ b/test/synthesized-nodes.test.ts @@ -23,7 +23,7 @@ const app: AnalysisInternal = { classes: {}, interfaces: {}, enums: {}, type_aliases: {}, namespaces: {}, variables: [], } as unknown as TSModule, }, - call_graph: [{ source: "src/x.foo", target: ANON, type: CALL_DEP, weight: 1, provenance: ["jelly"], tags: {} }], + call_graph: [{ source: "src/x.foo", target: ANON, type: CALL_DEP, weight: 1, provenance: ["defuse"], tags: {} }], external_symbols: {}, synthesized_callables: { [ANON]: { name: "", path: "src/x.ts", start_line: 3, start_column: 10 } }, }; diff --git a/test/tagged-templates.test.ts b/test/tagged-templates.test.ts new file mode 100644 index 0000000..0ea2452 --- /dev/null +++ b/test/tagged-templates.test.ts @@ -0,0 +1,56 @@ +/** + * Tagged template expressions are call sites (#98): `inline\`url(...)\`` must record a `call` + * body node and resolve a call edge to the tag — found missing by the vscode Joern ledger + * (cssValue.ts's `inline` idiom), then crash-guarded (tagged templates have no arguments list). + */ +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../src/core"; +import type { AnalysisOptions } from "../src/options"; + +const dir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-tagged-")); +fs.mkdirSync(path.join(dir, "src")); +fs.writeFileSync( + path.join(dir, "src", "x.ts"), + [ + "export function inline(strings: TemplateStringsArray, ...v: string[]): string { return ''; }", + "export function asCSSUrl(): string { return inline`url('x')`; }", + "export const top = inline`module-scope`;", + "declare const unknownTag: any;", + "export function throughLinker(): void { unknownTag`unresolved-tag`; }", + "export function mkSheet(): number { return 1; }", + "export function createRule(sel: string, sheet = mkSheet()): number { return sheet; }", + ].join("\n"), +); + +const opts = { + input: dir, output: null, emit: "json", appName: "tagged", neo4jUri: null, neo4jUser: "neo4j", + neo4jPassword: "", neo4jDatabase: null, analysisLevel: 2, graphs: [], graphFieldDepth: 3, + jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, phantoms: true, + cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "cants-tagged-cache-")), verbosity: 0, +} as AnalysisOptions; +const result = await analyze(opts); +fs.rmSync(dir, { recursive: true, force: true }); + +describe("tagged template calls (#98)", () => { + test("a tagged template resolves a call edge to its tag", () => { + expect(result.internal.call_graph.some((e) => e.source === "src/x.asCSSUrl" && e.target === "src/x.inline")).toBe(true); + }); + + test("a module-scope tagged template is attributed to the module", () => { + expect(result.internal.call_graph.some((e) => e.source === "src/x" && e.target === "src/x.inline")).toBe(true); + }); + + test("a parameter-default initializer call is attributed to the callable (#98)", () => { + expect(result.internal.call_graph.some((e) => e.source === "src/x.createRule" && e.target === "src/x.mkSheet")).toBe(true); + }); + + test("the tagged call is a body call node with a refined callee", () => { + const fn = result.application.application.symbol_table["src/x.ts"]?.functions["asCSSUrl"]; + const calls = Object.values(fn?.body ?? {}).filter((b) => b.kind === "call"); + expect(calls.length).toBe(1); + expect(calls[0]?.callee).toBe("can://typescript/tagged/src/x.ts/inline"); + }); +}); diff --git a/test/union-provider.test.ts b/test/union-provider.test.ts deleted file mode 100644 index d9f0cc9..0000000 --- a/test/union-provider.test.ts +++ /dev/null @@ -1,73 +0,0 @@ -/** - * Unit tests for the union merge (issue #11): tsc + jelly edges and external symbols must be - * combined — not discarded — with provenance preserved so consumers can still tell them apart. - */ -import { describe, expect, test } from "bun:test"; -import type { CallGraphResult } from "../src/semantic_analysis"; -import { mergeCallGraphs } from "../src/semantic_analysis"; -import { CALL_DEP, type TSCallEdge } from "../src/schema"; - -const edge = (source: string, target: string, provenance: string[], extra: Partial = {}): TSCallEdge => ({ - source, - target, - type: CALL_DEP, - weight: 1, - provenance, - tags: {}, - ...extra, -}); - -const result = ( - edges: TSCallEdge[], - external: CallGraphResult["external_symbols"] = {}, - synthesized: CallGraphResult["synthesized_callables"] = {}, -): CallGraphResult => ({ - edges, - external_symbols: external, - synthesized_callables: synthesized, -}); - -describe("mergeCallGraphs", () => { - test("keeps jelly-only edges (the bug: they used to be dropped)", () => { - const tsc = result([edge("a", "b", ["tsc"])]); - const jelly = result([edge("c", "d", ["jelly"])]); - const merged = mergeCallGraphs(tsc, jelly); - const keys = merged.edges.map((e) => `${e.source}->${e.target}`).sort(); - expect(keys).toEqual(["a->b", "c->d"]); - }); - - test("an edge found by both carries both provenances and summed weight", () => { - const tsc = result([edge("a", "b", ["tsc"], { weight: 2 })]); - const jelly = result([edge("a", "b", ["jelly"], { weight: 3 })]); - const merged = mergeCallGraphs(tsc, jelly); - expect(merged.edges).toHaveLength(1); - expect(merged.edges[0].provenance.sort()).toEqual(["jelly", "tsc"]); - expect(merged.edges[0].weight).toBe(5); - }); - - test("merges external symbols from both, tsc winning on conflict", () => { - const tsc = result([], { "pkg.foo": { name: "foo", module: "pkg" } }); - const jelly = result([], { - "pkg.foo": { name: "FOO-jelly", module: "pkg" }, - "pkg.bar": { name: "bar", module: "pkg" }, - }); - const merged = mergeCallGraphs(tsc, jelly); - expect(Object.keys(merged.external_symbols).sort()).toEqual(["pkg.bar", "pkg.foo"]); - expect(merged.external_symbols["pkg.foo"].name).toBe("foo"); // base (tsc) wins - }); - - test("unions synthesized (anonymous-callback) callables from both", () => { - const tsc = result([]); - const jelly = result([], {}, { "src/x.foo:<3:10>": { name: "", path: "src/x.ts", start_line: 3, start_column: 10 } }); - const merged = mergeCallGraphs(tsc, jelly); - expect(Object.keys(merged.synthesized_callables)).toEqual(["src/x.foo:<3:10>"]); - }); - - test("does not mutate the input results", () => { - const tsc = result([edge("a", "b", ["tsc"])]); - const jelly = result([edge("a", "b", ["jelly"])]); - mergeCallGraphs(tsc, jelly); - expect(tsc.edges[0].provenance).toEqual(["tsc"]); - expect(tsc.edges[0].weight).toBe(1); - }); -});