L3/L4 dataflow layer is serial in the driver; naive Ray fan-out measured 3.1x slower

[rahlk]

Problem

The L3/L4 dataflow layer runs serially in the driver process while only the symbol table is distributed over Ray. On odoo full (6,086 modules) the driver sat at 99% of a single core with 12.3 GB resident for ~7 minutes after the Ray phase completed. On erpnext it is ~60% of L4 wall time.

The obvious fix — fan build_function_pdgs out over Ray — was implemented and measured to be 3.1× slower, so this issue records the negative result and points at the real target.

What was tried (and reverted)

build_function_pdgs fanned out one Ray task per contiguous slice of modules (4 slices/core), oracle selected by name (ORACLE_SYNTACTIC / ORACLE_SCALPEL) because a factory closure cannot be pickled, results merged in submission order to preserve key insertion order.

Correctness was fine. Output equivalence on erpnext L4: identical callables (8,182), identical cfg (150,164) and cdg (83,043), ddg within 0.05% (1,804,611 vs 1,805,564), same prov mix, key order preserved. The residual delta sat inside the #146 noise floor (serial-vs-serial differed on 3 callables, serial-vs-parallel on 2).

Performance was not. Interleaved A/B on erpnext L4, --ray, same machine:

variant	wall	load at start
parallel dataflow	855s	9.57
serial dataflow (baseline)	272s	14.93

The parallel variant lost badly while running under lower load, so this is not a measurement artifact. Isolated micro-benchmarks did show phase 1 itself parallelizing well (flask 13.3s → 4.6s; erpnext-scale probe 310s → 19s), which means the loss is integration overhead, not the algorithm — most plausibly per-task worker startup and module imports (codeanalyzer + jedi + scalpel + pydantic) paid across ~80 tasks per run, on top of Ray's runtime_env provisioning that _ensure_ray configures.

Where the time actually is

Phase profile of L4 on erpnext (228.6s total run):

phase	time	parallelism
build_function_pdgs (×2: syntactic + Scalpel)	44.5s	embarrassingly parallel
compute_summaries	33.8s	whole-program fixpoint, serial
assemble_sdg	28.8s	global assembly, serial
callsite/nested-def wiring	~8.4s	parallel after a broadcast
emit_l3_body + emit_l4	13.5s	serial projection

So ~48% of the dataflow layer is inherently serial whole-program work. Even a free phase-1 parallelization caps out at roughly a 15–20% cut to L4 wall time — which is why the integration overhead was able to swallow it whole.

CAVEATS

• Any retry must be measured interleaved (A/B/A/B) with load recorded per run. Three separate measurement errors were made on this task — concurrent runs, falling-load ordering, and a micro-benchmark whose 310s serial figure cannot be reconciled with a 272s total run. Back-to-back ordering is not sufficient on this machine.
• Worker-pool reuse (long-lived Ray actors that import once and process many slices) is the only variant likely to beat serial, since it amortizes the per-task import cost that appears to dominate.
• The unexplained micro-benchmark discrepancy is worth resolving on its own: build_function_pdgs with the Scalpel oracle measured 310s standalone but the entire serial CLI run is 272s. Something differs between the standalone PyApplication and the CLI's, likely how often make_alias_oracle degrades to the type-based fallback.

DEFINITION OF DONE

• Either a measured net win on erpnext and odoo full under interleaved A/B, or a decision recorded that the dataflow layer stays serial and effort moves to the serial 48%.
• If parallelized: byte-equivalent output (modulo L2 call graph is not byte-reproducible: Jedi overload resolution for open() varies run to run #146) on flask, requests, erpnext.
• The compute_summaries / assemble_sdg serial tail is profiled at odoo-full scale, since it — not phase 1 — is the dominant cost at scale.