Skip to content

docs(engine): state and pin the concurrent-use guarantee - #332

Open
OmarAlJarrah wants to merge 3 commits into
mainfrom
docs/engine-concurrency-guarantee
Open

docs(engine): state and pin the concurrent-use guarantee#332
OmarAlJarrah wants to merge 3 commits into
mainfrom
docs/engine-concurrency-guarantee

Conversation

@OmarAlJarrah

@OmarAlJarrah OmarAlJarrah commented Aug 9, 2026

Copy link
Copy Markdown
Member

Summary

One engine.Engine can serve Run calls from several goroutines at once, but nothing said so
and nothing pinned it. An embedder had no way to tell, so the safe move was to over-construct —
an engine per request — and a later change that parked a cache or a scratch buffer on the
Engine would have broken the property with a green build.

Rather than take the "pure, reentrant stages" rule on trust, the guarantee was checked before
being written down. Every package-level var reachable from Run — 27 of them across the 22
packages in go list -deps ./engine — is a declaration-only table: keyword sets, reflect.Type
values, sentinel errors. None is assigned, indexed into, or appended to anywhere after package
init.

The shared value matters as much as the shared package. engine.New builds one
*openapi.Compiler and every goroutine dispatches through it, so a field on that struct would
break the guarantee just as surely as a package var. type Compiler struct{} — there is nothing
on it to race on.

The codebase does use package-level function variables as test seams — newEngine,
createOutput, renameOutput, harness.compile, reserializeJSON, encodeYAML — and those
are genuinely mutable package state; a test swapping one while another goroutine compiled would
race. But all of them live in cmd/morphic, cmd/morphic-harness, internal/harness and
ir/irtest, none of which appears in go list -deps ./engine. They do not narrow the guarantee.

So the guarantee is stated as broadly as it is true, and the one place it is genuinely narrower
is called out rather than papered over: compilers.Registry.Lookup is safe concurrently only
once registration has finished, because Register writes an unsynchronized map. engine.NewWith
satisfies that by construction — it fills a fresh registry and only then wraps it — but an
embedder registering a compiler after starting goroutines would not.

Doc comments on engine.Engine, engine.Engine.Run, compilers.Compiler, compilers.Registry
and pass.Validate, plus one new test. No behaviour change: strip the comment lines and the three
source files are identical to what they replace.

Test plan

engine/concurrency_test.go drives the whole 77-spec conformance corpus through a single shared
*engine.Engine from 8 goroutines, and requires every document to be byte-identical to a baseline
compiled with a fresh Engine per spec.

Two choices in it are load-bearing. It does not build an Engine per goroutine — that shares
nothing and proves nothing. And it does not stop at "no race reported": a data race is only one
way concurrency corrupts output, so the documents are compared, which also re-checks the
determinism invariant. The baseline uses unshared engines on purpose; a baseline drawn from the
shared engine would carry the same corruption as the concurrent runs and cancel it out.

Clean run:

$ go test ./engine -race -run TestEngine_ConcurrentRunSharesOneEngine -count=1 -v
=== RUN   TestEngine_ConcurrentRunSharesOneEngine
=== PAUSE TestEngine_ConcurrentRunSharesOneEngine
=== CONT  TestEngine_ConcurrentRunSharesOneEngine
--- PASS: TestEngine_ConcurrentRunSharesOneEngine (1.24s)
PASS
ok  	github.com/dexpace/morphic/engine	3.061s

Whole repo, race detector on: go test ./... -race -count=1 exits 0 with zero
WARNING: DATA RACE occurrences. The gate runs that build — make coverage passes -race — so
the first of the two properties is checked on every CI run rather than only by hand. The slowest
package under the detector is internal/harness at 8.6 s, against the coverage script's
-timeout 300s.

Proof the test bites

Four deliberate defects, each planted, observed, and reverted.

1. Per-instance state on the Engine — a lastFormat field assigned in Run, the shape a
memoizing Engine would take. The new test fails under -race:

WARNING: DATA RACE
--- FAIL: TestEngine_ConcurrentRunSharesOneEngine (1.08s)
    testing.go:1712: race detected during execution of test

Every other test in the package passes with this planted, because each builds its own Engine;
sharing one is what catches it. The package is also ok without -race, which is what makes
the detector step load-bearing rather than decorative.

2. A race-free wrong answer — a mutex-guarded memo on the Engine, keyed on source format
instead of on the spec, handing back a deep copy so no pointer is shared. Zero DATA RACE
reports, and the test still fails, on the comparison:

concurrency_test.go:97: worker 0 compiled allof-boolean-branch.yaml differently (-unshared +shared):
concurrency_test.go:97: worker 0 compiled allof-inline-merge.yaml differently (-unshared +shared):

This is the case a race-only test would wave through, and the reason the baseline is built from
unshared engines.

3. A degenerate baseline — every baseline entry compiled from the same spec, to confirm the
anti-vacuity guard is not decorative:

expected: 77
actual  : 1
Messages: baseline documents must all differ

4. A worker that stops partway through the corpus. This one reported the wrong thing: the
specs it never reached hold zero values, which the diff called "compiled differently". The
comparison loop now names that case for what it is, and says so:

concurrency_test.go:93: worker 0 never compiled allof-inline-residue.yaml

The audit

The package-var claim is the load-bearing one, so it was made by walking the AST rather than by
grepping: index, selector and star writes are resolved to their root identifier, and locals that
shadow a package var are excluded. It was then held to the standard it exists to enforce.
Planting a map-element write (formatTable["x"] = "y") and a sentinel reassignment
(errOptionChannels = nil) each produce exactly one finding, and a local shadowing a package var
produces none. A grep-based version of the same audit reported the same clean result and missed
both plants, which is why it is not the one the claim rests on.

Gate

make gate exits 0: gofmt, vet, lint, both nolint checks, build, the coverage counter's
verifier, coverage at 100%, fuzz and benchmarks.

Closes #82

…cy-guarantee

# Conflicts:
#	compilers/compilers.go
#	engine/engine.go
A worker that stopped partway through left zero values in its results, and the
comparison reported those as documents compiled differently — which sends a
reader looking for a corrupted lowering when nothing lowered at all. Planting an
early break made it print that for 74 of 77 specs. The loop now names the case
and says so.

The length check above it stays: it cannot fail while runCorpus returns either an
error or a full-length slice, and it is what keeps the indexing below from
panicking should that ever change. Its message no longer claims to be about
finishing the corpus, which is the check that was just added.

The Engine doc said the race property is pinned "only under -race" and left the
reader to wonder whether anything runs it. #344 landed -race in the coverage
step, so it does.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

engine: document and pin the concurrent-use guarantee

1 participant