Epic: trustworthy evaluation across CI, user projects, and public benchmarks

[jaakla]

Summary

Build one trustworthy evaluation system for three different decisions:

Tier	Decision	Primary evidence	Reader	Home	State
A. Regression CI	Did a change break the project contract or a checker?	deterministic fixtures plus isolated mutations	maintainers, every PR	evals/	foundation delivered by #7; follow-up fixes implemented
B. Project QA	Is this project internally consistent, reproducible, and supported by the available evidence on the user's data?	artifact checks, clean rerun, attested answers, declared metamorphic relations	end users and reviewers	shipped package and CLI	report semantics and attested expectations implemented locally; metamorphic and source extensions remain
C. Public benchmark	Does the skill improve artifact quality, and what quality/cost trade-off does each model configuration achieve?	paired trials on the same tasks, artifact-first scoring, complete run provenance	public benchmark readers	jaakla/OpenMapBench	interface remains

These tiers share check definitions where useful, but they do not share a score or denominator:

• Tier A evaluates the contract and the checkers.
• Tier B evaluates one project and reports the limits of the available evidence.
• Tier C evaluates an agent/model configuration across a controlled task set.

This epic completes Tier B and defines the narrow interoperability surface Tier C needs. It does not claim that generic checks can prove every numerical answer on arbitrary data.

Why this matters

Without this separation, a green number can mean several incompatible things: the fixtures still pass, the user's artifacts are structurally healthy, or a model performed well on a benchmark. Keeping the questions separate creates practical value:

• maintainers can refactor the skill and check library without accepting vacuous green tests;
• users can inspect real outputs rather than trusting manifest claims or assistant prose;
• reviewers can distinguish verified, reproducible, incomplete, and failed evidence;
• OpenMapBench can compare approaches without rewarding one canonical tool chain;
• failures found in real projects and public trials can become permanent regression coverage.

Non-negotiable evidence rules

1. A check passes only if it ran and established its predicate. Missing dependencies, unsupported formats, exceptions, and absent addressing information are not_testable, never passed or silently skipped.
2. A partially testable report is not presented as fully passed. Aggregate status must expose unavailable required checks and report tested/applicable coverage.
3. A pipeline cannot certify its own answer. A model-produced expectation remains unverified until an independent source or reviewer attests it.
4. Metamorphic relations are conditional. Each relation declares its preconditions, transformation, expected relation, tolerance, and applicability; no relation is assumed to hold for every GIS analysis.
5. Benchmark correctness is artifact-first. Trajectory metrics may be diagnostics, but a correct alternative implementation is not penalised for using different tools.
6. Secrets never enter project.yaml, logs, snapshots, or published benchmark artifacts. Connection discovery is read-only and user-controlled.
7. Reproducibility identifies the whole arm. A skill hash alone is insufficient; the task, data, checks, harness, runtime, model configuration, and pricing date are also recorded.

---

Delivered foundation

Tier A: regression CI

• A0 — deterministic fixture and live-eval foundation (Epic: Harden evals into a trustworthy Open-GIS agent benchmark #7)

  • Separate contract_ci, mutation_tests, agent_benchmark, and integration_visual score types.
  • Assertions grade files and structured records; assistant prose is not a correctness oracle.
  • Mutation cases use a known-healthy control and require one pinned failure code while isolation guards remain green.

• A1 — restore scheduled coverage for behavioural live cases (b43dc26, implemented on the current local main; publication pending)

  • Cases 070–073 no longer exist only as dead, manually invoked coverage.
  • These are negative user tasks—underspecification, contradiction, do-not-invent, and non-English—not checker mutations. They belong in Tier C eventually; scheduled execution is the bridge until C3.

• A2 — remove divergent skill-mode defaults (b43dc26, publication pending)

  • Programmatic callers must choose enabled or disabled explicitly; the benchmark cannot silently run a different arm from the CLI.

• A3 — stop leaving misleading root-owned results (b43dc26, publication pending)

  • Docker output belongs under evals/results/; stale root results are ignored and removed.

• A4 — document failure promotion (b43dc26, publication pending)

  • A plausible wrong project that survives the checks becomes a minimal gen.py --break= defect plus a paired mutation case.
  • This is the feedback loop that prevents the mutation set from covering only defects imagined at initial design time.

Tier B: project-QA foundation

• B1 — ship the check library (88ebde5)

  • Moved the reusable checks into openmapstack/checks/.
  • Added optional geo and visual dependencies; PyQGIS remains a system dependency and degrades explicitly to not_testable.
  • Preserved the fixture and mutation results while fixing a vacuous project.exists pass.
  • At delivery: 284 unit tests, 75% coverage over the relocated package, contract_ci 14/14, and mutation_tests 19/19.

• B2 — add openmapstack verify and share the clean-rerun implementation (350ea9e, 7de11d6, implemented locally; publication pending)

  • validate audits the manifest and bookkeeping; verify reads produced artifacts and derives its check plan from the manifest.
  • Checks include declared-file existence, provenance, geometry validity, CRS read from the dataset, evidence recomputation, presentation consistency, QGIS loading where available, and explicit not_testable outcomes.
  • --rerun rebuilds the project in an empty workspace and compares outputs and validation evidence.
  • Real-project use exposed a fixture blind spot: geodata checks assumed the geometry column was named geom. They now resolve typed and conventional geometry-column names.
  • At delivery: 298 unit tests, 76% check-package coverage, unchanged contract_ci 14/14, and mutation_tests 19/19 detected.

The test counts above describe the recorded delivery state; they are regression baselines, not claims about correctness on arbitrary projects.

---

Workstreams after the delivered foundation

B2.1 — harden report semantics and applicability (implemented locally; publication pending)

• Use aggregate precedence failed → warning → not_testable → passed, with a mixed passed/not_testable result reported as warning, not passed.
• Report executed, applicable, and not_testable counts; do not turn a raw check count into a claim about “grading power.”
• Preserve useful exit-code policy: default fails on definite failure; --strict also fails on warnings or unavailable required checks.
• Add fixtures proving that missing dependencies, an unsupported format, and a raised checker exception cannot produce an aggregate pass.
• Publish an applicability table for the automatic verification plan: plan condition, dependency, untestable causes, defect class, and regression evidence.

Value: users can tell the difference between “the project passed the checks” and “the environment could not inspect important parts of the project.”

B3 — attested expectations for project-specific answers (implemented locally; publication pending)

Add optional validation.expectations[] entries using the same namespaced checks as Tier A. An expectation records:

- id: parcel-count
  check: geodata.row_count
  args: {path: data/derived/parcels.parquet, equals: 1242}
  attestation:
    status: verified
    verified_by: "reviewer or authority"
    verified_against: "stable source locator"
    verified_at: "2026-08-30T00:00:00Z"
    evidence_sha256: "sha256:..."

• Define and validate the schema, including safe project-relative paths, simple SQL identifiers, and a five-check allowlist.
• Treat absent or incomplete attestation as warning/unverified, never passed, and do not execute the expected value.
• Require the skill/template to propose the expectation and ask for confirmation; it must not populate independent-verification fields from its own output.
• Bind review to the expectation digest and current input hash; invalidate changed expectations, inputs, or retained local evidence.
• Distinguish attested, unverified, stale_attestation, and invalid evidence in JSON and user-facing messages.

Value: project-specific numerical answers can be reviewed once and retained without pretending that self-entered metadata is an automatic oracle.

B4 — conditional metamorphic checks

Add a framework for no-golden-answer checks that perturb inputs or parameters and test a declared invariant. Each declaration must include:

• applicability and semantic preconditions;
• the controlled transformation;
• the expected relation between outputs;
• comparison keys and numeric/spatial tolerances;
• resource limits and cleanup behaviour;
• a reasoned not_testable result when the relation cannot be addressed safely.

Candidate relations, implemented only where their preconditions hold:

• input permutation invariance, with an explicit deterministic tie-breaking rule;

• duplicate resistance for analyses whose semantics require deduplication—not for valid counts or sums;

• CRS round-trip stability within a declared validity area and tolerance;

• positive-buffer monotonicity for a declared predicate and metric;

• subset additivity for disjoint partitions and genuinely additive measures;

• area-scale consistency for a specified projection/transformation model.

• Define runtime.implementation.parameters or an equivalent versioned parameter-addressing contract.

• Implement the framework and at least three relations with positive, deliberate-defect, and invalid-precondition tests.

• Ensure the clean rerun cannot mutate declared source data and that temporary variants are isolated and removed.

• Document counterexamples so projects do not enable invalid relations mechanically.

Value: important GIS failure modes—order-dependent joins, wrong CRS/units, inverted predicates, and silent row loss—can be detected without a frozen final answer, while avoiding false failures on analyses for which the relation is not valid.

B5 — user data and reproducible source pinning

Treat warehouse access as a connector and security problem, not only a documentation problem.

B5a: source/pin contract

• Represent a warehouse source with backend, account/project, database/schema/table, query digest, snapshot identifier or timestamp, schema digest, and retrieval metadata.
• Accept two pin classes:
  1. a user-approved local snapshot under data/source/ with a content hash; or
  2. backend time travel/snapshot identity with recorded retention limits.
• Report an expired or inaccessible snapshot as not_reproducible, never as pinned merely because a timestamp string exists.
• Update provenance.every_source_pinned to validate the pin class rather than only rejecting the literal latest.

B5b: safe connector pilot

• Define read-only discovery, secret redaction, query timeout, and row/byte/cost limits.
• Require explicit user approval before materialising data locally.
• Implement and test one local engine plus one warehouse path—DuckDB/local files and PostGIS are the initial reference pair.
• Add references/user-data-sources.md for credentials-by-reference, geometry/SRID discovery, snapshotting, and clean rerun.
• Document other backends only where the behaviour is verified; unsupported backends remain explicit follow-ups.

Value: users can work from real organisational data without putting credentials in the project or claiming reproducibility from a mutable latest table.

C1 — publish a narrow OpenMapBench interoperability contract

OpenMapBench already owns benchmark orchestration, run isolation, reporting, and provider adapters. OpenMapStack should not export a second benchmark harness for it to adopt.

• Expose a stable, versioned check API and machine-readable result schema.
• Keep agent adapters and leaderboard policy in OpenMapBench; avoid copying checker implementations in either direction.
• Define compatibility/version negotiation and a fixture proving OpenMapBench can consume a released OpenMapStack checker package.
• Keep benchmark setup failures outside the scored denominator while reporting them prominently.

Value: one implementation of project checks can serve both products without coupling the user-facing package to model-provider orchestration.

C2 — reproducible skill and arm snapshots

• Add openmapstack skill-snapshot --out DIR --json, generalising the existing controlled copy of SKILL.md, references/, and templates/.
• Record the snapshot content hash, but identify a published arm with the complete provenance tuple: task/data hashes, checker version, harness commit, runtime/container, GIS dependency versions, tool surface, model/provider revision, reasoning/sampling configuration, seed where supported, and price-catalog date.
• Make snapshots inspectable and reject symlinks or paths that escape the snapshot root.

Value: a benchmark result can be reproduced and interpreted after tags, prices, hosted model aliases, or local files change.

C3 — move behavioural benchmark ownership to OpenMapBench

• Make cases 070–073 canonical OpenMapBench tasks after the C1 contract is available.
• Keep a small OpenMapStack scheduled smoke subset only where it protects integration; do not maintain divergent copies as independent benchmarks.
• Run plain and oms as paired arms over the same task/version set with repeated trials and uncertainty.
• Report quality and cost as a Pareto trade-off; do not collapse them into a single “success per dollar” score.
• Record trajectory measures such as TAO/PEA only as diagnostics. Do not use exact tool-sequence matching as headline correctness.

Value: the public benchmark measures whether the skill improves final work, not whether an agent imitates one preferred implementation path.

---

Benchmark evidence policy

OpenMapBench should distinguish four evidence classes rather than treating all of them as interchangeable ground truth:

1. Authoritative answer: a published statistic or independently maintained source.
2. Frozen expert reference: a reviewed implementation and immutable reference artifact.
3. Metamorphic evidence: a declared relation holds under controlled perturbation; useful but not an absolute answer.
4. Differential diagnostic: independent engines agree or disagree. Disagreement is actionable; agreement alone is not proof of correctness.

VLM review may assess cartographic composition, legibility, and semantic presentation. It must have its own denominator, retain judge/version provenance, and be calibrated against blinded human review before publication. It must not turn an unverified analytical result into a correctness pass.

GABench remains useful for task taxonomy, datasets, and trajectory diagnostics. Its exact-sequence metrics should not be OpenMapBench's headline score because OpenMapStack intentionally enables valid alternative stacks and workflows.

---

Explicitly out of scope

• A guarantee that generic checks prove every analytical answer on arbitrary data.
• Treating the number of available checks as a weighted measure of correctness.
• Automatic trust in expectations written by the pipeline or model being evaluated.
• Silent credential extraction, credential storage in project files, or automatic export of warehouse data.
• Production-grade connectors for MotherDuck, BigQuery, Snowflake, Databricks, Redshift, Athena, Supabase, Iceberg, and Delta in this epic; each requires verified semantics and follow-up coverage.
• The cross-warehouse conformance suite; it remains tracked in evals/COVERAGE.md and should precede broad backend claims.
• Raster correctness scoring, true MultiPolygon edge coverage, nearest-neighbour tie semantics, and other existing evals/COVERAGE.md gaps unless pulled into a dedicated child issue.
• A hosted leaderboard service, benchmark governance organisation, or third-party dataset relicensing.
• Grading assistant prose or requiring one canonical library/tool trajectory.
• Using VLM judgement as a substitute for deterministic analytical verification.

---

Delivery order

Published foundation: A0, B1
  -> publish implemented A1–A4 and B2
  -> B2.1 report semantics and applicability
  -> B3 attested expectations
  -> B4 conditional metamorphic framework
  -> B5a source/pin contract
  -> B5b DuckDB + PostGIS connector pilot

In parallel after B2.1:
  C1 versioned checker/result interface
    -> C2 complete arm snapshot provenance
    -> C3 paired OpenMapBench migration

Continuous:
  Tier-C/project failure -> minimal Tier-A mutation when a checker gap is found

Definition of done

• Published commits and documentation identify which capabilities are released; the issue contains no unresolved local-only commit references.
• openmapstack verify runs against at least two projects whose data is absent from this repository's fixtures and emits stable text and JSON reports.
• Every planned check produces one of passed, warning, not_testable, or failed; no exception, missing dependency, unsupported artifact, or omitted addressing information becomes a pass.
• A report containing unavailable required checks cannot have aggregate status passed, and applicability/execution counts are visible.
• Project-specific expectations are either independently attested with durable evidence metadata or visibly unverified.
• At least three metamorphic relations have explicit preconditions, positive tests, deliberate-defect tests, and counterexample tests that reject invalid use.
• The source contract recognises a hashed local snapshot and a real PostGIS snapshot/pin path without storing credentials; expiry and access failures are represented honestly.
• OpenMapBench consumes the released check API/result schema without vendoring the check implementation.
• A paired plain/oms benchmark run records complete arm provenance, task parity, repeated-trial uncertainty, and dated cost estimates.
• Deterministic analytical correctness, metamorphic evidence, differential diagnostics, and visual judgement remain separate report dimensions and denominators.
• A newly discovered plausible-but-wrong project has a documented and tested path into a Tier-A mutation case.

Risks and controls

Risk	Control
False confidence from partial execution	non-pass aggregate for unavailable required checks; applicability reporting
Invalid metamorphic assumptions	declared preconditions, counterexample tests, explicit not_testable
Mutable or expired warehouse state	typed pin contract, retention metadata, honest not_reproducible result
Secret exposure or expensive queries	credentials by reference, redaction, read-only discovery, limits, explicit materialisation approval
Unfair benchmark comparisons	paired task sets, full arm provenance, repetitions and uncertainty
Harness/package coupling	small versioned checker and result interfaces; benchmark adapters remain in OpenMapBench
Benchmark-specific overfitting	failure promotion, held-out tasks, versioned task releases, separate public and development sets

Refs #7.

[jaakla]

A1–A4 done in b43dc26 (tier a), on top of B1 in 88ebde5.

Two of these were worse than the plan assumed:

• A1 — I planned to add cases 070–073 to the workflow's list. Looking at it, the list was pinned to 4 of 10 live-capable cases, so 003 and 006 were dark too. A hardcoded list can't notice a new case, so I dropped it entirely: the runner already selects every case declaring live mode and already exits 2 rather than publishing a green 0/0. A cases input narrows the run when that's wanted.
• A2 — the divergent skill_mode default turned out to be live-load-bearing, not cosmetic. Three existing tests were silently relying on the inferred disabled arm. Requiring it surfaced them; they now state their arm. Fixture and visual modes have no arm to choose, so the requirement is scoped to live only.

Next: B2, openmapstack verify.

[jaakla]

B2 done — openmapstack verify (350ea9e extracts the shared clean-rerun protocol into openmapstack/rerun.py; 7de11d6 adds the command).

Building it surfaced a defect in the check library itself, worth recording because it is the exact failure mode tier b exists to catch:

geodata.geometry_all_valid hardcoded the geometry column as geom, and dataset_crs_is defaulted to it. The eval generator always writes geom, so 39 cases and ~200 unit tests never noticed. But geometry is what GeoPandas and the GeoParquet spec write — so both checks reported not_testable on precisely the column naming real-world data uses. Fixed by resolving the column from the data.

A library validated only against fixtures a generator wrote will encode that generator's conventions as assumptions. That is an argument for running tier b against real projects early, not after the checks are "done".

Two real defects found in the shipped worked example

verify examples/tartu-development reports 38 passed, 2 not_testable (PyQGIS absent), 2 failed:

1. validation.run_record_matches — pipeline.py's hash in the run record does not match the file. openmapstack validate already reports this, but CI only smoke-tests the example with --preflight, which skips output, report and run-record checks — so it has never gated.
2. qgis.groups_match_manifest — the manifest declares 7 layer groups (candidates_tier1, catchments, user_overrides, …); the .qgz layer tree has 4 differently-named ones (Analysis Results, Educational Accessibility, Transportation & Overrides, Basemaps). Neither id nor title matches. Only verify catches this; validate does not.

Both predate this work. The README advertises the example as "a fully-worked reproducible project", and it currently fails its own validator — which undercuts the contract the skill teaches. Filing separately rather than folding into B2.