S0297-transform-tf-13: measure output conformance against pinned R4 and US Core, and publish it - #48
Conversation
…nd US Core, and publish it The guide's seven published v2 test messages are carried in the repository, transformed, and every resource of every resulting Bundle validated against the pinned FHIR R4 4.0.1 definitions and the pinned hl7.fhir.us.core 9.0.0 profiles. The outcome is committed as documentation/conformance/result.json and report.md, and `pnpm run conformance` regenerates both. Nothing about the library's output moved: no mapping, no emitted value, no issue code, no emit gate, and src/index.ts is byte-identical to the pin. This is an apparatus, so the result it publishes on day one is a measurement of the library as it stands: none of the seven messages produces a Bundle with zero error-severity results against R4 plus those profiles, and six of the seven are clean against base R4 alone. The README's old claim, that every emitted resource is "validated against @cosyte/fhir before it ships", is gone. It described a minimal internal required-element schema; the replacement says so and points at the measurement. Kept honest by three things. A hand-written claims register the run is held to in both directions, so a declared conformance that breaks and a declared non-conformance that stops reproducing both fail the build. A published list of which classes of check were performed and which were not. And two definition packages verified by sha256 at run time, with no network fetch and no fallback: an unobtainable package, an empty corpus, a zero-document run and an unparseable message each fail explicitly rather than reporting a pass. The corpus is inside the PHI gate rather than beside it: vendor/fhir-packages joins the scan walk, the allow-list gains narrow reviewed entries for the guide's own synthetic tokens, and a new suite plants a violator inside the corpus directory to prove the scanner still bites there. No detection logic changed and no existing gate assertion was weakened.
…ommand under test writes CI found this and no local run could. The unobtainable-input suite mirrors the pinned inputs into a throwaway root under os.tmpdir() and hard-linked them where the platform allowed it. The suite's own write() unlinked before writing, so it looked safe, but the thing under test WRITES: `pnpm run conformance` in its default mode writes result.json and report.md into whatever root it is given, with a plain writeFileSync this suite does not control. Linked, that wrote THROUGH into the repository's own committed artifacts and corrupted them for every later test in the run. It never reproduced in the dev container, where os.tmpdir() is a different filesystem so linkSync fails with EXDEV and the copy fallback took over. On the runner the two share a filesystem, the link succeeded, and `ci / verify` went red on both node 22 and 24. Reproduced directly with a scratch root on the same filesystem: with links the repository's result.json is MODIFIED by a write-mode run against the scratch root; with copies it is UNCHANGED. Every file is now a copy, and a new case digests the four mirrored repository files before the suite runs and asserts they are byte-identical afterwards, so this class cannot come back quietly.
…e report collapsing findings The README's lede still said the library "produces validated `@cosyte/fhir` model nodes" with no qualifier, which is the same claim the deleted line 149 sentence made, in the place a consumer meets first. It now carries the qualification where the claim is made: what "validated" means internally, that it is not a FHIR conformance statement, and where the measurement is published. The grader for that half of the criterion was a single-literal tripwire over the one sentence that had already been deleted, so it could not observe the sentence that violated the rule. It now scans the README by paragraph: any paragraph calling the output validated must be qualified in itself or in the next one, and the opening section must carry both the claim and its qualification. Both assertions were proven to fail against the unqualified lede before the fix went in. The published report's per-message tables were de-duplicated while the heading over them said "every message, every finding": three sibling Observations each missing `Observation.category` collapsed to one row under a sentence stating twelve results, so a reader counting rows read fewer findings than were measured. One row per finding per resource instance now, with the Bundle entry named, and a test that the rows count to the stated number. `result.json` is unchanged: the measurement did not move, only what the table shows of it. No `src/` change, no mapping, emitted value, issue code or emit-gate change; no dependency; no workflow change. The headline is still 0 of 7 against R4 plus US Core, 6 of 7 against base R4 alone.
aebd16b to
b593e64
Compare
|
Fix pass after impl-gate ordinal 1 ( F1 (blocking), closed. The README's opening paragraph still asserted unqualified that the library "produces validated F2 (advisory), addressed. The per-message tables were de-duplicated under a heading promising "every message, every finding" - ORU_R01 stated 12 error-severity results and printed 8 rows. Rather than reword the promise down to "every distinct finding", the table now prints one row per finding per resource instance, naming which Bundle entry it came from, so the rows count to the stated number. Branch rebased, content-neutral: the trees at Headline unchanged and unnarrowed: 0 of 7 against R4 plus US Core, 6 of 7 against base R4 alone. |
1 participant
Spec:
work/specs/S0297-transform-tf-13/spec.md(umbrella).What this is
The guide publishes seven v2 test messages. They are now carried in this
repository, transformed, and every resource of every resulting Bundle is
validated against the pinned FHIR R4 4.0.1 definitions and the pinned
hl7.fhir.us.core9.0.0 profiles. The outcome is committed asdocumentation/conformance/report.mdanddocumentation/conformance/result.json;pnpm run conformanceregenerates both, andpnpm testfails when either driftsfrom what a live run measures.
No mapping, no emitted value, no issue code and no emit gate changed, and
src/index.tsis byte-identical to770e8f4. This is an apparatus, so what itpublishes on day one is a measurement of the library as it stands.
The result
None of the seven messages produces a Bundle with zero error-severity results
against R4 plus those profiles. Against base R4 4.0.1 alone, six of the seven are
clean; the exception is
SIU_S12, whoseAppointmentisbookedwith no start,which R4's own
app-3invariant forbids. The profile findings are mostly oneshape: US Core requires elements the guide's segment maps publish no row for
(
Encounter.type,Coverage.relationship,Observation.category,DocumentReference.category,RelatedPerson.active), and this library leaves anungrounded element absent rather than guessing at it.
The README sentence that read as a conformance claim ("every emitted resource is
validated against
@cosyte/fhirbefore it ships") is gone. It described aminimal internal required-element schema; the replacement says so and points at
the measurement.
What keeps the number honest
test/_support/conformance-claims.jsonis hand written, never generated,and the run is held to it in BOTH directions: a pair declared conformant that
starts failing breaks the build, and a pair declared non-conformant that stops
failing breaks it too, so the register cannot rot into stale excuses.
external terminology resolution and mapping correctness among the latter, and
publishes the required-binding coverage (182 elements enforced, 13 not
evaluated) rather than implying it is complete.
count of profiles the package publishes for that type, so the denominator is
always visible. US Core publishes 26
Observationprofiles and every one istopic-scoped, so applying all of them would measure conformance to a set nothing
can conform to.
network fetch and no fallback. An absent, unreadable or mismatched package, an
empty corpus, a zero-document run and a message that cannot be parsed or
transformed each fail explicitly and name the thing.
The PHI gate
The corpus is inside the gate rather than beside it.
vendor/fhir-packagesjoins the scan walk (purely additive, and the two archives were measured clean on
the path route first), and
scripts/phi-allow-list.txtgains narrow, reviewedentries for the guide's own synthetic tokens, each with its provenance and its
cost written down. No detection logic in
scripts/phi-scan.tschanged and noexisting gate assertion was weakened: the only edit to that file is one line
added to
WALK_ROOT_NAMESplus its rationale, and all 82 existing gate tests passunchanged.
test/scripts/phi-scan-corpus.test.tsplants an uncovered violator inside thecorpus directory, on both routes, and asserts a non-zero exit, so "inside the
gate" is measured rather than asserted.
One consequence worth reading: the guide's
PID-19is a dashed placeholderSocial Security Number, and this scanner refuses a dashed shape in any tracked
file unconditionally, with no allow-list tag that can clear one. The corpus
therefore writes those digits in the page's own HTML numeric character
references, which reading the fixture reconstitutes exactly. It is a re-encoding
and not an edit: it is declared in the fixture's own
additionalEscapingblockwith its reason, and
corpus-provenance.test.tsasserts the reconstituted valueequals the published one. Nothing is redacted and no message is narrowed.
One defect found and fixed on the way
scripts/conformance/run.tsis a command: importing it runs it, and its defaultmode writes the published artifacts. The suite imported it to learn a path, so
every suite that read the committed result was regenerating that result first,
which made the drift check structurally incapable of failing. Caught by mutating
the committed artifact on purpose and watching the whole suite stay green. The
paths and the serializer now live in
scripts/conformance/artifacts.ts, which hasno side effect, and
main()runs only when the file is the process entry point.Verification
pnpm test: 46 files, 947 tests, 0 failures (was 39 / 880 at the pin; the 880existing tests are unchanged and still pass).
pnpm lint,pnpm typecheck,pnpm format:check,pnpm phi-scan,pnpm check:no-emdash,pnpm check:no-internal-refs,pnpm check:agent-notes:all green.
drifting the committed result, over-claiming in the register, repairing the
mangled corpus line, overstating the report's check classes, falsifying the
README headline, narrowing the scan walk, and dropping an allow-list entry.
No workflow, required check or branch-protection change. No change in any other
repository:
@cosyte/fhiris consumed unmodified as the vendored tarball italready was.