Publish text.redact-entities@1.0.0 (PII redaction via NER reuse) - #482
Merged
Merged
Conversation
Second pick from the post-#473 "Five Engines, What Next" downstream- capability review, after audio.transcribe-speech@1.1.0's VAD gate (#477). Reuses text.detect-entities' (#465) real DistilBERT-NER forward pass directly -- no new model, no new license/verification burden. New capability id, not a version bump: "redact" is a genuinely different action from "detect" for a caller (different intent, different output shape), unlike the VAD case where "transcribe, robustly" was still the same verb. capability-src/text-detect-entities gets the same lib/bin split proven on audio-detect-speech-segments for #477: engine logic (Model, encode, forward_logits, decode_entities, Entity, plus an always-available fixture_model() for dependents' own tests) moves to a new lib.rs, main.rs becomes a thin _start shim. text.detect-entities@1.0.0's published contract/artifact is untouched -- source-only refactor, all 19 existing tests pass unchanged. text-redact-entities takes a path dependency on that lib, runs the exact same encode/forward/decode pipeline, then substitutes a bracketed placeholder ([PER]/[ORG]/[LOC]/[MISC]) for each entity span instead of returning raw entity data. Verified against the real published model via the compiled wasm artifact under wasmtime: "Sarah Chen from Acme Corp will follow up with the Berlin team next week." redacts to "[PER] from [ORG][ORG][ORG] will follow up with the [LOC] team next week." -- correctly reproducing text.detect-entities' own documented BIO-fragmentation behavior on this exact sentence, verified directly rather than assumed to carry over. capability-src/text-redact-entities: 10 #[test] cases. cargo llvm-cov: functions=100%, lines=100%, regions=100%. New persona privacy-export-reviewer scaffolded (no existing persona fit "blank PII before text leaves a privacy boundary"). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Publishes
text.redact-entities@1.0.0-- PII redaction (PER/ORG/LOC/MISC bracketed placeholders) reusingtext.detect-entities' (#465) real DistilBERT-NER forward pass directly. Closes #481, the second pick from the post-#473 "Five Engines, What Next" downstream-capability review, afteraudio.transcribe-speech@1.1.0's VAD gate (#477).Why a new capability, not a version bump
Unlike the VAD-gate case (a version bump because "transcribe, robustly" was still the same verb), "redact" is a genuinely different action from "detect" for a caller -- different intent, different output shape (
redacted_textvs. rawentities). This is a new capability id.Shape
capability-src/text-detect-entitiesgets the same lib/bin split proven onaudio-detect-speech-segmentsfor audio.transcribe-speech@1.1.0 — gate Whisper on a Silero VAD speech-presence pre-check #477: engine logic (Model,encode,forward_logits,decode_entities,Entity, plus an always-availablefixture_model()for dependents' own tests) moves to a newlib.rs;main.rsbecomes a thin_startshim.text.detect-entities@1.0.0's published contract/artifact is untouched -- source-only refactor, all 19 existing tests pass unchanged.capability-src/text-redact-entitiestakes a path dependency on that lib, runs the exact same encode/forward/decode pipeline, then substitutes a bracketed placeholder ([PER]/[ORG]/[LOC]/[MISC]) for each entity span instead of returning raw entity data.dslim/distilbert-NER(Apache-2.0) license record astext.detect-entities.Verification
wasm32-unknown-unknown --features full-modelartifact underwasmtimeon real sentences:"Sarah Chen from Acme Corp will follow up with the Berlin team next week."redacts to"[PER] from [ORG][ORG][ORG] will follow up with the [LOC] team next week.", and"Elon Musk founded SpaceX in California."redacts to"[PER][PER] founded [ORG][ORG] in [LOC]."-- both correctly reproducingtext.detect-entities' own documented BIO-fragmentation behavior on these exact sentences (verified directly, not assumed to carry over).capability-src/text-redact-entities: 10#[test]cases (placeholder mapping, single/multi-span substitution, no-entities passthrough, defensive out-of-order-span handling, full-pipeline determinism, missing-text-field default);cargo llvm-covfunctions=100%, lines=100%, regions=100%.capability-src/text-detect-entities: all 19 pre-existing tests still pass post-refactor.Governing Spec
Project Item
https://github.com/orgs/traverse-framework/projects/3 -- #481
Definition of Done
capability-src/text-redact-entities/#![no_std]; ABI-clean (imports exactly{fd_read, fd_write, proc_exit});#[test]coverage functions=100%/lines®ions>=95%.capabilities/text/text.redact-entities/1.0.0/contract.json-- full I/O schema,ai.model_backed: true, spec-024risk.determinism_class: model_derived, spec-023authoring, provenance citing text.redact-entities@1.0.0 — PII redaction reusing the DistilBERT-NER engine #481.text.detect-entities@1.0.0contract/artifact untouched by the lib/bin source refactor.privacy-export-reviewerscaffolded viascripts/scaffold/new-persona.sh(no existing persona fit)..wasm,pre_pr_check.shgreen, human merge.Validation
bash scripts/ci/pre_pr_check.sh <this body>with BASE/HEAD wired as CI wires them.🤖 Generated with Claude Code