Merge pull request #5 from SecureToolsProject/feat/sprint-4-jpeg-cleaner-verification

✨[Feat] Sprint 4 JPEG Cleaner and Verification

Author: maruson08

CHANGELOG.md

@@ -6,6 +6,11 @@ All notable changes will be documented here. The project intends to follow seman
 
 ### Added
 
+- Deterministic JPEG Privacy Clean for whole EXIF, XMP, Photoshop/IPTC, and COM segments.
+- Checked single-allocation JPEG reconstruction preserving ICC, unknown APP, structural, scan, and trailing bytes.
+- Structured JPEG presence/absence verification and typed unsupported/incomplete-input errors.
+- Compact canonical coverage for determinism, idempotency, multiple scans and metadata instances, malformed TIFF removal, and exact subviews.
+
 - Shared bounded little- and big-endian TIFF/EXIF decoder.
 - Iterative IFD0, ExifIFD, GPSIFD, and next-IFD traversal.
 - IFD entry/depth limits and repeated-offset cycle protection.
@@ -17,6 +22,11 @@ All notable changes will be documented here. The project intends to follow seman
 - Bounded JPEG marker traversal and EXIF/XMP/ICC/IPTC container detection.
 - Binary boundary, JPEG container, TIFF endian, malformed, cycle, and integration tests.
 
+### Changed
+
+- JPEG parser records internal fill-aware rewrite ranges while retaining existing public source offsets.
+- Parse-limit validation is shared by inspection and cleaning.
+
 ### Foundation
 
 - Repository and TypeScript library scaffold.

README.md

@@ -12,9 +12,11 @@ Current implementation:
 - JPEG EXIF, XMP, ICC, Photoshop/IPTC, and comment container detection;
 - shared little- and big-endian TIFF/EXIF decoder;
 - iterative IFD0, ExifIFD, GPSIFD, and next-IFD traversal with cycle and depth protection;
-- common TIFF, EXIF, and GPS field decoding with exact rational values.
+- common TIFF, EXIF, and GPS field decoding with exact rational values;
+- deterministic whole-segment JPEG Privacy Clean with byte-preserving reconstruction;
+- structured JPEG verification for observable container presence or absence.
 
-Not implemented: MakerNote or thumbnail decoding, XMP/IPTC/ICC payload parsing, PNG/WebP container parsing, metadata cleaning, and verification.
+Not implemented: MakerNote or thumbnail decoding, XMP/IPTC/ICC payload parsing, PNG/WebP container parsing or cleaning, and PNG/WebP verification.
 
 ## Format status
 
@@ -38,11 +40,13 @@ import {
 
 GPS rational components remain exact numerator/denominator pairs; decimal coordinates are not derived. Unknown TIFF tags and MakerNote are represented structurally without dumping or recursively parsing their payloads.
 
-`cleanMetadata` and `verifyMetadata` still throw a typed `NotImplementedError`.
+`cleanMetadata` supports JPEG. Its default policy removes complete EXIF, standard/extended XMP, Photoshop/IPTC, and COM segments while preserving ICC, JFIF/JFXX, Adobe APP14, unknown APP segments, structural data, scan bytes, and trailing bytes. It returns a separate output, container-level change evidence, and an inspection report of that output.
+
+`verifyMetadata` supports JPEG expectations of `absent`, `present`, or `ignore` for EXIF, XMP, IPTC, comments, and ICC. The default checks the four privacy-clean removal targets. A single-file verification can observe presence or absence; it cannot prove that bytes came from an original file.
 
 ## Security philosophy
 
-Every byte is untrusted. All offsets are interpreted within bounded views, traversal is iterative and limited, repeated IFD offsets are rejected, and malformed entries recover without unchecked access. Unknown structures remain unknown and should be preserved by future cleaning. See the [security model](docs/security-model.md), [architecture](docs/architecture.md), and [cleaning policy](docs/cleaning-policy.md).
+Every byte is untrusted. All offsets are interpreted within bounded views, traversal is iterative and limited, repeated IFD offsets are rejected, and malformed entries recover without unchecked access. Unknown JPEG APP structures remain unknown and are preserved by cleaning. See the [security model](docs/security-model.md), [architecture](docs/architecture.md), and [cleaning policy](docs/cleaning-policy.md).
 
 ## Non-goals
 

docs/architecture.md

@@ -32,3 +32,17 @@ Unknown tags retain namespace, tag number, TIFF type, count, entry offset, and s
 - `container-partial`: JPEG traversal stopped on corruption, truncation, or a limit.
 - `metadata-partial`: JPEG container traversal completed and common TIFF/EXIF decoding was attempted; XMP/IPTC/ICC and unknown fields remain incomplete.
 - `metadata-inspected`: reserved for future broader decoders.
+
+## JPEG clean and verify flow
+
+```text
+input JPEG
+  → bounded JPEG parser and existing APP classification
+  → direct keep/remove policy
+  → checked retained ranges
+  → one output allocation and ordered byte copies
+  → inspectMetadata(output)
+  → structured verification checks
+```
+
+The parser remains the structural source of truth. Internal rewrite ranges include marker fill bytes associated with a removed marker while public source offsets retain their existing meaning. Cleaning does not invoke TIFF decoding on the source: a structurally bounded EXIF APP1 can be removed even if its TIFF body is malformed. The post-write inspection and verifier use the normal inspection layer.

docs/cleaning-policy.md

@@ -1,15 +1,23 @@
-# Cleaning Policy Direction
+# Cleaning Policy
 
-Cleaning is not implemented in Sprint 0. This document records the intended conservative policy for future work.
+JPEG Privacy Clean removes complete recognized metadata containers. It never rewrites TIFF/EXIF fields, XMP XML, IPTC blocks, comments, or ICC payloads.
 
-An initial privacy-clean mode should remove EXIF, GPS, XMP, IPTC, comments, and privacy-relevant textual metadata. It should preserve the encoded image payload, required container structures, ICC and other color profiles, rendering-critical metadata, and unknown structures unless the relevant format specification proves removal is safe.
+| JPEG structure                         | Default action |
+| -------------------------------------- | -------------- |
+| EXIF APP1                              | Remove         |
+| Standard XMP APP1                      | Remove         |
+| Extended XMP APP1                      | Remove         |
+| Photoshop/IPTC APP13                   | Remove         |
+| COM                                    | Remove         |
+| ICC APP2                               | Preserve       |
+| JFIF/JFXX APP0                         | Preserve       |
+| Adobe APP14                            | Preserve       |
+| Unknown APP                            | Preserve       |
+| Structural markers and image/scan data | Preserve       |
+| Data after EOI                         | Preserve       |
 
-For v0.1, whole EXIF containers are preferred over selective TIFF rewriting:
+Every recognized instance is handled independently and retained content keeps its original order and bytes. Unknown APP removal is intentionally unavailable in Sprint 4. Callers may override the four removal booleans and ICC preservation; `preserveColorProfiles` remains a deprecated alias for `preserveIcc`.
 
-```text
-JPEG APP1 EXIF → remove whole EXIF APP1
-PNG eXIf       → remove whole eXIf chunk
-WebP EXIF      → remove whole EXIF chunk
-```
+v0.1 removes the entire EXIF APP1, including malformed TIFF bodies whose JPEG segment boundary is valid. Selective GPS or tag rewriting and TIFF reserialization are deferred.
 
-Selective EXIF field rewriting is postponed. This reduces offset-rewrite complexity and makes cleaner behavior easier to audit. Unaffected bytes should remain byte-for-byte identical whenever the container format permits it, and output must be re-inspected rather than trusted merely because a write completed.
+`cleanMetadata` returns a new `Uint8Array`, container-level removed/preserved records, diagnostics, and an inspection report of the produced JPEG. A structurally incomplete JPEG is rejected before allocation. PNG, WebP, and unknown inputs return a typed unsupported-format error.

docs/format-support.md

@@ -11,7 +11,8 @@
 | MakerNote decoding          | Not supported          | Not supported  | Not supported  |
 | XMP payload decoding        | Not yet                | Not yet        | Not yet        |
 | IPTC/ICC payload decoding   | Not yet                | Not yet        | Not yet        |
-| Cleaning and verification   | Not yet                | Not yet        | Not yet        |
+| Whole-container cleaning    | Supported              | Not yet        | Not yet        |
+| Structured verification     | Supported              | Not yet        | Not yet        |
 
 ## TIFF/EXIF subset
 
@@ -28,3 +29,9 @@ Unknown tags remain structurally represented without speculative meaning or larg
 ## Remaining container support
 
 JPEG marker and scan traversal remains supported. XMP, ICC, and Photoshop/IPTC signatures are container-detected only. PNG requires its complete signature and WebP requires `RIFF....WEBP`; their chunks and metadata are not parsed yet.
+
+## JPEG cleaning and verification
+
+JPEG Privacy Clean removes recognized EXIF, standard/extended XMP, Photoshop/IPTC, and comment segments. ICC, JFIF/JFXX, Adobe APP14, unknown APP segments, structural markers, all scan data, and trailing bytes are retained. Structurally incomplete JPEGs are rejected; malformed TIFF inside a bounded removable EXIF segment does not block cleaning.
+
+Verification reports observable container presence or absence for EXIF, XMP, IPTC, comments, and ICC. It does not decode XMP/IPTC/ICC payloads or prove preservation from an original input.

docs/security-model.md

@@ -10,8 +10,8 @@ Binary metadata parsing processes attacker-controlled structures, sizes, offsets
 4. Core functions make no network requests and access no filesystem or DOM APIs.
 5. Image pixel payloads are never decoded.
 6. Unknown metadata is not deleted or assigned speculative meaning.
-7. ICC and color data will be preserved by default during future cleaning.
-8. Cleaner output must eventually be independently inspected and verified.
+7. JPEG cleaning preserves ICC, unknown APP, and rendering/container segments by default.
+8. Cleaner output is re-inspected before it is returned.
 9. Metadata absence never proves an image contains no private information.
 10. Steganography detection, malware scanning, visual redaction, and pixel privacy analysis are outside scope.
 
@@ -40,3 +40,11 @@ Every traversal or decoding loop has a validated finite count or advances a boun
 ## Environment and dependencies
 
 Core code is local-only and side-effect-free. It has no network, analytics, telemetry, filesystem, DOM, or pixel-codec behavior. The package has zero runtime dependencies.
+
+## JPEG cleaning properties
+
+Cleaning proceeds only after bounded traversal reaches EOI. Truncated lengths, invalid marker structure, unterminated scans, and segment-limit failures produce a typed `IncompleteJpegError`; no partial output is returned. TIFF validity is not required to remove a structurally bounded EXIF APP1.
+
+Removal uses checked, non-overlapping parser ranges. Output length is a safe integer no larger than input length, one output buffer is allocated, and retained ranges are copied in original order. Entropy-coded bytes, restart markers, retained marker fill, structural segments, and bytes after EOI are neither decoded nor regenerated. Exact `Uint8Array` views are honored and caller input is never mutated.
+
+The default policy preserves every ICC and unknown APP segment. Verification proves only the requested observable container state supported by inspection. It does not prove provenance, byte preservation without an original, absence of unknown metadata, or absence of personal information in pixels or unsupported structures.

src/core/errors.ts

@@ -1,10 +1,15 @@
+import type { Diagnostic } from "./diagnostics.js";
+
 export type SecureMetadataErrorCode =
   | "NOT_IMPLEMENTED"
   | "INVALID_OFFSET"
   | "INVALID_LENGTH"
   | "OUT_OF_BOUNDS"
   | "INVALID_LIMIT"
-  | "INPUT_LIMIT_EXCEEDED";
+  | "INPUT_LIMIT_EXCEEDED"
+  | "UNSUPPORTED_FORMAT"
+  | "INCOMPLETE_JPEG"
+  | "CLEAN_OUTPUT_SIZE_INVALID";
 
 export class SecureMetadataError extends Error {
   override readonly name: string = "SecureMetadataError";
@@ -75,3 +80,31 @@ export class InputLimitExceededError extends SecureMetadataError {
     );
   }
 }
+
+export class UnsupportedFormatError extends SecureMetadataError {
+  override readonly name: string = "UnsupportedFormatError";
+
+  constructor(
+    readonly operation: "cleanMetadata" | "verifyMetadata",
+    readonly format: "png" | "webp" | "unknown",
+  ) {
+    super(
+      `${operation} does not support ${format} input.`,
+      "UNSUPPORTED_FORMAT",
+    );
+  }
+}
+
+export class IncompleteJpegError extends SecureMetadataError {
+  override readonly name: string = "IncompleteJpegError";
+
+  constructor(
+    readonly operation: "cleanMetadata" | "verifyMetadata",
+    readonly diagnostics: readonly Diagnostic[],
+  ) {
+    super(
+      `${operation} requires a structurally complete JPEG ending at EOI.`,
+      "INCOMPLETE_JPEG",
+    );
+  }
+}

src/core/limits.ts

@@ -1,3 +1,5 @@
+import { InvalidParseLimitError } from "./errors.js";
+
 export interface ParseLimits {
   readonly maxInputBytes: number;
   readonly maxSegments: number;
@@ -21,3 +23,14 @@ export const DEFAULT_PARSE_LIMITS: Readonly<ParseLimits> = Object.freeze({
   maxDecompressedBytes: 16 * 1024 * 1024,
   maxDiagnostics: 256,
 });
+
+export function resolveParseLimit(
+  name: keyof ParseLimits,
+  configured: number | undefined,
+): number {
+  const value = configured ?? DEFAULT_PARSE_LIMITS[name];
+  if (!Number.isSafeInteger(value) || value < 0) {
+    throw new InvalidParseLimitError(name, value);
+  }
+  return value;
+}

src/core/types.ts

@@ -93,23 +93,54 @@ export interface MetadataReport {
 }
 
 export interface CleaningPolicy {
+  readonly removeExif?: boolean;
+  readonly removeXmp?: boolean;
+  readonly removeIptc?: boolean;
+  readonly removeComments?: boolean;
+  readonly preserveIcc?: boolean;
+  /** @deprecated Use preserveIcc. */
   readonly preserveColorProfiles?: boolean;
   readonly limits?: Partial<ParseLimits>;
 }
 
+export interface MetadataChange {
+  readonly namespace: MetadataNamespace;
+  readonly action: "removed" | "preserved";
+  readonly name: string;
+  readonly source: MetadataSource;
+}
+
 export interface CleanResult {
   readonly output: Uint8Array;
+  readonly format: "jpeg";
   readonly report: MetadataReport;
-  readonly removedEntryIds: readonly string[];
+  readonly removed: readonly MetadataChange[];
+  readonly preserved: readonly MetadataChange[];
+  readonly diagnostics: readonly Diagnostic[];
 }
 
+export type VerificationExpectation = "absent" | "present" | "ignore";
+
 export interface VerificationPolicy {
+  readonly exif?: VerificationExpectation;
+  readonly xmp?: VerificationExpectation;
+  readonly iptc?: VerificationExpectation;
+  readonly comments?: VerificationExpectation;
+  readonly icc?: VerificationExpectation;
   readonly requireNoPrivacyRelevantMetadata?: boolean;
   readonly limits?: Partial<ParseLimits>;
 }
 
+export interface VerificationCheck {
+  readonly namespace: "exif" | "xmp" | "iptc" | "jpeg-comment" | "icc";
+  readonly expected: Exclude<VerificationExpectation, "ignore">;
+  readonly actual: "absent" | "present";
+  readonly passed: boolean;
+}
+
 export interface VerificationResult {
   readonly valid: boolean;
+  readonly checks: readonly VerificationCheck[];
   readonly report: MetadataReport;
   readonly diagnostics: readonly Diagnostic[];
 }

src/index.ts

@@ -1,13 +1,18 @@
 export { inspectMetadata } from "./inspect.js";
-export { cleanMetadata } from "./policy/clean.js";
-export { verifyMetadata } from "./verify/verify.js";
+export { cleanMetadata, DEFAULT_JPEG_CLEANING_POLICY } from "./policy/clean.js";
+export {
+  DEFAULT_JPEG_VERIFICATION_POLICY,
+  verifyMetadata,
+} from "./verify/verify.js";
 
 export {
   BinaryBoundsError,
+  IncompleteJpegError,
   InputLimitExceededError,
   InvalidParseLimitError,
   NotImplementedError,
   SecureMetadataError,
+  UnsupportedFormatError,
 } from "./core/errors.js";
 export { DEFAULT_PARSE_LIMITS } from "./core/limits.js";
 
@@ -29,6 +34,7 @@ export type {
   InspectionStatus,
   InspectOptions,
   MetadataCategory,
+  MetadataChange,
   MetadataContainer,
   MetadataEntry,
   MetadataNamespace,
@@ -37,6 +43,8 @@ export type {
   MetadataValue,
   PrivacyRelevance,
   RationalValue,
+  VerificationCheck,
+  VerificationExpectation,
   VerificationPolicy,
   VerificationResult,
 } from "./core/types.js";