diff --git a/AGENTS.md b/AGENTS.md index c30149d5..1ad44c46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -70,7 +70,10 @@ Dependency edges (a crate depends on those to its right): hostile input, ancillary metadata surfaced as raw `MetadataBlock`-ready payloads (eXIf/iCCP/XMP/text) plus parsed gAMA/cHRM/sRGB/cICP. APNG out of scope (decodes as the default image). Differential oracle both directions: libpng, which also *generates* the - decoder's conformance fixtures. ← core, deflate (+ `miniz_oxide` for inflate). + decoder's conformance fixtures. ← core, deflate (+ `miniz_oxide` for inflate, and + **maintainer-approved `crc32fast`** for the chunk CRC that every encode pays on its critical + path — hardware CRC-32 on x86-64/aarch64, table fallback elsewhere including wasm32, and it + keeps its `unsafe` to itself, so gamut-png stays `#![deny(unsafe_code)]`). - **gamut-ifd** — TIFF/IFD container core (byte order, field types, IFD read/write); a low-level container primitive (sibling to bitstream), shared by `gamut-tiff` and EXIF metadata. ← core. Optional `bigtiff` feature adds 64-bit BigTIFF. Per-format metadata diff --git a/Cargo.lock b/Cargo.lock index fcdd8e15..cd522af1 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -833,9 +833,12 @@ dependencies = [ name = "gamut-png" version = "0.1.0" dependencies = [ + "crc32fast", + "divan", "gamut-codec-abi", "gamut-core", "gamut-deflate", + "gamut-png", "libpng-oracle", "miniz_oxide", ] diff --git a/README.md b/README.md index 57a77973..4a321e3c 100644 --- a/README.md +++ b/README.md @@ -214,6 +214,8 @@ cargo test --workspace | `mise run lint-fix` | Lint and auto-fix | | `mise run check-commits` | Check commits are Conventional Commits | | `mise run coverage` | Run tests with coverage (min 80%) | +| `mise run bench` | Run performance benchmarks (Divan; see [docs/benchmarking.md](docs/benchmarking.md)) | +| `mise run bench-test` | Run every bench once to prove it still executes (no timings) | | `mise run check-cross ` | Cross-compile-check the libs for a target (extended CI; master/manual) | | `mise run check-msrv` | Check the libs compile on the documented MSRV (extended CI; master/manual) | | `mise run versions` | List every crate's version | diff --git a/crates/gamut-cli/src/commands/inspect.rs b/crates/gamut-cli/src/commands/inspect.rs index 7debfb0b..dbb0bc7a 100644 --- a/crates/gamut-cli/src/commands/inspect.rs +++ b/crates/gamut-cli/src/commands/inspect.rs @@ -1,9 +1,40 @@ -//! `gamut inspect` — strict "deconstruct" of a TIFF or DNG (issues #197/#263). +//! `gamut inspect` — strict "deconstruct" of a TIFF, DNG or PNG (issues #197/#263/#224). //! //! Walks the entire container, classifies every byte into typed segments, and flags anything //! unrecognised (unknown tags, unknown field types, out-of-spec codes, unclassified bytes). //! Prints a report to stdout and exits non-zero when the file is not fully accounted for — //! usable as an archival CI gate. +//! +//! # What "fully accounted for" means, and what the exit code is +//! +//! Exit 0 is the file having nothing the walk can hold against it; exit 1 is a finding. Each +//! format states that in its own vocabulary, and the two are deliberately the same strength: +//! +//! - **TIFF / DNG** — `is_fully_accounted()`: every byte classified, *and* no unknown field +//! type, no unknown tag, and no anomaly. +//! - **PNG** — `is_verified()`: `is_intact()` (every byte classified, every chunk CRC valid, IEND +//! present, no trailing bytes after it, no truncated tail, nothing the filter scan found +//! damaging) *and* the filter scan actually ran. +//! +//! PNG's `is_fully_classified()` is **not** the gate, though it is printed: it is true by +//! construction for every file `deconstruct` accepts (a truncated tail and a trailer each get a +//! segment of their own, so the tiling still covers the file), and gating on it would exit 0 on a +//! truncated PNG. It exists so that a walk *bug* makes the predicate false. +//! +//! `is_intact()` is **not** the gate either, and the difference is the reason `is_verified` exists. +//! A PNG whose filter scan was skipped for budget is not *damaged* — nothing is known to be wrong +//! with it — so it is not a finding, and `intact: yes` is printed truthfully. But a corrupt zlib +//! payload under a valid CRC is damage only the scan can see, so an unread file is one this +//! command cannot vouch for, and exiting 0 on it would report this reader's budget as a property +//! of the file. Such a file exits non-zero saying it was not verified, distinctly from a damaged +//! one. To keep that rare, the walk's budget here is a gigabyte rather than the decoder's 64 MiB, +//! which is past any real image — at the decoder's budget every PNG over 4096x4096 RGBA8 would go +//! unread. +//! +//! For PNG the same walk answers a second question: **where did the bytes go?** The report carries +//! the per-chunk-type breakdown, the compressed IDAT total against the filtered stream it inflates +//! to, and the scanline filter distribution — which is what makes an encoder comparison possible +//! from the command line, on files this crate did not write. use std::path::PathBuf; @@ -21,7 +52,7 @@ const DNG_VERSION_TAG: u16 = 50706; /// Arguments for `gamut inspect`. #[derive(Args)] pub(crate) struct InspectArgs { - /// Input TIFF or DNG file. + /// Input TIFF, DNG or PNG file. input: PathBuf, /// Force the container format instead of auto-detecting it. #[arg(long, value_enum)] @@ -35,6 +66,8 @@ pub(crate) enum Format { Tiff, /// DNG (Adobe Digital Negative; gamut-dng). Dng, + /// PNG (gamut-png). + Png, } /// A format-agnostic view of a deconstruct report, for printing. @@ -56,9 +89,17 @@ pub(crate) fn run(args: &InspectArgs) -> Result<(), CliError> { })?; let format = args.format.unwrap_or_else(|| sniff(&data)); + // PNG's report is a different shape -- it has no IFD tree and no tag vocabulary, but it does + // carry compression figures the others have no equivalent for -- so it prints on its own path + // rather than being flattened into `Summary`. + if matches!(format, Format::Png) { + return inspect_png(&args.input, &data); + } + let summary = match format { Format::Dng => summarize_dng(gamut::dng::deconstruct(&data)?), Format::Tiff => summarize_tiff(gamut::tiff::deconstruct(&data)?), + Format::Png => unreachable!("handled above"), }; print_summary(&args.input, format, &summary); @@ -77,8 +118,15 @@ pub(crate) fn run(args: &InspectArgs) -> Result<(), CliError> { } } -/// Detects DNG vs TIFF: a DNG is a TIFF whose IFD 0 carries the mandatory `DNGVersion` tag. +/// The 8-byte PNG file signature (§5.2). +const PNG_SIGNATURE: [u8; 8] = [0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]; + +/// Detects PNG by signature, then DNG vs TIFF: a DNG is a TIFF whose IFD 0 carries the mandatory +/// `DNGVersion` tag. fn sniff(data: &[u8]) -> Format { + if data.starts_with(&PNG_SIGNATURE) { + return Format::Png; + } if let Ok(file) = gamut::tiff::read(data) && file .ifds @@ -326,15 +374,216 @@ fn print_ranges(label: &str, ranges: &[(u64, u64)]) { /// Prints a pre-formatted line list under `label`, truncating past [`MAX_LIST`]. fn print_lines(label: &str, lines: &[String]) { - if lines.is_empty() { + print_lines_of(label, lines, lines.len()); +} + +/// [`print_lines`], where `lines` is already truncated and `total` is how many there really are. +/// +/// Splitting the count from the list is what lets a caller whose list length is chosen by the +/// input build only the lines it will print while still reporting the true total. +fn print_lines_of(label: &str, lines: &[String], total: usize) { + if total == 0 { return; } - println!(" {label}: {}", lines.len()); + println!(" {label}: {total}"); for line in lines.iter().take(MAX_LIST) { println!(" - {line}"); } - if lines.len() > MAX_LIST { - println!(" … and {} more", lines.len() - MAX_LIST); + if total > lines.len() { + println!(" … and {} more", total - lines.len()); + } +} + +/// Deconstructs a PNG and prints where its bytes went, exiting non-zero when the file is not a +/// complete, undamaged datastream. +fn inspect_png(path: &std::path::Path, data: &[u8]) -> Result<(), CliError> { + use gamut::png::{FilterScan, FilterType, SegmentKind}; + + // Inspection budgets differently from decoding. `gamut::png::deconstruct`'s default matches + // the *decoder*'s, which guards a decode against hostile input; but a file this command + // declines to inflate is a file it cannot verify, and at the decoder's 64 MiB that is every + // PNG past 4096x4096 RGBA8 -- an ordinary photograph. Reading it is the whole job, so the + // ceiling is raised to a gigabyte: past any real image, short of unbounded. + let limits = gamut::png::DeconstructLimits::default().with_max_image_bytes(1 << 30); + let report = gamut::png::deconstruct_with_limits(data, limits)?; + let header = report.header; + + println!("{}: PNG", path.display()); + println!( + " image: {}x{} {:?} depth {}{}", + header.width, + header.height, + header.color_type, + header.bit_depth, + if header.interlaced { + ", Adam7 interlaced" + } else { + "" + } + ); + println!( + " size: {} bytes ({:.3} bits/pixel)", + report.file_len, + report.bits_per_pixel() + ); + println!( + " IDAT: {} bytes compressed from {} filtered ({:.1}%)", + report.idat_compressed, + report.filtered_len, + report.idat_ratio() * 100.0 + ); + println!( + " overhead: {} bytes, of which {} is chunk framing", + report.overhead_bytes(), + report.framing_bytes() + ); + + // Truncated like every other list here: a chunk type is four unvalidated bytes, so the number + // of distinct types is chosen by the input, not by the image. + println!(" chunks: {}", report.chunks.len()); + for stats in report.chunks.iter().take(MAX_LIST) { + println!( + " {} x{:<3} {:>9} payload + {:>4} framing{}", + String::from_utf8_lossy(&stats.chunk_type), + stats.count, + stats.payload_bytes, + stats.framing_bytes(), + if stats.is_ancillary() { + " (ancillary)" + } else { + "" + } + ); + } + if report.chunks.len() > MAX_LIST { + println!(" … and {} more", report.chunks.len() - MAX_LIST); + } + + match report.filters { + FilterScan::Counted(h) => { + let n = |f| h.count(f); + println!( + " filters: None {} / Sub {} / Up {} / Average {} / Paeth {} ({} scanlines)", + n(FilterType::None), + n(FilterType::Sub), + n(FilterType::Up), + n(FilterType::Average), + n(FilterType::Paeth), + h.total() + ); + } + FilterScan::Skipped(reason) => { + println!( + " filters: not counted — {}", + filter_skip_label(reason) + ); + } + } + + if report.passes.len() > 1 { + println!(" Adam7 passes:"); + for pass in &report.passes { + println!( + " {}: {}x{}, {} row bytes, {} filtered", + pass.index, pass.width, pass.height, pass.row_bytes, pass.filtered_len + ); + } + } + + // One damaged chunk yields one `String`, and the chunk count is chosen by the input, so the + // list is built under the same bound it is printed under: the total is counted separately and + // only the lines that will be shown are ever materialized. + let is_damaged_segment = |seg: &gamut::png::Segment| { + matches!( + seg.kind, + SegmentKind::Chunk { crc_ok: false, .. } + | SegmentKind::Truncated + | SegmentKind::Trailer + ) + }; + let mut findings = report + .segments + .iter() + .filter(|seg| is_damaged_segment(seg)) + .count(); + let mut damaged: Vec = report + .segments + .iter() + .filter(|seg| is_damaged_segment(seg)) + .take(MAX_LIST) + .map(|seg| match seg.kind { + SegmentKind::Chunk { chunk_type, .. } => format!( + "CRC mismatch in {} at offset {}", + String::from_utf8_lossy(&chunk_type), + seg.range.start + ), + SegmentKind::Truncated => format!( + "truncated from offset {} ({} bytes)", + seg.range.start, + seg.range.len() + ), + _ => format!( + "{} trailing bytes after IEND at offset {}", + seg.range.len(), + seg.range.start + ), + }) + .collect(); + // A skip the file itself caused is damage. An over-budget skip is not — nothing is known to be + // wrong with the file — but it is still a reason this command cannot vouch for it, which is a + // separate question the verdict below keeps separate. + if let FilterScan::Skipped(reason) = report.filters + && reason.is_damage() + { + findings += 1; + if damaged.len() < MAX_LIST { + damaged.push(format!( + "filters not counted — {}", + filter_skip_label(reason) + )); + } + } + print_lines_of("findings", &damaged, findings); + + println!(" classified: {}", yes_no(report.is_fully_classified())); + println!(" intact: {}", yes_no(report.is_intact())); + println!(" verified: {}", yes_no(report.is_verified())); + + // The gate is `is_verified`, not `is_intact`. `is_intact` is "nothing is known against this + // file", which a file whose IDAT was never inflated satisfies without anything having been + // read — and a corrupt zlib payload under a valid CRC is exactly the damage only the scan + // sees. An archival gate that passed such a file would be reporting the reader's budget as a + // property of the file. + if report.is_verified() { + Ok(()) + } else if report.is_intact() { + Err(CliError::NotFullyAccounted(format!( + "{}: not verified — {}", + path.display(), + report + .filters + .skipped() + .map_or("the filter scan did not run", filter_skip_label) + ))) + } else { + Err(CliError::NotFullyAccounted(format!( + "{}: not a complete, undamaged PNG datastream — {findings} finding(s)", + path.display(), + ))) + } +} + +/// Renders why a PNG's scanline filters were not counted. +fn filter_skip_label(reason: gamut::png::SkippedFilterScan) -> &'static str { + use gamut::png::SkippedFilterScan as Reason; + match reason { + Reason::OverBudget => "the image is larger than the reader's byte budget", + Reason::CorruptStream => "the IDAT stream is corrupt or truncated", + Reason::LengthMismatch => "the IDAT stream inflated to the wrong length", + Reason::UndefinedFilterCode => "a scanline carries an undefined filter code", + // `SkippedFilterScan` is non-exhaustive; describe future reasons generically. They are + // damage by default, so the finding is still raised. + _ => "the scan could not be trusted", } } @@ -343,6 +592,7 @@ fn format_name(format: Format) -> &'static str { match format { Format::Tiff => "TIFF", Format::Dng => "DNG", + Format::Png => "PNG", } } diff --git a/crates/gamut-cli/src/main.rs b/crates/gamut-cli/src/main.rs index debb0c3a..10000b26 100644 --- a/crates/gamut-cli/src/main.rs +++ b/crates/gamut-cli/src/main.rs @@ -62,7 +62,7 @@ struct Cli { enum Command { /// Decode an image (PNG/JPEG/PPM/WebP/JXL) and re-encode it as AVIF/WebP/TIFF/PNG/JXL/JPEG. Convert(commands::convert::ConvertArgs), - /// Strictly deconstruct a TIFF or DNG: account every byte and flag unknowns (gamut-tiff/gamut-dng). + /// Strictly deconstruct a TIFF, DNG or PNG: account every byte, flag unknowns, and for PNG report where the bytes went (gamut-tiff/gamut-dng/gamut-png). Inspect(commands::inspect::InspectArgs), /// Extract and inspect the embedded ICC colour profile of an image (gamut-icc). Icc(commands::icc::IccArgs), diff --git a/crates/gamut-png/Cargo.toml b/crates/gamut-png/Cargo.toml index 2abaa9d2..bf09549b 100644 --- a/crates/gamut-png/Cargo.toml +++ b/crates/gamut-png/Cargo.toml @@ -15,6 +15,14 @@ categories.workspace = true [lints] workspace = true +[features] +# Re-exports the encoder's pipeline stages (`src/stages.rs`) so `benches/encode.rs` -- an external +# crate, which can only see `pub` -- can time them one at a time (issue #224). Additive, +# `doc(hidden)`, no SemVer guarantee, and never enabled by the `gamut` umbrella, so the shipped +# surface and `mise run check-ffi-features` are unaffected. The module is re-exports only, so it +# adds no coverage regions and no mutants. +test-support = [] + [dependencies] gamut-core.workspace = true # The shared codestream-backend seam (issue #272): the `repr(C)` vtable + fallback contract the @@ -26,6 +34,11 @@ gamut-deflate.workspace = true # is deliberately encoder-only, and its docs bless miniz_oxide as the decode-side inflate (the same # choice gamut-dng made); revisiting an in-house inflater is tracked by issue #196. miniz_oxide = "0.8" +# CRC-32 (ISO-HDLC) for every chunk, IDAT included, so it is on the critical path of every encode. +# Hardware-accelerated (x86-64 PCLMULQDQ/AVX-512, aarch64 `crc32`) with a table fallback elsewhere, +# including wasm32. MIT/Apache-2.0, pure Rust, and it keeps its `unsafe` to itself -- gamut-png +# stays 100% safe Rust. Replaces a hand-written byte-at-a-time table loop; see `src/crc32.rs`. +crc32fast = "1.5" [dev-dependencies] # Differential cross-check oracle: a vendored, statically-linked libpng (built from the @@ -33,3 +46,12 @@ miniz_oxide = "0.8" # directions: it decodes the gamut encoder's output, and it generates the fixture corpus (and the # reference pixels) the gamut decoder is differentially checked against. libpng-oracle = { path = "../../tooling/libpng-oracle" } +# Benchmark harness (issue #149) plus this crate's own `test-support` feature, which the bench +# target needs to reach the pipeline stages. A self dev-dependency is the standard way to enable an +# own feature for tests and benches only; `mise run check-release-deps` skips self-edges. +divan.workspace = true +gamut-png = { path = ".", features = ["test-support"] } + +[[bench]] +name = "encode" +harness = false diff --git a/crates/gamut-png/README.md b/crates/gamut-png/README.md index 20985453..ab58f048 100644 --- a/crates/gamut-png/README.md +++ b/crates/gamut-png/README.md @@ -59,7 +59,12 @@ A differential oracle (`tooling/libpng-oracle`, a vendored static libpng) proves libpng decodes the encoder's output pixel-exact, and a libpng *reference encoder* generates the decoder's conformance fixtures (interlaced, sub-byte, forced-filter, metadata-laden) which both decoders must read identically — no vendored image corpus. A hand-crafted malformed-input corpus -pins the rejection policy, and output size is benchmarked against libpng at maximum compression. +pins the rejection policy. Output size is measured against libpng at zlib level 9 by +`cargo bench -p gamut-png`, and **enforced** by `tests/size_contract.rs`, whose per-case budgets +each carry a written justification — a regression in the crate's reason to exist fails the build. +`STATUS.md` records the measured table; gamut is smaller than libpng-9 on every corpus entry, by +28-85% wherever a reduction or a filter choice applies and by 0.2% on the incompressible noise row, +where there is nothing for either encoder to find. ## License diff --git a/crates/gamut-png/STATUS.md b/crates/gamut-png/STATUS.md index 23faf5c6..e463b63a 100644 --- a/crates/gamut-png/STATUS.md +++ b/crates/gamut-png/STATUS.md @@ -13,7 +13,8 @@ space optimisation behind the same chunk spine and CRC. FFI), in both directions: libpng decodes the encoder's output → pixel-exact with the source, and a libpng reference-encode entry point generates the decoder's conformance fixtures (interlaced, sub-byte, forced-filter, metadata-laden) that gamut-png and libpng must decode identically. Output -size is benchmarked against libpng at maximum compression. +size is measured against libpng at zlib level 9 by `cargo bench -p gamut-png` and enforced by +`tests/size_contract.rs` (see [Efficiency](#efficiency-issue-224)). **Out of scope:** Adam7 *encoding*, animation/APNG (gamut is image-first; the decoder reads an APNG's default image). Format-agnostic pixel conversion (grey↔RGB, alpha, 16↔8-bit) is @@ -37,6 +38,7 @@ opts into narrowing. That is distinct from the encoder's *lossless* auto-reduce | P8 | §11.3 | Metadata: eXIf, iCCP (deflate-compressed), iTXt-XMP (raw-bytes setters) | ✅ done | | P9 | §4.5 | **Space opt:** lossless palette/gray/alpha-drop reduction (size-estimate chosen) + brute-force filter strategy; extended to grey/grey-alpha/16-bit inputs with lossless 16→8 demotion and sub-byte grey packing (#338) | ✅ done | | P10 | — | CLI `gamut convert → .png`; umbrella `png` feature; final API review | ✅ done | +| E1 | #224 | **Efficiency:** `deconstruct` byte accounting; divan size/bpp + per-stage bench; libpng-9 size contract; opt-in transparent cleanup; palette-vs-native race; `crc32fast` and restructured filter kernels (see [Efficiency](#efficiency-issue-224)) | ✅ done | ## Decoder phases (issue #249) @@ -49,3 +51,128 @@ opts into narrowing. That is distinct from the encoder's *lossless* auto-reduce | D5 | §11.3 | Rich `decode()` → `DecodedPng`: raw eXIf/iCCP/XMP/text payloads (MetadataBlock-ready), parsed gAMA/cHRM/sRGB/cICP, metadata inflation budget | ✅ done | | D6 | — | libpng differential conformance suite over generated fixtures; malformed-input rejection corpus; mutation-gap closure | ✅ done | | D7 | §5, §11.3 | Pixel-free metadata entry point (issue #379): `metadata()` / `PngDecoder::metadata()` → `PngMetadata`, sharing one chunk-classification predicate with `decode()`; IDAT skipped by length, never read or inflated. Mirrors `gamut_jpeg::metadata` / `gamut_webp::metadata` | ✅ done | + +## Efficiency (issue #224) + +Correctness was settled long before efficiency was measured. This section is the measured state: +what the encoder achieves, what it costs, and — per axis — what it does not do yet. + +Everything here is produced by `cargo bench -p gamut-png`. What is *gated* is narrower, and +worth being precise about: `tests/size_contract.rs` asserts the size table -- every row including +`tiny_rgb8` and both `+clean` columns -- as a ratio against libpng-9 at 128×128, and pins +`with_transparent_cleanup` never costing bytes on any row. The throughput and per-heuristic tables +below are **reported, not gated**: timings cannot fail a build without making it flaky, which is +why CI runs the benches for compile-rot only ([#437]). One machine, so **read the ratios, not the +absolute times**. + +### Output size vs libpng at zlib level 9 + +256×256 unless noted, gamut at `Level::Best` + `FilterStrategy::BruteForce` + auto-reduce. +`+clean` additionally enables `with_transparent_cleanup`. Lower is better. + +| input | raw | default | best | +clean | libpng-9 | best/lp9 | bpp | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `gradient_rgb8` | 196 608 | 2 831 | 1 562 | 1 562 | 2 393 | **−34.7%** | 0.191 | +| `photo_rgb8` | 196 608 | 29 885 | 19 570 | 19 570 | 27 467 | **−28.8%** | 2.389 | +| `noise_rgb8` | 196 608 | 196 983 | 196 983 | 196 983 | 197 280 | −0.2% | 24.046 | +| `grey_as_rgb8` | 196 608 | 721 | 368 | 368 | 566 | **−35.0%** | 0.045 | +| `palette64_rgba8` | 262 144 | 1 274 | 726 | 688 | 1 102 | **−34.1%** | 0.089 | +| `sprite_rgba8` | 262 144 | 4 181 | 3 729 | **2 235** | 3 889 | −4.1% | 0.455 | +| `flat_rgba8` | 262 144 | 821 | 103 | 103 | 664 | **−84.5%** | 0.013 | +| `tiny_rgb8` (16×16) | 768 | 136 | 119 | 119 | 138 | **−13.8%** | 3.719 | + +gamut is smaller than libpng-9 on every row, though `noise_rgb8` is a 0.2% near-tie rather than a +win: incompressible input leaves both encoders emitting stored blocks, so that row's budget is the +one deliberately set above parity (1.02) and it is excluded from the win assertion. The margin is +thin where no reduction applies +(`gradient`, `tiny`) or nothing is compressible (`noise`), and large where a lawful +representation change is available that libpng does not attempt. + +### Filter heuristics (issue #480) + +`BruteForce` tries every whole-image strategy and keeps the smallest, so the size table above +cannot say *which* heuristic earned the win. IDAT bytes at `Level::Best`, each heuristic alone: + +| input | MinSumAbs | Entropy | Bigrams | winner | +| --- | --- | --- | --- | --- | +| `gradient_rgb8` | 2 215 | 2 215 | **1 505** | Bigrams | +| `photo_rgb8` | 25 364 | 22 427 | **19 513** | Bigrams | +| `noise_rgb8` | 196 890 | 196 890 | 196 890 | tie | +| `grey_as_rgb8` | **475** | 506 | 506 | MinSumAbs | +| `palette64_rgba8` | 990 | 899 | **770** | Bigrams | +| `sprite_rgba8` | **3 672** | 3 857 | 4 062 | MinSumAbs | +| `flat_rgba8` | **573** | 573 | 605 | MinSumAbs | +| `tiny_rgb8` | 79 | 79 | **62** | Bigrams | + +**Bigrams wins four rows by 22–32%; MinSumAbs wins three by 5–6%.** Both stay in the brute-force +set: neither dominates, and the margins run the wrong way to drop either. That matches oxipng +keeping MinSum at `-o 0`/`-o 6` while its default preset leads with Bigrams. + +**Entropy is never the unique winner**, and that is a recorded negative result. It beats MinSumAbs +on the photographic and palette rows but loses to Bigrams on both, and ties MinSumAbs elsewhere. +Since the brute-force set is resolved by taking the smallest, a candidate dominated everywhere +costs a full filter pass and a full DEFLATE for nothing — so it is not in that set. It stays +selectable: eight images is a corpus, not a proof. + +### Throughput + +| stage | before | after | | +| --- | --- | --- | --- | +| `crc32` | 420.8 MB/s | 8.996 GB/s | 21× | +| `filter_image` / None | 497.9 MB/s | 16.26 GB/s | 33× | +| `filter_image` / `Fixed(Paeth)` | 277.1 MB/s | 1.202 GB/s | 4.3× | +| `filter_image` / `MinSumAbs` | 46.7 MB/s | 265.8 MB/s | 5.7× | + +All safe Rust: `crc32fast` keeps its `unsafe` to itself, and the filter gains are structural +(hoisting a loop-invariant branch, equal-length subslices, one `match` per row instead of per +byte) plus removing a sixth redundant filter pass per scanline. + +### Per-axis state + +| # | Axis | State | +| --- | --- | --- | +| 1 | Filter selection | **partial** — MinSumAbs, Entropy and Bigrams per line, plus seven whole-image candidates each fully DEFLATEd. Bigrams is worth 22–32% where it wins (see above). Still missing: per-line trial deflate, `AtomicMin` pruning, and a two-tier cheap-trial codec. [#480] | +| 2 | DEFLATE quality | **good, ~2% behind zopfli**, and honestly documented in `gamut-deflate`. Two contained wins remain: an 8-byte-at-a-time match compare, and `parse_dp`'s single-distance relaxation. [#478], [#479] | +| 3 | Smallest lawful representation | **done** — grey, alpha-drop, ≤256 palette, 16→8, sub-byte, and a `tRNS` colour key for grey/truecolour. The key is worth ~7–9% on a contiguous transparent region, *not* the 25% the raw-byte arithmetic suggests: the alpha plane it removes is usually the most compressible plane in the image. | +| 4 | Palette optimization | **partial** — trailing-opaque `tRNS` trim, plus ordering: transparent entries first (so that trim cuts as far as §11.3.2.1 allows) then by luma. Worth −14.7% on the sprite row against +1.5% on `palette64`. Modified-Zeng ordering and caller-supplied palette cleanup remain. [#482] | +| 5 | Cleaning invisible data | **done** — `with_transparent_cleanup`, opt-in, on every alpha-carrying layout at 8 and 16 bits. Worth **40.1%** on the sprite row, and it is what makes a colour key reachable at all on a source whose invisible pixels carry different unseen colours. It is a *transform*, not a reduction, so it is **raced** rather than assumed: on `palette64_rgba8` cleaning measured −2.3% at 32×32, **+10.7% at 128×128** and −5.2% at 256×256, because zeroing invisible pixels that carry structure destroys bytes DEFLATE was compressing. `cleaned_or_plain` encodes both and keeps the smaller, so the knob can never cost bytes. | +| 6 | Metadata hygiene | **no policy** — the encoder emits exactly what the caller set, and `gamut convert` drops metadata on the PNG path. [#483] | +| 7 | Interlacing | **correctly none.** Adam7 costs 5–20%; out of scope by declaration. | +| 8 | Effort / speed / determinism | Output is byte-reproducible (no time, no randomness, and the one `HashMap` is never iterated). Three independent knobs, no composed dial. No parallelism. [#484] | +| 9 | Correctness / robustness | **covered** — 16-bit, odd dimensions, 1×1, CRC policy, malformed input. | + +### The cost model, and why it is a race + +`reduce::analyze8` chooses by comparing **raw** sizes, which does not predict compressed size when +one candidate's bytes are incompressible and the other's are not. A palette carries a `PLTE` (and +often `tRNS`) that DEFLATE cannot touch, while the pixels it replaces may compress by two orders of +magnitude. Measured on `palette64_rgba8`, whose palette candidate carries a flat 224 bytes of +`PLTE` + `tRNS` (192 + 8 payload, 24 framing) at every size — the fixture's colour count does not +depend on its side: + +| side | emitted | IDAT | PLTE+tRNS emitted | libpng-9 | +| --- | --- | --- | --- | --- | +| 128 | 364 | 307 | — palette declined | 405 | +| 160 | 465 | 408 | — palette declined | 572 | +| 192 | 563 | 506 | — palette declined | 707 | +| 256 | 726 | 445 | 224 | 1 102 | + +The raw-size estimate sees 16 664 against 65 536 and picks the palette by 4× **at every one of +these sizes**. The finished files disagree: the palette's 224 fixed bytes are incompressible while +the pixels they replace compress by two orders of magnitude, so indexing only pays once the image +is large enough to amortise them — the crossover sits between 192 and 256. So +`write_reduced_or_native` encodes both candidates and keeps the smaller, the same way +`FilterStrategy::BruteForce` already resolves filters — no tuned constant, and never worse than +either candidate alone. The three declined rows are the evidence: had the estimate been trusted, +each would have carried a palette and been larger. Only palette reductions pay for the second +encode; greyscale, alpha-drop and 16→8 demotion add no chunks, so for them the raw comparison is +sound. + +[#437]: https://github.com/visualcommons/gamut/issues/437 +[#478]: https://github.com/visualcommons/gamut/issues/478 +[#479]: https://github.com/visualcommons/gamut/issues/479 +[#480]: https://github.com/visualcommons/gamut/issues/480 +[#481]: https://github.com/visualcommons/gamut/issues/481 +[#482]: https://github.com/visualcommons/gamut/issues/482 +[#483]: https://github.com/visualcommons/gamut/issues/483 +[#484]: https://github.com/visualcommons/gamut/issues/484 diff --git a/crates/gamut-png/benches/encode.rs b/crates/gamut-png/benches/encode.rs new file mode 100644 index 00000000..821c5ed4 --- /dev/null +++ b/crates/gamut-png/benches/encode.rs @@ -0,0 +1,447 @@ +//! PNG encode size and throughput benchmarks (issues #224, #149). +//! +//! For a space-efficient encoder two things matter, and they trade against each other: the size it +//! achieves and the time it costs. So `cargo bench -p gamut-png` first prints two tables -- output +//! size and bits-per-pixel against libpng at maximum compression, then where the bytes went stage +//! by stage -- and only then runs the divan throughput benchmarks. +//! +//! Both tables are computed through [`gamut_png::deconstruct`], which reads any PNG whoever wrote +//! it. That is what makes the libpng column a like-for-like comparison rather than two encoders' +//! self-reports, and it is why the stage table can attribute a size difference to filtering, to +//! the colour-type choice, or to DEFLATE. +//! +//! Counters report bytes of *source* pixels per second, so figures are comparable with the other +//! codec suites. Run with `cargo bench -p gamut-png` (or `mise run bench`). The per-stage rows +//! need this crate's `test-support` feature, which its own dev-dependency on itself already +//! enables for every test and bench build -- there is no flag to pass. +//! +//! Intentionally tight: this measures **encoding**, on a generated 8-bit corpus, and nothing else. +//! There is no decode axis -- `PngDecoder`'s cost is a separate question against a separate +//! oracle, and folding the two into one aggregate would let a decode win mask an encode +//! regression. There is no ancillary-chunk axis: a metadata chunk costs its own payload plus +//! twelve bytes of framing, and the one piece of real work in the compressed ones (`iCCP`, +//! `zTXt`) is a `gamut-deflate` call that `gamut-deflate`'s own suite already measures -- none of +//! it is decided by the encoder's pixel path. There is no interlace axis because there is nothing +//! to measure: `ihdr::write` always emits interlace method 0, and Adam7 is a decode-side feature +//! here. What is left -- compression level, filter strategy, auto-reduce, and the corpus itself -- +//! are the four axes the encoder actually chooses between. + +use divan::counter::BytesCount; +use divan::{Bencher, black_box}; +use gamut_core::{Dimensions, EncodeImage, ImageRef, Rgb8, Rgba8}; +use gamut_png::{FilterStrategy, FilterType, Level, PngEncoder, deconstruct}; + +// The corpus lives with the size contract that asserts against it, so the budgets in +// `tests/size_contract.rs` and the table printed here can never describe different pixels. +#[path = "../tests/common/corpus.rs"] +mod corpus; + +use corpus::{ + flat_rgba, gradient_rgb, grey_as_rgb, noise_rgb, palette64_rgba, photo_rgb, sprite_rgba, +}; + +fn main() { + print_size_table(); + print_stage_table(); + print_heuristic_table(); + divan::main(); +} + +/// Side length of the square test images. +/// +/// 256 is the floor that means anything here: RGB at 256x256 is 192 KiB, roughly six times the +/// 32 KiB DEFLATE window, so LZ77 match behaviour is real. A 64x64 image fits *inside* the window +/// and would flatter both encoders equally, hiding the thing being measured. +const SIDE: u32 = 256; + +/// What a corpus entry is: named pixels in one of the two layouts the tables exercise. +enum Pixels { + /// 8-bit RGB, `SIDE x SIDE`. + Rgb(Vec), + /// 8-bit RGBA, `SIDE x SIDE`. + Rgba(Vec), +} + +/// One named corpus entry. +struct Case { + /// Short name, used as the table's row label and the divan argument. + name: &'static str, + /// Image width in pixels. + width: u32, + /// Image height in pixels. + height: u32, + /// The samples. + pixels: Pixels, +} + +impl Case { + /// Raw sample bytes -- the denominator every ratio in the tables is read against. + fn raw_len(&self) -> usize { + match &self.pixels { + Pixels::Rgb(v) | Pixels::Rgba(v) => v.len(), + } + } + + /// libpng's colour-type code for this entry's layout. + fn libpng_color_type(&self) -> u8 { + match self.pixels { + Pixels::Rgb(_) => libpng_oracle::COLOR_RGB, + Pixels::Rgba(_) => libpng_oracle::COLOR_RGBA, + } + } + + /// Encodes with gamut at the given knobs. + fn gamut(&self, level: Level, filter: FilterStrategy, auto_reduce: bool) -> Vec { + self.gamut_with(level, filter, auto_reduce, false) + } + + /// As [`Self::gamut`], with the opt-in transparent-colour cleanup as well. + fn gamut_with( + &self, + level: Level, + filter: FilterStrategy, + auto_reduce: bool, + cleanup: bool, + ) -> Vec { + let encoder = PngEncoder::new() + .with_compression(level) + .with_filter(filter) + .with_auto_reduce(auto_reduce) + .with_transparent_cleanup(cleanup); + let dims = Dimensions::new(self.width, self.height).expect("corpus dimensions are valid"); + let mut out = Vec::new(); + match &self.pixels { + Pixels::Rgb(v) => { + let image = ImageRef::::new(v, dims).expect("buffer matches dimensions"); + encoder.encode_image(image, &mut out).expect("encode"); + } + Pixels::Rgba(v) => { + let image = ImageRef::::new(v, dims).expect("buffer matches dimensions"); + encoder.encode_image(image, &mut out).expect("encode"); + } + } + out + } + + /// Encodes the *same source layout* with libpng at zlib level 9. + /// + /// Deliberately no `palette` option even for palettisable entries: handing libpng a palette + /// would hand it gamut's own reduction, and the comparison would stop measuring anything. + /// libpng's default adaptive filtering is left alone -- that is the honest baseline. + fn libpng9(&self) -> Vec { + let samples = match &self.pixels { + Pixels::Rgb(v) | Pixels::Rgba(v) => v.as_slice(), + }; + libpng_oracle::encode( + samples, + self.width, + self.height, + self.libpng_color_type(), + 8, + &libpng_oracle::EncodeOpts { + compression_level: Some(9), + ..libpng_oracle::EncodeOpts::default() + }, + ) + } +} + +/// The size-table corpus: one entry per axis that actually changes encoder behaviour. +fn corpus() -> Vec { + let rgb = |name, pixels| Case { + name, + width: SIDE, + height: SIDE, + pixels: Pixels::Rgb(pixels), + }; + let rgba = |name, pixels| Case { + name, + width: SIDE, + height: SIDE, + pixels: Pixels::Rgba(pixels), + }; + vec![ + rgb("gradient_rgb8", gradient_rgb(SIDE)), + rgb("photo_rgb8", photo_rgb(SIDE)), + rgb("noise_rgb8", noise_rgb(SIDE)), + rgb("grey_as_rgb8", grey_as_rgb(SIDE)), + rgba("palette64_rgba8", palette64_rgba(SIDE)), + rgba("sprite_rgba8", sprite_rgba(SIDE)), + rgba("flat_rgba8", flat_rgba(SIDE)), + // The regime where the signature and five chunks of framing dominate bits-per-pixel, and + // the only row where `overhead_bytes` is legible. + Case { + name: "tiny_rgb8", + width: 16, + height: 16, + pixels: Pixels::Rgb(gradient_rgb(16)), + }, + ] +} + +/// The knobs the size table reports gamut under: its default, and its smallest-output setting. +const BEST: (Level, FilterStrategy, bool) = (Level::Best, FilterStrategy::BruteForce, true); + +/// Prints output size and bits-per-pixel against libpng at zlib level 9. +fn print_size_table() { + println!( + "\ngamut-png output size, bytes (lower is better); bpp is the whole file over the pixel count:\n\n\ + {:<17} {:>9} {:>9} {:>9} {:>9} {:>9} {:>9} {:>7}", + "input", "raw", "default", "best", "+clean", "libpng-9", "best/lp9", "bpp" + ); + for case in corpus() { + let default = case.gamut(Level::Default, FilterStrategy::MinSumAbs, false); + let best = case.gamut(BEST.0, BEST.1, BEST.2); + // The opt-in cleanup is off in every other column; this one shows what it is worth. + let cleaned = case.gamut_with(BEST.0, BEST.1, BEST.2, true); + let libpng = case.libpng9(); + let delta = (best.len() as f64 / libpng.len().max(1) as f64 - 1.0) * 100.0; + let bpp = |bytes: &[u8]| bytes.len() as f64 * 8.0 / f64::from(case.width * case.height); + println!( + "{:<17} {:>9} {:>9} {:>9} {:>9} {:>9} {:>8.1}% {:>7.3}", + case.name, + case.raw_len(), + default.len(), + best.len(), + cleaned.len(), + libpng.len(), + delta, + bpp(&best), + ); + } +} + +/// Prints where gamut's bytes went, stage by stage -- every column read back out of the encoded +/// file through [`deconstruct`], so the table describes the artefact rather than the encoder's +/// own bookkeeping. +fn print_stage_table() { + println!( + "\nwhere the bytes went (gamut at Level::Best + BruteForce + auto-reduce):\n\n\ + {:<17} {:>14} {:>5} {:>10} {:>10} {:>7} {:>9} filters N/S/U/A/P", + "input", "type", "depth", "filtered", "idat", "deflate", "overhead" + ); + for case in corpus() { + let png = case.gamut(BEST.0, BEST.1, BEST.2); + let report = deconstruct(&png).expect("gamut's own output deconstructs"); + let filters = report.filters.histogram().map_or_else( + || "-".to_string(), + |h| { + let n = |f| h.count(f); + format!( + "{}/{}/{}/{}/{}", + n(FilterType::None), + n(FilterType::Sub), + n(FilterType::Up), + n(FilterType::Average), + n(FilterType::Paeth) + ) + }, + ); + println!( + "{:<17} {:>14} {:>5} {:>10} {:>10} {:>6.1}% {:>9} {}", + case.name, + format!("{:?}", report.header.color_type), + report.header.bit_depth, + report.filtered_len, + report.idat_compressed, + report.idat_ratio() * 100.0, + report.overhead_bytes(), + filters, + ); + } +} + +/// Prints what each per-scanline filter heuristic is worth on its own. +/// +/// `BruteForce` tries them all and keeps the smallest, so the aggregate table above cannot say +/// *which* one earned the win — and that is exactly the question issue #480 asks. oxipng's +/// evidence for demoting libpng's MinSum out of its default preset is a preset table, not +/// published byte counts, so gamut has to measure it on its own corpus. +fn print_heuristic_table() { + println!( + "\nper-scanline filter heuristic, IDAT bytes at Level::Best (lower is better):\n\n\ + {:<17} {:>10} {:>10} {:>10} winner", + "input", "MinSumAbs", "Entropy", "Bigrams" + ); + for case in corpus() { + let of = |filter| { + let png = case.gamut(Level::Best, filter, false); + deconstruct(&png) + .expect("gamut's own output deconstructs") + .idat_compressed + }; + let (msa, ent, big) = ( + of(FilterStrategy::MinSumAbs), + of(FilterStrategy::MinEntropy), + of(FilterStrategy::MinBigrams), + ); + let best = msa.min(ent).min(big); + // A three-way tie is a real outcome on incompressible input, and naming the first + // heuristic the winner there would record a preference the measurement did not find. + let winner = if msa == ent && ent == big { + "tie" + } else if best == msa { + "MinSumAbs" + } else if best == ent { + "Entropy" + } else { + "Bigrams" + }; + println!( + "{:<17} {:>10} {:>10} {:>10} {winner}", + case.name, msa, ent, big + ); + } +} + +fn case_named(name: &str) -> Case { + corpus() + .into_iter() + .find(|c| c.name == name) + .unwrap_or_else(|| panic!("unknown corpus entry {name}")) +} + +#[divan::bench(args = [Level::Fast, Level::Default, Level::Best])] +fn encode_level(bencher: Bencher, level: Level) { + let case = case_named("gradient_rgb8"); + bencher + .counter(BytesCount::new(case.raw_len())) + .bench_local(|| case.gamut(black_box(level), FilterStrategy::MinSumAbs, false)); +} + +#[divan::bench(args = [ + FilterStrategy::None, + FilterStrategy::Fixed(FilterType::Paeth), + FilterStrategy::MinSumAbs, + FilterStrategy::BruteForce, +])] +fn encode_filter_strategy(bencher: Bencher, filter: FilterStrategy) { + let case = case_named("gradient_rgb8"); + bencher + .counter(BytesCount::new(case.raw_len())) + .bench_local(|| case.gamut(Level::Default, black_box(filter), false)); +} + +/// Attributes the whole reduce stage without needing any seam into it: the same image encoded +/// with the analysis on and off. +#[divan::bench(args = [false, true])] +fn encode_auto_reduce(bencher: Bencher, auto_reduce: bool) { + let case = case_named("palette64_rgba8"); + bencher + .counter(BytesCount::new(case.raw_len())) + .bench_local(|| { + case.gamut( + Level::Default, + FilterStrategy::MinSumAbs, + black_box(auto_reduce), + ) + }); +} + +#[divan::bench(args = ["gradient_rgb8", "photo_rgb8", "noise_rgb8", "palette64_rgba8"])] +fn encode_corpus(bencher: Bencher, name: &str) { + let case = case_named(name); + bencher + .counter(BytesCount::new(case.raw_len())) + .bench_local(|| case.gamut(Level::Default, FilterStrategy::MinSumAbs, true)); +} + +/// Reading the accounting back out of a finished file -- the cost every table row pays. +#[divan::bench] +fn deconstruct_a_finished_png(bencher: Bencher) { + let case = case_named("gradient_rgb8"); + let png = case.gamut(Level::Default, FilterStrategy::MinSumAbs, false); + bencher + .counter(BytesCount::new(png.len())) + .bench_local(|| deconstruct(black_box(&png)).expect("deconstruct")); +} + +/// Per-stage rows. Behind `test-support` because a `benches/` target is a separate crate and the +/// encoder's stages are crate-private; see `gamut_png::stages`. +#[cfg(feature = "test-support")] +mod stages { + use gamut_png::stages; + + use super::{Bencher, BytesCount, Case, Pixels, SIDE, black_box, case_named, noise_rgb}; + + /// The filtered stride and row length an RGB8 image of `SIDE` presents. + const BPP: usize = 3; + const ROW_BYTES: usize = SIDE as usize * BPP; + + fn rgb_samples(case: &Case) -> &[u8] { + match &case.pixels { + Pixels::Rgb(v) | Pixels::Rgba(v) => v, + } + } + + #[divan::bench(args = [ + gamut_png::FilterStrategy::None, + gamut_png::FilterStrategy::Fixed(gamut_png::FilterType::Paeth), + gamut_png::FilterStrategy::MinSumAbs, + ])] + fn filter_image(bencher: Bencher, strategy: gamut_png::FilterStrategy) { + let case = case_named("gradient_rgb8"); + let samples = rgb_samples(&case).to_vec(); + bencher + .counter(BytesCount::new(samples.len())) + .bench_local(|| stages::filter_image(black_box(strategy), &samples, ROW_BYTES, BPP)); + } + + #[divan::bench(args = [1u8, 2, 4])] + fn pack_scanlines(bencher: Bencher, depth: u8) { + let samples = vec![1u8; (SIDE * SIDE) as usize]; + bencher + .counter(BytesCount::new(samples.len())) + .bench_local(|| { + stages::pack_scanlines(&samples, SIDE as usize, SIDE as usize, black_box(depth)) + }); + } + + /// Both sides of the auto-reduce early exit: a palettisable image, and one with far more than + /// 256 colours where the scan bails. + #[divan::bench(args = ["palettisable", "too_many_colors"])] + fn analyze8(bencher: Bencher, kind: &str) { + let case = case_named(if kind == "palettisable" { + "palette64_rgba8" + } else { + "photo_rgb8" + }); + let channels = match case.pixels { + Pixels::Rgb(_) => 3, + Pixels::Rgba(_) => 4, + }; + let samples = rgb_samples(&case).to_vec(); + bencher + .counter(BytesCount::new(samples.len())) + .bench_local(|| stages::analyze8(&samples, black_box(channels))); + } + + /// 16-bit analysis, with and without a lawful demotion available: every sample `k * 257` + /// demotes, an arbitrary one does not, and the two take different paths. + #[divan::bench(args = [true, false])] + fn analyze16(bencher: Bencher, demotable: bool) { + let n = (SIDE * SIDE) as usize; + let samples: Vec = (0..n) + .map(|i| { + let v = (i % 256) as u16; + if demotable { v * 257 } else { v * 257 + 1 } + }) + .collect(); + bencher + .counter(BytesCount::new(samples.len() * 2)) + .bench_local(|| stages::analyze16(&samples, black_box(1))); + } + + /// Runs over every IDAT byte, so it is on the critical path of every encode. + #[divan::bench] + fn crc32(bencher: Bencher) { + let data = noise_rgb(SIDE); + bencher + .counter(BytesCount::new(data.len())) + .bench_local(|| { + let mut crc = stages::Crc32::new(); + crc.update(black_box(&data)); + crc.finish() + }); + } +} diff --git a/crates/gamut-png/src/chunk.rs b/crates/gamut-png/src/chunk.rs index 5516e2a9..67e9cbb5 100644 --- a/crates/gamut-png/src/chunk.rs +++ b/crates/gamut-png/src/chunk.rs @@ -4,6 +4,8 @@ //! covers the type and data. All multi-byte integers in PNG are big-endian — the opposite of the //! DEFLATE/zlib payload the IDAT chunks carry. +use core::ops::Range; + use gamut_core::{Error, Result}; use crate::crc32::Crc32; @@ -30,6 +32,10 @@ pub(crate) struct RawChunk<'a> { pub data: &'a [u8], /// Whether the stored CRC-32 (computed over type + data, §5.5) matched. pub crc_ok: bool, + /// The chunk's whole span in the input, framing included: `12 + data.len()` bytes covering + /// the length, type, payload and CRC fields. Single-sourced from the offset the reader + /// already advances, so byte accounting cannot drift from framing. + pub range: Range, } impl RawChunk<'_> { @@ -105,13 +111,23 @@ impl<'a> ChunkReader<'a> { crc.update(data); let crc_ok = crc.finish().to_be_bytes() == stored; self.rest = rest; + let start = self.offset; self.offset += 12 + length as usize; Ok(Some(RawChunk { chunk_type, data, crc_ok, + range: start..self.offset, })) } + + /// The reader's cursor: the offset of the next chunk header, or — after [`next_chunk`] has + /// returned an error — the start of the malformed one. + /// + /// [`next_chunk`]: Self::next_chunk + pub(crate) fn offset(&self) -> usize { + self.offset + } } #[cfg(test)] diff --git a/crates/gamut-png/src/crc32.rs b/crates/gamut-png/src/crc32.rs index 2be972cd..09d2c2c2 100644 --- a/crates/gamut-png/src/crc32.rs +++ b/crates/gamut-png/src/crc32.rs @@ -3,53 +3,47 @@ //! This is the reflected CRC-32 with polynomial `0xEDB88320`, initial value all-ones, and a final //! ones-complement, computed over a chunk's **type and data** (not its length). zlib uses Adler-32, //! never this — so CRC-32 lives in the PNG crate, not in `gamut-deflate`. - -/// Precomputed byte-wise CRC table (built at compile time). -const TABLE: [u32; 256] = build_table(); - -const fn build_table() -> [u32; 256] { - let mut table = [0u32; 256]; - let mut n = 0usize; - while n < 256 { - let mut c = n as u32; - let mut k = 0; - while k < 8 { - c = if c & 1 != 0 { - 0xEDB8_8320 ^ (c >> 1) - } else { - c >> 1 - }; - k += 1; - } - table[n] = c; - n += 1; - } - table -} +//! +//! The arithmetic is [`crc32fast`]'s; this module is the PNG-shaped wrapper over it. The tests +//! below stay as a drift guard: they pin the polynomial this file's doc claims, so swapping the +//! backend for one computing a different CRC-32 variant (Castagnoli, say) fails here rather than +//! silently producing files no decoder accepts. /// An incremental CRC-32 accumulator. -pub(crate) struct Crc32 { - value: u32, -} +/// +/// Delegates to [`crc32fast`], which dispatches to PCLMULQDQ/AVX-512 on x86-64 and the `crc32` +/// instructions on aarch64, falling back to a table elsewhere (wasm32 included). The `unsafe` +/// that needs is entirely inside that crate; nothing here changes. +/// +/// This runs over every byte of every chunk, IDAT included, so it is on the critical path of +/// every encode. The byte-at-a-time table loop it replaces managed roughly 420 MB/s. +pub struct Crc32(crc32fast::Hasher); impl Crc32 { /// Starts a fresh CRC (register initialised to all ones). - pub(crate) fn new() -> Self { - Self { value: 0xFFFF_FFFF } + // No `Default` impl to pair with this: nothing in the crate would call it, so it would be an + // uncovered region and an unkillable mutant -- a delegation no test can reach. `new` is only + // `pub` so `crate::stages` can re-export it to the benchmark driver. + // `allow`, not `expect`: the lint only fires when `test-support` re-exports this type through + // `crate::stages`, so an `expect` is unfulfilled in a default-feature build and fails there + // instead. A `Default` impl would be dead delegation -- nothing in the crate calls it, so it + // would be an uncovered region and a mutant no test could kill. + #[allow( + clippy::new_without_default, + reason = "a Default impl here would be dead delegation: uncovered, and unkillable by any test" + )] + pub fn new() -> Self { + Self(crc32fast::Hasher::new()) } /// Folds `data` into the running CRC. - pub(crate) fn update(&mut self, data: &[u8]) { - let mut crc = self.value; - for &b in data { - crc = TABLE[((crc ^ u32::from(b)) & 0xff) as usize] ^ (crc >> 8); - } - self.value = crc; + pub fn update(&mut self, data: &[u8]) { + self.0.update(data); } /// Finalises the CRC (ones-complement of the register). - pub(crate) fn finish(self) -> u32 { - self.value ^ 0xFFFF_FFFF + pub fn finish(self) -> u32 { + self.0.finalize() } } diff --git a/crates/gamut-png/src/decoded.rs b/crates/gamut-png/src/decoded.rs index 2d1bbea7..903d7c55 100644 --- a/crates/gamut-png/src/decoded.rs +++ b/crates/gamut-png/src/decoded.rs @@ -21,7 +21,7 @@ use crate::inflate; use crate::palette::PngPalette; /// The parsed image header (IHDR, §11.2.1), reported as stored in the file. -#[derive(Debug, Clone, Copy)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[non_exhaustive] pub struct PngHeader { /// Image width in pixels. diff --git a/crates/gamut-png/src/decoder.rs b/crates/gamut-png/src/decoder.rs index cb95bbf0..85547f4e 100644 --- a/crates/gamut-png/src/decoder.rs +++ b/crates/gamut-png/src/decoder.rs @@ -35,7 +35,10 @@ use crate::palette::PngPalette; use crate::{adam7, inflate, pack}; /// Default cap on the decoded sample buffer: 64 MiB, a 4096×4096 RGBA8 image. -const DEFAULT_MAX_IMAGE_BYTES: usize = 64 << 20; +/// +/// `pub(crate)` because [`crate::deconstruct`] reports against the same budget: a file this +/// decoder decodes is one the report walk will inflate to count filters. +pub(crate) const DEFAULT_MAX_IMAGE_BYTES: usize = 64 << 20; /// Default cumulative cap on inflated metadata (iCCP/zTXt/iTXt) payloads: 16 MiB. const DEFAULT_MAX_METADATA_BYTES: usize = 16 << 20; /// The spec's own dimension bound (§11.2.1): width and height are 1 ..= 2³¹ − 1. @@ -360,17 +363,17 @@ impl PngDecoder { "PNG: image exceeds the dimension limit", )); } - let (width, height) = (header.width as usize, header.height as usize); - // Budget the *decoded* representation: one byte per sample below depth 16 (sub-byte - // depths are unpacked), two above. - let bytes_per_sample = if header.bit_depth == 16 { 2 } else { 1 }; - let native_bytes = width - .checked_mul(height) - .and_then(|pixels| pixels.checked_mul(header.color.channels())) - .and_then(|samples| samples.checked_mul(bytes_per_sample)) - .ok_or_else(|| { - Error::invalid_input(env!("CARGO_PKG_NAME"), "PNG: image dimensions overflow") - })?; + // Budget the *decoded* representation, via the one definition of that quantity, so the + // report walk in `crate::deconstruct` cannot come to budget a different one. + let native_bytes = ihdr::native_bytes( + header.width, + header.height, + header.color.channels(), + header.bit_depth, + ) + .ok_or_else(|| { + Error::invalid_input(env!("CARGO_PKG_NAME"), "PNG: image dimensions overflow") + })?; if native_bytes > self.max_image_bytes { return Err(Error::unsupported( env!("CARGO_PKG_NAME"), diff --git a/crates/gamut-png/src/deconstruct.rs b/crates/gamut-png/src/deconstruct.rs new file mode 100644 index 00000000..e70576c6 --- /dev/null +++ b/crates/gamut-png/src/deconstruct.rs @@ -0,0 +1,917 @@ +//! Where a PNG's bytes went (issue #224): every byte of the file classified into a typed +//! [`Segment`], plus the per-stage figures an encoder-efficiency comparison is built from. +//! +//! This is the measurement counterpart to [`crate::PngEncoder`]. It works on **any** PNG, not +//! just this crate's output, so the same numbers can be read off libpng's, oxipng's or +//! zopflipng's files and compared directly: bits per pixel, what the DEFLATE stage achieved in +//! isolation, how many bytes went to chunk framing, and which scanline filters the encoder +//! actually chose. +//! +//! # The every-byte invariant +//! +//! [`PngReport::segments`] is contiguous, non-overlapping, and covers `0..file_len` exactly. +//! It holds by construction, and [`PngReport::is_fully_classified`] re-derives it from the list +//! rather than storing a flag, so a walk bug makes the predicate false instead of silently +//! agreeing with itself. This mirrors [`gamut_isobmff::segments`]'s guarantee for ISOBMFF; PNG's +//! chunk stream needs its own walk (there are no boxes and no `meta` level), but the shape and +//! the names are deliberately the same. +//! +//! # What is an error and what is a finding +//! +//! Deliberately more tolerant than [`crate::PngDecoder::metadata`], and for the reason +//! [`gamut_dng::deconstruct`] gives: a measurement tool that refuses to measure is useless. +//! Unknown ancillary **and critical** chunks, CRC mismatches, a missing IEND, trailing bytes and +//! a truncated tail are all *reported*, never errors. Only a file with no header to report on — +//! bad signature, no first chunk, a first chunk that is not IHDR, or an unparsable IHDR — fails. + +use core::ops::Range; +use std::collections::HashMap; + +use gamut_core::{Error, Result}; + +use crate::chunk::{ChunkReader, RawChunk, SIGNATURE}; +use crate::decoded::PngHeader; +use crate::decoder::DEFAULT_MAX_IMAGE_BYTES; +use crate::filter::FilterType; +use crate::{adam7, ihdr, inflate}; + +/// Chunk framing overhead: 4 length bytes + 4 type bytes + 4 CRC bytes (§5.3). +const FRAMING: usize = 12; + +/// One contiguous run of the input file, tagged by what it holds ([`SegmentKind`]). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Segment { + /// The half-open byte range this segment occupies within the input (`start..end`). + pub range: Range, + /// What the bytes in [`range`](Self::range) are. + pub kind: SegmentKind, +} + +/// What a [`Segment`] holds. +/// +/// Non-exhaustive: a future revision may name a further region (an APNG frame span, say) without +/// a breaking change — match with a wildcard arm. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum SegmentKind { + /// The 8-byte PNG file signature (§5.2). Always the first segment. + Signature, + /// One complete chunk: 4 length bytes, 4 type bytes, the payload, 4 CRC bytes (§5.3), so the + /// segment is always `payload_len + 12` bytes long. + Chunk { + /// The chunk's four-character type, e.g. `*b"IDAT"`. Recognised and unrecognised types + /// alike appear here — critical ones included; the walk never drops a chunk. + chunk_type: [u8; 4], + /// The declared payload length, framing excluded. + payload_len: usize, + /// Whether the stored CRC-32 over type + payload matched (§5.5). A mismatch is reported, + /// never an error: §13.1 makes it recoverable in an ancillary chunk, and the framing is + /// intact either way, so the walk can keep going and account the rest of the file. + crc_ok: bool, + }, + /// Bytes after IEND. Not part of the datastream — §13.2 asks decoders to ignore them, so they + /// are surfaced here rather than silently dropped. + Trailer, + /// From the first chunk header that does not frame — truncated, or declaring a length that + /// overruns the input — to end of file. A file carrying one is not a complete PNG. + Truncated, +} + +/// Per-chunk-type totals, in first-appearance order. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub struct ChunkStats { + /// The chunk's four-character type. + pub chunk_type: [u8; 4], + /// How many chunks of this type the file carries. + pub count: usize, + /// Total payload bytes across those chunks — framing excluded. + pub payload_bytes: usize, +} + +impl ChunkStats { + /// Framing bytes these chunks cost: 12 per chunk (4 length + 4 type + 4 CRC, §5.3). + #[must_use] + pub fn framing_bytes(&self) -> usize { + self.count * FRAMING + } + + /// Payload plus framing — what this chunk type costs the file in total. + #[must_use] + pub fn total_bytes(&self) -> usize { + self.payload_bytes + self.framing_bytes() + } + + /// Whether the type is ancillary — bit 5 of the first byte set, i.e. lowercase (§5.4). + #[must_use] + pub fn is_ancillary(&self) -> bool { + self.chunk_type[0] & 0x20 != 0 + } +} + +/// One reduced image making up the filtered stream: an Adam7 pass (§8.1), or the whole image when +/// the file is not interlaced. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub struct PassStats { + /// Pass index in transmission order (`0..7`); always `0` when the file is not interlaced. + pub index: u8, + /// The reduced image's width in pixels. Never zero: an empty pass carries no bytes at all, + /// not even filter-type bytes (§7.3), so it is omitted entirely. + pub width: u32, + /// The reduced image's height in pixels. Never zero, for the same reason. + pub height: u32, + /// Bytes per scanline excluding the filter-type byte: `ceil(width × bits_per_pixel / 8)`, so + /// a sub-byte depth includes its row padding (§7.2). + pub row_bytes: usize, + /// This pass's contribution to the filtered stream: `height × (1 + row_bytes)`. + pub filtered_len: usize, +} + +/// How many scanlines chose each of the five filters (§9.1), summed over every pass. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct FilterHistogram { + counts: [u32; 5], +} + +impl FilterHistogram { + /// Scanlines that chose `filter`. + #[must_use] + pub fn count(self, filter: FilterType) -> u32 { + self.counts[filter as usize] + } + + /// Total scanlines — the sum over all five filters, and the image's scanline count. + #[must_use] + pub fn total(self) -> u32 { + self.counts.iter().sum() + } +} + +/// The outcome of the walk's optional filter scan: the counts, or why there are none. +/// +/// The scan is the one part of a report that has to inflate the IDAT stream, so it is the one +/// part that can be absent. Which is why the absence is *typed*: "no histogram" conflates a file +/// this reader declined to inflate with a file whose compressed data is broken, and only the +/// second is damage. [`is_damage`](Self::is_damage) answers that question once, for both +/// [`PngReport::is_intact`] and any caller that has to grade a file. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum FilterScan { + /// The IDAT stream inflated to the expected length and every scanline's filter code was read. + Counted(FilterHistogram), + /// No counts, for the stated reason. + Skipped(SkippedFilterScan), +} + +impl FilterScan { + /// The per-filter counts, if the scan ran. + #[must_use] + pub fn histogram(self) -> Option { + match self { + Self::Counted(histogram) => Some(histogram), + Self::Skipped(_) => None, + } + } + + /// Why there are no counts, if there are none. + #[must_use] + pub fn skipped(self) -> Option { + match self { + Self::Counted(_) => None, + Self::Skipped(reason) => Some(reason), + } + } + + /// Whether the scan actually ran, so the counts describe bytes this reader read. + /// + /// The complement of [`is_damage`](Self::is_damage) only for a scan that ran: a skip is + /// either damage or a budget refusal, and **neither is a verification**. A caller grading a + /// file — [`PngReport::is_verified`], an archival gate — asks this; a caller asking whether + /// anything is known to be *wrong* asks `is_damage`. + #[must_use] + pub fn is_counted(self) -> bool { + matches!(self, Self::Counted(_)) + } + + /// Whether the missing counts mean the *file* is damaged — see + /// [`SkippedFilterScan::is_damage`]. A scan that ran is never damage. + #[must_use] + pub fn is_damage(self) -> bool { + match self { + Self::Counted(_) => false, + Self::Skipped(reason) => reason.is_damage(), + } + } +} + +/// Why a [`FilterScan`] carries no counts. +/// +/// `#[repr(u8)]` with explicit discriminants, which are **permanent and append-only**: the value +/// is plain data a C caller reads by number, so a variant is never renumbered or removed. +/// Non-exhaustive — match with a wildcard arm, and prefer [`is_damage`](Self::is_damage) to +/// enumerating the reasons yourself. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[repr(u8)] +#[non_exhaustive] +pub enum SkippedFilterScan { + /// The image the header describes is larger than this reader's byte budget, so the walk + /// declined to inflate a stream a decode would refuse to allocate. **Nothing is known to be + /// wrong with the file** — it may be a perfectly sound very large PNG. + OverBudget = 0, + /// The IDAT stream is not a valid zlib stream, is truncated, or inflates past the length the + /// header implies. + CorruptStream = 1, + /// The stream inflated, but to a different length than the header implies, so the scanline + /// boundaries it describes are not where the filter bytes are. + LengthMismatch = 2, + /// A scanline's leading byte is not one of the five filter codes §9.1 defines. + UndefinedFilterCode = 3, +} + +impl SkippedFilterScan { + /// Whether this reason means the **file** is damaged, rather than merely unread. + /// + /// The single source of truth for that question, so no caller has to re-derive it from the + /// variant list. [`OverBudget`](Self::OverBudget) is the only reason that is not damage: it + /// describes the reader's budget, not the file. Every other reason is a statement about the + /// bytes, and a future reason is damage until it says otherwise. + #[must_use] + pub fn is_damage(self) -> bool { + !matches!(self, Self::OverBudget) + } +} + +/// Where a PNG's bytes went: a total byte accounting plus the figures an encoder-efficiency +/// comparison is built from. Produced by [`deconstruct`]. +/// +/// Non-exhaustive: report categories may be added without a breaking change. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct PngReport { + /// The input's total length in bytes — what [`segments`](Self::segments) together covers. + pub file_len: usize, + /// IHDR: dimensions, bit depth, colour type, interlace method. + pub header: PngHeader, + /// Every byte of the input in file order — contiguous, non-overlapping, covering + /// `0..file_len` exactly. See the [every-byte invariant](self#the-every-byte-invariant). + pub segments: Vec, + /// Per-chunk-type totals, in first-appearance order. + pub chunks: Vec, + /// The concatenated IDAT payload length: the zlib codestream, framing excluded. This is what + /// the encoder's compression stage produced, and the numerator of + /// [`idat_ratio`](Self::idat_ratio). + pub idat_compressed: usize, + /// The length that codestream inflates to — the filter-prefixed scanline stream. Derived from + /// IHDR alone (the sum over [`passes`](Self::passes) when interlaced), so it is known even + /// when [`filters`](Self::filters) was skipped. + pub filtered_len: usize, + /// The reduced images making up the filtered stream: one entry per non-empty Adam7 pass, or + /// exactly one entry for a non-interlaced image. + pub passes: Vec, + /// Scanlines per filter type, or the reason the IDAT stream was not scanned. Everything else + /// in this report is derived from framing and IHDR, so it survives whatever the reason is. + pub filters: FilterScan, +} + +impl PngReport { + /// **The headline law.** Whether the segments tile the input exactly: the first starts at 0, + /// each ends where the next starts, none is empty, and the last ends at `file_len`. + /// Re-derived from [`segments`](Self::segments) rather than stored. + #[must_use] + pub fn is_fully_classified(&self) -> bool { + let mut expected = 0usize; + for segment in &self.segments { + if segment.range.start != expected || segment.range.end <= segment.range.start { + return false; + } + expected = segment.range.end; + } + expected == self.file_len + } + + /// Whether every byte of this file belongs to a complete, undamaged PNG datastream: fully + /// classified, no [`SegmentKind::Truncated`] and no [`SegmentKind::Trailer`], every CRC + /// valid, IEND present, and nothing damaging found by the filter scan. + /// + /// A trailer counts against it even though §13.2 lets a *decoder* ignore trailing bytes, + /// because [`bits_per_pixel`](Self::bits_per_pixel) divides the whole file by the pixel + /// count: bytes outside the datastream still inflate the headline figure, so a size + /// comparison has to know they are there. + /// + /// Independent of whether every chunk type was *recognised* — an unknown critical chunk is + /// still accounted for. The filter conjunct is + /// [`!filters.is_damage()`](FilterScan::is_damage), not "the scan ran": a stream this reader + /// declined to inflate says nothing against the file, while a corrupt zlib payload under a + /// valid CRC is damage **only** the scan can see, so dropping the conjunct would stop + /// detecting it. + #[must_use] + pub fn is_intact(&self) -> bool { + self.is_fully_classified() + && !self.filters.is_damage() + && self.segments.iter().all(|segment| match segment.kind { + SegmentKind::Truncated | SegmentKind::Trailer => false, + SegmentKind::Chunk { crc_ok, .. } => crc_ok, + SegmentKind::Signature => true, + }) + && self.chunk(b"IEND").is_some() + } + + /// Whether this file is intact **and every byte of it was actually read**: `is_intact()` plus + /// [`FilterScan::is_counted`]. + /// + /// The distinction [`is_intact`](Self::is_intact) deliberately does not make. `is_intact` is + /// "nothing is known to be wrong", which a file whose IDAT was never inflated satisfies + /// vacuously — and a corrupt zlib payload under a valid CRC is damage *only* the scan can + /// see, so for an over-budget file `is_intact` is a statement about this reader's budget + /// rather than about the bytes. A gate that must not pass an unread file asks this instead; + /// a caller reporting what is known against a file keeps asking `is_intact`. + #[must_use] + pub fn is_verified(&self) -> bool { + self.is_intact() && self.filters.is_counted() + } + + /// **Stored bits per image pixel** — the space-efficiency figure of merit: the whole file, + /// framing and metadata included, over `width × height`. Distinct from the *uncompressed* + /// rate, which is `header.color_type.channels() × header.bit_depth`. + #[must_use] + pub fn bits_per_pixel(&self) -> f64 { + let pixels = f64::from(self.header.width) * f64::from(self.header.height); + // IHDR rejects a zero dimension, so `pixels >= 1.0` for any report that exists. + self.file_len as f64 * 8.0 / pixels + } + + /// The DEFLATE stage's compression ratio in isolation: `idat_compressed / filtered_len`. + /// Below 1.0 means the codestream compressed. Filtering and colour-type choice are *upstream* + /// of this number, which is what makes it the right lens for attributing a size difference to + /// the compressor rather than to the rest of the encoder. + /// + /// `0.0` when the filtered stream has no length. That is not a dead branch: IHDR admits + /// dimensions whose filtered stream overflows `usize` — 2³¹−1 square at RGBA16 is 2⁶⁵ bytes — + /// and [`deconstruct`] reports such a file rather than refusing it, leaving + /// [`filtered_len`](Self::filtered_len) zero. Thirteen header bytes reach it, so the guard is + /// what keeps `gamut inspect` from dividing by zero on a hostile file. + #[must_use] + pub fn idat_ratio(&self) -> f64 { + if self.filtered_len == 0 { + return 0.0; + } + self.idat_compressed as f64 / self.filtered_len as f64 + } + + /// Every byte that is not IDAT payload: the signature, all chunk framing, and every non-IDAT + /// payload. + #[must_use] + pub fn overhead_bytes(&self) -> usize { + self.file_len - self.idat_compressed + } + + /// Total chunk framing: 12 bytes per chunk in the file. + #[must_use] + pub fn framing_bytes(&self) -> usize { + self.chunks.iter().map(ChunkStats::framing_bytes).sum() + } + + /// The decoded image's byte cost — `width × height × channels`, doubled at depth 16 — or + /// `None` when that overflows `usize`. + /// + /// The quantity a decoder budgets, and the one this walk gates its filter scan on, so a + /// [`SkippedFilterScan::OverBudget`] report is exactly one whose `native_bytes` exceeds the + /// reader's budget. Distinct from [`filtered_len`](Self::filtered_len), which adds one filter + /// byte per scanline and counts sub-byte samples packed. + #[must_use] + pub fn native_bytes(&self) -> Option { + ihdr::native_bytes( + self.header.width, + self.header.height, + self.header.color_type.channels(), + self.header.bit_depth, + ) + } + + /// The stats for one chunk type, if the file carries it. + /// + /// A linear scan of [`chunks`](Self::chunks), so it costs O(distinct chunk types) per call — + /// bounded by the *types* the file carries, not by its chunk count. Looking up a handful of + /// types is what this is for; to summarise every type, iterate [`chunks`](Self::chunks) once + /// rather than calling this per type. + #[must_use] + pub fn chunk(&self, chunk_type: &[u8; 4]) -> Option { + self.chunks + .iter() + .find(|stats| &stats.chunk_type == chunk_type) + .copied() + } +} + +/// Accumulates the per-chunk-type totals of one walk, in time linear in the chunk count. +/// +/// A chunk type is four **unvalidated** bytes — [`crate::chunk`] reads them straight out of the +/// file and the walk never drops a chunk — so a hostile 12-byte-per-chunk file carries one +/// *distinct* type per chunk. Accumulating with a linear `find` over the types seen so far is +/// then quadratic in the file length: 4.8 MB of empty chunks took 40.9 s. The index makes each +/// chunk O(1), and `stats` keeps the first-appearance order [`PngReport::chunks`] documents. +/// +/// The keys are attacker-chosen, which is safe **because** [`HashMap`]'s default hasher is +/// SipHash-1-3 seeded per process: collisions cannot be precomputed against it. Do not swap in a +/// faster unseeded hasher (`FxHash`, `AHash` without a random seed) — that would reopen the +/// quadratic blow-up this type exists to close, by a different route. +struct ChunkTally { + /// One entry per distinct type, in first-appearance order. + stats: Vec, + /// Type → its index in `stats`. Dropped at the end of the walk; never surfaced. + index: HashMap<[u8; 4], usize>, +} + +impl ChunkTally { + /// An empty tally. + fn new() -> Self { + Self { + stats: Vec::new(), + index: HashMap::new(), + } + } + + /// Adds one chunk of `chunk_type` carrying `payload_len` payload bytes. + fn record(&mut self, chunk_type: [u8; 4], payload_len: usize) { + match self.index.get(&chunk_type) { + Some(&at) => { + self.stats[at].count += 1; + self.stats[at].payload_bytes += payload_len; + } + None => { + self.index.insert(chunk_type, self.stats.len()); + self.stats.push(ChunkStats { + chunk_type, + count: 1, + payload_bytes: payload_len, + }); + } + } + } + + /// The accumulated totals, in first-appearance order. + fn into_stats(self) -> Vec { + self.stats + } +} + +/// Classifies every byte of `png` and, where the IDAT stream is sound and within budget, counts +/// the scanline filter each row chose. +/// +/// Pixels are never reconstructed: no defiltering, no unpacking, no de-interlacing, no palette +/// resolution. The walk reads chunk framing and the IHDR, and inflates IDAT only to read one +/// filter byte per scanline. +/// +/// Works on any PNG, whichever encoder produced it, which is what makes the figures comparable +/// across encoders (issue #224). +/// +/// Walks under [`DeconstructLimits::default()`]. Use +/// [`deconstruct_with_limits`] to match a decoder you configured yourself. +/// +/// # Errors +/// +/// Returns [`Error::InvalidInput`] when there is no header to report on — a bad signature, no +/// first chunk, a first chunk that is not IHDR, or an IHDR whose payload is invalid — or when the +/// input carries more chunks than [`DeconstructLimits::max_chunks`] allows. Everything else is +/// **reported, not errored** — unknown ancillary *and critical* chunks, CRC mismatches, a missing +/// IEND, trailing bytes after IEND, a truncated tail, and a corrupt IDAT stream. +pub fn deconstruct(png: &[u8]) -> Result { + deconstruct_with_limits(png, DeconstructLimits::default()) +} + +/// The ceilings a [`deconstruct`] walk observes on attacker-chosen quantities. +/// +/// Every field is a quantity the *input* chooses, which is why each has a ceiling: a report is +/// routinely run over files from anywhere (`gamut inspect` is pointed at whatever is on disk), and +/// the crate's decoder already caps the same quantities for the same reason. +/// +/// Non-exhaustive: ceilings may be added without a breaking change. Build from +/// [`default()`](Self::default) and adjust the fields you care about. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub struct DeconstructLimits { + /// The largest decoded image, in bytes, whose IDAT stream is worth inflating to count + /// filters. Above it the scan is skipped as [`SkippedFilterScan::OverBudget`] and every other + /// figure is still reported, because everything else is derived from framing and IHDR. + /// + /// This is the quantity [`crate::PngDecoder::with_max_image_bytes`] budgets, and matching the + /// two is the point: a report is only "what a decode would have allocated" against a decoder + /// configured the same way. The default matches the decoder's default. + pub max_image_bytes: usize, + /// The largest number of chunks the walk will materialize into segments and per-type stats. + /// + /// A chunk costs 12 bytes of input and buys a `Segment` plus, for a type not seen before, a + /// `ChunkStats` and an index entry — so an input of unbounded chunk count is an input of + /// unbounded heap, at roughly an order of magnitude over the file size. The chunk *type* is + /// four unvalidated bytes, so the distinct-type count is attacker-chosen too. + /// + /// The default admits any plausible real file — a PNG at the ceiling is at least 12 MiB of + /// pure chunk framing — while bounding a crafted one. + pub max_chunks: usize, +} + +/// The chunk-count ceiling a default [`deconstruct`] walk observes. +/// +/// A PNG reaching it carries at least 12 MiB of chunk framing alone, which no real file does and a +/// crafted one reaches cheaply. +pub const DEFAULT_MAX_CHUNKS: usize = 1 << 20; + +impl Default for DeconstructLimits { + fn default() -> Self { + Self { + max_image_bytes: DEFAULT_MAX_IMAGE_BYTES, + max_chunks: DEFAULT_MAX_CHUNKS, + } + } +} + +impl DeconstructLimits { + /// Sets [`max_image_bytes`](Self::max_image_bytes). + /// + /// Builder methods rather than a struct literal, matching + /// [`PngDecoder::with_max_image_bytes`](crate::PngDecoder::with_max_image_bytes) — and + /// necessary as well as symmetrical, since a non-exhaustive struct cannot be built by literal + /// outside this crate at all. + #[must_use] + pub fn with_max_image_bytes(mut self, bytes: usize) -> Self { + self.max_image_bytes = bytes; + self + } + + /// Sets [`max_chunks`](Self::max_chunks). + #[must_use] + pub fn with_max_chunks(mut self, chunks: usize) -> Self { + self.max_chunks = chunks; + self + } +} + +/// [`deconstruct`], under caller-chosen [`DeconstructLimits`]. +/// +/// # Errors +/// +/// As [`deconstruct`], against `limits` rather than the defaults. +pub fn deconstruct_with_limits(png: &[u8], limits: DeconstructLimits) -> Result { + let mut reader = ChunkReader::new(png)?; + let mut segments = vec![Segment { + range: 0..SIGNATURE.len(), + kind: SegmentKind::Signature, + }]; + + let first = reader.next_chunk()?.ok_or_else(|| { + Error::invalid_input(env!("CARGO_PKG_NAME"), "PNG: no chunk after the signature") + })?; + if &first.chunk_type != b"IHDR" { + return Err(Error::invalid_input( + env!("CARGO_PKG_NAME"), + "PNG: first chunk is not IHDR", + )); + } + let native = ihdr::parse(first.data)?; + let header = PngHeader { + width: native.width, + height: native.height, + bit_depth: native.bit_depth, + color_type: native.color, + interlaced: native.interlaced, + }; + + let mut tally = ChunkTally::new(); + let mut idat = Vec::new(); + let mut saw_iend = false; + let push = |segments: &mut Vec, tally: &mut ChunkTally, chunk: &RawChunk| { + segments.push(Segment { + range: chunk.range.clone(), + kind: SegmentKind::Chunk { + chunk_type: chunk.chunk_type, + payload_len: chunk.data.len(), + crc_ok: chunk.crc_ok, + }, + }); + tally.record(chunk.chunk_type, chunk.data.len()); + }; + push(&mut segments, &mut tally, &first); + + loop { + match reader.next_chunk() { + Ok(None) => break, + Ok(Some(chunk)) => { + if &chunk.chunk_type == b"IDAT" { + idat.extend_from_slice(chunk.data); + } + let is_iend = &chunk.chunk_type == b"IEND"; + push(&mut segments, &mut tally, &chunk); + if segments.len() > limits.max_chunks { + return Err(Error::invalid_input( + env!("CARGO_PKG_NAME"), + "PNG: more chunks than the walk's ceiling admits", + )); + } + if is_iend { + saw_iend = true; + break; + } + } + // A header that does not frame ends the datastream; the rest of the file is + // accounted as one opaque run rather than dropped (§13.2's tolerance, extended to + // damage the spec does not describe). + Err(_) => { + // `next_chunk` returns `Ok(None)` when nothing is left, so reaching an error means + // bytes remain and this range is never empty. No guard: a `start < png.len()` + // check here can never be false, which makes it dead code and an equivalent + // mutant rather than a safety net. + let start = reader.offset(); + debug_assert!( + start < png.len(), + "a framing error leaves bytes unaccounted" + ); + segments.push(Segment { + range: start..png.len(), + kind: SegmentKind::Truncated, + }); + break; + } + } + } + if saw_iend && reader.offset() < png.len() { + segments.push(Segment { + range: reader.offset()..png.len(), + kind: SegmentKind::Trailer, + }); + } + + let passes = pass_stats(&native); + let filtered_len = adam7::expected_stream_len(&native).unwrap_or(0); + let filters = scan_filters( + &native, + &idat, + filtered_len, + &passes, + limits.max_image_bytes, + ); + + Ok(PngReport { + file_len: png.len(), + header, + segments, + chunks: tally.into_stats(), + idat_compressed: idat.len(), + filtered_len, + passes, + filters, + }) +} + +/// The reduced images making up the filtered stream, skipping empty passes exactly as +/// [`adam7::expected_stream_len`] does — so `filtered_len` is the sum of these and can be checked +/// against it rather than merely asserted. +fn pass_stats(header: &ihdr::Ihdr) -> Vec { + let mut out = Vec::new(); + let mut total = 0usize; + for (index, pass) in adam7::passes_for(header.interlaced).iter().enumerate() { + let (width, height) = adam7::pass_dimensions(pass, header.width, header.height); + if width == 0 || height == 0 { + continue; + } + let Some(row_bytes) = (width as usize) + .checked_mul(header.bits_per_pixel()) + .map(|bits| bits.div_ceil(8)) + else { + return Vec::new(); + }; + let Some(filtered_len) = row_bytes + .checked_add(1) + .and_then(|stride| (height as usize).checked_mul(stride)) + else { + return Vec::new(); + }; + // `adam7::expected_stream_len` fails on the seven-pass *sum* as well as on each pass, so + // this has to fail with it. Without the running check, a header whose passes each fit but + // whose total overflows leaves `filtered_len` saturated to 0 while `passes` still + // describes all seven -- a self-inconsistent report, and a `0.0%` ratio that reads as a + // measurement rather than as an overflow. + let Some(running) = total.checked_add(filtered_len) else { + return Vec::new(); + }; + total = running; + out.push(PassStats { + index: index as u8, + width, + height, + row_bytes, + filtered_len, + }); + } + out +} + +/// Whether this file's IDAT stream is worth inflating to count filters: whether the image its +/// header describes fits `max_image_bytes`. +/// +/// The budgeted quantity is [`ihdr::native_bytes`] — the decoded buffer — because that is exactly +/// what [`crate::PngDecoder`] budgets, so "a report never allocates more than a decode would" +/// holds by construction. Budgeting the *filtered* stream instead states the same intent over a +/// different number: the two differ by one filter byte per scanline, so a 4096×4096 RGBA8 image +/// is 67 108 864 native bytes (decodes on the default budget) and 67 112 960 filtered — and the +/// report declined to scan a file the decoder decodes, reporting it as damaged. +/// +/// Inflation is still bounded: the filtered stream is at most the native bytes plus one byte per +/// scanline, so a file that passes here inflates to under twice the budget. +/// +/// The budget is a parameter rather than a constant so the boundary is reachable from a unit test +/// without a 64 MiB fixture. +fn fits_decode_budget(header: &ihdr::Ihdr, max_image_bytes: usize) -> bool { + ihdr::native_bytes( + header.width, + header.height, + header.color.channels(), + header.bit_depth, + ) + .is_some_and(|native| native <= max_image_bytes) +} + +/// Inflates the IDAT stream and counts the filter byte leading each scanline. +/// +/// Every early return names its own reason, so a caller can tell a file this reader declined to +/// inflate from one whose compressed data is broken. Every other figure in the report is derived +/// from framing and IHDR, so it survives all of these. +fn scan_filters( + header: &ihdr::Ihdr, + idat: &[u8], + filtered_len: usize, + passes: &[PassStats], + max_image_bytes: usize, +) -> FilterScan { + if !fits_decode_budget(header, max_image_bytes) { + return FilterScan::Skipped(SkippedFilterScan::OverBudget); + } + let Ok(stream) = inflate::inflate_zlib(idat, filtered_len) else { + return FilterScan::Skipped(SkippedFilterScan::CorruptStream); + }; + if stream.len() != filtered_len { + return FilterScan::Skipped(SkippedFilterScan::LengthMismatch); + } + let mut counts = [0u32; 5]; + let mut at = 0usize; + for pass in passes { + for _ in 0..pass.height { + // The pass geometry sums to `filtered_len`, which the stream just matched, so this + // index is in range; a mismatch between the two is the same defect as a short stream. + let Some(&code) = stream.get(at) else { + return FilterScan::Skipped(SkippedFilterScan::LengthMismatch); + }; + let Some(filter) = FilterType::from_code(code) else { + return FilterScan::Skipped(SkippedFilterScan::UndefinedFilterCode); + }; + counts[filter as usize] += 1; + at += 1 + pass.row_bytes; + } + } + FilterScan::Counted(FilterHistogram { counts }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ColorType; + + /// A report with the given segment ranges and file length. Built by hand because + /// [`deconstruct`] cannot produce a malformed tiling: it is correct by construction, so every + /// negative case for [`PngReport::is_fully_classified`] has to be assembled here. That is also + /// why these live inline — the predicate is only falsifiable from inside the crate. + fn report_with(ranges: &[(usize, usize)], file_len: usize) -> PngReport { + PngReport { + file_len, + header: PngHeader { + width: 1, + height: 1, + bit_depth: 8, + color_type: ColorType::Truecolor, + interlaced: false, + }, + segments: ranges + .iter() + .map(|&(start, end)| Segment { + range: start..end, + kind: SegmentKind::Trailer, + }) + .collect(), + chunks: Vec::new(), + idat_compressed: 0, + filtered_len: 0, + passes: Vec::new(), + filters: FilterScan::Skipped(SkippedFilterScan::CorruptStream), + } + } + + #[test] + fn contiguous_segments_covering_the_file_are_fully_classified() { + assert!(report_with(&[(0, 8), (8, 20), (20, 33)], 33).is_fully_classified()); + } + + #[test] + fn a_gap_between_segments_is_not_fully_classified() { + // Every segment is non-empty and the last still reaches `file_len`, so only the + // start-chaining half of the predicate can reject this. + assert!(!report_with(&[(0, 8), (9, 33)], 33).is_fully_classified()); + } + + #[test] + fn an_empty_segment_is_not_fully_classified() { + // The mirror case: the chain is unbroken, so only the non-empty half can reject it. + assert!(!report_with(&[(0, 8), (8, 8), (8, 33)], 33).is_fully_classified()); + } + + #[test] + fn segments_must_start_at_zero_and_reach_the_end() { + assert!(!report_with(&[(4, 33)], 33).is_fully_classified()); + assert!(!report_with(&[(0, 20)], 33).is_fully_classified()); + assert!(!report_with(&[], 33).is_fully_classified()); + // ...and a zero-length file with no segments is vacuously covered. + assert!(report_with(&[], 0).is_fully_classified()); + } + + /// A header for the budget boundary, built directly: `ihdr::parse` would only add a byte + /// layout between the test and the quantity under test. + fn header(width: u32, height: u32, bit_depth: u8, color: ColorType) -> ihdr::Ihdr { + ihdr::Ihdr { + width, + height, + bit_depth, + color, + interlaced: false, + } + } + + #[test] + fn the_decode_budget_is_inclusive_and_measures_the_decoded_image() { + // 4096x4096 RGBA8 is exactly the decoder's default budget, so the walk must scan it. Its + // *filtered* stream is 67 112 960 bytes — 4096 more, one filter byte per scanline — which + // is how a cap stated over the filtered length came to decline an image that decodes. + let at_budget = header(4096, 4096, 8, ColorType::TruecolorAlpha); + assert!(fits_decode_budget(&at_budget, DEFAULT_MAX_IMAGE_BYTES)); + assert!(!fits_decode_budget(&at_budget, DEFAULT_MAX_IMAGE_BYTES - 1)); + assert!(fits_decode_budget(&at_budget, DEFAULT_MAX_IMAGE_BYTES + 1)); + // One pixel past the budget, at the same dimensions: the depth is the difference. + assert!(!fits_decode_budget( + &header(4096, 4096, 16, ColorType::TruecolorAlpha), + DEFAULT_MAX_IMAGE_BYTES + )); + // A header whose decoded size overflows `usize` is declined, not wrapped. + assert!(!fits_decode_budget( + &header(0x7FFF_FFFF, 0x7FFF_FFFF, 16, ColorType::TruecolorAlpha), + usize::MAX + )); + } + + #[test] + fn only_an_over_budget_scan_is_not_damage() { + // The single source of truth for `is_intact`'s filter conjunct: declining to inflate a + // stream is a statement about this reader's budget, everything else about the file. + assert!(!SkippedFilterScan::OverBudget.is_damage()); + for reason in [ + SkippedFilterScan::CorruptStream, + SkippedFilterScan::LengthMismatch, + SkippedFilterScan::UndefinedFilterCode, + ] { + assert!(reason.is_damage(), "{reason:?}"); + assert!(FilterScan::Skipped(reason).is_damage(), "{reason:?}"); + } + assert!(!FilterScan::Skipped(SkippedFilterScan::OverBudget).is_damage()); + let counted = FilterScan::Counted(FilterHistogram { + counts: [1, 0, 0, 0, 0], + }); + assert!(!counted.is_damage(), "a scan that ran is never damage"); + } + + #[test] + fn a_filter_scan_exposes_exactly_one_of_its_two_sides() { + // Built here because `FilterHistogram`'s counts are private, so the `Counted` side is + // only constructible from inside the crate. + let histogram = FilterHistogram { + counts: [1, 2, 0, 0, 0], + }; + let counted = FilterScan::Counted(histogram); + assert_eq!(counted.histogram(), Some(histogram)); + assert_eq!(counted.skipped(), None); + + let skipped = FilterScan::Skipped(SkippedFilterScan::OverBudget); + assert_eq!(skipped.histogram(), None); + assert_eq!(skipped.skipped(), Some(SkippedFilterScan::OverBudget)); + } + + #[test] + fn the_skip_reasons_keep_their_published_discriminants() { + // `#[repr(u8)]` plain data crossing the C ABI: these numbers are permanent and + // append-only, so a variant is never renumbered or removed, only added after the last. + assert_eq!(SkippedFilterScan::OverBudget as u8, 0); + assert_eq!(SkippedFilterScan::CorruptStream as u8, 1); + assert_eq!(SkippedFilterScan::LengthMismatch as u8, 2); + assert_eq!(SkippedFilterScan::UndefinedFilterCode as u8, 3); + } + + #[test] + fn an_overlap_is_not_fully_classified() { + assert!(!report_with(&[(0, 20), (10, 33)], 33).is_fully_classified()); + } +} diff --git a/crates/gamut-png/src/encoder.rs b/crates/gamut-png/src/encoder.rs index da9575c4..9757ca41 100644 --- a/crates/gamut-png/src/encoder.rs +++ b/crates/gamut-png/src/encoder.rs @@ -22,13 +22,22 @@ use crate::{ihdr, pack}; const IDAT_MAX: usize = 1 << 16; /// Whole-image filter strategies tried by [`FilterStrategy::BruteForce`]. -const BRUTE_FORCE_STRATEGIES: [FilterStrategy; 6] = [ +/// +/// [`FilterStrategy::MinEntropy`] is deliberately **not** here, and that was measured rather than +/// assumed. Across the benchmark corpus it is never the unique winner: it beats `MinSumAbs` on the +/// photographic and palette rows but loses to `MinBigrams` on both, and ties `MinSumAbs` elsewhere. +/// Since this list is resolved by taking the smallest result, a candidate that is dominated +/// everywhere costs a full filter pass and a full DEFLATE for nothing. It stays available as a +/// caller-selectable strategy — the corpus is eight images, not a proof — but it does not earn a +/// slot here. See `STATUS.md`'s heuristic table. +const BRUTE_FORCE_STRATEGIES: [FilterStrategy; 7] = [ FilterStrategy::None, FilterStrategy::Fixed(FilterType::Sub), FilterStrategy::Fixed(FilterType::Up), FilterStrategy::Fixed(FilterType::Average), FilterStrategy::Fixed(FilterType::Paeth), FilterStrategy::MinSumAbs, + FilterStrategy::MinBigrams, ]; /// A reusable PNG encoder. @@ -39,6 +48,7 @@ pub struct PngEncoder { filter: FilterStrategy, ancillary: Ancillary, auto_reduce: bool, + clean_transparent: bool, backends: Registry, } @@ -59,6 +69,7 @@ impl PngEncoder { filter: FilterStrategy::MinSumAbs, ancillary: Ancillary::default(), auto_reduce: false, + clean_transparent: false, backends: Registry::default(), } } @@ -122,6 +133,26 @@ impl PngEncoder { self } + /// Rewrites the colour channels of fully transparent pixels before encoding, so runs of + /// them compress instead of carrying whatever the source left there. + /// + /// Nothing a decoder renders changes -- at `alpha == 0` the colour channels are invisible by + /// definition -- but the stored samples do, so this is **not** lossless in the strict byte + /// sense [`with_auto_reduce`](Self::with_auto_reduce) keeps. That is why it is off by + /// default and separate from it: this crate's other reductions are exactly reversible, and + /// this one is only reversible in what you can see. + /// + /// Worth enabling for sprites, icons and UI assets, where invisible colour noise is common + /// and can cost real bytes. It applies to every layout that carries an alpha channel, at both + /// 8 and 16 bits per sample; a 16-bit pixel counts as invisible when its whole alpha sample is + /// zero, and all sixteen bits of each colour sample are cleared. No effect on an image with no + /// fully transparent pixel, or on a layout with no alpha channel. + #[must_use] + pub fn with_transparent_cleanup(mut self, enabled: bool) -> Self { + self.clean_transparent = enabled; + self + } + /// Enables automatic lossless reduction of any [`EncodeImage`] input to a smaller encoding /// when it does not change any pixel: greyscale (at the smallest exactly-representable bit /// depth), palette, alpha-channel drop, and 16→8 demotion when every sample's high and low @@ -341,15 +372,119 @@ impl PngEncoder { ) } + /// Encodes one 8-bit alpha-carrying sample buffer: the auto-reduce race if it applies, the + /// plain layout otherwise. + /// + /// Split out of the `EncodeImage` impls so [`cleaned_or_plain`](Self::cleaned_or_plain) can + /// run it twice over two different sample buffers. + fn encode_alpha8( + &self, + dims: Dimensions, + samples: &[u8], + channels: usize, + color: ColorType, + out: &mut Vec, + ) -> Result { + if self.auto_reduce + && let Some(reduced) = reduce::analyze8(samples, channels) + { + return self.write_reduced_or_native( + dims, + reduced, + |o| self.write_png((dims.width, dims.height), samples, color, 8, |_| {}, o), + out, + ); + } + self.write_png((dims.width, dims.height), samples, color, 8, |_| {}, out) + } + + /// The 16-bit twin of [`encode_alpha8`](Self::encode_alpha8). + fn encode_alpha16( + &self, + dims: Dimensions, + samples: &[u16], + channels: usize, + color: ColorType, + out: &mut Vec, + ) -> Result { + if self.auto_reduce + && let Some(reduced) = reduce::analyze16(samples, channels) + { + return self.write_reduced_or_native( + dims, + reduced, + |o| self.encode_16bit(dims, samples, color, o), + out, + ); + } + self.encode_16bit(dims, samples, color, out) + } + + /// Encodes the image both ways when cleaning changed something, and keeps the smaller file. + /// + /// Cleaning collapses every invisible pixel to one colour, which is what makes a palette or a + /// colour key reachable at all — worth ~31% on a sprite whose invisible pixels carry noise. + /// But it is a *transform*, not a reduction: it rewrites bytes DEFLATE was already + /// compressing. Where the invisible pixels carry structure — a gradient that continues under + /// the transparent region — zeroing them inserts a discontinuity that costs more than the + /// collapsed palette saves. Measured on `palette64_rgba8`, cleaning is worth −2.3% at 32x32, + /// **+10.7% at 128x128** and −5.2% at 256x256, with both candidates landing on the same + /// colour type throughout: the sign genuinely depends on the image. + /// + /// So the choice is raced rather than assumed, exactly as + /// [`write_reduced_or_native`](Self::write_reduced_or_native) races a palette against the + /// unreduced encoding, and for the same reason: no tuned constant can predict a compressed + /// size. [`with_transparent_cleanup`](Self::with_transparent_cleanup) therefore means "clean + /// where it pays", and enabling it can never cost bytes. + /// + /// A tie keeps the cleaned encoding, which carries less unseen data. + fn cleaned_or_plain( + &self, + cleaned: impl FnOnce(&mut Vec) -> Result, + plain: impl FnOnce(&mut Vec) -> Result, + out: &mut Vec, + ) -> Result { + let mut cleaned_encoding = Vec::new(); + cleaned(&mut cleaned_encoding)?; + let mut plain_encoding = Vec::new(); + plain(&mut plain_encoding)?; + + let winner = if prefers_plain(plain_encoding.len(), cleaned_encoding.len()) { + plain_encoding + } else { + cleaned_encoding + }; + out.extend_from_slice(&winner); + Ok(winner.len()) + } + + /// The cleaned samples, or `None` to use the caller's buffer unchanged — either because the + /// knob is off or because the image has no fully transparent pixel. + fn cleaned_samples(&self, samples: &[u8], channels: usize) -> Option> { + self.clean_transparent + .then(|| reduce::clean_transparent(samples, channels)) + .flatten() + } + + /// The 16-bit twin of [`cleaned_samples`](Self::cleaned_samples): the cleaned samples, or + /// `None` to use the caller's buffer unchanged. + fn cleaned_samples16(&self, samples: &[u16], channels: usize) -> Option> { + self.clean_transparent + .then(|| clean_transparent16(samples, channels)) + .flatten() + } + /// Encodes a 16-bit-per-sample image, serialising samples big-endian (PNG's network byte order). - fn encode_16bit>( + /// + /// Takes the samples rather than the [`ImageRef`] so the alpha layouts can hand over a cleaned + /// buffer (see [`cleaned_samples16`](Self::cleaned_samples16)). + fn encode_16bit( &self, - image: ImageRef<'_, P>, + dims: Dimensions, + samples: &[u16], color: ColorType, out: &mut Vec, ) -> Result { - let dims = image.dimensions(); - let samples = image.as_samples(); let mut bytes = Vec::with_capacity(samples.len() * 2); for &sample in samples { bytes.extend_from_slice(&sample.to_be_bytes()); @@ -440,6 +575,53 @@ impl PngEncoder { } } + /// Writes `reduced`, unless it is a palette encoding that turns out *larger* than encoding + /// the image untouched — in which case the untouched one wins. + /// + /// [`reduce::analyze8`] chooses by comparing **raw** sizes, and raw size does not predict + /// compressed size when one candidate's bytes are incompressible and the other's are not. A + /// palette carries a `PLTE` (and often `tRNS`) chunk that DEFLATE cannot touch, while the + /// pixels it replaces may compress by two orders of magnitude. On a 128x128 image with 64 + /// colours the estimate sees 16 664 bytes against 65 536 and picks the palette by 4x — and + /// the finished file is 451 bytes against 405. The crossover sits near 160x160, so the + /// estimate is right on large images and wrong on small ones. + /// + /// Rather than guess a correction factor, the two candidates are encoded and the smaller + /// kept. That is exactly what [`FilterStrategy::BruteForce`] already does for filters, it + /// needs no tuned constant, and it cannot be worse than either candidate alone. A tie keeps + /// the palette, which decodes with less work. + /// + /// Only the reductions that *carry a chunk* pay for the second encode — a palette's `PLTE` + /// (+ `tRNS`), or a colour key's `tRNS`. Greyscale, alpha-drop and 16→8 demotion add no chunks + /// at all, so for them the raw comparison is sound and this returns immediately. + fn write_reduced_or_native( + &self, + dims: Dimensions, + reduced: Reduced, + native: impl FnOnce(&mut Vec) -> Result, + out: &mut Vec, + ) -> Result { + let carries_chunks = matches!( + reduced, + Reduced::Indexed { .. } | Reduced::Rgb8Keyed { .. } | Reduced::GrayKeyed { .. } + ); + if !carries_chunks { + return self.write_reduced(dims, reduced, out); + } + let mut palette_encoding = Vec::new(); + self.write_reduced(dims, reduced, &mut palette_encoding)?; + let mut native_encoding = Vec::new(); + native(&mut native_encoding)?; + + let winner = if prefers_native(native_encoding.len(), palette_encoding.len()) { + native_encoding + } else { + palette_encoding + }; + out.extend_from_slice(&winner); + Ok(winner.len()) + } + /// Writes a reduced encoding chosen by [`reduce::analyze8`] / [`reduce::analyze16`]. fn write_reduced( &self, @@ -470,6 +652,28 @@ impl PngEncoder { Reduced::Rgb8(samples) => { self.write_png(wh, &samples, ColorType::Truecolor, 8, |_| {}, out) } + // §11.3.2.1: for truecolour, tRNS is three 16-bit big-endian samples naming the one + // colour a decoder renders as fully transparent. At depth 8 the high byte is zero. + Reduced::Rgb8Keyed { samples, key } => self.write_png( + wh, + &samples, + ColorType::Truecolor, + 8, + |out| { + let trns = [0, key[0], 0, key[1], 0, key[2]]; + chunk::write_chunk(out, *b"tRNS", &trns); + }, + out, + ), + // ...and for greyscale, one 16-bit big-endian sample. + Reduced::GrayKeyed { samples, key } => self.write_png( + wh, + &samples, + ColorType::Grayscale, + 8, + |out| chunk::write_chunk(out, *b"tRNS", &[0, key]), + out, + ), Reduced::Rgba8(samples) => { self.write_png(wh, &samples, ColorType::TruecolorAlpha, 8, |_| {}, out) } @@ -518,6 +722,54 @@ impl PngEncoder { } } +/// Whether the uncleaned encoding beats the cleaned one, for [`PngEncoder::cleaned_or_plain`]. +/// +/// **A tie keeps the cleaned encoding**, which carries less unseen data for the same bytes. Split +/// out for the same reason as [`prefers_native`]: engineering two encodings of the same image to +/// land on exactly equal lengths is not something a fixture can do reliably, so the tie is only +/// assertable here. +fn prefers_plain(plain_len: usize, cleaned_len: usize) -> bool { + plain_len < cleaned_len +} + +/// Whether the unreduced encoding beats the palette one, for [`PngEncoder::write_reduced_or_native`]. +/// +/// **A tie keeps the palette**, which decodes with less work for the same bytes. Split out because +/// engineering two encodings of the same image to land on exactly equal lengths is not something a +/// fixture can do reliably, so the tie is only assertable here. +fn prefers_native(native_len: usize, palette_len: usize) -> bool { + native_len < palette_len +} + +/// Zeroes the colour samples of every fully transparent pixel in a 16-bit interleaved buffer, +/// returning `None` when there is nothing to do (no alpha channel, or no fully transparent pixel) +/// so the caller can keep borrowing its own samples. +/// +/// The 8-bit twin is `reduce::clean_transparent`, which cannot serve here: it reads one-byte +/// samples with a one-byte stride, whereas a 16-bit pixel is invisible only when its *whole* alpha +/// sample is zero (both bytes of the stored big-endian pair), and clearing a colour sample must +/// clear all sixteen bits. Working on the `u16` samples rather than on the big-endian bytes +/// `PngEncoder::encode_16bit` emits keeps the ordering identical to the 8-bit paths — cleanup runs +/// first, so `reduce::analyze16` gets to see the collapsed invisible pixels. +fn clean_transparent16(samples: &[u16], channels: usize) -> Option> { + debug_assert!((1..=4).contains(&channels)); + if !channels.is_multiple_of(2) { + return None; // no alpha channel + } + let colour = channels - 1; // colour samples are everything before alpha + if !samples.chunks_exact(channels).any(|px| px[colour] == 0) { + return None; + } + + let mut out = samples.to_vec(); + for px in out.chunks_exact_mut(channels) { + if px[colour] == 0 { + px[..colour].fill(0); + } + } + Some(out) +} + /// Writes the zlib datastream as one or more consecutive IDAT chunks. fn write_idat(out: &mut Vec, zlib_stream: &[u8]) { if zlib_stream.is_empty() { @@ -536,7 +788,12 @@ impl EncodeImage for PngEncoder { if self.auto_reduce && let Some(reduced) = reduce::analyze8(image.as_samples(), 1) { - return self.write_reduced(image.dimensions(), reduced, out); + return self.write_reduced_or_native( + image.dimensions(), + reduced, + |o| self.encode_8bit(image, ColorType::Grayscale, o), + out, + ); } self.encode_8bit(image, ColorType::Grayscale, out) } @@ -566,69 +823,102 @@ impl EncodeImage for PngEncoder { if self.auto_reduce && let Some(reduced) = reduce::analyze8(image.as_samples(), 3) { - return self.write_reduced(image.dimensions(), reduced, out); + return self.write_reduced_or_native( + image.dimensions(), + reduced, + |o| self.encode_8bit(image, ColorType::Truecolor, o), + out, + ); } self.encode_8bit(image, ColorType::Truecolor, out) } } impl EncodeImage for PngEncoder { fn encode_image(&self, image: ImageRef<'_, Rgba8>, out: &mut Vec) -> Result { - if self.auto_reduce - && let Some(reduced) = reduce::analyze8(image.as_samples(), 4) - { - return self.write_reduced(image.dimensions(), reduced, out); + let dims = image.dimensions(); + let plain = image.as_samples(); + match self.cleaned_samples(plain, 4) { + Some(cleaned) => self.cleaned_or_plain( + |o| self.encode_alpha8(dims, &cleaned, 4, ColorType::TruecolorAlpha, o), + |o| self.encode_alpha8(dims, plain, 4, ColorType::TruecolorAlpha, o), + out, + ), + None => self.encode_alpha8(dims, plain, 4, ColorType::TruecolorAlpha, out), } - self.encode_8bit(image, ColorType::TruecolorAlpha, out) } } impl EncodeImage for PngEncoder { fn encode_image(&self, image: ImageRef<'_, GrayAlpha8>, out: &mut Vec) -> Result { - if self.auto_reduce - && let Some(reduced) = reduce::analyze8(image.as_samples(), 2) - { - return self.write_reduced(image.dimensions(), reduced, out); + let dims = image.dimensions(); + let plain = image.as_samples(); + match self.cleaned_samples(plain, 2) { + Some(cleaned) => self.cleaned_or_plain( + |o| self.encode_alpha8(dims, &cleaned, 2, ColorType::GrayscaleAlpha, o), + |o| self.encode_alpha8(dims, plain, 2, ColorType::GrayscaleAlpha, o), + out, + ), + None => self.encode_alpha8(dims, plain, 2, ColorType::GrayscaleAlpha, out), } - self.encode_8bit(image, ColorType::GrayscaleAlpha, out) } } impl EncodeImage for PngEncoder { fn encode_image(&self, image: ImageRef<'_, Gray16>, out: &mut Vec) -> Result { + let (dims, samples) = (image.dimensions(), image.as_samples()); if self.auto_reduce - && let Some(reduced) = reduce::analyze16(image.as_samples(), 1) + && let Some(reduced) = reduce::analyze16(samples, 1) { - return self.write_reduced(image.dimensions(), reduced, out); + return self.write_reduced_or_native( + dims, + reduced, + |o| self.encode_16bit(dims, samples, ColorType::Grayscale, o), + out, + ); } - self.encode_16bit(image, ColorType::Grayscale, out) + self.encode_16bit(dims, samples, ColorType::Grayscale, out) } } impl EncodeImage for PngEncoder { fn encode_image(&self, image: ImageRef<'_, Rgb16>, out: &mut Vec) -> Result { + let (dims, samples) = (image.dimensions(), image.as_samples()); if self.auto_reduce - && let Some(reduced) = reduce::analyze16(image.as_samples(), 3) + && let Some(reduced) = reduce::analyze16(samples, 3) { - return self.write_reduced(image.dimensions(), reduced, out); + return self.write_reduced_or_native( + dims, + reduced, + |o| self.encode_16bit(dims, samples, ColorType::Truecolor, o), + out, + ); } - self.encode_16bit(image, ColorType::Truecolor, out) + self.encode_16bit(dims, samples, ColorType::Truecolor, out) } } impl EncodeImage for PngEncoder { fn encode_image(&self, image: ImageRef<'_, Rgba16>, out: &mut Vec) -> Result { - if self.auto_reduce - && let Some(reduced) = reduce::analyze16(image.as_samples(), 4) - { - return self.write_reduced(image.dimensions(), reduced, out); + let dims = image.dimensions(); + let plain = image.as_samples(); + match self.cleaned_samples16(plain, 4) { + Some(cleaned) => self.cleaned_or_plain( + |o| self.encode_alpha16(dims, &cleaned, 4, ColorType::TruecolorAlpha, o), + |o| self.encode_alpha16(dims, plain, 4, ColorType::TruecolorAlpha, o), + out, + ), + None => self.encode_alpha16(dims, plain, 4, ColorType::TruecolorAlpha, out), } - self.encode_16bit(image, ColorType::TruecolorAlpha, out) } } impl EncodeImage for PngEncoder { fn encode_image(&self, image: ImageRef<'_, GrayAlpha16>, out: &mut Vec) -> Result { - if self.auto_reduce - && let Some(reduced) = reduce::analyze16(image.as_samples(), 2) - { - return self.write_reduced(image.dimensions(), reduced, out); + let dims = image.dimensions(); + let plain = image.as_samples(); + match self.cleaned_samples16(plain, 2) { + Some(cleaned) => self.cleaned_or_plain( + |o| self.encode_alpha16(dims, &cleaned, 2, ColorType::GrayscaleAlpha, o), + |o| self.encode_alpha16(dims, plain, 2, ColorType::GrayscaleAlpha, o), + out, + ), + None => self.encode_alpha16(dims, plain, 2, ColorType::GrayscaleAlpha, out), } - self.encode_16bit(image, ColorType::GrayscaleAlpha, out) } } @@ -698,14 +988,23 @@ mod tests { /// `1u8 << depth`, which overflows at 8. #[test] fn indexed_at_depth_eight_is_not_bit_packed() { - // 32 distinct opaque colours over 200 pixels: more than 16, so the index depth is 8, and - // cheap enough that the palette still beats raw RGB. + // 32 distinct opaque colours over 1024 pixels: more than 16, so the index depth is 8. + // + // Pseudo-random rather than cycling, and 1024 pixels rather than 200, because + // `write_reduced_or_native` races the palette against the unreduced encoding and keeps + // whichever is smaller. A period-32 cycle over 200 pixels compresses to an 82-byte RGB + // file, which a 96-byte `PLTE` cannot beat before a single index is written -- the race + // correctly declines the palette, and pinning `Indexed` there would assert the defect the + // race exists to fix. Shuffling denies DEFLATE the period and 1024 pixels amortise the + // palette: 479 bytes indexed against 525 unreduced. let mut rgb = Vec::new(); - for i in 0..200u32 { - let c = (i % 32) as u8; + for i in 0..1024u32 { + let mut h = i.wrapping_mul(2654435761); + h ^= h >> 15; + let c = (h % 32) as u8; rgb.extend_from_slice(&[c, c.wrapping_add(70), 90]); } - let img = ImageRef::::new(&rgb, Dimensions::new(200, 1).unwrap()).unwrap(); + let img = ImageRef::::new(&rgb, Dimensions::new(1024, 1).unwrap()).unwrap(); let mut png = Vec::new(); // Reduction is opt-in; without it the encoder writes the input layout unchanged and the // indexed path -- the one this test is about -- is never reached. @@ -763,6 +1062,20 @@ mod tests { assert_eq!(&appended[17..], &fresh[..], "the prefix is left untouched"); } + #[test] + fn a_tie_between_palette_and_native_keeps_the_palette() { + assert!(prefers_native(10, 11), "smaller native wins"); + assert!(!prefers_native(11, 10), "smaller palette wins"); + assert!(!prefers_native(10, 10), "a tie keeps the palette"); + } + + #[test] + fn a_tie_between_cleaned_and_plain_keeps_the_cleaned_encoding() { + assert!(prefers_plain(10, 11), "smaller plain wins"); + assert!(!prefers_plain(11, 10), "smaller cleaned wins"); + assert!(!prefers_plain(10, 10), "a tie keeps the cleaned encoding"); + } + #[test] fn brute_force_keeps_the_first_strategy_on_a_tie() { // A 1x1 image compresses to the same length under every strategy, so the tie-break is what @@ -803,4 +1116,48 @@ mod tests { let idats = out.windows(4).filter(|w| *w == b"IDAT").count(); assert!(idats >= 3, "expected multiple IDAT chunks, found {idats}"); } + + #[test] + fn cleaning_16_bit_pixels_needs_the_whole_alpha_sample_to_be_zero() { + // The byte-wise twin would read the big-endian pair `0x0001` as a zero high byte and + // wrongly call this pixel invisible; at `u16` width it is visible and must be untouched. + // The third pixel is the genuinely invisible one, and all three of its colour samples — + // both bytes of each — must be cleared. + let src: [u16; 12] = [ + 0x1234, 0x5678, 0x9ABC, 0xFFFF, // visible + 0x1111, 0x2222, 0x3333, 0x0001, // alpha 1: barely visible, must stay + 0x4444, 0x5555, 0x6666, 0x0000, // invisible: colour must go + ]; + let cleaned = clean_transparent16(&src, 4).expect("there is a transparent pixel"); + assert_eq!( + cleaned, + vec![ + 0x1234, 0x5678, 0x9ABC, 0xFFFF, // + 0x1111, 0x2222, 0x3333, 0x0001, // + 0, 0, 0, 0, + ] + ); + } + + #[test] + fn cleaning_16_bit_grey_alpha_zeroes_only_the_grey_sample() { + let src: [u16; 6] = [0xC800, 0xFFFF, 0x6F00, 0x0000, 0x5A00, 0x0001]; + let cleaned = clean_transparent16(&src, 2).expect("there is a transparent pixel"); + assert_eq!(cleaned, vec![0xC800, 0xFFFF, 0, 0, 0x5A00, 0x0001]); + } + + #[test] + fn cleaning_16_bit_declines_when_there_is_nothing_to_clean() { + let opaque: [u16; 8] = [1, 2, 3, 0xFFFF, 4, 5, 6, 0xFFFF]; + assert!( + clean_transparent16(&opaque, 4).is_none(), + "no fully transparent pixel" + ); + + // Odd channel counts have no alpha sample, so a zero there is a colour, not transparency. + let grey: [u16; 3] = [0, 7, 9]; + assert!(clean_transparent16(&grey, 1).is_none(), "no alpha channel"); + let rgb: [u16; 6] = [1, 2, 0, 4, 5, 6]; + assert!(clean_transparent16(&rgb, 3).is_none(), "no alpha channel"); + } } diff --git a/crates/gamut-png/src/filter.rs b/crates/gamut-png/src/filter.rs index 8c5c100b..d9177efa 100644 --- a/crates/gamut-png/src/filter.rs +++ b/crates/gamut-png/src/filter.rs @@ -22,7 +22,12 @@ pub enum FilterType { } /// How the encoder chooses a filter for each scanline (a space/time trade-off). +/// +/// Non-exhaustive: a heuristic is a measurement result, and this crate's own `STATUS.md` records +/// the corpus that decides which ones are worth shipping — so the set grows as that corpus does. +/// Match with a wildcard arm. #[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] pub enum FilterStrategy { /// Filter every scanline with [`FilterType::None`] (fastest; good for already-random data). None, @@ -31,6 +36,24 @@ pub enum FilterStrategy { /// Per scanline, pick the filter minimising the sum of absolute residuals — the standard /// libpng heuristic. A good size/speed balance and the default. MinSumAbs, + /// Per scanline, pick the filter whose residuals have the lowest Shannon entropy. + /// + /// Sum-of-absolutes asks "are these bytes small?"; entropy asks "are these bytes *repetitive*?" + /// — which is the question DEFLATE actually answers. A row of alternating 0 and 200 scores + /// badly under `MinSumAbs` and beautifully under this. + /// + /// The only strategy that scores in floating point. `f64::log2` is not required to be + /// correctly rounded, so this strategy's output is reproducible on a machine but not + /// guaranteed bit-identical across libm implementations — which is why it is absent from + /// [`BRUTE_FORCE_STRATEGIES`](crate::PngEncoder), keeping the default and `BruteForce` paths + /// integer-only and their output byte-exact everywhere. It never uniquely won a corpus row + /// (`STATUS.md`), so it is offered rather than chosen. + MinEntropy, + /// Per scanline, pick the filter producing the fewest distinct byte bigrams. + /// + /// A cheaper proxy for the same idea one order up: LZ77 matches runs, not single bytes, so + /// counting distinct adjacent pairs approximates how much of the row it can back-reference. + MinBigrams, /// Encode the whole image under several filter strategies, DEFLATE each, and keep the smallest. /// Pairs with [`Level::Best`](gamut_deflate::Level::Best) for maximum compression; slowest. BruteForce, @@ -70,21 +93,77 @@ fn paeth(a: u8, b: u8, c: u8) -> u8 { /// Forward-filters one scanline `cur` (with previous raw row `prev`, all zero for the first row) /// into `out` (which is overwritten to `cur.len()` bytes). +/// +/// Structured for the vectoriser rather than for brevity, because this is the encoder's hottest +/// loop -- `MinSumAbs` runs it five times per scanline and `BruteForce` up to ten. Three things +/// matter, and the straightforward version does none of them: +/// +/// * The `i >= bpp` test that picks between a real left-neighbour and an implicit zero is loop +/// invariant, so the row splits into a `bpp`-long prologue where `a` and `c` are zero and a +/// body where they are not. Testing it per byte defeats vectorisation outright. +/// * The body then reads five *equal-length* subslices, which lets the bounds checks fold away +/// instead of being re-proved for every index. +/// * The filter is matched once, outside the loop, so each arm is a straight-line kernel rather +/// than a branch per byte. And `out` is sized once, so there is no capacity check per `push`. fn filter_row(filter: FilterType, cur: &[u8], prev: &[u8], bpp: usize, out: &mut Vec) { + let n = cur.len(); out.clear(); - out.reserve(cur.len()); - for i in 0..cur.len() { - let a = if i >= bpp { cur[i - bpp] } else { 0 }; - let b = prev[i]; - let c = if i >= bpp { prev[i - bpp] } else { 0 }; - let residual = match filter { - FilterType::None => cur[i], - FilterType::Sub => cur[i].wrapping_sub(a), - FilterType::Up => cur[i].wrapping_sub(b), - FilterType::Average => cur[i].wrapping_sub(((u16::from(a) + u16::from(b)) / 2) as u8), - FilterType::Paeth => cur[i].wrapping_sub(paeth(a, b, c)), - }; - out.push(residual); + out.resize(n, 0); + let head = bpp.min(n); + + // Prologue: the first `bpp` bytes have no left neighbour, so `a == c == 0`. That collapses + // Sub to a copy and -- less obviously -- Paeth to Up, because `paeth(0, b, 0) == b` for every + // `b` (at `b == 0` all three distances tie and the spec's order picks `a`, which is also 0). + match filter { + FilterType::None | FilterType::Sub => out[..head].copy_from_slice(&cur[..head]), + FilterType::Up | FilterType::Paeth => { + for (d, (&x, &b)) in out[..head] + .iter_mut() + .zip(cur[..head].iter().zip(&prev[..head])) + { + *d = x.wrapping_sub(b); + } + } + FilterType::Average => { + for (d, (&x, &b)) in out[..head] + .iter_mut() + .zip(cur[..head].iter().zip(&prev[..head])) + { + *d = x.wrapping_sub(b / 2); + } + } + } + + // Body: `x` is the current byte, `a` the byte `bpp` to its left, `b` the byte above, `c` the + // byte above-left. All five slices are the same length by construction. + let m = n - head; + let dst = &mut out[head..]; + let x = &cur[head..]; + let a = &cur[..m]; + let b = &prev[head..]; + let c = &prev[..m]; + match filter { + FilterType::None => dst.copy_from_slice(x), + FilterType::Sub => { + for (d, (&x, &a)) in dst.iter_mut().zip(x.iter().zip(a)) { + *d = x.wrapping_sub(a); + } + } + FilterType::Up => { + for (d, (&x, &b)) in dst.iter_mut().zip(x.iter().zip(b)) { + *d = x.wrapping_sub(b); + } + } + FilterType::Average => { + for (d, ((&x, &a), &b)) in dst.iter_mut().zip(x.iter().zip(a).zip(b)) { + *d = x.wrapping_sub(((u16::from(a) + u16::from(b)) / 2) as u8); + } + } + FilterType::Paeth => { + for (d, (((&x, &a), &b), &c)) in dst.iter_mut().zip(x.iter().zip(a).zip(b).zip(c)) { + *d = x.wrapping_sub(paeth(a, b, c)); + } + } } } @@ -117,10 +196,110 @@ fn sum_abs(filtered: &[u8]) -> u64 { .sum() } +/// How a candidate row is judged. Lower is better for all three, so they are interchangeable in +/// [`choose_by`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Score { + /// Sum of absolute residuals, bytes read as signed magnitudes (libpng's heuristic). + SumAbs, + /// Shannon entropy of the byte histogram. + Entropy, + /// Count of distinct adjacent byte pairs. + Bigrams, +} + +/// Scratch a scorer needs, allocated once per image rather than per scanline. +/// +/// Hoisting saves the *allocation*; it does not on its own save the clearing, and the clearing is +/// the larger cost. The bigram set is 8 KiB, so wiping it wholesale would cost 8 KiB per candidate +/// and five candidates per scanline — 40 KiB of memset per row, independent of how long the row +/// is, which for any ordinary row is more work than the scoring. So the set records which words it +/// dirtied and clears only those: a row of `n` bytes touches at most `n - 1` of them. +struct Scratch { + /// Byte histogram for [`Score::Entropy`]. + histogram: [u32; 256], + /// One bit per (previous, current) byte pair for [`Score::Bigrams`]. + bigrams: Vec, + /// The indices of the `bigrams` words this scorer set, so the reset touches only them. Holds + /// each dirtied word exactly once — a word is pushed when it goes from all-zero to non-zero. + dirty: Vec, +} + +impl Scratch { + fn new() -> Self { + Self { + histogram: [0; 256], + bigrams: vec![0; 1 << 10], + dirty: Vec::new(), + } + } +} + +/// Scores a filtered row; lower is better in every variant, so candidates compare directly. +fn score(kind: Score, filtered: &[u8], scratch: &mut Scratch) -> u64 { + match kind { + Score::SumAbs => sum_abs(filtered), + Score::Entropy => { + scratch.histogram.fill(0); + for &b in filtered { + scratch.histogram[b as usize] += 1; + } + // Shannon entropy times the row length, `Σ c·log2(n/c)`, in 1/256ths of a bit so the + // comparison is integer-exact and the choice reproducible run to run. + // + // Stated this way every term is non-negative (`c ≤ n`) and the whole score is bounded + // by `8n·256` — a byte alphabet carries at most 8 bits — so lower is better directly, + // rather than by complementing against `u64::MAX`. That matters beyond tidiness: a + // score that can *reach* `u64::MAX` is a score that can collide with a sentinel, and + // this one did. + // + // Equivalent to the `n·log2(n) − Σ c·log2(c)` form: `n` is constant across a row's + // five candidates, so it cannot change the ranking either way. The `c == 1` terms + // contribute `0` there and `1·log2(n)` here, which is why the filter below is `c > 0` + // — a zero count is excluded because `0·log2(n/0)` is not a number, not because it + // contributes nothing. + let n = filtered.len() as f64; + let bits: f64 = scratch + .histogram + .iter() + .filter(|&&c| c > 0) + .map(|&c| f64::from(c) * (n / f64::from(c)).log2()) + .sum(); + (bits * 256.0) as u64 + } + Score::Bigrams => { + let mut distinct = 0u64; + for pair in filtered.windows(2) { + // The pair *is* a big-endian `u16`, so read it as one. Spelling it `a << 8 | b` + // costs two operators that carry no meaning of their own -- one of which has no + // behavioural variant at all, since the low byte of `a << 8` is zero and `|` is + // therefore indistinguishable from `^`. + let index = usize::from(u16::from_be_bytes([pair[0], pair[1]])); + let (word, bit) = (index >> 6, index & 63); + if scratch.bigrams[word] & (1 << bit) == 0 { + // Record the word the first time it leaves zero, so `dirty` lists each + // dirtied word once and the reset below is exact. + if scratch.bigrams[word] == 0 { + scratch.dirty.push(word); + } + scratch.bigrams[word] |= 1 << bit; + distinct += 1; + } + } + // Leave the set all-zero for the next candidate, touching only what was dirtied. + for &word in &scratch.dirty { + scratch.bigrams[word] = 0; + } + scratch.dirty.clear(); + distinct + } + } +} + /// Filters every scanline of `samples` (row-major, `row_bytes` per row) per `strategy`, producing /// the filter-prefixed byte stream that gets compressed: a filter-type byte then the filtered row, /// for each scanline. `bpp` is the filter stride (bytes per pixel, ≥1). -pub(crate) fn filter_image( +pub fn filter_image( strategy: FilterStrategy, samples: &[u8], row_bytes: usize, @@ -131,29 +310,65 @@ pub(crate) fn filter_image( let zero_row = vec![0u8; row_bytes]; let mut prev = zero_row.as_slice(); let mut scratch = Vec::with_capacity(row_bytes); + let mut chosen = Vec::with_capacity(row_bytes); + let mut aux = Scratch::new(); + // The per-scanline heuristics differ only in how they score a candidate. BruteForce is + // resolved to concrete strategies by the encoder; if it reaches here, fall back to MinSumAbs. + let adaptive = match strategy { + FilterStrategy::MinSumAbs | FilterStrategy::BruteForce => Some(Score::SumAbs), + FilterStrategy::MinEntropy => Some(Score::Entropy), + FilterStrategy::MinBigrams => Some(Score::Bigrams), + FilterStrategy::None | FilterStrategy::Fixed(_) => None, + }; for y in 0..height { let cur = &samples[y * row_bytes..(y + 1) * row_bytes]; - let filter = match strategy { - FilterStrategy::None => FilterType::None, - FilterStrategy::Fixed(f) => f, - // BruteForce is resolved to concrete strategies by the encoder; if it reaches here, fall - // back to the per-scanline heuristic. - FilterStrategy::MinSumAbs | FilterStrategy::BruteForce => { - choose_min_sum_abs(cur, prev, bpp, &mut scratch) + match adaptive { + Some(kind) => { + let filter = choose_by(kind, cur, prev, bpp, &mut scratch, &mut chosen, &mut aux); + out.push(filter as u8); + out.extend_from_slice(&chosen); } - }; - out.push(filter as u8); - filter_row(filter, cur, prev, bpp, &mut scratch); - out.extend_from_slice(&scratch); + None => { + let filter = match strategy { + FilterStrategy::Fixed(f) => f, + _ => FilterType::None, + }; + out.push(filter as u8); + filter_row(filter, cur, prev, bpp, &mut scratch); + out.extend_from_slice(&scratch); + } + } prev = cur; } out } -/// Picks the filter with the lowest sum-of-absolute-residuals for one scanline. -fn choose_min_sum_abs(cur: &[u8], prev: &[u8], bpp: usize, scratch: &mut Vec) -> FilterType { +/// Tries all five filters and keeps the one `kind` ranks lowest, leaving its bytes in +/// `best_bytes`. +/// +/// The first minimum wins, so a tie resolves to the earlier filter in None/Sub/Up/Average/Paeth +/// order — deterministic, which the byte-reproducibility contract depends on. +/// +/// Returning the winning bytes in `best_bytes` rather than just the winning filter is what makes +/// this five passes over the row instead of six: the caller would otherwise re-run [`filter_row`] +/// for the filter just chosen, having already computed exactly those bytes and thrown them away. +/// Keeping them costs one `memcpy` per improvement, against a full filter pass per scanline. +fn choose_by( + kind: Score, + cur: &[u8], + prev: &[u8], + bpp: usize, + scratch: &mut Vec, + best_bytes: &mut Vec, + aux: &mut Scratch, +) -> FilterType { let mut best = FilterType::None; - let mut best_score = u64::MAX; + // `None`, not a sentinel score. Seeding with `u64::MAX` and improving on a strict `<` leaves + // `best_bytes` unwritten when every candidate scores `u64::MAX` — and `filter_image` reuses + // that buffer across scanlines, so the row would be emitted with its predecessor's residuals + // under a filter byte of 0. `Option` makes "nothing chosen yet" unrepresentable as a score, so + // the first candidate is always taken whatever any scorer returns. + let mut best_score: Option = None; for filter in [ FilterType::None, FilterType::Sub, @@ -162,10 +377,12 @@ fn choose_min_sum_abs(cur: &[u8], prev: &[u8], bpp: usize, scratch: &mut Vec FilterType::Paeth, ] { filter_row(filter, cur, prev, bpp, scratch); - let score = sum_abs(scratch); - if score < best_score { - best_score = score; + let candidate = score(kind, scratch, aux); + if best_score.is_none_or(|best| candidate < best) { + best_score = Some(candidate); best = filter; + best_bytes.clear(); + best_bytes.extend_from_slice(scratch); } } best @@ -257,13 +474,139 @@ mod tests { assert_eq!(wide2, [128, 190]); // 1+floor(255/2)=128, then 255+floor((128+255)/2)=255+191 wraps to 190 } + /// A row that is *large* but *repetitive*: alternating 0 and 200 under `Sub`. + /// + /// This is the case the two new heuristics exist for. Sum-of-absolutes asks "are these bytes + /// small?" and rates it terribly; entropy and bigrams ask "are these bytes repetitive?", which + /// is the question DEFLATE actually answers. + #[test] + fn entropy_and_bigrams_prefer_repetition_where_sum_abs_prefers_smallness() { + let repetitive: Vec = (0..64).map(|i| if i % 2 == 0 { 0 } else { 200 }).collect(); + let varied: Vec = (0..64u8).map(|i| i / 8).collect(); + let mut aux = Scratch::new(); + + // Sum-of-absolutes: the varied row is far "smaller" and wins. + assert!( + score(Score::SumAbs, &varied, &mut aux) < score(Score::SumAbs, &repetitive, &mut aux) + ); + // Entropy and bigrams: the repetitive row has two symbols and one alternating pair, and + // wins by a mile. + assert!( + score(Score::Entropy, &repetitive, &mut aux) < score(Score::Entropy, &varied, &mut aux) + ); + assert!( + score(Score::Bigrams, &repetitive, &mut aux) < score(Score::Bigrams, &varied, &mut aux) + ); + } + + #[test] + fn the_bigram_score_counts_distinct_adjacent_pairs() { + let mut aux = Scratch::new(); + // (1,2), (2,1), (1,2), (2,1) -> two distinct pairs, however long the run. + assert_eq!(score(Score::Bigrams, &[1, 2, 1, 2, 1], &mut aux), 2); + // A constant row has exactly one. + assert_eq!(score(Score::Bigrams, &[7, 7, 7, 7], &mut aux), 1); + // Every pair distinct. + assert_eq!(score(Score::Bigrams, &[1, 2, 3, 4], &mut aux), 3); + // Fewer than two bytes has no pairs at all. + assert_eq!(score(Score::Bigrams, &[9], &mut aux), 0); + assert_eq!(score(Score::Bigrams, &[], &mut aux), 0); + // Distinct *pairs*, not distinct second bytes: (1,3), (3,2), (2,3) is three pairs over two + // distinct second bytes, so an index that dropped the high byte would report two. Every + // vector above happens to have as many pairs as second bytes, so none of them can tell. + assert_eq!(score(Score::Bigrams, &[1, 3, 2, 3], &mut aux), 3); + } + + #[test] + fn the_scratch_is_reusable_across_rows() { + // The histogram and bigram set are allocated once per image, so a stale one would silently + // score the wrong thing on every row after the first. + let mut aux = Scratch::new(); + let first = score(Score::Bigrams, &[1, 2, 3, 4], &mut aux); + let second = score(Score::Bigrams, &[7, 7, 7, 7], &mut aux); + assert_eq!(first, 3); + assert_eq!(second, 1, "the previous row's pairs must not carry over"); + + let a = score(Score::Entropy, &[0, 0, 0, 0], &mut aux); + let b = score(Score::Entropy, &[0, 1, 2, 3], &mut aux); + assert!(a < b, "a constant row must stay the lower-entropy one"); + } + + #[test] + fn a_universal_score_tie_keeps_the_first_filters_bytes() { + // Every candidate for this row has all-distinct bytes, so under a scorer that ranks by + // repetition they all score identically. `choose_by` must still emit the first candidate's + // bytes: before the `Option` seed it emitted none at all, and `filter_image` produced a + // one-byte stream for a two-byte row -- a PNG whose IDAT is shorter than its image. + assert_eq!( + filter_image(FilterStrategy::MinEntropy, &[1, 3], 2, 1), + [FilterType::None as u8, 1, 3] + ); + } + + #[test] + fn a_tied_row_does_not_reuse_the_previous_rows_residuals() { + // The companion failure to the one above, and the dangerous one: `filter_image` hoists the + // chosen-bytes buffer out of the row loop, so a row that chose nothing re-emitted its + // predecessor's residuals under a filter byte of 0 -- a structurally valid PNG carrying + // the wrong pixels, with no error anywhere. + assert_eq!( + filter_image(FilterStrategy::MinEntropy, &[0, 0, 0, 1], 2, 1), + [FilterType::None as u8, 0, 0, FilterType::None as u8, 0, 1] + ); + } + + #[test] + fn the_entropy_score_weights_each_symbol_by_how_often_it_occurs() { + // Entropy is `sum c*log2(n/c)`, not `sum log2(n/c)`: each symbol's surprise is weighted by + // how much of the row it accounts for. Drop the weighting and the score degenerates into + // something that mostly counts distinct symbols, which ranks these two rows the other way + // round. + // + // `balanced` is two symbols split evenly -- the worst case for a two-symbol row, a full + // bit per byte. `concentrated` has *more* distinct symbols but spends 14 of its 16 bytes + // on one of them, so it carries less information and must score lower. Unweighted it + // scores higher, because it has three log terms against two. + let mut aux = Scratch::new(); + let mut balanced = vec![0u8; 8]; + balanced.extend(std::iter::repeat_n(1u8, 8)); + let mut concentrated = vec![0u8; 14]; + concentrated.extend_from_slice(&[1, 2]); + + let balanced = score(Score::Entropy, &balanced, &mut aux); + let concentrated = score(Score::Entropy, &concentrated, &mut aux); + assert!( + concentrated < balanced, + "concentrated {concentrated} should score below balanced {balanced}" + ); + } + + #[test] + fn the_entropy_scale_separates_rows_closer_than_one_bit() { + // The scale is what makes the score integer-exact: these two rows carry 8.000 and 8.490 + // bits, which both floor to 8. Only multiplying by 256 before the cast keeps them apart, + // so this is the assertion that a `+ 256.0` or `/ 256.0` scale cannot satisfy. + let mut aux = Scratch::new(); + let even = score(Score::Entropy, &[0, 0, 0, 0, 1, 1, 1, 1], &mut aux); + let skewed = score(Score::Entropy, &[0, 0, 0, 0, 0, 0, 1, 2], &mut aux); + assert!(even < skewed, "{even} < {skewed}"); + } + #[test] fn min_sum_abs_prefers_flat_residuals() { // A horizontal gradient (each pixel = previous + k) filters to a constant under Sub, which // scores far below None. let row: Vec = (0..30u8).map(|i| i.wrapping_mul(3)).collect(); let prev = vec![0u8; row.len()]; - let chosen = choose_min_sum_abs(&row, &prev, 1, &mut Vec::new()); + let chosen = choose_by( + Score::SumAbs, + &row, + &prev, + 1, + &mut Vec::new(), + &mut Vec::new(), + &mut Scratch::new(), + ); assert_eq!(chosen, FilterType::Sub); } } diff --git a/crates/gamut-png/src/ihdr.rs b/crates/gamut-png/src/ihdr.rs index f70e759d..3347ffb6 100644 --- a/crates/gamut-png/src/ihdr.rs +++ b/crates/gamut-png/src/ihdr.rs @@ -41,6 +41,33 @@ impl Ihdr { } } +/// The decoded image's byte cost: `width × height × channels × (2 if the depth is 16 else 1)`. +/// +/// **The single definition** of the quantity PNG budgets. [`crate::PngDecoder`] bounds it before +/// allocating anything, and [`crate::deconstruct`] gates its optional IDAT inflation on the same +/// number, so "a report never allocates more than a decode would" holds structurally instead of +/// being asserted by two constants over two different quantities. +/// +/// It counts the **unpacked** buffer, which is what a decode produces: one byte per sample at +/// depths 1/2/4/8 (sub-byte samples are unpacked, §7.2), two at depth 16. The *filtered* stream is +/// a different, larger quantity — it adds one filter byte per scanline (§9.1) — so the two must +/// not be interchanged. +/// +/// `None` when the product overflows `usize`; the caller decides whether that is an error or a +/// refusal. +pub(crate) fn native_bytes( + width: u32, + height: u32, + channels: usize, + bit_depth: u8, +) -> Option { + let bytes_per_sample = if bit_depth == 16 { 2 } else { 1 }; + (width as usize) + .checked_mul(height as usize)? + .checked_mul(channels)? + .checked_mul(bytes_per_sample) +} + /// Parses and validates a 13-byte IHDR payload (PNG spec §11.2.1). /// /// # Errors @@ -149,6 +176,21 @@ mod tests { assert_eq!(parsed.bits_per_pixel(), 32); } + #[test] + fn native_bytes_counts_unpacked_samples() { + // 4096x4096 RGBA8 is exactly the decoder's 64 MiB default budget — the image the two + // budgets used to disagree about. + assert_eq!(native_bytes(4096, 4096, 4, 8), Some(64 << 20)); + // Depth 16 is the only depth that costs two bytes per sample... + assert_eq!(native_bytes(4096, 4096, 4, 16), Some(128 << 20)); + // ...and every sub-byte depth costs one, because a decode unpacks it (§7.2). A packed + // count would be eight times smaller here, and the row padding would round it up again. + assert_eq!(native_bytes(9, 4, 1, 1), Some(36)); + assert_eq!(native_bytes(9, 4, 1, 8), Some(36)); + // Overflow is refused rather than wrapped: 4 channels past the largest square. + assert_eq!(native_bytes(u32::MAX, u32::MAX, 4, 8), None); + } + #[test] fn parse_accepts_adam7() { let parsed = parse(&payload(3, 2, 8, 2, 1)).unwrap(); diff --git a/crates/gamut-png/src/lib.rs b/crates/gamut-png/src/lib.rs index 2b65a659..b2419d53 100644 --- a/crates/gamut-png/src/lib.rs +++ b/crates/gamut-png/src/lib.rs @@ -55,6 +55,7 @@ mod color; mod crc32; mod decoded; mod decoder; +mod deconstruct; mod encoder; mod filter; mod ihdr; @@ -62,6 +63,11 @@ mod inflate; mod pack; mod palette; mod reduce; +/// The encoder's pipeline stages, re-exported for the out-of-tree benchmark driver (issue #224). +/// Not part of the stable API; see `docs/benchmarking.md`. +#[cfg(feature = "test-support")] +#[doc(hidden)] +pub mod stages; pub use abi::{AbiDeflater, AbiInflater, CODEC_ID_ZLIB, PIXEL_FORMAT_FILTERED_BYTES}; pub use ancillary::{PhysicalUnit, SrgbIntent}; @@ -71,6 +77,10 @@ pub use decoded::{ Chromaticities, Cicp, DecodedPng, IccProfile, PngHeader, PngImage, PngMetadata, TextChunk, }; pub use decoder::{PngDecoder, TransparencyKey, metadata}; +pub use deconstruct::{ + ChunkStats, DEFAULT_MAX_CHUNKS, DeconstructLimits, FilterHistogram, FilterScan, PassStats, + PngReport, Segment, SegmentKind, SkippedFilterScan, deconstruct, deconstruct_with_limits, +}; pub use encoder::PngEncoder; pub use filter::{FilterStrategy, FilterType}; /// The DEFLATE compression level, accepted by [`PngEncoder::with_compression`]. diff --git a/crates/gamut-png/src/pack.rs b/crates/gamut-png/src/pack.rs index 87d5225b..da3c6bbd 100644 --- a/crates/gamut-png/src/pack.rs +++ b/crates/gamut-png/src/pack.rs @@ -17,12 +17,7 @@ pub(crate) fn gray8_scale(bit_depth: u8) -> u8 { /// Packs one-byte-per-sample `samples` (each value `< 2^bit_depth`) into MSB-first bit-packed, /// byte-padded scanlines. `bit_depth` must be 1, 2, or 4. -pub(crate) fn pack_scanlines( - samples: &[u8], - width: usize, - height: usize, - bit_depth: u8, -) -> Vec { +pub fn pack_scanlines(samples: &[u8], width: usize, height: usize, bit_depth: u8) -> Vec { debug_assert!(matches!(bit_depth, 1 | 2 | 4)); let depth = bit_depth as usize; let per_byte = 8 / depth; // samples packed per output byte: 8, 4, or 2 diff --git a/crates/gamut-png/src/reduce.rs b/crates/gamut-png/src/reduce.rs index fa154dde..98a76c8d 100644 --- a/crates/gamut-png/src/reduce.rs +++ b/crates/gamut-png/src/reduce.rs @@ -14,7 +14,7 @@ use std::collections::hash_map::Entry; use crate::pack::gray8_scale; /// A chosen reduced encoding for an image. -pub(crate) enum Reduced { +pub enum Reduced { /// Greyscale at depth 1, 2, 4, or 8 (R=G=B, fully opaque). `samples` holds one byte per pixel: /// the raw value at depth 8, the unscaled code (`value / gray8_scale(depth)`) below it. Gray { @@ -35,6 +35,24 @@ pub(crate) enum Reduced { GrayAlpha16Be(Vec), /// 16-bit RGB (alpha was fully opaque and dropped), pre-serialised big-endian. Rgb16Be(Vec), + /// 8-bit RGB plus a `tRNS` colour key (§11.3.2.1): the alpha channel was binary, every + /// transparent pixel shared one colour, and no opaque pixel used it, so that colour can stand + /// for "transparent" and the fourth channel disappears. + Rgb8Keyed { + /// One RGB triple per pixel. + samples: Vec, + /// The colour a decoder must render as fully transparent. + key: [u8; 3], + }, + /// Greyscale plus a `tRNS` colour key — the greyscale twin of [`Reduced::Rgb8Keyed`]. Always + /// depth 8: a sub-byte depth would have to scale the key too, and the saving over depth 8 is + /// smaller than the risk of getting that wrong. + GrayKeyed { + /// One grey sample per pixel. + samples: Vec, + /// The grey value a decoder must render as fully transparent. + key: u8, + }, /// Indexed colour with the smallest sufficient bit depth. Indexed { /// Index bit depth (1, 2, 4, or 8). @@ -58,6 +76,49 @@ pub(crate) fn index_bit_depth(palette_len: usize) -> u8 { } } +/// Zeroes the colour channels of every fully transparent pixel, leaving alpha alone. Returns +/// `None` when the image has no fully transparent pixel to clean. +/// +/// Nothing a decoder renders changes: at `alpha == 0` the colour channels are invisible by +/// definition. What changes is how well the image *compresses*, in three compounding ways: +/// +/// 1. Transparent pixels all become identical, so `Sub` and `Paeth` filter a run of them to +/// zeros instead of to whatever noise the source happened to carry. +/// 2. [`analyze8`] keys its palette on the whole RGBA quad, so two invisible pixels that differ +/// only in their unseen colour cost two palette entries today. This collapses every +/// transparent pixel to a single entry. +/// 3. It is the precondition for a `tRNS` colour key, which needs one colour to stand for +/// "transparent". +/// +/// One constant, not the neighbouring pixel's colour, and that choice was measured rather than +/// assumed. Inheriting the predecessor flattens a *run* just as well, but leaves every invisible +/// pixel a distinct RGBA quad, so (2) and (3) both fail: on an image alternating visible and +/// invisible pixels it collapsed nothing at all and saved zero bytes. +/// +/// This is *not* lossless in the strict byte sense the rest of this module keeps -- the stored +/// samples change -- which is why it is opt-in via +/// [`PngEncoder::with_transparent_cleanup`](crate::PngEncoder::with_transparent_cleanup) and off +/// by default. `channels` must be 2 (grey + alpha) or 4 (RGBA); layouts without an alpha channel +/// have nothing to clean and return `None`. +pub(crate) fn clean_transparent(pixels: &[u8], channels: usize) -> Option> { + debug_assert!((1..=4).contains(&channels)); + if !channels.is_multiple_of(2) { + return None; // no alpha channel + } + let colour = channels - 1; // colour channels are everything before alpha + if !pixels.chunks_exact(channels).any(|px| px[colour] == 0) { + return None; + } + + let mut out = pixels.to_vec(); + for px in out.chunks_exact_mut(channels) { + if px[colour] == 0 { + px[..colour].fill(0); + } + } + Some(out) +} + /// The RGBA quad a pixel of any supported layout presents: grey replicates into R=G=B, and layouts /// without an alpha channel (the odd channel counts) are opaque. fn pixel_key(px: &[u8], channels: usize) -> [u8; 4] { @@ -73,10 +134,93 @@ fn pixel_key(px: &[u8], channels: usize) -> [u8; 4] { } } +/// A `tRNS` chunk's cost for a greyscale image: one 16-bit sample plus 12 bytes of framing. +const GREY_KEY_COST: usize = 2 + 12; + +/// A `tRNS` chunk's cost for truecolour: three 16-bit samples plus 12 bytes of framing. +const RGB_KEY_COST: usize = 6 + 12; + +/// Whether a colour key could possibly apply, before paying for the scan that looks for one. +/// +/// Needs an alpha channel to drop (`channels` even) and something for it to be carrying: an +/// all-opaque image is better served by the plain alpha *drop*, which costs no chunk at all. +/// +/// Split out, like [`keyed_size`], because [`write_reduced_or_native`] races the winning estimate +/// against the unreduced encoding — so perturbing this decision usually changes which candidate is +/// *offered* without changing the bytes that finally win, which makes it invisible from outside. +/// +/// [`write_reduced_or_native`]: crate::PngEncoder +fn may_have_colour_key(all_opaque: bool, channels: usize) -> bool { + !all_opaque && channels.is_multiple_of(2) +} + +/// Raw bytes a colour-key encoding costs: one sample per pixel for greyscale or three for +/// truecolour, plus the `tRNS` chunk that makes it lawful. +fn keyed_size(pixel_count: usize, all_gray: bool) -> usize { + if all_gray { + pixel_count + GREY_KEY_COST + } else { + pixel_count * 3 + RGB_KEY_COST + } +} + +/// The colour that can stand for "transparent", if a `tRNS` colour key applies at all. +/// +/// Three conditions, all necessary (§11.3.2.1 gives a decoder exactly one transparent colour, not +/// a mask): +/// +/// 1. every alpha is 0 or 255 — a partially transparent pixel cannot be expressed by a key; +/// 2. at least one pixel is transparent — otherwise the plain alpha *drop* already applies and is +/// strictly better, since it costs no chunk; +/// 3. every transparent pixel shares one colour, and **no opaque pixel uses it** — otherwise the +/// key would erase a pixel that should be visible. +/// +/// Condition 3 is why +/// [`PngEncoder::with_transparent_cleanup`](crate::PngEncoder::with_transparent_cleanup) pairs +/// with this: it collapses every invisible pixel to one colour, which is precisely what a key +/// needs. Without it, a source whose transparent pixels carry different unseen colours has no key +/// available and keeps its alpha channel. +/// +/// Condition 2 needs no check of its own: `candidate` is assigned in the `alpha == 0` arm and +/// nowhere else, so "some pixel is transparent" is exactly `candidate.is_some()` and the +/// `candidate?` below discharges it. Callers reach here only through +/// [`may_have_colour_key`], which already requires `!all_opaque`, and any alpha that is neither 0 +/// nor 255 returns early — so in practice the `?` never fires; it is the total spelling of a +/// condition the caller gate has already established. +/// +/// Two passes rather than one: the candidate is not known until the first transparent pixel is +/// seen, so proving no *earlier* opaque pixel used it needs a second look. The second pass only +/// runs when the first has already established a candidate. +fn colour_key(pixels: &[u8], channels: usize) -> Option<[u8; 4]> { + debug_assert!(channels == 2 || channels == 4); + let mut candidate: Option<[u8; 4]> = None; + for px in pixels.chunks_exact(channels) { + let key = pixel_key(px, channels); + match key[3] { + 0 => match candidate { + // A second transparent colour: no single key can stand for both. + Some(seen) if seen[..3] != key[..3] => return None, + Some(_) => {} + None => candidate = Some(key), + }, + 255 => {} + // Partial transparency cannot be expressed as a colour key. + _ => return None, + } + } + let candidate = candidate?; + // The key must name a colour nothing visible uses. + let collides = pixels.chunks_exact(channels).any(|px| { + let key = pixel_key(px, channels); + key[3] == 255 && key[..3] == candidate[..3] + }); + (!collides).then_some(candidate) +} + /// Analyses interleaved 8-bit samples (`channels`: 1 = grey, 2 = grey+alpha, 3 = RGB, 4 = RGBA) /// and returns the smallest lossless reduction that beats the input encoding, or `None` to keep it /// as-is. -pub(crate) fn analyze8(pixels: &[u8], channels: usize) -> Option { +pub fn analyze8(pixels: &[u8], channels: usize) -> Option { debug_assert!((1..=4).contains(&channels)); let pixel_count = pixels.len() / channels; @@ -139,11 +283,16 @@ pub(crate) fn analyze8(pixels: &[u8], channels: usize) -> Option { } else { usize::MAX }; + let key = may_have_colour_key(all_opaque, channels) + .then(|| colour_key(pixels, channels)) + .flatten(); + let keyed_size = key.map_or(usize::MAX, |_| keyed_size(pixel_count, all_gray)); let best = palette_size .min(gray_size) .min(gray_alpha_size) - .min(rgb_size); + .min(rgb_size) + .min(keyed_size); if best >= input_size { return None; // no reduction is smaller } @@ -165,6 +314,27 @@ pub(crate) fn analyze8(pixels: &[u8], channels: usize) -> Option { out.push(key[3]); } Some(Reduced::GrayAlpha8(out)) + } else if let Some(key) = key + && best == keyed_size + { + if all_gray { + Some(Reduced::GrayKeyed { + samples: pixels + .chunks_exact(channels) + .map(|px| pixel_key(px, channels)[0]) + .collect(), + key: key[0], + }) + } else { + let mut out = Vec::with_capacity(pixel_count * 3); + for px in pixels.chunks_exact(channels) { + out.extend_from_slice(&pixel_key(px, channels)[0..3]); + } + Some(Reduced::Rgb8Keyed { + samples: out, + key: [key[0], key[1], key[2]], + }) + } } else if best == rgb_size { let mut out = Vec::with_capacity(pixel_count * 3); for px in pixels.chunks_exact(channels) { @@ -181,7 +351,7 @@ pub(crate) fn analyze8(pixels: &[u8], channels: usize) -> Option { /// widening) is demoted and re-analysed at 8 bits — the demotion alone halves the payload, so it /// always reduces. Otherwise only the 16-bit-native channel reductions (grey, alpha drop) apply; /// PNG has no 16-bit palette. -pub(crate) fn analyze16(samples: &[u16], channels: usize) -> Option { +pub fn analyze16(samples: &[u16], channels: usize) -> Option { debug_assert!((1..=4).contains(&channels)); if let Some(demoted) = demote16(samples) { let further = analyze8(&demoted, channels); @@ -246,6 +416,35 @@ fn be_bytes(samples: impl Iterator) -> Vec { samples.flat_map(u16::to_be_bytes).collect() } +/// Orders the palette so the encoding costs less, returning the entries in their new order. +/// +/// Index order is not free: it decides the `tRNS` chunk's length, and it decides what the row +/// filters see, since a filtered index stream is the *difference* between neighbouring indices. +/// Two rules, in priority order: +/// +/// 1. **Transparent entries first**, least opaque first. `tRNS` may be shorter than `PLTE` and +/// every omitted entry defaults to opaque (§11.3.2.1), so gathering the transparent entries at +/// the front makes the trailing-opaque trim below cut as much as it possibly can. First- +/// appearance order left them scattered, so one late transparent entry pinned the whole chunk +/// to full length. +/// 2. **Then by luminance.** Neighbouring indices become neighbouring brightnesses, so an image +/// with smooth shading produces small index deltas rather than the arbitrary jumps +/// raster-scan discovery order gives — which is what `Sub` and `Paeth` are good at. +/// +/// Rec. 601 luma, integer, because this only has to *order* entries and never round-trips through +/// a pixel. The full modified-Zeng ordering oxipng uses is a further step (#482). +fn ordered_palette(palette: &[[u8; 4]]) -> Vec<[u8; 4]> { + let mut out = palette.to_vec(); + out.sort_by_key(|c| { + let luma = 299 * u32::from(c[0]) + 587 * u32::from(c[1]) + 114 * u32::from(c[2]); + // Alpha first, so every transparent entry sorts ahead of every opaque one -- 255 is the + // maximum, so ordering by alpha *is* "opaque last" and a separate `c[3] == 255` component + // ahead of it can never change the order this returns. + (u32::from(c[3]), luma) + }); + out +} + /// Builds the indexed reduction from the collected palette. fn build_indexed( pixels: &[u8], @@ -253,19 +452,34 @@ fn build_indexed( palette: &[[u8; 4]], palette_index: &HashMap<[u8; 4], u8>, ) -> Reduced { + let ordered = ordered_palette(palette); + // Reindex through the new order. `palette_index` maps a colour to its *discovery* index, so + // this composes discovery -> colour -> final position. + let mut remap = vec![0u8; palette.len()]; + for (position, colour) in ordered.iter().enumerate() { + if let Some(&discovered) = palette_index.get(colour) { + remap[discovered as usize] = position as u8; + } + } let indices: Vec = pixels .chunks_exact(channels) - .map(|px| *palette_index.get(&pixel_key(px, channels)).unwrap_or(&0)) + .map(|px| { + let discovered = *palette_index.get(&pixel_key(px, channels)).unwrap_or(&0); + remap[discovered as usize] + }) .collect(); - let plte: Vec = palette.iter().flat_map(|c| [c[0], c[1], c[2]]).collect(); - let trns = if palette.iter().any(|c| c[3] != 255) { - let mut alphas: Vec = palette.iter().map(|c| c[3]).collect(); - // Trailing fully-opaque entries may be omitted (they default to opaque). + let plte: Vec = ordered.iter().flat_map(|c| [c[0], c[1], c[2]]).collect(); + let trns = if ordered.iter().any(|c| c[3] != 255) { + let mut alphas: Vec = ordered.iter().map(|c| c[3]).collect(); + // Trailing fully-opaque entries may be omitted (they default to opaque). With the + // transparent entries gathered at the front this now trims everything after them. // - // No length guard: this arm runs only when `needs_trns` held, i.e. some entry's alpha is - // not 255, so the `last() == 255` test always halts the loop before the vector empties. - // The `alphas.len() > 1` that used to be here therefore decided nothing, and `>` vs `>=` - // was an equivalent mutant no test could kill (#110) -- removed rather than excluded. + // No length guard: this arm runs only when some entry's alpha is not 255, so the + // `last() == 255` test always halts the loop before the vector empties. Ordering makes + // that argument stronger rather than weaker -- the entry that halts it is at index 0, so + // the loop stops with at least one element left. The `alphas.len() > 1` that used to be + // here therefore decided nothing, and `>` vs `>=` was an equivalent mutant no test could + // kill (#110) -- removed rather than excluded. while alphas.last() == Some(&255) { alphas.pop(); } @@ -274,7 +488,7 @@ fn build_indexed( None }; Reduced::Indexed { - depth: index_bit_depth(palette.len()), + depth: index_bit_depth(ordered.len()), indices, plte, trns, @@ -285,6 +499,68 @@ fn build_indexed( mod tests { use super::*; + #[test] + fn a_colour_key_is_only_possible_with_an_alpha_channel_carrying_something() { + assert!(may_have_colour_key(false, 4), "RGBA with transparency"); + assert!( + may_have_colour_key(false, 2), + "grey+alpha with transparency" + ); + // An all-opaque image drops the channel outright, which costs no chunk. + assert!(!may_have_colour_key(true, 4)); + // No alpha channel to drop in the first place. + assert!(!may_have_colour_key(false, 3)); + assert!(!may_have_colour_key(false, 1)); + } + + #[test] + fn the_keyed_cost_is_the_samples_plus_one_trns_chunk() { + // Greyscale: one byte per pixel, and a tRNS of one 16-bit sample plus 12 framing. + assert_eq!(keyed_size(100, true), 100 + 2 + 12); + // Truecolour: three bytes per pixel, and three 16-bit samples plus 12 framing. + assert_eq!(keyed_size(100, false), 300 + 6 + 12); + // The chunk is a flat cost -- it does not scale with the image. + assert_eq!(keyed_size(0, true), GREY_KEY_COST); + assert_eq!(keyed_size(0, false), RGB_KEY_COST); + } + + #[test] + fn cleaning_declines_when_there_is_nothing_invisible_to_clean() { + // `None` rather than an unchanged copy: the encoder must be able to tell "no work" from + // "work that happened to change nothing", or it allocates a whole image for nothing. + let opaque: Vec = (0..16u8).flat_map(|i| [i, i + 1, i + 2, 255]).collect(); + assert!(clean_transparent(&opaque, 4).is_none()); + + // Layouts with no alpha channel have nothing to clean, whatever the samples say. + assert!(clean_transparent(&opaque, 3).is_none()); + assert!(clean_transparent(&opaque, 1).is_none()); + } + + #[test] + fn cleaning_zeroes_invisible_colour_and_leaves_everything_else() { + let src: Vec = vec![ + 10, 20, 30, 255, // visible + 40, 50, 60, 0, // invisible: colour must go + 70, 80, 90, 128, // partially transparent: still visible, must stay + ]; + let cleaned = clean_transparent(&src, 4).expect("there is a transparent pixel"); + assert_eq!( + cleaned, + vec![ + 10, 20, 30, 255, // + 0, 0, 0, 0, // + 70, 80, 90, 128, + ] + ); + } + + #[test] + fn cleaning_grey_alpha_zeroes_only_the_grey_channel() { + let src: Vec = vec![200, 255, 111, 0, 90, 1]; + let cleaned = clean_transparent(&src, 2).expect("there is a transparent pixel"); + assert_eq!(cleaned, vec![200, 255, 0, 0, 90, 1]); + } + #[test] fn drops_opaque_alpha() { // Opaque, non-grey RGBA -> RGB. @@ -461,6 +737,74 @@ mod tests { assert!(analyze8(&rgb, 3).is_none()); } + /// Rec. 601 luma is the *only* thing separating these five opaque entries -- same alpha, so + /// the first two sort-key components tie -- and the fixture is chosen so collapsing any one of + /// the three weights from a multiply to an add returns a different order: + /// + /// | entry | `299*c0 + 587*c1 + 114*c2` | `299 +` | `587 +` | `114 +` | + /// | --- | --- | --- | --- | --- | + /// | `[255, 0, 0]` | 76 245 | 554 | 76 832 | 76 359 | + /// | `[0, 100, 0]` | 58 700 | 58 999 | 687 | 58 814 | + /// | `[0, 130, 0]` | 76 310 | 76 609 | 717 | 76 424 | + /// | `[0, 0, 255]` | 29 070 | 29 369 | 29 657 | 369 | + /// | `[0, 49, 0]` | 28 763 | 29 062 | 636 | 28 877 | + /// + /// Every other palette fixture in the crate happens to have discovery order equal to sorted + /// order, so none of them can tell [`ordered_palette`] from the identity, let alone tell one + /// weight from another. The input order below is deliberately not the expected order. + #[test] + fn the_palette_orders_by_rec_601_luma() { + let palette = [ + [255, 0, 0, 255], + [0, 100, 0, 255], + [0, 130, 0, 255], + [0, 0, 255, 255], + [0, 49, 0, 255], + ]; + assert_eq!( + ordered_palette(&palette), + vec![ + [0, 49, 0, 255], + [0, 0, 255, 255], + [0, 100, 0, 255], + [255, 0, 0, 255], + [0, 130, 0, 255], + ] + ); + } + + /// Rule 1 of [`ordered_palette`] earning its keep, end to end through [`build_indexed`]. + /// + /// The transparent entry is discovered *last* here: the raster scan meets opaque white, then + /// opaque red, and only then the invisible pixels. In discovery order the `tRNS` alphas would + /// be `[255, 255, 0]`, which the trailing-opaque trim cannot shorten at all -- one late + /// transparent entry pins the chunk to full length. Sorting transparent-first makes them + /// `[0, 255, 255]`, and the trim cuts two of the three. + #[test] + fn a_late_transparent_entry_moves_to_index_zero_and_shortens_trns() { + let mut rgba = Vec::new(); + for i in 0..80u32 { + if i % 2 == 0 { + rgba.extend_from_slice(&[255, 255, 255, 255]); // opaque white + } else { + rgba.extend_from_slice(&[200, 10, 10, 255]); // opaque red + } + } + rgba.extend_from_slice(&[0, 0, 0, 0].repeat(40)); // invisible, discovered last + + match analyze8(&rgba, 4) { + Some(Reduced::Indexed { plte, trns, .. }) => { + assert_eq!( + plte, + vec![0, 0, 0, 200, 10, 10, 255, 255, 255], + "transparent first, then opaque by luma" + ); + assert_eq!(trns, Some(vec![0]), "the trim reaches every opaque entry"); + } + _ => panic!("expected Indexed"), + } + } + #[test] fn palette_with_transparency_emits_trns() { let rgba = [ @@ -555,6 +899,56 @@ mod tests { } } + /// `Reduced::GrayKeyed` -- the greyscale twin of `Rgb8Keyed`, reachable and correct but + /// produced by nothing else in the suite, so `analyze8`'s `all_gray` split inside the keyed + /// arm had no test that could see it. + /// + /// Grey + binary alpha, every invisible pixel sharing grey 7, and 64 distinct opaque grey + /// levels 8..=71 that no invisible pixel can collide with. The estimates that race + /// (`pixel_count` = 256): + /// + /// - keyed: `256 + GREY_KEY_COST` = **270** + /// - grey + alpha: `256 * 2` = 512, which is also the input size, so it cannot win + /// - palette: 65 entries needs depth 8, `256 + 65 * 4 + 24` = 540 + /// + /// 65 entries is what keeps the palette out of the race: below 17 the index depth drops to 4 + /// and the palette would win on a fixture this large. + #[test] + fn binary_alpha_grey_reduces_to_a_colour_keyed_greyscale() { + let ga: Vec = (0..256u32) + .flat_map(|i| { + if i.is_multiple_of(5) { + [7, 0] // invisible, all one grey + } else { + [8 + (i % 64) as u8, 255] + } + }) + .collect(); + + match analyze8(&ga, 2) { + Some(Reduced::GrayKeyed { samples, key }) => { + assert_eq!(key, 7, "the one grey every invisible pixel carries"); + assert_eq!(samples.len(), 256, "one sample per pixel, alpha gone"); + // The key erases whatever wears it, so nothing visible may wear it. + for (i, px) in ga.as_chunks::<2>().0.iter().enumerate() { + if px[1] == 255 { + assert_ne!(samples[i], key, "visible pixel {i} would be erased"); + } + } + } + other => panic!( + "expected GrayKeyed, got {}", + match other { + Some(Reduced::Indexed { .. }) => "Indexed", + Some(Reduced::GrayAlpha8(_)) => "GrayAlpha8", + Some(Reduced::Rgb8Keyed { .. }) => "Rgb8Keyed", + Some(_) => "some other reduction", + None => "no reduction", + } + ), + } + } + #[test] fn grey_alpha_noise_keeps_its_encoding() { let ga: Vec = (0..600u32) diff --git a/crates/gamut-png/src/stages.rs b/crates/gamut-png/src/stages.rs new file mode 100644 index 00000000..b8a4c87f --- /dev/null +++ b/crates/gamut-png/src/stages.rs @@ -0,0 +1,22 @@ +//! The encoder's pipeline stages, exposed so they can be timed one at a time (issue #224). +//! +//! A `benches/` target compiles as a separate crate, so it can only reach `pub` items — and the +//! encoder's stages are all crate-private, by design. Rather than widen the shipped API or split +//! working code apart to be reachable, this module re-exports exactly the stage entry points a +//! benchmark drives, behind the `test-support` feature. +//! +//! **No SemVer guarantee.** This is gamut's own harness, not API to pin, and it is `doc(hidden)` +//! for that reason. The `gamut` umbrella never enables the feature, so the shipped surface and +//! `mise run check-ffi-features` are unaffected. +//! +//! It is deliberately re-exports and nothing else — no wrapper bodies. A wrapper would be an +//! executable line that no gate ever runs (bench targets carry `test = false`, so neither +//! `cargo test`, `cargo llvm-cov` nor `cargo mutants` reach them), which would both drag the +//! coverage floor and generate unkillable mutants. `.cargo/mutants.toml` already states the rule +//! this follows, in its `crates/gamut/**` entry: "pure feature-gated re-exports (no function +//! bodies), so it carries no logic of its own to mutate." + +pub use crate::crc32::Crc32; +pub use crate::filter::filter_image; +pub use crate::pack::pack_scanlines; +pub use crate::reduce::{Reduced, analyze8, analyze16}; diff --git a/crates/gamut-png/tests/accounting.rs b/crates/gamut-png/tests/accounting.rs new file mode 100644 index 00000000..fa098bf0 --- /dev/null +++ b/crates/gamut-png/tests/accounting.rs @@ -0,0 +1,814 @@ +//! Byte-accounting totality for [`gamut_png::deconstruct`] (issue #224): every PNG's segments +//! must tile `0..len` exactly, and the reported figures must match what the file actually holds. +//! +//! The fixtures come from **libpng**, not from gamut's encoder, wherever the claim is about +//! reading a foreign file: interlaced streams, forced filters and sub-byte depths are all things +//! `gamut_png::PngEncoder` cannot write, so a gamut-only corpus could not reach them, and a +//! filter histogram checked against gamut's own choice would be self-consistent rather than +//! correct. + +mod common; + +use std::time::Instant; + +use gamut_core::{Dimensions, EncodeImage, ImageRef, Rgb8, Rgba8}; +use gamut_png::{ + ChunkStats, DeconstructLimits, FilterScan, FilterStrategy, FilterType, PngEncoder, Segment, + SegmentKind, SkippedFilterScan, deconstruct, deconstruct_with_limits, +}; + +/// Folds over the segments asserting: non-empty, first starts at 0, each end chains to the next +/// start (contiguous, non-overlapping), and the last ends at `len` — the every-byte invariant. +/// Deliberately re-derived here rather than trusting `is_fully_classified`, which is the thing +/// under test. +fn assert_covers(segments: &[Segment], len: usize) { + assert!(!segments.is_empty(), "at least one segment"); + assert_eq!(segments[0].range.start, 0, "coverage starts at 0"); + for pair in segments.windows(2) { + assert_eq!( + pair[0].range.end, pair[1].range.start, + "segments are contiguous and non-overlapping" + ); + } + assert_eq!( + segments.last().expect("non-empty").range.end, + len, + "coverage runs to end of file" + ); + for s in segments { + assert!(s.range.end > s.range.start, "no empty segment: {s:?}"); + } +} + +/// A deterministic RGB pattern with enough local structure that filters differ between rows. +fn rgb(w: u32, h: u32) -> Vec { + let mut out = Vec::with_capacity((w * h * 3) as usize); + for y in 0..h { + for x in 0..w { + out.push((x ^ y) as u8); + out.push(x.wrapping_mul(3).wrapping_add(y) as u8); + out.push(x.wrapping_add(y.wrapping_mul(7)) as u8); + } + } + out +} + +fn encode_rgb(w: u32, h: u32) -> Vec { + let src = rgb(w, h); + let dims = Dimensions::new(w, h).expect("valid dimensions"); + let image = ImageRef::::new(&src, dims).expect("buffer matches dimensions"); + let mut png = Vec::new(); + PngEncoder::new() + .encode_image(image, &mut png) + .expect("encode"); + png +} + +#[test] +fn segments_tile_every_byte_of_a_gamut_encode() { + for (w, h) in [(1, 1), (17, 13), (64, 40)] { + let png = encode_rgb(w, h); + let report = deconstruct(&png).expect("deconstruct"); + assert_covers(&report.segments, png.len()); + assert!(report.is_fully_classified(), "{report:?}"); + assert!(report.is_intact(), "{report:?}"); + assert_eq!(report.file_len, png.len()); + } +} + +#[test] +fn segments_tile_every_byte_of_every_libpng_colour_type_and_depth() { + for &(color_type, depth) in common::TABLE_12 { + for interlace in [false, true] { + let png = common::libpng_fixture(17, 13, color_type, depth, interlace); + let report = deconstruct(&png).unwrap_or_else(|e| { + panic!("deconstruct ct={color_type} depth={depth} interlace={interlace}: {e:?}") + }); + assert_covers(&report.segments, png.len()); + assert!( + report.is_intact(), + "ct={color_type} depth={depth} interlace={interlace}: {report:?}" + ); + assert_eq!(report.header.bit_depth, depth); + assert_eq!(report.header.interlaced, interlace); + } + } +} + +#[test] +fn chunk_totals_match_an_independent_scan() { + let png = encode_rgb(40, 30); + let report = deconstruct(&png).expect("deconstruct"); + + // A naive second scan written here, so a defect in the walk's accumulation cannot agree with + // itself. 8 signature bytes, then `length || type || data || crc`. + let mut at = 8usize; + let mut seen: Vec<([u8; 4], usize, usize)> = Vec::new(); + while at + 12 <= png.len() { + let len = u32::from_be_bytes([png[at], png[at + 1], png[at + 2], png[at + 3]]) as usize; + let ty = [png[at + 4], png[at + 5], png[at + 6], png[at + 7]]; + match seen.iter_mut().find(|(t, _, _)| *t == ty) { + Some(entry) => { + entry.1 += 1; + entry.2 += len; + } + None => seen.push((ty, 1, len)), + } + at += 12 + len; + } + assert_eq!(at, png.len(), "the naive scan must consume the file too"); + + let got: Vec<_> = report + .chunks + .iter() + .map(|c| (c.chunk_type, c.count, c.payload_bytes)) + .collect(); + assert_eq!(got, seen, "chunk table, in first-appearance order"); + + // Signature + every chunk's payload and framing is the whole file. + let total: usize = report.chunks.iter().map(ChunkStats::total_bytes).sum(); + assert_eq!(total + 8, png.len()); + assert_eq!(report.framing_bytes(), report.chunks.len() * 12); +} + +/// A chunk type that appears more than once must accumulate, not overwrite. +/// +/// Every other fixture here carries at most one chunk of each type, so the accumulate arm of the +/// chunk table never ran: `count` stayed at the 1 it is inserted with and `payload_bytes` at the +/// first chunk's length, and no assertion could tell. +#[test] +fn repeated_chunk_types_accumulate_count_and_payload() { + let first: &[u8] = b"Author\0alice"; + let second: &[u8] = b"Comment\0a considerably longer comment"; + let png = common::png_from_chunks(&[ + common::chunk(b"IHDR", &common::ihdr_payload(4, 4, 8, 2, 0)), + common::chunk(b"tEXt", first), + common::chunk(b"tEXt", second), + common::chunk(b"IDAT", &common::zlib(&[0u8; 4 * (4 * 3 + 1)])), + common::chunk(b"IEND", &[]), + ]); + + let report = deconstruct(&png).expect("deconstruct"); + assert_covers(&report.segments, png.len()); + + let text = report.chunk(b"tEXt").expect("tEXt accounted"); + assert_eq!(text.count, 2, "both chunks counted"); + assert_eq!( + text.payload_bytes, + first.len() + second.len(), + "payloads summed, not overwritten" + ); + assert_eq!(text.framing_bytes(), 24, "12 framing bytes per chunk"); + assert_eq!(text.total_bytes(), first.len() + second.len() + 24); + // The table lists each type once, in first-appearance order. + assert_eq!( + report + .chunks + .iter() + .map(|c| c.chunk_type) + .collect::>(), + vec![*b"IHDR", *b"tEXt", *b"IDAT", *b"IEND"] + ); +} + +/// A stream large enough to split across several IDAT chunks: the same accumulation, on the path +/// that actually produces it in production rather than a hand-built file. +/// A synthetic chunk type for the quadratic regression fixture below: four lowercase letters, so +/// it is ancillary, private, and can never collide with `IHDR`, `IDAT` or `IEND`. 26⁴ = 456 976 +/// distinct types, comfortably more than the fixture uses. +fn synthetic_type(i: usize) -> [u8; 4] { + [ + b'a' + (i % 26) as u8, + b'a' + (i / 26 % 26) as u8, + b'a' + (i / 676 % 26) as u8, + b'a' + (i / 17_576 % 26) as u8, + ] +} + +/// Deconstruction must not slow down when every chunk type in the file is distinct. +/// +/// A chunk type is four unvalidated bytes and the walk never drops a chunk, so an attacker +/// chooses how many *distinct* types a file carries — one per 12-byte chunk, if they like. +/// Accumulating the per-type totals with a linear scan made this quadratic in the file length +/// (measured: 4.8 MB → 40.9 s), reachable from `gamut inspect` on an untrusted file. +/// +/// The claim asserted is not "fast" — an absolute wall-clock ceiling is flaky under `llvm-cov` +/// and parallel test binaries — but "the cost does not depend on how many distinct types the file +/// carries". The two halves are byte-for-byte the same length and carry the same number of +/// chunks, differing only in how many types those chunks use, and they run back to back in one +/// process under one load, so each calibrates the other. The fixed path measures ~2–4×; the +/// defect is three orders of magnitude worse, leaving ~5× of headroom above the fix and ~50× +/// below the defect. The structural assertions below mean it is not purely a timing test. +#[test] +fn the_chunk_tally_does_not_slow_down_when_every_type_is_distinct() { + /// Empty chunks between IHDR and IEND: 12 bytes each, so ~3.1 MB per half. + const CHUNKS: usize = 262_144; + + let build = |distinct: bool| { + let mut framed = Vec::with_capacity(CHUNKS + 2); + framed.push(common::chunk(b"IHDR", &common::ihdr_payload(1, 1, 8, 2, 0))); + framed.extend( + (0..CHUNKS).map(|i| common::chunk(&synthetic_type(if distinct { i } else { 0 }), &[])), + ); + framed.push(common::chunk(b"IEND", &[])); + common::png_from_chunks(&framed) + }; + let repeated = build(false); + let distinct = build(true); + assert_eq!( + repeated.len(), + distinct.len(), + "the two halves must be the same length, or the ratio compares two workloads" + ); + + let started = Instant::now(); + let repeated_report = deconstruct(&repeated).expect("deconstruct"); + let repeated_elapsed = started.elapsed(); + let started = Instant::now(); + let distinct_report = deconstruct(&distinct).expect("deconstruct"); + let distinct_elapsed = started.elapsed(); + + assert_eq!( + distinct_report.chunks.len(), + CHUNKS + 2, + "IHDR, one entry per distinct type, IEND" + ); + assert!( + distinct_report.chunks.iter().all(|stats| stats.count == 1), + "every synthetic type appears exactly once" + ); + assert_eq!( + repeated_report.chunks.len(), + 3, + "IHDR, the one repeated type, IEND" + ); + assert_eq!(repeated_report.chunks[1].count, CHUNKS); + + assert!( + distinct_elapsed < 20 * repeated_elapsed, + "distinct types cost {distinct_elapsed:?} against {repeated_elapsed:?} for the same \ + bytes with one type: the tally is scaling with the number of distinct types" + ); +} + +#[test] +fn a_multi_idat_encode_accumulates_every_idat() { + // Incompressible, so the zlib stream stays far above the 64 KiB per-chunk cap. + let (w, h) = (256u32, 256u32); + let src = common::corpus::noise_rgb(w); + let dims = Dimensions::new(w, h).expect("valid dimensions"); + let image = ImageRef::::new(&src, dims).expect("buffer matches dimensions"); + let mut png = Vec::new(); + PngEncoder::new() + .encode_image(image, &mut png) + .expect("encode"); + + let report = deconstruct(&png).expect("deconstruct"); + assert_covers(&report.segments, png.len()); + let idat = report.chunk(b"IDAT").expect("IDAT accounted"); + assert!(idat.count > 1, "the fixture must actually split: {idat:?}"); + assert_eq!( + idat.payload_bytes, report.idat_compressed, + "the chunk table and the compressed total are the same bytes counted twice" + ); + assert!(report.is_intact(), "{report:?}"); +} + +#[test] +fn trailing_bytes_after_iend_are_a_trailer() { + let mut png = encode_rgb(8, 8); + let clean = png.len(); + png.extend_from_slice(b"junk after the datastream"); + let report = deconstruct(&png).expect("deconstruct"); + + assert_covers(&report.segments, png.len()); + let last = report.segments.last().expect("non-empty"); + assert_eq!(last.kind, SegmentKind::Trailer); + assert_eq!(last.range, clean..png.len()); + // A trailer is not damage the walk failed to classify, but the file is not pristine. + assert!(report.is_fully_classified()); + assert!(!report.is_intact()); +} + +#[test] +fn a_truncated_tail_is_reported_not_an_error() { + let full = encode_rgb(24, 24); + // Cut inside the IDAT payload: the chunk header frames, but its data overruns the input. + let png = &full[..full.len() - 20]; + let report = deconstruct(png).expect("a truncated file still has a header to report on"); + + assert_covers(&report.segments, png.len()); + assert_eq!( + report.segments.last().expect("non-empty").kind, + SegmentKind::Truncated + ); + assert!(report.is_fully_classified()); + assert!(!report.is_intact(), "truncation is not intact"); + // Everything derived from IHDR survives the damage — that is the point of the split. + assert_eq!(report.header.width, 24); + assert!(report.filtered_len > 0); +} + +#[test] +fn unknown_ancillary_and_critical_chunks_are_accounted() { + let extra: [([u8; 4], &[u8]); 2] = [(*b"abCd", &[1, 2, 3]), (*b"ABCD", &[4])]; + let png = common::libpng_with_extra_chunks(12, 9, &extra); + let report = deconstruct(&png).expect("an unknown critical chunk is reported, not an error"); + + assert_covers(&report.segments, png.len()); + let ancillary = report.chunk(b"abCd").expect("unknown ancillary accounted"); + let critical = report.chunk(b"ABCD").expect("unknown critical accounted"); + assert_eq!(ancillary.payload_bytes, 3); + assert_eq!(critical.payload_bytes, 1); + assert!( + ancillary.is_ancillary(), + "lowercase first byte is ancillary" + ); + assert!(!critical.is_ancillary(), "uppercase first byte is critical"); +} + +#[test] +fn a_crc_mismatch_is_flagged_not_fatal() { + let mut png = encode_rgb(8, 8); + // Corrupt IEND's stored CRC, not any payload: every chunk still frames and IHDR still parses, + // so the only thing wrong with the file is a checksum. Corrupting a payload instead would + // make IHDR unparsable, which is a hard error and a different claim. + let last = png.len() - 1; + png[last] ^= 0xFF; + let report = deconstruct(&png).expect("a CRC mismatch is reported, not an error"); + + assert_covers(&report.segments, png.len()); + let bad = report + .segments + .iter() + .filter_map(|s| match s.kind { + SegmentKind::Chunk { + chunk_type, + crc_ok: false, + .. + } => Some(chunk_type), + _ => None, + }) + .collect::>(); + assert_eq!(bad, vec![*b"IEND"], "exactly the damaged chunk is flagged"); + assert!(report.is_fully_classified()); + assert!(!report.is_intact(), "a bad CRC is not intact"); +} + +#[test] +fn the_filter_histogram_matches_the_filter_libpng_was_forced_to_use() { + // libpng, not gamut, picks the filters here, so this cannot be satisfied by a self-consistent + // round trip: it is the differential half of the report's claim. + let forced = [ + (libpng_oracle::FILTER_NONE, FilterType::None), + (libpng_oracle::FILTER_SUB, FilterType::Sub), + (libpng_oracle::FILTER_UP, FilterType::Up), + (libpng_oracle::FILTER_AVG, FilterType::Average), + (libpng_oracle::FILTER_PAETH, FilterType::Paeth), + ]; + for (mask, expected) in &forced { + let png = common::libpng_forced_filter(20, 14, *mask); + let report = deconstruct(&png).expect("deconstruct"); + let filters = report + .filters + .histogram() + .expect("a sound IDAT stream yields a histogram"); + + assert_eq!(filters.total(), 14, "one filter byte per scanline"); + assert_eq!( + filters.count(*expected), + 14, + "every row used {expected:?} (mask {mask:#04x})" + ); + } +} + +/// The histogram must advance one scanline at a time. +/// +/// Every other histogram assertion here forces a single filter for the whole image, which cannot +/// tell a correct per-row walk from one that re-reads the same byte: both report `height` of the +/// one filter. This fixture's rows choose differently, so a stalled cursor collapses the +/// distribution to a single bucket and is visible. +#[test] +fn the_histogram_walks_each_scanline_not_the_first_one_repeatedly() { + const SIDE: u32 = 64; + let src = common::corpus::sprite_rgba(SIDE); + let dims = Dimensions::new(SIDE, SIDE).expect("valid dimensions"); + let image = ImageRef::::new(&src, dims).expect("buffer matches dimensions"); + let mut png = Vec::new(); + PngEncoder::new() + .with_filter(FilterStrategy::MinSumAbs) + .encode_image(image, &mut png) + .expect("encode"); + + let report = deconstruct(&png).expect("deconstruct"); + let h = report.filters.histogram().expect("sound stream"); + assert_eq!(h.total(), SIDE, "one filter byte per scanline"); + + let used = [ + FilterType::None, + FilterType::Sub, + FilterType::Up, + FilterType::Average, + FilterType::Paeth, + ] + .into_iter() + .filter(|&f| h.count(f) > 0) + .count(); + assert!( + used >= 2, + "this fixture's rows must not all choose the same filter, got {used} distinct" + ); +} + +#[test] +fn interlaced_filtered_length_is_the_per_pass_sum() { + // 5x3 and 1x1 leave several Adam7 passes empty; an empty pass contributes no bytes at all, + // not even a filter byte (§7.3). + for (w, h) in [(1, 1), (5, 3), (17, 13)] { + let png = common::libpng_fixture(w, h, libpng_oracle::COLOR_RGB, 8, true); + let report = deconstruct(&png).expect("deconstruct"); + + let summed: usize = report.passes.iter().map(|p| p.filtered_len).sum(); + assert_eq!( + summed, report.filtered_len, + "{w}x{h}: passes sum to the whole" + ); + assert!( + report.passes.iter().all(|p| p.width > 0 && p.height > 0), + "empty passes are omitted, not zero-sized: {:?}", + report.passes + ); + + let rows: u32 = report.passes.iter().map(|p| p.height).sum(); + assert_eq!( + report.filters.histogram().expect("sound stream").total(), + rows, + "{w}x{h}: one filter byte per scanline of every non-empty pass" + ); + } +} + +#[test] +fn sub_byte_row_padding_is_counted() { + // 5 pixels at depth 4 is 20 bits, which pads to 3 bytes per row -- `div_ceil`, not `/`. + let png = common::libpng_fixture(5, 3, libpng_oracle::COLOR_GRAY, 4, false); + let report = deconstruct(&png).expect("deconstruct"); + + assert_eq!(report.passes.len(), 1, "not interlaced"); + assert_eq!(report.passes[0].row_bytes, 3); + assert_eq!(report.filtered_len, 3 * (3 + 1)); +} + +#[test] +fn a_corrupt_zlib_stream_with_a_valid_crc_yields_no_histogram() { + // The only fixture that falsifies `is_intact`'s `filters.is_some()` conjunct on its own: + // framing is perfect, every CRC is valid, and only the compressed payload is nonsense. + let png = common::png_with_garbage_idat(16, 8); + let report = deconstruct(&png).expect("a corrupt IDAT is reported, not an error"); + + assert_covers(&report.segments, png.len()); + assert!(report.is_fully_classified()); + assert!( + report.segments.iter().all(|s| match s.kind { + SegmentKind::Chunk { crc_ok, .. } => crc_ok, + _ => true, + }), + "every CRC is valid in this fixture" + ); + assert_eq!( + report.filters, + FilterScan::Skipped(SkippedFilterScan::CorruptStream), + "the scan is the only casualty, and it names why" + ); + assert!( + report.filters.is_damage(), + "a corrupt payload is damage, not a budget refusal" + ); + assert!(!report.is_intact()); + // Framing- and IHDR-derived figures are unaffected. + assert_eq!(report.header.width, 16); + assert!(report.idat_compressed > 0); + assert!(report.filtered_len > 0); +} + +#[test] +fn an_undefined_filter_code_is_named_and_is_damage() { + // The fourth skip reason, and the only one with no fixture of its own: a stream that inflates + // to exactly the right length but whose scanline carries a filter code PNG SS9.1 does not + // define. Without this the `FilterType::from_code` guard can be deleted -- counting an + // undefined code as `None` and reporting a bogus histogram for a hostile file -- and every + // other assertion in the suite still passes. + let filtered = [9u8, 0, 0, 0, 0, 0, 0]; // 2x1 RGB8: one row, 6 bytes, filter code 9. + let png = common::png_from_chunks(&[ + common::chunk(b"IHDR", &common::ihdr_payload(2, 1, 8, 2, 0)), + common::chunk(b"IDAT", &common::zlib(&filtered)), + common::chunk(b"IEND", &[]), + ]); + let report = deconstruct(&png).expect("an undefined filter code is reported, not an error"); + + assert_covers(&report.segments, png.len()); + assert_eq!( + report.filtered_len, 7, + "one row of 6 bytes plus its filter byte" + ); + assert_eq!( + report.filters, + FilterScan::Skipped(SkippedFilterScan::UndefinedFilterCode), + "the scan names the undefined code rather than any other reason" + ); + assert!( + report.filters.is_damage(), + "an undefined filter code is a statement about the bytes" + ); + assert!(!report.is_intact()); +} + +#[test] +fn an_unread_file_is_intact_but_not_verified() { + // The distinction `is_verified` exists to make. Nothing is known to be wrong with an + // over-budget file, so `is_intact` is true -- but its IDAT was never inflated, so no claim + // about the compressed data has been checked and `is_verified` is false. Collapsing the two + // is what let an archival gate pass a file it never read. + let png = common::png_with_huge_ihdr(); + let report = deconstruct(&png).expect("deconstruct"); + + assert_eq!( + report.filters, + FilterScan::Skipped(SkippedFilterScan::OverBudget) + ); + assert!( + !report.filters.is_damage(), + "a budget refusal is not damage" + ); + assert!( + !report.filters.is_counted(), + "and it is not a reading either" + ); + assert!(report.is_intact(), "nothing is known against this file"); + assert!( + !report.is_verified(), + "but nothing about its compressed data was checked" + ); +} + +#[test] +fn the_chunk_ceiling_admits_exactly_its_own_count_and_refuses_one_more() { + // The chunk count is chosen by the input -- a chunk costs 12 bytes and buys a segment -- so + // the walk caps it. Asserted *at the boundary* rather than far past it: a file well over the + // ceiling is refused by `>`, `>=` and `==` alike, so only the exact count separates them. + // Eleven segments here: the signature, IHDR, eight fillers and IEND. + const SEGMENTS: usize = 11; + let mut chunks = vec![common::chunk(b"IHDR", &common::ihdr_payload(1, 1, 8, 0, 0))]; + for _ in 0..8 { + chunks.push(common::chunk(b"crUD", &[])); + } + chunks.push(common::chunk(b"IEND", &[])); + let png = common::png_from_chunks(&chunks); + + let exact = DeconstructLimits::default().with_max_chunks(SEGMENTS); + let report = deconstruct_with_limits(&png, exact) + .expect("a file of exactly the ceiling's size is admitted, not refused"); + assert_eq!(report.segments.len(), SEGMENTS); + assert!(report.is_fully_classified(), "and it reports normally"); + + let one_short = DeconstructLimits::default().with_max_chunks(SEGMENTS - 1); + let err = deconstruct_with_limits(&png, one_short) + .expect_err("one past the ceiling the walk refuses rather than allocating"); + assert!( + err.to_string().contains("more chunks"), + "the error names the ceiling it hit, got: {err}" + ); +} + +#[test] +fn the_image_budget_is_the_callers_to_set() { + // `with_max_image_bytes` has to be observable, or the walk silently keeps the decoder's + // default and `deconstruct_with_limits` is `deconstruct` with extra steps. A one-byte budget + // turns an ordinary small file -- comfortably scanned under the default -- into a refusal. + let png = common::minimal_png(); + assert!( + deconstruct(&png).expect("deconstruct").filters.is_counted(), + "the fixture is scanned under the default budget" + ); + + let stingy = DeconstructLimits::default().with_max_image_bytes(1); + let report = deconstruct_with_limits(&png, stingy).expect("a budget refusal is not an error"); + assert_eq!( + report.filters, + FilterScan::Skipped(SkippedFilterScan::OverBudget), + "the caller's budget decides, not the decoder's default" + ); +} + +#[test] +fn a_sound_file_is_both_read_and_verified() { + // The positive side of `is_counted` and `is_verified`. Without it both can be pinned to + // `false` by the negative cases alone -- an over-budget file satisfies every assertion they + // make -- and the verdict a gate depends on would be one that always says no. + let png = common::minimal_png(); + let report = deconstruct(&png).expect("deconstruct"); + + assert!( + report.filters.is_counted(), + "a sound stream is read, not skipped" + ); + assert!(report.filters.histogram().is_some(), "so it has counts"); + assert!(report.is_intact(), "and nothing is held against it"); + assert!( + report.is_verified(), + "which together with having been read is what verification means" + ); +} + +#[test] +fn an_over_budget_image_reports_everything_but_the_histogram() { + // A hand-built IHDR claiming 2^30 x 2^30 with a tiny IDAT: the image it implies is far past + // the decoder's byte budget, so the walk must decline to inflate rather than try. Without + // this the budget comparison is never exercised. + let png = common::png_with_huge_ihdr(); + let report = deconstruct(&png).expect("an oversized header is reported, not an error"); + + assert_covers(&report.segments, png.len()); + assert_eq!( + report.filters, + FilterScan::Skipped(SkippedFilterScan::OverBudget), + "declined: over the decoder's byte budget" + ); + assert!( + report.native_bytes().expect("representable") > (64 << 20), + "the implied image is huge" + ); + assert_eq!(report.header.width, 1 << 30); + // And so this file is *not* reported as damaged: nothing here can tell whether its IDAT is + // sound, and no decoder in the workspace could read it either, so claiming damage would be + // claiming knowledge the walk does not have. + assert!(!report.filters.is_damage()); + assert!(report.is_intact(), "{report:?}"); +} + +/// An image exactly at the decoder's byte budget must still be scanned. +/// +/// 4096x4096 RGBA8 is 67 108 864 native bytes — the default budget to the byte — but 67 112 960 +/// *filtered*, one more per scanline. A budget stated over the filtered stream therefore declined +/// it, and `is_intact` reported an image the decoder decodes as damaged. Cheap despite the +/// dimensions: nothing allocates `filtered_len`, and the 16-byte IDAT stops the scan at the +/// length check, so the reason is `LengthMismatch` — the file was scanned — and never +/// `OverBudget`. +#[test] +fn an_image_at_the_decoders_byte_budget_is_still_scanned() { + let png = common::png_from_chunks(&[ + common::chunk(b"IHDR", &common::ihdr_payload(4096, 4096, 8, 6, 0)), + common::chunk(b"IDAT", &common::zlib(&[0u8; 16])), + common::chunk(b"IEND", &[]), + ]); + let report = deconstruct(&png).expect("deconstruct"); + + assert_eq!( + report.native_bytes(), + Some(64 << 20), + "exactly the decoder's default budget" + ); + assert!( + report.filtered_len > 64 << 20, + "and past it once the filter bytes are counted: {}", + report.filtered_len + ); + assert_eq!( + report.filters, + FilterScan::Skipped(SkippedFilterScan::LengthMismatch), + "scanned, and stopped by this file's short stream — not declined for budget" + ); +} + +/// A header whose filtered stream overflows `usize` still reports, and its ratio is finite. +/// +/// §11.2.1 allows dimensions up to 2³¹−1 each, so 2³¹−1 square at RGBA16 implies 2⁶⁵ filtered +/// bytes: `filtered_len` saturates to 0 rather than wrapping, and `idat_ratio` would otherwise +/// divide by it. Thirteen header bytes reach this, and `gamut inspect` prints the ratio for every +/// file it reads, so the guard in `idat_ratio` is live code on a hostile-input path — not the dead +/// branch a filtered-stream budget would have made it. +#[test] +fn a_header_whose_stream_overflows_reports_a_zero_ratio_rather_than_dividing_by_it() { + let png = common::png_from_chunks(&[ + common::chunk( + b"IHDR", + &common::ihdr_payload(0x7FFF_FFFF, 0x7FFF_FFFF, 16, 6, 0), + ), + common::chunk(b"IDAT", &common::zlib(&[0u8; 8])), + common::chunk(b"IEND", &[]), + ]); + let report = deconstruct(&png).expect("an unrepresentable stream is reported, not an error"); + + assert_covers(&report.segments, png.len()); + assert_eq!( + report.filtered_len, 0, + "the implied stream is not representable" + ); + assert!(report.idat_compressed > 0, "there is a numerator to divide"); + assert_eq!( + report.idat_ratio(), + 0.0, + "no division by zero, and not an infinity" + ); + assert!( + report.passes.is_empty(), + "no pass geometry is representable either" + ); +} + +/// The interlaced twin of the case above, which is where the two overflow checks can disagree. +/// +/// Adam7 splits the image into seven smaller passes, so a header can be unrepresentable overall +/// while every individual pass fits `usize`. `adam7::expected_stream_len` fails on the seven-pass +/// *sum*, so `filtered_len` saturates to 0; `pass_stats` has to fail on the same sum or the report +/// contradicts itself -- seven passes described, and a `filtered_len` of 0 that `idat_ratio` then +/// reports as `0.0%` as though it were a measurement. +#[test] +fn an_interlaced_header_whose_passes_fit_but_whose_sum_does_not_reports_no_geometry() { + let png = common::png_from_chunks(&[ + common::chunk( + b"IHDR", + &common::ihdr_payload(0x7FFF_FFFF, 0x7FFF_FFFF, 16, 6, 1), + ), + common::chunk(b"IDAT", &common::zlib(&[0u8; 8])), + common::chunk(b"IEND", &[]), + ]); + let report = deconstruct(&png).expect("an unrepresentable stream is reported, not an error"); + + assert_covers(&report.segments, png.len()); + assert_eq!( + report.filtered_len, 0, + "the seven-pass sum is not representable" + ); + assert!( + report.passes.is_empty(), + "and the per-pass geometry must saturate with it, not describe seven passes \ + against a zero total" + ); + assert_eq!(report.idat_ratio(), 0.0, "no division by zero"); +} + +#[test] +fn a_file_with_no_header_to_report_on_is_an_error() { + assert!(deconstruct(&[]).is_err(), "empty input"); + assert!(deconstruct(b"not a png at all").is_err(), "bad signature"); + + let signature_only = common::SIGNATURE.to_vec(); + assert!(deconstruct(&signature_only).is_err(), "no chunk at all"); + + let mut first_not_ihdr = common::SIGNATURE.to_vec(); + first_not_ihdr.extend_from_slice(&common::chunk(b"gAMA", &45455u32.to_be_bytes())); + assert!( + deconstruct(&first_not_ihdr).is_err(), + "first chunk is not IHDR" + ); + + let mut bad_ihdr = common::SIGNATURE.to_vec(); + bad_ihdr.extend_from_slice(&common::chunk(b"IHDR", &[0u8; 13])); + assert!(deconstruct(&bad_ihdr).is_err(), "zero dimensions in IHDR"); +} + +#[test] +fn the_derived_ratios_are_the_stated_quotients() { + let png = encode_rgb(32, 24); + let report = deconstruct(&png).expect("deconstruct"); + + let pixels = f64::from(report.header.width) * f64::from(report.header.height); + assert!( + (report.bits_per_pixel() - (report.file_len as f64 * 8.0 / pixels)).abs() < 1e-9, + "bits_per_pixel is the whole file over the pixel count" + ); + assert!( + (report.idat_ratio() - (report.idat_compressed as f64 / report.filtered_len as f64)).abs() + < 1e-9, + "idat_ratio is IDAT over the filtered stream" + ); + assert_eq!( + report.overhead_bytes(), + report.file_len - report.idat_compressed + ); + // A real photo-ish pattern must actually compress, or the fixture is not measuring anything. + assert!(report.idat_ratio() < 1.0, "{}", report.idat_ratio()); +} + +#[test] +fn a_brute_force_encode_still_accounts_and_reports_its_filters() { + // The strategy that costs the most and is most likely to trip an accounting assumption. + let src = rgb(48, 32); + let dims = Dimensions::new(48, 32).expect("valid dimensions"); + let image = ImageRef::::new(&src, dims).expect("buffer matches dimensions"); + let mut png = Vec::new(); + PngEncoder::new() + .with_filter(FilterStrategy::BruteForce) + .encode_image(image, &mut png) + .expect("encode"); + + let report = deconstruct(&png).expect("deconstruct"); + assert_covers(&report.segments, png.len()); + assert!(report.is_intact()); + assert_eq!( + report.filters.histogram().expect("sound stream").total(), + 32 + ); +} diff --git a/crates/gamut-png/tests/backends.rs b/crates/gamut-png/tests/backends.rs index 6c649123..ed79077a 100644 --- a/crates/gamut-png/tests/backends.rs +++ b/crates/gamut-png/tests/backends.rs @@ -22,8 +22,17 @@ use gamut_png::{ // Byte-identical defaults // --------------------------------------------------------------------------------------------- -/// Bytes captured from the encoder **before** the seam existed. Pushing no backend must reproduce -/// them exactly: the registry is inert by construction, not merely "close enough". +/// Bytes captured from the encoder **before** the seam existed, except where a row's re-capture is +/// recorded below. Pushing no backend must reproduce them exactly: the registry is inert by +/// construction, not merely "close enough". +/// +/// This pins the *seam*, not the encoder — so a deliberate encoding improvement re-captures the +/// affected row, and the change is recorded here rather than being absorbed silently: +/// +/// * `rgb8_best_bruteforce`, issue #224: `FilterStrategy::MinBigrams` joined +/// `BRUTE_FORCE_STRATEGIES` and wins on this fixture, taking the IDAT from 36 bytes to 21. The +/// gate catching that is the point of it — an encoder change that made output *larger* would +/// look identical here, and would be a regression. const GOLDEN: [(&str, &str); 11] = [ ( "gray8", @@ -59,7 +68,7 @@ const GOLDEN: [(&str, &str); 11] = [ ), ( "rgb8_best_bruteforce", - "89504e470d0a1a0a0000000d49484452000000080000000808020000004b6d29dc000000244944415478da636160e713c5065856ac58418404828357079a14761d081e5ea3b0ca0100921322178646d81f0000000049454e44ae426082", + "89504e470d0a1a0a0000000d49484452000000080000000808020000004b6d29dc000000154944415478da636460e713c5069856e0008353020008cb701e6f73d8bc0000000049454e44ae426082", ), ( "rgb8_fast", diff --git a/crates/gamut-png/tests/colour_key.rs b/crates/gamut-png/tests/colour_key.rs new file mode 100644 index 00000000..b7e072e6 --- /dev/null +++ b/crates/gamut-png/tests/colour_key.rs @@ -0,0 +1,424 @@ +//! The `tRNS` colour key reduction (issue #224, axis 3): dropping a binary alpha channel by +//! naming one colour "transparent" (§11.3.2.1). +//! +//! This is a *lossless* reduction, so the claim is exact: libpng must decode the keyed file to +//! byte-identical RGBA. That is the only assertion that matters, and it is why every test here +//! goes through the oracle rather than round-tripping gamut against itself — the key is written +//! by gamut and interpreted by libpng, so a round trip could agree on a wrong convention. + +mod common; + +use gamut_core::{Dimensions, EncodeImage, Gray8, GrayAlpha8, ImageRef, Rgb8, Rgba8}; +use gamut_png::{FilterStrategy, Level, PngEncoder, deconstruct}; + +/// 128, not something smaller, and the reason is the whole design of the reduction. +/// +/// A colour key costs a flat 18-byte `tRNS` chunk that DEFLATE cannot touch, and buys an alpha +/// plane that usually compresses very well. So whether it wins is size-dependent, exactly as the +/// palette is: measured on this fixture the analysis offers `Rgb8Keyed` at every size, but +/// `write_reduced_or_native` only takes it once the chunk is amortised. Brute-force filtered at +/// `Level::Best`, keyed against plain RGBA: 32 declines it (279 against 274), 48 takes it (347 +/// against 353), and by 128 it is worth about 7% (863 against 926). +/// +/// That also matters for the *negative* tests below. Asserting "stayed RGBA" at a size where the +/// key would never have been taken anyway proves nothing; at 128 a valid key is taken, so RGBA +/// there is real evidence the reduction declined. The one test that needs the *losing* side of +/// that race says so and picks its own size. +const SIDE: u32 = 128; + +/// The 18 bytes a truecolour `tRNS` adds to an encoding: 4 length + 4 type + 6 payload + 4 CRC. +const TRNS_RGB_CHUNK: usize = 18; + +/// The 14 bytes a greyscale `tRNS` adds: the same framing over a single 16-bit sample. +const TRNS_GRAY_CHUNK: usize = 14; + +fn encode(samples: &[u8]) -> Vec { + encode_at(SIDE, samples) +} + +fn encode_at(side: u32, samples: &[u8]) -> Vec { + let dims = Dimensions::new(side, side).expect("valid dimensions"); + let image = ImageRef::::new(samples, dims).expect("buffer matches dimensions"); + let mut out = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .with_filter(FilterStrategy::BruteForce) + .with_auto_reduce(true) + .encode_image(image, &mut out) + .expect("encode"); + out +} + +/// Whether this pixel is outside the visible shape. +/// +/// A **contiguous** region, and that was measured rather than assumed. Scattering the +/// transparency instead — an avalanche hash over the pixel index — makes the key a net *loss*: +/// the invisible colour then interleaves with the visible gradient and wrecks the RGB channels' +/// compressibility, so `RGB + tRNS` came out at 14 886 bytes against plain RGBA's 14 319 and the +/// race correctly declined it. A solid transparent region keeps the colour channels smooth, which +/// is the shape real sprites and icons have and the shape where dropping the alpha plane pays. +fn outside(x: u32, y: u32) -> bool { + outside_at(x, y, SIDE) +} + +fn outside_at(x: u32, y: u32, side: u32) -> bool { + let cx = i64::from(x) - i64::from(side) / 2; + let cy = i64::from(y) - i64::from(side) / 2; + cx * cx + cy * cy >= (i64::from(side) * i64::from(side)) / 9 +} + +/// Binary alpha, one shared invisible colour, and enough distinct visible colours that a palette +/// is not on the table — so the colour key is the only reduction available. +fn keyable_rgba() -> Vec { + keyable_rgba_at(SIDE) +} + +fn keyable_rgba_at(side: u32) -> Vec { + let mut buf = Vec::with_capacity((side * side * 4) as usize); + for y in 0..side { + for x in 0..side { + if outside_at(x, y, side) { + // Invisible, all sharing one colour no visible pixel below can produce. + buf.extend_from_slice(&[1, 2, 3, 0]); + } else { + buf.extend_from_slice(&[(x * 2) as u8, (y * 2) as u8, 200, 255]); + } + } + } + buf +} + +#[test] +fn a_colour_key_drops_the_alpha_channel_losslessly() { + let src = keyable_rgba(); + let png = encode(&src); + let report = deconstruct(&png).expect("deconstruct"); + + assert_eq!( + libpng_oracle::decode(&png).color_type, + libpng_oracle::COLOR_RGB, + "the alpha channel is gone" + ); + assert!( + report.chunk(b"tRNS").is_some(), + "and a colour key replaced it" + ); + + // The whole claim: libpng renders the key, and every pixel comes back exactly. + let (_, _, rgba) = libpng_oracle::decode_rgba8(&png); + assert_eq!(rgba, src, "the colour key resolves losslessly"); +} + +#[test] +fn the_key_is_written_as_sixteen_bit_big_endian_samples() { + // §11.3.2.1: truecolour tRNS is three 16-bit big-endian samples, not three bytes. At depth 8 + // the high byte of each is zero — a decoder reading it as bytes would key on the wrong + // colour, and libpng's round trip above would fail rather than this, so pin the bytes too. + let png = encode(&keyable_rgba()); + let trns = read_chunk(&png, b"tRNS").expect("tRNS present"); + assert_eq!(trns, vec![0, 1, 0, 2, 0, 3], "the key is (1, 2, 3)"); +} + +#[test] +fn partial_transparency_keeps_the_alpha_channel() { + // A key can only say "fully transparent"; anything in between must keep a real alpha channel. + let mut src = keyable_rgba(); + src[7] = 128; // one pixel's alpha, neither 0 nor 255 + let png = encode(&src); + + assert_eq!( + libpng_oracle::decode(&png).color_type, + libpng_oracle::COLOR_RGBA, + "partial alpha is not expressible as a key" + ); + let (_, _, rgba) = libpng_oracle::decode_rgba8(&png); + assert_eq!(rgba, src); +} + +#[test] +fn a_colour_a_visible_pixel_uses_cannot_be_the_key() { + // The invisible pixels all share (0, 0, 200) — but so does a visible one. Keying on it would + // erase a pixel a viewer should see, so the reduction must decline. + let mut buf = Vec::with_capacity((SIDE * SIDE * 4) as usize); + for y in 0..SIDE { + for x in 0..SIDE { + if outside(x, y) { + buf.extend_from_slice(&[0, 0, 200, 0]); + } else { + buf.extend_from_slice(&[(x * 2) as u8, (y * 2) as u8, 200, 255]); + } + } + } + // Plant the collision on a pixel that is definitely visible: the centre. + let centre = ((SIDE / 2) * SIDE + SIDE / 2) as usize * 4; + buf[centre..centre + 4].copy_from_slice(&[0, 0, 200, 255]); + + let png = encode(&buf); + assert_eq!( + libpng_oracle::decode(&png).color_type, + libpng_oracle::COLOR_RGBA, + "the only candidate key is in use by a visible pixel" + ); + let (_, _, rgba) = libpng_oracle::decode_rgba8(&png); + assert_eq!(rgba, buf); +} + +#[test] +fn two_different_invisible_colours_have_no_single_key() { + let mut src = keyable_rgba(); + // A second transparent colour: no one key can stand for both. + src[0..4].copy_from_slice(&[9, 9, 9, 0]); + let png = encode(&src); + + assert_eq!( + libpng_oracle::decode(&png).color_type, + libpng_oracle::COLOR_RGBA, + "two invisible colours cannot share one key" + ); + let (_, _, rgba) = libpng_oracle::decode_rgba8(&png); + assert_eq!(rgba, src); +} + +#[test] +fn cleanup_makes_an_unkeyable_image_keyable() { + // The compounding case the cleanup pass exists for: transparent pixels carrying different + // unseen colours have no key, until cleaning collapses them to one. + let src = common::corpus::sprite_rgba(SIDE); + let dims = Dimensions::new(SIDE, SIDE).expect("valid dimensions"); + + let mut plain = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .with_auto_reduce(true) + .encode_image( + ImageRef::::new(&src, dims).expect("buffer"), + &mut plain, + ) + .expect("encode"); + + let mut cleaned = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .with_auto_reduce(true) + .with_transparent_cleanup(true) + .encode_image( + ImageRef::::new(&src, dims).expect("buffer"), + &mut cleaned, + ) + .expect("encode"); + + // Whatever each lands on, the visible pixels must survive both. + for png in [&plain, &cleaned] { + let (_, _, rgba) = libpng_oracle::decode_rgba8(png); + for (a, b) in rgba.as_chunks::<4>().0.iter().zip(src.as_chunks::<4>().0) { + if b[3] != 0 { + assert_eq!(a, b, "a visible pixel changed"); + } + assert_eq!(a[3], b[3], "alpha changed"); + } + } + assert!( + cleaned.len() <= plain.len(), + "cleaning must not cost bytes: {} vs {}", + cleaned.len(), + plain.len() + ); +} + +/// The payload of the first chunk of this type, if present. +fn read_chunk(png: &[u8], want: &[u8; 4]) -> Option> { + let mut at = 8usize; + while at + 12 <= png.len() { + let len = u32::from_be_bytes([png[at], png[at + 1], png[at + 2], png[at + 3]]) as usize; + let ty = &png[at + 4..at + 8]; + if ty == want { + return Some(png[at + 8..at + 8 + len].to_vec()); + } + at += 12 + len; + } + None +} + +/// Binary alpha over a grey ramp: the greyscale twin of [`keyable_rgba`]. Grey 7 stands for +/// "invisible" and the visible ramp starts at 8, so no opaque pixel can collide with the key, and +/// 200 distinct visible levels keep a palette out of the race. +fn keyable_grey_alpha() -> Vec { + let mut buf = Vec::with_capacity((SIDE * SIDE * 2) as usize); + for y in 0..SIDE { + for x in 0..SIDE { + if outside(x, y) { + buf.extend_from_slice(&[7, 0]); + } else { + buf.extend_from_slice(&[8 + ((x + y) % 200) as u8, 255]); + } + } + } + buf +} + +/// The greyscale twin of [`a_colour_key_drops_the_alpha_channel_losslessly`], covering +/// `Reduced::GrayKeyed` -- reachable and correct, but produced by nothing else in the suite, so +/// the encoder's arm for it (the `ColorType::Grayscale` choice, and the single 16-bit big-endian +/// `tRNS` sample) had no test that could see it. +/// +/// The win is thinner here than for truecolour: dropping the alpha plane saves one byte per pixel +/// rather than three, while the `tRNS` chunk still costs a flat 14. It is a win regardless -- +/// measured at `SIDE`, brute-force filtered at `Level::Best`, 499 bytes keyed against 626 as +/// `GrayAlpha8`, about 20% -- and it stayed a win at every square from 32 to 256, so no size +/// threshold is needed on this side. +/// +/// The key is grey 7 rather than 0 deliberately: a `tRNS` written little-endian would read +/// `[7, 0]`, which a key of 0 could not tell from the correct `[0, 7]`. +#[test] +fn a_greyscale_colour_key_drops_the_alpha_channel_losslessly() { + let src = keyable_grey_alpha(); + let dims = Dimensions::new(SIDE, SIDE).expect("valid dimensions"); + let mut png = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .with_filter(FilterStrategy::BruteForce) + .with_auto_reduce(true) + .encode_image( + ImageRef::::new(&src, dims).expect("buffer matches dimensions"), + &mut png, + ) + .expect("encode"); + + let dec = libpng_oracle::decode(&png); + assert_eq!( + dec.color_type, + libpng_oracle::COLOR_GRAY, + "the alpha plane is gone" + ); + assert_eq!(dec.bit_depth, 8, "a keyed grey is always depth 8"); + assert_eq!( + read_chunk(&png, b"tRNS").expect("tRNS present"), + vec![0, 7], + "one 16-bit big-endian sample naming grey 7" + ); + + // The whole claim: libpng renders the key, and every pixel comes back exactly. + let (_, _, rgba) = libpng_oracle::decode_rgba8(&png); + let expected: Vec = src + .as_chunks::<2>() + .0 + .iter() + .flat_map(|px| { + let grey = if px[1] == 0 { 7 } else { px[0] }; + [grey, grey, grey, px[1]] + }) + .collect(); + assert_eq!(rgba, expected, "the grey colour key resolves losslessly"); +} + +/// The greyscale twin of [`a_colour_key_that_would_cost_bytes_is_declined`], and the only test +/// that can see the `GrayKeyed` member of `write_reduced_or_native`'s `carries_chunks` set. +/// +/// [`a_greyscale_colour_key_drops_the_alpha_channel_losslessly`] proves `GrayKeyed` is *reachable*, +/// but its fixture wins at every size, so dropping `GrayKeyed` from `carries_chunks` -- emitting +/// the keyed file with no race -- would not change its result. Losing needs a thinner saving: the +/// `tRNS` costs a flat 14 bytes while dropping the alpha plane saves only one byte per pixel, so a +/// mostly-opaque image is where the fixed cost wins. A quarter-width transparent border at 16x16 +/// measures 88 bytes as `GrayAlpha8` against 97 for the key, and the encoder must emit the 88. +#[test] +fn a_greyscale_colour_key_that_would_cost_bytes_is_declined() { + const SMALL: u32 = 16; + // Mostly opaque, so the alpha plane the key removes is cheap to keep. Grey 7 is the invisible + // colour and the visible ramp starts at 8, so the key is valid -- only its cost declines it. + let mut src = Vec::with_capacity((SMALL * SMALL * 2) as usize); + for y in 0..SMALL { + for x in 0..SMALL { + if x < SMALL / 4 || y < SMALL / 4 { + src.extend_from_slice(&[7, 0]); + } else { + src.extend_from_slice(&[8 + ((x + y) % 200) as u8, 255]); + } + } + } + let dims = Dimensions::new(SMALL, SMALL).expect("valid dimensions"); + let encoder = || { + PngEncoder::new() + .with_compression(Level::Best) + .with_filter(FilterStrategy::BruteForce) + }; + let mut chosen = Vec::new(); + encoder() + .with_auto_reduce(true) + .encode_image( + ImageRef::::new(&src, dims).expect("buffer matches dimensions"), + &mut chosen, + ) + .expect("encode"); + assert_eq!( + libpng_oracle::decode(&chosen).color_type, + libpng_oracle::COLOR_GRAY_ALPHA, + "the key is valid at this size, so only its cost can have declined it" + ); + + // What the key would have cost: the grey plane alone through the same configuration, plus the + // flat `tRNS`. Reproducible from outside exactly as the truecolour twin does it. + let grey: Vec = src.as_chunks::<2>().0.iter().map(|px| px[0]).collect(); + let mut keyed = Vec::new(); + encoder() + .with_auto_reduce(false) + .encode_image( + ImageRef::::new(&grey, dims).expect("buffer matches dimensions"), + &mut keyed, + ) + .expect("encode"); + let keyed_len = keyed.len() + TRNS_GRAY_CHUNK; + assert!( + keyed_len > chosen.len(), + "the declined candidate must really be the larger one: keyed {keyed_len} vs GrayAlpha8 {}", + chosen.len() + ); +} + +/// The *losing* side of the race in `write_reduced_or_native`, which its `carries_chunks` set +/// exists for. +/// +/// The other negative tests here stay RGBA because no key was ever *offered* -- partial alpha, two +/// invisible colours, a collision with a visible pixel. This one offers a perfectly valid key and +/// has it declined on size alone, which is the only way the `Rgb8Keyed` member of `carries_chunks` +/// is observable: drop it and the encoder would emit the larger keyed file without racing it. +/// +/// Measured on `keyable_rgba_at(32)`, brute-force filtered at `Level::Best`: plain RGBA is 274 +/// bytes and `RGB + tRNS` is 279 (261 for the RGB stream plus the flat 18-byte chunk). 32 is the +/// largest square where the key loses -- by 48 it already wins, 347 against 353. +#[test] +fn a_colour_key_that_would_cost_bytes_is_declined() { + const SMALL: u32 = 32; + let src = keyable_rgba_at(SMALL); + let chosen = encode_at(SMALL, &src); + assert_eq!( + libpng_oracle::decode(&chosen).color_type, + libpng_oracle::COLOR_RGBA, + "the key is valid at this size, so only its cost can have declined it" + ); + + // What the key would have cost. The encoder's `Rgb8Keyed` arm is the RGB stream through this + // same configuration plus one `tRNS`, so the losing candidate is reproducible from outside. + let rgb: Vec = src + .as_chunks::<4>() + .0 + .iter() + .flat_map(|px| [px[0], px[1], px[2]]) + .collect(); + let dims = Dimensions::new(SMALL, SMALL).expect("valid dimensions"); + let mut keyed = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .with_filter(FilterStrategy::BruteForce) + .with_auto_reduce(false) + .encode_image( + ImageRef::::new(&rgb, dims).expect("buffer matches dimensions"), + &mut keyed, + ) + .expect("encode"); + let keyed_len = keyed.len() + TRNS_RGB_CHUNK; + assert!( + keyed_len > chosen.len(), + "the declined candidate must really be the larger one: keyed {keyed_len} vs RGBA {}", + chosen.len() + ); +} diff --git a/crates/gamut-png/tests/common/corpus.rs b/crates/gamut-png/tests/common/corpus.rs new file mode 100644 index 00000000..9883db50 --- /dev/null +++ b/crates/gamut-png/tests/common/corpus.rs @@ -0,0 +1,143 @@ +//! The efficiency corpus (issue #224): deterministic image generators shared by +//! `benches/encode.rs` and `tests/size_contract.rs`. +//! +//! One file, included by both, because the size contract's budgets are only meaningful if they +//! are measured on the same pixels the benchmark table reports. Two copies would drift, and the +//! drift would be invisible — a budget that no longer describes the row it names. +//! +//! Dependency-free on purpose: the benchmark includes it with `#[path]`, so it must not reach for +//! anything outside `core`/`alloc`. +//! +//! Each generator is one axis of encoder behaviour, and no two overlap. There is no vendored +//! image corpus in this crate (`README.md` says so), so every fixture is generated. + +#![allow(dead_code)] + +/// A deterministic, non-trivial RGB gradient — the workspace's shared bench pattern. Avoids the +/// all-constant fast paths so the measured work reflects realistic entropy. +pub fn gradient_rgb(side: u32) -> Vec { + let mut buf = vec![0u8; (side * side * 3) as usize]; + for y in 0..side { + for x in 0..side { + let i = ((y * side + x) * 3) as usize; + buf[i] = (x ^ y) as u8; + buf[i + 1] = x.wrapping_mul(3).wrapping_add(y) as u8; + buf[i + 2] = x.wrapping_add(y.wrapping_mul(7)) as u8; + } + } + buf +} + +/// Smooth, photograph-like content: three triangle waves at co-prime periods standing in for +/// sinusoids (no floating point in a fixture). Palette-hostile and 16-bit-hostile, so no reduction +/// applies and the whole residual is filtering plus DEFLATE — the row that measures the +/// compressor rather than the analysis. +pub fn photo_rgb(side: u32) -> Vec { + let tri = |v: i64, period: i64| { + let m = v.rem_euclid(period * 2); + let up = if m < period { m } else { period * 2 - m }; + (up * 255 / period) as u8 + }; + let mut buf = vec![0u8; (side * side * 3) as usize]; + for y in 0..side { + for x in 0..side { + let i = ((y * side + x) * 3) as usize; + let (xi, yi) = (i64::from(x), i64::from(y)); + buf[i] = tri(xi + yi, 61); + buf[i + 1] = tri(xi * 2 - yi, 43); + buf[i + 2] = tri(xi + yi * 3, 97); + } + } + buf +} + +/// Incompressible: a full avalanche mix of the byte index. Pins that the encoder does not +/// *expand* random data by more than stored-block framing, and drives `FilterType::None`. +/// +/// Deliberately not the plain `i * 2654435761 >> 24` that `gamut-deflate`'s bench uses. Over a +/// dense index that top byte changes only once every few hundred `i`, so a "noise" row built that +/// way compresses roughly 97x and measures nothing at all. +pub fn noise_rgb(side: u32) -> Vec { + (0..(side * side * 3)) + .map(|i: u32| { + let mut v = i.wrapping_add(0x9E37_79B9); + v ^= v >> 16; + v = v.wrapping_mul(0x21F0_AAAD); + v ^= v >> 15; + v = v.wrapping_mul(0x735A_2D97); + v ^= v >> 15; + v as u8 + }) + .collect() +} + +/// A greyscale ramp presented as RGB: R=G=B everywhere, so the grey reduction applies and two +/// channels disappear before DEFLATE runs. +pub fn grey_as_rgb(side: u32) -> Vec { + let mut buf = vec![0u8; (side * side * 3) as usize]; + for y in 0..side { + for x in 0..side { + let i = ((y * side + x) * 3) as usize; + let v = ((x + y) % 256) as u8; + buf[i] = v; + buf[i + 1] = v; + buf[i + 2] = v; + } + } + buf +} + +/// Exactly 64 distinct colours over two alpha levels: the indexed + tRNS path, which is the +/// biggest structural lever this crate has over libpng-9 (libpng does not auto-palettise). +pub fn palette64_rgba(side: u32) -> Vec { + let mut buf = vec![0u8; (side * side * 4) as usize]; + for y in 0..side { + for x in 0..side { + let i = ((y * side + x) * 4) as usize; + let idx = ((x / 8 + y / 8 * 8) % 64) as u8; + buf[i] = idx.wrapping_mul(4); + buf[i + 1] = idx.wrapping_mul(9); + buf[i + 2] = 255 - idx.wrapping_mul(3); + buf[i + 3] = if idx.is_multiple_of(8) { 0 } else { 255 }; + } + } + buf +} + +/// A sprite: binary alpha, where the fully transparent pixels carry *different* RGB values. +/// +/// That invisible colour noise is what the palette build keys on today, so this is the only entry +/// that can see the dirty-alpha and tRNS-colour-key axes. It is the row to watch when either +/// lands. +pub fn sprite_rgba(side: u32) -> Vec { + let mut buf = vec![0u8; (side * side * 4) as usize]; + let r2 = (i64::from(side) * i64::from(side)) / 9; + for y in 0..side { + for x in 0..side { + let i = ((y * side + x) * 4) as usize; + let cx = i64::from(x) - i64::from(side) / 2; + let cy = i64::from(y) - i64::from(side) / 2; + if cx * cx + cy * cy < r2 { + buf[i] = (x ^ y) as u8; + buf[i + 1] = 0x40; + buf[i + 2] = 0xC0; + buf[i + 3] = 255; + } else { + // Invisible, and deliberately not constant. + buf[i] = x as u8; + buf[i + 1] = y as u8; + buf[i + 2] = (x ^ y) as u8; + buf[i + 3] = 0; + } + } + } + buf +} + +/// One fully opaque colour: the compressible extreme, where the whole reduce cascade applies and +/// chunk framing is most of what is left to measure. +pub fn flat_rgba(side: u32) -> Vec { + (0..(side * side)) + .flat_map(|_| [0x2E, 0x86, 0xC1, 0xFF]) + .collect() +} diff --git a/crates/gamut-png/tests/common/mod.rs b/crates/gamut-png/tests/common/mod.rs index f6c4030b..c571d18a 100644 --- a/crates/gamut-png/tests/common/mod.rs +++ b/crates/gamut-png/tests/common/mod.rs @@ -3,6 +3,9 @@ //! CRC-32 so the builders do not depend on the crate under test. #![allow(dead_code)] // each integration-test binary uses its own subset +/// The efficiency corpus, shared with `benches/encode.rs` (issue #224). +pub mod corpus; + /// The 8-byte PNG signature. pub const SIGNATURE: [u8; 8] = [0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]; @@ -146,3 +149,90 @@ pub fn tiny_exif() -> Vec { 0x49, 0x49, 0x2A, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ] } + +/// Every valid Table-12 colour-type/bit-depth pair, flattened (libpng's `COLOR_*` codes). +pub const TABLE_12: &[(u8, u8)] = &[ + (libpng_oracle::COLOR_GRAY, 1), + (libpng_oracle::COLOR_GRAY, 2), + (libpng_oracle::COLOR_GRAY, 4), + (libpng_oracle::COLOR_GRAY, 8), + (libpng_oracle::COLOR_GRAY, 16), + (libpng_oracle::COLOR_PALETTE, 1), + (libpng_oracle::COLOR_PALETTE, 2), + (libpng_oracle::COLOR_PALETTE, 4), + (libpng_oracle::COLOR_PALETTE, 8), + (libpng_oracle::COLOR_RGB, 8), + (libpng_oracle::COLOR_RGB, 16), + (libpng_oracle::COLOR_GRAY_ALPHA, 8), + (libpng_oracle::COLOR_GRAY_ALPHA, 16), + (libpng_oracle::COLOR_RGBA, 8), + (libpng_oracle::COLOR_RGBA, 16), +]; + +/// A full-size palette for an indexed fixture at `depth`. +fn full_palette(depth: u8) -> Vec<[u8; 3]> { + (0..(1usize << depth)) + .map(|i| [i as u8, (i * 7 + 3) as u8, 255 - i as u8]) + .collect() +} + +/// Encodes a deterministic fixture with libpng (full-size palette for indexed depths). +pub fn libpng_fixture( + width: u32, + height: u32, + color_type: u8, + depth: u8, + interlace: bool, +) -> Vec { + let pixels = sample_bytes(width, height, color_type, depth, 11); + let palette = full_palette(depth); + let opts = libpng_oracle::EncodeOpts { + interlace, + palette: (color_type == libpng_oracle::COLOR_PALETTE).then_some(&palette), + ..libpng_oracle::EncodeOpts::default() + }; + libpng_oracle::encode(&pixels, width, height, color_type, depth, &opts) +} + +/// An 8-bit RGB fixture libpng wrote with exactly one filter on every scanline. `mask` is one of +/// libpng's `FILTER_*` bits, so the *oracle* chooses the filter, not gamut. +pub fn libpng_forced_filter(width: u32, height: u32, mask: u8) -> Vec { + let pixels = sample_bytes(width, height, libpng_oracle::COLOR_RGB, 8, 11); + let opts = libpng_oracle::EncodeOpts { + filters: Some(mask), + ..libpng_oracle::EncodeOpts::default() + }; + libpng_oracle::encode(&pixels, width, height, libpng_oracle::COLOR_RGB, 8, &opts) +} + +/// An 8-bit RGB fixture carrying extra raw chunks written verbatim after IHDR — used for chunk +/// types this crate does not recognise, ancillary and critical alike. +pub fn libpng_with_extra_chunks(width: u32, height: u32, extra: &[([u8; 4], &[u8])]) -> Vec { + let pixels = sample_bytes(width, height, libpng_oracle::COLOR_RGB, 8, 11); + let opts = libpng_oracle::EncodeOpts { + extra_chunks: extra, + ..libpng_oracle::EncodeOpts::default() + }; + libpng_oracle::encode(&pixels, width, height, libpng_oracle::COLOR_RGB, 8, &opts) +} + +/// A structurally perfect PNG whose IDAT payload is not a zlib stream. Every CRC is valid, so +/// only the *compressed data* is damaged — the one input that isolates a decompression failure +/// from a framing failure. +pub fn png_with_garbage_idat(width: u32, height: u32) -> Vec { + png_from_chunks(&[ + chunk(b"IHDR", &ihdr_payload(width, height, 8, 2, 0)), + chunk(b"IDAT", b"this is not a zlib stream"), + chunk(b"IEND", &[]), + ]) +} + +/// A PNG whose IHDR claims 2^30 x 2^30 with a tiny IDAT: the filtered stream it implies is far +/// past any sane inflation budget, so a reader must decline rather than attempt it. +pub fn png_with_huge_ihdr() -> Vec { + png_from_chunks(&[ + chunk(b"IHDR", &ihdr_payload(1 << 30, 1 << 30, 8, 2, 0)), + chunk(b"IDAT", &zlib(&[0u8; 16])), + chunk(b"IEND", &[]), + ]) +} diff --git a/crates/gamut-png/tests/oracle.rs b/crates/gamut-png/tests/oracle.rs index 218914c3..f2f8b478 100644 --- a/crates/gamut-png/tests/oracle.rs +++ b/crates/gamut-png/tests/oracle.rs @@ -406,9 +406,14 @@ fn auto_reduce_cases() -> (Dimensions, [AutoReduceCase; 3]) { expected_type: libpng_oracle::COLOR_GRAY, }, AutoReduceCase { + // Three colours repeating with period 3: DEFLATE squeezes the RGBA stream to + // less than the palette encoding's PLTE + tRNS + framing costs on its own, so + // `write_reduced_or_native` keeps the unreduced form. That is the smaller file, + // which is the contract; `a_palette_is_chosen_when_it_actually_wins` covers the + // other side of that race, and `reduce`'s own unit tests pin the analysis. name: "palette", rgba: palette, - expected_type: libpng_oracle::COLOR_PALETTE, + expected_type: libpng_oracle::COLOR_RGBA, }, AutoReduceCase { name: "opaque", @@ -446,6 +451,54 @@ fn auto_reduce_picks_the_colour_type_the_pixels_allow() { } } +/// The palette side of `write_reduced_or_native`'s race. +/// +/// A palette costs a flat `PLTE` (+ `tRNS`) that DEFLATE cannot compress, so whether it wins is +/// size-dependent: the fixed cost has to be amortised over enough pixels. At 32x32 it is not, and +/// the cases above keep the unreduced form; at 192x192 with the same colour count it is, and the +/// encoder must take the palette. Without this test the palette encoding path would only ever be +/// exercised where it loses. +#[test] +fn a_palette_is_chosen_when_it_actually_wins() { + let (w, h) = (192u32, 192u32); + let dims = Dimensions::new(w, h).unwrap(); + // 64 distinct colours in 8x8 blocks: too many for RGBA to compress away, few enough to index. + let mut src = Vec::with_capacity((w * h * 4) as usize); + for y in 0..h { + for x in 0..w { + let idx = ((x / 8 + y / 8 * 8) % 64) as u8; + src.extend_from_slice(&[ + idx.wrapping_mul(4), + idx.wrapping_mul(9), + 255 - idx.wrapping_mul(3), + 255, + ]); + } + } + + let reduced = encode_auto_reduced(&src, dims); + assert_eq!( + libpng_oracle::decode(&reduced).color_type, + libpng_oracle::COLOR_PALETTE, + "the palette wins once its fixed cost is amortised" + ); + + let mut plain = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .encode_image(ImageRef::::new(&src, dims).unwrap(), &mut plain) + .expect("encode"); + assert!( + reduced.len() < plain.len(), + "and it is smaller: {} vs {}", + reduced.len(), + plain.len() + ); + + let (_, _, rgba) = libpng_oracle::decode_rgba8(&reduced); + assert_eq!(rgba, src, "the palette resolves losslessly"); +} + #[test] fn auto_reduce_is_lossless() { // The claim that makes the reduction safe to enable at all: whatever colour type it chose, @@ -516,18 +569,26 @@ fn extended_auto_reduce_covers_grey_and_sixteen_bit_inputs() { // packed one. The depth/pixel checks above pin the contract that matters. } - // Low-cardinality grey off the scale grid -> a grey palette at 2 bits. + // Low-cardinality grey off the scale grid. `reduce::analyze8` offers a 2-bit grey palette, + // but on a fixture this small and this regular the plain 8-bit grey stream compresses to less + // than the palette's PLTE and framing, so `write_reduced_or_native` keeps grey. Asserted + // exactly: no input reaches this line and comes back paletted, so admitting that as an + // alternative would be a branch nothing can take. The size at which a palette does win, and + // is packed below 8 bits, is covered by its own test at the end of this file. let off_grid: Vec = (0..n).map(|i| [5u8, 9, 200][i % 3]).collect(); let mut png = Vec::new(); encoder() .encode_image(ImageRef::::new(&off_grid, dims).unwrap(), &mut png) .expect("encode"); let dec = libpng_oracle::decode(&png); - assert_eq!(dec.color_type, libpng_oracle::COLOR_PALETTE); - assert_eq!(dec.bit_depth, 2); + assert_eq!( + dec.color_type, + libpng_oracle::COLOR_GRAY, + "off-grid grey stays grey at this size" + ); let (_, _, rgba) = libpng_oracle::decode_rgba8(&png); let expected: Vec = off_grid.iter().flat_map(|&v| [v, v, v, 255]).collect(); - assert_eq!(rgba, expected, "grey palette resolves losslessly"); + assert_eq!(rgba, expected, "off-grid grey resolves losslessly"); // GrayAlpha8 with an all-opaque alpha channel -> plain 8-bit grey. let ga: Vec = (0..n).flat_map(|i| [(i % 89) as u8, 255]).collect(); @@ -635,3 +696,117 @@ fn solid_image_round_trips() { let dec = libpng_oracle::decode(&png); assert_eq!(dec.pixels, src); } + +#[test] +fn every_filter_strategy_survives_the_libpng_round_trip() { + // The end-to-end pin whose absence hid a silent-corruption defect: `MinEntropy` was scored but + // never encoded with, so nothing noticed that a row whose candidates all tied emitted its + // predecessor's residuals. Sweeping the whole enum means a new strategy cannot land unproven. + // + // Deliberately narrow: 3x7 is the smallest corpus size whose rows are short enough for an + // all-distinct-bytes tie, which is exactly the case that used to break. + let (w, h) = (3, 7); + let src = rgb_pattern(w, h); + let dims = Dimensions::new(w, h).unwrap(); + for strategy in [ + FilterStrategy::None, + FilterStrategy::Fixed(FilterType::None), + FilterStrategy::Fixed(FilterType::Sub), + FilterStrategy::Fixed(FilterType::Up), + FilterStrategy::Fixed(FilterType::Average), + FilterStrategy::Fixed(FilterType::Paeth), + FilterStrategy::MinSumAbs, + FilterStrategy::MinEntropy, + FilterStrategy::MinBigrams, + FilterStrategy::BruteForce, + ] { + let mut png = Vec::new(); + PngEncoder::new() + .with_filter(strategy) + .encode_image(ImageRef::::new(&src, dims).unwrap(), &mut png) + .expect("encode"); + let dec = libpng_oracle::decode(&png); + assert_eq!(dec.pixels, src, "{strategy:?} did not round-trip"); + } +} + +/// Sub-byte indexed auto-reduce: the palette wins *and* its index depth drops below 8. +/// +/// `a_palette_is_chosen_when_it_actually_wins` needs 64 colours to make the palette win, which is +/// depth 8 -- so the encoder's `depth < 8` path into `pack::pack_scanlines`, and +/// `reduce::index_bit_depth`'s `3..=4 => 2` arm, were only reached by inputs whose palette the +/// race then declined. +/// +/// Four colours, and **pseudo-random** rather than blocked. Blocked, the RGBA stream compresses +/// away and `write_reduced_or_native` correctly keeps it -- which is exactly why the 64-colour +/// fixture needed 64 colours. Scattered, the four-symbol stream is near its entropy either way, +/// so the 2-bit packing is the whole difference. Measured at 192x192, `Level::Best`: 9500 bytes +/// indexed (36 864 pixels at two bits is 9216 of payload) against 19 135 as RGBA, about 50%. +#[test] +fn a_small_palette_is_packed_to_a_sub_byte_index_depth() { + let (w, h) = (192u32, 192u32); + let dims = Dimensions::new(w, h).unwrap(); + const PALETTE: [[u8; 4]; 4] = [ + [220, 30, 40, 255], + [30, 200, 60, 255], + [40, 60, 210, 255], + [200, 190, 20, 255], + ]; + let mut src = Vec::with_capacity((w * h * 4) as usize); + for y in 0..h { + for x in 0..w { + // A finalizer-quality avalanche over the pixel index. A cheaper mix (one multiply + // and a shift) is periodic in x, and DEFLATE finds the period: the same fixture came + // out at 272 bytes, which would have proved nothing about packing. + let mut hash = y * w + x; + hash ^= hash >> 16; + hash = hash.wrapping_mul(0x7feb_352d); + hash ^= hash >> 15; + hash = hash.wrapping_mul(0x846c_a68b); + hash ^= hash >> 16; + src.extend_from_slice(&PALETTE[(hash & 3) as usize]); + } + } + + let reduced = encode_auto_reduced(&src, dims); + let dec = libpng_oracle::decode(&reduced); + assert_eq!( + dec.color_type, + libpng_oracle::COLOR_PALETTE, + "four colours over 36 864 pixels is a palette" + ); + assert_eq!(dec.bit_depth, 2, "and four entries need only two bits"); + assert_eq!( + read_chunk(&reduced, b"PLTE").expect("PLTE present").len(), + 12, + "four RGB triples" + ); + + let mut plain = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .encode_image(ImageRef::::new(&src, dims).unwrap(), &mut plain) + .expect("encode"); + assert!( + reduced.len() < plain.len(), + "packed indices beat RGBA: {} vs {}", + reduced.len(), + plain.len() + ); + + let (_, _, rgba) = libpng_oracle::decode_rgba8(&reduced); + assert_eq!(rgba, src, "the packed palette resolves losslessly"); +} + +/// The payload of the first chunk of this type, if present. +fn read_chunk(png: &[u8], want: &[u8; 4]) -> Option> { + let mut at = 8usize; + while at + 12 <= png.len() { + let len = u32::from_be_bytes([png[at], png[at + 1], png[at + 2], png[at + 3]]) as usize; + if &png[at + 4..at + 8] == want { + return Some(png[at + 8..at + 8 + len].to_vec()); + } + at += 12 + len; + } + None +} diff --git a/crates/gamut-png/tests/size_contract.rs b/crates/gamut-png/tests/size_contract.rs new file mode 100644 index 00000000..0bad0a4b --- /dev/null +++ b/crates/gamut-png/tests/size_contract.rs @@ -0,0 +1,381 @@ +//! The size contract (issue #224): gamut's output measured against libpng at zlib level 9, with a +//! per-case budget that each carries its own written justification. +//! +//! `README.md` and `STATUS.md` have long claimed "output size is benchmarked against libpng at +//! maximum compression". `benches/encode.rs` now prints that comparison, but a bench asserts +//! nothing and is not in the per-PR gate. This file is what makes the claim enforceable: a +//! regression in the crate's reason to exist fails the build, which is the same mechanism +//! `gamut-deflate`'s ratio contract and `gamut-webp/tests/effort.rs` use. +//! +//! Budgets are *measured*, not aspirational, and they are one-sided. The table below records what +//! each case actually achieves alongside what is asserted, so drift shows up in review rather +//! than as a surprise red build. Deliberately no "budgets are still tight" assertion: it would +//! fail on a libpng point release for no correctness reason. + +mod common; + +use gamut_core::{Dimensions, EncodeImage, ImageRef, Rgb8, Rgba8}; +use gamut_png::{FilterStrategy, Level, PngEncoder, deconstruct}; + +/// One case's size budget against libpng at zlib level 9. +struct Budget { + /// Row label; matches `benches/encode.rs`, plus a `+clean` suffix where this row differs from + /// its neighbour only by [`PngEncoder::with_transparent_cleanup`]. + name: &'static str, + /// Corpus generator key. Distinct from `name` so a cleaned row can share a fixture with its + /// uncleaned twin rather than duplicating the pixels. + fixture: &'static str, + /// The square side to measure at. + side: u32, + /// Whether to enable [`PngEncoder::with_transparent_cleanup`]. + cleanup: bool, + /// The most gamut's file may measure as a fraction of libpng's. `1.00` reads "never larger". + /// + /// Derived, not chosen: `measured × (1 + headroom)` rounded up to two decimals, where the + /// headroom is 5% unless this row's `why` names the other component whose drift it absorbs. + max_ratio: f64, + /// What the case actually measures at this revision, so drift is visible in review. + /// + /// To refresh the whole table after an encoder change: set every `max_ratio` to `2.00`, run + /// `cargo test -p gamut-png --test size_contract + /// gamut_never_exceeds_its_size_budget_against_libpng9 -- --exact --nocapture`, paste each + /// printed ratio back into `measured`, then re-derive `max_ratio` by the rule above. + measured: f64, + /// Why this number and not a tighter one — which stage spends the bytes. + why: &'static str, +} + +/// Every budget carries its justification, and every `max_ratio` is derived from the `measured` +/// beside it rather than chosen -- see [`Budget::max_ratio`]. +/// +/// Measured at 128x128 (a quarter of the bench's pixel count, so the suite stays quick enough for +/// the coverage and mutation lanes) except `tiny_rgb8`, which is the bench's own 16x16 row. +/// +/// These ratios are **not** comparable with the bench's 256x256 figures and must be read +/// separately. Every fixed cost -- the signature, IHDR, PLTE/tRNS, IEND, and DEFLATE's own framing +/// -- is amortised over a quarter as many pixels here, which systematically disadvantages exactly +/// the rows where a reduction wins: the gap runs to about 30 percentage points on `gradient_rgb8` +/// and `palette64_rgba8`. `STATUS.md` records the 256x256 table; this one gates. +const BUDGETS: &[Budget] = &[ + Budget { + name: "gradient_rgb8", + fixture: "gradient_rgb8", + side: 128, + cleanup: false, + max_ratio: 0.82, + measured: 0.772, + why: "no reduction applies, so this is filtering plus DEFLATE against libpng's own \ + adaptive filtering. The margin is thin by nature -- both encoders are doing the \ + same job -- so the budget only guards against losing outright.", + }, + Budget { + name: "photo_rgb8", + fixture: "photo_rgb8", + side: 128, + cleanup: false, + max_ratio: 0.83, + measured: 0.731, + why: "smooth photographic content: palette-hostile, so again pure filtering + DEFLATE, \ + and the win is the optimal parse. Coupled to gamut-deflate's own Best/z9 column by \ + construction: if that regresses, this row moves with it, so it carries 13% headroom \ + where the others carry 5%.", + }, + Budget { + name: "noise_rgb8", + fixture: "noise_rgb8", + side: 128, + cleanup: false, + max_ratio: 1.02, + measured: 0.998, + why: "incompressible, so both encoders fall back to stored blocks and the file is \ + slightly larger than the raw samples. Above 1.0 because there is nothing to win \ + here, not because we lose; the 2% margin covers stored-block framing only.", + }, + Budget { + name: "grey_as_rgb8", + fixture: "grey_as_rgb8", + side: 128, + cleanup: false, + max_ratio: 0.62, + measured: 0.582, + why: "R=G=B everywhere, so auto-reduce drops two channels before DEFLATE runs. A \ + structural win libpng does not attempt.", + }, + Budget { + name: "flat_rgba8", + fixture: "flat_rgba8", + side: 128, + cleanup: false, + max_ratio: 0.36, + measured: 0.321, + why: "one opaque colour: the reduce cascade collapses it to depth-1 indexed, and chunk \ + framing is most of what remains. 10% headroom because at ~100 bytes total a single \ + byte moves the ratio by about a percent.", + }, + Budget { + name: "sprite_rgba8", + fixture: "sprite_rgba8", + side: 128, + cleanup: false, + max_ratio: 0.99, + measured: 0.963, + why: "binary alpha over invisible colour noise. The reduce cascade now reaches this \ + case -- `write_reduced_or_native` races an `RGB`+`tRNS` colour key against the \ + unreduced encoding and keeps whichever is smaller -- so the budget is a real one \ + rather than the placeholder 1.00 it carried while those axes were missing. \ + Tightening it was #481's stated acceptance test. 2% headroom, not 5%: at 0.963 the \ + usual 5% rounds past 1.00, which would give up the very claim this row exists to \ + make.", + }, + Budget { + name: "sprite_rgba8 +clean", + fixture: "sprite_rgba8", + side: 128, + cleanup: true, + max_ratio: 0.70, + measured: 0.665, + why: "the same pixels with `with_transparent_cleanup`, which collapses every invisible \ + pixel to one colour and so makes the palette reachable. This is the row that gates \ + the `+clean` column STATUS.md publishes; without it the headline cleanup result was \ + measured by a bench and asserted by nothing.", + }, + Budget { + name: "palette64_rgba8", + fixture: "palette64_rgba8", + side: 128, + cleanup: false, + max_ratio: 0.95, + measured: 0.899, + why: "64 colours over two alpha levels. The palette encoding wins outright at 256x256 \ + but loses at this size, because PLTE + tRNS is a flat 224 incompressible bytes \ + against pixels that compress ~160x; `write_reduced_or_native` encodes both and \ + keeps the smaller, so the row measures whichever is actually better here -- at \ + 128x128 that is the unreduced encoding, which carries no PLTE at all. The race is \ + what makes the outcome stable enough to budget below 1.00.", + }, + Budget { + name: "palette64_rgba8 +clean", + fixture: "palette64_rgba8", + side: 128, + cleanup: true, + max_ratio: 0.95, + measured: 0.899, + why: "the row where cleaning does not pay, and therefore is not done. Collapsing the \ + transparent entries shortens PLTE and tRNS, but it also rewrites pixels that were \ + compressing well, and at 128x128 the second effect wins: cleaning measured 403 \ + bytes against the uncleaned 364. `cleaned_or_plain` races the two and keeps the \ + smaller, so this row now measures exactly what `palette64_rgba8` does, and the \ + budget is the same. That equality is the assertion -- it is what \ + `with_transparent_cleanup` never costing bytes looks like from here, and it is \ + pinned as a law for every row by `cleanup_never_costs_bytes_on_any_corpus_row`.", + }, + Budget { + name: "tiny_rgb8", + fixture: "tiny_rgb8", + side: 16, + cleanup: false, + max_ratio: 0.95, + measured: 0.862, + why: "the regime where the signature and five chunks of framing dominate, and the only \ + row where `overhead_bytes` is legible. Reported by the bench and, until now, gated \ + by nothing. Same 10% headroom as `flat_rgba8`, for the same reason.", + }, +]; + +/// Half the bench's side, so this file stays fast enough for the coverage and mutation lanes. +const SIDE: u32 = 128; + +/// The pixels for a budget row, and how many channels they carry. +fn pixels(fixture: &str, side: u32) -> (Vec, usize) { + match fixture { + "gradient_rgb8" => (common::corpus::gradient_rgb(side), 3), + "photo_rgb8" => (common::corpus::photo_rgb(side), 3), + "noise_rgb8" => (common::corpus::noise_rgb(side), 3), + "grey_as_rgb8" => (common::corpus::grey_as_rgb(side), 3), + "palette64_rgba8" => (common::corpus::palette64_rgba(side), 4), + "sprite_rgba8" => (common::corpus::sprite_rgba(side), 4), + "flat_rgba8" => (common::corpus::flat_rgba(side), 4), + // The bench's 16x16 row: the regime where chunk framing dominates bits-per-pixel. + "tiny_rgb8" => (common::corpus::gradient_rgb(side), 3), + other => panic!("unknown corpus fixture {other}"), + } +} + +/// Encodes at the crate's smallest-output settings. +/// +/// `BruteForce`'s candidate set is integer-only -- `MinEntropy` is deliberately not in it -- so no +/// `f64::log2` enters the gated path and these ratios are machine-independent as well as stable +/// run to run. +fn gamut_best(samples: &[u8], channels: usize, side: u32, cleanup: bool) -> Vec { + let encoder = PngEncoder::new() + .with_compression(Level::Best) + .with_filter(FilterStrategy::BruteForce) + .with_auto_reduce(true) + .with_transparent_cleanup(cleanup); + let dims = Dimensions::new(side, side).expect("valid dimensions"); + let mut out = Vec::new(); + if channels == 3 { + let image = ImageRef::::new(samples, dims).expect("buffer matches dimensions"); + encoder.encode_image(image, &mut out).expect("encode"); + } else { + let image = ImageRef::::new(samples, dims).expect("buffer matches dimensions"); + encoder.encode_image(image, &mut out).expect("encode"); + } + out +} + +/// The same source layout through libpng at zlib level 9 — no palette hint, default adaptive +/// filtering. Handing libpng a palette would hand it gamut's own reduction. +fn libpng9(samples: &[u8], channels: usize, side: u32) -> Vec { + let color_type = if channels == 3 { + libpng_oracle::COLOR_RGB + } else { + libpng_oracle::COLOR_RGBA + }; + libpng_oracle::encode( + samples, + side, + side, + color_type, + 8, + &libpng_oracle::EncodeOpts { + compression_level: Some(9), + ..libpng_oracle::EncodeOpts::default() + }, + ) +} + +#[test] +fn gamut_never_exceeds_its_size_budget_against_libpng9() { + for budget in BUDGETS { + let (samples, channels) = pixels(budget.fixture, budget.side); + let ours = gamut_best(&samples, channels, budget.side, budget.cleanup); + let theirs = libpng9(&samples, channels, budget.side); + let ratio = ours.len() as f64 / theirs.len() as f64; + // Printed, not just asserted: the `measured` column is only honest if refreshing it is a + // paste rather than a re-derivation. `cargo test` captures this on success. + println!( + "{:<22} {:>7} / {:>7} = {ratio:.3} (budget {:.2}, recorded {:.3})", + budget.name, + ours.len(), + theirs.len(), + budget.max_ratio, + budget.measured, + ); + assert!( + ratio <= budget.max_ratio, + "{}: {} bytes vs libpng-9's {} = {ratio:.3}, budget {:.2} (measured {:.2} when set)\n {}", + budget.name, + ours.len(), + theirs.len(), + budget.max_ratio, + budget.measured, + budget.why, + ); + } +} + +#[test] +fn gamut_beats_libpng9_where_it_claims_to() { + // "We win here" and "we do not lose too much there" are different claims, so they are + // different tests. The winning set is listed explicitly rather than derived from + // `max_ratio < 1.0`: a budget loosened past 1.0 during a regression would otherwise drop out + // of this test silently, which is exactly when it should fail. + const WINS: &[&str] = &[ + "gradient_rgb8", + "photo_rgb8", + "grey_as_rgb8", + "flat_rgba8", + "sprite_rgba8", + "palette64_rgba8", + ]; + for budget in BUDGETS.iter().filter(|b| WINS.contains(&b.name)) { + let (samples, channels) = pixels(budget.fixture, budget.side); + let ours = gamut_best(&samples, channels, budget.side, budget.cleanup); + let theirs = libpng9(&samples, channels, budget.side); + assert!( + ours.len() < theirs.len(), + "{}: claims a structural win but measured {} vs {}", + budget.name, + ours.len(), + theirs.len(), + ); + } +} + +#[test] +fn the_codestream_is_no_larger_where_both_encoders_choose_the_same_representation() { + // The reason `deconstruct` is a dependency of this file: it reads the IDAT total out of both + // encoders' output, so the comparison is over codestreams rather than whole files, with + // framing and chunk differences excluded. + // + // This is deliberately *not* an attribution to DEFLATE. Landing on the same colour type and + // depth makes `filtered_len` identical -- it is a function of IHDR alone -- but not the + // filtered *bytes*: gamut runs `BruteForce` (MinBigrams wins `gradient_rgb8`) while libpng + // runs its own adaptive heuristic, so the two compress different inputs. What is asserted is + // the combined result of filtering and DEFLATE, which is what the size claim rests on anyway; + // isolating the DEFLATE stage would mean re-filtering libpng's pixels with gamut's own + // choices first. Only the rows where no reduction applies can be compared at all. + for name in ["gradient_rgb8", "photo_rgb8"] { + let (samples, channels) = pixels(name, SIDE); + let ours = gamut_best(&samples, channels, SIDE, false); + let theirs = libpng9(&samples, channels, SIDE); + let (a, b) = ( + deconstruct(&ours).expect("gamut output deconstructs"), + deconstruct(&theirs).expect("libpng output deconstructs"), + ); + + assert_eq!( + (a.header.color_type, a.header.bit_depth), + (b.header.color_type, b.header.bit_depth), + "{name}: attribution only holds when both land on the same representation", + ); + assert_eq!( + a.filtered_len, b.filtered_len, + "{name}: same representation means an identical filtered stream length", + ); + assert!( + a.idat_compressed <= b.idat_compressed, + "{name}: gamut's codestream is {} bytes against libpng-9's {}", + a.idat_compressed, + b.idat_compressed, + ); + } +} + +#[test] +fn encoded_size_is_deterministic() { + // Without this the budget table is measuring noise rather than the encoder. + for budget in BUDGETS { + let (samples, channels) = pixels(budget.fixture, budget.side); + let first = gamut_best(&samples, channels, budget.side, budget.cleanup); + let second = gamut_best(&samples, channels, budget.side, budget.cleanup); + assert_eq!(first, second, "{}: encode is not reproducible", budget.name); + } +} + +#[test] +fn cleanup_never_costs_bytes_on_any_corpus_row() { + // The gate on `with_transparent_cleanup`'s central claim. It is only true because the encoder + // *races* the cleaned and uncleaned encodings and keeps the smaller: cleaning is a transform, + // not a reduction, and on a fixture whose invisible pixels carry structure rather than noise + // it destroys compressible bytes. Measured before the race, on `palette64_rgba8`, cleaning was + // worth -2.3% at 32x32, +10.7% at 128x128 and -5.2% at 256x256 -- with both candidates landing + // on the same colour type, so the sign was a property of the image, not of the reduction. + // + // A law rather than a budget, so it covers every row and every side, and needs no constant. + for budget in BUDGETS.iter().filter(|b| !b.cleanup) { + let (samples, channels) = pixels(budget.fixture, budget.side); + let plain = gamut_best(&samples, channels, budget.side, false); + let cleaned = gamut_best(&samples, channels, budget.side, true); + assert!( + cleaned.len() <= plain.len(), + "{}: cleanup cost {} bytes ({} -> {}); the race in `cleaned_or_plain` should have \ + kept the uncleaned encoding", + budget.name, + cleaned.len() - plain.len(), + plain.len(), + cleaned.len(), + ); + } +} diff --git a/crates/gamut-png/tests/transparent_cleanup.rs b/crates/gamut-png/tests/transparent_cleanup.rs new file mode 100644 index 00000000..3eda1a3b --- /dev/null +++ b/crates/gamut-png/tests/transparent_cleanup.rs @@ -0,0 +1,309 @@ +//! `PngEncoder::with_transparent_cleanup` (issue #224): rewriting the colour of invisible pixels. +//! +//! The claim has two halves and they need different techniques. That nothing *visible* changes is +//! a differential claim, checked by decoding with libpng and comparing every pixel a viewer could +//! see. That it actually pays is a size claim, checked against the same image encoded without it. +//! +//! Both halves matter: a cleanup that changed a visible pixel would be a correctness bug, and one +//! that saved no bytes would be churn. + +mod common; + +use gamut_core::{Dimensions, EncodeImage, GrayAlpha16, ImageRef, Rgba8, Rgba16}; +use gamut_png::{FilterStrategy, Level, PngEncoder}; + +const SIDE: u32 = 64; + +fn encode(samples: &[u8], cleanup: bool, auto_reduce: bool) -> Vec { + let dims = Dimensions::new(SIDE, SIDE).expect("valid dimensions"); + let image = ImageRef::::new(samples, dims).expect("buffer matches dimensions"); + let mut out = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .with_filter(FilterStrategy::BruteForce) + .with_auto_reduce(auto_reduce) + .with_transparent_cleanup(cleanup) + .encode_image(image, &mut out) + .expect("encode"); + out +} + +#[test] +fn every_visible_pixel_survives_cleanup_unchanged() { + // libpng decodes both files; every pixel with a non-zero alpha must be byte-identical, and + // every alpha must be identical everywhere. Only the colour under alpha == 0 may differ. + let src = common::corpus::sprite_rgba(SIDE); + let plain = libpng_oracle::decode_rgba8(&encode(&src, false, false)).2; + let cleaned = libpng_oracle::decode_rgba8(&encode(&src, true, false)).2; + + assert_eq!(plain.len(), cleaned.len()); + let mut invisible_changed = 0usize; + let (plain_px, _) = plain.as_chunks::<4>(); + let (clean_px, _) = cleaned.as_chunks::<4>(); + for (i, (a, b)) in plain_px.iter().zip(clean_px).enumerate() { + assert_eq!(a[3], b[3], "pixel {i}: alpha must never change"); + if a[3] == 0 { + if a[..3] != b[..3] { + invisible_changed += 1; + } + } else { + assert_eq!(a, b, "pixel {i} is visible and must be byte-identical"); + } + } + assert!( + invisible_changed > 0, + "the fixture must actually exercise the cleanup" + ); +} + +#[test] +fn cleanup_shrinks_an_image_with_invisible_colour_noise() { + let src = common::corpus::sprite_rgba(SIDE); + let plain = encode(&src, false, false); + let cleaned = encode(&src, true, false); + assert!( + cleaned.len() < plain.len(), + "cleanup should pay on a sprite: {} vs {}", + cleaned.len(), + plain.len() + ); +} + +#[test] +fn cleanup_is_inert_on_a_fully_opaque_image() { + // No fully transparent pixel means nothing to rewrite, and the output must be byte-identical + // rather than merely the same size — this is what pins that the pass is a no-op, not a + // re-encode that happens to land on the same length. + let src = common::corpus::flat_rgba(SIDE); + assert_eq!(encode(&src, false, true), encode(&src, true, true)); +} + +#[test] +fn cleanup_collapses_invisible_pixels_into_one_palette_entry() { + // The compounding effect: `analyze8` keys its palette on the whole RGBA quad, so invisible + // pixels that differ only in unseen colour cost an entry each. This fixture has 64 visible + // colours and 64 *distinct* invisible ones, which is over the 256-entry cliff only in the + // sense that it doubles the table; cleaning collapses the invisible half. + let mut src = vec![0u8; (SIDE * SIDE * 4) as usize]; + for (i, px) in src.as_chunks_mut::<4>().0.iter_mut().enumerate() { + let v = (i % 64) as u8; + if i % 2 == 0 { + px.copy_from_slice(&[v, v, v, 255]); + } else { + // Invisible, and every one a different colour. + px.copy_from_slice(&[v.wrapping_mul(3), v.wrapping_add(7), 200 - v, 0]); + } + } + let plain = encode(&src, false, true); + let cleaned = encode(&src, true, true); + assert!( + cleaned.len() < plain.len(), + "collapsing the invisible half should shrink the palette: {} vs {}", + cleaned.len(), + plain.len() + ); +} + +#[test] +fn cleanup_is_off_by_default() { + // The default must stay byte-for-byte lossless, so an encoder that was never asked for + // cleanup must produce exactly what it produced before this feature existed. + let src = common::corpus::sprite_rgba(SIDE); + let dims = Dimensions::new(SIDE, SIDE).expect("valid dimensions"); + let image = ImageRef::::new(&src, dims).expect("buffer matches dimensions"); + let mut default_out = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .with_filter(FilterStrategy::BruteForce) + .encode_image(image, &mut default_out) + .expect("encode"); + assert_eq!(default_out, encode(&src, false, false)); +} + +// --- 16-bit layouts ------------------------------------------------------------------------- +// +// `Rgba16` and `GrayAlpha16` carry an alpha channel and can carry fully transparent pixels, so +// the knob's documented behaviour applies to them too. The oracle here is `libpng_oracle::decode` +// rather than `decode_rgba8`: the simplified reader would scale 16-bit samples down to 8 bits and +// hide exactly the low byte a byte-wise cleanup would get wrong. + +/// A 16-bit sprite: an opaque disc over fully transparent pixels whose colour samples vary in +/// *both* bytes, so a cleanup that only cleared high bytes would leave compressible noise behind. +fn sprite_rgba16(side: u32) -> Vec { + let mut buf = vec![0u16; (side * side * 4) as usize]; + let r2 = (i64::from(side) * i64::from(side)) / 9; + for y in 0..side { + for x in 0..side { + let i = ((y * side + x) * 4) as usize; + let cx = i64::from(x) - i64::from(side) / 2; + let cy = i64::from(y) - i64::from(side) / 2; + if cx * cx + cy * cy < r2 { + buf[i] = u16::from((x ^ y) as u8) * 257; + buf[i + 1] = 0x4040; + buf[i + 2] = 0xC0C0; + buf[i + 3] = u16::MAX; + } else { + // Invisible, and deliberately not constant in either byte of any sample. + buf[i] = (x as u16).wrapping_mul(1103); + buf[i + 1] = (y as u16).wrapping_mul(2749); + buf[i + 2] = ((x ^ y) as u16).wrapping_mul(7919); + buf[i + 3] = 0; + } + } + } + buf +} + +/// The [`sprite_rgba16`] shape in two channels: an opaque grey band over invisible grey noise. +fn sprite_gray_alpha16(side: u32) -> Vec { + let mut buf = vec![0u16; (side * side * 2) as usize]; + for y in 0..side { + for x in 0..side { + let i = ((y * side + x) * 2) as usize; + if x % 8 < 5 { + buf[i] = u16::from((y % 32) as u8) * 2048; + buf[i + 1] = u16::MAX; + } else { + buf[i] = (x as u16).wrapping_mul(6151) ^ (y as u16).wrapping_mul(769); + buf[i + 1] = 0; + } + } + } + buf +} + +fn encode_rgba16(samples: &[u16], cleanup: bool) -> Vec { + let dims = Dimensions::new(SIDE, SIDE).expect("valid dimensions"); + let image = ImageRef::::new(samples, dims).expect("buffer matches dimensions"); + let mut out = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .with_filter(FilterStrategy::BruteForce) + .with_transparent_cleanup(cleanup) + .encode_image(image, &mut out) + .expect("encode"); + out +} + +fn encode_gray_alpha16(samples: &[u16], cleanup: bool) -> Vec { + let dims = Dimensions::new(SIDE, SIDE).expect("valid dimensions"); + let image = ImageRef::::new(samples, dims).expect("buffer matches dimensions"); + let mut out = Vec::new(); + PngEncoder::new() + .with_compression(Level::Best) + .with_filter(FilterStrategy::BruteForce) + .with_transparent_cleanup(cleanup) + .encode_image(image, &mut out) + .expect("encode"); + out +} + +/// The decoded 16-bit samples, as big-endian pairs reassembled into `u16`. +fn decode16(png: &[u8], channels: usize) -> Vec { + let decoded = libpng_oracle::decode(png); + assert_eq!(decoded.bit_depth, 16, "the 16-bit path must stay 16-bit"); + assert_eq!( + decoded.pixels.len(), + (SIDE * SIDE) as usize * channels * 2, + "unexpected layout" + ); + decoded + .pixels + .as_chunks::<2>() + .0 + .iter() + .map(|&p| u16::from_be_bytes(p)) + .collect() +} + +#[test] +fn every_visible_rgba16_pixel_survives_cleanup_unchanged() { + let src = sprite_rgba16(SIDE); + let plain = decode16(&encode_rgba16(&src, false), 4); + let cleaned = decode16(&encode_rgba16(&src, true), 4); + + let mut invisible_changed = 0usize; + let (plain_px, _) = plain.as_chunks::<4>(); + let (clean_px, _) = cleaned.as_chunks::<4>(); + for (i, (a, b)) in plain_px.iter().zip(clean_px).enumerate() { + assert_eq!(a[3], b[3], "pixel {i}: alpha must never change"); + if a[3] == 0 { + assert_eq!( + &b[..3], + &[0, 0, 0], + "pixel {i}: invisible colour must be zeroed" + ); + if a[..3] != b[..3] { + invisible_changed += 1; + } + } else { + assert_eq!(a, b, "pixel {i} is visible and must be sample-identical"); + } + } + assert!( + invisible_changed > 0, + "the fixture must actually exercise the cleanup" + ); +} + +#[test] +fn every_visible_gray_alpha16_pixel_survives_cleanup_unchanged() { + let src = sprite_gray_alpha16(SIDE); + let plain = decode16(&encode_gray_alpha16(&src, false), 2); + let cleaned = decode16(&encode_gray_alpha16(&src, true), 2); + + let mut invisible_changed = 0usize; + let (plain_px, _) = plain.as_chunks::<2>(); + let (clean_px, _) = cleaned.as_chunks::<2>(); + for (i, (a, b)) in plain_px.iter().zip(clean_px).enumerate() { + assert_eq!(a[1], b[1], "pixel {i}: alpha must never change"); + if a[1] == 0 { + assert_eq!(b[0], 0, "pixel {i}: invisible grey must be zeroed"); + if a[0] != b[0] { + invisible_changed += 1; + } + } else { + assert_eq!(a, b, "pixel {i} is visible and must be sample-identical"); + } + } + assert!( + invisible_changed > 0, + "the fixture must actually exercise the cleanup" + ); +} + +#[test] +fn cleanup_shrinks_a_16_bit_image_with_invisible_noise() { + // The knob's whole justification is that invisible noise costs real bytes, and it costs twice + // as many of them per sample at 16 bits. Both fixtures carry it, so on both the cleaned + // encoding must come out strictly smaller — the same claim + // `cleanup_shrinks_an_image_with_invisible_colour_noise` makes for `Rgba8`. + let rgba = sprite_rgba16(SIDE); + assert!( + encode_rgba16(&rgba, true).len() < encode_rgba16(&rgba, false).len(), + "rgba16: {} vs {}", + encode_rgba16(&rgba, true).len(), + encode_rgba16(&rgba, false).len() + ); + let grey = sprite_gray_alpha16(SIDE); + assert!( + encode_gray_alpha16(&grey, true).len() < encode_gray_alpha16(&grey, false).len(), + "gray-alpha16: {} vs {}", + encode_gray_alpha16(&grey, true).len(), + encode_gray_alpha16(&grey, false).len() + ); +} + +#[test] +fn cleanup_is_inert_on_a_fully_opaque_16_bit_image() { + // No fully transparent pixel means the pass must not even copy the buffer: byte-identical + // output, not merely equal length. + let opaque: Vec = (0..(SIDE * SIDE)) + .flat_map(|i| [i as u16, 0x8686, 0xC1C1, u16::MAX]) + .collect(); + assert_eq!( + encode_rgba16(&opaque, false), + encode_rgba16(&opaque, true), + "rgba16" + ); +} diff --git a/docs/README.md b/docs/README.md index 25725652..9a261f87 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,6 +9,7 @@ Anything not listed here is descriptive, not binding. | -------- | ------------- | | [`non-image-media.md`](non-image-media.md) | Whether gamut implements a given audio/video/other-media surface, the crate topology that work lands in, and the [#217]/[#216] roadmaps. Decides scope questions; authorizes no work. | | [`mutation-testing.md`](mutation-testing.md) | How a mutation survey is invoked and what bounds it: the single entry point, the memory budget every parallelism dial is derived from, the guards, and the refusals. What counts as an acceptable survivor is `AGENTS.md`'s rule. | +| [`benchmarking.md`](benchmarking.md) | Where a benchmark lives, what a size or ratio table must record, and where a measured number is kept. What CI does with benches, and what it deliberately does not. Whether a size claim is *enforced* is `testing.md`'s. | | [`testing.md`](testing.md) | Where a test lives and what it may reach, which technique it uses, the per-crate authority table, and the contract by which one law drives both a pinned-seed property test and the fuzz tier. The scope and technique *rules* are `AGENTS.md`'s. | ## Elsewhere in the repo diff --git a/docs/benchmarking.md b/docs/benchmarking.md new file mode 100644 index 00000000..d16542b5 --- /dev/null +++ b/docs/benchmarking.md @@ -0,0 +1,128 @@ +# Benchmarking + +Normative for **where a benchmark lives, what a size or ratio table must record, and where a +measured number is kept**. + +Not normative for: whether a size claim is *enforced* — that is a test, and +[`testing.md`](testing.md) places it (the "size / effort contract" row of its technique table). +Nor for the prose a crate uses to describe its own performance, which is that crate's `README.md`. + +> **A benchmark reports. A test asserts. Only the test can fail a build.** + +## Where a benchmark lives + +One file per crate under `crates//benches/`, named for the thing measured (`codec.rs`, +`compression.rs`, `encode.rs`, `pipeline.rs`), declared with `harness = false` and +`divan.workspace = true` in `[dev-dependencies]`. Sixteen crates ship one. + +```toml +[dev-dependencies] +divan.workspace = true + +[[bench]] +name = "encode" +harness = false +``` + +Do **not** use `required-features`. `mise run bench` is `cargo bench --workspace` with no features, +so a bench behind a required feature silently never runs. Gate the feature-dependent *benchmarks* +inside the file instead, and say so in the module doc. + +## What it must state + +Every bench opens with a module doc that names the subject, the issue, what the counter unit means, +and how to run it. The house phrase is "Intentionally tight:", introducing why *these* axes and not +others. + +Counter units are fixed by kind, so figures are comparable across suites: + +| kind | counter | over | +| --- | --- | --- | +| codec encode/decode | `BytesCount` | **source pixel** bytes | +| compressor | `BytesCount` | input bytes | +| container / parser | `BytesCount` | payload bytes | +| byte-oriented codec pipeline stage | `BytesCount` | bytes the stage consumes | +| per-pixel or per-sample kernel *whose item is not a byte* | `ItemsCount` | items | +| one-off construction cost | none | — | + +The last two rows split on what the kernel's natural unit actually is, because the counter is what +makes a figure comparable and a figure is only comparable to figures in the same unit. A stage +inside a codec pipeline — CRC, scanline packing, filtering, a colour-type scan — consumes the +byte stream the enclosing encoder consumes, so counting its bytes puts it in the same unit as the +crate's own encode benchmark and its size table, and a per-stage figure can be read against the +whole. `gamut-png`'s stage benches are all of this kind. `ItemsCount` is for a kernel whose item is +*not* a byte and would be lost by counting bytes: `gamut-dsp` counts transform coefficients, +`gamut-tonemap` `f32` samples, `gamut-color` `f64` samples and pixels, `gamut-bitstream` coded +symbols, `gamut-cmm` transformed pixels. Bytes per second would say nothing about any of those. + +Fixtures are **generated, never vendored**, and each generator documents the one axis it exists +for. Size them against the algorithm, not for speed: `gamut-png`'s corpus is 256×256 because RGB at +that size is ~6× the DEFLATE window, and a 64×64 image fits *inside* it and would flatter every +encoder equally. + +## Size and ratio tables + +A crate whose reason to exist is output size prints a table before `divan::main()`: + +```rust +fn main() { + print_size_table(); + divan::main(); +} +``` + +The table names its baseline, marks the direction ("lower is better"), and carries a percentage +delta column against that baseline. Where the crate has an oracle, the baseline is the oracle at +its strongest setting — `zlib -9` for `gamut-deflate`, libpng at compression level 9 for +`gamut-png` — configured to do the *same job* on the *same input*. Handing the baseline an +optimisation that is the crate's own contribution does not measure anything. + +Prefer deriving the columns from a reader that works on **any** file rather than from the +encoder's own bookkeeping. `gamut-png`'s table goes through `gamut_png::deconstruct`, which is what +makes its libpng column a measurement rather than two encoders' self-reports. + +## Where a measured number is kept + +In the crate's **`STATUS.md`**, which `docs/README.md` makes normative for implemented state. +Record the invocation, the fixture size, and the caveat that one machine means the ratios are the +result — `gamut-cmm/STATUS.md` and `gamut-png/STATUS.md` are the models. A number in a `README.md` +is a summary of that table, never the source. + +Record negative results too. A heuristic that did not beat the one it was meant to replace is a +finding, and re-deriving it later costs more than writing it down. + +## What CI does + +- **Every PR**: `mise run lint` is `cargo clippy --workspace --all-targets --all-features`, and + `--all-targets` includes benches. They compile, so they cannot rot silently. +- **Extended lane**: `mise run bench-test` is `cargo bench --workspace --benches -- --test`, which + runs every benchmark **once** to prove it still executes — a bench that compiles and then panics + in setup used to be invisible. It takes no timings and asserts no thresholds (issue #437). + +So a benchmark is compiled and executed by CI, and its *numbers* are not gated. Whether they should +be is open, for the reason #437 records: the numbers would come from preemptible shared runners. +Until then, a claim that must not regress belongs in a test — see [`testing.md`](testing.md). + +## Running one + +```bash +mise run bench # the whole workspace +mise run bench-test # run each once, no timings (what Extended does) +cargo bench -p gamut-png # one crate +cargo bench -p gamut-png --bench encode -- --sample-count 50 +``` + +Divan flags must target one harness directly: the per-crate libtest stubs reject them, so +`cargo bench -p --bench -- ` is the form that works. + +## Reaching a crate's internals + +A `benches/` target compiles as a separate crate and sees only `pub` items, which most pipeline +stages are not. The convention is a `test-support` feature exposing a `#[doc(hidden)]` module of +**re-exports only** — no wrapper bodies, which would be executable lines no gate ever runs (bench +targets carry `test = false`) and so would both drag the coverage floor and generate unkillable +mutants. `gamut_png::stages` is the model; the feature is never enabled by the `gamut` umbrella, so +the shipped surface and `mise run check-ffi-features` are unaffected. + +[#437]: https://github.com/visualcommons/gamut/issues/437 +[#149]: https://github.com/visualcommons/gamut/issues/149 diff --git a/docs/testing.md b/docs/testing.md index c38d701a..631126ed 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -77,7 +77,7 @@ the first one that can falsify the claim. | **conformance** | the specification ships vectors | | **pin / drift guard** | an artifact must still equal an authority (`gamut-iptc/tests/techreference.rs`, `gamut-jxl-sys/tests/version.rs`) | | **null-change invariance** | output must be *unchanged*, correctness belonging elsewhere (`gamut-webp/tests/default_bytes.rs`) | -| **size / effort contract** | an encoder knob's ladder is monotonic, deterministic, and correctness-independent (`gamut-webp/tests/effort.rs`) | +| **size / effort contract** | an encoder knob's ladder is monotonic, deterministic, and correctness-independent (`gamut-webp/tests/effort.rs`), or its output stays within a budget against the crate's oracle (`crates/gamut-png/tests/size_contract.rs`) | | **robustness** | input is hostile; the claim is "no panic, bounded allocation, typed error" | A hand-written sweep over five sizes is a property test written badly. A property asserting one @@ -194,7 +194,7 @@ mutation gates, and the stubs carry no function bodies. | gamut-icc | Little-CMS | differential | `IccProfile::parse` ☐ | | gamut-cmm | Little-CMS | differential | — | | gamut-deflate | zlib | differential | — | -| gamut-png | libpng (both directions) | differential + conformance | `PngDecoder` ☐ | +| gamut-png | libpng (both directions) | differential + conformance + size contract | `PngDecoder` ☐ | | gamut-jpeg | libjpeg-turbo | differential + exact-byte | `JpegDecoder` ☐ | | gamut-tiff | libtiff | differential | `TiffDecoder` ☐ | | gamut-dng | Adobe DNG SDK; libtiff (container) | conformance + differential | `DngDecoder` ☐ | @@ -228,3 +228,8 @@ Compile-time assertions (`gamut-codec-abi/src/lib.rs`'s `const _` ABI pins), the gates, doctests (`mise run test-doc`), benchmarks, and the excluded `tooling/gamut-dng-real-conformance` tier. These are real checks; they are simply not tests this document places or classifies. + +Benchmarks have their own document, [`benchmarking.md`](benchmarking.md): where one lives, what its +tables must record, and where the numbers are kept. The boundary is that a benchmark reports and +only a test can fail a build, so a size claim that must not regress is a **size / effort contract** +here — `gamut-png/tests/size_contract.rs` and `gamut-webp/tests/effort.rs` — not a bench. diff --git a/mise.toml b/mise.toml index 50ccb405..a862b21a 100644 --- a/mise.toml +++ b/mise.toml @@ -238,7 +238,7 @@ run = "cargo bench --workspace" # an `unwrap` on an encode that now fails -- was invisible. # # `--benches` restricts this to bench targets. Without it `cargo bench` also builds every lib and -# test target in release just to look for `#[bench]` functions that do not exist here (all 15 +# test target in release just to look for `#[bench]` functions that do not exist here (all 16 # benches are `harness = false` Divan), which is a second full release build for nothing. [tasks.bench-test] description = "Run every bench once to prove it still executes (no timings; issue #437)"