Run-level diagnostic for bot-protection interference (spec v0.6.0 `bot-protection-interference`)

[dacharyc]

Context

Spec v0.6.0 adds the bot-protection-interference check (Category 7) and the "Bot Protection Degrading Scan Reliability" interaction effect, grounded in a production case: a CDN's behavioral bot management responded to a sustained scan by accepting connections, returning headers, and holding response bodies open indefinitely. Single-request probes looked healthy throughout, and enforcement decayed after a cooldown.

The HTTP client now guards body reads with the request timeout, so a tarpitted run completes instead of hanging, and stalled bodies surface as per-page fetch errors with the message Body read timed out after <n>ms (response stalled; server may be rate-limiting or tarpitting automated clients).

Problem

Those errors are scattered across per-check ; N failed to fetch suffixes, and errored pages are excluded from scoring denominators. A heavily tarpitted run therefore produces a confident-looking score computed on whatever sample survived, with no run-level indication that the sample collapsed.

Proposed work

1. Run-level interaction diagnostic: aggregate fetch failures across all checks; when the failure rate crosses a threshold (spec suggests ~20% of page fetches), emit an interaction diagnostic along the lines of: "N% of page fetches failed or timed out. The site may be rate-limiting or tarpitting automated clients; scores reflect only the M pages that responded." Body-timeout errors are already distinguishable by message, so the diagnostic can separate stalls (tarpit signature) from ordinary fetch failures.
2. Implement bot-protection-interference as a scored check per spec v0.6.0: heuristic, evaluated as a byproduct of the normal scan (stalled bodies, challenge-page heuristics in fetched content, failure rate climbing as the scan progresses). Pass/warn/fail per the spec's result levels.
3. Wire the check into the Checks Summary/registry, scoring weights, and scorecard output.

Notes

• Detection is stateful on the target side: a clean run does not prove absence, and results legitimately vary across runs. The check description in the spec covers this caveat.
• Challenge-page detection (e.g. "verifying your browser" interstitials served as 200) can likely reuse soft-404-style heuristics.