Skip to content

feat: emit per-check progress to stderr during runs - #108

Open
danielcalvolopez wants to merge 1 commit into
agent-ecosystem:mainfrom
danielcalvolopez:emit-progress-to-stderr
Open

danielcalvolopez wants to merge 1 commit into
agent-ecosystem:mainfrom
danielcalvolopez:emit-progress-to-stderr

Conversation

@danielcalvolopez

Copy link
Copy Markdown

Closes #105

What changed

Between the Running checks on <site>... banner and the final report, afdocs emitted nothing, so on a slow or hostile site there was no way to distinguish "working through many slow requests" from "hung" without inspecting the process from outside. This adds lightweight check-level progress on stderr:

Running checks on docs.alkimi.org...
[1/23] llms-txt-exists... done (259ms)
...
[7/23] llms-txt-directive-md... done (33 tested, 2 fetch errors, 8s)
...
[23/23] auth-alternative-access... skipped
  • Runner: RunnerOptions gains an optional onProgress callback that fires a start and complete event per selected check, carrying index/total, the CheckResult, and wall-clock duration. The library API stays output-agnostic; event types are exported from the public API.
  • CLI: renders events as numbered stderr lines. The start fragment is written without a newline, so a stalled check leaves its name visible on screen — you can spot a hang within one check's duration. Completion lines include the tested-page count and the fetch-error count from the check's existing details, doubling as an early live signal of the bot-protection interference pattern from Run-level diagnostic for bot-protection interference (spec v0.6.0 bot-protection-interference) #104.
  • -q, --quiet: suppresses progress and the banner for clean CI logs.
  • Progress goes to stderr for every format, so parseable stdout (json, piped scorecard) is never contaminated.

Per the issue's notes, this implements check-level granularity only (the best effort/value point); per-page ticks within multi-page checks are left as a possible follow-up.

Testing

  • 15 new tests: runner event emission (ordering, numbering, skip/dependency-skip/error paths), progress-line formatting, and CLI wiring (--quiet, json stdout stays parseable). Full suite: 1360 passing.
  • Verified live against real sites: default run shows progress, --quiet produces zero stderr bytes, -f json > report.json stays parseable while progress streams to the terminal.
  • Docs updated in docs/reference/cli.md.

Between the "Running checks on..." banner and the final report, afdocs
was silent, making a slow run indistinguishable from a hung one. The
runner now exposes an onProgress callback on RunnerOptions that fires
start/complete events per selected check, and the CLI renders them as
numbered stderr lines:

  [7/24] llms-txt-directive-html... done (47 tested, 3 fetch errors, 12s)

The start fragment is written without a trailing newline so a stalled
check leaves its name visible on screen. Completion lines include the
tested-page and fetch-error counts from check details, doubling as a
live signal of bot-protection interference (agent-ecosystem#104). Progress goes to
stderr for every output format so json/scorecard stdout stays
parseable; the new -q/--quiet flag suppresses it (and the banner) for
clean CI logs.

Closes agent-ecosystem#105
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Emit progress to stderr during long runs

1 participant