This repository collects community-submitted telemetry about how different Claude models behave in real Claude Code sessions: orchestration follow-through, escalation frequency, prose structure, and coding output. Each submission is a set of aggregate counts extracted from a contributor's own session transcripts, on their own machine, using an open extraction skill.
The collected data feeds the aggregate published alongside 3dl.dev/coding-vs-orchestration.html. Inclusion in the aggregate is a separate, deliberate curation step, not an automatic result of opening a pull request.
The extraction logic lives in skill/SKILL.md. Install it into
~/.claude/skills/model-telemetry/ so it loads as a Claude Code skill, or
paste its contents directly into a Claude Code session as a prompt.
The skill reads your local ~/.claude/projects/ transcripts, computes
counts and ratios, and writes a metrics.json file. It runs entirely on
your machine and takes about 20 seconds. It does not upload anything and it
does not require network access. The output contains counts only: no
transcript text, no file paths, and no project names.
Submission is optional. The article's charts at
3dl.dev/coding-vs-orchestration.html
re-render from a local metrics.json: drop the file on the panel in the
final section and every chart follows your data. The file is read in the
browser and is not uploaded. Nothing in that flow touches this repository.
- Fork this repository.
- Run the extraction skill to produce
metrics.json. - Wrap it in the submission shape described in
skill/SKILL.mdand save it assubmissions/<github-handle>-<YYYY-MM-DD>.json, where<github-handle>matches your GitHub account. - Open a pull request adding that one file.
CI validates the submission against schema/metrics.schema.json on every
pull request that touches submissions/**. A pull request that fails
validation cannot merge.
Merging a pull request adds a file to the submissions/ directory. It does
not add the submission to the published aggregate.
Inclusion is allowlist-based. A maintainer reviews a merged submission and,
if it passes the checks in CONTRIBUTING.md, adds its filename to
accepted.json. The aggregate script (scripts/aggregate.py) only reads
files listed in accepted.json. Submissions not listed there are present
in the repository but excluded from the published numbers by default.
Both submissions/ and accepted.json are public, so every inclusion
decision is visible and auditable after the fact.
scripts/aggregate.py reads the accepted submissions and writes
aggregate.json. For each metric and each model, it reports the median,
the first and third quartiles, and the sample size (n), not a pooled mean.
Small-n groups are reported as small-n; they are not hidden or merged into
a larger group to look more confident.
This data is self-reported and unverifiable. Submitters run the extraction script themselves and submit whatever counts it produces. Nobody checks a submission against the underlying transcripts, because the transcripts never leave the submitter's machine.
Every published number reflects only the set of submissions a maintainer
chose to include in accepted.json, not every submission that was ever
opened as a pull request. This curation step exists to filter out
implausible or malformed data, but it does not make the underlying counts
independently verified.
The aggregate reports distributions, not means. A median and quartiles describe a spread; a single mean can hide that spread and can be pulled by outliers.
Per-hour rates depend heavily on workload: how many hours a submitter worked, what kind of tasks they ran, and how many machines they merged data from. Per-turn structural ratios, such as the share of dispatching turns that end with workers still running, are less sensitive to workload and are the more comparable figures across submitters.
The submitted and aggregated data in this repository is released under
CC0-1.0. See LICENSE.
The scripts and skill in this repository are released under MIT. See
LICENSE-SCRIPTS. Copyright 2026 Third Division Labs.