Replication package for the paper "How Rust Projects Use Macros: A Large-Scale Empirical Study," accepted as a Full Paper at SBCARS 2026 (20th Brazilian Symposium on Software Components, Architectures, and Reuse), co-located with CBSoft 2026, September 8–11, 2026, São Paulo, SP, Brazil.
Authors: Leopoldo Teixeira (lmt@cin.ufpe.br), Nilo Drumond (nbmcd@cin.ufpe.br) — Centro de Informática, Universidade Federal de Pernambuco, Brazil.
Paper PDF:
paper.pdf— camera-ready version submitted for proceedings preparation, in this repository.
Proceedings DOI: (to be added once the SBCARS 2026 / CBSoft 2026 proceedings are published)
Macros are a central feature of the Rust programming language, enabling metaprogramming and compile-time code generation. However, there is limited empirical understanding into how they are used in practice. Using a descriptive survey design, we analyze macro usage across the 99 most popular open-source Rust repositories on GitHub, comprising 4,001 crates and 2,215,517 total invocations, with a custom Tree-sitter-based static analysis pipeline. We address three research questions: (RQ1) how are invocations distributed across macro categories (declarative, derive, attribute, function-like); (RQ2) how concentrated is macro usage around a small set of widely-used ecosystem crates; and (RQ3) how does invocation density vary across project domains. We find that declarative macros dominate macro definitions while function-like and declarative call sites jointly dominate macro invocations. Derive and attribute macros are concentrated among a small set of widely-used ecosystem crates: three crates alone account for 31.9% of user-defined attribute invocations. Finally, educational and AI/LLM-oriented projects exhibit higher density than general-purpose application code.
.
├── src/ Rust scraper/analyzer pipeline (tree-sitter-based)
├── graphql/ GitHub GraphQL schema + query used by src/github.rs
├── scripts/ Python scripts for statistics, tables, and visualization
│ ├── generate_numbers.py populate <out-dir>/numbers.tex and <out-dir>/tables/ (--out-dir, default "paper")
│ ├── generate_report.py generate interactive HTML report
│ ├── stats_tests.py Kruskal-Wallis + Mann-Whitney for RQ3
│ ├── score_validation.py inter-rater κ and per-category precision
│ ├── draw_validation_sample.py draw stratified validation sample
│ └── generate_domain_labels.py browser-based domain labeling helper
├── tools/
│ ├── domain_labeler.html browser-based tool used to produce data/domain_labels_final.csv
│ └── validation_labeler.html browser-based tool used to produce the validation_sample_*_labeled.csv files
├── corpus/
│ ├── sbcars-2026/ Frozen corpus for this submission
│ └── latest/ Symlink-equivalent: mirrors sbcars-2026/ now, overwritten by a fresh run
├── data/ Frozen analysis outputs used in the paper
│ ├── results.csv per-repo macro counts and LOC
│ ├── data.json full structured output (HTML report)
│ ├── crates.ron discovered crate paths (input to generate_numbers.py)
│ ├── domain_labels_final.csv domain label for each of the 99 repos
│ ├── validation_sample_answer_key.csv answer key for the sample
│ ├── validation_sample_pass_author1_labeled.csv author 1 labels
│ ├── validation_sample_pass_author2_labeled.csv external researcher labels
│ ├── validation_stats.json κ and per-category precision
│ ├── stats_tests.json Kruskal-Wallis + Mann-Whitney results
│ └── generated/ written by generate_numbers.py --out-dir data/generated (gitignored)
├── docker/entrypoint.sh dispatches `docker run ferrometer <action>` (see "Running via Docker")
├── Dockerfile
├── Cargo.toml
├── Cargo.lock
├── requirements.txt
├── CITATION.cff citation metadata for this software and the paper (see "Citing This Artifact")
├── LICENSE MIT
└── README.md (this file)
Alternative: Docker. If you'd rather not install Rust/Python/cloc
locally, skip straight to Running via Docker below —
it packages everything in this section into one image.
- Rust stable ≥ 1.96 — install via rustup.rs
cloc≥ 1.96 onPATH— only needed for the full pipeline re-run- GitHub personal access token — only needed for Step 1 of the full pipeline
(querying the GitHub GraphQL API for the top-100 repo list); not needed when
using the pinned corpus in
corpus/latest/ - ~20 GB disk space if cloning all repositories
- Python ≥ 3.10
- Install dependencies once:
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txtrequirements.txt pins: plotly, pandas, numpy, statsmodels, scipy.
All Python scripts run on a standard laptop with no GPU. The full Rust pipeline
(clone + analysis of all repos) takes roughly 1–2 hours depending on network
speed. Using the frozen data/ outputs — the recommended path for artifact
evaluation — completes in under 5 minutes (possibly less).
After setting up the Python environment above, verify it:
source .venv/bin/activate
python scripts/stats_tests.py
# Expected last line: results written to data/stats_tests.jsonVerify the Rust build:
cargo build --release --locked 2>&1 | tail -1
# Expected: Finished `release` profile [optimized] target(s) in ...Cargo.lock is committed and pins exact dependency versions — always build
with --locked so the tool doesn't silently resolve newer dependencies over
time.
The Dockerfile bundles the Rust toolchain, cloc, and the Python
environment, so no local install is needed. Build it once:
docker build -t ferrometer .Then run one action at a time. Mount data/ so generated outputs land back
on the host instead of staying inside the throwaway container:
docker run --rm -v "$(pwd)/data:/app/data" ferrometer <action>| Action | What it does | Needs network/token? |
|---|---|---|
rq1 |
Reproduce RQ1 numbers/tables → data/generated/ |
no |
rq2 |
Reproduce RQ2 numbers/tables → data/generated/ (same underlying script as rq1) |
no |
rq3 |
Reproduce RQ3 density statistics → data/stats_tests.json |
no |
validation |
Reproduce classifier validation (κ, precision) → data/validation_stats.json |
no |
report |
Generate the interactive HTML report → data/report.html |
no |
all |
Run rq1, rq3, validation, and report in sequence |
no |
build |
cargo build --release --locked, to verify the tool compiles |
no |
full-pipeline |
Re-run the full scraper against the pinned corpus | yes — requires -e GITHUB_TOKEN=... and ~20 GB free under data/repos/ |
shell |
Drop into an interactive bash shell in the image |
no |
help (default) |
Print the action list | no |
Example — reproduce everything that doesn't need cloning, in one shot:
docker run --rm -v "$(pwd)/data:/app/data" ferrometer allExample — full pipeline re-run (slow; needs a GitHub token):
docker run --rm -v "$(pwd)/data:/app/data" \
-e GITHUB_TOKEN=<your-personal-access-token> \
ferrometer full-pipelinerq1/rq2 write to data/generated/ rather than paper/numbers.tex —
this repository doesn't include paper/ (see Repository
Organization); scripts/generate_numbers.py
takes a --out-dir flag for exactly this reason, defaulting to paper for
the manuscript repo.
All three RQs can be reproduced from the frozen data/ directory without
re-running the scraper — either directly (commands below) or via
Docker. The full pipeline re-run is optional and
described at the end of this section.
RQ1 — How are macro invocations distributed across categories, and which macros are most frequently invoked?
Primary data: data/results.csv (columns declarative_count,
derive_count, user_attr_count, builtin_count) and data/data.json
(per-macro top-N counts).
source .venv/bin/activate
python scripts/generate_numbers.py --out-dir data/generated--out-dir defaults to paper (the manuscript repo's layout); this repo has
no paper/ directory, so point it at data/generated instead — same
content, different destination.
Outputs written (paths below assume --out-dir data/generated; the
manuscript repo writes the same files under paper/ instead):
| File | Contents |
|---|---|
data/generated/numbers.tex |
All \newcommand values cited in the paper |
data/generated/tables/top-builtins.tex |
Table of top built-in attribute invocations |
data/generated/tables/top-user-attrs.tex |
Table of top user-defined attribute invocations |
data/generated/tables/top-fl-decl.tex |
Table of top function-like/declarative invocations |
data/generated/tables/top-derive.tex |
Table of top derive macro invocations |
data/generated/tables/top-nonstd-fnlike.tex |
Top non-std function-like/declarative |
data/generated/tables/top-nonstd-derive.tex |
Top non-std derive macros |
Key numbers to verify against the paper:
\newcommand |
Expected value |
|---|---|
\totalRepos |
99 |
\totalCrates |
4,001 |
\totalInvocations |
2,215,517 |
Known limitation:
declarative_count==function_like_countin every row ofresults.csv. Both columns countmacro_invocationAST nodes because tree-sitter cannot syntactically distinguish declarative (macro_rules!-defined) from function-like proc macro call sites at the invocation level without macro expansion. This is documented in Section 3.4 of the paper and quantified by the inter-rater validation (Section 3.5).
Primary data: data/data.json (per-macro invocation counts aggregated
across the corpus, with std_origin flags distinguishing standard-library from
ecosystem macros).
source .venv/bin/activate
python scripts/generate_numbers.py --out-dir data/generated # idempotent — safe to re-runRQ2 findings are captured in the \topN* and \nonstd* commands written to
data/generated/numbers.tex, and in data/generated/tables/top-nonstd-*.tex.
Primary data: data/results.csv joined with data/domain_labels_final.csv
(99 repos labeled into 6 domains: systems-cli 29, application 20, ai-llm 17,
library 17, devtools 11, educational 5).
source .venv/bin/activate
python scripts/stats_tests.pyOutput written: data/stats_tests.json
Key numbers to verify against the paper:
| Metric | Expected value |
|---|---|
| Kruskal-Wallis H | 25.39 |
| Kruskal-Wallis df | 5 |
| Kruskal-Wallis p | 0.00012 |
| Significant pairs (Bonferroni-corrected) | 3 of 15 |
| Sensitivity H (excl. educational repos) | 20.26 |
| Sensitivity p | 0.00044 |
| Sensitivity significant pairs | 2 of 10 |
Reproduces the inter-rater reliability results reported in Section 3.5.
Primary data: data/validation_sample_pass_author1_labeled.csv,
data/validation_sample_pass_author2_labeled.csv,
data/validation_sample_answer_key.csv.
source .venv/bin/activate
python scripts/score_validation.py --sep ";"Output written: data/validation_stats.json
Key numbers to verify against the paper:
| Metric | Expected value |
|---|---|
| Cohen's κ | 0.917 ("Almost perfect") |
| Precision — derive | 100% |
| Precision — user-defined attribute | 75% |
| Precision — built-in attribute | 67.6% |
| Precision — declarative + function-like | 34.1% (conflation artifact — see Section 3.4) |
Items excluded as unsure |
9 |
The 200-item stratified sample was drawn with SEED=20260619 via
scripts/draw_validation_sample.py. The sample and both authors' labels are
committed to data/ so the scoring step can be reproduced without re-drawing.
Generates a Plotly-based interactive report with all per-repo and per-macro charts from the paper.
source .venv/bin/activate
python scripts/generate_report.py
open data/report.html # macOS, use xdg-open on LinuxRe-runs the scraper from scratch against the pinned corpus. The pipeline is
resumable: delete data/state.ron to restart, or set individual step
timestamps to None to re-run only that step.
export GITHUB_TOKEN=<your-personal-access-token>
cargo run --releaseCorpus pinning: corpus/sbcars-2026/ is the authoritative frozen snapshot
for this submission — the exact HEAD commit per repo used in the paper.
The tool reads from corpus/latest/ at runtime; corpus/latest/
currently mirrors corpus/sbcars-2026/ exactly. A fresh cargo run would
overwrite corpus/latest/ with a new GitHub query, so copy corpus/sbcars-2026/
over corpus/latest/ first if you want to reproduce the paper's exact corpus:
cp corpus/sbcars-2026/repos.json corpus/latest/repos.json
cp corpus/sbcars-2026/snapshot.json corpus/latest/snapshot.jsonPipeline steps (each is skipped if already recorded in data/state.ron):
github::get_most_popular_repos— loads fromcorpus/latest/repos.jsongithub::clone_repos— clones intodata/repos/<owner>.<name>/cloc::cloc_repos— runscloc --include-lang=Rustper repocrate_paths::find_crate_paths— discovers all crate roots (Cargo.toml)analyzis::analyze_crates— tree-sitter AST walk, counts all macro definitions and invocationscount_code::count_crates_code— counts source lines per crate- Writes
data/results.csvanddata/data.json
After the pipeline completes, run the Python scripts above to regenerate all derived outputs.
Citation metadata is in CITATION.cff (GitHub renders a
"Cite this repository" button from it). It includes both the software
itself and the accompanying paper as a preferred-citation — prefer citing
the paper unless you're specifically referencing this codebase/version.
This repository is connected to Zenodo: each tagged GitHub Release mints a new archival snapshot with its own DOI.
- Concept DOI (always resolves to the latest version): 10.5281/zenodo.21866344
- v1.0.0 DOI (this specific archived release):
The accompanying paper does not yet have its own DOI — the SBCARS 2026 /
CBSoft 2026 proceedings are still being prepared. It will be added here and
to CITATION.cff's preferred-citation once available.
Released under the MIT License — see LICENSE for the full text.