Which of the layouts did you use for the Cranelift benchmarking?

[bjorn3]

Hi @strager, I just saw your blog post at https://quick-lint-js.com/blog/cpp-vs-rust-build-times/. On the benchmark with my project rustc_codegen_cranelift you reported a slowdown. I would like to reproduce this result locally to see if I can find why this happens and try to fix it. I couldn't find which of the layouts under "Workspace and test layouts" you used during this benchmark. Do you remember which one you used? In any case thanks for benchmarking. While I'm disappointed that it caused compilation to slow down, it is an invaluable data point for future improvements.

[strager]

I couldn't find which of the layouts under "Workspace and test layouts" you used during this benchmark.

The Cranelift chart uses the rust directory:

cpp-vs-rust/tools/bench-build-charts.py

Line 260 in f8d3134

project="rust",

The rust directory has the "workspace; many test exes" layout.

[bjorn3]

Thanks!

[bjorn3]

I managed to reproduce the issue at least for cargo build (cargo test failed with cg_clif (rustc_codegen_cranelift) on the AArch64 due to a couple of missing simd intrinsics). Turns out the rustup wrapper is at fault. For the non-cranelift benchmarks you avoided it by calling rustup which rustc once beforehand, but for the cranelift benchmarks cargo-clif internally used the rustup wrappers. This accounts for almost 600ms difference total, which is way too high IMHO.

I benchmarked the following configurations:

• RUSTC='$(rustup which rustc)' $(rustup which cargo) build: The configuration for the non-cranelift full build, except without testing.
• RUSTC='$(rustup which rustc)' RUSTFLAGS='-Zcodegen-backend=$(pwd)/../../dist/lib/librustc_codegen_cranelift.so' $(rustup which cargo) build: The configuration equivalent to the above command, except with Cranelift instead of the LLVM backend. (cargo-clif/rustc-clif change the sysroot to a cg_clif compiled one and set -Cpanic=abort too, but this one is closest to what the non-cranelift build does)
• RUSTC='$(pwd)/../../dist/rustc-clif' ../../dist/cargo-clif build: The configuration you used for the cranelift build build, except without testing.

$ hyperfine --warmup 5 --prepare "cargo clean" --runs 20 "RUSTC='$(rustup which rustc)' $(rustup which cargo) build" "RUSTC='$(rustup which rustc)' RUSTFLAGS='-Zcodegen-backend=$(pwd)/../../dist/lib/librustc_codegen_cranelift.so' $(rustup which cargo) build" "RUSTC='$(pwd)/../../dist/rustc-clif' ../../dist/cargo-clif build"
Benchmark 1: RUSTC='/home/gh-bjorn3/.rustup/toolchains/nightly-2022-12-13-aarch64-unknown-linux-gnu/bin/rustc' /home/gh-bjorn3/.rustup/toolchains/nightly-2022-12-13-aarch64-unknown-linux-gnu/bin/cargo build
  Time (mean ± σ):      3.491 s ±  0.059 s    [User: 6.863 s, System: 1.334 s]
  Range (min … max):    3.429 s …  3.635 s    20 runs
 
Benchmark 2: RUSTC='/home/gh-bjorn3/.rustup/toolchains/nightly-2022-12-13-aarch64-unknown-linux-gnu/bin/rustc' RUSTFLAGS='-Zcodegen-backend=/home/gh-bjorn3/cg_clif/cpp-vs-rust/rust/../../dist/lib/librustc_codegen_cranelift.so' /home/gh-bjorn3/.rustup/toolchains/nightly-2022-12-13-aarch64-unknown-linux-gnu/bin/cargo build
  Time (mean ± σ):      3.281 s ±  0.033 s    [User: 4.691 s, System: 1.200 s]
  Range (min … max):    3.244 s …  3.374 s    20 runs
 
Benchmark 3: RUSTC='/home/gh-bjorn3/cg_clif/cpp-vs-rust/rust/../../dist/rustc-clif' ../../dist/cargo-clif build
  Time (mean ± σ):      3.892 s ±  0.048 s    [User: 5.273 s, System: 1.411 s]
  Range (min … max):    3.822 s …  4.037 s    20 runs
 
Summary
  'RUSTC='/home/gh-bjorn3/.rustup/toolchains/nightly-2022-12-13-aarch64-unknown-linux-gnu/bin/rustc' RUSTFLAGS='-Zcodegen-backend=/home/gh-bjorn3/cg_clif/cpp-vs-rust/rust/../../dist/lib/librustc_codegen_cranelift.so' /home/gh-bjorn3/.rustup/toolchains/nightly-2022-12-13-aarch64-unknown-linux-gnu/bin/cargo build' ran
    1.06 ± 0.02 times faster than 'RUSTC='/home/gh-bjorn3/.rustup/toolchains/nightly-2022-12-13-aarch64-unknown-linux-gnu/bin/rustc' /home/gh-bjorn3/.rustup/toolchains/nightly-2022-12-13-aarch64-unknown-linux-gnu/bin/cargo build'
    1.19 ± 0.02 times faster than 'RUSTC='/home/gh-bjorn3/cg_clif/cpp-vs-rust/rust/../../dist/rustc-clif' ../../dist/cargo-clif build'

(note: all benchmarks use nightly-2022-12-13, cg_clif is currently stuck on this version due to a change in rustc that broke things for cg_clif, there is an open PR to fix this)

[strager]

@bjorn3 Thanks for looking into it. I mentioned your findings in the blog post. I can't re-run the benchmark and redo the analyses right now. Therefore I just posted a notice on the blog, not updated data.

[bjorn3]

A notice is fine by me. Thanks! I will leave this issue open in case you want to do re-run the benchmark at a later time. If not, that is fine too. Feel free to close this issue in that case.