Skip to content
12 changes: 6 additions & 6 deletions .github/actions/authenticate-github-fetches/action.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
name: Authenticate git fetches of GitHub repos
description: >-
Rewrite github.com URLs so git-CLI fetches carry the job's ephemeral token.
GitHub throttles anonymous git HTTPS traffic per source IP, and Warp runners
share egress IPs across tenants, so unauthenticated clones of public repos
can fail with a 401 ("could not read Username"). The rewrite covers every
fetch that goes through the git CLI: Lake dependency clones, Nix's eval-time
`builtins.fetchGit`, script clones, and cargo git dependencies when
CARGO_NET_GIT_FETCH_WITH_CLI is set.
GitHub throttles anonymous git HTTPS traffic per source IP, and managed
runner pools can share egress IPs across tenants, so unauthenticated clones
of public repos can fail with a 401 ("could not read Username"). The rewrite
covers every fetch that goes through the git CLI: Lake dependency clones,
Nix's eval-time `builtins.fetchGit`, script clones, and cargo git dependencies
when CARGO_NET_GIT_FETCH_WITH_CLI is set.

runs:
using: composite
Expand Down
2 changes: 1 addition & 1 deletion .github/actions/bencher-track/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ inputs:
description: Bencher testbed slug.
required: true
workload:
description: Workload key for the `refs/bencher/<workload>` anchor (the backend testbed minus its runner-arch suffix, e.g. zisk-check-execute).
description: Workload key for the `refs/bencher/<workload>` anchor (the backend testbed minus its runner-arch suffix, e.g. ooc-check).
required: true
file:
description: Bencher Metric Format JSON file to upload.
Expand Down
4 changes: 2 additions & 2 deletions .github/actions/install-sp1/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@ runs:
- name: Install system build deps
shell: bash
run: |
# Some warpbuild images ship an unreachable azure mirror that hangs
# Some runner images ship an unreachable Azure mirror that hangs
# `apt-get update`; drop it first (no-op elsewhere).
sudo sed -i '/azure\.archive\.ubuntu\.com/d' /etc/apt/apt-mirrors.txt 2>/dev/null || true
export DEBIAN_FRONTEND=noninteractive
# The Ubuntu EC2 mirrors sporadically 502 / throttle on these large
# Ubuntu package mirrors sporadically 502 / throttle on these large
# fetches (qemu-system alone pulls hundreds of MB), so retry the whole
# update+install: `Acquire::Retries` absorbs single-download blips and
# `--fix-missing` resumes a partially fetched set.
Expand Down
12 changes: 6 additions & 6 deletions .github/actions/install-zisk/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,11 @@ runs:
- name: Install system build deps
shell: bash
run: |
# Some warpbuild images ship an unreachable azure mirror that hangs
# Some runner images ship an unreachable Azure mirror that hangs
# `apt-get update`; drop it first (no-op elsewhere).
sudo sed -i '/azure\.archive\.ubuntu\.com/d' /etc/apt/apt-mirrors.txt 2>/dev/null || true
export DEBIAN_FRONTEND=noninteractive
# The Ubuntu EC2 mirrors sporadically 502 / throttle on these large
# Ubuntu package mirrors sporadically 502 / throttle on these large
# fetches (qemu-system alone pulls hundreds of MB), so retry the whole
# update+install: `Acquire::Retries` absorbs single-download blips and
# `--fix-missing` resumes a partially fetched set.
Expand Down Expand Up @@ -90,7 +90,7 @@ runs:
"$HOME/.zisk/bin/cargo-zisk" toolchain install --toolchain-version zisk-0.5.1
echo "$HOME/.zisk/bin" >> "$GITHUB_PATH"
# Pre-build the proofman C++ sys crate ALONE so its build script runs
# exactly once before any parallel zisk-host build. zisk-host pulls
# exactly once before zisk-host's parallel Cargo build. zisk-host pulls
# zisk-sdk as both a dependency and a build-dependency, so cargo compiles
# proofman-starks-lib-c as two units whose build scripts can run
# CONCURRENTLY — and both run `make` inside the SHARED
Expand All @@ -117,9 +117,9 @@ runs:
# reuse a stale key. `aws s3 cp` (multipart, no creds — public bucket)
# holds throughput steady where a single-stream `curl` swings by an
# order of magnitude; `curl` is the fallback for hosts without the AWS
# CLI. Deliberately NOT actions/cache'd: the runner fleet is AWS-hosted,
# so multipart S3 outruns the Actions cache service — and the tarball
# would crowd the repo's cache quota besides.
# CLI. Deliberately omitted from the shared runner cache: the multi-GB
# tarball would crowd build artifacts, while its public S3 source supports
# multipart downloads directly.
- name: Fetch proving key from S3, extract, regenerate const-trees
if: inputs.proving-key == 'true'
shell: bash
Expand Down
154 changes: 33 additions & 121 deletions .github/actions/log-cpu/action.yml
Original file line number Diff line number Diff line change
@@ -1,140 +1,52 @@
name: Log CPU information
description: >-
Log a benchmark host's CPU and effective cgroup allocation, optionally
recording the report, printing native-binary build provenance, and warning
when the build and measurement CPUs differ.
description: Log the current host's CPU and effective cgroup allocation

inputs:
label:
description: Heading and context stored in a newly recorded CPU report
description: Heading and context for the CPU report
required: false
default: CPU information
output-file:
description: Optional path at which to record the current host report
required: false
provenance-file:
description: Optional native-binary build-host report to print
required: false
provenance-label:
description: Subject named in provenance mismatch warnings
required: false
default: Native benchmark binaries
warning-file:
description: Optional Markdown file to which provenance warnings are appended
required: false
log-current:
description: Whether to log the current host
required: false
default: "true"

outputs:
summary:
description: CPU model, effective vCPU count, and total RAM
value: ${{ steps.report.outputs.summary }}

runs:
using: composite
steps:
# PR benchmarks execute trusted workflow YAML from the default branch but
# load this action from the PR checkout. Ensure a freshly checked-out base
# has its pinned Rust toolchain before Lake invokes Cargo.
- name: Bootstrap base Rust toolchain
if: inputs.label == 'Base benchmark binary build CPU'
- id: report
shell: bash
run: |
set -euo pipefail

toolchain_file=base/rust-toolchain.toml
[ -f "$toolchain_file" ] || { echo "::error::$toolchain_file is missing"; exit 1; }
channel=$(awk -F '"' '/^[[:space:]]*channel[[:space:]]*=/ { print $2; exit }' "$toolchain_file")
profile=$(awk -F '"' '/^[[:space:]]*profile[[:space:]]*=/ { print $2; exit }' "$toolchain_file")
if [[ ! "$channel" =~ ^[A-Za-z0-9._+-]+$ ]]; then
echo "::error::$toolchain_file has an invalid Rust channel"
exit 1
fi
case "${profile:-default}" in
minimal|default|complete) ;;
*) echo "::error::$toolchain_file has an invalid Rust profile"; exit 1 ;;
esac

rustup run "$channel" rustc --version >/dev/null 2>&1 && exit 0
echo "Installing Rust $channel (${profile:-default} profile) for the fresh base build"
rustup toolchain install "$channel" --profile "${profile:-default}" --no-self-update

- shell: bash
env:
CPU_LABEL: ${{ inputs.label }}
CPU_OUTPUT_FILE: ${{ inputs.output-file }}
CPU_PROVENANCE_FILE: ${{ inputs.provenance-file }}
CPU_PROVENANCE_LABEL: ${{ inputs.provenance-label }}
CPU_WARNING_FILE: ${{ inputs.warning-file }}
LOG_CURRENT_CPU: ${{ inputs.log-current }}
run: |
set -euo pipefail
export LC_ALL=C

resolve_path() {
if [[ "$1" == \~/* ]]; then
printf '%s/%s\n' "$HOME" "${1:2}"
else
printf '%s\n' "$1"
fi
}

model_name() {
awk -F ': *' '/^Model name:/ { print $2; exit }' "$@"
}

current_cpu_model=$(lscpu | model_name)

cpu_report() {
echo "Context: $CPU_LABEL"
echo "Runner: ${RUNNER_NAME:-unknown} (${RUNNER_OS:-unknown}/${RUNNER_ARCH:-unknown})"
echo "Kernel: $(uname -srvm)"
echo "nproc: $(nproc)"
if [ -r /sys/fs/cgroup/cpu.max ]; then
echo "cgroup cpu.max: $(< /sys/fs/cgroup/cpu.max)"
fi
if [ -r /sys/fs/cgroup/cpuset.cpus.effective ]; then
echo "cgroup cpuset.cpus.effective: $(< /sys/fs/cgroup/cpuset.cpus.effective)"
fi
grep -m1 '^Cpus_allowed_list:' /proc/self/status || true
lscpu
flags=$({ grep -m1 -oE 'avx2|avx512[a-z0-9_]*' /proc/cpuinfo || true; } \
| sort -u | tr '\n' ' ')
echo "AVX flags: ${flags:-absent}"
}

if [ "$LOG_CURRENT_CPU" = true ]; then
echo "::group::$CPU_LABEL"
if [ -n "$CPU_OUTPUT_FILE" ]; then
output_file=$(resolve_path "$CPU_OUTPUT_FILE")
mkdir -p "$(dirname "$output_file")"
cpu_report | tee "$output_file"
else
cpu_report
fi
echo "::endgroup::"
cpu_info=$(lscpu)
cpu_model=$(printf '%s\n' "$cpu_info" | awk -F ': *' '/^Model name:/ { print $2; exit }')
vcpus=$(nproc)
ram_kib=$(awk '/^MemTotal:/ { print $2; exit }' /proc/meminfo)
ram=$(awk -v kib="$ram_kib" 'BEGIN { printf "%.1f GiB", kib / 1048576 }')

echo "::group::$CPU_LABEL"
echo "Context: $CPU_LABEL"
echo "Runner: ${RUNNER_NAME:-unknown} (${RUNNER_OS:-unknown}/${RUNNER_ARCH:-unknown})"
echo "Kernel: $(uname -srvm)"
echo "nproc: $vcpus"
echo "RAM: $ram"
if [ -r /sys/fs/cgroup/cpu.max ]; then
echo "cgroup cpu.max: $(< /sys/fs/cgroup/cpu.max)"
fi

if [ -n "$CPU_PROVENANCE_FILE" ]; then
provenance_file=$(resolve_path "$CPU_PROVENANCE_FILE")
echo "::group::Native benchmark binary build CPU"
if [ -s "$provenance_file" ]; then
cat "$provenance_file"
else
echo "::warning::Binary build CPU provenance is unavailable (the cache may predate CPU reports)"
fi
echo "::endgroup::"

provenance_model=$(model_name "$provenance_file" 2>/dev/null || true)
warning=""
if [ -z "$provenance_model" ] || [ -z "$current_cpu_model" ]; then
warning=$(printf '**CPU provenance unavailable for %s:** one report has no CPU model, so native-code compatibility cannot be checked.' "$CPU_PROVENANCE_LABEL")
elif [ "$provenance_model" != "$current_cpu_model" ]; then
warning=$(printf "**CPU model mismatch for %s:** built on \`%s\`; measured on \`%s\`. Native Rust code uses \`-Ctarget-cpu=native\`." "$CPU_PROVENANCE_LABEL" "$provenance_model" "$current_cpu_model")
fi
if [ -n "$warning" ]; then
echo "::warning::$warning"
if [ -n "$CPU_WARNING_FILE" ]; then
warning_file=$(resolve_path "$CPU_WARNING_FILE")
mkdir -p "$(dirname "$warning_file")"
printf '%s\n' "- $warning" >> "$warning_file"
fi
fi
if [ -r /sys/fs/cgroup/cpuset.cpus.effective ]; then
echo "cgroup cpuset.cpus.effective: $(< /sys/fs/cgroup/cpuset.cpus.effective)"
fi
grep -m1 '^Cpus_allowed_list:' /proc/self/status || true
printf '%s\n' "$cpu_info"
flags=$({ grep -m1 -oE 'avx2|avx512[a-z0-9_]*' /proc/cpuinfo || true; } \
| sort -u | tr '\n' ' ')
echo "AVX flags: ${flags:-absent}"
echo "::endgroup::"

printf 'summary=CPU: `%s` · vCPUs: `%s` · RAM: `%s`\n' \
"${cpu_model:-unknown}" "$vcpus" "$ram" >> "$GITHUB_OUTPUT"
34 changes: 34 additions & 0 deletions .github/actions/setup-lake-cache/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Set up Namespace build cache
description: >-
Mount .lake, Namespace's built-in cache presets, and optional build
directories on the shared cache volume.

inputs:
cache-modes:
description: Built-in Namespace cache presets to mount (for example, rust or nix)
required: false
default: rust
additional-paths:
description: Paths to mount in addition to the repository's .lake directory
required: false

outputs:
cache-hit:
description: Whether Namespace restored a prior cache volume
value: ${{ steps.cache.outputs.cache-hit }}

runs:
using: composite
steps:
- name: Create Lake build directory
shell: bash
run: mkdir -p "${{ github.workspace }}/.lake"

- name: Mount Namespace build cache
id: cache
uses: namespacelabs/nscloud-cache-action@v1
with:
cache: ${{ inputs.cache-modes }}
path: |
${{ github.workspace }}/.lake
${{ inputs.additional-paths }}
Loading
Loading