fix(ci): give each checkout its own bazel output base in container-run.sh - #11441
Merged
Conversation
…n.sh
Containers started from different checkouts (git worktrees or clones)
all mount the checkout at /ic and share ~/.cache, so they used a single
bazel output base (md5("/ic")). Bazel's client only recognizes a live
server by its pid and /proc start time, which it cannot see across the
containers' PID namespaces, so the second container started another
server in the same output base and the first server halted itself:
Server terminated abruptly (error code: 14, error message:
'recvmsg:Connection reset by peer', ...)
With --network=host the second client could even be served by the first
container's server, i.e. run against the other checkout.
container-run.sh now points $BAZELRC at a generated rc file that sets
--output_base to ~/.cache/bazel/_bazel_ubuntu/<basename>-<sha of the
host path>, keeping the install base and the repository caches shared
and leaving the rest of the bazel configuration (.bazelrc, user.bazelrc)
untouched. For linked git worktrees it also bind-mounts the main
repository's .git directory at its host path so that git works inside
the container, and disables gc's automatic worktree pruning there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unsanitized or excessively long checkout names can break every Bazel invocation or prevent container startup.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Isolates Bazel output bases per checkout and adds linked Git worktree support.
Changes:
- Generates a checkout-specific Bazel rc file.
- Mounts linked worktrees’ common Git directory.
- Documents concurrent checkout behavior and limitations.
File summaries
| File | Description |
|---|---|
ci/container/README.md |
Documents parallel checkout behavior and cache usage. |
ci/container/container-run.sh |
Configures isolated output bases and worktree mounts. The checkout basename must be sanitized and bounded before use in rc paths and filenames. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 1
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
… key The basename is copied into the generated `startup --output_base=...` rc line, into the rc file name passed via the comma-separated $BAZELRC variable, and into a directory name, so whitespace, quotes, backslashes or commas in a checkout name would break bazel in that container. Keep only [A-Za-z0-9._-], cap it at 64 characters, and rely on the path hash for uniqueness. Keys of ordinary checkout names are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
Both reviewed files have no unresolved issues, and the changes were verified for concurrent checkouts and linked worktrees.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Balanced
rzakrzyk
approved these changes
Sep 4, 2026
basvandijk
added a commit
that referenced
this pull request
Sep 4, 2026
… checkout
The dev container of .devcontainer/devcontainer.json still used bazel's
default output base, md5("/ic"), so two dev containers opened on two
checkouts (e.g. two git worktrees) killed each other's bazel server the
same way container-run.sh containers did before #11441.
The initializeCommand now runs ci/container/devcontainer-initialize.sh on
the host before the container is created. It creates the bind-mounted
dirs/files as before and writes a bazelrc with
`startup --output_base=~/.cache/bazel/_bazel_ubuntu/devcontainer-<key>`
under ~/.cache/container-run/devcontainer<checkout path>/, which
containerEnv's BAZELRC points at through the ~/.cache mount. The key
computation is shared with container-run.sh via the new
ci/container/bazel-output-base-key.sh (keys are unchanged). The
"devcontainer-" prefix keeps the dev container's output base distinct from
the container-run.sh one of the same checkout: they run in separate PID
namespaces, so the Bazel extension's background queries would otherwise
kill a server running in the other container.
Existing dev containers need one "Rebuild Container" for the new
containerEnv to take effect and then start with a cold output base. Git in
a dev container opened on a linked worktree remains unsupported (VS Code's
experimental worktree mount is skipped when workspaceMount is customized,
microsoft/vscode-remote-release#11478); this is documented in the README.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Running
./ci/container/container-run.shfrom two checkouts (e.g. a git worktree next to the main checkout) and invoking bazel in both kills the first bazel server:Every checkout is mounted at
/icand shares~/.cache, so all containers use the same output base (6d065581...ismd5("/ic")). Each container has its own PID namespace, and Bazel's client only recognizes a live server by its pid and/proc/<pid>/statstart time, so the client in the second container treats the first container's server as dead (restart_reason=pid_file_but_no_server), starts a second server in the same output base, and its launcher rewritesserver.pid.txt. The first server'sPidFileWatcherthen halts it. Because the containers use--network=host, the second client can even end up connected to the first container's server and run its command against the other checkout (this happens when both servers get the same pid number in their namespaces). The same collision occurs between a VS Code devcontainer and acontainer-run.shcontainer on different checkouts.Changes
container-run.shgives every checkout its own bazel output base,~/.cache/bazel/_bazel_ubuntu/<basename>-<sha256 of the host path>, by pointingBAZELRCat a generated one-line rc file under<cache dir>/container-run/. Bazel reads$BAZELRCin addition to the workspace.bazelrc(including itsuser.bazelrcimport) and~/.bazelrc, so the rest of the configuration is unchanged; the install base, the repository cache and the repo contents cache remain shared. The variable only exists inside the container, so host-side bazel is unaffected..gitdirectory is bind-mounted at its host path so that git works inside the container (--config=stamped,ci/scripts/rust-lint.sh,ic-admin's build script under cargo). gc's automatic worktree pruning is disabled in the container (gc.worktreePruneExpire=never) because linked worktrees look prunable from inside it.Notes
~/.cache/bazel/_bazel_ubuntu/6d065581cce7ad9076e3b8db2b3afaf0can be deleted afterwards.podman execfor a second shell..devcontainer/devcontainer.jsonis unchanged and keeps the default output base. It no longer collides withcontainer-run.shcontainers, but two devcontainers on different checkouts still would; that can be a follow-up.Testing
Verified on a devenv with an isolated cache dir (
-c) from the main checkout and from a linked worktree: each container got its own output base; a server started from the main checkout survived a second container from the worktree starting its own server, and itsserver_pidwas unchanged afterwards; inside the worktree containergit rev-parse HEAD,git status,bazel/workspace_status.sh --stampandgit config gc.worktreePruneExpire(never) all worked.bazel run //pre-commit:shfmt-checkpasses.🤖 Generated with Claude Code