Verify packed PEX cache entries before reuse.

[apetti1920]

Problem

PEXBuilder._build_packedapp admits an existing bootstrap-zip or packed-wheel cache entry on atomic_directory(...).is_finalized() alone, which is just os.path.exists of the target dir, and then safe_copys the cached zip into the PEX being built — outside the atomic block, without inspecting it.

If an entry is ever finalized while incomplete, it is reused verbatim by every later build sharing that PEX_ROOT. A single bad write becomes a permanent, deterministic failure for every subsequent build:

RUN PEX_TOOLS=1 python3 /binary-deps.pex venv --scope=deps --compile /bin/app
pex.dist_metadata.MetadataError: Failed to determine project name and version
  for distribution at .../.deps/<wheel>.whl

or, when it is the bootstrap zip that is short, ModuleNotFoundError: No module named 'pex.version'.

We hit this across a large CI fleet. The tell that it is not a bad dependency is that the missing artifact differs every time — we saw botocore, grpcio, rpds_py, ConfigArgParse, aiobotocore, and in one case pex.version itself. The victim is positional, not packaged. Because the cache entry is stable, retries reproduce the failure byte-for-byte; one of our builds was retried 11 times and failed identically each time.

Note the corrupt artifacts are structurally valid zips with correct central directories, so testzip() and namelist() both pass on them. They are simply missing members.

Fix

cache_zip records the digest of the zip inside the atomic_directory work dir, so it lands through the same atomic rename as the zip it describes, and verifies it before reuse. A mismatched entry is discarded and re-created once; a second failure raises rather than shipping a bad PEX.

An entry written before this existed carries no digest and is treated as unverified, so caches already poisoned heal on first contact. That costs one re-pack per stale entry.

Scope

This addresses reuse of a bad entry, which is what turns one bad write into a permanent failure for everyone sharing the cache. It deliberately does not claim to prevent creation of one: the digest is taken from whatever create_zip wrote, so a short write that still yields a structurally valid zip is only caught by the emptiness guard. The creation-side race appears to live in atomic_directory's EEXIST recovery, where safe_mkdir(work_dir, clean=True) cannot distinguish a dead writer from a live one — its own comment says as much. That felt like a maintainer call rather than something to fold in here, so I have left it alone.

Verification

• New test_cache_zip_rejects_incomplete_entry covers cold build, warm reuse without rebuild, a poisoned-but-structurally-valid entry being discarded and re-created, and an entry with no recorded digest being re-created. test_cache_zip_raises_when_no_zip_produced covers the loud-failure path.
• End to end, reproducing the original failure: build a --layout packed PEX, replace the cached packed wheel with a valid zip whose .dist-info is gone, rebuild. Before this change the build exits 0 and silently ships a corrupt PEX whose venv --scope=deps then fails with the MetadataError above; after it, the entry is discarded, re-created, and the PEX is correct.
• tests/test_pex_builder.py and tests/test_atomic_directory.py pass (46 passed). Two test_build_compression cases error with SystemExit: Pex tests must be run via testing/bin/runtests.py both with and without this change, so they are pre-existing to my environment rather than caused here.
• black and isort clean. I could not get uv run dev-cmd format lint typecheck to bootstrap its venv locally (its pip install -U pip step fails in my environment), so please treat CI as the authority on the full gate.

Happy to adjust the approach — including moving the check inside atomic_directory instead, if you would rather it be generic across all cache users.

[jsirois]

@apetti1920 please provide more context on your Pex use. Is it direct or indirect, say via Pants? The current scheme is very deliberate. Pex uses atomic_directory to ensure bad cache dirs can never be written in the 1st place and relies on this for speed of reads. Absent bugs, a bad cache dir can only be created by another entity deleting cache dir contents or feeding Pex bad inputs for the cache dir. Pex considers the former outside the bounds of what it guards against and purposefully trusts its own cache. The latter is a possibility with, for example, --venv-repository (as opposed to using Pip, which is the default resolver) if the venv is malformed and Pex does not currently check for malformed venvs - it trusts you give it a good one. With that understood by you I'd like to understand if your bad cache directories are from a Pex bug - in which case I'd like to fix that root bug instead - or from an external party.

For example of (likely) external bad, see: pantsbuild/pants#23657

Here Pex is being fed corrupt uv venvs as the input source of pre-installed wheels.

[jsirois]

@apetti1920 your profile links klaviyo via Linked In; so I assume this is Pants. If that's true, are you using the uv resolver and what Pants version is this?

[goodwin-klaviyo]

We're on pants 2.33.0 Checking on UV

[jsirois]

It would be this option in pants.toml: https://www.pantsbuild.org/dev/reference/subsystems/python#resolver:

[python]
resolver = "uv"

[goodwin-klaviyo]

Ok yep we're not using that.

[apetti1920]

@apetti1920 please provide more context on your Pex use. Is it direct or indirect, say via Pants? The current scheme is very deliberate. Pex uses atomic_directory to ensure bad cache dirs can never be written in the 1st place and relies on this for speed of reads. Absent bugs, a bad cache dir can only be created by another entity deleting cache dir contents or feeding Pex bad inputs for the cache dir. Pex considers the former outside the bounds of what it guards against and purposefully trusts its own cache. The latter is a possibility with, for example, --venv-repository (as opposed to using Pip, which is the default resolver) if the venv is malformed and Pex does not currently check for malformed venvs - it trusts you give it a good one. With that understood by you I'd like to understand if your bad cache directories are from a Pex bug - in which case I'd like to fix that root bug instead - or from an external party.

For example of (likely) external bad, see: pantsbuild/pants#23657

Here Pex is being fed corrupt uv venvs as the input source of pre-installed wheels.

@jsirois Via Pants 2.33.0, PEX version: 2.97.3, pinned by us (Pants 2.33's own floor is 2.97.1).

The UV resolver is unset so we're on the default (Resolver.pex, pip via PEX). We have no --venv-repository anywhere. uv appears in our repo only as a lockfile-generation tool (a uv pip compile alias) and as a Pants tool resolve it never produces the pre-installed wheels PEX consumes. So I don't think pantsbuild/pants#23657 applies to us; that one is specifically PEX reading a venv another uv sync is mid-write and we never hand PEX a venv as an input source.

Our environment is alittle unusual in that PEX_ROOT is a Pants append only named cache which is shared by every concurrently executing pex process on the machine and never pruned or invalidated by Pants. These are long-lived CI agents that serve many jobs in parallel and builds are routinely SIGKILLed mid-flight by the merge-queue. So high concurrency against one PEX_ROOT plus abrupt termination. Nothing external should write to or deletes from PEX_ROOT so it shouldnt be the case that another entity deleted cache dir contents.
which is why I suspect its corruption is in PEX's own caches, not in anything we feed it
- packed_wheels entries: a structurally valid zip, correct central directory but no .dist-info, so we see MetadataError at PEX_TOOLS=1 ... venv --scope=deps.
- One production failure was a short .bootstrap missing pex/version.py → ModuleNotFoundError: No module named 'pex.version' . That's PEX's own code, with no resolver input involved at all.
- The affected distribution differed every time rather than tied to any particular input.

We also see PEX's own warning in the logs immediately before these:

After obtaining an exclusive lock on .../packed_wheels/1/<hash>/.defN.atomic_directory.lck,
failed to establish a work directory at .../packed_wheels/1/<hash>/defN.lck.work
due to: [Errno 17] File exists
Continuing to forcibly re-create the work directory at ...

If you'd rather fix the root cause than the defensive check I'm happy to close it. I wrote it to stop a reuse loop with one bad write becoming a permanent failure for every subsequent build on that machine but it does not prevent creation of a bad entry. If the creation path gets fixed most of its value goes away, though it might still be useful for the case of SIGKILLS

[jsirois]

Our environment is alittle unusual in that PEX_ROOT is a Pants append only named cache which is shared by every concurrently executing pex process on the machine and never pruned or invalidated by Pants.

That is actually the typical Pants scenario! On your local machine your named cache directory is never pruned unless you do so manually and a single Pants invocation can run many Pex processes in parallel. As a result, there were many Pex bug fixes in the early days of Pants use of Pex as it exercised parallel Pex invocations.

If the creation path gets fixed most of its value goes away, though it might still be useful for the case of SIGKILLS

The atomic_directory works like so:

1. obtain flock lock on a file associated with the final directory
2. if and only if final directory does not exist, create a work directory next to it
3. populate the work directory and not the final directory
4. if and only if 3 completes without raising, do an atomic rename of the work directory to the final directory
5. unlock the flock lock

So, if the SIGKILL happens at any point before 4, there is never a rename and so partial population of the work dir will never be seen at the final directory rename location.

So all a SIGKILL can do is prevent a rename of a work dir to the final dir, in which case the work dir will be left around for the next try, and you'd see what you report:

After obtaining an exclusive lock on .../packed_wheels/1/<hash>/.defN.atomic_directory.lck,
failed to establish a work directory at .../packed_wheels/1/<hash>/defN.lck.work
due to: [Errno 17] File exists
Continuing to forcibly re-create the work directory at ...

This is as noted in the comment here:

pex/pex/atomic_directory.py

Lines 259 to 285 in e0d238b

	# If there is an error making the work_dir that means that either file-locking guarantees have
	# failed somehow and another process has the lock and has made the work_dir already or else a
	# process holding the lock ended abnormally.
	try:
	os.mkdir(atomic_dir.work_dir)
	except OSError as e:
	ident = "[pid:{pid}, tid:{tid}, cwd:{cwd}]".format(
	pid=os.getpid(), tid=threading.current_thread().ident, cwd=os.getcwd()
	)
	pex_warnings.warn(
	"{ident}: After obtaining an exclusive lock on {lockfile}, failed to establish a work "
	"directory at {workdir} due to: {err}".format(
	ident=ident,
	lockfile=atomic_dir.lockfile,
	workdir=atomic_dir.work_dir,
	err=e,
	),
	)
	if e.errno != errno.EEXIST:
	raise
	pex_warnings.warn(
	"{ident}: Continuing to forcibly re-create the work directory at {workdir}.".format(
	ident=ident,
	workdir=atomic_dir.work_dir,
	)
	)
	safe_mkdir(atomic_dir.work_dir, clean=True)

I.E.: workdir cleanup indicates a forceable kill of a prior attempt to populate the atomic directory (OK), or else a failed flock lock scenario where two processes hold the same lock (not OK!).

Its the latter that I suspect in this scenario because I've reviewed this locking code so many times now over the last many years. The known cases where flock fails are network filesystems like NFS and other exotic storage configuration scenarios. I'm currently focused on your use of docker overlayfs2 for the Pants named-caches directory mount from the host to confirm that style of mount does or does not allow flock to be implemented faithfully.

[jsirois]

@apetti1920 and @goodwin-klaviyo I'm definitely confused. Does this solve your CI issue? IIUC it does not solve a bad cache write, it just enshrines it further with a hash; so it can only detect cache tampering away from the original write - regardless of whether that original write was good or bad. I.E.: it detects a changed cache entry but says nothing more - could have changed from good to bad or bad to good or bad1 to bad2.

[apetti1920]

@apetti1920 and @goodwin-klaviyo I'm definitely confused. Does this solve your CI issue? IIUC it does not solve a bad cache write, it just enshrines it further with a hash; so it can only detect cache tampering away from the original write - regardless of whether that original write was good or bad. I.E.: it detects a changed cache entry but says nothing more - could have changed from good to bad or bad to good or bad1 to bad2.

@jsirois we are building the forked version now and going to use it in our CI to test if it fixes the cache issues we are seeing, will report back

[jsirois]

@apetti1920 ok, great. Per my analysis above, this can't do anything except detect a cache tamper or lock race AFAICT, but more data will be good.

I guess a pertinent question is also re:

RUN PEX_TOOLS=1 python3 /binary-deps.pex venv --scope=deps --compile /bin/app
pex.dist_metadata.MetadataError: Failed to determine project name and version
  for distribution at .../.deps/<wheel>.whl

Is that the only context you see the error in? If so, how is that docker image built? Is it in CI as well? If so, is it inline in the build, i.e.: via docker in docker IIUC?

[jsirois]

@apetti1920 and @goodwin-klaviyo the only thing I've been able to come up with is this scenario, which is still half baked as noted:

1. Pex process is killed while populating an atomic_directory work_dir, leaving it laying around and partial.
2. Later Pex process goes through current safe_mkdir(atomic_dir.work_dir, clean=True) which is silent when the underlying clean fails (it uses safe_rmtree which is really just shutil.rmtree(..., ignore_errors=True)) and the clean does in fact fail / is partial.

That much is solid - could happen if prior Pex process ran as root, say, and later Pex process ran with less permissions.

The next part is hand-waves and strange perms (722), but observe:

:; mkdir -p /tmp/foo/bar/baz
:; chmod 722 /tmp/foo/bar/baz
:; touch /tmp/foo/bar/baz/spam
:; tree -pug /tmp/foo/
[drwxrwxr-x jsirois  jsirois ]  /tmp/foo/
└── [drwxrwxr-x jsirois  jsirois ]  bar
    └── [drwx-w--w- jsirois  jsirois ]  baz
        └── [-rw-rw-r-- jsirois  jsirois ]  spam

3 directories, 1 file
:; sudo -u mail python -c '
import os
for root, dirs, files in os.walk("/tmp/foo"):
    for d in dirs:
        print(os.path.join(root, d))
    for f in files:
        print(os.path.join(root, f))
'
/tmp/foo/bar
/tmp/foo/bar/baz
:; echo $?
0

So the mail user can walk the /tmp/foo tree created by jsirois without errors, but it cannot see the /tmp/foo/bar/baz dir contents. I.E.: It can write but not read:

:; sudo -u mail echo "write but no read" > /tmp/foo/bar/baz/spam
:; sudo -u mail cat /tmp/foo/bar/baz/spam
cat: /tmp/foo/bar/baz/spam: Permission denied
:; cat /tmp/foo/bar/baz/spam
write but no read

Hand waves aside, this change should stand on its own as an overall behavior ~noop, with better diagnostics about the state of any stale work_dir: #3263.
Please have a look and chime in.

[apetti1920]

@apetti1920 and @goodwin-klaviyo the only thing I've been able to come up with is this scenario, which is still half baked as noted:

1. Pex process is killed while populating an `atomic_directory` work_dir, leaving it laying around and partial.

2. Later Pex process goes through current `safe_mkdir(atomic_dir.work_dir, clean=True)` which is silent when the underlying clean fails (it uses `safe_rmtree` which is really just `shutil.rmtree(..., ignore_errors=True)`) and the clean does in fact fail / is partial.

That much is solid - could happen if prior Pex process ran as root, say, and later Pex process ran with less permissions.

The next part is hand-waves and strange perms (722), but observe:

:; mkdir -p /tmp/foo/bar/baz
:; chmod 722 /tmp/foo/bar/baz
:; touch /tmp/foo/bar/baz/spam
:; tree -pug /tmp/foo/
[drwxrwxr-x jsirois  jsirois ]  /tmp/foo/
└── [drwxrwxr-x jsirois  jsirois ]  bar
    └── [drwx-w--w- jsirois  jsirois ]  baz
        └── [-rw-rw-r-- jsirois  jsirois ]  spam

3 directories, 1 file
:; sudo -u mail python -c '
import os
for root, dirs, files in os.walk("/tmp/foo"):
    for d in dirs:
        print(os.path.join(root, d))
    for f in files:
        print(os.path.join(root, f))
'
/tmp/foo/bar
/tmp/foo/bar/baz
:; echo $?
0

So the mail user can walk the /tmp/foo tree created by jsirois without errors, but it cannot see the /tmp/foo/bar/baz dir contents. I.E.: It can write but not read:

:; sudo -u mail echo "write but no read" > /tmp/foo/bar/baz/spam
:; sudo -u mail cat /tmp/foo/bar/baz/spam
cat: /tmp/foo/bar/baz/spam: Permission denied
:; cat /tmp/foo/bar/baz/spam
write but no read

Hand waves aside, this change should stand on its own as an overall behavior ~noop, with better diagnostics about the state of any stale work_dir: #3263. Please have a look and chime in.

@jsirois Heres what I found after digging throug. Summary up front, details below.

I think I've got the mechanism and I believe #3263 may address it directly. Pants routinely cancels in-flight pex processes when a sibling target fails. Each cancellation kills a pex mid-atomic_directory populate, leaving a stale .lck.work directory behind in the shared PEX_ROOT. A later pex hits EEXIST on that leftover directory, warns, and calls safe_mkdir(work_dir, clean=True), whose shutil.rmtree(..., ignore_errors=True) silently swallows partial cleanup failures. It then populates into the still-dirty directory, and finalize() renames the mixture into place. The resulting packed_wheels entry is a structurally valid .whl zip that's missing members, which ships inside the packed PEX as .deps/<wheel>.whl and blows up in the container at venv --scope=deps.

Why I don't think it's a lock race: every stale-work-dir warning cluster I found carries a single pid; different threads, different lock targets but never two pids contending on the same lock anywhere in the logs. That's evidence against a flock race and for your abnormal-termination branch instead.

The unabridged warning (packed_wheels, pex 2.97.3):

[pid:486, tid:137910866794304, cwd:/tmp/pants-sandbox-FgoQb1]: After obtaining an exclusive lock on
.../named_caches/pex_root/packed_wheels/1/d2d96d11…/.defN.atomic_directory.lck, failed to establish
a work directory at .../packed_wheels/1/d2d96d11…/defN.lck.work due to: [Errno 17] File exists

defN isn't anonymization (I confirmed against a live agent that it's the real compression-variant directory name).

The kill source: cancellations like this show up in the same jobs, both docker and non-docker:

Canceled: Building 143 requirements for .../app-deps.pex ... (1.5s)
Canceled: Building build_backend.pex from resource://.../setuptools.lock (0.9s)
Canceled: Building twine.pex from resource://.../twine.lock (0.7s)

So this isn't a rare crash artifact, Pants is cancelling siblings when one target fails happens on nearly every failing build, which supplies a high-frequency source of stale work dirs without needing an actual process crash.

Full chain, end to end:

1. A target fails, Pants cancels in-flight sibling pex builds, and those processes die mid-populate.
2. Stale .lck.work dirs are left behind in the durable, host-shared PEX_ROOT.
3. A later pex hits EEXIST, warns, and calls safe_mkdir(work_dir, clean=True) -> safe_rmtree -> shutil.rmtree(ignore_errors=True) (pex/common.py:620), which is silent on partial failure.
4. It populates into the still-dirty dir, and finalize() renames the mixture into place.
5. The packed_wheels entry is a valid zip missing members, which gets copied to .deps/<wheel>.whl.
6. venv --scope=deps in the container is the first thing that reads every wheel, so that's where MetadataError surfaces.

A couple of supporting details worth flagging: .deps/<wheel>.whl is a symlink into installed_wheels/, also populated via atomic_directory (pex/layout.py:186-201), so both cache directories are exposed to this bug. And the .layout.json with record_relpath we see in the corrupted grpcio artifact (pex/installed_wheel.py:27,39,51) can't appear in a raw PyPI wheel. Its presence is positive evidence that the installed-wheel-chroot path actually ran, not just a downloaded wheel being reused as-is.

To answer your two questions:

Is PEX_TOOLS=1 ... venv --scope=deps the only place we see this, and does that mean anything? Yes, it's the only place, but I don't think it's meaningful. The generated Dockerfile is just COPY <pex> /binary-deps.pex followed by RUN ... venv --scope=deps (k-repo build_support/plugins/macros.py:1070-1073), with no PEX_ROOT cache mount, so the container root is fresh on every retry. Same wheel and same pex hash across all three retries of a given build; different wheels across different builds (grpcio, then aiobotocore on a later build). That tells me the artifact ships corrupt from the build step and the venv step is simply the first thing that reads every wheel in .deps/ and not a distinguishing environment.

How is the image built? in CI, DinD? It's Pants' docker_image via our klaviyo_pex_docker macro, built inline in CI, not docker-in-docker. We bind-mount the host daemon socket (/var/run/docker.sock:/var/run/docker.sock, DOCKER_HOST=unix:///var/run/docker.sock). The daemon runs overlay2 with userns-remap: buildkite-agent, data-root under /mnt/ephemeral/docker/<uid>.<gid>.

On the uid-mismatch idea you raised: it's structurally possible but I couldn't confirm it happening. /etc/subuid maps container uid 0 to the agent's own uid (buildkite-agent:$(id -u buildkite-agent):1) and our CI containers run as root (no user: override in the plugin config, pip.conf mounted to /root/.pip/). So container-root writes land as buildkite-agent, same uid, no mismatch. A real mismatch would need a non-root container user landing in the 100000+ range, and I didn't find one in our setup.

A few infra notes in case they're relevant to your read on this: I verified via findmnt on live agents that the Pants named cache has never lived on overlayfs, it's always been plain ext4, on /, /mnt/ephemeral, /tmp or the builds volume. Docker itself has used overlay2 for several years unchanged; only its data-root moved since then. I think an earlier pass at this investigation conflated the cache mount with the Docker storage driver and they're separate. I also confirmed the build tree is reachable via two paths that are inode-identical, so flock contends correctly across them.

Separately, our CI mixes host and containerized Pants environments in a single invocation, only python 3.10 is overridden to run on the host (//:build_machine), while default and python_3_12 remain docker_environments, all sharing the same named_caches. And we have local_cache = false / remote_cache_read/write = true with bazel-remote over gRPC, namespaced only by Pants version, so if a corrupt-but-exit-0 artifact ever got uploaded there, it'd be served to every agent. That's a real amplifier in theory but a sampled healthy run showed 0 read errors and 0 missing digests, so I haven't caught it in the act.

On the cache path itself: we had the named cache set to a durable, host-shared path from for a few days a couple weeks ago to test then reverted back to worktree-local (which gets wiped every checkout by the agent's git-clean-flags = -ffxdq). One of the builds I looked at still failed after that revert, so the durable-shared-cache setup isn't the whole story though it may still have been a contributing factor while it was active.

One question back this PR. after it falls back to a random work_dir, does the original stale .lck.work ever get cleaned up, or does every subsequent process just repeat the failed-rmtree -> warn -> random-dir path indefinitely? If it's the latter, might be worth a small follow-up to at least surface it as something actionable rather than a recurring silent warning.

[jsirois]

@apetti1920 at a very high level I want to stress that this:

Pants routinely cancels in-flight pex processes when a sibling target fails. Each cancellation kills a pex mid-atomic_directory populate, leaving a stale .lck.work directory behind in the shared PEX_ROOT. A later pex hits EEXIST on that leftover directory, warns, and calls safe_mkdir(work_dir, clean=True), whose shutil.rmtree(..., ignore_errors=True) silently swallows partial cleanup failures. It then populates into the still-dirty directory, and finalize() renames the mixture into place. The resulting packed_wheels entry is a structurally valid .whl zip that's missing members, which ships inside the packed PEX as .deps/.whl and blows up in the container at venv --scope=deps.

Has been true for Pants and Pex for a very long time on desktop and in CI; so this is a poor explanation candidate from the get go since we don't have reports pouring in of this issue. Your report is currently singular. This is why I've been focusing hard on what is unique about your CI environment. None of your paragraph above is unique to your CI in any way. For example, @goodwin-klaviyo mentioned you run on m8gd.4xlarge and m8id.4xlarge instances in Slack - those have 16 vcpus which is < a typical dev laptop these days. I have 16 hyperthreads on my 5 year old laptop for example. Developers hammer Pants harder than your CI does IOW.

4. It populates into the still-dirty dir, and finalize() renames the mixture into place.

@apetti1920 the issue is, unless there is a perms weirdness similar to the one I pointed out, the population into the dirty directory succeeds (or else atomic dir would not exit cleanly and "publish" the result) and so the previously dirty work dir is now fully populated! Its only if there are write perms for part of that dir but not read perms, would a later read see partial. As such - I do not think this is a valid explanation unless you can verify wonky perms / ownership or you can come up with some other mechanism where a partial work_dir is re-populated sucessfully, but that somehow results in a partial re-populate. That is the very sticky wicket here!

built inline in CI, not docker-in-docker. We bind-mount the host daemon socket

Bind-mounting the host daemon socket into a docker image == dnd (docker in docker) - you only need to do this (bind the docker daemon socker into a container) to run docker from within a docker container.

One question back this PR. after it falls back to a random work_dir, does the original stale .lck.work ever get cleaned up, or does every subsequent process just repeat the failed-rmtree -> warn -> random-dir path indefinitely? If it's the latter, might be worth a small follow-up to at least surface it as something actionable rather than a recurring silent warning.

The .lck.work is never cleaned, but as soon as there is a successful work dir publish, no work is ever done again for the cache entry. So if the random work dir population works, thats the last time there is ever anything except a pure cache read for the entry. If the random dir population fails, Pex lets the raised error bubble - and so you'll know.

[jsirois]

So, @apetti1920 and @goodwin-klaviyo if #3263 does fix your issue, I'm claiming that is proof you have wonky perms / ownership in CI - that would be the only way that PR could fix anything as I understand the universe. To check that, you'd look for mixed ownership of files in pants named caches in CI. Probably root and some other user, but you know your CI setup best obviously.

FWIW - the thing that got me even thinking about that was this: https://github.com/buildkite-plugins/docker-buildkite-plugin#propagate-uid-gid-optional-boolean

I have never used BuildKite - but someone mentioned it - I think @goodwin-klaviyo - and it got me researching the service. That bit caught my eye and got a mixed ownership bug in my brain.

[jsirois]

Alright @apetti1920 & @goodwin-klaviyo the #3263 diagnostics for unexpected perms are now available here: https://github.com/pex-tool/pex/releases/tag/v2.101.2

[apetti1920]

Alright @apetti1920 & @goodwin-klaviyo the #3263 diagnostics for unexpected perms are now available here: v2.101.2 (release)

@jsirois Thank you we will try that update on our end as well to test and really appreciate you looking into and pushing back this hard on all of this, we're just trying to get our own devs unblocked.

Two corrections up front: you were right that a cache_zip entry can't be left partial by re-population, chroot.zip opens with mode="w", so a stale zip gets truncated on re-populate. The original "populate into the dirty dir and publish a mixture" story for packed_wheels doesn't hold after looking more into it. You're also right that bind-mounting the host docker socket into a container is DinD by definition.

Some way a partial work_dir gets re-populated successfully but still ends up partial does exist one layer down from where we first looked. packed_wheels zips are built from a chroot of symlinks into installed_wheels/, and installed_wheels is populated through atomic_directory (pex/pip/installation.py:226) by code in pex/pep_427.py:1119-1131 with three skip-on-exists paths: safe_copy(..., overwrite=False), elif not os.path.exists(dst_file), and a symlink branch that swallows EEXIST outright. A stale file that survives safe_mkdir(work_dir, clean=True)'s shutil.rmtree(..., ignore_errors=True) isn't overwritten, nothing raises and finalize() publishes the result anyway. A short installed_wheels entry then yields a structurally valid but incomplete packed_wheels zip one level up.

On permissions: our first guess was mixed ownership/uid, based on where we thought the cache lived. That was the wrong path. The real pex_root lives on a host-shared /tmp path set by an agent-level hook, and everything touching it runs as root inside the container. Ownership is uniform, so root's rmtree isn't permission-blocked and the mixed-uid theory doesn't hold for us. What's still unproven is what makes that rmtree partially fail at all under uniform ownership, our current leading candidate is a cancel race where a killed-but-not-yet-reaped pex process is still writing while its successor cleans up, so the rmdir hits ENOTEMPTY and gets silently swallowed.

A few things we confirmed directly in our logs: stale .lck.work dirs do survive into later processes here, cancellations reliably precede corruption within seconds across every build we examined and in the clearest case the corrupted wheel is unrelated to the target that actually failed. A poisoned shared cache entry, not a per-target bug, reproduced identically across all three automatic retries.

Last piece, and we think it's the real answer to "why is your report singular": on Aug 18 we deliberately pointed pex_root at a host-shared, durable /tmp path, because our hosts serve many sequential jobs before recycling. Before that change, pex_root didn't survive past a single job and a poisoned entry just got destroyed with the container or wiped by git clean before anyone noticed. This failure mode was probably always structurally possible, exactly as you said from the start. What's different for us is that most Pants setups have a pex_root scoped to one container or checkout, where a poisoned entry evaporates unseen. We removed that self-healing property on purpose, for durability and because pex_root is excluded from the Pants action key, a poisoned entry then propagates to every agent and branch through our remote cache. That's what makes cache-verification-on-reuse (#3262) load-bearing for us specifically, not just a nicety on top of a fix for the populate bug. Your .lck.work answer checks out against the code on our end too.

Still open on our side: the exact trigger for the partial rmtree (cancel race is our best guess, not confirmed), and whether a separate disk-pressure cleanup process could independently produce the same "directory exists, contents partial" symptom by reaping an already-published entry. Chasing both down next. Thanks again for looking into this with us.

[jsirois]

Some way a partial work_dir gets re-populated successfully but still ends up partial does exist one layer down from where we first looked. packed_wheels zips are built from a chroot of symlinks into installed_wheels/, and installed_wheels is populated through atomic_directory (pex/pip/installation.py:226) by code in pex/pep_427.py:1119-1131 with three skip-on-exists paths: safe_copy(..., overwrite=False), elif not os.path.exists(dst_file), and a symlink branch that swallows EEXIST outright. A stale file that survives safe_mkdir(work_dir, clean=True)'s shutil.rmtree(..., ignore_errors=True) isn't overwritten, nothing raises and finalize() publishes the result anyway. A short installed_wheels entry then yields a structurally valid but incomplete packed_wheels zip one level up.

@apetti1920 a skip on exists would mean, for example, you could never get ModuleNotFoundError: No module named 'pex.version', since pex/version.py would exist. What you might get is invalid content inside pex/version.py such that importing would fail due to truncated Python syntax or else import would work but symbols would be missing if the truncation were luckily clean. Again, you have not identified a novel circumstance of your CI - this is a standard code path in all Pants uses of Pex that would affect desktop use too. Worse, your scenario only explains partial file contents and not missing files. Your revealed error set in CI includes completely missing files.

on Aug 18 we deliberately pointed pex_root at a host-shared, durable /tmp path, because our hosts serve many sequential jobs before recycling. Before that change, pex_root didn't survive past a single job and a poisoned entry just got destroyed with the container or wiped by git clean before anyone noticed. This failure mode was probably always structurally possible, exactly as you said from the start. What's different for us is that most Pants setups have a pex_root scoped to one container or checkout, where a poisoned entry evaporates unseen.

A durable PEX_ROOT is exactly the scenario for every developer and as I pointed out above, developer machines are as beefy or beefier than CI machines - they hammer a persistent PEX_ROOT more than CI. So why aren't developers seeing this? What is different for them?

You have not addressed, AFAICT, how this change does anything other than hash a bad cache entry when a bad cache entry is written successfully.