VirtioFS serves symlinks with st_size=0 on cold path lookups (Sandboxes microVM mounts)

[JStonevalley]

Summary

Through a VirtioFS mount, symlinks are served with st_size = 0 once the virtiofs server first enumerates their directory on behalf of the guest — which, for a sandbox/boot, is how every pre-existing symlink gets its attributes cached. readlink(2) still returns the correct target, and the macOS host reports the correct size. Creating or touching the file on the host immediately restores the correct size in the guest (after the guest's ~1s attribute cache TTL); recreating the file from the guest also works. A symlink created on the host inside an already-enumerated directory reports correctly, and the poisoned state returns whenever the guest enumerates the containing directories again (observed with guest-side find walks) — both point at the directory-enumeration path in the server: its batch attribute fetch appears to drop symlink size.

Environment

• Docker Desktop 4.89.0, engine 29.7.2, macOS 26.6.2 (Apple Silicon)
• Docker Sandboxes (sbx v0.39.0) microVM, guest kernel 7.0.12 aarch64, workspace mounted via virtiofs (host /Users/... virtiofs rw,relatime from /proc/mounts in the guest)
• Guest git 2.53.0; statx is ENOSYS in the guest, so all stat traffic uses newfstatat
• Verified with python os.lstat and coreutils stat inside the guest

Steps to reproduce

With any sbx sandbox whose workspace is a macOS host directory (paths are identical on both sides of the mount):

WS=~/path/to/workspace            # the mounted workspace
mkdir -p "$WS/.sbx-probe/repro"   # a directory NEW to the VM
ln -s a-target "$WS/.sbx-probe/repro/link"

stat -f %z "$WS/.sbx-probe/repro/link"   # host -> 8 (target-string length, correct)

sbx exec <sandbox> -- sh -c "cd '$WS/.sbx-probe/repro' && readlink link && stat -c %s link"
# guest -> a-target    (readlink works)
# guest -> 0           <- the bug

touch -h "$WS/.sbx-probe/repro/link"   # host-side event
sleep 2                                # guest attr-cache TTL
sbx exec <sandbox> -- sh -c "stat -c %s '$WS/.sbx-probe/repro/link'"   # guest -> 8, fixed

Expected behavior

stat inside the guest returns the symlink's target-string length, matching the host.

Actual behavior

It returns 0 until a host-side event on the file (touch -h) or a guest-side create repopulates the attribute. Guest-side enumeration of the containing directory re-poisons it.

Impact

git inside the sandbox flags every pre-existing tracked symlink as modified and renders it as a deletion plus an empty file, because git (a) compares st_size against its index under every core.checkStat mode and (b) treats zero-size worktree entries as empty content in diff_populate_filespec without ever calling readlink. The phantom modifications cannot be cleared with git update-index --refresh, and they reappear after every VM boot. Any other tool that trusts st_size (rsync, tar, ...) is similarly affected. We currently work around it by touch -h-ing tracked symlinks from the host at every sandbox start.

Diagnostics: not uploaded — happy to provide on request.

[JStonevalley]

Correction: the original report had the trigger and the cure backwards

I filed this describing directory enumeration as the poison. That is wrong, and it would have made this impossible to reproduce from the steps as written. Apologies. Re-measured on the same setup (Docker Desktop 4.89.0, engine 29.7.2, macOS 26.6.2 Apple Silicon; Docker Sandboxes sbx v0.39.0 microVM, guest kernel 7.0.12 aarch64, guest git 2.53.0; statx is ENOSYS in the guest so all stat traffic is newfstatat).

What actually happens

The virtiofs server answers a cold path lookup of a symlink — one it holds no cached attribute record for — with st_size = 0, and caches that zero. It appears not to call readlink on that path to compute the length. readlink(2) itself keeps returning the correct target throughout, and the host reports the correct size.

Size is the only thing affected, and nothing else varies it. Across a matrix of 14 symlinks (target lengths 5–40, targets that exist and targets that dangle), all 14 report 0 on a first targeted lstat in a directory the guest has never touched — no enumeration involved. Regular files in the same directories always report correct sizes.

Two things restore the truth, and they cover different states:

• A host-side event on the file (touch -h <link>) replaces a cached zero with the correct size, for that file only. Sticky, and it does not disturb correct siblings.
• A guest-side readdir (ls -la, find) makes the server rescan entries it holds no record for and report them truthfully. It does not clear an existing zero record — asked for one of those, the server serves the zero and ls -la prints 0.

And one thing re-breaks it:

• A host-side event on a directory (touch <dir>) drops the server's records for everything inside it, so the next access to each is a cold lookup → 0 again. A per-file touch -h issued in the same window is coalesced into that directory event and does not take. In that state only a guest readdir recovers the directory: I measured four symlinks stuck at st_size=0 across 26s of repeated host-side touching, cured instantly by one ls -la of their directories.

In one case a directory touch left the directory enumerating as empty inside the guest — ls -1a returned 2 entries against 59 on the host — until a host event on a file inside it. That persisted for at least 10s and did not self-heal.

There is also a shorter-lived guest-side layer: immediately after a host touch, ls -la and lstat() can disagree on the same path in the same VM (observed 15 vs 0, converging ~10s later).

Corrected reproducer

WS=/path/to/mounted/workspace
mkdir -p "$WS/probe/new-dir"        # a directory the guest has never touched
ln -s a-target "$WS/probe/new-dir/link"

stat -f %z "$WS/probe/new-dir/link"                 # host -> 8, correct

# first guest access is a targeted lstat, no enumeration:
sbx exec <sandbox> -- stat -c %s "$WS/probe/new-dir/link"   # -> 0   <-- the bug
sbx exec <sandbox> -- readlink  "$WS/probe/new-dir/link"    # -> a-target, correct

touch -h "$WS/probe/new-dir/link" && sleep 2
sbx exec <sandbox> -- stat -c %s "$WS/probe/new-dir/link"   # -> 8, fixed

Timing note: if the guest's first lookup lands well after the ln -s, the FSEvent from the creation may already have populated a truthful record and the probe reads 8. Create the link before the guest has ever looked at the directory.

To see the directory-event variant, take a workspace whose symlinks currently read correctly, touch their containing directories on the host, and re-stat from the guest: all of them go to 0.

Impact (unchanged)

git status / git diff inside the sandbox flags every pre-existing tracked symlink as modified and renders it as a deletion plus an empty file, because git compares st_size against its index under every core.checkStat mode, and diff_populate_filespec short-circuits a zero-size worktree entry to empty content without calling readlink. git update-index --refresh cannot clear it — its own content check hits the same short-circuit. Any other tool that trusts st_size (rsync, tar, …) is affected the same way.

Expected: a cold lookup of a symlink returns the target-string length, as the host does, without needing a prior host event or a readdir.