Skip to content

feat: follow symlinked files and directories in the scan and the watcher (#935, #770) - #288

Merged
sunerpy merged 5 commits into
mainfrom
sync/v1.6.1-symlinks
Oct 1, 2026
Merged

sunerpy merged 5 commits into
mainfrom
sync/v1.6.1-symlinks

Conversation

@sunerpy

@sunerpy sunerpy commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR follows symlinked files and directories in the scan (upstream #935) and in the watcher (#770). Following them by default, including targets outside the project, was the owner's decision on 2026-10-01. The design passed the kirocodex plan-convergence review in round 3 (blocking items 6 → 1 → 0).

It also corrects the ledger. The 2026-06-24 sync recorded #935 as ALREADY-HAVE, but the official v0.51.0 binary indexed only src/real/a.ts from {src/real/a.ts, src/afile.ts -> real/a.ts, src/linkdir -> real, src/extlink -> ../../outside/lib}. The scan never followed a symlink.

Scan (crates/codegraph-extract/src/links.rs, engine.rs)

  • A symlink is judged by what it resolves to, at its logical path, under the same ignore, include and extension rules as a real entry.
  • Links are followed in hop levels, so a directory is scanned once, under the logical path with the fewest symlinks. Ties go to the lexicographically smallest path, and the real tree always wins. The result depends on the filesystem only, never on read_dir order.
  • These links are not followed:
    • a link to the project root or one of its ancestors;
    • a link into the project's .git or a reserved index root;
    • a missing or unreadable target.
  • ScanProjectResult gains links, linked_dirs and file_aliases.

Sync stays equal to index --force (crates/codegraph-watch/src/link_state.rs)

  • A retargeted link can put a different file with the same size and mtime at an indexed path. Every full build therefore records the links it followed in project_metadata key followed_links; no DDL change.
  • Full sync and the pending inventory re-read every path behind a link that is new, retargeted or retyped. They do the same under every link when the record is missing, which is the case for any older index.

Watcher (#770)

  • The scan alone decides which links are followed; the watcher only narrows that set with its stricter policy. It never watches a directory the scan did not walk through that exact logical path.
  • Linux: one NonRecursive watch per linked logical directory.
  • macOS/Windows: one Recursive supplemental watch per directory link, capped at 256. FSEvents' canonical event paths are mapped back to logical paths.
  • Topology changes: creating, removing or retargeting a link, or a new directory that holds one, re-reads the links, re-registers every link-reached watch, and schedules one full sync.
  • Removal: a known directory that is no longer a directory counts as removed whatever the notify hint. inotify reports a deleted symlink as a file removal.
  • Aliases: an edit to a file-link target also re-indexes the link.
  • Known limitation (as upstream): a file link whose target lies outside every scanned directory is not watched; the next full sync picks it up.

CI

  • A new macOS Watcher job runs the symlink scan and watcher tests on macos-15, the only backend no other job runs. It is a required need of CI Success, and ci-gate.test.sh follows the new needs line.
  • Symlink tests fail loudly in CI if a link cannot be created, so Windows CI exercises them rather than skipping.

Docs

  • docs/cli.md: the scope rules above.
  • UPSTREAM.md: the 1.6.0 → 1.6.1 callout keeps its v0.51.0 boundary and marks #770 as landed afterwards, and a new dated entry supersedes the #935 row without rewriting it.
  • The dated v1.6.1 audit header goes back to its original sentence: docs(upstream): close the v1.6.1 sync and advance tracked parity #285 had rewritten it, against docs/AGENTS.md's append-only rule.

Verification

  • New tests, all green:
    • crates/codegraph-extract/tests/scan_symlinks.rs (8): precedence, the hop-level counterexample from the plan review, skipped targets, ignore rules at the logical path, creation-order independence, aliases, unreadable targets. Against the pre-change scan, 6 failed; the two skip-rule tests are negative controls that hold either way.
    • crates/codegraph-cli/tests/symlink_sync.rs (4, through the real binary): indexing through links, a removed link, same-stat directory and file retargets, and an index without the record. With the re-read rule forced off, the three retarget and record tests fail because status reports nothing modified.
    • Eight new symlink_* watcher tests: supplemental selection and cap, canonical-to-logical mapping, the re-registration plan, the removal hint, scan/watch parity (.cache, .codegraph-sources), and, on a real watcher, an edit behind a link, alias fan-out, and link creation, retarget and removal.
  • Goldens: all 19 re-indexable corpora regenerated with this build are byte-identical.
  • Smoke: on the probe layout, the PR build indexes src/afile.ts, src/extlink/out.ts and src/real/a.ts where v0.51.0 indexed only src/real/a.ts. followed_links is recorded, and sync then status report nothing pending.
  • Pre-push gate (make pre-ci): 3,927 tests passed, 0 failed; clippy -D warnings, fmt, guardrail, workflow lint, ci-gate harness (11/11) and archive smoke clean.

BEGIN_COMMIT_OVERRIDE
feat(scan): follow symlinked files and directories (#935)
fix(sync): re-read files behind new, retargeted or unrecorded symlinks
feat(watch): follow the scan's symlinks with live watches (#770)
END_COMMIT_OVERRIDE

🤖 Generated with Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d7a8cde018

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +10 to +11
- **Tracked release:** [`UPSTREAM.md`](UPSTREAM.md) keeps `v1.6.0` until the
ported rows ship in a release and the downloaded binaries are verified.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore the verified v1.6.1 audit record

These lines rewrite the dated audit's already-verified conclusion from “advanced to v1.6.1 after v0.51.0 shipped” back to the obsolete pre-release state “keeps v1.6.0,” contradicting the current ledger and its 2026-10-01 closeout. Restore the historical text and place any later correction in a new ledger entry instead of altering append-only evidence.

AGENTS.md reference: docs/AGENTS.md:L19-L21

Useful? React with 👍 / 👎.

known_dirs.extend(next_links.watched_dirs.iter().cloned());
link_state = next_links;
supplemental = next_supplemental;
full_sync_pending = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Arm the debounce when topology alone requires a full sync

When a newly moved-in real directory already contains a symlink, collect_watch_tree sets link_topology_changed, but the directory event adds no source path to pending. This line sets full_sync_pending, while the subsequent deadline is armed only when pending is nonempty, so the promised full reconcile never runs and the linked subtree remains absent until an unrelated event or manual sync.

AGENTS.md reference: AGENTS.md:L39-L41

Useful? React with 👍 / 👎.

Comment on lines +650 to +652
for dir in unwatch {
// Best effort: a removed link's watch is often already gone.
let _ = watcher.unwatch(&project_root.join(dir));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve watches when a link becomes a real directory

On per-directory backends, if a followed directory symlink is atomically replaced by a real directory at the same logical path, the event handler first registers watches for the new real tree, but this loop then unwatches every path from the old link state. Because the new link state contains no corresponding link watches, those paths are not re-registered, and later edits below the replacement directory are silently missed.

AGENTS.md reference: AGENTS.md:L39-L41

Useful? React with 👍 / 👎.

@codecov

codecov Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.51429% with 48 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/codegraph-watch/src/watcher.rs 91.90% 43 Missing ⚠️
crates/codegraph-extract/src/engine.rs 96.87% 4 Missing ⚠️
crates/codegraph-extract/src/links.rs 98.27% 1 Missing ⚠️

❌ Your patch check has failed because the patch coverage (94.51%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #288      +/-   ##
==========================================
- Coverage   95.29%   95.28%   -0.02%     
==========================================
  Files         158      160       +2     
  Lines       97628    98419     +791     
==========================================
+ Hits        93037    93779     +742     
- Misses       4591     4640      +49     
Files with missing lines Coverage Δ
crates/codegraph-cli/src/main.rs 90.91% <100.00%> (+<0.01%) ⬆️
crates/codegraph-watch/src/lib.rs 97.72% <ø> (ø)
crates/codegraph-watch/src/link_state.rs 100.00% <100.00%> (ø)
crates/codegraph-watch/src/migrate.rs 83.00% <100.00%> (+0.34%) ⬆️
crates/codegraph-watch/src/sync.rs 97.90% <100.00%> (+0.02%) ⬆️
crates/codegraph-extract/src/links.rs 98.27% <98.27%> (ø)
crates/codegraph-extract/src/engine.rs 96.41% <96.87%> (+0.09%) ⬆️
crates/codegraph-watch/src/watcher.rs 94.58% <91.90%> (-0.69%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sunerpy
sunerpy merged commit a10ede3 into main Oct 1, 2026
10 checks passed
@sunerpy
sunerpy deleted the sync/v1.6.1-symlinks branch October 1, 2026 09:40
sunerpy added a commit that referenced this pull request Oct 1, 2026
Current alignment now says #286, #288 and #289 shipped in codegraph-rs v0.52.0, closing the v0.51.0 exceptions #770 and #1829/#1878; a new dated entry records the release run, digest and black-box acceptance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant