fix: Reduce inotify watch count via gitignore-aware directory walking on Linux#12398
Open
antoinelyset wants to merge 1 commit intovercel:mainfrom
Open
fix: Reduce inotify watch count via gitignore-aware directory walking on Linux#12398antoinelyset wants to merge 1 commit intovercel:mainfrom
antoinelyset wants to merge 1 commit intovercel:mainfrom
Conversation
Contributor
|
@antoinelyset is attempting to deploy a commit to the Vercel Team on Vercel. A member of the Team first needs to authorize it. |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| .git_ignore(true) | ||
| .git_global(false) // skip ~/.gitignore — not predictable in CI | ||
| .follow_links(false) | ||
| .filter_entry(|e| e.file_name() != OsStr::new(".git")) |
Contributor
There was a problem hiding this comment.
WalkBuilder defaults require_git to true, causing .gitignore files to be silently ignored when no .git directory is found in the walk root or ancestors.
… on Linux On Linux, turbo's file watcher creates one inotify watch per directory using recursive directory walking. In monorepos with large node_modules, this easily exceeds the OS inotify limit (e.g. GKE default of 12,288), causing `turbo watch` to fail with "file watcher failed to start". Replace `walkdir::WalkDir` with `ignore::WalkBuilder` in the Linux-only `manually_add_recursive_watches()` so that gitignored directories (primarily node_modules) are skipped during inotify watch setup. This reduces the watch count from ~13k+ to just the project's own directories. Also adds `ensure_watched_up_to_root()` to guarantee the cookie directory (.turbo/cookies) gets inotify watches even when .turbo is gitignored. Co-Authored-By: Claude Opus 4.6 <[email protected]>
antoinelyset
force-pushed
the
fix/gitignore-aware-inotify-watches
branch
from
1a7024f to
6354e63
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hello Turborepo! I ran into a limitation of watch mode on Linux. In our monorepo,
node_modulescreates enough directories to exceed the OS inotify limit (e.g. GKE default of 12,288), causingturbo watchto fail withERROR file watcher failed to start.I gave it a try with the help of Claude Code — I didn't let it code blindly, I guided it carefully, used TDD, and built a Docker-based reproduction to verify the fix end-to-end.
Known limitation — looking for guidance
In the end it's not done because I discovered that using
.gitignoreto filter directories might be too broad.Skipping all gitignored directories means turbo loses inotify watches on output directories like
out/anddist/— which are typically gitignored but still need to be watched.hash_watcherrelies on inotify events to detect when output files change and recompute hashes. Without those watches, hashes go stale. This causes 8 test failures on Linux:globwatchertests (timeout — no events from gitignored output dirs)hash_watchertests (test_negative_inputs,test_inputs_with_turbo_defaults— hashes not updated because modification events fromdist//out/never arrive)Possible approaches — would love your input:
node_modulesskip — Only skipnode_modules(and.git) instead of relying on.gitignore..gitignorefor the initial skip, then add watches back for directories that match taskoutputsfromturbo.json..gitignore.What this does
Replaces
WalkDirwithignore::WalkBuilder(.git_ignore(true)) in the Linux-onlymanually_add_recursive_watches(), so gitignored directories are skipped during inotify watch registration.Current Design choices
hash_watcherunchanged.ensure_watched_up_to_root()ensures.turbo/cookiesgets watches even when.turbois gitignored.Reproduction
ERROR file watcher failed to startRepro repo: antoinelyset/turbo-inotify-repro