Indexing, caching & performance

The alien indexing works in a pretty simple manner - it simply shells out to a command that returns the list of files within a project. For version-controlled projects by default Projectile will use the VCS itself to obtain the list of files. As an example, here is the command that Projectile uses for Git projects:

For every supported VCS there’s a matching Projectile defcustom holding the command to invoke for it:

There are also two Git-specific commands for listing submodules and ignored files:

For non-VCS projects Projectile will invoke whatever is in projectile-generic-command. The default chooses fd when it’s installed and falls back to find:

By default, fd is also used inside Git repositories (instead of git ls-files), because git ls-files has the limitation that it lists deleted files until the deletions are staged. With fd, deleted files disappear from the listing immediately; with git ls-files, Projectile post-filters the listing against git ls-files -zd to hide deletions until they’re staged. You can control this with projectile-git-use-fd:

You can also customize the path to the fd executable and the arguments passed to it:

The hybrid method runs the same external command as alien and then post-processes the result with Projectile’s filtering rules. In other words, it’s alien plus a second pass that applies:

This is the right choice when your VCS lists more files than you want Projectile to surface (e.g. you want to drop *.elc even though they’re checked in), or when you rely on .projectile to keep/ignore subtrees inside an otherwise large repository. It’s slower than alien because of the extra pass, but it remains substantially faster than native for big projects since the file list itself is still produced by the external tool.

Not all Projectile features apply to every indexing method. The table below summarises which configuration knobs take effect under which mode:

Since indexing a big project is not exactly quick (especially in Emacs Lisp), Projectile supports caching of the project’s files. The caching is enabled by default whenever native indexing is enabled.

To enable caching unconditionally use this snippet of code:

At this point you can try out a Projectile command such as s-p f (M-x projectile-find-file RET).

Running C-u s-p f will invalidate the cache prior to prompting you for a file to jump to.

Pressing s-p z will add the currently visited file to the cache for current project. Generally files created outside Emacs will be added to the cache automatically the first time you open them.

Normally the cache lasts for the duration of your Emacs session. If you want to cache to persist between Emacs sessions you should set this option to 'persistent.

Now the project cache is persistent and will be preserved during Emacs restarts. Each project gets its own cache file, that will be placed in the root folder of the project. The name of the cache file is .projectile-cache.eld by default, but you can tweak it if you want to:

The cache file will be loaded automatically in memory the first time you trigger a "find file" operation for the project it belongs to.

You can purge an individual file from the cache with M-x projectile-purge-file-from-cache or an entire directory with M-x projectile-purge-dir-from-cache.

If the caches of many projects have gone stale at once (something you’d typically notice with commands operating on several projects, like projectile-find-file-in-known-projects), you can invalidate them all in one go with M-x projectile-invalidate-cache-all. Remote (TRAMP) projects are skipped to avoid a potentially slow connection round-trip per project; invalidate those individually with projectile-invalidate-cache.

When projectile-mode is enabled Projectile will auto-update the project cache when files within are added or deleted from within Emacs (via file hooks). This behavior can be disabled like this:

You can also set the project files cache to expire after a given number of seconds:

By default projectile-files-cache-expire is nil, meaning the cache never expires automatically.

One last thing - the project cache will be auto-invalidated if you’re using .projectile and its last modification time is more recent than the time at which the cache file was last updated.

The file hooks described above only see what happens inside Emacs. Files created, deleted or renamed by external tools (a git pull, a code generator, another editor) still leave the cache stale until you run projectile-invalidate-cache by hand. If you’d rather have Projectile notice such changes itself, it can watch the directories of your cached projects using filesystem notifications:

Whenever a project’s file list is cached, Projectile then registers a file-notify watch for each of the project’s directories and applies the resulting events to the cached list: new files are run through the usual ignore filtering before being added, deleted files (and directories) are removed, renames are handled as both. Events are debounced, so bursts (say, a branch switch touching hundreds of files) are folded into a single update. Anything that can’t be applied incrementally with confidence falls back to invalidating the project’s cache, which simply gets rebuilt on the next file listing.

Be aware of the trade-offs before enabling it:

Watches are dropped automatically when a project’s cache is invalidated (and re-armed when it’s next filled), when the option is disabled via Customize or setopt, when projectile-mode is turned off (turning the mode back on re-arms them for all cached projects), when a project is removed from the known projects and on Emacs shutdown.

The first "find file" in a large or remote project has to index it. With the alien and hybrid indexing methods Projectile does this without freezing Emacs: the indexing command runs asynchronously and is awaited in a way that keeps redisplay going and leaves C-g live, so you can abort a slow index (for instance one stuck on an unresponsive remote host) instead of staring at a frozen editor. The resulting file list is exactly what the synchronous indexer would produce.

This is on by default and controlled by projectile-async-indexing:

You can also warm the cache ahead of time, without blocking, using M-x projectile-index-project-async:

It runs the project’s indexing command via an asynchronous process and populates the same cache that a regular projectile-find-file reads, so a later "find file" finds the cache already warm instead of indexing on the spot. You could, for instance, warm a project right after switching to it:

The asynchronous building blocks it’s made of - projectile-files-via-ext-command-async and projectile-dir-files-alien-async - are public, so a streaming file finder (for example one built on top of consult) can drive Projectile’s indexing command asynchronously and feed results into its own UI. See the next section.

If you’d rather have your own asynchronous/streaming UI (à la fzf, consult or affe) list a project’s files instead of projectile-find-file, projectile-project-files-producer hands you everything you need to run Projectile’s own indexing command yourself:

Run :command in :directory and split its output on :separator. A minimal (non-streaming) example built on the public asynchronous runner:

A real streaming finder would instead consume the process output incrementally (filtering as you type) rather than collecting it. For git projects, note that :command lists the main worktree only; if you want byte-for-byte the same set as projectile-find-file (submodule files folded in, deleted-but-unstaged files removed) drive projectile-dir-files-alien-async instead.

Projectile does many file existence checks since that is how it identifies a project root. Normally this is fine, however in some situations the file system speed is much slower than usual and can make Emacs "freeze" for extended periods of time when opening files and browsing directories.

The most common example would be interfacing with remote systems using TRAMP/ssh. By default all remote file existence checks are cached.

To disable the remote file-exists cache, use this snippet of code:

To change the remote file exists cache expire to 10 minutes use this snippet of code:

You can also enable the cache for local file systems, that is normally not needed but possible: