org-agenda-files-track

Fine-track org-agenda-files to speed-up org-agenda.

When an agenda buffer is built, Emacs visits each file listed in org-agenda-files. In case your tasks or events are recorded in an ever-extending journal and/or roam directories, org-agenda can become sluggish.

This package aims to dynamically update the org-agenda-files variable by appending/deleting a candidate org file when it is saved. This limits the number of files to visit when building the agenda. The agenda buffer thus builds faster.

Two different packages are provided :

• org-agenda-files-track uses the native org-element API. In this case, candidate selection is governed by the org-agenda-files-track-predicate.
• org-agenda-files-track-ql uses the org-ql backend. In this case, candidate selection logic is extracted from org-agenda-custom-commands and org-ql-views. Once the cache of org-ql is built, using org-ql-view commands instead of org-agenda should additionally speed-up approximately 5 times. See org-ql’s readme for details.

Configuration

org-agenda-files-track

You should redefine the org-agenda-files-track-predicate to align with your desired org-agenda-files . Typically, this involves matching content compatible with org-agenda and org-agenda-custom-commands. Test the predicate using pp-eval-expression on (org-agenda-files-track-file-p) with an opened candidate org file.

org-agenda-files-track-ql

If you want to continue using org-agenda alongside org-ql, rather than org-ql-view, you should (setq org-agenda-include-diary nil) and use pure org-ql-block commands in your org-agenda-custom-commands. With org-ql integration, there’s no need for explicit predicate definition.

Initially defining or redefining org-agenda-files

Redefining org-agenda-files is probably necessary when org-agenda-files-predicate or org-agenda-custom-commands is changed. One simple method is to record all possible org files and then cleanup which files are agenda-files. This is the part that will take time. For instance:

(defun my/org-agenda-files-track-init ()
  "(Re)initialize dynamic agenda files.

This can take a long time, so it is recommended to run this only
on installation and when first tasks are added to many files via
methods the save hook cannot detect, like file synchronization."
  (interactive)
  ;; ;; uncomment if storing org-agenda-files in file
  ;; (make-empty-file org-agenda-files 'force)
  (org-store-new-agenda-file-list
   (directory-files-recursively
    org-directory (rx ".org" eos) nil
    ;; ignore hidden directories like .git and .attach
    (lambda (subdir)
      (not (eq ?. (string-to-char (file-name-nondirectory subdir)))))))
  ;; use ql here if desired
  (org-agenda-files-track-cleanup-files 'full)
  (message "Initialized agenda files"))

Both values of org-agenda-files (a file path or a list of files) are supported, since we use a native org-agenda saving method. You will likely want to set org-agenda-files to a single filename to persist org-agenda-files across Emacs sessions:

(setq org-agenda-files
      (expand-file-name "org-agenda-files.txt" (xdg-cache-home)))

Changelog

It’s only getting started!

MAYBE TODO: redefine org-agenda-files-track-predicate to match all possible org-agenda commands, current default config is quite minimalist. Maybe that could be done automatically by parsing org-agenda-custom-commands and org-agenda, but then it amounts to re-create a subset of org-ql.

MAYBE TODO: investigate integration with org-super-agenda.

MAYBE TODO: investigate integration with consult-org-agenda.

Relation to other files / packages

The original source of inspiration for this package is the post Task management with org-roam Vol. 5: Dynamic and fast agenda by Boris Buliga. But since I’d like to use my org-roam files for knowledge management and keep task management separate, I decided to make a package out of it to build it.

In the process of writing the package, I discovered Armin Darvish’s implementation which basically does the same thing, on which I added a few improvements (namely: native save, org-ql integration, more general utility functions).

This package functions both on builtin functionality and with org-ql clarity and speed improvements.

At some point, this package family prefix was org-dynamic-agenda but settled for org-agenda-files-track for clarity.

Finally, the same package could be done with org-roam and org-roam-ql (with some more work, could be faster (you might not need more speed than here though) since you can read from the database without having to parse files), but it mingles knowledge and task management.