libmonitor

[flatcap]

libmonitor will be a system-agnostic API for filesystem monitoring.

Currently, NeoMutt has some trivial Inotify (Linux) support.

Kqueue (BSD/macOS) isn't supported yet.

In the future, it might be possible to upgrade to using libuv (cross-platform).

1. Using Inotify
2. Current NeoMutt
3. Future NeoMutt

[More info coming soon]

[flatcap]

Using Inotify

inotifywait

Most distros have an inotify-tools package that contains the inotifywait tool.

# Watch the current directory
inotifywait -m .

Sample output, as subdirectory imap is read:

./ OPEN,ISDIR                imap
./ ACCESS,ISDIR              imap
./ CLOSE_NOWRITE,CLOSE,ISDIR imap

Kernel Config

The Inotify system exposes some config via /proc

Limits on my Fedora 38 box:

/proc/sys/fs/inotify/max_queued_events  16384
/proc/sys/fs/inotify/max_user_instances 128
/proc/sys/fs/inotify/max_user_watches   59459

They're also available using sysctl:

# Get values
sysctl fs.inotify.max_queued_events
sysctl fs.inotify.max_user_instances
sysctl fs.inotify.max_user_watches

# Temporarily set values (until reboot)
sysctl fs.inotify.max_user_watches=65536

To permanently change a value, create a file: /etc/sysctl.d/watches.conf containing:

fs.inotify.max_user_watches = 10485760

Programs using inotify expose info in /proc/pid/fdinfo

# The file descriptor of inotify, in process 495980, is 3
cat /proc/495980/fdinfo/3

The program is watching 3 files/dirs (watch descriptors 1-3):

pos:    0
flags:  04000
mnt_id: 15
ino:    1060
inotify wd:3 ino:f8ad03 sdev:25 mask:3f8 ignored_mask:0 fhandle-bytes:14 fhandle-type:4d f_handle:03adf80000000000030100000000000045340500
inotify wd:2 ino:f8ad02 sdev:25 mask:3f8 ignored_mask:0 fhandle-bytes:14 fhandle-type:4d f_handle:02adf80000000000030100000000000045340500
inotify wd:1 ino:f8ad01 sdev:25 mask:3f8 ignored_mask:0 fhandle-bytes:14 fhandle-type:4d f_handle:01adf80000000000030100000000000045340500

Sample Code

Branch [flatcap/monitor] contains an early work-in-progress.

It contains a new library, libmonitor and a sample program test-inotify.c

After building NeoMutt, build the test program with:

gcc -I. -o test-inotify test-inotify.c libmonitor.a libmutt.a -lpcre2-8

Then run the test program on some files:

./test-inotify ma*.c

You will have 5 seconds to do something in another window, e.g.

touch ma*.c

Sample output:

Add: mailcap.c
	watch: wd 1
Add: maillist.c
	watch: wd 2
Add: main.c
	watch: wd 3
file change(s) detected
ioctl expects: 144 bytes
read returns: 144 bytes
+ detail: wd=1 mask=0x20
EVENT on 0x13e96f0 (73,18205611)
	IN_OPEN 
Callback1: 0x13e92a0, wd 1, me 0x0020, 42
+ detail: wd=1 mask=0x4
EVENT on 0x13e96f0 (73,18205611)
	IN_ATTRIB 
Callback1: 0x13e92a0, wd 1, me 0x0004, 42
+ detail: wd=1 mask=0x8
EVENT on 0x13e96f0 (73,18205611)
	IN_CLOSE_WRITE 
Callback1: 0x13e92a0, wd 1, me 0x0008, 42
+ detail: wd=2 mask=0x20
EVENT on 0x13e9870 (73,18205621)
	IN_OPEN 
Callback2: 0x13e92a0, wd 2, me 0x0020, 42
+ detail: wd=2 mask=0x4
EVENT on 0x13e9870 (73,18205621)
	IN_ATTRIB 
Callback2: 0x13e92a0, wd 2, me 0x0004, 42
+ detail: wd=2 mask=0x8
EVENT on 0x13e9870 (73,18205621)
	IN_CLOSE_WRITE 
Callback2: 0x13e92a0, wd 2, me 0x0008, 42
+ detail: wd=3 mask=0x20
EVENT on 0x13e98a0 (73,19104442)
	IN_OPEN 
Callback3: 0x13e92a0, wd 3, me 0x0020, 42
+ detail: wd=3 mask=0x4
EVENT on 0x13e98a0 (73,19104442)
	IN_ATTRIB 
Callback3: 0x13e92a0, wd 3, me 0x0004, 42
+ detail: wd=3 mask=0x8
EVENT on 0x13e98a0 (73,19104442)
	IN_CLOSE_WRITE 
Callback3: 0x13e92a0, wd 3, me 0x0008, 42
ioctl expects: 0 bytes
read returns: -1 bytes
removing: wd 1
removing: wd 2
removing: wd 3

How it Works

The test program calls monitor_init() which calls inotify_init1() giving us a file descriptor (fd).

Next, it calls monitor_watch_file() on each command line argument.
We pass a callback function and some private data to this function.
monitor_watch_file() calls inotify_add_watch() giving us a watch descriptor (wd) for each.

Finally, it calls monitor_poll() which calls the standard unix command, poll() on the inotify fd.
Here, we use ioctl(FIONREAD) to query how much data is available.
Alternatively, we could have read a buffer load and repeated.

The data waiting on the fd is a set of struct inotify_event.
Once decoded, the callback function is called for each event.

Note: Each watch uses 1080 bytes of unswappable kernel memory

[flatcap]

Current NeoMutt

Currently, NeoMutt has some Inotify support.
It's poorly implemented and in the wrong place.

Watches are added in two places:

• When the mailboxes command is parsed
• When the Index is opened

NeoMutt watches Maildir's new/ directory.

Polling for changes happens deep within mutt_getch().
If it finds some, it modifies two global variables:

bool MonitorFilesChanged;   // Some monitored file/dir changed
bool MonitorContextChanged; // Current mailbox changed

When mutt_getch() returns, the Index can call mx_mbox_check() leading to maildir_check().
With almost zero information, all the Maildir can do is rescan the directories.

[flatcap]

Future NeoMutt

Overview

To improve NeoMutt, we want to:

• Create a system-agnostic monitoring library

  • Create a monitoring API
  • Create new constants to hide the implementation
  • Add Kqueue support for BSD/macOS

• Reduce dependencies

  • Inotify symbols/functions only used in monitor/inotify.c
  • Kqueue symbols/functions only used in monitor/kqueue.c
  • Monitor symbols/functions only used in the Maildir / Mbox backends

• Communicate everything we can to the backend

  • The monitor knows exactly which file/dir changed, so tell the backend
  • The backend needs to efficiently integrate the changes

• Update the GUI efficiently

  • The backend needs to notify the GUI of the exact changes,
    e.g. Two new emails
  • The GUI needs to efficiently integrate the change,
    e.g. Insert the new emails into the thread tree
    (without disturbing anything else)

[roccoblues]

The kqueue monitor requires a file descriptor to be opened for every file being watched. Monitoring directories only is not enough because the callback gets no information about the changed files. That makes it unsuitable for our usecase.

Here's a good overview of the different monitors and limitations: https://github.com/emcrisostomo/fswatch#limitations.