Any way to restrict/disable file completion for slow filesystems? (fuse, etc)

[yump]

So, tab completion is great most of the time, when poking around the filesystem is very cheap, as Linux's VFS cache usually ensures it is. However, sometimes I encounter conditions where it is not cheap. These include:

1. Sshfs, and maybe other network filesystems, especially over wi-fi, especially when the host on the other end is an Android device that likes to go into high-latency low power modes.
2. Fuse-mounted borg backups. Each archive within a repository is presented as a subdirectory of the mount point, and looking inside too many of them causes large memory usage.
3. Mountpoints or cross-filesystem symlinks that require spinning up a hard drive to look on the other end. Takes about 10 seconds and the HDD will make an annoying sound for the next half hour or so until it times out again.

In these cases, pressing tab a few times to refine a filename completion can cause fish to make syscalls that incur high costs.

Ideally, there'd be a way to stop the file completion looking deeper than the current directory unless the last character is a path separator. It should probably be a user defined function to query whether completing through a particular path is allowed, because of the many and varied things that might make completion ill-advised.

This is similar to issue #4402, but my concern is about the performance implications of deep completion rather than the presentation.

P.S. The borg backup thing might be caused by PR #7586.

[faho]

Any way to restrict/disable file completion for slow filesystems?

There is not, no.

Ideally, there'd be a way to stop the file completion looking deeper than the current directory unless the last character is a path separator

It'll only look deeper for cd, in order to suggest a unique hierarchy - cd foo/bar/baz in case foo/ and foo/bar/ are otherwise empty. Once it finds a directory that has something else than exactly one directory in it it'll stop

In other cases, the normal file completions will only look at one component without recursion.

P.S. The borg backup thing might be caused by PR #7586.

It probably isn't, because that's about what happens when you run cd foo/. It'll just try to chdir() instead of trying a stat() first - so it would probably improve performance instead of making it worse because, if it works, it'll now only do one syscall instead of two. It has nothing to do with completions.

Ideally, there'd be a way to stop the file completion looking deeper than the current directory unless the last character is a path separator

Frankly: This isn't the sort of customization we do. It's a weird knob for extremely special cases that most people won't need, and even those who need it would probably not find it. The general solution for this is #23 - simply making completions not block.

[faho]

Here's what we can do:

We can optimize this. We can in fact skip the stat/open if the system supports d_type, so we get all the info from readdir().

I have a branch, I'm going to ask you to check if it improves things. I tested it and it cut down the stat calls by 1000 (i.e. no stat) on a tmpfs.

Here: https://github.com/faho/fish-shell/tree/keep-dir-information

(not ready to merge, there's a failing test regarding a symlink - turns out we just can't skip non-dirs earlier because of logical paths)

[yump]

Using your branch, I don't see the directory opening happening on sshfs, but it still does happen in tmpfs and btrfs. However, I did find two weird things:

1. The test in the commit message, complete -C "cd /tmp/completions_test/" doesn't reproduce the problem, even when run at the interactive prompt. The directory opening only happens, as far as I can tell, when the tab key is pressed.

2. strace -C seems to misreport the amount of time each syscall takes, compared to the full dump. With an sshfs mount over sufficiently slow wifi, I can literally watch the syscalls scroll by.

I published what I've collected so far. Particularly, not the relative timings in strace_wifi.txt. Each openat() takes tens of ms, but this doesn't show up in strace -C at all.

[faho]

the packaged version (3.5.0)

Oh, you're on Fedora? (they've neglected to package version 3.5.1 for some reason)

The test in the commit message, complete -C "cd /tmp/completions_test/" doesn't reproduce the problem

The exact command is complete -C"cd /tmp/completion_test/". No space after the -C and no "s" after "completion".

Check that you used the correct command and path.

the patched branch seems possibly flakier

That's unlikely, and the sort of observation that really needs to be backed by data.

Each openat() takes tens of ms, but this doesn't show up in strace -C at all.

I get almost no openat() even with 3.5.1 or master before the patch. In particular I do not get openat() of each of the directories. Check your config.

In particular, $CDPATH would be of interest.

[faho]

Ah, I figured out where the openat() occurs - it's the autosuggestions, which do, in fact, descend into all these directories.

However: That happens on a background thread, and doesn't slow down fish itself. Autosuggestions are cancelled when they are no longer relevant to the commandline at hand or after 500ms, but if they don't finish they don't finish, and you can just go on. You can add a usleep() there and you'll see it doesn't actually block anything.

We don't propagate the timeout to the descending logic, so it'll keep running even once the timeout has been reached, and will only really be stopped once the commandline changes. But given that fish keeps trucking on, that doesn't seem to be causing problems with the completions being slow.

@ridiculousfish would it make sense to stop the autosuggestion performer once the timeout is reached? It appears we're not doing that at the moment (or at least simply inserting a usleep() in wildcard::descend_unique_hierarchy doesn't appear to trigger it)

[yump]

Yeah, I went down a rabbit hole with gdb and traced the difference between interactive and complete -C to expand_flag::special_for_cd_autosuggestion, but I wanted to reduce it to a minimal example and learn more before saying anything. Sorry for any duplicate work that may have resulted from my lack of communication.

If this behavior had been different in 3.5.1, It wouldn't have been the first time I was bitten by an older version packaged in Fedora. That in mind, I've emailed the [email protected] about the new version, and set github to notify me about new fish releases, so if necessary I can ping them again next time.

However: That happens on a background thread, and doesn't slow down fish itself.

Fair, but should the effect on fish itself be the only concern? A non-cached read from a spun-down HDD still spins it up again, and opening a borg archive in FUSE causes it to populate the file index for that archive in memory. It is, of course, possible to work around that and avoid OOM situations by passing --last 5 or somesuch to borg mount, at the cost of not being able to see the entire backup history at once.

Thanks for helping me troubleshoot this, and for your great work on fish in general.