Why do I need to set -U XDG_DATA_DIRS /usr/local/share in order to get fish to pick up locally installed functions?

[ioquatix]

I'm currently maintaining a fork of https://github.com/ioquatix/chruby-fish and I'm having trouble figuring out what is the correct way to handle the functions that get installed by this package.

This package installs functions by default into /usr/local/share/fish/vendor_functions.d and /usr/local/share/fish/vendor_conf.d.

On my macOS system, it doesn't seem to pick these paths up automatically, but according to what I read in the documentation it should.

In order to pick up these scripts, I need to do this:

set -U XDG_DATA_DIRS /usr/local/share

This feels a bit odd to me. Am I doing the right thing here?

[kidonng]

How did you install fish? What is the content of $__fish_data_dir/__fish_build_paths.fish on your machine? The content of that file is used to load the vendored scripts and functions. If you are using Homebrew it should automatically pick up stuff under Homebrew's prefix.

My best bet is that you are using a Apple Silicon machine, where Homebrew will use /opt/homebrew as its prefix, and hence stuff under /usr/local will not be loaded automatically.

Regardless of the cause of the problem, you can use $XDG_DATA_DIRS. It is there for a reason.

[ioquatix]

I am using MacPorts but also confirmed this can be an issue on Homebrew.

The contents of $__fish_data_dir/__fish_build_paths.fish:

> cat $__fish_data_dir/__fish_build_paths.fish
# This file was generated from a corresponding .in file. DO NOT MANUALLY EDIT THIS FILE!
# This function is generated by the autotools build to "compile in"
# the local "vendor" completions, functions and configuration conf.
# It is sourced by share/config.fish, if it exists.

set __extra_completionsdir /opt/local/share/fish/vendor_completions.d
set __extra_functionsdir /opt/local/share/fish/vendor_functions.d
set __extra_confdir /opt/local/share/fish/vendor_conf.d

The prefix for MacPorts would be /opt/local.

Setting XDG_DATA_DIRS to /usr/local/share worked correctly, so if that's the right solution I'll continue to do it. I'm not sure are there any other implications of setting that variable?

[page-down]

When XDG_DATA_DIRS is empty, many software will fall back to the default path of the system or package manager, which, depending on your situation, will probably contain /opt/local/share.

When you set this environment variable, other software that relies on it will not work properly.

If you only expect fish to work properly and you know that other software does not rely on the environment variable at all, then you can set it that way.

Otherwise, you may want to set it to /opt/local/share:/usr/local/share:/usr/share. This depends on your system and environment, the above paths is just a guess based on your description.

And for files with the same name fish will load only the first one in order, so you have to make sure that the path priority is correct and that fish works properly + other software works properly.

[ioquatix]

What you've suggested makes sense. It seems unreasonable for fish to know about all the "standard" prefixes when one has chosen to install it into a specific prefix, in this case /opt/local. It seems totally reasonable to me it shouldn't consider /usr/local unless it was manually added to that list.

I'm not sure if there is a better way to do this than XDG_DATA_DIRS. It might make sense for us to have some way to add search paths/roots without impacting other parts of the system.

[krobelus]

Yeah I think XDG_DATA_DIRS is the right knob. It's not that much different from PATH. If you have /usr/local/bin to your PATH, adding /usr/loca/share to XDG_DATA_DIRS makes sense.

[page-down]

The problem with XDG_DATA_DIRS is the dependence of other software on its fallback value when it is empty.

• The software will use its own fallback value when its value is empty, which will ensure that the data that comes with the software is loaded correctly.
• The fallback value of XDG_DATA_DIRS is decided by the software or distribution when it is packaged.
• When a user installs different package managers and uses software from both package managers, it is likely that there are multiple locations of data dirs.
• There are environments where the value is empty: for example, macOS does not have this environment variable by default, and in some user environments on Linux systems it is not set.
• The user needs to know all the locations that will be used and set the full path correctly, not just append the additional fish vendored_conf.d paths that are needed.

Example:

• The user uses package manager X . XDG_DATA_DIRS falls back to /usr/local/share by default.
• And also package manager Y, All software in this package manager XDG_DATA_DIRS falls back to /opt/local/share by default.
• The user installs Software A via X, B via Y.
• The user compiles Software K from source and install into ~/.local , which provides the fish-related ~/.local/K.app/fish/vendored_completions.d.
• At this point the user wants to use the K, A, B software normally in macOS (XDG_DATA_DIRS is empty by default and relies entirely on fallback values).
• The user cannot simply add ~/.local/K.app/share to XDG_DATA_DIRS, he has to understand the situation and add all the paths: ~/.local/K.app/share:/opt/local/share:/usr/local/share:/usr/share.

And when the developer of software K wants to add its path to XDG_DATA_DIRS and let the fish load additional files, it can't be done.

• The software K does not have permission and does not expect to install to the system global, so it can't be installed to /usr/local/share/fish/vendored_completions.d.
• The software K does not know the paths corresponding to the user's package manager X and Y, so it cannot provide the full paths either.

The current workaround is to provide a single path to XDG_DATA_DIRS and then clean up after fish starts, making sure that environment variable is empty (or the original env value) under macOS. So at this point fish loads ~/.local/K.app/share/fish/vendored_completions.d from K, and also makes sure that A and B are fully functional (fallback to their default paths).

This issue was recently encountered while working with kitty terminal shell integration.

This problem could be solved if fish provided separate environment variables to load directories. Do you have any better suggestions for this? Thank you.

[krobelus]

On Sat, Nov 27, 2021 at 05:53:02AM -0800, page-down wrote: The problem with `XDG_DATA_DIRS` is the dependence of other software on its fallback value when it is empty. - The software will use its own fallback value when its value is empty, which will ensure that the data that comes with the software is loaded correctly. - The fallback value of `XDG_DATA_DIRS` is decided by the software or distribution when it is packaged.

I don't think applications or distributions are supposed to come up with random fallback values. The spec [1] says they "should use /usr/local/share/:/usr/share" [1]: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

- When a user installs different package managers and uses software from both package managers, it is likely that there are multiple locations of data dirs. - There are environments where the value is empty: for example, macOS does not have this environment variable by default, and in some user environments on Linux systems it is not set.

I guess because a fresh system doesn't have a ~/.local/share ? but as soon as you install something there you might want to add it to XDG_DATA_DIRS.

- The user needs to know all the locations that will be used and set the full path correctly, not just append the additional `fish vendored_conf.d paths` that are needed. Example: - The user uses package manager X . `XDG_DATA_DIRS` falls back to `/usr/local/share` by default. - And also package manager Y, All software in this package manager `XDG_DATA_DIRS` falls back to `/opt/local/share` by default. - The user installs Software A via X, B via Y. - The user compiles Software K from source and install into `~/.local` , which provides the fish-related `~/.local/K.app/fish/vendored_completions.d`. - At this point the user wants to use the K, A, B software normally in macOS (`XDG_DATA_DIRS` is empty by default and relies entirely on fallback values). - The user cannot simply add `~/.local/K.app/share` to `XDG_DATA_DIRS`, he has to understand the situation and add all the paths: `~/.local/K.app/share:/opt/local/share:/usr/local/share:/usr/share`.

Yes. I don't see the problem here. That's what happens with flatpak on Linux (the variables are automatically added by flatpak's profile scripts).

And when the developer of software K wants to add its path to `XDG_DATA_DIRS` and let the fish load additional files, it can't be done.

applications shouldn't write XDG_DATA_DIRS, only read them. The OS/package manager/user is supposed to set that.

- The software K does not have permission and does not expect to install to the system global, so it can't be installed to `/usr/local/share/fish/vendored_completions.d`.

Shouldn't it rather install to $PREFIX/share/fish/vendored_completions.d where $PREFIX is ~/.local in your example? Note that there is also "pkgconf fish --variable completionsdir" but that's usually for things that are installed system-wide.

- The software K does not know the paths corresponding to the user's package manager X and Y, so it cannot provide the full paths either. The current workaround is to provide a single path to `XDG_DATA_DIRS` and then clean up after fish starts, making sure that environment variable is empty (or the original env value) under macOS. So at this point fish loads `~/.local/K.app/share/fish/vendored_completions.d` from K, and also makes sure that A and B are fully functional (fallback to their default paths).

This sounds like you are installing to $XDG_DATA_DIRS[1]. I don't think you should. I believe this variable is purely a runtime configuration variable (e.g. the user has full control over what happens even after everything is compiled & installed).

This issue was recently encountered while working with kitty terminal shell integration. This problem could be solved if fish provided separate environment variables to load directories. Do you have any better suggestions for this? Thank you.

I don't think I have understood the complete problem yet, so bear with me. These issues are super tricky, maybe we can simplify it somehow.

[page-down]

I apologize for not being able to describe it very well. Thank you for taking the time to reply.

related with: https://github.com/kovidgoyal/kitty
Disclaimer: I am only one of the participants. I only stand for the developer myself and not for the project.

In short: how to make a terminal run fish while letting
fish to execute integration scripts (in order to output escape codes when outputting prompt via event func) and get completion without writing to user files when running.

Due to the limited time and effort of the maintainer, the official release of the project only provides compiled tarball, which is expected to work wherever the user extracts it.

Initially, the integration and completion script is soft-linked at runtime to ~/.config/fish/completions/ etc. before running fish. However, this can be intrusive to the user configuration directory.
This was later changed to: provide the configuration and completion scripts contributed by this software (vendored) via XDG_DATA_DIRS before running fish. Then it was found that when that environment variable was empty, various situations would occur.

The spec [1] says they "should use /usr/local/share/:/usr/share"

That was my initial thought too, but it doesn't apply to reality. Compromises need to be made.
As far as I know, some package managers are installed to /opt, not all are /usr/local, /usr. We need to achieve maximum compatibility. We have to make sure that everything is fine even when the user is running the software in these locations.

... add all the paths ...

Yes. I don't see the problem here. That's what happens with flatpak on Linux ...

They are able to do so because they stand on the system and software distribution side. On the developer's side, there is no effective way to know all the system and pkg manager's path and build the correct fallback path. We want to bring the best out-of-the-box experience for users, without the need for manual user configuration.

applications shouldn't write XDG_DATA_DIRS

In general yes, but in this scenario it might be understandable for the program to provide the required script (vendored completions, .etc.) for fish via XDG_DATA_DIRS. It is also self-contained without the need to use container packaging and does not require writing to other user directory locations.

Shouldn't it rather install to $PREFIX/share ...
This sounds like you are installing to $XDG_DATA_DIRS ... the user has full control over ...

This is a runtime variable that the user has full control over, the software simply adds its own vendored script path to this variable before running fish, which is the main purpose of this variable.

This seems very straightforward, however in real life scenarios it is not as simple as appending a path.

This issue originated from:
kovidgoyal/kitty#3948 (comment)

Some subsequent discussions are continued in the following PR:
kovidgoyal/kitty#4199

The current final implementation is: Add itself's vendored path to XDG_DATA_DIRS before running fish, then restore XDG_DATA_DIRS when fish runs the prompt for the first time. (empty value, or user-specified value)

As you said

These issues are super tricky, maybe we can simplify it somehow.

This is just to discuss whether a better way can be provided to dynamically run initialization configuration scripts via environment variable. It would be very simple and straightforward to provide a fish-exclusive environment variable to solve this problem. Kind of like zsh ZDOTDIR, but for the additional fish vendored conf.d, compl.d.

[zanchey]

Then you're looking at adding support for multiple ZDOTDIRs, as kitty will presumably not be the only software that wants to use it! Which just leads us back to a fish-specific version of XDG_DATA_DIRS.

I really think XDG_DATA_DIRS is the right control for this. I don't think kitty should add itself, then remove itself - that will lead to hard-to-debug failure cases.

iTerm2 manages this by having an "install shell integration" which does some magic to config.fish on a per-user basis. I don't love that, but it seems to be fairly robust.

[page-down]

... Which just leads us back to a fish-specific version of XDG_DATA_DIRS.

Only a separate FISH_VENDORED_DIRS can avoid conflicts with other software's dependent on XDG_DATA_DIRS, especially on platforms that default to empty (unset), such as macOS.

I really think XDG_DATA_DIRS is the right control for this. I don't think kitty should add itself, then remove itself - that will lead to hard-to-debug failure cases.

Spec-wise, this is the most correct environment variable. This is indeed a hacky workaround for now, which requires better integration support from fish.

iTerm2 manages this by having an "install shell integration" which does some magic to config.fish on a per-user basis. I don't love that, but it seems to be fairly robust.

This is how it was done before.
Not the best practice since it modifies user files. Better to just load it on demand, depending on the environment it is running in -- running in kitty -- to load the appropriate features. Users can manually configure this into config.fish or conf.d if they want.

[kovidgoyal]

Let me add a little clarity. Terminal emulators like kitty or iTerm2 have features that a shell can make use of to improve the user experience. Sadly shell developers have shown no interest in implementing the integration for these features, for years now, so they languish undiscovered by most users in the ecosystem.

In lieu of shell developers being willing implement the features themselves, terminals are left with the only option being, to inject some code into the shells to enable the integration needed for these features. To make this strategy work it would be good is shell developers at least provided some robust way for the program invoking the shell to inject code into the shell (aka source a file) as early in the shell's startup as possible.

There are two techniques that can be used for this. Either a command line argument or a environment variable. Since the terminal is responsible for invoking the shell, it can control either.

A well designed environment variable for this purpose would behave as follows:

1. Be of the form BOOTSTRAP_SHELL_<shell name>=when:/path/to/file/to/source. Here when should be either start or end indicating the file should be sourced before any other files or after
2. The env var should be documented as intended for use only by programs hosting the shell, such as terminal emulators. It should in particular be undefined behavior for user or system shell rc files to manipulate this env var in anyway
3. If the file specified does not exist it should be ignored (maybe log the error somewhere)

A command line argument has the disadvantage of not being passed to sub shells, terminal multiplxers, etc.

[zanchey]

Sadly shell developers have shown no interest in implementing the integration for these features

I disagree with your framing but will agree that relatively few features have been merged across the major command-line shells (fish being considered major is certainly debatable).

I can see the merits of an environment variable, noting that fish already supports this with a command-line argument (-C).