Installing a Rust release

[zanchey]

This issue is to discuss how to drive the installation process of the rewritten-in-rust fish.

cargo install is not suitable for a complete, working fish install. The following tasks are required:

• Generate documentation
• Install binaries, completions, functions, tools, documentation, translations, and the pkgconfig file
• Support installation using both prefix and DESTDIR options

It may also:

• Generate source tarballs
• Build Mac apps and installers

Some options:

• Do nothing. Teach fish to run from the cargo build directory, publish to crates.rs, and leave everything else up to distributors (I do not like this option)
• Continue to use CMake. All the logic is already there, and Corrosion may not be necessary. The actual CMake parts can be slimmed down significantly.
• Write a small Makefile. Portable and well-understood, but may end up requiring a configure step.
• Autotools strikes back: if we're going to need a configure step beyond what cargo provides, should we just reach back to our old friends automake/autoconf? (I would prefer to stick with CMake)
• Write all the tasks in cargo xtask patterns. Major downside is that there's a lot of things that Make/CMake provide for "free" that would need re-implementation in Rust.

https://github.com/axodotdev/cargo-dist is cool, but does not really solve the problem of making a source tarball that can then be built by distrbutions.

[Conan-Kudo]

As a package maintainer of fish in Fedora and an ardent user of fish shell, I would prefer that you keep using CMake. It's straightforward and already does what you need, and works well with packaging and shipping fish in distributions.

[zanchey]

Regrettably, there will be some work required for distributors; Fedora probably has it easiest as the imperative nature of RPM scripts means that including the macros that handle build dependencies and so on is straightforward. But at a minimum there will need to be additions made to handle the requirements of cargo being online and package builders being offline.

CMake does not support much in the way of Rust by default; Corrosion is helping but requires either vendoring or online access to work properly and the method of getting CMake variables into the Rust build seems fragile to me.

[Conan-Kudo]

Fedora has a method of being able to tell Cargo to use a local index of offline crates packaged as RPMs, so that's not an issue.

[cdluminate]

I maintain fish for Debian family (including Ubuntu etc). Feel free to at me here on GitHub, or reach out throught email. I'll have to check debcargo and get back to you soon.

[faho]

It's probably worth looking at an example of how rust programs are packaged today:

https://gitlab.archlinux.org/archlinux/packaging/packages/ripgrep/-/blob/main/PKGBUILD

This will

• Use the github-generated tarball
• run cargo --locked so it uses the exact deps from the lockfile
• install the binary by hand
• mkdir the required directory structure by hand
• Run rg --generate to get completions and the one man page by hand, into the correct place
• install the license files by hand

The package manager will then tar all that up and add the metadata to make a package, which it will then, on install, extract.

For fish, a similar procedure would involve roughly, as far as I can tell:

• run cargo build
• run sphinx-build (with trickery to get it to use the newly built fish_indent)
• run a script to create __fish_build_paths, pkgconfig and other generated files
• install all the files into the necessary directory structure.

Note that none of this would be done simply with cargo install, and so fish on crates.io won't be a thing. Doing it would lead to a semi-functional and incomplete fish.

This procedure seems pretty involved. It might be possible to create a script or xtask thing, but I'm not sure how well that works and how many people will try cargo install and get a broken install.

[zanchey]

Continuing to use CMake is looking increasingly favourable. It would be worth looking into whether all of Corrosion is needed, or whether CMake could be taught to drive cargo without too much trouble.

I don't think there's much to be gained from rewriting fish_test_helper from C++ to Rust, so work on this can start.

[mqudsi]

I think trying to (force a) move from CMake right now isn’t necessary - there’s no rush to move away from a dependency we already have (and have already sorted out for release management purposes with the various package managers and distros). But, purely theoretically speaking, if we’re able to move all compilation-related stuff from CMake to build.rs over time (getting rid of corrosion in the process), we absolutely could get away with just a Makefile and build.rs without needing autotools or any other stuff that drives people away from Makefiles to CMake or whatever else in the first place (i.e. just Make as a cross-platform, lightweight, always available; familiar, and distro-friendly task runner rather than an actual build system).

In that context, I don’t see the benefit of xtask over a Makefile (and see other drawbacks).

[Be-ing]

It would be worth looking into whether all of Corrosion is needed, or whether CMake could be taught to drive cargo without too much trouble.

Corrosion handles a lot for you that you probably don't want to bother with, for example integrating how CMake handles cross compilation with how Cargo handles it. I'd caution against calling Cargo directly from CMake without Corrosion.

[Be-ing]

just a Makefile and build.rs

There's a major limitation with build.rs that you'd bump into: there's intentionally no way for build.rs to say "generate this file and put it here" and have an external tool know where "here" is. There's a way to hack around this though: have an external tool (Make, CMake, a shell script, or whatever) set an environment variable telling build.rs where to put generated files when calling cargo build and read that environment variable from build.rs.

[matklad]

If I may chime in:

• As far as I am aware, there isn't a standard pattern for installing Cargo-build applications into --prefix the UNIX way, so some amount of "do it yourself" is required
• This creates a problem --- a packager would probably prefer to avoid learning project-specific entry point for installation
• So, driving the installation process through standard "make install" (or a CMake equivalent) is beneficial for packagers
• At the same time, CMake&make are extra dependencies, which are not exactly lightweight --- CMake is ugly (that's aesthetics argument, but it's hard to argue with :), and Make is not entirely cross-platform (GNUMake vs BSDMake, dependency on host's /bin/sh, Windows exists). Though, for UNIX environment (and fish is a UNIX application), this doesn't matter all that much.
• Rust is a general purpose programming language. Writing "shell-like" glue code to run this tool, than that tool, then copying files from here to there works fine with Rust. And Rust is more elegant and cross platform than CMake/make. So, writing installation script in Rust is also an option.

My personal decision tree here:

• If I don't care much about "improving" installation, and don't want my packagers to spend time upgrading their packaging recopies, I'd stick with CMake with exactly same interface as today, except pruned down to just installation task.
• If I want lean installation with few dependencies, and only care about UNIX, I'd write a Makefile to orchestrate installation process.
• If I am a Rust zealot who just hates accidental dependencies on UNIX environment which make my software less portable, I'd:
  • write installation script in Rust
    • so that it is invoked as cargo run --bin install-script or some such
    • and installs all the files into a local directory, using ./out as a prefix: ./out/bin/fish, ./out/man/man1/fish.1, ./out/lib, etc
  • provide a thin Makefile which runs cargo run --bin install-script to do all the heavy lifting, and then sudo cp the resulting files into user-specified prefix (global /usr by default).

---

Additional thing here is that, as far as I understand, fish also depends on native libraries (pcre & ncurses). I think the biggest benefit that CMake provides is actually finding those libraries and allowing the packager to substitute them in a nice way. Cargo, through custom build.rs, of course allows the packager to control these as well, but there isn't a declarative, standard way to do so. Eg, different rust projects might end up using different mechanisms for customization, for example, input environment variables vs pkg-config.

[faho]

I broadly agree with your conclusion (specifically the first), but I think I disagree in some of the details.

I know you've done a bit to popularise the "xtask" pattern, where you add another rust binary that is run via cargo to do stuff, and that's nice for "simpler" projects, but I'm afraid the complexity we would need there makes my head spin.

E.g. zellij has what I believe classifies as a "big" xtask, and they copy three asset files, one of which is a man page. For comparison we have about the same amount of lines of .fish as .rs, and we have a quarter of that in docs that we want as both man pages and html, plus auxilliary files like pkg-config.

I don't see any handling of things like DESTDIR/PREFIX/DATADIR (which fish uses at runtime to find out where the functions and stuff are). I have no idea how cross-compilation would work, especially when you have e.g. the docs depending on a working fish_indent on the generating system (tho to be fair I don't really know how we handle that today). They also end up shelling out in a bunch of places anyway, which... well maybe I'm a shellscript snob, but that always felt nicer in a shellscript to me.

And Rust is more elegant and cross platform than CMake/make

Rust is more elegant generally than CMake, but CMake is the better build tool. And cmake is cross-platform enough that it runs everywhere we currently support fish, and we're not planning on supporting Windows.

And it being "ugly" doesn't matter so much when most of the work writing any cmake script is already done.

I'd stick with CMake with exactly same interface as today, except pruned down to just installation task.

That's quickly looking better and better. I think we can get to a point where you can cargo build and get a working fish for testing (I have a WIP branch on my computer), but you run cmake to get it all packaged up and generate the docs and such. I'd like to get the tests, including our script-integration and interactive tests hooked up to cargo as well.

I don't see a winning argument for switching back to makefiles - the cmake dependency isn't a problem in practice (especially not for distributions) and we switched for a reason. If anyone has a compelling argument I would like to hear it, and what make features specifically we would want. Part of would prefer writing a shellscript over a makefile but that's probably unpopular.

And of course: If something happens in cargo or as an extension to it, we can always revisit this decision and pick a nicer, rustier, build tool.

[mqudsi]

Starting with what I said before: we already have CMake and removing it doesn’t especially further the particular interests of the project in the same way that removing C++ from the project did, and there is no shame in making one (or many) release with a bastardized build system inspired by the project’s C/C++ roots but with the modernized rust toolchain also used in parts.

That said:

CMake is the better build tool.

Fully agreed, but

. I think we can get to a point where you can cargo build and get a working fish for testing

I want to get here too.

I don't see a winning argument for switching back to makefiles - the cmake dependency isn't a problem in practice (especially not for distributions) and we switched for a reason.

If we get to the point where we can build all the binaries, correctly configured, without CMake doing more than providing the parameters (what and where), then it’s being a better build tool is irrelevant. Which nullifies the advantage of CMake over Makefiles or over a rust runner.

I am - and always have been - partial to the beauty, simplicity, and universality of make followed by make install (I’ve released a BSD/Linux-compatible Makefile to drive Cargo and use it for most of my rust projects). It’s the lightest dependency for any project and can do things that don’t depend on the remaining components of the build toolchain being available, and if you’re using it as a task runner, it’s perfectly adequate. And it’s perfectly configurable to drive whatever you want behind the scenes, from shell scripts to secondary build systems.

(Look at how non-packagers can use Make now to drive fish’s CMake and the C++ build toolchain without knowing a thing about how the build system actually works, and it gracefully-ish handles the missing baseline components of the build system by telling users what they need to install to get started.)

[faho]

Which nullifies the advantage of CMake over Makefiles or over a rust runner.

Well, no, because: CMake already exists. We already have it. Most of the work we would have to do on it is to remove stuff, not add anything. (we would have to add to build.rs if we want cargo build to be more useful, but we'd have to do that regardless)

Plus, unlike you, I've never really liked makefiles, and we've always disagreed on how unknown cmake really is (I don't think running cmake .; make; sudo make install is that foreign to people, or too much to ask).

But if you want to write a Makefile, and it manages to be nicer than our current cmake system, we can of course merge it. I would want it to have rough feature-parity tho.

In particular it should support detection of curses and pcre, the output variables (DESTDIR/PREFIX etc), installing, out-of-tree builds and building with/without localization or docs. Cross-compilation would also be great to have (but again I've never tested our cross-compilation with cmake, I'm not sure if people actually use it).

Ideally it wouldn't require autotools.

---

(side note: it would be really cool if we found a rust-replacement for curses - we mostly use it to read terminfo, which as I understand it should be possible without linking to libcurses)

[mqudsi]

(side note: it would be really cool if we found a rust-replacement for curses - we mostly use it to read terminfo, which as I understand it should be possible without linking to libcurses)

This was my initial approach and I think there's a committed version of the riir branch that does just that, but unfortunately it turned out that many distros no longer ship a terminfo/termcap database or ship an out-of-date one. ncurses went from reading that to hard-coding its own equivalent in the library itself at some point, killing that as an option :(

edit: not in the riir branch, but its own repo here: https://github.com/mqudsi/termcap

[faho]

So it appears to be looking like "hollowed-out cmake, unless something better materializes".

In that case we would have to figure out how to make it easy to package.

Looking at arch's rust package guidelines I see recommendations to run cargo fetch --locked in the "prepare" step and then cargo build --frozen when actually building.

Is that something we can accomodate?

I'm assuming cargo fetch --locked works as-is, and you would run cmake -DCARGO_FLAGS=--frozen -DCMAKE_BUILD_TYPE=release to get a locked build? (tbh I'm not entirely clear on when cargo decides to update the dependencies to begin with)

How would this work on systems where the builder has no internet access?

[matklad]

tbh I'm not entirely clear on when cargo decides to update the dependencies to begin with

In two cases:

• you run cargo update, explicitly asking Cargo to update the lockfile
• Cargo.lock is incompatible with Cargo.toml (eg, you've added a new dependency to Cargo.toml)

[faho]

Alright, some changes since we last updated this:

• Fish no longer relies on (n)curses - Remove (n)curses dependency #10269
• PCRE2 is built via the pcre2-sys crate, we got out of that

So building is now a lot easier than it was.

---

The next part I admit is a bit spicy:

I would like to support installation via cargo install as a regular crate on crates.io.

I believe it is possible to get something usable (if not perfect), and I believe it is a nice, cross-platform way to install fish especially for people already used to rust.

This requires doing a lot of work via build.rs instead of cmake, and I believe it is possible to drop cmake almost entirely.

To that end I've been working on my fish-installer branch.

What that currently offers is this:

• You do cargo install --path .
• You run ~/.cargo/bin/fish
• It tells you that it is missing the asset files and you should run fish_installer
• You do that, it asks for confirmation before it installs files into ~/.local/share/fish/install
• You restart fish
• You now have a working install with the functions, completions and fish_config. Running help opens the online version
• When you update fish it tells you to run fish_installer again

I will admit this is not perfect. It is unfortunate that there is this second step of running fish_installer. We could technically embed the asset files in the fish binary and either extract them automatically (which is a bit naughty) or just read them from inside the binary (note: rust-embed supports reading from the filesystem in debug-mode, so you don't need to rebuild fish every time you change a script during development). The main downside of that is a binary that is two to three times as large.

There are also some missing bits, like the html docs, the pkg-config file, the vendor directories (they're bound to DATADIR so this fish would never read e.g. /usr/share/fish/vendor_confdirs - I'm unsure if this makes sense in general or if we should change it).

I believe it is possible to get the manpages and translations included as well by generating them in build.rs. HTML docs are basically impossible because they rely on fish_indent for highlighting and build.rs runs before that is built, there is no other place to generate the docs.

At that point, cmake is basically relegated to:

• build the html docs
• generate the pkg-config file
• copy the files to their locations (unless you use what cargo uses by default)
• Auxilliary tasks like running the test suite or making a mac package

At which point replacing it with an "install.sh" shellscript becomes tractable. (and yes, my preference is for a shellscript, since cargo handles the actual build and so make-et-al's main advantage of skipping work does not come into play)

[zanchey]

I believe it is possible to get something usable (if not perfect), and I believe it is a nice, cross-platform way to install fish especially for people already used to rust.

I'm just not convinced that the target market (people with a Rust toolchain installed but not using Nix or Homebrew which make installation very easy) is that big, but I'm not hugely opposed to any of the steps you've planned.

I don't want to underestimate how much distribution packages hate custom workflows (cf #10181 (comment)) but if it's as simple as cargo build; ./install.sh maybe that will be acceptable.

[faho]

I'm just not convinced that the target market (people with a Rust toolchain installed but not using Nix or Homebrew which make installation very easy) is that big

Note that this also enables creating statically-linked self-installable binaries.

I have, right now, a fish that is 100% statically linked (against musl), and I can copy it to a linux (with matching architecture) and run fish --install, and I have a usable fish.

This would remove the need for appimages, and it's enabled by what enables cargo install to work.

I don't want to underestimate how much distribution packages hate custom workflows (cf #10181 (comment)) but if it's as simple as cargo build; ./install.sh maybe that will be acceptable.

Alternatively we can keep cmake for them, it just won't do a lot.

[Be-ing]

I don't want to underestimate how much distribution packages hate custom workflows (cf #10181 (comment)) but if it's as simple as cargo build; ./install.sh maybe that will be acceptable.

Custom workflows are inevitable as long as Cargo provides no canonical solution how to do this. Using CMake for installation is something packagers know how to handle, but in a sense it's still a strange custom workflow for a Rust program.

[faho]

By "custom" zanchey was, of course, referring to something alien to packagers.

CMake would not count, because packagers know how to drive it. (unless we required you to pass things you typically don't)