Since when SILE 0.14.x on Arch no longer uses the default system Lua?

[Omikhleia]

Simple Docker file:

FROM docker.io/library/archlinux:latest
RUN pacman --needed --noconfirm -Syu

# Lua luarocks
RUN pacman --noconfirm -S luarocks git

# SILE
RUN pacman --noconfirm -S sile

WORKDIR /data
ENTRYPOINT ["sile"]

Build downloaded docker.io/library/archlinux:latest (which I didn't have), performed the other things.

$ docker run -it --entrypoint bash --volume "$(pwd):/data" --user "$(id -u):$(id -g)" silenext
[I have no name!@3707faf042e2 data]$ lua -v
Lua 5.4.6  Copyright (C) 1994-2023 Lua.org, PUC-Rio
[I have no name!@3707faf042e2 data]$ sile --version
SILE v0.14.13 (LuaJIT 2.1.1700008891)

Q1. Since when SILE 0.14.x bundles or requires its own version of Lua? Up to recently, SILE was using the system's default , and I had understood that the switch to LuaJIT would come in 0.15 (along with other rusty stuff)... Sounds weird to see it in 0.14.13, 'cause it breaks my workflow, it seems... and the release notes said nought about such a drastic change...

Well let's go any way and add at the end of this Dockerfile:

# Resilient packages and classes for SILE
RUN luarocks install resilient.sile
RUN luarocks list

Aha, there are installed for 5.4 indeed and are not found. Let's add --lua-version=5.1 in both commands there...
Now installs for 5.1, but with warning s(about things not explicitly required by my rocks):

Missing dependencies for luaepnf 0.3-2:
   lpeg >= 0.8 (not installed)
Missing dependencies for penlight 1.13.1-1:
   luafilesystem (not installed)

Q2. Probably some issues with luarocks itself? Can't say, that's a bit frightening but let's cross fingers, I never tried (regular) 5.1 before...

Next step is to check if it works... Let's alias running this docker image too resilient and proceed...

Using the test.sil from #1921 (reply in thread)

$ resilient test.sil 
SILE v0.14.13 (LuaJIT 2.1.1700008891)
<test.sil> as sil
[1]

Yay! Worked (though I don't get the discussion there at all, then: it seems this SILE with LuaJit finds 5.1 module...).

Using the example from https://mastodon.social/@omikhleia/111518778130369008

$ resilient -u inputters.silm mastodonsample.silm 
SILE v0.14.13 (LuaJIT 2.1.1700008891)

! Unexpected Lua error

error summary:
	Processing at: txt.silm: in <snippet>:
		[[]]
	Using code at: /usr/share/lua/5.1/sile/resilient-tinyyaml.lua:777: bad argument #1 to 'sfind' (string expected, got nil)

Wow, seem I'll have to investigate even 3rd party code. Which I didn't expect in 0.14 and won't be able to do quickly.
Doh, this will teach me not to advocate for SILE on social networks ;) ... at best.

Getting SILE right for 3rd party devs is hard.

[alerque]

The SILE provided Docker images that build SILE from source using Arch system dependencies still use Lua 5.4 (until v0.15.0 comes out). You can build Docker images using that as a base if you like. The Arch Linux system package switched from Lua 5.4 to LuaJIT here.

You can read which version of Lua you should install modules for, then install them like so:

$ sile -q -e 'print(SILE.lua_version); os.exit()' | read lua_version
$ luarocks --lua-version $lua_version install resilient.sile

Yes, LuaJIT uses modules in the same format as Lua 5.1 builds them.

We've been building and testing SILE against 4 versions of Lua for years now so that downstream distros and users can use whatever suits them best. If you want to write modules that work across the broader ecosystem you should at least be testing 5.1 since that has the most limiting API. The vast majority of 3rd party Lua code (such as on LuaRocks) is 5.1 compatible for just this reason.

The most likely culprit for the error you show there last is probably trying to concatenate a Nil. You have to cast tostring() for that to work in Lua 5.1, it won't automatically cast to a string when concatenating.

[alerque]

Also just a pointer, you can install some of your packages from the AUR such as silex. The PKGBUILDs for those show how to do a system installation matching Arc Linux's SILE package Lua expectations.

[Omikhleia]

Also just a pointer, you can install some of your packages from the AUR such as silex. The PKGBUILDs for those show how to do a system installation matching Arc Linux's SILE package Lua expectations.

Why there would be non-official AUR distributions of my modules fully eludes me, when Luarocks was the advertised way to distribute such 3rd-party packages. How these will be maintained (updated etc.) eludes me too ;)

[Omikhleia]

Ping?

[alerque]

Sorry this did fall out of my notifications. It's not a SILE question per-se more of a general software packaging, Linux distribution, and specifically Arch Linux & LuaRocks question. Yes there are a lot of moving parts here so I'm not surprised it creates some confusion. I'll try to de-conflate some different parts here, but if any of this helps you make sense of things and you feel it should be documented for SILE users it would probably be good to adapt this for the manual or something. I'm probably not a good one to do that because I've been involved in the Linux packaging scene for so long I take a lot of things for granted that isn't always apparent to folks starting out.

We have a huge mix of parties and tooling here, many of them wearing multiple hats. For parties we have at least:

• The upstream SILE project (me, you, other contributors, etc.)
• 3rd party module developers (me, you, others)
• Distribution developer teams (Nix, openSUSE, Fedora, Arch Linux (which I happen to be a member of), etc.)
• 3rd party packaging ecosystem developers and their users (Homebrew, Nix, AUR, etc)
• End users of SILE.

For tooling we have at least:

• Source repos and upstream source releases (this repo, your module repos, etc)
• Containerization systems and their registries (docker w/ Docker Hub or GHCR, nix w/ Nixpkgs, GH Actions w/ marketplace, etc.)
• Language specific build tooling and their registries (luarocks with luarocks.org, cargo with crates.io, etc.)
• First party package managers and their repositories (Arch's pacman with distro repos, Fedora's dnf and )
• 3rd party packages managers and/or related repositories (Arch's AUR, Fedora's COPR, Homebrew, etc.)

In reality it is even more complex because many of these points are actually whole ecosystems of themselves. Arch Linux isn't the end of the line, there are at least half a dozen derivative distros that package SILE downstream from Arch (with and without packaging modifications) that get more parties and tooling involved (see e.g. repology).

But back to your specific case:

SILE as an application as a general assumption expects 3rd party modules to be organized in the way luarocks (the tool) organizes them. This is designed to play nice with lua the language and it's module system (which can of course be used independently from luarocks, luarocks being tooling to make the management easier. As such SILE end users can use luarocks (the tool) and-or https://luarocks.org (the registry) to find and manage SILE modules. 3rd party package developers like yourself can use either/both the LuaRocks tool for publishing and registry for distribution. Either developers or end users are welcome to forgo either or both ends of that and do things manually (for example luarocks can install from GitHub sources or local files without using the registry, or you can manually extract and place stuff downloaded from the registry without using the tool.

That pretty much covers you as a package developer. SILE has passed all of it's "plugin"/"package"/"module" management off on luarocks as a tool that plays nice with Lua sources and integrates well with SILE's expectations.

But that's not the end of the story for end users. People have lots of different preferences here and some projects call for different workflows. Skipping over manual methods like copying files by hand... The luarocks tool itself can be used to manage packages (rocks) installed per project, per user, or per system. End users on any system are welcome to use their system as they please, but to me personally (with either my Linux user hat or my distro developer hat) using any tool other than the official distro tooling to install anything system-wide is a big no-no. My base system and the things it provides globally should only ever be touched by the official distro package manager (whatever that is for a given distro). I know npm and cargo and luarocks and almost every other language specific built tool includes the ability to install to the root system (--global or --sysetm or whatever) but I think this is crazy and never let that happen. Either I install things to a project or to my user using these other tools or use the system tooling for my system, but no mixing and matching. (Notably it is also a generally accepted no-no for distro tooling to mess with anything in a user's home directory outside the system scope.)

SILE uses GNU Autotools for configuration and compilation (./configure, make). This tooling is also capable of installing (sudo make install). I never use it for that last step for this or any other project. If I install something to my system I use the system packages instead. In my case that is Arch Linux most of the time, so if official ones exist (as they do for sile) then I use that. In this case I happen to be the person responsible for official distro packaging, but that is irrelevant to my point. If they don't exist then I make (or use) an AUR recipe to wrap an upstream project's build and install into a system package. The AUR is an open repository (open to any contributors with about 5 seconds of signup and light moderation) of recipes to fetch, configure, and build upstream projects and turn them into system packages. They can then be managed with the official distro package tooling.

Why does this matter to me as an end user?

• Because I don't want anything breaking my host system. If I do anything dubious I'll do so in a container or virtual machine or at least as a non-privileged user or whatever. Nothing gets access to root/system stuff except the packaging system that was made for it.
• I am assured of being able to uninstall things cleanly, replicate the system conditions on other systems, etc.
• Dependencies are handled better. For example lets theorize about a currently non-existent 3rd party SILE package that handles mermaid graphs. The 3rd party project will have a bunch of Lua code and per SILE's suggestion probably is setup to be installed via LuaRocks. It's probably even distributed on https://luarocks.org. Lets say that in order to run it needs SILE (obviously), the external CLI tool for mermaid, and also a Lua library or two. As an example lets say it needs a Lua JSON library for whatever reason. The LuaRocks tooling can provide the dependency information for the Lua libraries such that luarocks install sile-mermaid also installs dkjson and any other needed Lua modules in the relevant place. But it cannot check for or provide the mermaid CLI itself, you would just document that in the project README as something the user needed to install. Enter a distro package (or AUR recipe). A distro package could come along and wrap the luarocks install, but also provide the other dependencies from the system making sure they all worked together out of the box. For example if it was an Arch Linux package (or AUR) one would pacman -S sile-mermain and they would automatically get lua-dkjson, sile, mermaid, and anything else they needed.

So who is responsible for the AUR packages of your projects? Well the AUR is open to anybody and generally works on a first-come-first-dips moderation system. I was the first one along interested in your modules being packaged their so I posted them. As the initial poster I'm also marked as the 'maintainer', but I can also add others to manage them or disown them and leave them up for grabs. Also others can flag them for various reasons and if their concerns don't get met moderators can orphan them and leave them open for whatever contributor comes along next to pick up. This happens when people loose interest in updating packages or they are broken or whatever.

For the moment I'm happy posting version bumps to your packages in the AUR. I'm also happy to add you as a co-maintainer to them if you are interested, but unless you use them yourself there is no particular reason you have to care. If you look under the hood they are just a pretty thin wrapper around luarocks doing an installation to a specific directory that is then transformed into a system package.

Other distros have similar systems too but none of us have to care really.

[alerque]

Also as a side note, Arch doesn't really have a "default" lua. It has completely parallel packaging for 5 Lua versions and users can use any of them (or even all of them). A few project depend on specific versions and packaging reflects that, others don't care. For those that don't care Arch packagers pick one: usually either the newest (Lua 5.4) or fastest (LuaJIT) depending on the project.

This is actually pretty common for Lua, most distros handle Lua similarly. As the Arch Linux packager for SILE I made the decision to switch from the newest available Lua (which wasn't providing any benefit to SILE over any other version) to the much faster LuaJIT. Other packagers have made similar choices (for example NeoVIM or MPV being built against LuaJIT). Some other distros have made similar choices when possible, others not of course.

This will be slightly more complicated for SILE v0.15.0 as we'll also be shipping a Lua interpreter while still having optional compatibility for that being any of 6 different versions of 3 different upstream "Lua" fork projects.