Add support for libsixel

[HalosGhost]

libsixel is an ANSI-compatible library for SIXEL/DEC graphics painting in a terminal. This allows for richer interface design and better integration between the text-based and graphical environments.

There is at least one person who has already started work on wrapping libsixel for use in Rust, though a different wrapper may be preferrable.

xterm whose terminfo alacritty defaults to already supports libsixel (through a compile-time option).

[aslpavel]

Sixel support would be really nice to have. By the way it would not require any binding for libsixel format is pretty simple to implement directly.

[crocket]

sixel doesn't support 24-bit colors yet, but it is still good.

[aslpavel]

Sixel does support 24-bit, it is just palette based. It is just not very efficient if you have too many distinct colors.

[aurelg]

Extra motivation: it seems lsix relies on sixel.

[vifino]

Hey. I would love sixel support in alacritty as well.
Have there been any attempts yet?
Sounds like something I could try to implement, but unfortunately I'm pretty busy.
Might get around to it in a few weeks?

[chrisduerr]

Have there been any attempts yet?

There have not.

Sounds like something I could try to implement, but unfortunately I'm pretty busy.
Might get around to it in a few weeks?

If you want to give it a shot, feel free and let me know if you need any help.

[Aaron1011]

I'd be interested in working on this.

[chrisduerr]

@Aaron1011 If you have any specific questions, please let me know either here or on the Alacritty IRC on freenode.

[vifino]

If anything, implementing both is a good idea. Sixel has quite a few nice uses.
It is not just displaying images. Plots for example: Sixel backend for Python's Matplotlib.

[mbaz]

Sevaral scientific plotting packages support sixels (example), which is nice because you don't need a browser or additional windows to see the plots. However, very few terminals, and no modern terminals, support it. It'd be nice to have a modern, fast terminal support this.

[mbaz]

@leotaku You're right that most of those programs generate a PNG and then translate it to sixels, and that the kitty protocol is likely to be a better, more modern solution. To me as a user, the important feature is to get images on the terminal, in a way that is as standard, well-documented and widely supported as possible.

[chrisduerr]

Sixel is a standardized and accepted protocol. Kitty's own protocol is not.

[crocket]

You better create a new issue about kitty protocol.

[HalosGhost]

@leotaku, I line up with @chrisduerr. Sixel has been a standard for an incredibly long time; more than that, it is wholly backwards-compatible. Every other format I've seen for images-on-terminal (including kitty, iTerm2, terminology, and others) is propretary, is not interoperable and/or is not backwards compatible.

Sixel is the only reasonable option. It also has the benefit of having several high-quality implementations out there already.

[crocket]

Is sixel going to support 24bit color space? The current color space is 8bit.

[HalosGhost]

@crocket, I believe you were a part of this thread: saitoha/libsixel#44

tl;dr: most terminal implementations of true-color are actually also not backwards-compatible, and so, in principle, sixel cannot reasonably be compatible with them and maintain its backwards-compatibility. However, there is the --high-color option for more colors.

[jmriego]

It's true that Sixel is palette-based and that makes it not the best solution if you need truecolor. But is that something that should be the first priority? If it's not 100% required I don't really see why even try moving to another system.

Kitty, iTerm2 and others do their own thing completely and they could make breaking changes in the future. Should alacritty be tied to their changes? Or if they make changes should we stick with whatever version of the protocol alacritty is on and not being able to use apps depending on their new protocol version?

Sixel is the only thing that can be considered a standard at the moment for graphics in the terminal. Sure, it can be improved on. But in my opinion it's better to stick to Sixel and if anyone wants to improve the Sixel protocol that's a different discussion but we would be in a better position to continue without breaking anything

[emmanueltouzery]

From what I've seen the kitty authors want their protocol to be a standard and certainly document it as such:
https://sw.kovidgoyal.net/kitty/graphics-protocol.html

[crocket]

Is there anyone willing to implement any terminal graphics protocol in alacritty?
I'd welcome any protocol at this point.

[HalosGhost]

Looks like the new version of iTerm2 includes sixel support.

[ghost]

I just finished implementing sixel read support for my TUI library, and it was surprisingly straightforward. The code to convert a string of sixel data to a bitmap image is here, and the client code for the Sixel class is here.

I have done very little for performance on the decoder. But when using the Swing backend, performance is still OK, as seen here. The snake image looks bad only because byzanz used a poor palette creating the demo gif.

I was a bit taken aback how quickly it came together. It's very fair to say that the "decode sixel into bitmap" part is the easy bit, the hard bit is the "stick image data into a text cell, and when that is present blit the image to screen rather than the character".

[rabite0]

Lacking support for graphics is the reason I switched over to kitty... I think ultimately the question of which API to support isn't really all that important. At least in the sense that there's technically no reason to only support a single one, since they're only trivially different for the most part. The way I see it there are 3 kinds of APIs/formats/protocols:

• SIXEL: The tried and true standard, supported by multiple terminals and a "wide" range of software. It's basically its own image format with in-band transmission to the terminal. A bit slow, and inefficient, but with transparent network support.
• urxvt/iterm/etc: Basically just an escape code with a path to an image file and some metadata like dimensions and where to draw to. Some variations with in-band base64 encoded transmission or some such exist, but for the most part they're similar enough that they're almost interchangeable by a string substitution.Limited network support, depending on the exact implementation.
• kitty: This one is the most complex with multiple ways to transmit images, both in and out-of-band, among them a very fast shared memory path, which allows for things like really nice and fast video playback with very little overhead. Supports both (A)RGB raw image and png, via path, in-band transmission and shm, with optional zlib compression.

Ultimately just adding SIXEL support requires decoding it and uploading it as a texture to draw it, but at that point decoding png/jpg/whatever and hooking up another escape code isn't really going to add much more work on top of that. The only thing that really changes is the front-end. So adding urxvt/iterm/we support would be almost free, with the advantage of being faster/more efficient and possibly higher quality, since it eliminates the redundant encoding/decoding step in between. The hard part is adding any kind of graphics support at all in the first place, I think.

kitty's protocol is the most complex with multiple ways to transmit image data, redrawing images multiple times at different positions without retransmission, drawing over and under text, with alpha blending, scaling and so on, but it's also the most complete and efficient way to draw graphics on a terminal. It's a bit overkill IMO, but not bad per se. This is the only one requiring substantially more work to add.

Sooo... Since this has been open for quite a while and no has one dropped any code yet, I've been thinking about looking into it myself during the upcoming spring holiday season, starting with SIXEL or whatever is easiest to get up and running at all and working my way up from there. No promises though ;)

[chrisduerr]

The notion that once one protocol is implemented we could just tack on the other formats in a few minutes of work is entirely false. The different formats might share some similarities, since after all they all have the same purpose, however supporting multiple of them would lead to significant additional code that would likely go unused for many, which would however still have to be maintained.

Most of these protocols also have some serious flaws, making it pointless to just rush into it and try to implement some crappy protocol that isn't going to get any actual use out of it. Which is why most of the existing protocols are used very, very rarely, with people even preferring the hacky barely working w3m solution over them.

[rabite0]

A few minutes might be a stretch, but given that images works at all, adding another protocol within a few hours doesn't seem implausible. I don't know alacritty's codebase well enough to say for sure, but I'd imagine adding any kind of image protocol to take much longer, relatively speaking. Not saying that supporting too many is a good thing necessarily...

But as far as the amount of code goes there's nothing in urxvt's protocol you wouldn't also need for kitty's protocol, for example. Same for iterm2. Everything it does would also be required for what kitty's protocol does, except for the literal escape code (and the download stuff...). Not sure what else there is, vector graphics is a thing, too, I guess. I was just saying that because you seemed fine with adding kitty's protocol which is kind of a superset of the others as far as I can tell.

If you only want a single protocol in alacritty that's understandable, too. I might also be overlooking details that invalidate some of my assumptions...

[kchibisov]

I've got a pretty solid reference to the various formats here

I was talking about prompt marks, as in OSC 133.

[amano-kenji]

I found your image protocol. Where do I find a new keyboard protocol? Let's move the discussion there.

[christianparpart]

I found your image protocol.

that's draft but will definitely be resumed to be worked on as soon as i'm done with the current building blocks

Where do I find a new keyboard protocol? Let's move the discussion there.

it all started here: http://www.leonerd.org.uk/hacks/fixterms/
also note the existence of modifyOtherKeys from xterm,
then there is Kitty's version: https://sw.kovidgoyal.net/kitty/keyboard-protocol/

[csdvrx]

I found that tmux is considering integrating the sixel protocol has branches that supports sixel protocol

See also https://github.com/csdvrx/sixel-tmux which as a fallback and to support scrolling "downconverts" sixel images to ANSI colored unicode :

[FergusFettes]

@csdvrx that link is broken

but i checked out your repo anyway-- looks nice, but there are no installation instructions?

[csdvrx]

You need to compile as you would normally compile tmux.

Usually it means running bash autogen.sh then ./configure

For configure you may want to pass options, ask which options are supported with --help

[amano-kenji]

How likely is sixel to become a terminal image standard? It is a suboptimal protocol, compared to kitty, and kitty itself is more complex than it should be.

Because there is no terminal image standard, no TUI library used in practice supports image. There are just experimental TUI libraries which plays with terminal image.

I feel that a real terminal image standard which is neither sixel nor kitty may emerge in the future.

[csdvrx]

@amano-kenji your guess is as good as mine.

I live in the present and I care about having access right now to most tools that offer terminal images, so I use sixels because it just works.

[dankamongmen]

How likely is sixel to become a terminal image standard?

it IMHO already is, to whatever degree anything will be. any terminal which supports the kitty protocol (only kitty itself and wezterm iirc) supports sixel as well (though using the kitty protocol is desirable anywhere it's supported).

It is a suboptimal protocol, compared to kitty, and kitty itself is more complex than it should be.

i agree with the former, but not the latter. Notcurses exploits almost the entirety of the kitty protocol (and indeed i lobbied to have a feature added, see kovidgoyal/kitty#3809). it might not be necessary for "just blit this image to the terminal", but for any kind of more advanced work, it's all pretty easily justified. see https://github.com/dankamongmen/notcurses/blob/master/src/lib/kitty.c to see how all we use it.

Because there is no terminal image standard, no TUI library used in practice supports image.

as of version 3.0 notcurses offers complete, robust, and abstracted bitmap support. see the ncvisual API: https://github.com/dankamongmen/notcurses/blob/442cfea046ae4c62d52f536356a53d48dad554e7/include/notcurses/notcurses.h#L3258

I feel that a real terminal image standard which is neither sixel nor kitty may emerge in the future.

having worked several months to integrate the various protocols (see https://nick-black.com/dankwiki/index.php?title=Theory_and_Practice_of_Sprixels), i sat down to design my own protocol from the viewpoint of a library implementor. it ended up looking so similar to kitty's that i put it aside. https://nick-black.com/dankwiki/index.php/Spriteful_TErminal_GrAphics_Protocol

[chrisduerr]

for any kind of more advanced work, it's all pretty easily justified

Which is completely out of place for terminal emulators.

[dankamongmen]

for any kind of more advanced work, it's all pretty easily justified
Which is completely out of place for terminal emulators.

if all you want is "blit this image to this place", i'd take a look at the "1337" protocol of iTerm2.

[amano-kenji]

Kitty can change on whims of one man. It's just a proprietary protocol.

ANSI escape codes are a standard. A standard cannot change due to one man's whims.

As far as I know, there is no modern terminal image standard that is free from personal whims.

[amano-kenji]

OSC1337 doesn't document a way to pick a place where an image is printed.

[nixpulvis]

I'll just say this, because I was out in traffic for too long today.

Don't block the box!

[FergusFettes]

@csdvrx Yes I figured, but the branch is so diverged from the main that I couldn't figure it out!

Have you tried zellij btw? It works with sixels and this branch of alacritty: https://github.com/zellij-org/zellij

So I think I will just switch to that. zellij plus this alacritty branch is a quickstart multiplexing with sixels!

[hholst80]

@csdvrx Yes I figured, but the branch is so diverged from the main that I couldn't figure it out!

Have you tried zellij btw? It works with sixels and this branch of alacritty: https://github.com/zellij-org/zellij

So I think I will just switch to that. zellij plus this alacritty branch is a quickstart multiplexing with sixels!

How did you make it work? What version of alacrity does it need? of zellij?

[FergusFettes]

@hholst80 IIRC:

1. Support for graphics in the terminal #4763 this is the alacritty branch corresponding to this issue
2. I got the latest zellij pre-built from here https://github.com/zellij-org/zellij/releases/tag/v0.36.0
3. I installed libsixel from here: https://github.com/saitoha/libsixel/releases/tag/v1.8.6

running ubuntu 22.04

works like a dream, but i guess its possible i just got lucky with the combinations

[mobsenpai]

So, will it ever get implemented?

[alexshpilkin]

To add to the protocol documentation links: aside from the graphics programming volume of VT330/VT340 Programmer’s Reference (EK-VT3XX-GP-001), the sixel protocol is also documented from the implementer’s point of view in chapter 9 of the DEC-internal Video Systems Reference Manual (DEC STD 070-9, EL-00070-09).

[amano-kenji]

If you can't wait, you can use other terminal emulators.

[chrisduerr]

Considering its complexity, it is unlikely it will be merged. However I wouldn't rule it out entirely.

[RetroTechCorner]

sixel support would be great (although I know it's not a wish-list...)