LZMA Support Proposal

[thomas5280]

Hello, and thank you for fflate, which I have followed for several years now. Having watched the compression space for about 20 years, including all the ports of the original zlib and especially including JS implementations, I think fflate represents a correct and good 'end state' for the long lineage of this functionality. Well done and thank you.

I'd like to ask about LZMA. I ask because I think this decompression method stands right up there with DEFLATE as an important contribution for the world, and it also has its own set of very elegant properties from a CS/algorithm standpoint. Of course, we all know in terms of sheer use, standard DEFLATE (and DEFLATE64) are the most common and likely will be going forward, and that's OK.

LZMA as you know is an available compression method in ZIP (Compression method 14 since v6.3 around 2006). In my view, any serious ZIP processor should support it (WinZip and others did in fact make LZMA archives with 'best compression' specified).

But more importantly, LZMA is used in several important other contents: 7-zip (.7z), XZ (.tar.xz) (uses LZMA2, a wrapper around LZMA); standalone LZMA (not really applicable to everyday use), and a somewhat obscure format LZIP (*.lz, *.tar.lz) which is basically improved GZIP in terms of archive header but using LZMA (my personal favorite). https://www.nongnu.org/lzip/

I myself have two working implementations of LZMA in JavaScript and would be happy to collaborate. These are inelegant implementations, pieced together ports from various places, and the second one is taken primarily from the C code in info-zip (a good starting point). Both pass an extensive set of test files that I have.

The same discussion applies to BZIP2, which is useful both in context of ZIP (dating back earlier than LZMA to v4.3) and also *.tar.bz2.

I see you've just tackled ZSTD. Would you be willing to tackle LZMA next? I think an fflate-style flzma would be a meaningful contribution to the world.

Tom

[101arrowz]

Thanks for the kind words! I mainly made fzstd for fun, not because I had any need for it, but I will look into the LZMA format and see if I can make a good JS implementation. However, from my experience LZMA and BZIP2 are far, far less common than DEFLATE so if it's a lot of engineering work I may only make a decompressor, like I did with fzstd, and it may take quite a while. I'll update here if I make any progress. Feel free to share your JS implementations for LZMA by the way, they could be useful starting points.

[thomas5280]

Thank you for the thoughtful response. Yes, I agree that a decompress-only mode is appropriate and useful for LZMA (and, down the road, BZIP2). And, I think decompression on both of these is simpler algorithmically (a typical result).

The goal here would be simply to enable support for decoding the many 7z, ZIP-LZMA, and tar.xz files out there. Encoders for all of these are still in active use. My feeling is that the world does not need another LZMA encoder beyond what we have (including LZIP).

I'll share my LZMA implementation with you! It's pretty fast and I believe is accurate, it's a bit strange since it has a pseudo streaming/chunking wrapper around it but this should feed into your async approaches (at least in concept).

[thomas5280]

One last follow up, LZMA2, a wrapper around LZMA, is really the gold standard here, but is not super well documented outside of reference implementations. Fortunately I've got that figured out, including all of tricky conversions needed in JavaScript vs. other languages in terms of 32 bit integer handling, notably Math.imul().

[101arrowz]

Sounds good, I'll take a stab at it next week.

[larabr]

Hey @101arrowz , is bzip2 now supported? I've seen it mentioned in the README but I couldn't find the implementation.

[Tsury]

Hey @101arrowz, I would also like to know whether bzip2 decompression is possible, I, too, am confused by the readme file. Using the opportunity to tag you again since it's been ~6 months since the last tag.

[101arrowz]

BZIP2 and LZMA are not supported by fflate, and I don't plan to change that in the future.

These algorithms are mentioned in the README here:

// This function will almost always have to be called. It is used to support
// compression algorithms such as BZIP2 or LZMA in ZIP files if just DEFLATE
// is not enough (though it almost always is).
// If your ZIP files are not compressed, this line is not needed.
unzipper.register(fflate.UnzipInflate);

What I meant by that is that you can add support for a custom compression algorithm if you implement it and register() with the unzipper instance, not that UnzipInflate supports either of those algorithms. I can see now that's very unclear from how it's written; sorry about that!

[Tsury]

@larabr You are welcome to check my new repo, streaming-bz2-decompress. Not sure it will fit your needs though as it's pretty specific.