[Bug?]: custom yarn checksum is bad for file sharing

[milahu]

Self-service

• I'd be willing to implement a fix

Describe the bug

yarn 2 introduced a custom checksum field in yarn.lock
but the checksum is not the sha512 hash ("integrity") of the original downloaded tgz file

problem: the custom yarn checksum is hard to reproduce and validate
see also Yarn v2+ lockfile, how the validate new checksum

this is problematic for tools like npmlock2nix that want to use the yarn.lock file
to download tgz files (and git commits)
and then run yarn in offline mode, to build a node_modules tree
because such tools need the integrity values of downloaded files

when such tools cannot use yarn.lock
then they need to invent their own lockfiles
which means: download all tgz files to get their sha integrity

custom yarn checksum is bad for file sharing

"file sharing" as in:
different nodejs package managers (npm, pnpm, yarn)
use the same tgz files to build their node_modules trees
with nix, all these tgz files are cached in /nix/store/

so now when yarn introduces a new archive format
to repack the original tgz files into yarn zip files
then the /nix/store/ has tgz files and zip files for the same node packages
(and even different zip files with different compression levels... #6068)
and these different files cannot be shared between different package managers
bottom line: more disk space is used

so ...

at least more documentation would be nice (i found zero)
why the custom checksum?
why not use sha512 integrity of tgz files?
why not cache the original tgz files? (does yarn cache the zip files?)

in my first impression, this looks like a bad tradeoff.
what exactly is the benefit of the custom yarn checksum?

(feel free to move this to a discussion)

To reproduce

.

Environment

yarn 2.x to yarn 4.2.2

Additional context

similar issues #6105 #5136

[arcanis]

different nodejs package managers (npm, pnpm, yarn)

Why should Yarn/npm/pnpm care about their cache being compatible with other package managers?

Yarn keeps the checksum of what's in its cache (ie what it actually uses when installing packages). The tgz are an intermediary format that disappears right after we fetch them (for various reasons, one of them being that tgz has bad random access capabilities).

[milahu]

Yarn keeps the checksum of what's in its cache (ie what it actually uses when installing packages).

The tgz are an intermediary format that disappears right after we fetch them

ok, this adds one layer of complexity
but i fail to see the benefits over caching the original tgz files (or git clone trees)

documentation is missing...
removing tgz integrity values from lockfiles is a radical change from npm or pnpm
so more transparency would be nice

tgz has bad random access

yes, but same for zip?
sqlite or squashfs or indexed archives would give fast random read access
but then, why? the archives will be extracted anyway

Why should Yarn/npm/pnpm care

to reduce global disk use, by agreeing on the same source of truth
at least with nix this makes sense, where npm/pnpm/yarnv1 share the same tgz files

[purepani]

but i fail to see the benefits over caching the original tgz files
In the case of yarn, they have a global cache, so the problems that hashing after re-compressing solves are:

1. Allowing them to use a different format(zip) which is better for their global cache.
2. Allowing the lock file to validate against that cache.

I agree that this is basically what nix is doing and had some confusion when looking into this too, and I generally think that keeping the tgz hashes would be a better thing for the yarn maintainers to do, but this also should be a problem that nix should find a way to work around in another way.

I'm sure you've considered the design implications @arcanis, but for the reasons mentioned in the linked issues, and other unseen ones that could easily crop up, having the hash match the sha-512 one in the npm cache can get rid the problems.

One solution to generate a package manager independent lock file would be to grab the sha-512 off of the npm api(see https://registry.npmjs.org/svelte-language-server/0.16.9 under dist.integrity). I was trying it earlier and it's the same as the nix hash, so it should work to avoid manually downloading the entire package at least.
I do think it would be better for yarn maintainers to store the hash based on the sha-512 to avoid the platform-dependent issues that are cropping up, but if that's not a design choice that yarn is willing to make then the above solution is at least functional.

[milahu]

Allowing them to use a different format(zip) which is better for their global cache.

why repack from tgz to zip?

why is zip better than tgz?

i only heard the rumor of "fast random read access"
but in my simple benchmark, tar xf is 5x faster than unzip

unzip    0.266s
tar xf   0.041s
sqlite   0.050s

npm is based on tgz, git is based on gzip, ...
zstd would give better performance

benchmark

$ cd $(mktemp -d -p /run/user/$UID)

$ df -h $PWD
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            12G  699M   12G   6% /run/user/1000

$ mkdir pkg

$ time for i in {0..9}; do head -c$((10*1024*1024)) /dev/urandom >pkg/file$i.bin; done

real    0m2.113s
user    0m0.080s
sys     0m1.957s

$ time zip -0 -q -r pkg.level0.zip pkg/

real    0m1.523s
user    0m1.147s
sys     0m0.294s

$ time tar cf pkg.tar.gz pkg

real    0m0.275s
user    0m0.032s
sys     0m0.233s

$ du -sh *
100M    pkg
101M    pkg.tar.gz
101M    pkg.level0.zip

$ rm -rf pkg

$ time unzip -q pkg.level0.zip pkg/file9.bin

real    0m0.266s
user    0m0.223s
sys     0m0.039s

$ ls pkg
file9.bin

$ rm -rf pkg

$ time tar xf pkg.tar.gz -- pkg/file9.bin

real    0m0.041s
user    0m0.005s
sys     0m0.028s

$ ls pkg
file9.bin

$ rm -rf pkg

$ ./pkg.py
output_db_path 'pkg.db'
done pkg.db -- last_file_id is 10

$ ./pkg.py read 10
extracted file pkg/file9.bin (b'k\x92g\xa0\x05\x1d)\xdes\x0f'...) in 0.050516605377197266 seconds

pkg.py

#!/usr/bin/env python3

import os
import sys
import sqlite3
import glob
import time

output_db_path = "pkg.db"
table_name = "files"
src_files_glob = "pkg/**"



if len(sys.argv) > 1 and sys.argv[1] == "read":
    file_id = int(sys.argv[2])
    assert os.path.exists(output_db_path), f"error: missing input file: {output_db_path}"
    # benchmark random read access
    t1 = time.time()
    connection = sqlite3.connect(output_db_path)
    cursor = connection.cursor()
    query = f"select path, content from {table_name} where id = ?"
    args = (file_id,)
    path, content = cursor.execute(query, args).fetchone()
    t2 = time.time()
    print(f"extracted file {path} ({repr(content[0:10])}...) in {t2-t1} seconds")
    connection.close()
    sys.exit()



print("output_db_path", repr(output_db_path))
assert os.path.exists(output_db_path) == False, f"error: output exists: {output_db_path}"
os.makedirs(os.path.dirname(output_db_path) or ".", exist_ok=True)

connection = sqlite3.connect(output_db_path)
cursor = connection.cursor()

sqlite_page_size = 2**12 # 4096 = 4K = default
cursor.executescript(f"PRAGMA page_size = {sqlite_page_size}; VACUUM;")

cursor.execute("PRAGMA count_changes=OFF")

cursor.execute(
    f"CREATE TABLE {table_name} (\n"
    f"  id INTEGER PRIMARY KEY,\n"
    f"  path TEXT,\n"
    f"  content BLOB\n"
    f")"
)

file_id = -1

for path in glob.glob(src_files_glob, recursive=True, include_hidden=True):
    if not os.path.isfile(path):
        continue
    file_id += 1
    query = f"insert into {table_name} (path, content) values (?, ?)"
    with open(path, "rb") as f:
        content = f.read()
    args = (path, content)
    cursor.execute(query, args)

last_file_id = file_id

connection.commit()
connection.close()
print(f"done {output_db_path} -- last_file_id is {last_file_id}")

sha-512

sha512 is a waste of disk space, because sha256 is not broken
but thats a separate issue, which also affects npm