Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build-release: only zip files, not directories #22

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

illwieckz
Copy link
Member

Zip format does not require to store directory to store files in a directory. Storing directory is only required for storing directory metadata (like file modification date, etc.)

Not storing directories produces smaller zip files, the directories are automatically created when unzipping the files.

@slipher
Copy link
Contributor

slipher commented Sep 10, 2022

This saves what, 2 KB out of a 600M release? Seems not worth it.

@illwieckz
Copy link
Member Author

illwieckz commented Sep 11, 2022

At some point we may want to extend that package function anyway. For example we use validate-release to check that our engine zips don't ship weird permissions that would be written on users' computers when unpacking our zips. So we may want to fix those permissions there once for all. On a side note not shipping the folders even remove the need to ship proper permissions on folders.

I also want to check if the macos engine zip isn't expected to ship symlinks (the macos deps file is already know to do that), preventing 7z to dereference them would save file space as well.

Right now when releasing the game I recompress all our dpk and zip files with my drt-zipbrute tool from there. It has an option to rewrite the permissions of files in zip (so they can pass through validate-release without failing), but it also offers many optimization techniques to reduce zip file size:

  • it can touch all files in a zip to the same date: that doesn't improve file compression in zip itself but that improves zip file compression in zip, like engine/dpk/symbols zip getting better recompressed in universal zip,
  • it can remove useless folder entries, not only it save file space by itself, it removes the need for having proper folder permissions, and of course the time unification a zip-in-zip optimization is just even not needed,
  • it makes sure files are stored in the same order in the zip on every run (improves reproducibility),
  • it can recompress separately each file with different tools (7z, zopfli…) and keep the smallest file for every file.

This not only make our zip valid for validate-release, it just produces the smallest zip possible.

@illwieckz
Copy link
Member Author

illwieckz commented Sep 11, 2022

I forgot to mention in the PR description that the sort call also makes sure the files are stored in the same order in the zip on every run, this improves reproducibility.

Zip format does not require to store directory to store files
in a directory. Storing directory is only required for storing
directory metadata (like file modification date, etc.)

Not storing directories produces smaller zip files, the directories
are automatically created when unzipping the files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants