Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: warn that preallocation disables compression on btrfs #1692

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

martinetd
Copy link

(foreword: I open PRs as issues when it's not much work to write, feel free to close this and adjust documentation otherwise if that is better for you)

Rationale:

Current aria2c documentation for --file-allocation states:

If you are using newer file systems such as ext4 (with extents support),
btrfs, xfs or NTFS(MinGW build only), falloc is your best choice. It
allocates large(few GiB) files almost instantly.

For btrfs, it might be worth noting that preallocating a file with falloc() will disable cow for writes in the preallocated area, and has the side-effect of also disabling file compression for that file.
(I'm not exactly sure what would happen with zero-filling manually, but even prealloc uses fallocate now if supported apparently so this is moot -- either cow is still supported and the writes are thrown away, with the compression autodetection mechanism thrown down the floor (e.g. likely compressing even if data isn't compressible) or cow is disabled and compression gone)

This has been a subtle issue for nixos for a while:
NixOS/nix#3550

While they are not using aria2c, that illustrates quite well that knowledge of how btrfs compression works is quite sparse so a note documenting the tradeoff (potential speed vs. compression) in the documentation would probably make sense; hence this commit.

test:

$ aria2c --file-allocation=trunc https://codewreck.org/10Mo.zero && sync && sudo compsize 10Mo.zero && rm 10Mo.zero 

10/04 17:31:31 [NOTICE] Downloading 1 item(s)
[#d91dbb 8.5MiB/10MiB(85%) CN:1 DL:1.7MiB]                                                                                                                             
10/04 17:31:36 [NOTICE] Download complete: /tmp/10Mo.zero

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
d91dbb|OK  |   1.7MiB/s|/tmp/10Mo.zero

Status Legend:
(OK):download completed.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL        3%      320K          10M          10M       
zstd         3%      320K          10M          10M       

$ aria2c https://codewreck.org/10Mo.zero && sync && sudo compsize 10Mo.zero && rm 10Mo.zero 

10/04 17:31:41 [NOTICE] Downloading 1 item(s)
[#0a3873 8.3MiB/10MiB(83%) CN:1 DL:1.4MiB ETA:1s]                                                                                                                      
10/04 17:31:48 [NOTICE] Download complete: /tmp/10Mo.zero

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
0a3873|OK  |   1.5MiB/s|/tmp/10Mo.zero

Status Legend:
(OK):download completed.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%       10M          10M          10M       
none       100%       10M          10M          10M       

Original commit message:

falloc() will disable cow for writes in the preallocated area, and
has the side-effect of also disabling file compression for that file.

If user wants compression they should be advised to use trunc or none
file-allocation instead.

@martinetd
Copy link
Author

(FWIW the build error looks like a stalled build on macos that I'm fairly sure is unrelated -- I'll force push once to retrigger build...)

falloc() will disable cow for writes in the preallocated area, and
has the side-effect of also disabling file compression for that file.

If user wants compression they should be advised to use trunc or none
file-allocation instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant