Support runtime chunk deduplication #1507
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jiangliu
requested review from
liubogithub,
luodw and
changweige
and removed request for
a team
jiangliu
force-pushed
the
runtime-chunk-dedup
branch
4 times, most recently
from
7cc1aa1
to
ac55d88
Compare
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #1507 +/- ##
==========================================
+ Coverage 62.72% 62.74% +0.02%
==========================================
Files 129 129
Lines 44153 44360 +207
Branches 44153 44360 +207
==========================================
+ Hits 27695 27834 +139
- Misses 15087 15144 +57
- Partials 1371 1382 +11
|
jiangliu
force-pushed
the
runtime-chunk-dedup
branch
4 times, most recently
from
bc4403c
to
946d8a0
Compare
3 tasks
jiangliu
force-pushed
the
runtime-chunk-dedup
branch
2 times, most recently
from
13de7e8
to
40415e5
Compare
Desiki-high
reviewed
Dec 9, 2023
jiangliu
force-pushed
the
runtime-chunk-dedup
branch
3 times, most recently
from
5403291
to
1c695cf
Compare
ccx1024cc
reviewed
Dec 13, 2023
ccx1024cc
reviewed
Dec 13, 2023
jiangliu
force-pushed
the
runtime-chunk-dedup
branch
from
1c695cf
to
d199ab8
Compare
Add helper copy_file_range() which: - avoid copy data into userspace - may support reflink on xfs etc Signed-off-by: Jiang Liu <[email protected]>
Implement CasManager to support chunk dedup at runtime. The manager provides to major interfaces: - add chunk data to the CAS database - check whether a chunk exists in CAS database and copy it to blob file by copy_file_range() if the chunk exists. Signed-off-by: Jiang Liu <[email protected]>
Enable chunk deduplication for file cache. It works in this way: - When a chunk is not in blob cache file yet, inquery CAS database whether other blob data files have the required chunk. If there's duplicated data chunk in other data files, copy the chunk data into current blob cache file by using copy_file_range(). - After downloading a data chunk from remote, save file/offset/chunk-id into CAS database, so it can be reused later. Signed-off-by: Jiang Liu <[email protected]>
Add smoking test case for chunk dedup. Signed-off-by: Jiang Liu <[email protected]>
Add documentation for cas. Signed-off-by: Jiang Liu <[email protected]>
jiangliu
force-pushed
the
runtime-chunk-dedup
branch
from
d199ab8
to
7d287c9
Compare
imeoer
reviewed
Dec 19, 2023
|
|
||
| // Verify lower layer mounted by nydusd | ||
| ctx.Env.BootstrapPath = lowerBootstrap | ||
| tool.Verify(t, ctx, lowerLayer.FileTree) |
There was a problem hiding this comment.
We may need a way to check if the CAS works.
imeoer
reviewed
Dec 19, 2023
| @@ -240,9 +240,16 @@ impl FileCacheEntry { | |||
| }; | |||
| let blob_compressed_size = Self::get_blob_size(&reader, &blob_info)?; | |||
|
|
|||
| // Turn off chunk deduplication in case of tarfs. | |||
| let cas_mgr = if is_tarfs { | |||
There was a problem hiding this comment.
Maybe give a warnning message about conflict feature here.
| @@ -208,11 +209,18 @@ impl FileCacheEntry { | |||
| } else { | |||
| reader.clone() | |||
| }; | |||
| // Turn off chunk deduplication in case of cache data encryption is enabled or is tarfs. | |||
| let cas_mgr = if mgr.cache_encrypted || mgr.cache_raw_data || is_tarfs { | |||
There was a problem hiding this comment.
Maybe give a warn message about conflict feature here.
| If a record with the same chunk digest already exists, it will be reused. | ||
| We call such a system as CAS (Content Addressable Storage). | ||
|
|
||
| ## Chunk Deduplication by Using CAS as L2 Cache |
There was a problem hiding this comment.
This still seems to be an experimental feature, do we still need to consider the cas.db record recycling?
|
Hi all, I tried out this feature and it seems to work as expected. Is there something preventing it from being merged? |
cc @jiangliu any updates we can continue? :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Details
This PR enhances
nydusdto support runtime chunk deduplication. It works in this way:copy_file_range().So there are two types of chunk deduplication:
copy_file_range()will optimize to use reference instead of data copy, thus reduce local storage consuption.Types of changes
What types of changes does your PullRequest introduce? Put an
xin all the boxes that apply:Checklist
Go over all the following points, and put an
xin all the boxes that apply.