Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENG-349] Archive/Zip #2356

Open
brxken128 opened this issue Apr 19, 2024 · 10 comments
Open

[ENG-349] Archive/Zip #2356

brxken128 opened this issue Apr 19, 2024 · 10 comments
Labels
core Feature Created by Linear-GitHub Sync Urgent Created by Linear-GitHub Sync

Comments

@brxken128
Copy link
Contributor

We need a job to archive file(s) and directories. Currently we aren't able to multi-select items within the explorer, so I'll design this job around one object or directory (like the others). There are some thoughts about implementing multi-select support for FS jobs below¹.

We can offer ZSTD and possibly a few other compression algorithms (and levels!), with the zip crate. I have experience with this crate already, and I feel as though it's better than the alternatives. There are a few quirks for Windows machines (mostly \ issues) that I'll have to ensure are thoroughly tested.

Eventually I feel as though we should design our own zip crate - it's not the easiest of tasks but we'd definitely benefit from it in the long-run.

We will also have to implement zip-slip protection for unzipping.

Additionally, we need to ensure that the supported compression algorithms won't cause any build issues, as I believe some of them are as annoying as FFmpeg - I'll need to double check this though.

¹ As a further note, multiple input/source object support probably be achieved by taking a Vec<{path_id}> and iterating through them to create steps for the job. We can use this for copy, erase, cut, delete and pretty much all actions due to the job system.

From SyncLinear.com | ENG-349

@brxken128 brxken128 added core Feature Created by Linear-GitHub Sync Urgent Created by Linear-GitHub Sync labels Apr 19, 2024
@brxken128
Copy link
Contributor Author

Agreed with @ericson.ds999, we don't need to broke this job to steps, it can be a 1 big bulky zip and unzip job that does it thing through the library. I don't think pausing it or remembering where it is makes sense here too. Let's not make this complicated and keep it as simple as possible. It just needs work.

@brxken128
Copy link
Contributor Author

I don't think that this job can be broken in steps at all. Steps are only needed when we can interrupt the job to be resumed later, which isn't the case. But the new task system enables periodically pauses with a mutable state between them. Without the restriction of being needed to be serializable. On a compression task we would be able to pause it in memory, but if the app was closed, it would need to be started from the beginning.

@brxken128
Copy link
Contributor Author

Oh sorry I missed that part, thanks @brxken128! Do you know about this @ericson.ds999?

@brxken128
Copy link
Contributor Author

and the current job system doesn't allow for a mutable object to be shared between job steps (which is what the zip crate would require)

@brxken128
Copy link
Contributor Author

Why is this blocked?

@brxken128
Copy link
Contributor Author

Yeah, we still need zip-slip protection as well as special path handling on Windows (most likely), and the current job system doesn't allow for a mutable object to be shared between job steps (which is what the zip crate would require).

@brxken128
Copy link
Contributor Author

As mentioned on Slack, plan is to use a already ready zip crate

@brxken128
Copy link
Contributor Author

Might not be blocked but not a priority atm.

@brxken128
Copy link
Contributor Author

I'm pretty sure this is still not possible given the current state of the job system. I will look further into it, but we need a mutable state which is not serialised. Maybe a serde(skip)will work, but the job will not be resumable at all.

@brxken128
Copy link
Contributor Author

I'm currently unable to continue with this as we're not able to mutably share objects between job steps, when the objects in question cannot be serialized/deserialized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Feature Created by Linear-GitHub Sync Urgent Created by Linear-GitHub Sync
Projects
None yet
Development

No branches or pull requests

1 participant