Welcome to CompactGUI Discussions!

[Iridium-IO]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

[Uranite]

Can't wait for v3

[Vitaliuz]

v3 isn't configurable (to add/remove poorly compressed file types, ect), you can't put it in the right-click menu (using archaic "select a folder" in 2022 is absurd, and there's a bunch of GUIs already that can't use the RMB menu or drag-and-drop as well - Compactor, for example), and it's causing a ton of slowdown due to a lot (16, in my case, for example) of parallel write operations on top of that (i.e. isn't mechanical HDD friendly, and can easily kill a cheap SSD/NVMe with a weak controller). And you can't even compress the folder once again (for example, if the 5 out of 100 files have been modified, so only those 5 files will be compressed, which happens to, you know ... Steam games after an update) - it just proudly shows me the percentage saved.
And the styling of the main window (what's left of it, that is) is kinda weird - I thought the GUI was aimed at PC's with screens in landscape mode, not portrait.

Thanks, but I'll stick to v2.6.2, which doesn't have any of these downsides.
As with this kind of "improvements" - the fate of the GUI is ... imminent.

[Iridium-IO]

v3 isn't complete yet - did you notice the alpha tag?

• Context menu integration is coming back
• drag and drop is coming back
• HDD performance is about on par with 2.6.2 but I am going to limit the thread count to 1 for mechanical drives anyway - I take your point there actually, the more small files there are the slower a HDD will be when trying to run this. I'll fix it!
• The 16 processes you see very likely aren't all writing to the drive at once, the GC doesn't collect them instantly when the compact function is completed. Nonetheless, even a basic SATA SSD's controller is capable of at least 32 "simultaneous" operations - however I believe everything passed through Windows is handed off sequentially to the controller anyway (the simultaneous part comes from the controllers queue capabilities and not a true parallel write, SSDs batch and cache writes to minimise unnecessary amplification). Benchmarking software such as CrystalDiskMark hit it a lot harder, and because of how many process calls are made, this program is CPU limited anyway as soon as you try to compress many small files in a row. Run the native compact.exe function on a whole folder and you'll see that CompactGUI doesn't complete actions any faster than it does

[kwencel]

I haven't used v3 yet, but if it still invokes compact.exe under the hood, I would suggest leaving that idea behind and use the Windows API directly. There is a project called Compactor which does exactly that, however it lacks context menu integration like CompactGUI. It's compression speed is much greater because it does not have the overhead of spawning a compact.exe instance for every file it compresses.

[Iridium-IO]

It's something I've wanted to do for ages (I've been following this thread) but I hate using PInvoke especially in vb.net, there's a lot of stumbling blocks that don't exist in C# and other languages which makes it slow going.
I had a test piece built 2 years ago which kinda worked, but because it was in a scrap project file I never took the time to make it legible so when I checked it before rewriting v3, it was a mess. I'm slowly piecing it back together again.

However, you might find that v3 works almost as fast as just compressing via compact.exe anyway :). By spawning parallel processes the overhead of each process.Start() is somewhat mitigated as there's always a process (or several) actively compressing while another loads. The main bottleneck comes from rapid loading of tiny files, which v3 gets around by skipping files smaller than the disk's allocation size as they won't be compressible (by default 4kb).

I will also say Compactor's creator is a far more talented programmer using a better language too :)

[Iridium-IO]

Okay so I was bored today.

tl;dr

expect to see a new alpha using the Windows API soon.
I had grossly overestimated how hard it would be to use (I thought I'd have to be responsible for rewriting each file safely, which I don't have to do!)

The long version:

I just spent the last 6ish hours trying to get this working.
I thought I'd try reading the file data first, to avoid potential damage by a failed write. Seemed like a reasonable approach right? Wrong.
So, Microsoft provides this documentation for the read entry point of WofAPI. Looked pretty straightforward, so I cobbled together this DLLImport:

    <DllImport("WofUtil.dll")>
    Public Function WofIsExternalFile(Filepath As String) As Integer
    End Function

Rather than using all the optional out parameters, I figured I'd get the main bit working first.
Seems simple, right? Well, this is the bit that stumped me for most of the last 5 hours. At first I thought it was because the entrypoint expected an LPCWSTR instead of a regular string. But no amount of marshalling would fix the error I kept getting:

system.accessviolationexception: attempted to read or write protected memory

I was ready to tear my hair out.
Wait! If the file is not compressed, it works and returns 0 as expected. But if the file is compressed, I get the above error all over again.
At this point I started thinking - maybe it needs unsafe memory access (or something, what did I know) to find the sparse file?
Maybe I should write a library in C# that I can then call across?

1. I've never used C# before
2. That didn't work anyway, I couldn't even get it to work on an uncompressed file

So now I knew it wasn't working in C# either, and that's Microsoft's favourite child (not the cripple that they've left VB become).

Maybe the problem wasn't with me, but with the docs?

And here lies the problem:
 

 

You see those four unassuming, innocent words next to each parameter that says optional?

OPTIONAL MY ARSE

All that time wasted, and all I had to do was use every parameter and it works fine

And then I turned to the actual problem (so I thought) of how to safely recreate a file as sparse.
This is a literal non-issue. I was just looking at the wrong docs. That code I was scratching up two years ago? Yeah it was recreating the existing file in chunks of 4K/8K/16K/32K manually depending on which compression mode you chose, and replacing the original. Unsafe as hell. Completely unnecessary.
So now I have a working version of CompactGUI that doesn't use compact.exe, I just need to test it for safety (for example working out where the force and ignore flags come from in compact.exe

[Iridium-IO]

@kwencel try out the new release :)

[Werve]

Hello,
I want to share some ideas, since the part of the exclusion list by file type is not yet added wouldn't a dynamic system based on the entropy value of the input file be better (making the solution more universal than an extension list that maybe doesn't include that particular one or that file with wrong extension).
The threshold value could be based on community feedback (as is the case with the game database).
Or a system (but it would greatly increase the total compression time) that decompresses the compressed file if at a certain percentage (20, 50 ?) of compression of the file it has not reached a user-defined compression threshold.
So as to make the process self-assessing the compressibility of the source file.

[Iridium-IO]

wouldn't a dynamic system based on the entropy value of the input file be better

It would indeed, but I wouldn't know how to make that accurate, seems to be a lot of issues around that online as it is.
An alternative would be to grab a few random chunks out of each file and do a mini "test run" compression on them?

decompresses the compressed file if at a certain percentage (20, 50 ?) of compression of the file it has not reached a user-defined compression threshold.
should be doable, but actually compressing through the windows API does automatically skip files pretty quickly - it tries briefly and if a certain threshold isn't crossed it aborts the compression.

[Werve]

It would indeed, but I wouldn't know how to make that accurate, seems to be a lot of issues around that online as it is. An alternative would be to grab a few random chunks out of each file and do a mini "test run" compression on them?

Regarding entropy https://github.com/horsicq/Detect-It-Easy/ tool can calculate the entropy and says if the file is likely compressed,
but yes i think that a chunks compression test should be enough to reduce false positive (good file candidates).