Refreshing the database is so slow.

[formlessdao]

Hi so it seems your app is the best we have on Linux that is similar to Everything from VoidTools.
Sadly when refreshing the database when I start the app it takes very long to refresh and that is way inferior to how fast Everything works on Windows.
Why is it so much slower?
Also how good is NTFS support?

[cboxdoerfer]

Hi,

Sadly when refreshing the database when I start the app it takes very long to refresh and that is way inferior to how fast Everything works on Windows.
Why is it so much slower?

Everything achieves the fast indexing speeds by using the MFT (Master File Table) to quickly iterate over every file/folder on disk and the USN Journal to keep the index up to date. On Linux filesystems there's nothing like a MFT or USN journal, so FSearch needs to traverse the filesystem while indexing, which is much slower, especially on spinning drives. There's some room of improvement in FSearch, especially for really fast SSDs, which might benefit from parallel indexing, but it'll never reach the speed of Everything, unless the Linux kernel adds something similar to the MFT or USN, which I doubt.

Also how good is NTFS support?

There's no special NTFS support, it's as good or bad as the filesystem driver you're using. Most likely you're using the NTFS-3G FUSE driver, which is quite slow, so indexing times will be even worse than with other filesystems. However recently a new NTFS driver was added to the Linux kernel, which is supposedly much faster, but I haven't tried it yet.

[formlessdao]

Hi,

Sadly when refreshing the database when I start the app it takes very long to refresh and that is way inferior to how fast Everything works on Windows.
Why is it so much slower?

Everything achieves the fast indexing speeds by using the MFT (Master File Table) to quickly iterate over every file/folder on disk and the USN Journal to keep the index up to date. On Linux filesystems there's nothing like a MFT or USN journal, so FSearch needs to traverse the filesystem while indexing, which is much slower, especially on spinning drives. There's some room of improvement in FSearch, especially for really fast SSDs, which might benefit from parallel indexing, but it'll never reach the speed of Everything, unless the Linux kernel adds something similar to the MFT or USN, which I doubt.

Also how good is NTFS support?

There's no special NTFS support, it's as good or bad as the filesystem driver you're using. Most likely you're using the NTFS-3G FUSE driver, which is quite slow, so indexing times will be even worse than with other filesystems. However recently a new NTFS driver was added to the Linux kernel, which is supposedly much faster, but I haven't tried it yet.

I'll be honest, I would be willing to use only NTFS partitions on my current Linux distro for all my personal files(only my linux instalation would be EXT4) if you could implement the same type of search(MFT+USN) as available on Windows.

[cboxdoerfer]

I'll be honest, I would be willing to use only NTFS partitions on my current Linux distro for all my personal files(only my linux instalation would be EXT4) if you could implement the same type of search(MFT+USN) as available on Windows.

I don't use NTFS partitions, so adding support for MFT and USN is not something I'm personally interested in. And since I doubt that people would be willing to pay me enough to do it, the only hope is that someone else implements it. Patches are of course welcome.

[formlessdao]

I'll be honest, I would be willing to use only NTFS partitions on my current Linux distro for all my personal files(only my linux instalation would be EXT4) if you could implement the same type of search(MFT+USN) as available on Windows.

I don't use NTFS partitions, so adding support for MFT and USN is not something I'm personally interested in. And since I doubt that people would be willing to pay me enough to do it, the only hope is that someone else implements it. Patches are of course welcome.

What filesystem has the best ability in finding the file I'm looking for the fastest among many files, in my case between 200.000-300.000 files?
Do you offer the same kind of support to all filesystems?
XFS,ZFS,EXT4...do you optimize how Fsearch works just as well on all of them or did you just choose one filesystem to optimize for?

[formlessdao]

I'll be honest, I would be willing to use only NTFS partitions on my current Linux distro for all my personal files(only my linux instalation would be EXT4) if you could implement the same type of search(MFT+USN) as available on Windows.

I don't use NTFS partitions, so adding support for MFT and USN is not something I'm personally interested in. And since I doubt that people would be willing to pay me enough to do it, the only hope is that someone else implements it. Patches are of course welcome.

What filesystem has the best ability in finding the file I'm looking for the fastest among many files, in my case between 200.000-300.000 files?
Do you offer the same kind of support to all filesystems?
XFS,ZFS,EXT4...do you optimize how Fsearch works just as well on all of them or did you just choose one filesystem to optimize for?

[cboxdoerfer]

What filesystem has the best ability in finding the file I'm looking for the fastest among many files, in my case between 200.000-300.000 files?

The filesystem has no impact on the search performance, because the database is stored in RAM. However the filesystem has an impact on the time it takes to load the database when you launch FSearch and when the database gets updated, but unless you're using a FUSE or network filesystem the difference probably won't be noticeable.

Do you offer the same kind of support to all filesystems?

Yes.

XFS,ZFS,EXT4...do you optimize how Fsearch works just as well on all of them or did you just choose one filesystem to optimize for?

There's no filesystem in particular FSearch was optimized for, it's supposed work efficiently on all of them.

[gorman42]

On Linux filesystems there's nothing like a MFT or USN journal, so FSearch needs to traverse the filesystem while indexing, which is much slower, especially on spinning drives. There's some room of improvement in FSearch, especially for really fast SSDs, which might benefit from parallel indexing, but it'll never reach the speed of Everything, unless the Linux kernel adds something similar to the MFT or USN, which I doubt.

Hi! Another ex-Windows user that came upon your program after searching for an Everything alternative. I have read this discussion, but it dates back to more than a year ago.

In the meantime I see on your roadmap that you plan on including inotify support. I briefly read about it and I am wondering how close that would bring fsearch to how Everything works under Windows. Would inotify support work with a background service always running? I hope so and, most of all, I hope you are still planning on working on it.

Thanks for your program!

[cboxdoerfer]

Hi, yes I'm currently working on the inotify support. The most important differences between it and Everything (with MFT and USN) are:

• Before the monitoring can start all the to-be-monitored folders need to be re-indexed (this needs to be done every time FSearch is launched)
• The monitoring requires more resources (RAM and CPU time)
• There's a configurable limit to the amount folders which can be monitored (fs.inotify.max_user_watches)
• There are some rare race conditions which can't be avoided, which means in some cases file system events will be missed (and the database index will become inconsistent, which will trigger a rescan)

Would inotify support work with a background service always running?

Yes, that's possible and will be implemented. However, the background service needs to perform a rescan before it can start monitoring the file system as well.

I hope so and, most of all, I hope you are still planning on working on it.

Yes, if you're curious you can try out the current progress; you can find it in the wip/fs_monitoring branch. Inotify support is already working there, but database saving/loading is disabled for the moment, since the database format is being worked on so it can support configurable index properties (size, creation time, ...).

[gorman42]

Well, a bunch of good news. Great!

I am not exactly sure about the implementation you are describing, so I'll ask some more questions. Hoping not to bother you too much.

When you say before monitoring can start a re-index is necessary, do you mean every time the machine is rebooted? Since you say every time FSearch is launched. Or is it just for the first time and then the index is saved somehow?

I am not well versed in Linux. Don't know how to go about testing a different branch. My main problem, currently, is that I had a rather complex home server configuration under Windows. I have managed to replicate it in Linux Mint, even adding functionalities I had postponed for the switch. But now I risk forgetting how I did what.

So, for instance... in FSearch case I see I added your fsearch-daily PPA. And it works fine. I fear I would mess up things by switching to a different branch (which I suppose would amount to adding a different branch). For instance, can the two versions coexist?

Thanks again for developing this, it's an incredible time saver for me (as Everything was under Windows).

[cboxdoerfer]

I am not exactly sure about the implementation you are describing, so I'll ask some more questions. Hoping not to bother you too much.

Not at all.

When you say before monitoring can start a re-index is necessary, do you mean every time the machine is rebooted? Since you say every time FSearch is launched. Or is it just for the first time and then the index is saved somehow?

Yes, every time FSearch is launched, so at least once per reboot. The reason for that is that inotify only provides file system events while FSearch is running, hence all files/folders which were created/modified/deleted before or after FSearch was running need to be detected with a rescan instead. Everything can avoid those rescans due to the USN Journal, but unfortunately on Linux there's nothing like that.

So, for instance... in FSearch case I see I added your fsearch-daily PPA. And it works fine. I fear I would mess up things by switching to a different branch (which I suppose would amount to adding a different branch). For instance, can the two versions coexist?

At the moment they can coexist, because there aren't any breaking changes to the config file and database format yet, but that will change soon. So unless you don't mind any future breakage, you should probably stick to the daily PPA for now. Eventually those changes will be added to the daily PPA anyway, once they've became more stable.

Thanks again for developing this, it's an incredible time saver for me (as Everything was under Windows).

Thanks, I'm glad you like it. :)

[gorman42]

Ok. All clear. I'll follow your suggestion and stick to the daily PPA.
I'll wait for the work to be completed and I commend you for further developing this project. :-)

[Nekkowe]

Oh, just to clarify, does that mean that after both the background service and inotify support are implemented, it'll go like this -

1. Reboot
2. Background service is launched automatically at startup, reindexes all files, and watches for further changes with inotify
3. User launches fsearch once they need to find something
4. Since the index for all files is already kept up-to-date by the background service, fsearch is immediately ready to retrieve results?

That sounds wonderful, frankly - the long wait to index everything after opening fsearch is the one hangup I've been having with it.

[q2333gh]

this post explains some of it .
https://unix.stackexchange.com/questions/31063/is-there-a-file-search-engine-like-everything-in-linux?answertab=modifieddesc#tab-top

[atrugiel69]

I'm using 0.3.alpha0, PPA-nightly. I have noticed refreshing the database has become noticeably faster. Is this something that changed in Fsearch or should I be looking at other parts of my system?

Some other things have changed in my system that might influence it but... it seems too much faster for being an external cause.

Edit: I thought about asking because I have received several updates for Fsearch, recently.