FUSE implementation

[gwik]

Hi there,

I've been using conserve for some time now, thanks for the this great tool.

I started implementing a FUSE mount tool that you can find on crates.io.
It's an early release and there are still many things to do but some people might find it useful.

Cheers,

Antonin

[sourcefrog]

That's great, thanks for sharing it!

There is probably a lot of scope to improve performance of the core Conserve library for the kind of random access this would generate, both by caching and by more efficient traversal to given paths. Maybe also in the fuse wrapper, e.g. by holding the block addresses in the entry you find while listing a file. Of course perhaps performance is fine for casual looking around or retrieval of a few files just as it is.

[gwik]

I didn't focus on performance just yet but I'd like it to be rather efficient.

Maybe also in the fuse wrapper, e.g. by holding the block addresses in the entry you find while listing a file

That's basically how it works. I maintain a "filesystem tree", which is tree index of integers (inode) that maps to a conserve's IndexEntry that contains the block addresses.
Upon traversal the directory children are loaded into the index, so the subsequent operations are very cheap.
For now the entries are always kept in the index after they are loaded but the FUSE api interface would allow to unload them when they are no longer needed.

The least efficient part for now is probably the listing of directory children.
I use the ReadTree::iter_entries method on the StoredTree, passing a subtree.

However, if I understood correctly, passing a substree only allows to skip the entries that doesn't match the prefix.
It seems you already thought about improving this:

conserve/src/index.rs

Lines 453 to 455 in 2418116

	// TODO: We could be smarter about skipping ahead if nothing
	// in this page matches; or terminating early if we know
	// nothing else in the index can be under this subtree.

I'll try to make a patch that does what the comment suggests.

If we want to get this one step further, then one way would be to have a hunk index with low and high apath, maybe in a new header of each hunk, a bit like what parquet does with its row groups, although making it separate is probably way easier. Just thinking out loud...

[sourcefrog]

If we want to get this one step further, then one way would be to have a hunk index with low and high apath, maybe in a new header of each hunk, a bit like what parquet does with its row groups, although making it separate is probably way easier. Just thinking out loud...

Yes, I think that would be a good change too. It would fit naturally with storing the index in hash-addressed blocks, although it could be done separately. In doing this I think it also makes sense to allow those apaths to identify just a sub-range of the index block, so that we can reuse partial index blocks.