-
Notifications
You must be signed in to change notification settings - Fork 33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to optimize config for a huge movie collection track ? #48
Comments
Interesting to hear your experiences. I have indeed tested Olaf with 800GB of music and have not experimented with larger datasets or non-music (less information rich) signals. Some insights/pointers/possible optimisations:
Good luck with your project |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I rewrite the ruby layer to request only the C code, I use the default (stock) config to store cached acoustic fingerprint to a custom directory with a php CLI only script (only exec() line can interest you) :
(TMP operation on tmpfs RAM)
And to build the B+ tree I use a loop to load all ~1h40/2h (average movie duration) audio tracks :
I can generate all my audio track fingerprint from movies, all gz are lightweight, but integration into B+ tree part is too slow and db become too huge. even on my i9 64GB DDR5 / PCIE SSD machine :(
How you can store 340 days of audio (around 800GB of mp3s) inside a 15GB database ? mine grow way faster than this with your default config https://github.com/JorenSix/Olaf/blob/master/src/olaf_config.c .... and I have about 4000 days (10 years !) of sound to index !!!!! game over lol But with an estimation based on your result, this must enter inside 150GB database, good for me, but this is not the case (I grow rapidly over multiples terabytes and show as a snail to rebuild the B+ tree from all lightweight fingerprints.csv.gz)
The text was updated successfully, but these errors were encountered: