Feature: SQLite WASM builds

[jlarmstrongiv]

Describe what you are looking for

Run usearch extensions for SQLite in the browser with SQLite wasm like https://github.com/nalgeon/sqlean.js

Can you contribute to the implementation?

• I can contribute

Is your feature request specific to a certain interface?

Other bindings

Contact Details

Ping me in Discord

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[ashvardanian]

I think we should update the release.yml to use WASI SDK 21 over the currently used 20. Moreover, we should avoid using the SDK directly, and instead pass it as a CMake toolchain, as documented.

[jonathanpv]

If we get this, my app for local rag would be solved. Although am concerned about data persistence, my use case is users upload textbooks and can perform vector similarity on individual textbooks instead of lets say a user uploads 10,000 textbooks, my project atm will do textbook by textbook to avoid overloading the memory.

How would 1: data persistance and 2: memory be handled with wasm + sqlite + usearch based approach?

Conceptually I would like something like

Query: who are the authors
Textbook: Calculus textbook selected, the sqlite id is textbook_id_here
Sqlite: target the table textbook_id_here and query it using unum search

I've read OPFS + wasm may be a great solution to this but in general will the entire db will all be populated in memory. There seems to not be a single non-memory solution to vector search in browser. Disk-ANN works in C and other languages, im assuming solely due to file system and non browser restrictions. However, I assume a browser-js native implementation can work complete with hnsw and vector search but without multithreading as i believe multithread is not possible in browser-js. Given OPFS is files and supports reading the bytes its basically gives freedom to talk to unum indexes, avoiding the posisblitiy of using wasm all together.

Similar projects I've found are here:
https://github.com/askorama/orama

They allow client based vector search by being js native
https://github.com/askorama/orama/blob/main/packages/docs/open-source/usage/search/vector-search.md#L5L102

And heres how they persist data
https://github.com/askorama/orama/blob/44836b3f2132061b907015f18bea334f9dd4478b/packages/docs/open-source/plugins/plugin-data-persistence.md#L5L104

However for sqlite approach theres also this repo that saves sqlite
storage in indexedDB (which apprently OPFS supersedes in performance but heres the repo nonetheless)
https://github.com/jlongster/absurd-sql

Theres also work by the official sqlite team to address this persisting storage
https://sqlite.org/wasm/doc/trunk/persistence.md

Given this information perhaps we can just have native js browser based OPFS solution to query unum files index.unumsearch calculus.unumsearch
and my app can just be query("my calculus question", opfs.file(namehere)) on unum file per textbook

Interested in more discussion here

[jlarmstrongiv]

While it’s still being developed, it appears this library https://github.com/asg017/sqlite-vec will support vector search with sqlite in wasm. I hope usearch sqlite extensions support wasm in the future too!