-
Notifications
You must be signed in to change notification settings - Fork 175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Roadmap 2024 #2079
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Vector search
We continue to work at making vector search performant at very high scales (10 billion vectors).
V2 File Format
We are working on the next version of our Lance file format. The key improvements are null support in all types, columnar compression, and better performance with large blobs.
Table format
We want to make table maintenance automatic, if opted into. This would make it much easier to manage tables, since you would not need to manually call operations like compact, re-index, and cleanup.
Full text search
We will work on native full text search indices for Lance. LanceDB's current implementation of FTS doesn't support incremental updates or working in Node or Rust. Nor does it work on object storage. Implementing FTS at the Lance level will remove these limitations.
Read / write integration
To make Lance a great choice for AI data lakes, we will be adding connectors to two of the most popular distributed frameworks, Spark and Ray.
Query
Finally, there are smaller features we'd like to implement to improve queries. Providing a general SQL front-end for SELECT queries will provide native support for aggregations as well as provide an easy way to implement table views. Bitmap scalar indices will provide much better filter performance on columns with few distinct values.
sql
method to execute arbitrary SELECT statements_distance
column _distance filter pushdown #2096The text was updated successfully, but these errors were encountered: