Can DuckDB pushdown list_cosine_similarity and list_distance functions to LanceDB?

[dmackenthun]

Hi all,

Are there any vector search functions like list_cosine_similarity and list_distance that can be pushed down from DuckDB into LanceDB? I have a use case where I would like to do a similarity search within the context of a DuckDB SQL query. LanceDB is an intriguing option for persisting embeddings and performing semantic search, however, I need to be able to execute complex joins, nested queries, and filters within the same query.

Is there a different way to do this type of search in a DuckDB query that references a LanceDB table?

[wjones127]

We don't support pushdown like that, but you could define a Python UDF in DuckDB and do the computations in DuckDB.