-
Notifications
You must be signed in to change notification settings - Fork 211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bug(python): update function is not accepting scalars #1228
Comments
The patch here is very simple, and could even be done in user code: from lancedb.util import value_to_sql
@value_to_sql.register(pa.Scalar)
def _(value: pa.Scalar) -> str:
return value_to_sql(value.as_py()) |
Here is a test case from typing import List
import ollama
import lancedb
from lancedb.pydantic import LanceModel, Vector
def embeddings(data, model = "mxbai-embed-large") -> List[float]:
embed = ollama.embeddings(model,data)
return list(embed["embedding"])
HELLO = embeddings("hello")
class Doc(LanceModel):
filename: str
text: str
vector: Vector(len(HELLO))
def update(self, text):
self.text = text
self.vector = embeddings(text)
def newdoc(text,filename):
return Doc( filename=filename, text=text, vector=embeddings(text) )
def docsfrom(docs):
return [
Doc(
text=docs["text"][idx].as_py(),
filename=docs["filename"][idx].as_py(),
vector=docs["vector"][idx].as_py().copy(),
)
for idx in range(len(docs))
]
db = lancedb.connect('./lancedb')
if "test_table" in db.table_names():
db.drop_table("test_table")
table = db.create_table('test_table', schema=Doc)
data = [newdoc("Hello world", "doc1"),
newdoc("Goodbye world", "doc2")]
table.add(data,mode="overwrite")
found = docsfrom(table.search(embeddings("Hello world")).limit(1).to_arrow())[0]
found.update("This is a list of integer 10,11,12")
criteria=f"filename==\"{found.filename}\""
table.update(criteria, {"vector":found.vector}) |
added import pyarrow as pa
@value_to_sql.register(pa.Scalar)
def _(value: pa.Scalar) -> str:
return value_to_sql(value.as_py()) Still get |
I think, for pydantic support, we need to handle whatever |
Indeed. It's sealed inside of a closure, so we need to refactor that to allow the conversion. |
Bizarre, the following breaks def embeddings(data, model = "mxbai-embed-large") -> List[float]:
embed = ollama.embeddings(model,data)
real = list(embed["embedding"])
fake = [0.0] * len(real)
# comment out copy and works
for i in range(len(real)):
fake[i] = real[i]
return fake |
LanceDB version
v0.6.7
What happened?
I want to use
update
to change the value of a fixed size list. If I pass a list of python floats as a value then it works. If I pass aFixedSizeListScalar
as a value then it does not work. Supporting this latter case seems useful sinceFixedSizeListScalar
is very easy to get from querying the database (see reproduction).Are there known steps to reproduce?
The text was updated successfully, but these errors were encountered: