Add new rows in existing lance dataset

[jrabary]

Is it possible to add new element in existing lance dataset ? Example, in case we want to annotate existing image dataset.

[wjones127]

If you are looking to add new columns to existing rows, you can use the merge() method.

For example, if I have an existing dataset with columns x and y, I can add a z column by providing a new dataframe that has x and z to the merge method. This will join based on the x column and save the new y column in the appropriate rows.

>>> import lance
>>> import pyarrow as pa
>>> df = pa.table({'x': [1, 2, 3], 'y': ['a', 'b', 'c']})
>>> dataset = lance.write_dataset(df, "dataset")
>>> dataset.to_table().to_pandas()
   x  y
0  1  a
1  2  b
2  3  c
>>> new_df = pa.table({'x': [1, 2, 3], 'z': ['d', 'e', 'f']})
>>> dataset.merge(new_df, 'x')
>>> dataset.to_table().to_pandas()
   x  y  z
0  1  a  d
1  2  b  e
2  3  c  f

Does that look like it will work in your use case?

[jrabary]

Thanks for your reply.
We are more it the case where we have a table with fixed columns and want to append a new row or modify an existing row.
Also, I saw somewhere in the documentation that lance has a versioning feature. Is this kind of modification can be handle with Lance.

[wjones127]

We are more it the case where we have a table with fixed columns and want to append a new row or modify an existing row.

Thanks for the clarification. We support appending data, but not updating rows yet. We will be releasing an update soon that will allow deleting rows, so at that point you will be able to delete rows and insert replacements. In a few weeks from now, we plan to start work on built-in row update support.

[jrabary]

I've just found this from the doc so append seems to be handled. Remains, how to update a row.

df = pd.DataFrame({"a": [10]})
tbl = pa.Table.from_pandas(df)
dataset = lance.write_dataset(tbl, "/tmp/test.lance", mode="append")