bug(python): create_table() Fails on exFAT filesystem on Windows

[turc1656]

LanceDB version

0.6.8

What happened?

I have NTFS and exFAT drives on my Windows 10 device. There seems to be an issue with only the exFAT drives. NTFS has no issues running the same, very basic test code. I have tried using Python 3.10 and 3.12. Both have the same issue.

The error that results is:

OSError: LanceError(IO): Generic LocalFileSystem error: Unable to copy file from H:\LanceDB\lanceTEST\my_table.lance\_versions\.tmp_1.manifest_525df977-88a3-421c-b046-de32a7ebd1d7 to H:\LanceDB\lanceTEST\my_table.lance\_versions\1.manifest: Incorrect function. (os error 1), D:\a\lance\lance\rust\lance-table\src\io\commit.rs:620:54

Are there known steps to reproduce?

In a fresh virtual environment:

pip install lancedb
pip install pandas

Then run this test script with the 'uri' variable pointed to a location on an exFAT formatted drive.

import lancedb


def test_function():
    uri = "H:/LanceDB/lanceTEST"
    db = lancedb.connect(uri)
    table = db.create_table("my_table",
                            data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
                                  {"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
    table = db.open_table("my_table")
    result = table.search([100, 100]).limit(2).to_list()
    result = table.search([2, 5]).limit(2).to_pandas()
    print("Done")


if __name__ == '__main__':
    test_function()

It should fail on the call to db.create_table()

[westonpace]

The crux of Lance's "inter-process transaction model" is the ability to do a "copy if not exists" operation on the filesystem. If two writers are competing to make a change, and both run "copy if not exists", then only one must succeed.

For a local filesystem, this translates to Rust's hard_link function. On Windows, this is implemented with the OS call CreateHardLink and from the docs we can see there are quite a few restrictions:

This function is only supported on the NTFS file system, and only for files, not directories.

which can be different names in the same directory, or the same or different names in different directories. However, all hard links to a file must be on the same volume.

Note that SMB 3.0 does not support creation of hard links on shares with continuous availability capability.

It sounds like hard links are not supported on the exFAT filesystem, which explains the error you are getting. Unfortunately, this does mean that you will need to find some other mechanism for transaction control. This is the same situation we have in S3 which also does not support an atomic "copy if not exists" operation. We have, in lance (not yet in lancedb), the ability to specify a custom commit handler to work around this limitation (e.g. we often use dynamodb + s3 so that dynamodb can provide the inter-process atomic commit and s3 provides the storage). I think the solution will have two parts:

• Either we need to enhance our detection capabilities so that we recognize this is an unsupported operation and fallback to the unsafe commit handler automatically or we need to allow users to specifically request the unsafe commit handler in some kind of advanced config setting.
• We need to expose the commit handler to lancedb (feat: enable dynamodb commit in python sdk #534 is related)

[fcakyon]

Do you have any plans to fix this issue @westonpace? LanceDB fiftyone integration doesn't work because of this problem: voxel51/fiftyone#4404

[westonpace]

@fcakyon I think, at a minimum, exposing the commit handler to lancedb would be a good idea, as this would allow users to supply their own external locking mechanism. This is on our 2024 roadmap but I don't have any immediate plans to work on it.