[Feature Request]: Auto-save during indexing

[GownerCode]

Feature Description

I suggest implementing an autosave feature:

embeddings.index(data, autosave={"interval": 3600, "save_path": "/home/user/index"})

Something like this should save the index every interval seconds to save_path.

Reason

When one processes a large dataset, indexing can take a long time. The naive approach:

embeddings.index(my_data)
embeddings.save(my_save_path)

embeddings.index(...) takes a long time. When working with a database or other network dependent data retrieval, something may go wrong during the index call which means the save call is never reached, all progress is lost.

Value of Feature

It would allow for the ability to continue an index that has failed for reasons other than bad data.

[davidmezzetti]

Thank you for this idea. I'll take a look.

Would it make sense to save per number of records vs time? Most checkpoints I'm familiar with tend to be based on data volume.

[GownerCode]

You're right, data volume makes more sense. Thanks for looking into it!

[dustyatx]

I'd yes and this.. It would be better if we can treat it like a check point and in the event that the indexing fails, be able to restart it from the last check point. I've had issues where there were non-strings in a field that was trying to create vector index on and it breaks after many hours of running. It would be great to be able to identify the error, correct any issue with the data and start the indexing from the last check point position.