Modifying an embedder should not always recompute all the vectors

[ManyTheFish]

Related product team resources: PRD (internal only)

⚠️ this issue depends on #4480 to be implemented

Summary

This issue is a subset of the work implementing the settings diff-indexing enhancement.

When adding or changing an embedder, the whole vector store is erased to be recomputed totally during indexing.
It should be possible to avoid that erase and only compute/delete the modified data to save computing time.

⚠️ The main goal of this issue is to minimize the number of vectors to recompute during indexing. Erasing the DB to recompute it is sometimes the best approach because computing a difference implies computing twice the number of vectors for the data.

Current implementation

When modifying the settings, the first step is to compute the changes in the embedding settings, in the current implementation, we erase the database if any changes imply any modification in the vector store.
Then later in the indexing process, the embeddings pipeline is ran if any changes has been made in the embeddings settings, this relies on the InnerSettingsDiff structure that should be enhanced to be able to compute more precisely the changes in the embedders.
In this extraction pipeline, only the new settings are used, because the database is currently erased so no need to compute the old settings in order to remove a part of the data, but this pipeline should be reworked to support both settings versions.

TODO

• Changes to 1 embedder should not trigger the reindexing of all embedders. Only the modified embedder should be reindexed
  • In the case where one embedder is deleted, the code currently benefits from the "reindex everything" behavior in that the associated index of embedders after the deleted embedder is modified to not leave a "hole" in the list of embedders. A free list is required to support incremental deleting of embedder.
• Currently, changes to searchable, displayable, sortable, filterable, etc. do not need to trigger a reindexing. In the future, the properties of a field might be accessible to the document template, madking reindexing mandatory in this case. (related to Create SettingsDiff structure and run extractions based on it #4480)
• A change to the apiKey (for OpenAI models) should not trigger a reindexing operation of the modified embedder.
• When modifying the documentTemplate, a reindexing is necessary, but would be more minimal if comparing the rendered versions of both the old and the new template, and only regenerating embeddings for documents where the rendered version actually changed.
• Future extensions:
  1. distributionShift: should not trigger a reindexing operation.
  2. distance: needs reindexing

Related Benchmarks:

• settings-add-embeddings.json
• movies-subset-hf-embeddings.json
• a new benchmark modifying the embeddings may be added