Avoid regenerating embeddings at dump import #4606
Labels
experimental feature
Related to an experimental feature
impacts docs
This issue involves changes in the Meilisearch's documentation
impacts integrations
This issue involves changes in the Meilisearch's integrations
missing usage in PRD
Description of the feature usage is missing in the PRD
performance
Related to the performance in term of search/indexation speed or RAM/CPU/Disk consumption
Milestone
Comments
curquiza
added
impacts docs
curquiza
changed the title
curquiza
added
the
performance
meili-bors bot
added a commit
that referenced
this issue
May 22, 2024
4633: Allow to mark vectors as "userProvided" r=Kerollmops a=dureuill # Pull Request ## Related issue Fixes #4606 ## What does this PR do? [See usage in PRD](https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697#deb96fb0595947bda7d4a371100326eb) - Extends the shape of the special `_vectors` field in documents. - previously, the `_vectors` field had to be an object, with each field the name of a configured embedder, and each value either `null`, an embedding (array of numbers), or an array of embeddings. - In this PR, the value of an embedder in the `_vectors` field can additionally be an object. The object has two fields: 1. `embeddings`: `null`, an embedding (array of numbers), or an array of embeddings. 2. `userProvided`: a boolean indicating if the vector was provided by the user. - The previous form `embedder_or_array_of_embedders` is semantically equivalent to: ```json { "embeddings": embedder_or_array_of_embedders, "userProvided": true } ``` - During the indexing step, the subfields and values of the `_vectors` field that have `userProvided` set to **false** are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template. - This allows **importing** embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change. - The dump process now uses this ability: it enriches the `_vectors` fields of documents with the embeddings that were autogenerated, marking them as not `userProvided`. This allows importing the vectors from a dump without regenerating them. ### Tests This PR adds the following tests - Long-needed hybrid search tests of a simple hf embedder - Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content. - Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected Co-authored-by: Louis Dureuil <[email protected]>
meili-bors bot
added a commit
that referenced
this issue
May 22, 2024
4633: Allow to mark vectors as "userProvided" r=Kerollmops a=dureuill # Pull Request ## Related issue Fixes #4606 ## What does this PR do? [See usage in PRD](https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697#deb96fb0595947bda7d4a371100326eb) - Extends the shape of the special `_vectors` field in documents. - previously, the `_vectors` field had to be an object, with each field the name of a configured embedder, and each value either `null`, an embedding (array of numbers), or an array of embeddings. - In this PR, the value of an embedder in the `_vectors` field can additionally be an object. The object has two fields: 1. `embeddings`: `null`, an embedding (array of numbers), or an array of embeddings. 2. `userProvided`: a boolean indicating if the vector was provided by the user. - The previous form `embedder_or_array_of_embedders` is semantically equivalent to: ```json { "embeddings": embedder_or_array_of_embedders, "userProvided": true } ``` - During the indexing step, the subfields and values of the `_vectors` field that have `userProvided` set to **false** are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template. - This allows **importing** embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change. - The dump process now uses this ability: it enriches the `_vectors` fields of documents with the embeddings that were autogenerated, marking them as not `userProvided`. This allows importing the vectors from a dump without regenerating them. ### Tests This PR adds the following tests - Long-needed hybrid search tests of a simple hf embedder - Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content. - Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected Co-authored-by: Louis Dureuil <[email protected]>
meili-bors bot
added a commit
that referenced
this issue
May 22, 2024
4633: Allow to mark vectors as "userProvided" r=Kerollmops a=dureuill # Pull Request ## Related issue Fixes #4606 ## What does this PR do? [See usage in PRD](https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697#deb96fb0595947bda7d4a371100326eb) - Extends the shape of the special `_vectors` field in documents. - previously, the `_vectors` field had to be an object, with each field the name of a configured embedder, and each value either `null`, an embedding (array of numbers), or an array of embeddings. - In this PR, the value of an embedder in the `_vectors` field can additionally be an object. The object has two fields: 1. `embeddings`: `null`, an embedding (array of numbers), or an array of embeddings. 2. `userProvided`: a boolean indicating if the vector was provided by the user. - The previous form `embedder_or_array_of_embedders` is semantically equivalent to: ```json { "embeddings": embedder_or_array_of_embedders, "userProvided": true } ``` - During the indexing step, the subfields and values of the `_vectors` field that have `userProvided` set to **false** are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template. - This allows **importing** embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change. - The dump process now uses this ability: it enriches the `_vectors` fields of documents with the embeddings that were autogenerated, marking them as not `userProvided`. This allows importing the vectors from a dump without regenerating them. ### Tests This PR adds the following tests - Long-needed hybrid search tests of a simple hf embedder - Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content. - Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected Co-authored-by: Louis Dureuil <[email protected]>
meili-bors bot
added a commit
that referenced
this issue
May 22, 2024
4633: Allow to mark vectors as "userProvided" r=Kerollmops a=dureuill # Pull Request ## Related issue Fixes #4606 ## What does this PR do? [See usage in PRD](https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697#deb96fb0595947bda7d4a371100326eb) - Extends the shape of the special `_vectors` field in documents. - previously, the `_vectors` field had to be an object, with each field the name of a configured embedder, and each value either `null`, an embedding (array of numbers), or an array of embeddings. - In this PR, the value of an embedder in the `_vectors` field can additionally be an object. The object has two fields: 1. `embeddings`: `null`, an embedding (array of numbers), or an array of embeddings. 2. `userProvided`: a boolean indicating if the vector was provided by the user. - The previous form `embedder_or_array_of_embedders` is semantically equivalent to: ```json { "embeddings": embedder_or_array_of_embedders, "userProvided": true } ``` - During the indexing step, the subfields and values of the `_vectors` field that have `userProvided` set to **false** are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template. - This allows **importing** embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change. - The dump process now uses this ability: it enriches the `_vectors` fields of documents with the embeddings that were autogenerated, marking them as not `userProvided`. This allows importing the vectors from a dump without regenerating them. ### Tests This PR adds the following tests - Long-needed hybrid search tests of a simple hf embedder - Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content. - Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected Co-authored-by: Louis Dureuil <[email protected]>
meili-bors bot
added a commit
that referenced
this issue
May 22, 2024
4633: Allow to mark vectors as "userProvided" r=dureuill a=dureuill # Pull Request ## Related issue Fixes #4606 ## What does this PR do? [See usage in PRD](https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697#deb96fb0595947bda7d4a371100326eb) - Extends the shape of the special `_vectors` field in documents. - previously, the `_vectors` field had to be an object, with each field the name of a configured embedder, and each value either `null`, an embedding (array of numbers), or an array of embeddings. - In this PR, the value of an embedder in the `_vectors` field can additionally be an object. The object has two fields: 1. `embeddings`: `null`, an embedding (array of numbers), or an array of embeddings. 2. `userProvided`: a boolean indicating if the vector was provided by the user. - The previous form `embedder_or_array_of_embedders` is semantically equivalent to: ```json { "embeddings": embedder_or_array_of_embedders, "userProvided": true } ``` - During the indexing step, the subfields and values of the `_vectors` field that have `userProvided` set to **false** are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template. - This allows **importing** embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change. - The dump process now uses this ability: it enriches the `_vectors` fields of documents with the embeddings that were autogenerated, marking them as not `userProvided`. This allows importing the vectors from a dump without regenerating them. ### Tests This PR adds the following tests - Long-needed hybrid search tests of a simple hf embedder - Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content. - Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected Co-authored-by: Louis Dureuil <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Related product team resources: PRD (internal only)
Related product discussion: https://github.com/orgs/meilisearch/discussions/677
Motivation
Improve AI search performance
Usage
Avoid regenerating embeddings at dump import
https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697?pvs=74
TODO
Reminders when modifying the Setting API
test_setting_routesmacro/settings/my-new-settingconfigurated in themake_setting_routesmacro/settingsroute configurated in theupdate_allfunction/settingsroute serializing, e.g., enums case can be different (camelCasein route andPascalCasein the dump)Special cases when adding a setting for an experimental feature
Unsetwhen returned from the index in this situation. See an example)validate_settingsfunction)Impacted teams
@meilisearch/docs-team @meilisearch/integration-team
The text was updated successfully, but these errors were encountered: