Hybrid search with experiemental features using Azure Open AI Embedding API as an embedder

[miguelisidoro]

Hello,

I am trying to configure a Meilisearch instance that we are hosting in Azure (PaaS) to use the Azure Open AI Embeddings API to generate the embeddings in Meilisearch.

We followed the documentation at https://www.meilisearch.com/docs/learn/experimental/vector_search#generate-auto-embeddings and we are trying to use the REST option with the Azure Open AI embeddings API.

We are using the following request to configure Azure Open AI Embedding API as Meilisearch embedder.

But we are getting the following error:

The Azure Open AI api key we are using is correct and if we make a request to Azure Open AI API directly with the same api key, we get a valid vector back.

What are we doing wrong?

Thanks

[dureuill]

Hello!

Looking at Azure's reference documentation, it looks like Authentication by API key uses the api-key header. Meilisearch's REST embedder always uses the Authorization: Bearer <api-key> scheme.

This is unfortunate. If you have access to "Microsoft Entra ID authentication" you can retry with one as it looks like this method uses the Authorization header.

By the way, looking at your embedder config, it looks like some parameters are incorrect. You can't see that now that you have the authentication issue, but they will be an issue after the auth issue is solved.

1. You can't override the model dimensions for the ada model, so dimensions should not appear as part of your query, unless you're not actually using the ada model but one of the embedding models compatible with dimension reduction
2. The dimensions for the ada model is 1536, so the dimensions at the root of your object should be set to that value if using the ada model
3. The path to the input field in inputField should read ["input"]
4. inputType can be set to textArray for better performance
5. The model name does not appear in the parameters of the request in the reference docs of Azure, so model shouldn't be in query
6. Unless all your documents are all very small, and you want all their fields embedded, you should define a custom documentTemplate.

[miguelisidoro]

@dureuill thanks for the response.

This is unfortunate indeed.
I took a look at https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity and it seems that auth with Microsoft Entra ID is a good option to make requests to the Azure Open AI APIs but the Bearer token has to be obtained every time we want to make a call to Azure Open AI APIs, quite different from having an API Key that is always the same throughout API calls to the Azure Open AI APIs. This way, we cannot set a fixed api key to create the Meilisearch embedder and have that api key to be used in every call to the Azure Open AI Embedding API to create the vector in our Meilisearch hybrid search request.

Using Microsoft Entra ID Authentication, what api key should we set in the Meilisearch create embedder API call to the Azure Open AI Embedding API? How would we solve the problem I mention above?

Another question:

Even if I set the vector in my hybrid search, I get the following error:

Is there any way to avoid the embedder to be a requirement when a "q" and a vector are passed in the Meilisearch hybrid search request? If yes, can you supply a hybrid search example to the POST {{url}}/indexes/{{indexUID}}/search endpoint? The only documentation I found is available at https://github.com/orgs/meilisearch/discussions/677 and it is sparse on details.

Thanks

[dureuill]

Hey @miguelisidoro, you're welcome 😊

but the Bearer token has to be obtained every time we want to make a call to Azure Open AI API

Ah yes, you won't be able to use this option then.
It looks like you unfortunately cannot use the REST embedder source in Meilisearch with Azure Open AI until we add some fashion of custom (authorization) header support to the API.

What you can do instead is the following:

1. declare your embedder (named "default" for instance) with a userProvided source and the dimensions equal to the number of floats returned by the Azure OpenAI API embedding requests.
2. put a _vectors field in each of your documents, with value an object whose field is the name of your embedder (for example, "default"), and the value of that field the embedding for this document as a vector of floats (of the same dimension as declared in the settings).
3. In search, put both q and a vector (still of the same dimensions as declared in the settings) to enjoy hybrid search :-)

---

Is there any way to avoid the embedder to be a requirement when a "q" and a vector are passed in the Meilisearch hybrid search request?

Since 1.6, it is mandatory to specify an embedder, but it can be of the "userProvided" type as described above. You can find more information in the docs. A full vector search with user provided embeddings would be like so, and to perform a hybrid search you'd just use the q parameter.

Hope this clarifies things a bit for you, feel free to ask further questions you'd have ☺️

[miguelisidoro]

Hello @dureuill,

Once again thanks for your response :)

It looks like you unfortunately cannot use the REST embedder source in Meilisearch with Azure Open AI until we add some fashion of custom (authorization) header support to the API.

When do you think this can be available? It would be great to have this as soon as possible because the user provided embedding is a solution but harder to implement and a little bit less elegant :).

I have a doubt however: even if you add custom (authorization) header support to the API, don't we always have to add the vector into the indexed documents?
Or is it the embedder that takes care of this? Meaning:

• When indexing the content, we don't supply the vector but the vector is generated by the embedder
• When searching, we don't need to supply the vector and the vector for the "q" parameter is generated by the embedder

Since 1.6, it is mandatory to specify an embedder, but it can be of the "userProvided" type as described above. You can find more information in the docs. A full vector search with user provided embeddings would be like so, and to perform a hybrid search you'd just use the q parameter.

Earlier you said that a "q" parameter and a vector would be necessary when using user provided embedding. But here you say that a "q" parameter is enough. Can you clarify?

Thanks

[dureuill]

When do you think this can be available?

This needs prioritization, I can't answer right now.

the user provided embedding is a solution but harder to implement and a little bit less elegant :).

Agreed

I have a doubt however: even if you add custom (authorization) header support to the API, don't we always have to add the vector into the indexed documents?

No you don't have to for embedders that are not of source "userProvided".

Or is it the embedder that takes care of this? Meaning:

• When indexing the content, we don't supply the vector but the vector is generated by the
• When searching, we don't need to supply the vector and the vector for the "q" parameter is generated by the embedder

Yes, that is exactly so for "openAI", "rest", "huggingFace" and "ollama" embedders.

Earlier you said that a "q" parameter and a vector would be necessary when using user provided embedding. But here you say that a "q" parameter is enough. Can you clarify?

Ah, sorry for not being clear. The query I linked already contains vector (because it describes a full semantic search), so I meant that you only have to add q on top of it. After doing so, you have simultaneously both of q and vector in the query, which is required for hybrid search with user-provided embeddings.
I hope I'm making sense :-)

[miguelisidoro]

@dureuill thanks!

Have a nice weekend.

[miguelisidoro]

Hello @dureuill ,

We are thinking on another approach that is doing a custom REST API that serves as an embedder.
Can you confirm that with a custom REST API as an embedder:

• When indexing the content, we don't supply the vector but the vector is generated by the embedder
• When searching, we don't need to supply the vector and the vector for the "q" parameter is generated by the embedder
• Indexing times are much faster because Meilisearch is doing the work by calling our custom API to generate the embedding vector without any time penalty instead of us having to generate the embedding vector that will slow down the indexing process

One of our concerns is that we will need to rebuild our existing search index and if we use a user provided approach, for each document, we will need to call the Azure Open AI embedding API to generate the embedding vector that will be set in the _vectors field of the index.

If on the other hand, use a REST API approach using a custom REST API that calls Azure Open AI Embedding API, we don't need to call the Azure Open AI Embedding API directly and indexing times are much faster.

Can you confirm what I said above?

About the format of the response required by Meilisearch when using a custom REST API as an embedder. Can you give us an example of a response that we would have to supply so that it can be processed correctly by Meilisearch?

About the request itself, how does Meilisearch do a request to the custom REST API as an embedder? How do I know how data is passed to the embedder? Is it a GET or a POST (I assume POST)? Can you give us an example of a request in the following situations?

• When content is indexed
• When performing a search

Another thing: in https://www.meilisearch.com/docs/learn/experimental/vector_search#generate-auto-embeddings, REST Option, it is referred the following:

"model is a mandatory field indicating a compatible model.

documentTemplate is an optional field. Use it to customize the data you send to the embedder. It is highly recommended you configure a custom template for your documents."

What are we supposed to set in the model field?

documentTemplate although optional, is recommended. Are there any examples of document templates in the documentation or that you can supply?

Thanks

[curquiza]

Answered here: https://github.com/orgs/meilisearch/discussions/748