Calling index search inside Triton python backend

[riyaj8888]

Right now we are having two models one for embedding and other for reranking.
For new query we generate embedding and for that embedding we do index search on other server.
After retrieval of few k documents from index search we pass these (query, took docs) to another Triton endpoint of reranking.

My question is can we integrate these all components together.
We can create ensemble of embedding+ reranking but how can we add indexing or search over index in this pipeline.

Thanks 🙏

[oandreeva-nv]

Let me outline the process in the way I understand it, feel free to correct me.

For this task potentially you can either re-build an index based on the documents to re-use it, or de-serialize it from external service.

After that, one option is to write a python model and utilize a cuVS library. The latter one has apis to build an index, please check with their docs to see if it fits to your needs. This library also provides a variety of vector search algorithms to choose from as well as specifying k for top-k.

Then, the last step for this model is to combine initial request with retrieved top-k embedding and prepare a response, which will be passed to the next stage of your ensemble.

Let me know how this sound to you, happy to discuss further.