Multiple Filters using Chroma().as_retriever

[jacobbamio]

Hello everyone!

I've started using Langchain and ChromaDB a few days ago, but I'm facing an issue I cannot solve.

I have a VectorStore that contains multiple pdfs and associated metadata. My goal is to pre-filter in multiple ways.

Looking into the documentation the only example about filters is using just one filter. There's other methods like "get" that include "where" arguments, which has examples of using more than one filter.

I tested this code for one filter:

retriever = DB.as_retriever(search_kwargs={"k": int(get_env_var('TARGET_SOURCE_CHUNKS')),
                                           "filter": {"user_id": user_id}})

Which worked without any issues.

The real problem appears when I try to use more than one filter. I have tried it in different ways:

First:

retriever = DB.as_retriever(search_kwargs={"k": int(get_env_var('TARGET_SOURCE_CHUNKS')),
                                           "filter": {"user_id": user_id, "category_id":cat_id}})

Second:

retriever = DB.as_retriever(search_kwargs={"k": int(get_env_var('TARGET_SOURCE_CHUNKS')),
                                           "filter":[ {"user_id": user_id}, {"category_id":cat_id}]})

Also wanted to receive even more than one user at the same time:

retriever = DB.as_retriever(search_kwargs={"k": int(get_env_var('TARGET_SOURCE_CHUNKS')),
                                           "filter": {"user_id": [user_id_1, user_id_2]}})

Nothing worked.

I'd appreciate your help in order to achieve filtering in different metadata keys, and also more than one value for each metadata key.

Thanks in advance!

[dosubot[bot]]

🤖

Hello,

Thank you for using LangChain and ChromaDB. I understand you're having trouble with multiple filters using the as_retriever method.

Based on the issues and solutions I found in the LangChain repository, it seems that the filter argument in the as_retriever method should be able to handle multiple filters. However, the syntax you're using might not be correct.

Here's how you can use multiple filters:

retriever = DB.as_retriever(search_kwargs={"k": int(get_env_var('TARGET_SOURCE_CHUNKS')),
                                           "filter":{'$or': [{'user_id': {'$eq': user_id}},
                                                              {'category_id': {'$eq': cat_id}}]}})

This will return documents that match either the user_id or the category_id.

If you want to filter documents that match both user_id and category_id, you can use the $and operator:

retriever = DB.as_retriever(search_kwargs={"k": int(get_env_var('TARGET_SOURCE_CHUNKS')),
                                           "filter":{'$and': [{'user_id': {'$eq': user_id}},
                                                              {'category_id': {'$eq': cat_id}}]}})

For filtering on a list of values, you can use the following format:

retriever = DB.as_retriever(search_kwargs={"k": int(get_env_var('TARGET_SOURCE_CHUNKS')),
                                           "filter":{'user_id': {'$in': [user_id_1, user_id_2]}}})

This will return documents that have a user_id that matches either user_id_1 or user_id_2.

Please note that the $eq and $in operators are used to specify the conditions for the filters. The $eq operator checks for equality, and the $in operator checks if a value is in a list of values.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• Issue: Create way to do filters using VectorStore
• How to pass filter down to Chroma db when using ConversationalRetrievalChain
• can't seem to add filters in ConversationalRetrievalChain
• libs/langchain/langchain/vectorstores/singlestoredb.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[keiru517]

Hello
Filtering on a list of values gives me this error.
retriever=db.as_retriever(search_kwargs={'filter':{'keywords':{'$in':['appointment', 'asthma']}}}),
Expected where operator to be one of $tt, $gte, $lt, $lte, $ne, $eg, got $in

How to fix that?
Thanks

[zhangZZZ-v]

I had the same problem, did you solve it please?

[jacobbamio]

Hi guys,

If I'm understanding well your problem, you want to filter in your vectorstore those keywords that match 'asthma' and 'appointment', am I right?

Could you try the following code and see if it works for you?

retriever = db.as_retriever(search_kwargs={'filter':{'keywords':['asthma', 'appointment']}})

Let me know if don't and I'll see if I can help you!

[keiru517]

In my opinion, we cannot add metadata in the format of array, we can add it as the string format so if I add metadata, it shows {keywords: '['asthma', 'appointment']'}.
I want to add 2 keywords (asthma and appointment) to an embedding and filter it by one of these keyword.
How to do this?
How can I add metadata?
Now I added 2 keywords like this. {keyword1: 'asthma', keyword2:'appointment'} and filter this with loop to check if the given keyword matches with keyword1 or keyword2 or not. But I think this is not the solution
Thanks

[keiru517]

Hi, @jacobbamio
That code is not working
It gives the following error
Expected where value to be a str, int, float, or operator expression, got ['Certificate']

[zhangZZZ-v]

@dosu-bot.
Hello
Filtering on a list of values gives me this error.

retriever=db.as_retriever(search_kwargs={'filter':{'keywords':{'$in':['appointment', 'asthma']}}}),
Expected where operator to be one of $tt, $gte, $lt, $lte, $ne, $eg, got $in

How to fix that?
Thanks

[keiru517]

This is not working

[KINJALMARU16]

retriever = DB.as_retriever(search_kwargs={"k": int(get_env_var('TARGET_SOURCE_CHUNKS')),
"filter":{'user_id': {'$in': [user_id_1, user_id_2]}}})
this is not working for me. I want to search from multiple documents, and they have different metadata.
did anyone find a way to do that?

[chelun86]

Have you solved this problem? I have the same confusion

[keiru517]

Here is what I want to do
Let's say there is a chunk with this metadata. {'keyword': 'certificate'}
And I want to filter this chunk with these array ['asthma', 'certificate']
In this case, the chunk has keyword as 'certificate' so it should be in the filtered data.
But if I run the following code,
as_retriever(search_kwargs={'filter':{"keyword": ['asthma', 'certificate']})
it gives

Expected where value to be a str, int, float, or operator expression, got ['Certificate']
error

And if I run the folloiwng code,
retriever=db.as_retriever(search_kwargs={'filter':{'keywords':{'$in':['appointment', 'asthma']}}}
then I got

Expected where operator to be one of $tt, $gte, $lt, $lte, $ne, $eg, got $in

How to solve this?
Thanks

[Swalloow]

retriever = db.as_retriever(search_kwargs={'filter':{'keywords': {'in': ['asthma', 'appointment']}}})

use in instead of $in.
it depends on vectorstore. works in pgvector.

[Robs-Git-Hub]

I am trying to get this to work in a jupyter notebook at the moment and being unsuccesful. Would you mind posting your code so I can see how you are populating the metadata?

At the moment this retrieves appropriate results:

db.as_retriever(search_kwargs={
    'k': 2, 
    'filter': {'topic':'foo'}
}).get_relevant_documents(query)

While this retrieves nothing when I would expect it to retrieve the same result as above.

db.as_retriever(search_kwargs={
    'k': 2, 
    'filter': {'topic':{'$eq':'foo'}}
}).get_relevant_documents(query)

I have tried both '$eq' and 'eq'. I have my cmetadata column set to jsonb.

[gkourogiorgas]

What is this functionality based on? Is it native python, custom or some package?

[ad092]

How is this working for Qdrant?

 raise UnexpectedResponse.for_response(response)
qdrant_client.http.exceptions.UnexpectedResponse: Unexpected Response: 400 (Bad Request)
Raw response content:
b'{"status":{"error":"Format error in JSON body: data did not match any variant of untagged enum Condition at line 1 column 8643"},"time":0.0}'

do i have to use the qdrant specific syntax for filtering?

[ad092]

I got it...sorry.

Here for anyone who is struggling:

sources:
#4718
#5446

#16466

from qdrant_client.http import models
from qdrant_client import QdrantClient
from langchain_community.vectorstores import Qdrant

client = QdrantClient(
    host="localhost",
    port=6333,
)
vectorstore = Qdrant(
    client=client,
    collection_name=collection_name,
    embeddings=embed_model
)

retriever = vectorstore.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={
        "k": 6,
        "score_threshold": 0.01,
        "filter": models.Filter(
            must=[
                models.FieldCondition(
                    key="metadata.release_year",
                    range=models.Range(
                        gt=None,
                        gte=1990,
                        lt=None,
                        lte=2024,
                    )
                )
            ]
        )
    },
)