How do I use the MetadataFilter to filter by multiple values?

[kolaente]

Using the MetadataFilter, how can I filter by multiple values?

Consider the following: I have multiple sources for documents. These are saved as source_id in the embedding's metadata. When retriving content, I want to filter based on that id. Currently, I do this:

    metadata_filters = [
        MetadataFilter(key="source_id", value=s_id)
        for s_id in bot.source_ids
    ]

    return index.as_chat_engine(
        chat_mode=ChatMode.CONDENSE_PLUS_CONTEXT,
        similarity_top_k=5,
        system_prompt=bot.system_prompt,
        context_prompt=context_prompt,
        condense_prompt=condense_prompt,
        filters=MetadataFilters(filters=metadata_filters, condition=FilterCondition.OR),
    )

(notice the FilterCondition.OR)

Ideally, I'd like to use the FilterOperator.IN filter instead, like this:

    return index.as_chat_engine(
        chat_mode=ChatMode.CONDENSE_PLUS_CONTEXT,
        similarity_top_k=5,
        system_prompt=bot.system_prompt,
        context_prompt=context_prompt,
        condense_prompt=condense_prompt,
        filters=MetadataFilters(filters=[MetadataFilter(key="source_id", value=bot.source_ids, operator=FilterOperator.IN)]),
    )

But it looks like the value of the filter must always be a str, int or float.

Is it possible at all what I want to achive?

[dosubot[bot]]

🤖

Hey @kolaente! Good to see you tackling new challenges with LlamaIndex. How's everything going?

Based on the information available in the LlamaIndex repository, it seems that the FilterOperator.IN operator is designed to work with an array of values. However, the value field in MetadataFilter is defined as a Union[StrictInt, StrictFloat, StrictStr], which means it can accept either a string, integer, or float. This suggests that the MetadataFilter class might not support a list of values for the value field when the IN operator is used.

Here is an example of how you might use the FilterOperator.IN operator:

from llama_index.core.vector_stores.types import MetadataFilter, FilterOperator

# Create a MetadataFilter instance
filter = MetadataFilter(
    key="source_id", 
    value=["id1", "id2", "id3"],  # List of source_id values to filter by
    operator=FilterOperator.IN
)

However, this might not work as expected because the value field in MetadataFilter is not designed to accept a list of values. You might need to modify the MetadataFilter class to accept a list of values for the value field when the IN operator is used.

Please note that modifying the MetadataFilter class might have unintended side effects and should be done with caution. It's also important to thoroughly test any changes to ensure they work as expected.

I hope this helps! If you have any other questions or need further clarification, please let me know.

Sources

• llama-index-core/llama_index/core/vector_stores/types.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[kolaente]

How would I need to modify the MetadataFilter class to make it accept a list of values?