Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow multiple metadata keys on RedisVectorStoreFilterType #5015

Open
5 tasks done
mauriciocirelli opened this issue Apr 8, 2024 · 1 comment · May be fixed by #5028
Open
5 tasks done

Allow multiple metadata keys on RedisVectorStoreFilterType #5015

mauriciocirelli opened this issue Apr 8, 2024 · 1 comment · May be fixed by #5028
Labels
auto:bug Related to a bug, vulnerability, unexpected error with an existing feature auto:improvement Medium size change to existing code to handle new use-cases

Comments

@mauriciocirelli
Copy link
Contributor

mauriciocirelli commented Apr 8, 2024

Checked other resources

  • I added a very descriptive title to this issue.
  • I searched the LangChain.js documentation with the integrated search.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain.js rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

// Add a sample document with some metadata fields:
vectorStore.addDocument(new Document({ pageContent: "Page Content!", metadata: { x: "y", z: 1, w: ["a", "b", "c"] } }))

// Narrow results with one metadata field:
vectorStore.similaritySearchWithScore("test string", 5, { x:  "y" }))

// Narrow results with two metadata fields (AND):
vectorStore.similaritySearchWithScore("test string", 5, { x:  "y", z: 1 }))

// Narrow results which contains an element on a metadata field list:
vectorStore.similaritySearchWithScore("test string", 5, { w: "a" }))

Error Message and Stack Trace (if applicable)

No response

Description

This is an issue created from the following discussion:

How to use RedisVectorStoreFilterType?

Current implementation of RedisVectorStore saves the Document's metadata object as a JSON string in the metadata-key field.

This makes it impossible to search for multiple fields.

In order to make the RedisVectorStore more compatible to the Document metadata object, we need to change how metadata fields are stored and filtered.

A use case for this is a vector store of books, which metadata with fields such as author (string), year (number) and tags (list of strings) would narrow the vector search and improve its results.

We can achieve this with MemoryVectorStore filter function, and I am pretty sure we can do the same with Redis.

I can work on a PR for this if this interests the community.

Thank you.

System Info

[email protected] | MIT | deps: 17 | versions: 262

@dosubot dosubot bot added auto:bug Related to a bug, vulnerability, unexpected error with an existing feature auto:improvement Medium size change to existing code to handle new use-cases labels Apr 8, 2024
@mauriciocirelli
Copy link
Contributor Author

Perhaps an easier (and yet more flexible) way to do the filtering is just passing a Redis search string in the filter:

vectorStore.similaritySearchWithScore("test string", 5, "@x:(y)")

mauriciocirelli pushed a commit to mauriciocirelli/langchainjs that referenced this issue Apr 9, 2024
mauriciocirelli pushed a commit to mauriciocirelli/langchainjs that referenced this issue Apr 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto:bug Related to a bug, vulnerability, unexpected error with an existing feature auto:improvement Medium size change to existing code to handle new use-cases
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant