Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Pinecone should support storing metadata #435

Open
nsadeh opened this issue Jan 2, 2024 · 2 comments · May be fixed by #437 or #955
Open

[FEATURE] Pinecone should support storing metadata #435

nsadeh opened this issue Jan 2, 2024 · 2 comments · May be fixed by #437 or #955
Labels
enhancement New feature or request good first issue Good for newcomers P2 High priority

Comments

@nsadeh
Copy link

nsadeh commented Jan 2, 2024

Please provide as much details as possible, this will help us to deliver a fix as soon as possible.
Thank you!

Describe the bug
A clear and concise description of what the bug is.

PineconeEmbeddingStore, the Pinecone implementation of Embedding store, is a TextSegment embeddings store. Since the latter has a Metadata attribute, it stands to reason that if a client called the add method using TextSegments with metadata, that the metadata would be uploaded to Pinecone. The actual behavior is that the Metadata gets dropped.

Log and Stack trace
Please provide a log and a stack trace (with exception), if applicable.

N/A

To Reproduce
Please provide a relevant code snippets to reproduce this bug.

  1. Instantiate a Pinecone Client, index, and an embedding model
  2. Create a TextSegment with non-trivial metadata, for example

(excuse my Java, I am a Clojure programmer)

Metadata m = new Metadata(Map.of(
    "a", "b",
    "c", "d"
));

TextSegment t = TextSegment.from("I Am a text segment, m)
  1. Create embeddings for this text (although you can actually pass in a random vector for this repro) and invoke .add with the segment to upload to Pinecone
  2. The metadata will not make it to Pinecone

Expected behavior
A clear and concise description of what you expected to happen.
Expecting the metadata to appear in the Pinecone console/be retrieved with Query requests

Please complete the following information:

  • LangChain4j version: e.g. 0.24.0
  • Java version:o openjdk version "21.0.1" 2023-10-17
  • Spring Boot version (if applicable): e.g. 2.7.14

Additional context
Add any other context about the problem here.
This appears to be just omitted from the interaction with Pinecone:

I may try to make a pull request. In the meantime I will have to write my own Pinecone integration with their SDK.

@nsadeh nsadeh added the bug Something isn't working label Jan 2, 2024
@nsadeh nsadeh linked a pull request Jan 2, 2024 that will close this issue
@nsadeh
Copy link
Author

nsadeh commented Jan 2, 2024

See #437 for a fix suggestion

@langchain4j langchain4j changed the title [BUG] Pinecone doesn't add metadata to request [FEATURE] Pinecone should support storing metadata Apr 11, 2024
@langchain4j langchain4j added enhancement New feature or request P2 High priority and removed bug Something isn't working labels Apr 11, 2024
rgrebski pushed a commit to Stepwisepl/langchain4j that referenced this issue Apr 17, 2024
@rgrebski
Copy link

rgrebski commented Apr 17, 2024

Created PR #955 for that issue.
Please check my comments in the PR itself 🙏

Btw - If we manage to get this merged soon - I have already done the upgrade from pinecone-client 0.6.0 to 1.0.0.
I will create separate PR once this one gets merged

@langchain4j langchain4j added the good first issue Good for newcomers label Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers P2 High priority
Projects
None yet
3 participants