Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: MongoDB doc_store add Index #13078

Open
danerlt opened this issue Apr 24, 2024 · 0 comments
Open

[Feature Request]: MongoDB doc_store add Index #13078

danerlt opened this issue Apr 24, 2024 · 0 comments
Labels
enhancement New feature or request triage Issue needs to be triaged/prioritized

Comments

@danerlt
Copy link

danerlt commented Apr 24, 2024

Feature Description

I am using MongoDB as the doc_store, and the structure stored within it is as follows:

{
  "_id": "2c5ae478-327c-4620-894e-f1a9cefde5b8",
  "__data__": {
    "id_": "2c5ae478-327c-4620-894e-f1a9cefde5b8",
    "embedding": null,
    "metadata": {
      "dataset_id": "3b3da4dc-09d2-43bc-8b46-b1a990c040ff",
      "account_id": "14fe321d-6768-4d9d-bbcf-d81a0a7fa51d",
      "file_id": "bb3c585c-18ca-4b45-97b6-e95b43145428",
      "file_name": "test.csv",
      "ref_doc_id": "bb3c585c-18ca-4b45-97b6-e95b43145428",
      "doc_id": "bb3c585c-18ca-4b45-97b6-e95b43145428"
    },
    "excluded_embed_metadata_keys": [],
    "excluded_llm_metadata_keys": [],
    "relationships": {},
    "text": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "start_char_idx": null,
    "end_char_idx": null,
    "text_template": "{metadata_str}\n\n{content}",
    "metadata_template": "{key}: {value}",
    "metadata_seperator": "\n",
    "class_name": "Document"
  },
  "__type__": "4"
}

When I query nodes, I am able to filter by fields within the metadata field. I want to create an index on the metadata field to speed up retrieval.

I want to add an _initialize function in the MongoDBKVStore to create an index on the metadata field, similar to how it is done in the PostgresKVStore, but in the _initialize function, the task is to create the index.

Reason

Indexes support efficient execution of queries in MongoDB. Without indexes, MongoDB must scan every document in a collection to return query results. If an appropriate index exists for a query, MongoDB uses the index to limit the number of documents it must scan.

Value of Feature

No response

@danerlt danerlt added enhancement New feature or request triage Issue needs to be triaged/prioritized labels Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request triage Issue needs to be triaged/prioritized
Projects
None yet
Development

No branches or pull requests

1 participant