Search taking upwards of 40s?

[lrojas94]

First off, hello and thank you all for your hard work!

We've been using meilisearch for a few months now, and have found it to work fairly ok here and there, however, we've run into really really slow searches as our index has began to grow and we've began to add new ways to sort / search.

We have a search index that has around 4M documents. This index also has:

• 4 searchable attributes
• 5 sortable attributes
• 6 filterable attributes

This has lead to 2 main issues:

• Updating the index settings takes AGES. Lately, we've not been able to get it to finish and just end up reindexing the entire thing instead which has proven to at least be faster. By updating, I mean, either adding or removing sortable, searchable or filterable attributes. Note that when creating a new index, this is not a problem, hence why we just create a new one which seems to work far faster.
• Search is extremely slow - The general number is upwards of 10s, but can take even 40-50s for single word searches (i.e, Lion). Our dataset is in english if that helps some.

The machine we are using to store meiliserach has about 32GB of RAM (of which, around 5GB are being constantly used). A limit of 8 CPU cores, and we're doing 4 for indexing. In terms of how we add documents, we add/update about every 10 minutes and that might be 1 to 2k documents at a time.

Indexing speed isn't too bad. The main concern is search, which is quite hindered.

We're on Meilisearch 1.3.0

[curquiza]

Hello @lrojas94

Thank for taking the time to write this!

For indexing: I recommend you use Meilisearch v1.6.X when it's out (15/01/2024). We made a lot of improvements since v1.3.0. In v1.6.0 you will be even able to customize the proximity ranking rules settings (proximityPrecision) which can drastically decrease your indexing time and the size taken on your disk.

Here is an overview of what will come in v1.6.0

For the search, no matter your version, this is indeed not expected.
Is it possible you send us your dataset (by email) and the query you do to trigger this slowness? Or a access to the machine? The purpose is to be able to reproduce the slowness, explain it and/or fix it.

Thank you again for your feedback 🙏

[lrojas94]

Thank you for the super quick response!

We're planning to migrate to 1.6 for sure, thank you for letting me know the release date!

As for dataset access, would access to the Meilisearch dashboard be helpful? The dataset is a bit big to send over an email - so I'm happy to give you a read key to the meili dashboard if that helps (?). If so, let me know the email address you preffer and I can do that shortly :)

[curquiza]

You can use my email address in my GitHub profile
Without the dataset, it might be complex for the developers to debug 😕 I'll let you know

Can you send the settings you have?
And the kind of documents you have (how many fields, which number of words per field..)?

[lrojas94]

For sure:

{
    "displayedAttributes": [
        "*"
    ],
    "searchableAttributes": [
        "meta.prompt",
        "generationProcess",
        "tags.name",
        "user.username"
    ],
    "filterableAttributes": [
        "aspectRatio",
        "baseModel",
        "createdAtUnix",
        "generationTool",
        "tags.name",
        "user.username"
    ],
    "sortableAttributes": [
        "createdAt",
        "stats.collectedCountAllTime",
        "stats.commentCountAllTime",
        "stats.reactionCountAllTime",
        "stats.tippedAmountCountAllTime"
    ],
    "rankingRules": [
        "words",
        "typo",
        "proximity",
        "attribute",
        "sort",
        "exactness"
    ],
    "stopWords": [],
    "synonyms": {},
    "distinctAttribute": null,
    "typoTolerance": {
        "enabled": true,
        "minWordSizeForTypos": {
            "oneTypo": 5,
            "twoTypos": 9
        },
        "disableOnWords": [],
        "disableOnAttributes": []
    },
    "faceting": {
        "maxValuesPerFacet": 100,
        "sortFacetValuesBy": {
            "*": "alpha"
        }
    },
    "pagination": {
        "maxTotalHits": 1000
    }
}

[curquiza]

Hello again Luis,

Thanks for the access you sent by email, and for the settings.

Do you use an SSD or HDD disk? Where is hosted Meilisearch?

[lrojas94]

It's being hosted on Digital Ocean's Standard droplet, 8 vCPU/32GB machine.

[curquiza]

Ok, so nothing that sould be wrong.
Definitely we cannot debug without having your dataset 😢
Is it possible for you to provide a way for us to get the dump of your instance? Would be really appreciated 🙏
Again sorry for the trouble

[lrojas94]

I believe I will be able to send this dump to you today 👍 Even the dump itself took a while to create, but perhaps that's fairly normal. duration":"PT32969.497581966S"

[curquiza]

Unfortunately, since the search is long, and the indexing is long too, I expected the dump creation to be long too 😅

[lrojas94]

I've sent you the dump - hope you guys manage to figure something out 🙏

[curquiza]

Hello @lrojas94
@ManyTheFish made investigation and a fix should available in the release coming on Monday 😊

Thanks again for your help in this issue 🙏

See the related PR: #4313

[lrojas94]

🙏 Thank you for being so fast at investigating and dealing with it! Looking forward to Monday!