initial load of index documents - errors with more than 2000 records #877
beardsleym
started this conversation in
General
Replies: 1 comment
-
@beardsleym I have indexed myself datasets of 20k documents in one single update, but those documents were small and simple. So I don't think there is a problem purely coming from the number of documents. But indeed, one single update with thousands of heavy documents may require huge amounts of RAM available. The alternative of iterating and indexing the documents one by one seems very cost-effective too. What I normally do is trying to index those documents in batches of, for example, 1000 for each request. This normally works fast and smooth. Have you tried this? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi guys, so great to find MeiliSearch.
I used the 1-click DigitalOcean droplet which was so simple and I was suitably impressed, so now I have spun up MeiliSeacrh on a 1GB VM in Oracle Cloud (free-tier) for longer-term use as per the 'production' docs.
On both, I really struggled initially getting my data from Firestore to MeiliSearch. I'll share a few my different attempts. All running on my local Mac machine.
This would only work if I limited to 2000 documents from Firestore at a time.
This worked up till about 1956 records, then I received an error from MeiliSearch, I think it was along the lines of rate limit exceeded.
This returned a 413 error (request entity too large) from MeiliSearch. I tried to change the upload limit as suggested in the docs, --http-payload-size-limit=100000000, but that has had no effect, I couldn't actually work out how to confirm the limit that is in place.
FYI, I have an Algolia account and I checked that the .json file was ok by uploading it in their console and all was fine.
Beta Was this translation helpful? Give feedback.
All reactions