-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature request] Store scraped data #99
Comments
well, there is an option to cache responses using Redis. Caching for infinite time, however, would blow up RAM usage. Furthermore, it would be akin to keeping stale answers, which is not optimal. So, I don't think it's a sound idea. if you're worried about a particular answer not existing in the future, you can always use archive.org and save the answer forever. As for instances being rate limited, I'm exploring ways to minimise it. |
Why not caching on disk, of not with MySQL or Postgres then with SQLite for example, which would easily allow something like a month of cache ? Thanks |
Hello,
It would be nice of Quetre to store scraped data in a database so that it doesn't ever need to scrape it again.
Here's what happened to me today :
Thanks
The text was updated successfully, but these errors were encountered: