New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide an addon for Broad Crawls #6331
Labels
Comments
Are all of those settings safe to enable for all broad crawls? |
I'm not 100% sure. It seems we should review them. Also, we should ensure that the users can change the values we set in the addon. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There are common practices for broad crawls, explained here: https://docs.scrapy.org/en/latest/topics/broad-crawls.html. It involves modifying many settings. It seems we can provide a Scrapy addon to simplify all of that.
It can start as a third-party package, but I think Scrapy itself may also benefit from such addon.
The text was updated successfully, but these errors were encountered: