This repository has been archived by the owner on Mar 9, 2021. It is now read-only.
Prevent spider from crawling certain directories or files #67
Comments
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I've got some PHP files to prevent scrapers & form spammers. If a spider ignores my robots.txt file they are blacklisted. Currently, glyphhanger doesn't respect robots.txt and tries to crawl the directories or files.
An option like --spider-nocrawl=/wp-content/uploads/forbiddenfolder&&/wp-content/uploads/forbidden-file.php would be nice!
The text was updated successfully, but these errors were encountered: