Skip to content

dreadedhamish/scrapy-cloudflare-middleware

 
 

Repository files navigation

PyPI

Scrapy "CloudFlare" middleware

A Scrapy middleware to bypass the CloudFlare's anti-bot protection, based on cloudflare-scrape.

Installation

pip install scrapy_cloudflare_middleware

Usage

Add the middleware to your DOWNLOADER_MIDDLEWARES settings

DOWNLOADER_MIDDLEWARES = {
    # The priority of 560 is important, because we want this middleware to kick in just before the scrapy built-in `RetryMiddleware`.
    'scrapy_cloudflare_middleware.middlewares.CloudFlareMiddleware': 560
}

Done. Happy scraping !

About

A Scrapy middleware to bypass the CloudFlare's anti-bot protection

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%