Read this in other languages: Russian, हिन्दी, 中國人

Fast and simple crawler

How it works?

Вit's very simple: your bot massively signs your account in response, people follow you.

The order of preparation and work with the bot

1. Clone the repository or download the archive from github or using the following commands on the command line

   $ cmd
   $ git clone https://github.com/BEPb/github_bot
   $ cd github_bot
   

2. Create a Python virtual environment.

3. Install all necessary packages for our code to work using the following command:

   pip install -r requirements.txt
   

4. create a project called nameproject

scrapy startproject nameproject

5. after which you will have a folder with the name of this project and in it the minimum necessary files and dependencies

    scrapy.cfg #deploy configuration file
    nameproject/ # project's Python module, you'll import your code from here
        __init__.py
        items.py # project items definition file
        middlewares.py # project middlewares file
        pipelines.py # project pipelines file
        settings.py # project settings file
        spiders/ # a directory where you'll later put your spiders
            __init__.py

6. go to our project folder

cd nameproject

7. create a quotes_spider.py file in the spiders/ folder and write in it who and how we cheat
8. launch our crawler

scrapy crawl quotes

9. as a result of the execution, two new files were created: quotes-1.html and quotes-2.html with content for the corresponding URLs, as our parse method specifies.
10. use shell selectors

scrapy shell 'https://quotes.toscrape.com/page/1/'

11. view all 'title' objects using css. The result of executing response.css('title') is similar to list object named SelectorList which is a list of Selector objects that wrap XML/HTML elements and allow you to perform additional queries to refine the selection or retrieve data.

response.css('title')

12. and in order to view the list, specify the getall () method

response.css('title::text').getall()

13. the same can be done with xpath

response.xpath('//title/text()').get()

14. and now take div tags with class quote

response.css("div.quote")

15. take only the first element in the list

response.css("div.quote")[0]

16. in order to get the class in the tag, use the following command:

quote.css("span.text::text").get()
quote.css("small.author::text").get()

17. and this is how we will display the complete list of the class of the div tag

response.css("div.quote").css("div.tags a.tag::text").getall()

18. this is how we save the result in json format, where the -O command line switch overwrites any existing file;

scrapy crawl quotes -O quotes.json

19. and this is how we save the result in csv format

scrapy crawl quotes -O quotes.csv

20. The following command writes line by line using the .jl format

scrapy crawl quotes -o quotes.jl