How to write web crawlers in V language

[Avey777]

Python has many crawler frameworks

Scrapy
PySpider
Selenium
Python-goose

Is it possible for me to do this in V language?
Do you want to provide an official network library? After all, V language currently involves many fields, and there is still a blank space in the field of web crawlers

[JalonSolov]

This should be a fairly simple task for any competent programmer.

Look at examples/web_crawler/ and examples/news_fetcher.v as starting points.

[MedLabs]

No need to use a scraping framework, just use net.html

[Avey777]

ok

[Avey777]

When displaying data in pages, you need to click on the specified page number to obtain the data. How to perform a "click" operation on an element?

[MedLabs]

usually the pagination links have the same URL with only the page number changer example.com/foo/bar?page=[PAGE_NUMBER]
check how many pages there are on the website and run a for loop like:

all_pages := 69
mut p := 1
for p < all_pages {
    url := "example.com/foo/bar?page=${p}"
    ...do your stuff...
    p++
}

[Avey777]

https://github.com/playwright-community/playwright-go