[SOLVED] web-scrapping feature is mature I think

[Owyn]

I've noticed stable RSSGuard already supports script arguments with brackets instead of old #,
and many if not most of my RSSfeeds are already being done by my css2rss https://github.com/Owyn/CSS2RSS scrapper,
I've added multiple features and options into it and I like the result very much
I've also tried to make my readme file very user-friendly and easy to understand

Could you perhaps add a link to CSS2RSS.py script into your readme at an appropiate place? I'm not sure myself where cuz I don't want to just bury it deep into the documentation,
I think we need to emphasize the feature of RSSGuard to scrape websites (with it) now as it is rather simply and works good

Maybe add it into the main README.md file after "feed services" that you can easily scrape websites now with CSS selectors?

Cuz many people would like to scrape websites I think (well I did for sure)

[Owyn]

btw, filters are using javascript (without the need for user to install anything), why can we not use javascript scripts as post-process scripts too?

[Owyn]

if bundled JS could make a html document from html text (or maybe html could be passed as an object to the js scrapper?), that should be enough to then query css selectors and do everything needed to make a json format feed

[martinrotter]

Yes, JS core bundled with Qt support only ecma262.

https://www.ecma-international.org/publications-and-standards/standards/ecma-262/

It really does not offer many "standard library" features. I do not think that some DOM implementation is there.

[martinrotter]

SO bundled JS is really good as "scripting" language but that's it, no DOM support, not utility functions etc.

[Owyn]

That's a shame, I could rewrite my scrapper into js pretty quicky if it did support DOM, without the need for python it'd probably start running much more quickly and rssguard users wanting to scrape wouldn't have had to install python

[martinrotter]

Well. Users can use any tool they have preinstalled, for example PowerShell on windows.

[martinrotter]

If you are using a NoWebEngine version of the RSSGuard (like I am), then no javaScripts would run on scrapped pages, so sites which populate their content with javaScripts wouldn't be able to get scrapped, instead their starting version (what you'd see in right click -> view page source) would get scrapped.

Scraping feature works exactly the same in nowebengine and in webengine versions, no javascript is evaluated on "scrapped" data.

[Owyn]

If you are using a NoWebEngine version of the RSSGuard (like I am), then no javaScripts would run on scrapped pages, so sites which populate their content with javaScripts wouldn't be able to get scrapped, instead their starting version (what you'd see in right click -> view page source) would get scrapped.

Scraping feature works exactly the same in nowebengine and in webengine versions, no javascript is evaluated on "scrapped" data.

weird, I remember you saying scraping used webengine of the rssguard 🤔

[martinrotter]

If you are using a NoWebEngine version of the RSSGuard (like I am), then no javaScripts would run on scrapped pages, so sites which populate their content with javaScripts wouldn't be able to get scrapped, instead their starting version (what you'd see in right click -> view page source) would get scrapped.

Scraping feature works exactly the same in nowebengine and in webengine versions, no javascript is evaluated on "scrapped" data.

weird, I remember you saying scraping used webengine of the rssguard 🤔

I planned to somehow integrate, but its not that easy. It sharess for example cookies at this point.

[martinrotter]

I will enhance "web scraping" part of docs.