[SOLVED] RSS Guard supports plugins, can scrape websites with scripts

[martinrotter]

@Owyn

Saw your ticket on quiterss repo. I would like to inform you that RSS Guard does support those features you requested for quiterss. Documentation:

https://github.com/martinrotter/rssguard/blob/master/resources/docs/Documentation.md#scrap
https://github.com/martinrotter/rssguard/blob/master/resources/docs/Documentation.md#fltr

[Owyn]

It doesn't seem to support any accessibility at all, look at this forced super-small unchangable text size, it even ignores windows zoom and text size settings (you can see that with menu text size which was left alone in comparison with enforced small text size in the feed categories list and app settings):

I looked through all settings but there was nothing about text size or scaling

edit: searching through issues helped find the magic setting but not much, settings font is still unchangable:

[martinrotter]

You want to be able to sent FONT SIZE application-wide? For all texts in app? Also, what is "windows zoom"?

[Owyn]

yea, that's how it usually is in most applications, a setting in "Interface" for that

windows zoom level:

[Owyn]

Also internal web-browser seems to have hardcoded colors and ignore windows (contrast) theme colors entirely:

[martinrotter]

1. Internal web browser uses colors from active "Skin" which is changeable in settings. There is dark skin (dark, or nudus-dark) and light skin available.
2. You can zoom internal web browser and zoom state is remembered.
3. You can change font size globally by using environment variable QT_SCALE_FACTOR. Set it either globally in your PC, or locally for RSS Guard for example to value 2. Result likes like pic below. Here is some related documentation.

[martinrotter]

4. To follow Windows zoom settings automatically, related docs.

[Owyn]

Internal web browser uses colors from active "Skin" which is changeable in settings.

no it doesn't (I restarted):

[martinrotter]

Yes, it does, set it for example to "nudus-dark" and restart.

[martinrotter]

[Owyn]

Indeed, seems like not all dark themes are actually dark 🤔

[Owyn]

When i imported all my old feeds (without fetching metadata cuz else it wouldn't have added feeds which are currently offline), now none of my feeds have a favicon unless I click "fetch metadata" on each and every feed one by one? there is no "fetch metadata for all" button??

[Owyn]

I tried to make a log file like docs say :

but the log file simply wasn't created anywhere after the program ran (write rights on the folder are free)

[martinrotter]

that sounds weird, double check for the log, should be there

[martinrotter]

also, "Kumo desu ga" is the, just with fetched title "Nyaa....."

as for favicons, let me check

[Owyn]

That also seems weird, why did it reset my custom feed name by itself?

[martinrotter]

That also seems weird, why did it reset my custom feed name by itself?

Probably because metadata were fetched during import?

[Owyn]

how do I make article text in center not at right? I don't need read\importance stuff, how do I disable that panel?

[martinrotter]

[martinrotter]

how do I make article text in center not at right

Article text is formatted with skin CSS, almost all skins format texts "justified" -> not aligned to right, ellaborate what exactly you mean.

[Owyn]

That was it, now text is in the center (of the monitor)
I was expecting I could do that by right clicking the panel and choosing "hide" as most apps do

[martinrotter]

which most apps, can you name them?

[Owyn]

sorry, it's just that last such toolbar I remember was hidden that way iirc,
but yea, view menu -> show\hide is also the usual way

[Owyn]

My colors look pretty colorful for a dark theme, the blue color especially seems rather unreadable

[martinrotter]

because that is the UX approach all modal dialogs use, you can either OK or CANCEL, you can use ESC button.

[Owyn]

but why disable the [x]? ppl love pressing [x]... 😢

[martinrotter]

if more people will show interest in X buttons I will implement, so far, you are the first one requesting this

[Owyn]

my parents for example never press ok\cancel, they always press [x] 😄

[martinrotter]

Yes, they won't be typical RSS Guard users I guess.

[Owyn]

@Owyn

Saw your ticket on quiterss repo. I would like to inform you that RSS Guard does support those features you requested for quiterss. Documentation:

https://github.com/martinrotter/rssguard/blob/master/resources/docs/Documentation.md#scrap https://github.com/martinrotter/rssguard/blob/master/resources/docs/Documentation.md#fltr

I tried one of the examples php#tweeper.php#-v#0#https://twitter.com/NSACareers and it didn't work:

I wish it was something simple here to parse websites into RSS, like feed-me-up-scotty https://feed-me-up-scotty.vincenttunru.com/ (maybe you could incorporate that script into RSSguard? the source is on the gitlab) , there I just had to specify a CSS selector where all the needed <a> links are and it made those links into feed articles:

[EvilGod]
title = "I'm an Evil God chapters"
url = "https://mangahub.io/manga/i-m-an-evil-god/"
entrySelector = "ul.list-group li"
titleSelector = "a span"
linkSelector = "a"
contentSelector = "a span"

Output: https://github.com/Owyn/feeds/blob/gh-pages/EvilGod.xml

how do I do the same here? examples seem to be very lacking for practical use...

[Owyn]

If not, then you will simply be good with Python

Most websites work fine without the JS as far as I've experienced, everything needed usually is in the page source,

As I've checked crawler scripts in python ( https://github.com/martinrotter/rssguard/tree/master/resources/scripts/scrapers ) - and all of them seem to be in raw python without any soup?

why not just use those good examples you've posted - beautiful soup \ pyppeteer ?

soup itself even seems to have a built-in support for CSS selectors it seems https://stackoverflow.com/questions/11465555/can-we-use-xpath-with-beautifulsoup

[martinrotter]

found it in releases! thank you

but can it at least display images? I don't need much, just images cuz some feeds are simple a feed of one image per item where artists post their new works

In documentation example image - images seem to be gone

No, at this point it only show clickable links which you can display in external app/browser. If you want a way to show images, post ticket pls.

[martinrotter]

As I've checked crawler scripts in python ( https://github.com/martinrotter/rssguard/tree/master/resources/scripts/scrapers ) - and all of them seem to be in raw python without any soup?

Some of those are using https://newspaper.readthedocs.io/en/latest/ and some of them are using BeautifulSoup. It is really just a bunch of scripts to be used as inspiration.

[Owyn]

It is really just a bunch of scripts to be used as inspiration.

I mean, the soup can be presented as a default to-go way to do things easily here if it's good

[martinrotter]

bsoup works nicely, you should used that if sites do not use JS

[Owyn]

about the main theme of the topic again -

plugins, can scrape websites with scripts

but can it use compiled executables instead of scripts? cuz scripts can be compiled, or just some non-scripting language compiled. and also compiled scripts'd be faster than non-compiled ones, and users wouldn't need to install any development environment like scripting languages onto their PC

[martinrotter]

If course it can. It can call just about any valid executable.

[Owyn]

That would be a great way for the users

[martinrotter]

As I said before, you can call just anything.

[Owyn]

The # character is there to separate interpreter and individual arguments. I had to select some character as separator because simply using space is not that easy as it might sound, as sometimes space could be a part of an argument itself.

That's a problem, since # sign is used for Id CSS selectors 😢 so I can't pass CSS selectors as arguments now when I'm pretty much done with my python script analogue of feed-me-up

why not just us " and ' just like how all passing of parameters are done everywhere?

and 2nd) - does the json format RSSguard uses for feed supports baseUrl? if links \ favicons are relative (especially favicons, cuz most has relative links)

if it's a post-proccess script and it gets no url at all, so RSSguard should provide the baseUrl itself then... cuz now relative links in json don't work

[Owyn]

Here, I made this for RSSGuard: https://github.com/Owyn/CSS2RSS

[martinrotter]

EDIT: I FOUND A WAY. Will you test it after I implement?

[martinrotter]

@Owyn OK I pushed a fix - 66159f4.

Now, you don't have to use separators at ALL. If some of your arguments contain spaces, then enclose it with double quotes (NOT single quotes).

Pls, test and let me know, it is compilling RN.

[Owyn]

Just a question... where should I look for compiled dev builds?

[martinrotter]

https://github.com/martinrotter/rssguard/releases/tag/devbuild

right now they are not there, it is compilling RN, wait about 15 minutes, they appear as files there

[Owyn]

Pressed "what are scripts"? then pressed it again, - the window is still out of bounds, and since the [x] is disabled you can't even close it with the mouse now without dragging it

but new param passing using quotes works