Possibility of a website scraping using the internal browser and JS?

[DawEdhel]

My main problem is the data I want to retrieve is hidden behind an authorization process and the page is powered by React framework.
I've already tried Puppeteer, but to no avail.

Is it possible to do the following steps or in any similar way?

1. Login in advance to a target URL using the internal browser;
2. Provide the URL to the feed's source field and force the app to retrieve the DOM source after a "idle network" event via the internal browser using the existing cookies;
3. Provide a JS script to the feed's postscript field to execute on the DOM source and pass the result of the script (e.g. JSON) to the app to parse;

[martinrotter]

This actually doable idea. Will check it out but should be possible via QWebEnginePage

[martinrotter]

OK, I pushed the code and development build is now compilling. Please test when compiled and report back.

1. I added new source type "Built-in web browser". Simply select that and put in your URL.
2. Make sure to be "logged-in" that URL in built-in browser before step 1.

Here is how the dialog setup looks (do not mind the third red rectangle, that is only for my purposes for that specific testing website I used).

3. The feature works in both "Fetch it now" testing mode and real live mode.
4. The testing website I use provides the HTML only when user is logged-in and when fetched with this new feature it correctly returns expected data (which I additionally parse with "css2rss").

The post-process field yes. Read documentation. Simply when the data is obtained from the source with one of methods (URL, local file, Script or new Built-in web browser URL) then the data is fed to provided post-process script via console standard input/output mechanism).

You can likely use node to execute some JS on HTML data I guess you need to write custom Node script which:

1. Reads standard input (the obtained data).
2. Creates DOM from the input.
3. Runs the JS you need.
4. Prints the modified DOM to the standard output (which is then read back by RSS guard and parsed as a feed).

So your point "3" is entirely in your direction, all the tools necessary are already there in RSS Guard.

For point 2 - the part of retrieving the webpage via internal browser is implemented as per I wrote above. But, what exactly you mean by
after a "idle network" event via the internal browser. You mean that internal web browser should wait until some specific JS event/message arrives? If so, how exactly this event looks like?

[martinrotter]

I just got an idea, will implement and we can then retest.

[martinrotter]

OK so some summary:

• - Authentication, cookies, storage - this all seems to work very nicely, state is remembered, logins too.
• - JavaScript itself runs without problems and the internal JS script I wrote is able to detect "idle" moments very well.
• - Problem is that some websites use JS techniques which run JavaScript only when page becomes physically visible/drawn. The solution to this is veeeeery problematic as the only way is literally showing the temporary "web browser" somewhere in the app which is just unacceptable for vast majority of users. Just tested this on Reddit subreddits. If I hack intermediate web browser and display it while the feed fetches the data from it, then all HTML elements are generated by JS properly just as we expect and I get all reddit data. But when the "web browser" is created but hidden, well, the JS magic simply does not run.

So, implementation of this feature will work for some JS-enabled sites, but not for all sites. Not sure what approach proper Chrome headless mode uses, if it works better or not.

I would likely recommend to stick to those solutions, @DawEdhel what you think.

[DawEdhel]

Problem is that some websites use JS techniques which run JavaScript only when page becomes physically visible/drawn.

I've made some research about this, and my question is: have you tried dealing with Document.visibilityState property?

The Document.visibilityState read-only property returns the visibility of the document. It can be used to check whether the document is in the background or in a minimized window, or is otherwise not visible to the user.

Here is one of examples of its spoofing. There is also this solution.

——————

Overall, it's still a better solution over URL page source's option in some scenarios, so I'll take anything there is.

P.S. I'm still not sure what to do with my latest 'unresponsive result' testing.

[martinrotter]

@DawEdhel I pushed new code, cleaned the feature up and fixed some related bugs which could be responsible for your unresponsiveness.

I have also tested it with some websites and it indeed works. I also tweaked built in JS and now it automatically triggers 1 scrolling to bottom, which in some cases forces the page to load more of its JS-enabled content.

Please, test latest devbuild when it compiles, I will take a look on the spoofing thing you sent.

EDIT: With my limited JS knowledge I have tested both your suggestions - neither seems to work sadly - tested with Youtube.

[DawEdhel]

@martinrotter

I've downloaded latest devbuild-07eb444d9 and tested it, and indeed, I could successfully retrieve a populated with data HTML of my initially problematic example, which makes initial post of mine resolved.

But there were some caveats: when I wrote a script to parse HTML and provide a JSON feed, then pressed "Fetch it now" button, I got an "Error: feed format not recognized". Initially thought I wasn't following the feed's format, but couldn't find anything. Then I've tried to create a new feed, using a working-in-the-stable-build local file with JSON, and I got the same error.

I managed to circumvent this error by doing the following:

1. Supply into the "Add new feed" window any URL source with a RSS feed, press the "Fetch it now" button, then OK button.
2. Select it to edit, change all needed fields to expect JSON from the built-in browser and post-script, then OK button.
3. "Fetch selected" it, and articles list gets populated successfully with no errors.

Looks like "Fetch it now" and "Fetch selected" buttons follow different pathways for some reason in this dev build.

P.S. I would like to look into the Youtube problem. Which part of it couldn't be retrieved? Videos on its main page?

[martinrotter]

Maybe provide your setup first so we can find what was wrong when you was testing that JSON.

As for youtube yes, the page loaded, but video are simply not there

[DawEdhel]

Fresh unpacking of rssguard-devbuild-07eb444d9-win10.7z with no previous data (portable).

Inside the "Add new feed" window:
Type: JSON
Source: Local file with a path target
File's contents as an example: "{ "title": "Test", "items": [] }"

"Fetch it now" responds with an "Error: feed format not recognized".
"Application log" window's messages:
time=" 1113.736" type="warning" -> core: Feed guessing error: 'XML is not well-formed, Start tag expected.'.
time=" 1113.736" type="warning" -> core: Feed guessing error: 'XML is not well-formed, Start tag expected.'.
time=" 1113.737" type="warning" -> core: Feed guessing error: 'XML is not well-formed, Start tag expected.'.
time=" 1113.741" type="warning" -> core: Feed guessing error: 'not a JSON feed'.
time=" 1113.742" type="warning" -> core: Feed guessing error: 'XML is not well-formed, Start tag expected.'.

[martinrotter]

That is not a valid JSON feed 1.1.