Rendora + Angular 6 ?

[Murena7]

Hi guys, first i want say - big thanks for you job!
I try to use rendora with angular 6 - SPA, and all my attempts to make it works well, do not succeed.
On backend i use Node.js - and they return angular dist.
For testing I successfully setup and run rendora+headless chrome. I set target:url and backend:url - my local site host - for example: url: "http://localhost:80". Next i try open http://localhost:3001 - and my site open good and all works well, but if i set some crawlers bot user agent and reload page - i wait about 20-30 sec and page not load correctly (missed css style and some other) and not load successfully. I think may be i do some wrong, can somebody please write some instruction how setup rendora with angular 6 + node.js or tell what i do wrong ? I will be grateful. (p.s. now i use rendertr*n - they work well but - solution have low performance and no cache.)

[Murena7]

My Config:

debug: true
listen:
    port: 3001
target:
    url: "http://localhost:80"

backend:
    url: "http://localhost:80"

headless:
    waitAfterDOMLoad: 3000
    internal:
      url: http://127.0.0.1:9222
filters:
    userAgent:
        defaultPolicy: blacklist
        exceptions:
            keywords:
                - bingbot
                - yandex
                - google
server:
    enable: true

[geokb]

Hi @Murena7 and thanks for the nice words, as for the issue I see nothing suspicious in your config file, the latency should be a little higher than 3 seconds since you set waitAfterDOMLoad to 3000. I am actually suspicious of something else that could be the cause of this problem, is your server at localhost:80 serving also the static files? if so, rendora may be trying to render also the static files using the headless Chrome and there is no DOM event fired for those requests, you get stuck waiting for the final response "forever" (or actually until rendora cancels the operation using a timer whose timeout that can be configured via headless.timeout), to make sure that this is really the problem, please try to whitelist only the homepage, i.e. put this in your config file:

filters:
    paths:
        defaultPolicy: blacklist
        exceptions:
            exact:
                - /
    userAgent:
        defaultPolicy: blacklist
        exceptions:
            keywords:
                - bingbot
                - yandex
                - google

and try loading the homepage with for example curl curl http://127.0.0.1:3001 and curl -A "BingBot" http://127.0.0.1:3001 to see if it now works

Frankly I might deal with this this as a bug, I intentionally didn't check for the Accept http request header which tells if the needed response is HTML or not since I am not really sure if all crawlers add it. Also, I assume that most real world cases use a frontend server (e.g. nginx) for static files

[Murena7]

I reproduce your solution, so if i try curl http://127.0.0.1:3001 - return html page without js render, if i try curl -A "BingBot" http://127.0.0.1:3001 - return part js render page not fully render. So returned only part rendered page without any js request to side data api. My project for testing generate 70% Dom elements on the page after js make request to side API (to other domains). So rendora prerendered/show js page without any js request for the data - i think the problem is with this. May be i not fully understand how rendora work. Can you please also describe - what different between target and backend url ? if i understand correct - rendora read user-agent and if is user - return all what returned target url (without js render) - if this bot - render target url and return this ? I use rendertr*n and he render all correct. May be i not correct understand rendora, please describe. thanks

[Murena7]

Hi, guys ) I think i know why - rendora not work same in the case of my project as rendertrn. In faq rendertrn i found some info:

Auto detecting loading function
The service detects when a page has loaded by looking at the page load event, ensuring there are no outstanding network requests and that the page has had ample time to render.

May be you can implements some same - in rendora ?

[geokb]

@Murena7 I apologize for responding so late but my last 2 weeks were tough so I couldn't respond.

The service detects when a page has loaded by looking at the page load event, ensuring there are no outstanding network requests and that the page has had ample time to render

yeah, there is already an issue #5 for that and I guess this will be my top priority feature for now. But I guess this still wouldn't solve your problem since you applied enough latency via waitAfterDOMLoad and the final DOM after the additional requests have been made and rendered should be ready anyway. Are you sure that the headless Chrome instance is able to address that server? (e.g. if you are running the headless chrome docker container in some other network space that cannot see your localhost in your host)