Skip to content
This repository has been archived by the owner on Jun 10, 2024. It is now read-only.

PhantomJS can not fetcher the result of js_script #962

Open
ldybyz opened this issue Nov 14, 2020 · 1 comment
Open

PhantomJS can not fetcher the result of js_script #962

ldybyz opened this issue Nov 14, 2020 · 1 comment

Comments

@ldybyz
Copy link

ldybyz commented Nov 14, 2020

from pyspider.libs.base_handler import *
class Handler(BaseHandler):
crawl_config = {
}
@every(minutes=24 * 60)
def on_start(self):
self.crawl('https://movie.douban.com/explore#!type=movie&tag=%E7%83%AD%E9%97%A8&sort=recommend&page_limit=20&page_start=0',
fetch_type='js', js_script="""
function() {
setTimeout("$('.more').click();console.log('finish');", 2000);
}""", callback=self.phantomjs_parser)

@config(age=10 * 24 * 60 * 60)
def phantomjs_parser(self, response):
    return [{
        "rate": x('p strong').text(),
        "url": x.attr.href,
    } for x in response.doc('a.item').items()]

this is the code,I try a lot and the js_scriptis run,but the data is just the page first load

@gtarpenning
Copy link

phantomjs_1 | phantomjs fetcher running on port 25555

Screen Shot 2020-12-10 at 10 08 00 AM

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants