Run the Scraper with Travis CI

[titieo]

Hi guys. I want to integrate meilisearch to my vuepress site. I'm using Intel core 2 dou (macbook pro 2009) so I can't use docs-scraper on my laptop I choose to use travis ci. Here is my configuration (other things used for deploying to gh-page are deleted).

language: node_js
node_js:
  - lts/*
cache:
  npm: true
services:
  - docker
install:
  - yarn install
jobs:
  include:
    - name: Scraper
      script:
      - docker run -t --rm \
          --network=host \
          -e MEILISEARCH_HOST_URL='https://loctran016.gitlab.io/present/' \
          -e MEILISEARCH_API_KEY='myMasterKey' \
          -v docs-scraper.config.json:/docs-scraper/config.json \
          getmeili/docs-scraper:latest pipenv run ./docs_scraper config.json

But travis ci reported me a problem:

Is there any way to fix this. Thanks

• This is my repo
• The log as well

[curquiza]

Hello @LocTran016!

Hard to answer without any other information... Does Travis provide any other trace of the issue? docs-scraper is written in python and a traceback is displayed when an exception is raised.

Is https://loctran016.gitlab.io/present/ really a MeiliSearch instance? Because when I try to access this URL, it does not seem to be a MeiliSearch running.

What I should get when I try to access the health route:

curl -X GET -I https://demos.meilisearch.com/health
HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Mon, 12 Oct 2020 09:27:52 GMT
Content-Length: 0
Connection: keep-alive

What I really get with your URL:

$ curl -X GET -I https://loctran016.gitlab.io/present/health
HTTP/2 404
content-type: text/html; charset=utf-8
vary: Origin
content-length: 26782
date: Mon, 12 Oct 2020 09:28:11 GMT

Plus --network=host is only useful to say to docker the outside MeilSearch instance is in the local machine that hosted the docker, so it should not be useful in your case.

You should take a look on this article to use docs-scraper.

Tell me if it helps!

PS: I don't recommend you making your master-key publicly visible as you do right now. Everyone could push to your MeiliSearch instance if they are aware of this 😉

[curquiza]

Right below the sentence you highlight there is this sentence: Following on from the example at the first step with a link to the first step. You should click on it. This first step is an example of how to run a MeiliSearch instance and you would be able to define the master-key at this step.

In the docs I provided, there are three steps to do:

• Run a MeiliSearch instance
• Run the scraper
• Integrate the search bar

You should take a look at the entire article, not only the README of docs-scraper (the one you screenshot).

The MEILISEARCH_HOST_URL is the URL of your MeiliSearch instance. It depends on where you host it.

The scraper is linked to MeiliSearch and cannot work if no MeiliSearch are running.

[titieo]

I've followed the guide but there is a error (I've already installed pipenv and run pipenv install) :

Trans-MacBook-Pro:present loctran016$ pipenv run ./docs_scraper docs-scraper.config.json
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/core.py", line 1635, in system_which
    assert c.return_code == 0
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/cli/command.py", line 448, in run
    do_run(
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/core.py", line 2547, in do_run
    run_fn(*run_args, **run_kwargs)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/core.py", line 2474, in do_run_posix
    command_path = system_which(script.command)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/core.py", line 1637, in system_which
    result = fallback_which(command, allow_global=True)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/core.py", line 1583, in fallback_which
    result = find_python(finder, command)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 2196, in find_python
    result = finder.find_python_version(line)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/vendor/pythonfinder/pythonfinder.py", line 262, in find_python_version
    version_dict = self.parse_major(major, minor=minor, patch=patch, arch=arch)
  File "/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv/vendor/pythonfinder/pythonfinder.py", line 209, in parse_major
    int(version_dict["major"]) if version_dict.get("major") is not None else major
ValueError: invalid literal for int() with base 10: './docs_scraper'
Trans-MacBook-Pro:present loctran016$ 

[curquiza]

Have you a MeiliSearch instance running (locally or not)? How do you launch it? Because, as I said, your URL provided is not a MeiliSearch instance.

[titieo]

Oh, sorry. I've read this doc several times but I don't know what does the doc mean. Should I just follow the guide here (mean copy and paste the code or I have to change http://127.0.0.1:7700 to the link to my site. Btw, do I have to install a SDK. Thanks

[curquiza]

Should I just follow the guide here

This guide is perfect to install and run a MeiliSearch instance, but you have the same suggestion in the docs I provided in the part "Run a MeiliSearch instance".

I have to change http://127.0.0.1:7700 to the link to my site

No this URL should be the URL of a MeiliSearch instance not your website.

Btw, do I have to install a SDK.

You have no SDK to install to run a MeiliSearch instance and scrape your content. If you want to integrate a search bar to your website, we recommend installing docs-searchbar.js. But again, all is detailed in the article I linked you 😉