Two problems with doc-scraper

[kappa-wingman]

Hello guys, I am using Pelican with MeiliSearch. I am not a front end developer but got a small problem with doc-scraper.
Overall I am very satisfy with MeiliSearch 👍

Problem 1:
I have problem understanding the doc-selector and global in the doc-scraper.

Problem 2:
These are my headings/anchors in the article.
When I type 'hosting' in the search box, the text stack together.
From WordPress to Pelican (h1)
Choosing the Static Site Generator (h2)
Customization of the theme (h2)
Choices of hosting service provider (h2)

Here is my config

{
  "index_uid": "docs",
  "sitemap_urls": [
    "https://www.kappawingman.com/sitemap.xml"
  ],
  "start_urls": [
    "https://www.kappawingman.com"
  ],
  "selectors": {
    "lvl0": {
      "selector": "#sb_Article_TOC",
      "global": true,
      "default_value": "Article"
    },
    "lvl1": "h1",
    "lvl2": "h2",
    "text": "p, li"
  },
  "strip_chars": " .,;:#",
  "scrap_start_urls": false,
  "nb_hits": 1137
}

Only articles like this with TOC would have a left sidebar.

Thanks.

[curquiza]

Hello @kappa-wingman!

Glad to read you are satisfied with MeiliSearch 😄

About your issues with docs-scraper, I'm going to give you a quick first answer, but be sure I will take time to answer better tomorrow or this weekend if my current answer does not help you.

I know, the docs-scraper README is not really detailed yet: issues are open for that, so be sure we plan to improve this README.
But you can find a better explanation in the article I wrote about "How to integrate a relevant search bar to your docs". Sorry If you have already found this article but you still don't understand how this is works.

To complete, here is the config file we use in production for our own documentation. It might help.

Tell me if you were already aware of both of these links or if they do not help!

[kappa-wingman]

Thanks for reply.

Yes, I am aware of the two links you had mentioned. Actually my current config is copied from those links.
Maybe vuepress (I haven't use it) has structure like "lvlX": ".docs-content .docs-lvlX" for every articles, which is their standard.

For Pelican SSG, there more cases. The official Pelican did not provide a standard way to generate table of content. They use plugins to generate html. Markdown has their own plugin or extensions. Restructuredtext had another plugin. I am using RST. There is no standard way to display the TOC. I put it on the sidebar or one may choose to display it at the top of the article.

Back to the question.
Looks like this setting
"lvl2": {
"selector": ".entry-content .section .toc-backref ",
"global": true,
"default_value": "Blog headings"
},
or this setting
"lvl2": "h2"

would get all the headings in 'h2' tag in a single line (stacked together).
I would like the search to display h2 tags individually.

Sorry, I may not be expressing the problem very well on this.
But at least I got the searching works okay with 'h1' headings and the content of the articles now.

[curquiza]

Hello again @kappa-wingman!

I'm not sure to really understand what exactly you wanted to change, so I followed this idea: find the best config file to scrape your website so that your website will provide a search bar with a great user experience 🙂

TLDR;

Here is the config file I suggest:

{
  "index_uid": "docs",
  "start_urls": [
    "https://www.kappawingman.com"
  ],
  "stop_urls": [
    "https://www.kappawingman.com/archives.html",
    "https://www.kappawingman.com/tags.html",
    "https://www.kappawingman.com/tags/",
    "https://www.kappawingman.com/categories.html",
    "https://www.kappawingman.com/categories/",
    "https://www.kappawingman.com/category/storage.html",
    "https://www.kappawingman.com/category/webdev.html",
    "https://www.kappawingman.com/category/siteinfo.html",
    "https://www.kappawingman.com/category/security.html"
  ],
  "selectors": {
    "lvl0": {
      "selector": ".navbar-nav .active",
      "global": true,
      "default_value": "Kappa ITC Wingman"
    },
    "lvl1": "#content h1",
    "lvl2": "#content h2",
    "text": "#content p, #content li"
  },
  "custom_settings": {
    "synonyms": {
      "static site generator": ["ssg"],
      "ssg": ["static site generator"]
    },
    "stopWords": ["of", "the", "for", "from"]
  },
  "scrap_start_urls": false,
  "nb_hits": 226
}

⚠️ Run this file with the v0.9.3 or later!

Explanation

What was the issues I noticed?

Here is the main "issue" with your website: there is the same content duplicated many times because you have a summary of each article on the home page, on the category pages, etc... It makes it harder to scrape because it would lead to duplicate contents in MeiliSearch, so in your search bar, like this:

The only article contents we want to scrape are the "final" and complete articles in this kind of URL: /posts/the_category/the_date/the_title (ex: https://www.kappawingman.com/posts/webdev/2020/06/11/further-improved-the-new-theme). So, the first step was to exclude all the contents that describe the article but that is not the complete article. That's why I use the stop_urls field.

Even after this first step, there is still an issue with the intermediated pages in .../post/... URL (like https://www.kappawingman.com/posts/2020/may/). The scraper also passes on them but I cannot add them to stop_urls because you could add new articles in the future on different dates: these URLs are dynamically added.
But I fixed this issue with selectors. Indeed, I noticed only the "real" article page contains an HTML tag with a content id. That's why you can see #content a the beginning of each selector. #content h1 means "take h1 content but only in the HTML tag with the id content". Thanks to that, no "summary" articles on the other pages were scraped!

More about selectors

Because you told me the selectors weren't clear to you, I will try to explain them better.

Selectors are needed to tell the scraper "I want to get the content in this HTML tag/id/class". This HTML tag/id/tag is a selector.
Once this content is scraped, it will be displayed this way in the search bar:

lvl0 selector in your config file

This selector is a little bit different from the other selectors in your config file. As you can notice in the picture above, the lvl0 corresponds to the main title in the search bar, so I wanted to make it relevant for your search bar!
Again I use the selectors: ".navbar-nav .active" means "take the content in the class active that is itself in the class navbar-nav". This corresponds to the activated tab in the top navigation bar of your website!
Now, instead of just "Article" you have "Webdev", "Security" (for your articles), or even "About" or "Privacy Policy". Thanks to that, the users understand better the results of the search.

The global: true means you want the same lvl0 (so, the same main title) for all the contents extracted from the same page.

What I removed

I removed the sitemap_urls field because the sitemap of your website does not contain any link that the scraper can not access from the start_urls.

I also removed the strip_chars field. This field is useful if a title contains characters you don't want to appear in the search. For example, if the title is "# Hello World", you don't want to get the # but just "Hello World", so you put # in the strip_chars array.

Tips to custom your MeiliSearch

You can notice I added custom_settings in your config file.

• the synonyms SGG <=> Static Site Generator allows the user to find all the results containing "Static Site Generator" with only typing "SSG" (and the opposite).
  Here is the dedicated page about synonyms in the official documentation.

• stop words: because your website provides articles with structured sentences, I advise you to add stop words. Indeed, the search-engine would not be "spoiled" by these tiny words and would focus on the main words, rendering more accurate results. Your website does not contain enough content to show you how useful is it, but be sure it will be when you will have more and more articles.
  Here is the dedicated page about stop-words in the official documentation.
  You can find more complete lists of English stop-words like this one.

Both of these settings are not mandatory, and your search bar would still work perfectly without them.

The result

You can now search the articles by title:

You can also find the article sub-titles (h2):

And get relevant results by just typing a word:

---

Your config file is now perfect to provide a relevant search bar!
I hope it's what you expected!
Feel free to ask any questions.

Your blog is really great by the way! I hope you'll keep adding content 😉

Edit

You can notice the highlight is sometimes buggy on the screenshots I did: it's because I tested on an old MeiliSearch version. The bug is fixed right now 😉

[kappa-wingman]

Hello @curquiza,

Thanks for the reply and explanation.
I added some comment and reference so it may be useful to other users that use Pelican SSG and Meilisearch.

1. About the duplicated content:
   It is the nature of the original Pelican theme (pelican-bootstrap3).
   They would put the same content in tags, categories, archives and the summary page.

The duplicated content would also cause problem with Google search.
Part of the content (upper part of the article) would have duplicated search result.

For search bots, I tried to change that behavior with robots.txt
https://www.kappawingman.com/robots.txt

Maybe docs-scaraper could have an option to read the robots.txt for crawl exclusion.

2. about the id="content"
   That's was added in the original pelican-bootstrap3 theme, for reference:
   Line 71
   https://github.com/getpelican/pelican-themes/blob/master/pelican-bootstrap3/templates/article.html#L71

3. Using the 'nav-link active' as the selector is superb!
   Now the pages (About / Privacy Policy) is also handled nicely.

4. Included is my config for docs-scraper for reference

{
  "index_uid": "docs",
  "start_urls": [
    "https://www.kappawingman.com"
  ],
  "stop_urls": [
    "https://www.kappawingman.com/archives.html",
    "https://www.kappawingman.com/posts/2020/",
    "https://www.kappawingman.com/tags.html",
    "https://www.kappawingman.com/tags/",
    "https://www.kappawingman.com/categories.html",
    "https://www.kappawingman.com/categories/",
    "https://www.kappawingman.com/category/storage.html",
    "https://www.kappawingman.com/category/webdev.html",
    "https://www.kappawingman.com/category/siteinfo.html",
    "https://www.kappawingman.com/category/security.html",
    "https://www.kappawingman.com/authors/",
    "https://www.kappawingman.com/authors/kappa.html"
  ],
  "selectors": {
    "lvl0": {
      "selector": ".navbar-nav .active",
      "global": true,
      "default_value": "Kappa ICT Wingman"
    },
    "lvl1": "#content h1",
    "lvl2": "#content h2",
    "text": "#content p, #content li"
  },
  "custom_settings": {
    "synonyms": {
      "static site generator": [
        "ssg"
      ],
      "ssg": [
        "static site generator"
      ]
    },
    "stopWords": [
      "a", "and", "as", "at", "be", "but", "by",
      "do", "does", "doesn't", "for", "from",
      "in", "is", "it", "no", "nor", "not",
      "of", "off", "on", "or",
      "so", "should", "than", "that", "that's", "the",
      "then", "there", "there's", "these",
      "this", "those", "to", "too",
      "up", "was", "wasn't", "what", "what's", "when", "when's",
      "where", "where's", "which", "while", "who", "who's",
      "with", "won't", "would", "wouldn't"
    ]
  },
  "scrap_start_urls": false,
  "nb_hits": 232
}

Finally, thanks Clémentine again for the help. 👍
The result from the from search bar looks much better than before.

[curquiza]

Maybe docs-scaraper could have an option to read the robots.txt for crawl exclusion.

That would be a good idea!! I keep that in mind! 🙂

Thanks for all your explanations about Pelican. I wasn't aware of this tool before!

Glad to read you like your new search bar! I see that is already in production in your website 🚀 Soooo cool!!!

[kappa-wingman]

Yes, it's nice to get a search bar with instant search. 🥇

Just found out Google had a specific behavior about robots.txt and noindex.
In order for noindex to be effective, Google crawl bots need to read a document first. So the document itself could not be blocked in robots.txt After the Google crawl bot read the document, if it had the noindex metadata, it would not index it.

Previously, even I had set robots.txt, I could not stop Google bot to discover and index other web pages automatically. So I end up adding this in some of my html:

<meta name="robots" content="noindex">

So it seems logical for docs-scraper could have an option to disable indexing for web pages with the noindex meta tag. Edit: On second thought, it seems complicated and 'it's a Google specific way'. Maybe checking robots.txt alone is enough, since robots.txt is a standard used by websites.

For reference, here is the official doc about it in Google.