Update crawler.ts: Add hostname check to keep crawler on the same domain. #15
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added hostname check before adding a url to the queue to make sure we don't follow social links to the end of the earth if editing the URLs we're crawling. If the found URL has the same hostname as the starting URL, it will add that URL to the queue. Thanks for the project!
|
Tested and works like a charm. I'd also like to extend crawler to take an optional regex, what do you think? |
HarounAns
suggested changes
Sep 24, 2023
src/app/api/crawl/crawler.ts
Outdated
| @@ -1,5 +1,5 @@ | |||
| import cheerio from 'cheerio'; | |||
There was a problem hiding this comment.
your linter shouldnt be making changes for this PR
dougwithseismic
commented
Sep 25, 2023
athrael-soju
reviewed
Oct 9, 2023
| @@ -80,8 +102,20 @@ class Crawler { | |||
|
|
|||
| private extractUrls(html: string, baseUrl: string): string[] { | |||
| const $ = cheerio.load(html); | |||
There was a problem hiding this comment.
Minor observation: This will give a 'deprecated' warning. You can do import { load } from 'cheerio'; and use the load by itself instead.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
The existing crawler logic adds any found URLs to the queue without checking if they belong to the same domain as the starting URL. This could result in the crawler venturing off into unrelated domains, especially social links, which may not be desirable for the scope of the crawl.
Solution
Introduced a hostname check in the addNewUrlsToQueue function. This ensures that only URLs that have the same hostname as the starting URL are added to the queue for crawling. This feature helps in restricting the crawler within the scope of the initial domain, thereby making the crawl more focused and efficient.
Type of Change
New feature (non-breaking change which adds functionality)
Test Plan
Run the crawler on a starting URL that contains both internal and external links.
Verify that only URLs belonging to the same hostname as the starting URL are added to the queue.
Optionally, you can print the queue or keep logs to verify that the URLs are indeed from the same hostname.