scripts/check-more-info-urls.py: add script #12506
Conversation
github-actions
bot
added
documentation
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
vitorhcl
force-pushed
the
add-detect-broken-links-script
branch
from
9b7b17b
to
6474534
Compare
This comment was marked as outdated.
This comment was marked as outdated.
vitorhcl
changed the title
Co-authored-by: Sebastiaan Speck <[email protected]>
This comment was marked as outdated.
This comment was marked as outdated.
|
Hi @vitorhcl, Any updates on this? |
|
Hi @kbdharun, thanks for pinging me. I'm gonna try to do the pending fixes and documentation until Monday, but I'll leave the regex filter for another PR. |
Anyone knows why this is returning an error? Is it because of asynchronous functions? |
This comment was marked as outdated.
This comment was marked as outdated.
vitorhcl
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
Probably yeah, will check the script locally and maybe fix this issue. Edit. That didn't take a lot of time, fixed the issue and also updated the README file. It seems like some functions were imported but not actually used so removed it, for the e unused exception variable, I added it to the i.e. diff --git a/scripts/check-more-info-urls.py b/scripts/check-more-info-urls.py
index 5d055e9a5bd3f..847232bdef3ab 100644
--- a/scripts/check-more-info-urls.py
+++ b/scripts/check-more-info-urls.py
@@ -2,22 +2,19 @@
# SPDX-License-Identifier: MIT
"""
-A Python script to check for bad (HTTP status code different than 200) "More information" URLs accross all pages.
+A Python script to check for bad (HTTP status code different than 200) "More information" URLs across all pages.
-These bad codes tipically indicate a not found page or a redirection. They are written to bad-urls.txt with their respective URLs.
+These bad codes typically indicate a page not found or a redirection. They are written to bad-urls.txt with their respective URLs.
Usage:
python3 scripts/check-more-info-urls.py
"""
-import random
import re
import asyncio
-import sys
-from aiofile import AIOFile, Reader, Writer
import aiohttp.client_exceptions
from aioconsole import aprint
-from aiofile import async_open
+from aiofile import AIOFile, Writer
from aiopath import AsyncPath
MAX_CONCURRENCY = 500
@@ -62,7 +59,7 @@ async def process_file(
try:
content = await f.read()
except Exception as e:
- await aprint(file.parts[-3:])
+ await aprint(f"Error: {e}, File: {file.parts[-3:]}")
return
url = extract_url(content)Feel free to check it out and modify my changes @vitorhcl. |
Signed-off-by: K.B.Dharun Krishna <[email protected]>
|
@kbdharun your change LGTM, thank you for the fixes. |
|
Are you going to implement the domain rotation or do you want me to do that? |
Feel free to do it, I assigned myself for the previous change (and to sort this PR separately under my notifications 😅 ). |
- Write any HTTP status code != 200 to the output file. - Use better names for some functions. - Improve the initial page getting output wording and formatting. - Remove some unnecessary functions that makes the initial pages getting faster. - Write the output URLs to a CSV file, as the previous format was very similar. - Colorize the displayed HTTP status codes. The corresponding colors are documented in the CodeColors class.
…ntation - Extract URL parsing and bad URLs writing from main() to find_and_write_bad_urls() - Rename some functions, variables and parameters - Improve the documentation of the functions - Document the missing return types
Before, find_all_pages was getting all pages multiple times while iterating over the platforms too, which was hugely slowing down the page finding.
|
@vitorhcl is this PR ready for review? Or should it become draft until it is ready for review? |
Hmm it should become draft until it's ready for merge. PS: My 3 previous commits have bodies that explain each change. |
TODO: