How to get medium.com archives?

[gerroon]

HI

I tried to archive https://medium.com/@xsmith-dev/getting-started-with-yarn-3-and-typescript-125e7b537e6c but it fails. Is this a site restriction or something else?

archivebox_1  | [*] [2023-12-18 18:21:58] Archiving 1/6 URLs from added set...
archivebox_1  |         Extractor timed out after 67s.
archivebox_1  |         Run to see full output:
archivebox_1  |             cd /data/archive/1702923714.13198;
archivebox_1  |             /usr/bin/chromium-browser --headless=new --no-sandbox --no-zygote --disable-dev-shm-usage --disable-software-rasterizer --run-all-compositor-stages-before-draw --hide-scrollbars --window-size=1440,2000 --autoplay-policy=no-user-gesture-required --no-first-run --use-fake-ui-for-media-stream --use-fake-device-for-media-stream --disable-sync "--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/)" --window-size=1440,2000 --dump-dom https://medium.com/@xsmith-dev/getting-started-with-yarn-3-and-typescript-125e7b537e6c
archivebox_1  |
archivebox_1  |       > wget
archivebox_1  | Internal Server Error: /add/
archivebox_1  | Traceback (most recent call last):
archivebox_1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
archivebox_1  |     response = get_response(request)
archivebox_1  |                ^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 181, in _get_response
archivebox_1  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
archivebox_1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 70, in view
archivebox_1  |     return self.dispatch(request, *args, **kwargs)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/usr/local/lib/python3.11/site-packages/django/utils/decorators.py", line 43, in _wrapper
archivebox_1  |     return bound_method(*args, **kwargs)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
archivebox_1  |     return view_func(*args, **kwargs)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/mixins.py", line 109, in dispatch
archivebox_1  |     return super().dispatch(request, *args, **kwargs)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 98, in dispatch
archivebox_1  |     return handler(request, *args, **kwargs)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/edit.py", line 142, in post
archivebox_1  |     return self.form_valid(form)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/app/archivebox/core/views.py", line 288, in form_valid
archivebox_1  |     add(**input_kwargs)
archivebox_1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox_1  |     return func(*args, **kwargs)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/app/archivebox/main.py", line 678, in add
archivebox_1  |     archive_links(new_links, overwrite=False, **archive_kwargs)
archivebox_1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox_1  |     return func(*args, **kwargs)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/app/archivebox/extractors/__init__.py", line 236, in archive_links
archivebox_1  |     archive_link(to_archive, overwrite=overwrite, methods=methods, out_dir=Path(link.link_dir))
archivebox_1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox_1  |     return func(*args, **kwargs)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/app/archivebox/extractors/__init__.py", line 199, in archive_link
archivebox_1  |     write_link_details(link, out_dir=out_dir, skip_sql_index=False)
archivebox_1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox_1  |     return func(*args, **kwargs)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/app/archivebox/index/__init__.py", line 335, in write_link_details
archivebox_1  |     write_json_link_details(link, out_dir=out_dir)
archivebox_1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox_1  |     return func(*args, **kwargs)
archivebox_1  |            ^^^^^^^^^^^^^^^^^^^^^

archivebox_1  |   File "/app/archivebox/index/json.py", line 99, in write_json_link_details
archivebox_1  |     atomic_write(str(path), link._asdict(extended=True))
archivebox_1  |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/app/archivebox/index/schema.py", line 193, in _asdict
archivebox_1  |     'snapshot_id': self.snapshot_id,
archivebox_1  |                    ^^^^^^^^^^^^^^^^
archivebox_1  |   File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 48, in __get__
archivebox_1  |     res = instance.__dict__[self.name] = self.func(instance)
archivebox_1  |                                          ^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/app/archivebox/index/schema.py", line 265, in snapshot_id
archivebox_1  |     return str(Snapshot.objects.only('id').get(url=self.url).id)
archivebox_1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox_1  |   File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 429, in get
archivebox_1  |     raise self.model.DoesNotExist(
archivebox_1  | core.models.Snapshot.DoesNotExist: Snapshot matching query does not exist.
archivebox_1  | "POST /add/ HTTP/1.1" 500 145
archivebox_1  |         Extractor timed out after 62s.
archivebox_1  |         Run to see full output:
archivebox_1  |             cd /data/archive/1702923714.13198;
archivebox_1  |             wget --no-verbose --adjust-extension --convert-links --force-directories --backup-converted --span-hosts --no-parent -e robots=off --timeout=60 --restrict-file-names=windows --warc-file=/data/archive/1702923714.13198/warc/1702924084 --page-requisites "--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.1 (+https://github.com/ArchiveBox/ArchiveBox/) wget/GNU Wget 1.21.3" --compression=auto https://medium.com/@xsmith-dev/getting-started-with-yarn-3-and-typescript-125e7b537e6c
archivebox_1  |
archivebox_1  |       > title
archivebox_1  |         Extractor failed:
archivebox_1  |             SSLError HTTPSConnectionPool(host='medium.com', port=443): Max retries exceeded with url: /@xsmith-dev/getting-started-with-yarn-3-and-typescript-125e7b537e6c (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)')))
archivebox_1  |         Run to see full output:
archivebox_1  |             cd /data/archive/1702923714.13198;
archivebox_1  |             curl --silent --location --compressed --max-time 60 --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.1 (+https://github.com/ArchiveBox/ArchiveBox/) curl/curl 8.4.0 (x86_64-pc-linux-gnu)" https://medium.com/@xsmith-dev/getting-started-with-yarn-3-and-typescript-125e7b537e6c
archivebox_1  |
archivebox_1  |       > readability
archivebox_1  |         Extractor failed:
archivebox_1  |             SSLError HTTPSConnectionPool(host='medium.com', port=443): Max retries exceeded with url: /@xsmith-dev/getting-started-with-yarn-3-and-typescript-125e7b537e6c (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)')))
archivebox_1  |         Run to see full output:
archivebox_1  |             cd /data/archive/1702923714.13198;
archivebox_1  |             curl ./{singlefile,dom}.html
archivebox_1  |
archivebox_1  |       > mercury
archivebox_1  |         Extractor timed out after 85s.
archivebox_1  |         Run to see full output:
archivebox_1  |             cd /data/archive/1702923714.13198;
archivebox_1  |             /app/node_modules/@postlight/parser/cli.js https://medium.com/@xsmith-dev/getting-started-with-yarn-3-and-typescript-125e7b537e6c --format=text
archivebox_1  |
archivebox_1  |       > htmltotext
archivebox_1  |         Extractor failed:
archivebox_1  |             SSLError HTTPSConnectionPool(host='javascript.plainenglish.io', port=443): Max retries exceeded with url: /getting-started-with-yarn-3-and-typescript-125e7b537e6c (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)')))
archivebox_1  |         Run to see full output:
archivebox_1  |             cd /data/archive/1702923714.13198;
archivebox_1  |             "(internal) archivebox.extractors.htmltotext" ./{singlefile,dom}.html
archivebox_1  |
archivebox_1  |       > media

[pirate]

try disabling SSL checking with archivebox config --set CHECK_SSL_VALIDITY=False and increasing the timeout with archivebox config --set TIMEOUT=120

[gerroon]

Thanks that seems to work. Is there a way to avoid these popups which block the page? I cant pass them in the archive (the bottom)

[pirate]

For the popups you can set up a chrome profile and cookies.txt file. Hide the popups while logged in with that profile, and then use it for archiving and the popups should stay hidden.

Instructions here: https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile

We may add more advanced automatic popup and adblocking in the future, follow these issues:

• Add ability to run JS scripts during archiving with Playwright/Puppeteer #51
• Architecture: Block ads and trackers during archiving #211