Some websites block archival process.

[luigifcruz]

Some websites are inaccessible to the archival process. Not sure if a solution is feasible.

• https://www.digikey.com/en/products/detail/analog-devices-inc./DC1524A-A/4890577
• https://www.trulia.com/building/caspian-260-brooklyn-basin-way-oakland-ca-94606-2710623398
• https://www.apartments.com/caspian-oakland-ca/pp0w7es (this one never gets added, it just hangs)
• https://indico.fnal.gov/event/46955/contributions/204594/attachments/138539/173497/GPU_Direct_IO_with_HDF5.pdf (can't download the PDF)

[daniel31x13]

This is because some websites prevent automated requests from accessing their content.
There's an upcoming file upload feature which lets users take the screenshot manually.

[luigifcruz]

Cool, file uploading is a very welcome feature and will mostly fix this problem.

I'm still a bit curious about how the websites are identifying this as an automated request. It can't be IP flagging because this is using my home IP address. Perhaps User-Agent matching?

[daniel31x13]

Oh, if that's the case you can add the following to the env file, it should fix most of the links:

AUTOSCROLL_TIMEOUT=1000 # Amount in seconds (default 30)
NEXT_PUBLIC_MAX_FILE_SIZE=1000 # Amount in MB (default 30)
IGNORE_UNAUTHORIZED_CA=true
IGNORE_HTTPS_ERRORS=true

[daniel31x13]

Also for the third link, just merged a hotfix in v2.5.2, please note that you'll need to add the following to the env file as well: IGNORE_URL_SIZE_LIMIT=true.

[luigifcruz]

The patch for the third like indeed works. The URL is added to the system. But it also breaks the archival of every URL. I noticed a new error popping up:

Linkwarden    | [1] Processing link https://hackaday.com/2024/04/14/hackaday-links-april-14-2024/ for user 1
Linkwarden    | [1] Something went wrong while retrieving the file size.

[luigifcruz]

Regarding the other variables, they unfortunately didn't help in preventing the detection of an automated request. 😕

[daniel31x13]

But it also breaks the archival of every URL.

Forgot to add a single line, fixed in v2.5.3.

[luigifcruz]

Thanks! I'll test this soon.

Regarding the bot detection circumvention, looks like you are using the vanilla Playwright which makes it very easy for the website to detect an automated request. I tried the example of Puppeteer Stealth plugin for Playwright and was able to successfully circumvent bot detection in 2 of the 3 links. Looking at the archival code, very little modification would be necessary to support plugins.

[daniel31x13]

That’s a great news!
If you want, make a PR for this and I’ll be merging it.
Otherwise I’ll be getting to it soon.