Pause gracefully if OSError (No space left on device)

[TheTechRobo]

Instead of crashing the crawl, why don't we simply pause and retry every so often if the computer has no space left?

[ivan]

I believe wpull and ludios_wpull do this if psutil is installed (psutil>=2.0,<=4.2 is in its requirements). I think I didn't include the requirement in grab-site for compatibility with Windows or macOS? Not sure.

https://github.com/ArchiveTeam/grab-site/blob/6269289a2ca874bae52f116016ca54dc8887d0cc/extra_docs/pause_resume_grab_sites.sh is the workaround I use to avoid crashes right now.

I agree this should be better and work by default.

[TheTechRobo]

Well, a 50GB crawl I had crashed. :/

Just installed it for the future.

[TheTechRobo]

Nope, didn't do anything. I have psutil yet my 33GB crawl just crashed because I ran out of space.

[TheTechRobo]

Where does this error pop up, I can't remember but I want to fix this.

Do you think just putting it in a try block would work and catch OSErrors, and pause the crawl, or should we do something else?