-
-
Notifications
You must be signed in to change notification settings - Fork 125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scrape only missing resources, not entire files #344
Comments
I figured out a way to do this in case anyone else ever runs in to this:
|
This does seem to require rehashing each file though, which is suboptimal given that the cache already has the hashes in quickid? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the feature / request
Currently, if I have a full set of media scraped, except for videos, and I want to scrape videos, Skyscraper will refresh the entire cache, and re-scrape everything I've already got. Ideally, Skyscraper could understand / have the ability to only scrape missing resources (currently the documentation only shows how to scrape entire files based on missing resources, not the individual resources themselves)
Additional context
This would save unnecessary hammering on screenscraper.fr, spurious API calls, and bandwidth
The text was updated successfully, but these errors were encountered: