Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scrape only missing resources, not entire files #344

Open
jmt-gh opened this issue May 24, 2022 · 2 comments
Open

Scrape only missing resources, not entire files #344

jmt-gh opened this issue May 24, 2022 · 2 comments

Comments

@jmt-gh
Copy link

jmt-gh commented May 24, 2022

Describe the feature / request
Currently, if I have a full set of media scraped, except for videos, and I want to scrape videos, Skyscraper will refresh the entire cache, and re-scrape everything I've already got. Ideally, Skyscraper could understand / have the ability to only scrape missing resources (currently the documentation only shows how to scrape entire files based on missing resources, not the individual resources themselves)

Additional context
This would save unnecessary hammering on screenscraper.fr, spurious API calls, and bandwidth

@jmt-gh
Copy link
Author

jmt-gh commented May 25, 2022

I figured out a way to do this in case anyone else ever runs in to this:

  1. do your scraping, generate your media, etc. etc.
  2. decide you want to scrape a new resource, like video
  3. in your config, set cacheCovers, cacheScreenshots, etc. to false
  4. in your config, set whatever you resource you want to true (e.g videos)
  5. run the scraper with --refresh. it should only be fetching videos (you can tell by the API requests remaining going down by 1 each game now, instead of 4 or 5 if you had everything enabled)
  6. generate your media / gamelist
  7. your media folder and gamelist should now have everything in it!

@jmt-gh
Copy link
Author

jmt-gh commented May 25, 2022

This does seem to require rehashing each file though, which is suboptimal given that the cache already has the hashes in quickid?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant