[VENDOR REQUEST]: Digi-Key (US)

[PhillippOhlandt]

Vendor Name

Digi-Key

Vendor Country

US

Vendor URL

https://www.digikey.com/

Product Links

https://rpilocator.com/?vendor=digikeyus

Additional information

https://github.com/nerves-metal-detector/nerves_metal_detector/blob/master/CONTRIBUTING.md#adding-new-vendors-and-products-from-rpilocator

There is also a German site which could be covered as an additional vendor as well.

[bflatau]

@PhillippOhlandt I'll try this one next.

[PhillippOhlandt]

@bflatau alright. Vendor name is DigiKeyUs and it uses application/ld+json. But it is also showing the count of available items which needs to be parsed from the HTML (not inside json). Unfortunately, there is no meta information in the HTML so you need to parse the XX In Stock text.

It's as simple as that (as long as the shop doesn't put anything in front of the number).

iex(1)> Integer.parse "72 In Stock"
{72, " In Stock"}

Please cast 0 to nil.

[bflatau]

@PhillippOhlandt, have you come across this error?

[error] Product update job failed!

Job args:
%NervesMetalDetector.Vendors.DigikeyUs.ProductUpdate{url: "https://www.digikey.com/en/products/detail/raspberry-pi/SC0693/13530916", sku: "CM4101016"}

Error reason:
{:load_body, {:error, %HTTPoison.Error{reason: :timeout, id: nil}}}

If not, I'll try doing some research...

[PhillippOhlandt]

@bflatau Does it happen every time?

[bflatau]

@PhillippOhlandt yes every time :(

[PhillippOhlandt]

@bflatau Ok, found it. I already suspected that and it took some testing. Basically, they want to be smart asses and prevent simple scrapers. Unfortunately (for them), all they really can check for are the request headers. Those are easy to "fake" tho (you can see that in the .livemd file for getting the items list from rpilocator).

What you can do now is have this list of headers defined before or after the options variable in the fetch_availability/1 function.

headers = [
  {"Accept",
   "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"},
  {"User-Agent",
   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36"},
  {"Accept-Language", "en-US,en;q=0.9"},
  {"Accept-Encoding", ""}
]

Those 4 headers are actually required to get the request to work. Now you can pass these headers to the HTTPoison.get/3 function when loading the body, like so:

HTTPoison.get(url, headers, options)

[bflatau]

Ah awesome! I'll check this out in the next few days. Yeah I was thinking, if I'm getting the products in the livebook, there's gotta be SOME way that this should be working...

Thanks for all the help, I'm learning a ton!

[PhillippOhlandt]

@bflatau Hey, are you still planning on doing this one?

[bflatau]

Hey @PhillippOhlandt!

Sorry I've been bogged down with some life stuff--I've actually been thinking about this project lately and wanting to get back into it. If you want to take care of it, you won't hurt my feelings, but I hope to make some progress this weekend. If I don't get anything done by this weekend, just assume that I'll contribute when I can, but don't factor me into any of your decisions.

Thanks for checking in

--Ben

[PhillippOhlandt]

Hey @bflatau, sorry for the late reply. No worries, I will leave this for you for now.