/scrape public domain info from social media sites (like privacy policy)

[Gbillington1]

I'm using Firecrawl in my application to scrape privacy policies for many websites. It works great for most cases but fails with a 403 error when trying to scrape what Firecrawl considers "social media" sites. I got this error when trying to scrape the privacy policy of twitter (ironically x.com works), and Instagram:

https://twitter.com/en/privacy
https://help.instagram.com/155833707900388

I'm assuming that these sites are being blocked by some blacklist on your side, but it would be awsome if I could scrape pages that don't necessarily relate to the data on the platforms. I want the information about the privacy policy, which Is public domain info that isn't related to the data that is stored on the social platforms. If you guys could make it possible for me to scrape these types of sites it would be greatly appreciated!

[nickscamara]

Hey! Thanks for the feedback! Very good point, we will look into it on monday!

[nickscamara]

ccing @rafaelsideguide

[Gbillington1]

Awesome, thanks!

[nickscamara]

Hey @Gbillington1, just submitted a pr that might help with this for now. If you have a better idea, or other ways we can do this lmk.

[nickscamara]

@Gbillington1 Merged!