Doesn't work with nitter.net

[vosian]

Nitter is an alternative frontend to twitter, I'm guessing the way it retrieves images makes bandwidth hero miss them, so they end up not being compressed.

https://github.com/zedeus/nitter/

[changhaitravis]

Twitter images seem to get proxied from the nitter servers so that they avoid hotlinking to twitter servers. Firefox bandwidth hero is set to scout the request and determines it's best not to forward to your bandwidth-hero server to proxy. This one's kind of due to unexpected behavior on their image proxy, which if they fixed it would cause the firefox bandwidth hero to behave more like chrome b-h.

Chrome/Chromium bandwidth hero dives right in and seems to infinitely ping pong some combination of 302 and 307 responses between the bandwidth hero proxy and the nitter image proxy.

This line of code in bandwidth-hero-proxy src/redirect.js line 9 using "encodeURI" may also be causing issues.

res.setHeader('location', encodeURI(req.params.url))

Since it seems to be double-encoding the previously encoded parts of the url, preventing a successful redirect.

encodeURI("https://nitter.net/pic/https%3A%2F%2Fpbs.twimg.com%2Ftweet_video_thumb%2FESZQUfiUEAANy0y.jpg");
"https://nitter.net/pic/https%253A%252F%252Fpbs.twimg.com%252Ftweet_video_thumb%252FESZQUfiUEAANy0y.jpg"

However, fixing the above's won't actually fix the fact that it's not compressing the images, since it's still probably getting a 400+ HTTP Error somehow when the bandwidth-hero proxy server requests it so it falls back to a 302 redirect.

[vosian]

So this would require fixing the setHeader function here and change the how nitter proxies the images in order for B-H to work?

I guess to fix the double encoding that it'd have to check whether it is already encoded or not, somehow I think there could be lots of edge cases for that. Special characters that are encoded use %, but I guess there could also be sites that have % somewhere in their urls. I don't know how easy it'd be to attempt to parse the url, there are probably characters that require a lot more than the % and two characters I normally see.

But that still leaves the problem of the link being usable, would adding some way to detect redirections add too much complexity to the project?

[vosian]

I think that the following changes to redirect.js would make this work, I wouldn't expect the decoding or regex to introduce any problems, but considering I just tried a few strings using your query as an example (as I don't know how req.params.url is actually formatted) I can't guarantee it.

function redirect(req, res) {
  if (res.headersSent) return

  res.setHeader('content-length', 0)
  res.removeHeader('cache-control')
  res.removeHeader('expires')
  res.removeHeader('date')
  res.removeHeader('etag')
  url = decodeURIComponent(req.params.url)             // ensure that the url doesn't get double encoded
  urlMatch = url.match(/(https?:\/\/)(?!.*\1)[a-z0-9]+([\-\.]{1}[a-z0-9]+)*\.[a-z]{2,5}(:[0-9]{1,5})?(\/.*)?/)[0] // if there's only one url it'll return the same, if there's more than one (determined by the existance of http://) it'll return the last instance
  if(urlMatch) url = urlMatch // so that if regex returns null it doesn't overwrite the url
  res.setHeader('location', encodeURI(url))
  res.status(302).end()
}

module.exports = redirect

[vosian]

I made a fork with that code along a few console.log() and deployed another heroku instance, I can see redirect being used when using nitter, but only for a handful of items like https://nitter.net/apple-touch-icon.png the actual images taken from twitter aren't there, so I think for these links redirect.js isn't being used at all @changhaitravis

[changhaitravis]

Hi, can you confirm if you're using the Chrome or Firefox extension?
It really sounds like you're using the firefox extension.

For chrome, I don't think the current code should be causing an issue. The redirect.js confusion is on my side since I use a fork of bandwidth-hero with google cloud functions, and specifically that is having some issues right now. (sorry about sending you on a wild goose chase)

For firefox, there's a HEAD request that's sent that's not being handled by their proxy properly (404), and I can't really fix it outside of a whitelist or exposing the HEAD request scouting/recon functionality as optional (this is probably a good practice anyways).

[vosian]

indeed I'm using firefox, it's just byt the way you had phrased your original message it sounded like chrome itself had problems handling this (just different problem). What would changing how the head request is handled entail?

[changhaitravis]

If you have your own signed build of the firefox extension, you'd change in

bandwidth-hero/src/background.js

Line 132 in aa8536e

if (!isFirefox()) return { redirectUrl }

to return { redirectUrl }
and delete everything after that line up to line 151

[changhaitravis]

optimally the UI for the firefox version would just have a checkbox to toggle that functionality on and off. I can add that to my existing pull request #26, but idk when that'd get merged. Could also start a new branch and put that in as its own pull request.

[vosian]

Unfortunately I don't have my own build, so I'll have to wait, I guess

[vosian]

@changhaitravis I got a friend to build me the extension with the changes you suggested, now BH attempts to proxy the images but fails as expected, the problem is that it fails with my changes as well, checking the logs on heroku it appears the links are getting to redirect.js and then stripped from the https://nitter.net/pic/ part (I check by using console.log(res.getHeader('location'))), yet, the images don't load in the browser, and when I check the requests they are still prefixed by https://nitter.net/pic/ so it's like what in res isn't actually used

[changhaitravis]

if it made it to redirect.js, then I think that means that you're running into the same issue I ran into on google cloud functions. You can try my proxy at https://dizzy.fenesisu.moe/bandwidth-hero that's using this code https://github.com/changhaitravis/bandwidth-hero-proxy-gcf/tree/upstream-backport hosted on a linux VM server instead of as serverless or PaaS, if it works you can try deploying that on heroku, but I think you'll run into the same issue.

[vosian]

even trying to change it within proxy itself before it reached the redirect, also tried changing the req.params.url value itself to no avail, in the browser it's still getting prefixed. At this point I'll make the uneducated guess that the problem may lay somewhere in "request" itself

[vosian]

still nothing, attempted to make my own proxy call from within redirect after changing req.param.url to no avail. Also, @changhaitravis I think there's no ping-pong ing going on here, I only see each redirected link once, I mentioned this as you menitoned it should be going back and forth between 302 and 307, unless that doesn't happen in the proxy itself but in the browser.