How To Rate Limit When Multiple Users are from the Same IP Address?

[szarkowicz]

Hi,

I have done some extensive searching and can not find the way to rate limit by unique user on a API that does not require a user to be logged in to use it. I have public facing API and I want to rate limit it from being spammed but if alot of users are at a single location and the rate limit uses the IP address, that IP Address is going to look the same to the server and it could end up blocking some of the users if they hit the IP address request limit.

All the example online just say use IP Address but I can not see that working if multiple users are connected to the same network.

Am I missing something for the rate limit?

Thanks in advance.

Sean

[animir]

@szarkowicz Hi, you can rate limit your API only if you have some unique keys. It may be IP address, but you don't want to use it. Can you use something else as unique identifier for a user?

[szarkowicz]

Hi @animir ,

I will provide some more info with my real world use case, we run large events that our players and spectators and connecting to the locations wifi, these players and spectators can pull up the sports bracket which calls the API to get the latest matchups at anytime. Some our events have over 2000 players and they are constantly refreshing the brackets to see where they are. This is a public link that is shared on social media which requires no login or user info to view the bracket. If I use IP it will get rate limited because it all would be the same coming from the same public IP address.

I am not sure how to get anything unique about the user when they do not have to be logged in to view these brackets.

I have tried to google this problem or similar issues and I can not find any solution that addresses large amount of users from the same location. They all mention a user ID but our users do not have to be logged in.

Any suggestions / help would be greatly appreciated.

[animir]

@szarkowicz This is going to be a little more complex.
You should make that user ID. Provide public access to API endpoint, only if user has one.
For that, there should be one more endpoint /get-user-id, which would return some string like UUIDv4, which is unique enough for this purpose.

User's browser or application (not sure, how they trigger API endpoints), should send that generated user ID with every request in specific HTTP Header or Cookie. That's not all, because who would stop malicious user from getting infinite number of user IDs from /get-user-id endpoint?

Now /get-user-id should be rate limited by IP. Limit should be defined by nature of your application and/or statistic. You should know, how many maximum users would get data from your API from single IP address.

Note, you can start collecting data about amount of unique users by IP address with rate-limiter-flexible also, since it stores amount of requests per period of time already.

I hope, this helps.

[szarkowicz]

@animir thanks for the suggestion / help

The API is called from an Angular site.

I will work on this solution over the next couple days. I can post some skeleton code here when I am done to and I can even add a section to modules read-me about how to rate limit when multiple users are behind the same public IP to possibly help others who might run across the same limitations / requirements. Would that be something others might want to see?

Thanks again

[animir]

@szarkowicz Hi, were you able to implement this approach?

[szarkowicz]

Hi Animir,

Not yet - I got pulled into another production issue that I have been working to resolve. Hopefully this week I will be able to get back to the rate-limit. As soon as I have it working I will post here.

Thanks for checking.

[yuvapoojary]

Generate a unique user ID when they enter your website/specific route and pass that ID in the body/header of http requests.
After that , in the server side you combine the ip address and the unique user ID forming a single key and consume that key in the ratelimiter.
For example

const middleware = (req, res, next) => {
  const ip = req.ip;
  const userId = req.body.userId; 
  const key = ip + userId;
  rateLimiter.consume(key)
  .then(...)
  .catch (...)
};

However the user can easily change their id if u pass it in the body without authentication.
If the page/route is not sensitivity, may be this is fine.

May be this is also possible with cookie/session.

[eazylaykzy]

Hi @animir ,

I will provide some more info with my real world use case, we run large events that our players and spectators and connecting to the locations wifi, these players and spectators can pull up the sports bracket which calls the API to get the latest matchups at anytime. Some our events have over 2000 players and they are constantly refreshing the brackets to see where they are. This is a public link that is shared on social media which requires no login or user info to view the bracket. If I use IP it will get rate limited because it all would be the same coming from the same public IP address.

I am not sure how to get anything unique about the user when they do not have to be logged in to view these brackets.

I have tried to google this problem or similar issues and I can not find any solution that addresses large amount of users from the same location. They all mention a user ID but our users do not have to be logged in.

Any suggestions / help would be greatly appreciated.

Like @animir stated, it will be a little bit complex, what I will suggest you use to get a unique identifier is store a hash on the local storage of any user trying to access the application, and send it alongside the call to the API when they try to pull up the sports bracket, the downside to this is, it can be cleared by a real malicious user, in this case I'd advice you use this premium library fingerprintjs, they did a great job with the library, it generates a unique identifier for the browser used, even while in incognito, and VPN (like they claim), you can test it out to see if it suits your use case.

[timo-klarshift]

If the user is malicious they would bypass fingerprints by using other tools than a browser.

[Kinuseka]

Session-cookie based identity, unique identity + IP identity

Per identity lets say will have 5 attempts but since cookie based identity is easily bypassed
We can have a wider rate-limit on the IP side with 50 attempts in let say 5 minutes.

This unfortunately affects other people on the herd but should slow down any brute force attempts by gross elimination of the herd.

Other than that there is no other way to accurately identify a user without intrusive methods (getting their unique ID via computer specification, kernel ID, OS). But if your application is very sensitive and requires at most monitoring and regulation then you may try these methods.