How to best stop hammering of my web server api?

[JeremyCanada26]

I have a web app that uses an API that I wrote to communicate with my MySQL database that was written in PHP. Currently, each user of the API is required to authenticate before using the API with a username/password. Every API request is logged into a Requests_History table which stores the API request type, the timestamp and the userId of the requester.

 

How can I easily stop hammering of the API via too many requests in a given time period?

 

1. first violation within 24 hours, 2 minute ban

2. second violation within 24 hours, 10 minute ban

3. second violation within 24 hours, 1 hour ban

[requinix]

You mean besides looking at the last access time in that table?

 

Example: check the number of accesses in the last $timelimit. If it's beyond some threshold then deny access for $period.

If you want adaptive measures, stick some kind of "threat level" counter in the user table (wherever the userID comes from). When authenticating, get that counter and adjust your $period accordingly. Should probably attach a "last threat at" timestamp (updated when appropriate) so the threat level counter can be lowered over time.

[JeremyCanada26]

How do I stop the case where the user sends 50 requests in 1 second? Each request still takes up a mysql connection so doesn't that mean 50 connections?

[requinix]

You can have your web server limit connections by IP address - $x hits in $y time, with a bad response if they hit faster than that. But then what if you want (eg) "premium" customers to use it as much as they want?*

 

Otherwise if you're running on 'nix (not Windows) then there are some other approaches.

 

* Then you should offer a way of running multiple queries with one request.

[JeremyCanada26]

Yes, I'm running ubuntu 10.4 and using a LAMP setup. I came across an apache module designed specifically for limiting requests by IP address and it detects hammering and eventually forwards the IP out to the firewall. However, this solution reportedly works very well on a single machine only.

 

In my development setup, it also would work very well since i'm using a single machine. However, When I launch, I'll be switching to 2 servers in a load balancing setup and probably scale upward from there as needed and so the triggering mechanism doesn't work very well with that kind of setup.

 

 

[requinix]

Can you put the IP filtering on the load balancer?

[jdavidbakr]

if you have access to iptables you can set up an iptables recent rule to prevent more than a certain number of requests from any IP in a given amount of time.  It's not entry-level iptables stuff but it's not too terribly difficult (just be careful that you don't lock yourself out  )

[JeremyCanada26]

I'm using amazon web services(amazon ec2). So my firewall is on their end. They do have an API available for add/removing IPs though so that's good.