How to Stop an Apache DDoS Attack with mod_evasive

server_ddos


The first inkling that I had a problem with a DDoS (Distributed Denial of Service) attack was a note sent to my inbox:

lfd on server1.myhostname.com: High 5 minute load average alert – 89.14

Apache DDoS

My initial thought was that a site on my server was getting Slashdotted or encountering the Digg or Reddit effect. I run Chartbeat on several sites where this occasionally happens and I will usually get an alert from them first. A quick look at the Extended status page from Apache showed that I had a much different kind of problem.

If a site is getting a lot of “good” natural traffic you will see a few things:

  • clients will be requesting all kinds of files from your site as a normal web browser would, and
  • the referring agent will show the link of the sending site, which you can verify.

In my case I had about 400 or so IP addresses requesting “/” from a little trafficked site of mine. Fortunately my Apache is well-tuned enough that it did not take the server down, which was crucial for diagnosing this problem. Otherwise the entire server may down and your only option is to reboot and stop apache from starting on boot. My apache logs showed the requests:

1
2
3
4
5
6
7
8
9
10
186.58.179.33 - - [21/Oct/2010:00:10:06 -0400] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.21022; .NET
CLR 3.5.30729; .NET CLR 3.0.30618)"
189.76.197.117 - - [21/Oct/2010:00:10:06 -0400] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.19) Gecko/20081201 Firefox/2.0.0.19"
186.58.179.33 - - [21/Oct/2010:00:10:06 -0400] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.21022; .NET
CLR 3.5.30729; .NET CLR 3.0.30618)"
186.6.168.11 - - [21/Oct/2010:00:10:07 -0400] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 2000) Opera 6.03 [en]"
197.0.165.121 - - [21/Oct/2010:00:10:07 -0400] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/0.4.154.25 Safari/525.19"
189.76.197.117 - - [21/Oct/2010:00:10:07 -0400] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.19) Gecko/20081201 Firefox/2.0.0.19"
197.0.165.121 - - [21/Oct/2010:00:10:07 -0400] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/0.4.154.25 Safari/525.19"
186.6.168.11 - - [21/Oct/2010:00:10:07 -0400] "GET / HTTP/1.1" 200 12474 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 2000) Opera 6.03 [en]"

I found out this is from a type of DoS called a sloworis attack. In this attack, a HTTP request is made to the server and the connection is ‘held open’, making multiple requests. A compromised botnet is used to hammer the server from all over the world on many different connections and IP addresses. From the standpoint of Apache, these are legitimate connections despite the frequency of them. It simulates someone sitting at a browser and hitting the refresh command a few times a second. While not a devastating attack with regards to bandwidth, it ties up your server and rejects legitimate connections.

Fortunately after some asking around I found the mod_evasive Apache module. This module is a very basic one that has a simple function: it will keep a hash table of IPs and pages requested and when a threshold is reached on a page or site it will “block” the IP with a 403 “Forbidden” error.

Installing the module is easy:

  1. Download the module onto your server:

    1
    # wget http://www.zdziarski.com/blog/wp-content/uploads/2010/02/mod_evasive_1.10.1.tar.gz
  2. Run the apache apxs command on the module which will compile it install it into your httpd.conf file (for Apache 2.0 – you are running 2.0 aren’t you?):
    1
    # apxs -cia mod_evasive20.c
  3. Set up the configuration file. You can enter the following into your httpd.conf main server configuration (for Apache 2.0 again)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    <IfModule mod_evasive20.c>
    DOSHashTableSize 3097
    DOSPageCount 3
    DOSSiteCount 100
    DOSPageInterval 3
    DOSSiteInterval 5
    DOSBlockingPeriod 300
    DOSLogDir "/var/log/httpd/modevasive/"
    DOSEmailNotify your@emailaddress.com
    </IfModule>

Now this works great to throw a forbidden page to the client, but that client is still taking up a slot on your server. Once all of your slots are filled up, other requests are queue and therefore your server is lethargic to non-responsive to real requests.

To fix this problem the mod_evasive module has a DOSSystemCommand option. Using this option you can have your server execute a command when a client trips the mod_evasive triggers. I use ConfigServerFirewall (csf) on my server, so I added the following command which specifies a 1 hour firewall ban on the IP, which effectively drops all of its traffic to the server:

1
DOSSystemCommand "/usr/bin/sudo /usr/sbin/csf -td %s 3600"

But wait a second! Apache doesn’t have access to the firewall normally. This is my one reservation about this proceedure. You need to give Apache access to the firewall programs (as root) via sudo so that it can execute this firewall block. This has other security implications, especially if you are on a multitenant server. We use visudo to do this.

1
# visudo

Add the following to the file:

1
2
3
User_Alias      APACHE = apache
Cmnd_Alias      FIREWALL = /sbin/iptables, /usr/sbin/csf, /sbin/ifconfig, /sbin/route
APACHE  ALL = (ALL) NOPASSWD: FIREWALL

Where apache is the apache user (typically web, www, apache or httpd – this depends on the system) and the FIREWALL binaries are the ones used in the csf script.

Now my system watches for 3 or more connections to the same page in 3 seconds and not only serves a forbidden response to them but will drop their traffic completely via iptables. Within about 30 seconds my server load was back to normal, serving connections faithfully just like any other time.

This module worked great in my situation, but DDoS attacks come in many flavors and sizes. You may have luck with the mod_qos Apache module, where you can fine tune connections to certain pages. If an attack is bad enough, it is possible you will need to move to a hardware based solution because you can only do so much at the server level. A hardware firewall mixed with load balancers, caches like varnish, and other tricks can help to mitigate these DoS attacks.

  • John

    I had a much larger attack. Apache/mod_evasive were not able to stop. Had to use an appliance from IntruGuard. That was pretty smooth.

  • Nice – yeah this works up until a certain point but you are still loading tasks through Apache. It worked for me on a lower-end server with 2gb ram and it was about a 400 IP attack – but what if you had 1000, 10000 or more IPs working at it?

    Looked up the IntruGuard appliance – looks sweet but at $10,000 seems like a lot for small organizations. Are there any ‘cheaper’ ways to do that type of work? Seems like there should be an open source solution you could install on hardware with 2 nics, similar to some of the other security appliances/firewalls I’ve seen.

  • Pingback: DDOS by several IPs with one connection - Admins Goodies()

  • Pingback: ?? ????? ?????????? (VPS) ??????? | ???? VPS()