Oct 182015

Again... sigh.

I've recently relocated some web sites and blogs I manage to EC2 so I've been watching my logs a little closer than usual. That's how I detected a ginormous number of hits for wp-login.php from certain IP's:

They're not getting in because of some alternate security measures I've put in place, but it's annoying to see 47K redirects over the course of about 2 1/2 hours... so I installed the mod_evasive module for Apache.

The module itself was reasonably easy to install and configure... but I had to think a little bit about how to take advantage of its ability to take some action when an IP is blocked. Sure, I could just add a rule to my .htaccess files, but that doesn't prevent the traffic from hitting my sites. What I really wanted was a way to block them at the edge of my VPC. With the AWS command line interface, a bit of shell programming and mod_evasive's DOSSystemCommand setting, I got exactly what I needed.

It should be pretty easy to follow what's going on (read the comments)... I've reserved rules 1 through 999 for the hackers. AWS processes rules in order so I changed the ID for the default ALLOW rule from 100 to 1000 to make this work. The script won't add a rule for an IP if one already exists.

At the same time I create the ACL, I create another temporary script that will revert the DOS after 3 days have passed. This lets me recycle the designated range of rule ID's.

Once the script was known to be working, I modified the mod_evasive configuration to trigger the script:

It didn't work out of the gate, though. The first issue was that I originally configured DOSLogDir as /var/log/httpd/modevasive, but didn't create it. On my system, /var/log/httpd is owned and writable by root. My Apache modules are writing as apache. The following commands took care of the problem:

Note that if logging is failing, sending email notifications will fail as well. It seems that logging failures terminates the current operation. If you're concerned about whether logging is working, try a grep for evasive in /var/log/messages. If you have a permissions issue, you'll see messages like this one:

The second issue is that /bin/mail does not exist on my system. To resolve that, I created a symlink to /usr/sbin/sendmail:

If everything is working, you should see output in your system log:

With this fix, I see about 20-25 requests before the ACL kicks in and the hacker can no longer reach any port on my server.


Zdziarski's Blog of Things
How To Stop An Apache DDOS Attack With mod_evasive