A TINKERER'S SOLUTION TO DEALING WITH WEB SERVER DOS ATTACKS
------------------------------------------------------------


A while back I found that my web server (Apache2) was on occasion being brought 
to its knees by what were effectively DoS attacks. Repeated requests to the 
server opened so may Apache2 child processes that I'd have to shut it down and 
restart it. A little research revealed that these attacks came from a variety 
of foreign IP addresses, and when they came from domestic sources they were 
usually from addresses owned by the notorious Zhou Pizhong.

My server runs Ubuntu Linux, although the solution I found may be adaptable to 
work quite well on other similar Unix-like platforms. Several packages need to 
be installed. They're easily available from most, if not all distributions and 
may already be installed on your servers:

Monit - utility for monitoring services on a Unix system
fail2ban - a set of server and client programs to limit brute force authentication attempts
Python - an interpreted, interactive, object-oriented programming language
Lynx - a general purpose distributed information browser for the World Wide Web
Gawk - pattern scanning and processing language

Additionally, you'll need to configure your Apache2 web server to allow access 
from localhost to the pseudo-website 'server-status'. How this is done depends 
on your OS and distribution, but generally involves enabling the Apache2 status 
module (mod_status.c) and allowing access to the status page from the localhost 
address, 127.0.0.1, using the URL http://localhost/server-status. This is 
Apache system admin SOP and should be easy for anyone familiar with Apache 
server administration.  Other web servers may have similar mechanisms to 
display the list of currently running server child processes and the IP 
addresses to which they're connected.  If your web server is not Apache2 and 
the status informaion is in a different format, you'll also need to know how to 
use gawk (a.k.a. awk) to extract connecting IP addresses from the server status 
report.

The Apache2 server here is using mpm_prefork. Because I run a very small 
hosting service, the maximum number of Apache2 child processes, 
MaxRequestWorkers, is set to 150.  This number may be much larger for a large 
commercial hosting service; nonetheless, the same techniques apply.

The next link in the chain is a very stable and flexible service monitor daemon 
called monit. I use monit to make sure that every system service essential for 
our customers is running as it should.  Monit looks at the process table, 
and/or does rudamentary connectivity tests to make sure that these services are 
alive and doing their jobs. Monit is capable of counting the number of Apache2 
child processes and taking a specified action if this number exceeds a given 
threshold. The monit configuration stanza which does this is as follows:

  check process apache2 with pidfile /var/run/apache2/apache2.pid
     start program = "/etc/init.d/apache2 start"
     stop program = "/usr/local/sbin/apache_spy.sh"
     if children > 90 then restart
     if failed host linode.fmp.com port 80 protocol http
        and request "/index.html"
        then restart
  
This instructs monit to transfer control to /usr/local/sbin/apache_spy.sh if 
the number of Apache2 child processes exceeds 90. This figure can be adjusted up 
or down as needed, but should never exceed MaxRequestWorkers as set in the 
Apache2 configuration. This stanza also does a basic check to make sure the 
server is up and running, but it's the count of Apache2 children that we're 
interested in here. If the count of children exceeds 90, then control is passed 
to /usr/local/sbin/apache_spy.sh.

apache_spy.sh is a short shell script, as follows:

  #!/bin/bash
  # Runs if monit detects an overrun in the number of Apache2 child processes
  # and passes a list of connecting IP addresses to apache_dos_counter.py.
  #
  # See also /usr/local/sbin/apache_dos_counter.py,
  #
  # /etc/fail2ban/jail.local and /etc/fail2ban/filter.d/apache-dos.conf
  #
  # also, see /etc/monit/conf.d/apache2 which invokes this script
  #
  FN="Apache_restart_$(/bin/date +'%F').txt"
  /usr/bin/lynx -width=200 -dump http://localhost/server-status > /tmp/$FN
  /etc/init.d/apache2 stop
  cat /tmp/$FN | /usr/bin/mailx -s"Apache DoS Attack!" sysadmin@fmp.com
  cat /tmp/$FN | awk '{printf $11"\n"}'|egrep \
    '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' | \
    /usr/local/sbin/apache_dos_counter.py

This script invokes the text-based web browser lynx to request the 
server-status page (http://localhost/server-status) from the server and writes 
it to a text file in /tmp with a filename indicating the date of its creation, 
e.g.  "Apache_restart_2018-01-31.txt". The script could easily pipe the 
server-status page directly to the next link in the chain, 
apache_dos_counter.py, however saving it to a file preserves the state of the 
server at the time of the attack for manual analysis at a later time. If a 
server other than apache2 is in use, then the awk invocation (used to parse 
connecting IP addresses from the server status report) will need to be adjusted 
according to the format of this output.

This list of connecting IP addresses is piped to the Python (v2) script 
apache_dos_counter.py:

  #!/usr/bin/python
  #
  # See also, /usr/local/sbin/apache_spy.sh and /etc/monit/conf.d/apache2
  #
  """Accepts a list of IP addresses on stdin and logs a line to /var/log/authlog if the
  count of any single address exceeds mincount."""
  
  import sys
  import syslog as S
  
  # Change mincount as necessary to make the whole system work
  mincount = 60
  
  ipArray = {}
  nlines = sys.stdin.readlines()
  for line in nlines:
      ip = line.rstrip()
      if ipArray.has_key(ip):
          ipArray[ip] += 1
      else:
          ipArray[ip] = 1
  
  for ipn in ipArray:
      if ipArray[ipn] >= mincount:
          S.syslog(S.LOG_AUTH | S.LOG_WARNING, "Apache DOS Attempt: %s" % ipn)
	  sys.exit(0)

This script tallies the list of IP addresses, assigning a count to each one 
according to the number of times it occurs in the list. If any address occurs 
more than mincount times, the script causes a log entry to be written to 
/var/log/auth.log with the text "Apache DOS Attempt" followed by the first IP 
address which occurred more than mincount times in the tally. The Python 
variable mincount must be less than the trigger value in the above monit stanza 
(in this case 90) and assumes that the DoS attacker is coming in using a single 
IP address and that other IP addresses haven't already loaded up the apache2 
child process roster so that this sequence of actions is triggered when no 
single IP address exceeds a tally count of mincount. If an attacker is using 
multiple IP addresses simultaneously then this process may fail, and some other 
method of detecting an attack, such as counting addresses from the same /24 
address group, must be used. This algorithm is perhaps the weakest link in this 
method overall.

The final piece of this DoS trap is the fail2ban daemon. fail2ban scans system 
log files looking for identifying strings indicating brute-force attacks on 
services. If an attack is identified, fail2ban writes a rule to the kernel 
iptables INPUT chain in the default "filter" table so that further packets from 
the IP identified as the DoS attack source in apache_dos_counter.py are 
rejected with "icmp-port-unreachable" indicating that there is no valid network 
path for packets from the attacking client to reach the web server.

The fail2ban configurations are as follows:

In jail.local, we have the following stanza:

  [apache-dos]
  
  enabled  = true
  port     = http,https
  filter   = apache-dos
  logpath  = /var/log/auth.log
  maxretry = 0
  findtime = 31536000
  bantime  = 31536000

This invokes the fail2ban filter apache-dos to parse /var/log/auth.log, and if 
the filter finds a match, the offending IP address is banned for a year. 

The filter rule, in fail2ban's filter.d directory, is called apache-dos.conf 
and is as follows:

  [INCLUDES]
  
  [Definition]
  
  failregex = ^.*Apache DOS Attempt: <HOST>$
  
  ignoreregex =

So fail2ban triggers on the tag string inserted into auth.log by 
apache_dos_counter.py and identifies the associated IP address ("<HOST>") as 
deserving of blocking.

This algorithm basically relies on two daemon utilities, monit and fail2ban, 
which are extremely useful for a variety of security and system stability 
tasks. What I've outlined here simply harnesses them in series to deal 
effectively with a DoS attack on the web server. Both of these daemons have 
other tasks on FMP's server, monit monitoring several services, and fail2ban 
parsing the mail log and syslog for indications of brute-force attacks on the 
ssh and authenticated smtp services.

There may be better, simpler or more elegant ways of doing this job, but this 
one works for us, and in the spirit of "if it works, don't fix it", I haven't 
tried to improve on a working system once we have it working well enough to be 
effective and safe.

Any questions? Email Lindsay Haisley <fmouse@fmp.com>