Protecting Against Distributed DOS Attacks

DDOS attacks are much harder to initiate and nearly impossible to stop. A DDOS attack begins with the penetration of hundreds or even thousands of weakly secured machines. These machines can then be directed to attack a single host based on the whims of the attacker.

With the advent of DSL and cable modem, millions of people are enjoying Internet access with virtually no speed restrictions. In their rush to get online, many of those people neglect even the most basic security. Since the vast majority of these people run Microsoft operating systems, they tend to get hit with worms and viruses rather quickly. Once the machine has been infiltrated, quite often the worm or virus installs a program on the victim's machine that instructs it to quietly call home and announce that it is now ready to do the master's bidding.

At the whim of the master, the infected machines can now be used to focus a concentrated stream of garbage data at a selected host. In concert with thousands of other infected machines, a scriptkiddie now has the power take down nearly any site on the Internet.

Detecting a DDOS is similar to detecting a DOS attack. One or more of the following signs are likely to be present:

Sustained saturated data link
No reduction in link saturation during off-peak hours
Hundreds or even thousands of simultaneous network connections
Extremely slow system performance

To determine if your data link is saturated, the act of pinging an outside host can tell much of the story. Much higher than usual latency is a dead giveaway. Normal ping latency (that is, the time it takes for a ping response to come back from a remote host) looks like the following:

# ping www.example.com
PING www.example.com (192.0.34.166) from 10.0.0.11: 56(84) bytes of data
64 bytes from 192.0.34.166: icmp_seq=1 ttl=49 time=40.1 ms
64 bytes from 192.0.34.166: icmp_seq=2 ttl=49 time=42.5 ms
64 bytes from 192.0.34.166: icmp_seq=3 ttl=49 time=39.5 ms
64 bytes from 192.0.34.166: icmp_seq=4 ttl=49 time=38.4 ms
64 bytes from 192.0.34.166: icmp_seq=5 ttl=49 time=39.0 ms
   
--- www.example.com ping statistics ---
5 packets transmitted, 5 received, 0% loss, time 4035ms
rtt min/avg/max/mdev = 38.472/39.971/42.584/1.432 ms

In the preceding example, the average time for a ping packet to make the round trip was about 39 thousandths of a second.

A ping to a nearly saturated link will look like the following:

# ping www.example.com
PING www.example.com (192.0.34.166): from 10.0.0.11: 56(84)bytes of data
64 bytes from 192.0.34.166: icmp_seq=1 ttl=62 time=1252 ms
64 bytes from 192.0.34.166: icmp_seq=2 ttl=62 time=1218 ms
64 bytes from 192.0.34.166: icmp_seq=3 ttl=62 time=1290 ms
64 bytes from 192.0.34.166: icmp_seq=4 ttl=62 time=1288 ms
64 bytes from 192.0.34.166: icmp_seq=5 ttl=62 time=1241 ms
   
--- www.example.com ping statistics ---
6 packets transmitted, 5 received, 0% loss, time 5032ms
rtt min/avg/max/mdev = 1218.059/1258.384/1290.861/28.000 ms

In this example, a ping packet took, on average, 1.3 seconds to make the round trip. From the first example to the second example, latency increased by a factor of 31! A data link that goes from working normally to slowing down by a factor of 31 is a clear sign that link utilization should be investigated.

For a more accurate measure of data throughput, a tool such as ttcp can be used. To test your connection with ttcp you must have installed the ttcp RPM package on machines inside and outside of your network. (The ttcp package comes on CD #3 included with this book.) If you are not sure if the package is installed, simply type ttcp at a command prompt. You should see something like the following:

# ttcp
Usage: ttcp -t [-options] host [ < in ]
       ttcp -r [-options > out]
Common options:
     -l ##     length of bufs read from or written to network (default 8192)
     -u     use UDP instead of TCP
     -p ##     port number to send to or listen at (default 5001)
     -s     -t: source a pattern to network
          -r: sink (discard) all data from network
     -A     align the start of buffers to this modulus (default 16384)
     -O     start buffers at this offset from the modulus (default 0)
     -v     verbose: print more statistics
     -d     set SO_DEBUG socket option
     -b ##     set socket buffer size (if supported)
     -f X     format for rate: k,K = kilo{bit,byte}; m,M = mega; g,G = giga
Options specific to -t:
     -n##     number of source bufs written to network (default 2048)
     -D     don't buffer TCP writes (sets TCP_NODELAY socket option)
Options specific to -r:
     -B     for -s, only output full blocks as specified by -l (for TAR)
     -T     "touch": access each byte as it's read

The first step is to start up a receiver process on the server machine:

# ttcp -rs
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket

The –r flag denotes that the server machine will be the receiver. The –s flag, in conjunction with the –r flag, tells ttcp that we want to ignore any received data.

The next step is to have someone outside of your data link, with a network link close to the same speed as yours, set up a ttcp sending process:

# ttcp -ts server.example.com
ttcp-t: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp  ->
server.example.com
ttcp-t: socket
ttcp-t: connect

Let the process run for a few minutes and then press Ctrl+C on the transmitting side to stop the testing. The receiving side will then take a moment to calculate and present the results:

# ttcp -rs
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 64.223.17.21
ttcp-r: 2102496 bytes in 70.02 real seconds = 29.32 KB/sec +++
ttcp-r: 1226 I/O calls, msec/call = 58.49, calls/sec = 17.51
ttcp-r: 0.0user 0.0sys 1:10real 0% 0i+0d 0maxrss 0+2pf 0+0csw

In this example, the average bandwidth between the two hosts was 29.32 kilobytes per second. On a link suffering from a DDOS, this number would be a mere fraction of the actual bandwidth the data link is rated for.

If the data link is indeed saturated, the next step is to determine where the connections are coming from. A very effective way of doing this is with the netstat command, which is included as part of the base Red Hat Linux install. Type the following to see connection information:

# netstat –tupn

Table 14-1 describes each of the netstat parameters used here.

Table 14-1: netstat Parameters
Parameter	Description
-t, --tcp	Show TCP socket connections.
-u, --udp	Show UDP socket connections.
-p, --program	Show the PID and name of the program to which each socket belongs.
-n, --numeric	Show numerical address instead of trying to determine symbolic host, port, or user names.

The following is an example of what the output might look like:

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address      Foreign Address      State      PID/Program name
tcp        0      0 65.213.7.96:22     13.29.132.19:12545   ESTABLISHED 32376/sshd
tcp        0    224 65.213.7.96:22     13.29.210.13:29250   ESTABLISHED 13858/sshd
tcp        0      0 65.213.7.96:6667   13.29.194.190:33452  ESTABLISHED 1870/ircd
tcp        0      0 65.213.7.96:6667   216.39.144.152:42709 ESTABLISHED 1870/ircd
tcp        0      0 65.213.7.96:42352  67.113.1.99:53       TIME_WAIT   -
tcp        0      0 65.213.7.96:42354  83.152.6.9:113       TIME_WAIT   -
tcp        0      0 65.213.7.96:42351  83.152.6.9:113       TIME_WAIT   -
tcp        0      0 127.0.0.1:42355    127.0.0.1:783        TIME_WAIT   -
tcp        0      0 127.0.0.1:783      127.0.0.1:42353      TIME_WAIT   -
tcp        0      0 65.213.7.96:42348  19.15.11.1:25        TIME_WAIT   -

The output is organized into columns defined as follows:

Proto — Protocol used by the socket.
Recv-Q — The number of bytes not yet copied by the user program attached to this socket.
Send-Q — The number of bytes not acknowledged by the host.
Local Address — Address and port number of the local end of the socket.
Foreign Address — Address and port number of the remote end of the socket.
State — Current state of the socket. Table 14-2 provides a list of socket states.

PID/Program name — Process ID and program name of the process that owns the socket.

Table 14-2: Socket States
State	Description
ESTABLISHED	Socket has an established connection.
SYN_SENT	Socket actively trying to establish a connection.
SYN_RECV	Connection request received from the network.
FIN_WAIT1	Socket closed and shutting down.
FIN_WAIT2	Socket is waiting for remote end to shut down.
TIME_WAIT	Socket is waiting after closing to handle packets still in the network.
CLOSED	Socket is not being used.
CLOSE_WAIT	The remote end has shutdown, waiting for the socket to close.
LAST_ACK	The remote end has shut down, and the socket is closed, waiting for acknowledgement.
LISTEN	Socket is waiting for an incoming connection.
CLOSING	Both sides of the connection are shut down but not all of our data has been sent.
UNKNOWN	The state of the socket is unknown.

During a DOS attack, the foreign address is usually the same for each connection. In this case, it is a simple matter of typing the foreign IP address into the search form over at http://www.arin.net/whois/ so you can alert your ISP.

During a DDOS, the foreign address will likely be different for each connection. In this case, it is impossible to track down all of the offenders, as there will likely be thousands of them. The best way to defend yourself is to contact your Internet provider and see if it can filter the traffic at its border routers.