22.4 Case Studies

The following stories are all true. In the first case, the names and a few details have been changed to protect people's jobs. The second and third stories are based on actual cases that took place at Vineyard.NET, an Internet Service Provider that is partly owned by one of the authors.

22.4.1 Rootkit

Late one night, a part-time computer consultant at a Seattle-based firm logged into one of the computers that he occasionally used. The system seemed sluggish, so he ran the top command to get an idea of what was slowing the system. The consultant noticed that a program called vs was consuming a large amount of system resources. The program was running as superuser.

Something didn't look right. To get more information, the consultant ran the ps command. That's when things got stranger still?the mysterious program didn't appear when ps was run. So the consultant used the top command again, and, sure enough, the vs program was still running.

The consultant suspected a break-in. He started looking around the filesystem using the Emacs dired command and found the vs program in a directory called /var/.e. That certainly didn't look right?why was a program running in a hidden directory on the /var/ partition? So the consultant went to his shell window, did a chdir( ) to the /var directory, and then did a ls -a. But the ls program didn't show the directory /var/.e. Nevertheless, the program was definitely there: it was still visible from the Emacs dired command.

The consultant was now pretty sure that somebody had broken into the computer. And the attack seemed sophisticated because system commands appeared to have been altered to hide evidence of the break-in. Not wanting to let the break-in proceed further, he wanted to shut down the computer. But he was afraid that the attacker might have booby-trapped the /etc/halt command to destroy traces of the break-in. So before shutting down the system, he used the tar command to make a copy of the directory /var/.e, as well as of the directories /bin and /etc. As soon as the tar file was made, he copied it to another computer and halted the system.

The following morning, the consultant analyzed the tar file and made the following observations:

  • Somebody had broken into the system.

  • The program /bin/login had been modified so that anybody on the Internet could log into the root account by typing a special password.

  • The /var/.e/vs program that had been left running was a password-sniffing program. It listened on the company's local area network for users typing their passwords; these passwords were then sent to another computer elsewhere on the Internet.

  • The program /bin/ls had been modified so that it would not display the directory /var/.e.

  • The program /bin/ps had been modified so it would not display the vs program.

  • The inode creation dates and the modification times on the files /bin/ls, /bin/ps, and /bin/login had been reset to their original dates before the modifications took place. The checksums for the modified commands (as computed with the sum command) matched those of the original, unmodified versions. But a comparison of the programs with a backup made the previous month revealed that the programs had been changed.

It was 10:00 p.m. when the break-in was discovered. Nevertheless, the consultant telephoned the system manager at home. When he did, he discovered something else:

  • The computer's system manager had known about the break-in for three days, but had not done anything about it. The reason: she feared that the intruder had created numerous holes in their system's security, and was afraid that if she angered the intruder, that person might take revenge by deleting important files or shutting down the system.

In retrospect, this was a poor decision. Allowing the intruder to stay on the system let him collect more passwords from users of the system. The delay also allowed for plenty of time to make further modifications to the system. If the system was only somewhat compromised before, it was in all likelihood thoroughly compromised now!

Leaving the intruder alone also left the company in a precarious legal position. If the intruder used the system to break in anywhere else, the company might be held partially liable in a lawsuit because they left the intruder with free run of the compromised system.

So what should the system manager have done when she first discovered the break-in? Basically, the same thing as what the consultant did: take a snapshot of the system to tape or another disk, isolate the system, and then investigate. If the staff was worried about some significant files being damaged, they should have done a complete backup right away to preserve whatever they could. If the system had been booby-trapped and a power failure occurred, they would have lost everything as surely as if they had shut down the system themselves.

This case is typical of many Unix break-ins. The attackers have access to one of many "rootkits" used to break into systems, install password sniffers, and alter system programs to hide their presence. Many of the users of these toolkits are quite ignorant of how they work.

22.4.2 Warez

On May 19, 1998, employees at Vineyard.NET noticed that the primary web server was acting very strangely. Processes were inexplicably terminating. Sometimes web pages were being displayed, but other times they were not. The system itself, an Intel-based Unix system running BSD/OS Version 3.2, had a very low load, but its response was incredibly sluggish.

The staff spent half an hour in an attempt to diagnose the problem. They ran the top command to see if there were any processes that were eating up the computer's available CPU; no process was using more than its fair share. (The ISP had previously had a problem with "runaway" processes occasionally bogging down the system.) They checked the amount of free disk space and memory, but storage was not a problem. As a last resort, the staff rebooted the computer. For a while, that seemed to solve the problem. But slowly, the sluggishness and strange behavior returned.

One of the tools used by Vineyard.NET to monitor its systems was the Multi-Router Traffic Grapher, better known as MRTG. As the employees started looking at more and more systems in an attempt to figure out what was going on, somebody looked at the MRTG traffic graph (see Figure 22-1). The graph indicated that Vineyard.NET's outgoing traffic was much higher than normal.

Figure 22-1. Vineyard.NET MRTG monitor

Clearly, the data was coming from somewhere. But where?

The netstat command (shown in Example 22-3 and illustrated in Figure 22-1) revealed that there were a large number of FTP connections originating from the Vineyard.NET server to other hosts on the Internet. This seemed quite strange. Each connection was accompanied by a connection on port 20, the FTP data port, indicating that a file transfer was in progress

Example 22-3. netstat shows active connections in progress during the warez incident
% netstat | grep ESTABLISHED
tcp        0      0  VINEYARD.NET.pop       ASY5.VINEYARD.NE.2117  ESTABLISHED
tcp        0   8500  VINEYARD.NET.http      srry01m05-128.bc.1505  ESTABLISHED
tcp        0   7168  VINEYARD.NET.http      hd62-160.hil.com.2033  ESTABLISHED
tcp        0   8192  VINEYARD.NET.http     ESTABLISHED
tcp        0   7552  VINEYARD.NET.20        hades.osc.epsilo.2943  ESTABLISHED
tcp        0   6952  VINEYARD.NET.http      ww-tl01.proxy.ao.37672 ESTABLISHED
tcp        0   7096  VINEYARD.NET.20        spc-isp-mon-uas-.1042  ESTABLISHED
tcp        0   7680  VINEYARD.NET.1117      r18m69.cybercabl.1177  ESTABLISHED
tcp        0   8496  VINEYARD.NET.http      cs206-32.student.1069  ESTABLISHED
tcp        0   8192  VINEYARD.NET.20        wend10.swol.de.1328    ESTABLISHED
tcp        0      0  SMTP4.VINEYARD.N.erpc  ANNEX1.VINEYARD..1024  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       dns1.bit-net.com.2268  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       spc-isp-mon-uas-.1037  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       kenny26.zip.com..1033  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       sladl3p24.ozemai.1676  ESTABLISHED
tcp        0   8760  VINEYARD.NET.pop       ASY10.VINEYARD.N.1043  ESTABLISHED
tcp        0   7360  VINEYARD.NET.20    ESTABLISHED
tcp        0   7340  VINEYARD.NET.1093      ESTABLISHED
tcp        0   7928  VINEYARD.NET.20        semicon.chungbuk.41987 ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       r18m69.cybercabl.1155  ESTABLISHED
tcp        0   8616  VINEYARD.NET.20        ppp068.0.mmtl.vi.1144  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       hades.osc.epsilo.2532  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       wend10.swol.de.1319    ESTABLISHED
tcp        0   7296  VINEYARD.NET.1076      slip-32-100-165-.2700  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       slip-32-100-165-.2699  ESTABLISHED
tcp        0   6656  VINEYARD.NET.1075      friday.datasourc.3024  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp    ESTABLISHED
tcp        0   8512  VINEYARD.NET.1067      slip-32-100-165-.2698  ESTABLISHED
tcp        0   7168  VINEYARD.NET.20        ezvl-78ppp122.ep.3783  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       ezvl-78ppp122.ep.3782  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       slip-32-100-165-.2695  ESTABLISHED
tcp        0   7168  VINEYARD.NET.20        t2o64p89.telia.c.1858  ESTABLISHED
tcp        0   7680  VINEYARD.NET.1043      r18m69.cybercabl.1123  ESTABLISHED
tcp        0      0  VINEYARD.NET.ftp       r18m69.cybercabl.1122  ESTABLISHED
tcp        0   8112  VINEYARD.NET.20        proxy1.fm.intel..3166  ESTABLISHED
tcp        0   6656  VINEYARD.NET.20        slmel21p25.ozema.1207  ESTABLISHED
tcp        0   8312  VINEYARD.NET.20        ing079.ing.hb.se.1297  ESTABLISHED

At this point the staff looked at the processes again. This time, however, instead of simply running top or looking at the first page of the ps aux output, they looked at all of the processes. There were 106 copies of the FTP daemon running, as shown in Example 22-4. From the FTP commands, it was evident that most of these individuals were downloading files with names like /calibreX/Win98.Final-PWA/pwa98cbl.zip\r\n from the directory /open.

Example 22-4. The process list during the warez incident
% ps aux
simsong   1770 86.4  2.0  5184 5212  p3  R     5:34PM    4:47.73 /usr/local/bin/perl /usr/
 -v (report.www)
root     24659 31.4  0.0     0    0  ??  Z     4:19PM    0:00.00 (admin_server)
root      2345  2.0  0.1   220  284  ??  S    31Dec69    0:00.02 (ping)
root      1406  0.0  0.0     0    0  ??  Z     5:32PM    0:00.00 (junkbuster)
root         0  0.0  0.0     0    0  ??  DLs  Mon01PM    0:00.30 (swapper)
root         1  0.0  0.1   148  288  ??  Ss   Mon01PM    0:01.63 /sbin/init
root         2  0.0  0.0     0   12  ??  DL   Mon01PM    0:00.01 (pagedaemon)
root        15  0.0  0.0    68   64  ??  Is   Mon01PM    0:00.00 asyncd 2
root        17  0.0  0.0    68   64  ??  Is   Mon01PM    0:00.02 asyncd 2
root        26  0.0  0.8   748 2008  ??  Ss   Mon01PM    0:00.67 mfs -o rw -s 40960 /dev/
sd0b /tmp (mount_mfs)
root        51  0.0  0.1   268  296  ??  Ss   Mon01PM    0:02.92 gettyd -s
root        62  0.0  0.1   160  340  ??  Ss   Mon01PM    1:19.11 syslogd
daemon      65  0.0  0.1   112  184  ??  Ss   Mon01PM    0:01.36 portmap
root        72  0.0  0.1   216  300  ??  Ss   Mon01PM    0:01.34 mountd
root        74  0.0  0.1   144  288  ??  Is   Mon01PM    0:00.01 nfsd-master (nfsd)
root        76  0.0  0.0    76  100  ??  I    Mon01PM    0:00.00 nfsd-server (nfsd)
root        77  0.0  0.0    76  100  ??  I    Mon01PM    0:00.04 nfsd-server (nfsd)
root        78  0.0  0.0    76  100  ??  I    Mon01PM    0:00.00 nfsd-server (nfsd)
root        79  0.0  0.0    76  100  ??  I    Mon01PM    0:00.00 nfsd-server (nfsd)
root        80  0.0  0.0    76  100  ??  I    Mon01PM    0:00.00 nfsd-server (nfsd)
root        81  0.0  0.0    76  100  ??  I    Mon01PM    0:00.01 nfsd-server (nfsd)
root       120  0.0  0.0    72   76  ??  Ss   Mon01PM    0:31.99 update
root       122  0.0  0.1   340  268  ??  Ss   Mon01PM    0:02.77 cron
ftp      11452  0.0  0.2   288  480  ??  S     2:07PM    0:00.27 cmodem85.lancite.net: 
anonymous/getright@: RE
TR /open/  /calibreX/Win98.Final-PWA/pwa98cbi.zip\r\n (ftpd)
ftp      11559  0.0  0.2   288  480  ??  S     2:09PM    0:00.28 cmodem85.lancite.net: 
anonymous/getright@: RE
TR /open/  /calibreX/Win98.Final-PWA/pwa98cbg.zip\r\n (ftpd)
ftp      13154  0.0  0.2   288  480  ??  S     2:21PM    0:00.24 cmodem85.lancite.net: 
anonymous/getright@: RE
TR /open/  /calibreX/Win98.Final-PWA/pwa98cbj.zip\r\n (ftpd)
ftp      13238  0.0  0.2   288  480  ??  S     2:22PM    0:00.25 cmodem85.lancite.net: 
anonymous/getright@: RE
TR /open/ /calibreX/Win98.Final-PWA/Microsoft_WIndows98_FINAL_Retail_Full_Setup-PWA/
ftp      13381  0.0  0.2   740  496  ??  S     2:23PM    0:00.71 port2d07.h2o.or.jp: 
anonymous/ta@tvr.com: IDL
E (ftpd)
ftp      13750  0.0  0.2   288  480  ??  S     2:26PM    0:12.34 port2d07.h2o.or.jp: 
anonymous/ta@tvr.com: RET
R /open/  /calibreX/Win98.Final-PWA/pwa98cbe.zip\r\n (ftpd)
ftp      13909  0.0  0.2   288  480  ??  S     2:28PM    0:00.25 cmodem85.lancite.net: 
anonymous/getright@: RE
TR /open/  /calibreX/Win98.Final-PWA/pwa98cbk.zip\r\n (ftpd)
ftp      14891  0.0  0.2   288  480  ??  S     2:38PM    0:00.24 cmodem85.lancite.net: 
anonymous/getright@: RE
TR /open/  /calibreX/Win98.Final-PWA/pwa98cbl.zip\r\n (ftpd)
ftp      15702  0.0  0.2   740  496  ??  S     2:45PM    0:23.34 dialup2-52.itv.se: 
RETR pwa98cbl.zip\r\n (ftpd)
ftp      16299  0.0  0.2   300  492  ??  I     2:52PM    0:04.61 anonymous/
bpftpuser@bpftp.com: I
DLE (ftpd)
ftp      17530  0.0  0.2   740  496  ??  S     3:06PM    0:01.05 client-151-197-126-3.
bellatlantic.net: anonym
ous/busta@rhymes.edu: RETR /open/  /calibreX/Win98.Final-PWA/pwa98cbh.zip\r\n (ftpd)
ftp      17713  0.0  0.2   740  496  ??  S     3:08PM    0:06.06 du126.str.ptd.net: 
anonymous/guest@: RETR pwa
98cbe.zip\r\n (ftpd)
ftp      18259  0.0  0.2   740  496  ??  S     3:14PM    0:01.18 p9-term1-and.netdirect.
net: anonymous/guest@:
 RETR pwa98cbk.zip\r\n (ftpd)
ftp      19393  0.0  0.2   740  496  ??  S     3:26PM    0:01.55 sc9-14-54.thegrid.net: 
ftp.com: RETR /open/  /calibreX/Win98.Final-PWA/pwa98cbk.zip\r\n (ftpd)
ftp      20077  0.0  0.2   740  496  ??  S     3:31PM    0:05.66 apus.osir.hihm.no: 
anonymous/anon@: RETR pwa9

A quick look at the FTP log file /var/log/xferlog confirmed this suspicion, as shown in Example 22-5.

Example 22-5. An excerpt from /var/log/xferlog/open
Tue May 19 17:35:58 1998 1 friday.datasource.net 7045 /open/_ _/calibreX/Win98.Final-PWA/
Microsoft_WIndows98_FINAL_Retail_Full_Setup-PWA/PWA.NFO b _ o a mozilla@ ftp 0 *
Tue May 19 17:36:27 1998 1933 ing079.ing.hb.se 5130019 /open/_ _/calibreX/Win98.Final-PWA/
pwa98cbe.zip b _ o a F_rnamn.Efternamn@Linje.ing.hb.se ftp 0 *
Tue May 19 17:36:30 1998 2522 semicon.chungbuk.ac.kr 5130606 /open/_ _/calibreX/Win98.
Final-PWA/pwa98cbc.zip b _ o a semicon@semicon.chungbuk.ac.kr ftp 0 *
Tue May 19 17:36:32 1998 1945 ppp068.0.mmtl.videotron.net 3467331 /open/_ _/calibreX/
Win98.Final-PWA/pwa98cba.zip b _ o a leegend@hotmail.com ftp 0 
Tue May 19 17:36:37 1998 1 semicon.chungbuk.ac.kr 0 /open/_ _/calibreX/Win98.Final-PWA/
pwa98cbd.good.zip b _ o a semicon@semicon.chungbuk.ac.kr ftp 0 *
Tue May 19 17:36:41 1998 2072 dialup2-52.itv.se 5130477 /open/_ _/calibreX/Win98.Final-
PWA/pwa98cbk.zip b _ o a dont@bother.me ftp 0 *
Tue May 19 17:37:19 1998 1 wend10.swol.de 7045 /open/_ _/calibreX/Win98.Final-PWA/
Microsoft_WIndows98_FINAL_Retail_Full_Setup-PWA/PWA.NFO b _ o a king@hotmail.com ftp 0 *

Evidently, someone had uploaded a copy of the Windows 98 distribution disk and this disk was now being downloaded all over the world. Why the interest in Windows 98? Because this incident took place on May 19, 1998, and Windows 98 wasn't scheduled to ship until May 25, 1998. Somebody had leaked a copy of Windows 98. That copy had been uploaded to Vineyard.NET and stored in an unprotected directory named /open. Re-examining the MRTG graph in Figure 22-1, it was apparent that the upload had happened at 19:00 on the previous night.

The location of the Windows 98 archive must have been sent out early the next morning. This information was passed along to the Internet piracy community, and many individuals all over the world started downloading the archive. An analysis of the Vineyard.NET log files later revealed that the copy of Windows 98 was downloaded to 129 different sites on the Internet.

The massive outflow of data from the server was responsible, in part, for the system being so sluggish. The large number of simultaneous FTP sessions, and the fact that many of these sessions were going through a Perl-based, Web-to-FTP gateway, further dragged down the system.

Vineyard.NET's first response was to change the permissions on the /open directory to 000 and move it to another location. The FTP server was temporarily disabled. Finally, the Web-to-FTP gateway was temporarily disabled as well so that the system would not be slowed down by so many people repeatedly using that rather inefficient piece of software. Once these measures were taken, the system immediately returned to normal.

Next, Vineyard.NET's staff looked at the files that were downloaded to see if they were, in fact, a copy of Windows 98. An initial list of files looked somewhat suspicious:

# ls -l
total 73
drwxr-xr-x  3 ftp  wheel    512 May 18  1998   
-rw-r--r--  1 ftp  wheel   6762 Apr 29  1998 SLG.TXT
-rw-r--r--  1 ftp  wheel  65113 May 16  1998 lynx272ssleay.zip
drwxr-xr-x  3 ftp  wheel    512 May 19  1998 nothing_here

It turns out that the real action isn't in the directory called nothing_here, but in the first directory?the one with a name consisting of two spaces.

Look again at the directory list. This time, the -F option to ls is used:

# ls -lF | cat -v
total 73
drwxr-xr-x  3 ftp  wheel    512 May 18  1998   /
-rw-r--r--  1 ftp  wheel   6762 Apr 29  1998 SLG.TXT
-rw-r--r--  1 ftp  wheel  65113 May 16  1998 lynx272ssleay.zip
drwxr-xr-x  3 ftp  wheel    512 May 19  1998 nothing_here/

We can change into this directory and see what's inside it:

# cd '  '
# ls -l
total 1
drwxr-xr-x  3 ftp  wheel  512 May 19  1998 calibreX
# cd calibreX/
# ls -l
total 1
drwxr-xr-x  3 ftp  wheel  1024 May 19  1998 Win98.Final-PWA
# cd Win98.Final-PWA/
# ls -l
total 76505
drwxr-xr-x  2 ftp  wheel      512 May 19  1998 Microsoft_WIndows98_FINAL_Retail_Full_
-rw-r--r--  1 ftp  wheel     7045 May 19  1998 PWA.NFO
-rw-r--r--  1 ftp  wheel      832 May 19  1998 file_id.diz
-rw-r--r--  1 ftp  wheel  5133067 May 19  1998 pwa98cba.zip
-rw-r--r--  1 ftp  wheel  5134860 May 19  1998 pwa98cbb.zip
-rw-r--r--  1 ftp  wheel  5130606 May 19  1998 pwa98cbc.zip
-rw-r--r--  1 ftp  wheel        0 May 19  1998 pwa98cbd.good.zip
-rw-r--r--  1 ftp  wheel  5130019 May 19  1998 pwa98cbd.zip
-rw-r--r--  1 ftp  wheel  5130019 May 19  1998 pwa98cbe.zip
-rw-r--r--  1 ftp  wheel  5130019 May 19  1998 pwa98cbf.zip
-rw-r--r--  1 ftp  wheel  5130019 May 19  1998 pwa98cbg.zip
-rw-r--r--  1 ftp  wheel  5130019 May 19  1998 pwa98cbh.zip
-rw-r--r--  1 ftp  wheel  5130019 May 19  1998 pwa98cbi.zip
-rw-r--r--  1 ftp  wheel  5133267 May 19  1998 pwa98cbj.zip
-rw-r--r--  1 ftp  wheel  5130477 May 19  1998 pwa98cbk.zip
-rw-r--r--  1 ftp  wheel  5130019 May 19  1998 pwa98cbl.zip
-rw-r--r--  1 ftp  wheel  5130477 May 19  1998 pwa98cbm.zip
-rw-r--r--  1 ftp  wheel  5130477 May 19  1998 pwa98cbn.zip
-rw-r--r--  1 ftp  wheel  5130019 May 19  1998 pwa98cbo.zip
-rw-r--r--  1 ftp  wheel  1154560 May 19  1998 pwa98cbp.zip

How thoughtful! A full copy of Windows 98, with the full retail setup. This is described in the file PWA.NFO:

# cat PWA.NFO 

              ÜûÜ                          Üß 
             ûûÜßßÜ              Ü      Üß ° 
             ûûûûûûÜßÜ          Üß   Üß   ° ÜÜ
            ûûûûûûû       Üß ±      °  ûûÜÜ        .
           Üûûûûûûûûûû     Üß  °     û °  û ûûûûûßßÜÜ   
            ßßûûûûûûûûû  Üß   °       °   ûûß  Ü  ßÜß
               ßûûûûûûûû   û  °       û°    ßÜûû   ßÜ
               ûûûûûû   ° ß  Üß  ° ß    Ü ßûß     ûÜ
              ûûûûûû    û°  ß Üß ß ÜßÜß     °Ü   ÜÜ  ßÜ
 ÄÄÄÄÄÄÄÄÄÄ Üß ûûûûûÄÄÄ°   Üß   Üûßß ÄÄÄÄÄ ±° ß   Ä ûû ÄÄÄ-ÄÄÄ-Äú ú
            ßÜûûûûûß    ûÜß     Üß    ßÜ    ±  û      ±
            Üûûûûß     ß                  ±°      Üß ° 
          Üûûßß ÜÜ    Üß      .            ±°   ßÜ    ßÜ °
         ß Ü    ßßÜÜÜ                    ±°     Üß     ßÜ 
         ±  ±        ßßßÜ               ±°°  Üß         ß
         Ü   û        °Üß                ß  °ß
          ßß  ±°    °Üß                    ßß     ..R.Noble <MiRAGE>
                û °°°Üß
                û   ß     ... Pirates With Attitudes
              Üß           Proudly Presents ...
º [ Windows '98 Final - CAB disks ]                            May 17, 1998 º
º Supplier .....: PWA Gods              Type .....: Operating System       º
º Cracker ......: N/A                   Video ....:                        º
º Packager .....: Murmillius            Audio ....:                        º
º Protection ...: Serial Number         # Disks ..: 21 x 5meg              º

        Here it is: Windows '98 Final release - Retail Full Install!

        While every other group will be bringing you so many good programs
        for this operating system, it's PWA that brings you the OS itself. 
        It is fortunately for the user community that this is the case or 
        you would probably have ended up with a ripped down release
        from some other lame group missing important system files like 
        KRNL386.exe, because disklimits are more important nowadays to these
        people than a working release.

        People that still believe in quality have only one option :
        Pirates with Attitudes.

        Greets fly out to all Hardworking people in PWA.

        Note to other groups:

        This *IS* final.  When you get your CD's, it's very likely the file
        dates will differ as the dates get time stamped on the CD at pressing 
        time, however if you CRC check the CD with this release, you'll see the 
        files are all identical (make sure you use the FULL RETAIL CD to check, 
        not an upgrade or OEM version).  We're just going to wait to release 
        tnis to see if MS decides to make any last minute changes because 
        of the stuff going on with the US Dept. of Justice.

        Retail FULL Install Key: K4HVD-Q9TJ8-6CRX9-C9G28

        Install Notes: 

        The way this has been zipped up is as follows:

        1 ZIP file labeled RETAIL FULL SETUP
        21 ZIP files labeled Win98 CABs
        Several ZIP files labelled online service clients (only necessary for AOL,
        MSN, Compuserve, etc).

        You need to download the CABS and the RETAIL SETUP and unzip/unrar
        everything into one directory.  The reason for this is that as soon
        as I get install keys, I can release RETAIL UPGRADE, OEM FULL and
        OEM UPGRADE versions and they will only take 4 meg each (the CAB zips
        are generic thruout all these versions, I can just package up the 
        differences in seperate zips to save everyone space and time).  You just
        unzip whichever one you want into the same directory as the generic
        CAB zips.

                                    -/ This is PWA \-
                                     < 16 May 1998 >

 Council ......... Code3, Dark Lord, Dream Weaver, Murmillius, Rambone,
 Senior Members .. Akasha, Gumby, Mercy, Oyl Patch, Stoned, The Technic 
 Members ......... Acidman, Aironz, Angelface, BadBrad, Bunter, Chowdery,
                   Codebreaker, Corv8, DaPhantm, Disc Killer, Disk Killer, 
                   DJpaul, El Cid, EzD, Frost, Guip, Ico, IceB, Ivan, 
                   The Judge, Kewe, Lost Soul, Magellan, Marduk, Muadib, Nagumo,   
                   Ofd, Patriarch, Prozac, Ryu, Shadowman, SilverB, Skylark,  
                   neTyx, Single Minded, SpyNut, Sugar, The Jerk, Vampyre,  
                   Virtual Power, Voltan, Warlock
             If you are going to do it ... Do it with an ATTITUDE!

                PWA..... a juggernaut that rolls on thru 1998

  *  Please note that PWA is NOT accepting pay sites of any nature.. We're *
  *  in this for fun and entertainment, not to try to make ourselves rich. *

  *  PWA also does not accept new BBS', FTP sites, net couriers, graphics  *
  *  artists or programmers (including PPE's... PCB, may it rest in peace) *

 Support the software companies! If you enjoy using a program or using a  
 Util, consider buying it! Someone has to make it worth the programmer's  
 effort to keep up the high standards.. They made it, so they DESERVE it! 
# The follow-up

Following the incident, Vineyard.NET decided to report this incident to the proper authorities. As this was an act of copyright infringement involving a Microsoft product, we decided to call Microsoft's anti-piracy help line. The conversation went something like this:

Microsoft: "Microsoft Anti-Piracy Line. Can I help you?"

Vineyard.NET: "Yes, I have a pirate copy of Windows 98."

Microsoft: "Well, I'd like you to look at the box and let me know if you see the security hologram. Do you know what a hologram looks like?"

Vineyard.NET: "No, I don't think you understand. I have a copy of Windows 98. The program that you're launching in a hundred-million-dollar extravaganza in a week's time."

Microsoft: "Yes, I understand that. You have a copy of Windows 98. Is it on disk or floppy?"

Vineyard.NET: "It's on my computer. Somebody uploaded it to my computer over the Internet."

Microsoft: "You mean, you don't really have a copy?"

This seemed useless, so we hung up on Microsoft and called the Boston office of the FBI. Nobody was in the office who could handle the case, so we left a message and asked for them to call us back.

Finally, we examined our log files to determine the location from which the files had been uploaded. It turned out to be a machine at Pace University named knox.pace.edu. A phone call to the network administrator at Pace revealed that this was the school's firewall. The administrator said that his firewall logs would reveal who it was.

The next day, the administrator called back. He said that they had determined the student whose network connection was used to upload the files. That student would receive immediate disciplinary action.

We thought that this was a bit harsh. After all, it's possible that the student's computer was being used by another student. But the network administrator seemed positive that he knew what was going on, and wasn't interested in any "alternative theories" about what might have happened. He knew that he had the guilty party.

As it turns out, he had only the tip of the iceberg.

On May 4, 2000, the U.S. Justice Department announced an indictment against 12 individuals who were allegedly part of the group that called itself Pirates With Attitude and 5 individuals at Intel who had supplied the group with software in exchange for access to PWA's archive of more than 5,000 programs. International in scope, PWA had archives located on compromised systems throughout the United States, Canada, and Europe. Scott R. Lassar, United States Attorney for the Northern District of Illinois, called PWA "one of the oldest and most sophisticated networks of software pirates anywhere in the world."

Undoubtedly, it was this Intel connection that was responsible for the copy of Windows 98 discovered at Vineyard.NET. This explains the use of internal terminology in the PWA.NFO file and the high quality of these warez. One wonders what their attitude will be as convicted federal felons?

22.4.3 faxsurvey

On October 7, 1998, an employee at Vineyard.NET noticed that the user http was logged into the company's primary web server, as shown in Example 22-6.

Example 22-6. An intruder is discovered
Script started on Wed Oct  7 20:54:21 1998 
bash-2.02# w
 8:57PM  up 27 days, 14:19, 5 users, load averages: 0.28, 0.33, 0.35
USER     TTY FROM              LOGIN@  IDLE WHAT
http     p0  KRLDB110-06.spli Tue02AM 1days /bin/sh
simsong  p1  asy12.vineyard.n  8:42PM    15 -tcsh (tcsh)
ericx    p2  mac-ewb.vineyard  8:46PM     0 script
ericx    p3  mac-ewb.vineyard  8:46PM    11 top
ericx    p4  mac-ewb.vineyard  8:53PM     1 sleep 5

This computer was running the BSDI v3.1 operating system with all patches as released by the vendor. The web server was a version of the Apache web server named Stronghold. Broadly defined, the computer was a "federal interest computer" under the law because the computer was used to initiate Automated Clearing House electronic funds transfers from customer accounts. To assist in these funds transfers, the computer held credit card and bank account information. (Fortunately, that information on the computer was stored in an encrypted format.)

In all likelihood, a user logged in as http could be the result of two things. First, it could be a member of the ISP's staff who is using the http account for debugging. Alternatively, it could be an attacker who had found some way to break into the http account but had been unable to gain additional access. Because the user http was logged in from a computer whose name began with KRLDB110-06.spli, it appeared to the staff that this was a case of unauthorized access.

When the intrusion was discovered, one of the staff members immediately started the Unix program script to record his actions.

As evidenced in Example 22-6, the intruder appeared to be idle for more than a day. The original intrusion had taken place on Tuesday at 2:00 a.m.

The next step was to use the Unix ps command to list all of the processes currently running on the computer. Two processes were out of place: two copies of the /bin/sh shell that were being run by http. Both of those shells had been started on the previous day, one at 2:00 a.m., the other at 4:00 a.m., as shown by the last two lines of Example 22-7.

Example 22-7. Processes that were running on the compromised computer
bash-2.02# ps auxww
root     11766  3.0  0.0     0    0  ??  Z    23Sep98    0:00.00 (admin_server)
root      3763  1.0  0.0     0    0  ??  Z     2:03PM    0:00.00 (junkbuster)
mail     18120  1.3  0.3   816  724  ??  S     8:56PM    0:00.64 smap
root     17573  1.0  0.0     0    0  ??  Z    11:03AM    0:00.00 (admin_server)
root        16  0.0  0.0    68   64  ??  Is   10Sep98    0:00.00 asyncd 2
root        18  0.0  0.0    68   64  ??  Is   10Sep98    0:00.02 asyncd 2
root        28  0.0  8.0   748 20680  ??  Ss   10Sep98   0:16.32 mfs -o rw -s 40960 /dev/
 sd0b /tmp (mount_mfs)
root        53  0.0  0.1   268  296  ??  Ss   10Sep98    0:38.23 gettyd -s
root     18670  0.0  0.5   560 1276  ??  S    Tue02AM    0:04.77 (xterm) 
http     18671  0.0  0.1   244  276  p0  Is   Tue02AM    0:02.23 /bin/sh
http     26225  0.0  0.1   236  276  p0  I+   Tue04AM    0:00.07 /bin/sh

Apparently, the intruder had broken in and then, for some reason, had given up. As there appeared to be no immediate urgency, the ISP carefully formulated a plan of action:

  1. Do not alert the intruder about what is happening.

  2. Determine the intruder's source IP address.

  3. Use the Unix kill command to stop the intruder's processes. This signal would prevent the processes from running while leaving a copy in memory.

  4. Make a copy of the intruder's processes using the Unix gcore command.

  5. Place a rule on the ISP router to block packets from the intruder's ISP.

  6. Kill the intruder's processes unequivocally with kill -9.

  7. Determine how the intruder had broken in and fix the hole.

  8. Alert law enforcement.

To trace the intruder, the ISP tried using the Unix netstat command. This turned up a new piece of information. The intruder had not broken in with Telnet or SSH; instead, there was an X11 connection from the web server (Apache.Vineyard.NET) to an X server running on the intruder's computer! This is made clear by the bold line in Example 22-8.

Example 22-8. Active TCP connections to Vineyard.NET during the attack
bash-2.02# netstat -a
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp        0      0  VINEYARD.NET.http      nhv-ct4-09.ix.ne.1137  SYN_RCVD
tcp        0      0  VINEYARD.NET.http      nhv-ct4-09.ix.ne.1136  SYN_RCVD
tcp        0      0  VINEYARD.NET.http      nhv-ct4-09.ix.ne.1135  SYN_RCVD
tcp        0      0  VINEYARD.NET.http      DSY27.VINEYARD.N.1079  SYN_RCVD
tcp        0   2456  VINEYARD.NET.http      nhv-ct4-09.ix.ne.1134  ESTABLISHED
tcp        0   2268  VINEYARD.NET.http      DSY27.VINEYARD.N.1078  ESTABLISHED
tcp        0   2522  VINEYARD.NET.http    ESTABLISHED
tcp        0   8192  VINEYARD.NET.http      host-209-214-118.1785  ESTABLISHED
tcp        0   4916  VINEYARD.NET.http      host-209-214-118.1784  ESTABLISHED
tcp        0      0  VINEYARD.NET.http      host-209-214-118.1783  ESTABLISHED
tcp        0      0  VINEYARD.NET.http      ASY14.VINEYARD.N.1163  FIN_WAIT_2
tcp        0      0  VINEYARD.NET.smtp    ESTABLISHED
tcp        0   3157  VINEYARD.NET.pop       ASY5.VINEYARD.NE.1027  ESTABLISHED
tcp        0      0  APACHE.VINEYARD..ssh   MAC-EWB.VINEYARD.2050  ESTABLISHED
tcp        0      0  VINEYARD.NET.http      host-209-214-118.1782  FIN_WAIT_2
tcp        0      0  VINEYARD.NET.http      host-209-214-118.1781  FIN_WAIT_2
tcp        0      0  VINEYARD.NET.http      host-209-214-118.1775  FIN_WAIT_2
tcp        0      0  VINEYARD.NET.http      56k-2234.hey.net.1099  FIN_WAIT_2
tcp        0      0  VINEYARD.NET.https     ESY8.VINEYARD.NE.1557  FIN_WAIT_2
tcp        0      0  APACHE.VINEYARD..smtp  m28.boston.juno..54519 ESTABLISHED
tcp        0      0  APACHE.VINEYARD..ssh   MAC-EWB.VINEYARD.nfs   ESTABLISHED
tcp        0    328  APACHE.VINEYARD..ssh   MAC-EWB.VINEYARD.2048  ESTABLISHED
tcp        0      0  VINEYARD.NET.http      ASY14.VINEYARD.N.1162  FIN_WAIT_2
tcp        0      0  VINEYARD.NET.http      ASY14.VINEYARD.N.1160  FIN_WAIT_2
tcp        0      0  NEXT.VINEYARD.NE.ssh   ASY12.VINEYARD.N.1047  ESTABLISHED
tcp        0   7300  VINEYARD.NET.pop       DSY27.VINEYARD.N.1061  ESTABLISHED
tcp        0      0  NEXT.VINEYARD.NE.imap2 ASY12.VINEYARD.N.1041  ESTABLISHED
tcp        0      0  VINEYARD.NET.3290      VINEYARD.NET.imap2     CLOSE_WAIT
tcp        0      0  VINEYARD.NET.ssh       simsong.ne.media.1017  ESTABLISHED
tcp        0      0  APACHE.VINEYARD..3098  KRLDB110-06.spli.X11   ESTABLISHED
tcp     8760      0  VINEYARD.NET.1022      BACKUP.VINEYARD..ssh   ESTABLISHED
tcp        0      0  LOCALHOST.VINEYA.4778  *.*                    LISTEN
tcp        0      0  LOCALHOST.VINEYA.domai *.*                    LISTEN
tcp        0      0  NET10.VINEYARD.N.domai *.*                    LISTEN
tcp        0      0  SMTP4.VINEYARD.N.domai *.*                    LISTEN

The ISP concluded that the attacker had used a vulnerability in a CGI script to spawn an xterm back to his remote machine. To test this hypothesis, the ISP did a quick search through its web server logs, shown in Example 22-9.

Example 22-9. Searching through the web server's logs
% grep -i krldb110-06 /vni/apache/log/access_log:
1krldb110-06.splitrock.net - - [06/Oct/1998:02:53:48 -0400] "GET /cgi-bin/
phf?Qname=me%0als%20-lFa HTTP/1.0" 404 - "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows
98)" "/htdocs/biz/captiva"
2krldb110-06.splitrock.net - - [06/Oct/1998:02:53:50 -0400] "GET /cgi-bin/faxsurvey?ls%20-
lFa HTTP/1.0" 200 5469 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)" "/htdocs/biz/
3krldb110-06.splitrock.net - - [06/Oct/1998:02:53:52 -0400] "GET /cgi-bin/view-source?../.
./../../../../../../etc/passwd HTTP/1.0" 404 - "-" "Mozilla/4.0 (compatible; MSIE 4.01; 
Windows 98)" "/htdocs/biz/captiva"
4krldb110-06.splitrock.net - - [06/Oct/1998:02:53:53 -0400] "GET /cgi-bin/htmlscript?../..
/../../../../../../etc/passwd HTTP/1.0" 404 - "-" "Mozilla/4.0 (compatible; MSIE 4.01; 
Windows 98)" "/htdocs/biz/captiva"
5krldb110-06.splitrock.net - - [06/Oct/1998:02:53:54 -0400] "GET /cgi-bin/campas?%0als%20-
lFa HTTP/1.0" 404 - "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)" "/htdocs/biz/
6krldb110-06.splitrock.net - - [06/Oct/1998:02:53:55 -0400] "GET /cgi-bin/handler/useless_
shit;ls%20-lFa|?data=Download HTTP/1.0" 404 - "-" "Mozilla/4.0 (compatible; MSIE 4.01; 
Windows 98)" "/htdocs/biz/captiva"
7krldb110-06.splitrock.net - - [06/Oct/1998:02:53:56 -0400] "GET /cgi-bin/php.cgi?/etc/
passwd HTTP/1.0" 404 - "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)" "/htdocs/biz/
8krldb110-06.splitrock.net - - [06/Oct/1998:02:54:30 -0400] "GET /cgi-bin/faxsurvey?ls%20-
lFa HTTP/1.1" 200 5516 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)" "/htdocs/biz/
9krldb110-06.splitrock.net - - [06/Oct/1998:02:54:44 -0400] "GET /cgi-bin/
faxsurvey?uname%20-a HTTP/1.1" 200 461 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 
98)" "/htdocs/biz/captiva"
10krldb110-06.splitrock.net - - [06/Oct/1998:02:55:03 -0400] "GET /cgi-bin/faxsurvey?id 
HTTP/1.1" 200 381 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)" "/htdocs/biz/
11krldb110-06.splitrock.net - - [06/Oct/1998:02:55:39 -0400] "GET /cgi-bin/
faxsurvey?cat%20/etc/passwd HTTP/1.1" 200 79467 "-" "Mozilla/4.0 (compatible; MSIE 4.01; 
Windows 98)" "/htdocs/biz/captiva"
12krldb110-06.splitrock.net - - [06/Oct/1998:02:55:44 -0400] "GET /cgi-bin/
faxsurvey?ls%20-lFa%20/usr/ HTTP/1.1" 200 1701 "-" "Mozilla/4.0 (compatible; MSIE 4.01; 
Windows 98)" "/htdocs/biz/captiva"
13krldb110-06.splitrock.net - - [06/Oct/1998:04:31:55 -0400] "GET /cgi-bin/faxsurvey?id 
HTTP/1.1" 200 381 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)" "/htdocs/web.
14krldb110-06.splitrock.net - - [06/Oct/1998:04:32:01 -0400] "GET /cgi-bin/faxsurvey?pwd 
HTTP/1.1" 200 305 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)" "/htdocs/web.
15krldb110-06.splitrock.net - - [06/Oct/1998:04:32:08 -0400] "GET /cgi-bin/faxsurvey?/bin/
pwd HTTP/1.1" 200 305 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)" "/htdocs/web.
16krldb110-06.splitrock.net - - [06/Oct/1998:04:32:33 -0400] "GET /cgi-bin/
faxsurvey?ls%20-lFa HTTP/1.1" 200 5516 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 
98)" "/htdocs/web.vineyard.net"
17krldb110-06.splitrock.net - - [06/Oct/1998:04:32:55 -0400] "GET /cgi-bin/
faxsurvey?ls%20-lFa%20../conf/ HTTP/1.1" 200 305 "-" "Mozilla/4.0 (compatible; MSIE 4.01;
Windows 98)" "/htdocs/web.vineyard.net"

Notice that the first seven lines each occur within a few seconds of each other. It appears that the attacker is using an automated tool that checks for CGI vulnerabilities. In the next 10 lines the attacker exploits a vulnerability in the faxsurvey script. This was almost certainly done with a different tool; one indication is that the version of the HTTP protocol that the client supports changes from HTTP/1.0 to HTTP/1.1.

The web server log file revealed that the full hostname of the attacker was krldb110-06.splitrock.net. Using the Unix host command, this address could be translated into an actual IP address:

% host krldb110-06.splitrock.net
krldb110-06.splitrock.net has address

By inspecting the log file, it appears that the script /cgi-bin/faxsurvey has a bug that allows the attacker to execute arbitrary commands. (Otherwise, why else would the attacker keep sending URLs calling the same script with different arguments?) If this is true, then the following commands must have been executed by the attacker:

ls -lFa
ls -lFa
uname -a
cat /etc/passwd
ls -lFa /usr/
ls -lFa
ls -lFa ../conf/

It is not clear from the log files how the attacker was able to go from executing these commands to executing the xterm command. But is very clear that the xterm command was executed, as evidenced by the http entry in the output of the w command, the running (xterm) process, and the X11 entry in the netstat command.

At this point, the ISP searched for the attacker's hostname in other log files. A suspicious result was found in the messages log file?apparently, the attacker had attempted to exploit a POP or qpopper error, as evidenced in Example 22-10.

Example 22-10. The attacker apparently attempted to exploit a bug in the qpopper command
% grep -i krldb110-06 *
messages:Oct  6 03:38:29 apache popper.bsdos[22312]: @KRLDB110-06.splitrock.net: -ERR POP

To preserve the record of the attacker's processes, they were stopped, gcoreed, and sent a hard kill:

% kill -STOP 18671 26225
% gcore 18671 /tmp/attack1.core
% gcore 26225 /tmp/attack2.core
% kill -9 18671 26225

Following this, a rule was added to the ISP's routers to block access from the attacker's IP addresses. Permissions on the faxsurvey script were changed to 000, pending an investigation. A few days later, the script was removed from the web server.

The attacked ISP contacted SplitRock Services, Inc., the ISP that was responsible for the IP address. It was determined that SplitRock operated several modem pools that were provided to another ISP (Prodigy) on a leasing arrangement. SplitRock was asked to preserve its log files so that they could be used in a future legal investigation.

By using the Unix strings command over the files attack1.core and attack2.core it was possible to extract significantly more information about the attacker. One group of strings was from the shell history which was, effectively, a list of the commands that the attacker had typed. Example 22-11 shows the attacker's downloading of a "rootkit." The attacker appears to be trying to get a buffer overflow attack to work properly. Example 22-12 shows another group of strings, probably indicating commands typed by the attacker. In this sequence, the attacker appears to be attacking the computer's IMAP server (port 143) with a well-known buffer overflow exploit. If this exploit had worked, the attacker would have gained superuser access.

Example 22-11. A list of strings found in the attack1.core file which probably correspond to commands that were typed by the attacker
-lFa                                     gcc -o s s.c         
st2.c                                    ftp
cron.c                                   gcc -o s st2.c
cxterm.c                                 ./s concole
x2.c                                     t .s
qpush.c                                  .121
cat t.c                                  qpush.c
cat .c                                   ppp.c
cat s.c                                  t2.c
gc c                                     cron.c
ls -lFa                                  cxterm.c
./s -v c2                                tcsh
./s p0                                   x2.c
ls -lFa /                                README
cat .s                                   README.debian
ls -lFa                                  qpush
cat /w                                   qpush.c
ls -lFa /                                qpush.c.old
cat .s                                   gf: not found
_=.s                                     /tmp
$ : not found                            mfs:28
gcc -o s steal.c                         /bin/sh
ls -lFa *.c
Example 22-12. Another list of strings found in the attack2.core file indicating an IMAP attack
/bin/sh                         /bin/sh
/bin/sh                         inetd.conf
/etc/inetd.conf                 t) | telnet 127.1 143
qpush.c                         cd /etc
/usr/bin/gcc                    cat .s
n/gcc                           which pwd
./cc                            ls -lFa
expr                            expr $L + 1
done                            ls -lFa
./cc -10                        ./cc

The second kind of strings found in the core file corresponded to shell environment variables (Example 22-13). Many of these variables corresponded to variables that would be set by the Apache web server for a process spawned from a CGI script?confirming that the shell was, in fact, the result of a CGI attack. Note that the SCRIPT_FILENAME points to the vulnerability being with the faxsurvey script, and the QUERY_STRING was a request to run an xterm to a remote system. This block confirmed that the CGI script responsible for the intrusion was the faxsurvey script.

Example 22-13. A second block of strings found in the core file that likely corresponds to shell environment variables
HTTP_USER_AGENT=Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)
HTTP_ACCEPT=application/vnd.ms-excel, application/msword, application/vnd.ms-powerpoint, 
SERVER_SOFTWARE=Stronghold/2.2 Apache/1.2.5 C2NetUS/2002
TERMCAP=xterm|vi|xterm-ic|xterm-vi|xterm with insert character instead of insert mode:  :
al@:dl@:im=:ei=:mi@:ic=\E[@:   :AL=\E[%dL:DC=\E[%dP:DL=\E[
%dM:DO=\E[%dB:IC=\E[%d@:UP=\E[%dA:      :al=\E[L:am:    :bs:cd=\E[J:ce=\E[K:cl=\E[H\E[2J:
cm=\E[%i%d;%dH:co#80:  :cs=\E[%i%d;%dr:ct=\E[3k:       :dc
=\E[P:dl=\E[M:  :im=\E[4h:ei=\E[4l:mi:  :ho=\E[H:       :is=\E[m\E[?7h\E[?1;3;4l\E[4l:  :
rs=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;3;4;6l\E[4l\E<:        :kb
=^H:kd=\EOB:ke=\E[?1l\E>:    :k1=\E[11~:k2=\E[12~:k3=\E[13~:k4=\E[14~:k5=\E[15~:     :
k6=\E[16~:k7=\E[17~:k8=\E[18~: :kl=\EOD:km:kn#8:kr=\EOC:ks
=\E[?1h\E=:ku=\EOA:      :li#24:md=\E[1m:me=\E[m:mr=\E[7m:ms:nd=\E[C:pt: :sc=\E7:rc=\E8:
sf=\n:so=\E[7m:se=\E[m:sr=\EM:   :te=\E[2J\E[?47l\E8:ti=\E7\
E[?47h: :up=\E[A:us=\E[4m:ue=\E[m:xn:

The victim ISP could have used an X tool to attack the attacker. Specifically, the attacker's screen could have been eavesdropped on, and X keyboard events could have been fed to remote xterms. Although such a turnabout attack might be considered fair play, it would likely have been illegal under existing computer crime statutes.

After the intrusion, the victim ISP contacted the Boston office of the Federal Bureau of Investigation. The ISP was informed that the Boston office had a damage threshold of $8,000 that needed to be exceeded before an investigation could be opened. As this threshold had not been met, no investigation could take place. While such minimums are understandable, they are unfortunate for two reasons:

  • Many attacks are conducted by relatively young offenders, who might cease such activity if they received a warning or, at most, a suspended sentence. The lack of any official investigation and follow-up only encourages these attackers to engage in larger and larger crimes until they are responsible for serious damage.

  • In this case, the attacker appeared to be quite sophisticated. It's quite possible that the attacker was engaged in other illegal activities that usually go by without anyone noticing. There are many cases in which the investigation of relatively small crimes have led law enforcement agencies to significant criminal enterprises. For example, it was a $.75 accounting discrepancy that caused Cliff Stoll to track down a computer hacker who was ultimately found to be breaking into U.S. commercial and military computers at the behest of the Soviet Union (a story detailed in Stoll's classic hacker thriller, The Cuckoo's Egg).

As it turns out, the vulnerability in the faxsurvey script had been reported over the BugTraq mailing list nearly three months prior to the attack (see Example 22-14). Either nobody from the ISP had been reading the BugTraq mailing list, or else no one was aware that the faxsurvey script had been installed.

Example 22-14. The BugTraq report for the faxsurvey script
Date:         Tue, 4 Aug 1998 07:41:24 -0700
Reply-To:     dod@muenster.net
From:         Tom <dod@MUENSTER.NET>
Subject:      remote exploit in faxsurvey cgi-script


There exist a bug in the 'faxsurvey' CGI-Script, which allows an attacker
to execute any command s/he wants with the permissions of the HTTP-Server.

All the attacker has to do is type
in his favorite Web-Browser to get a copy of your Password-File.

All S.u.S.E. 5.1 and 5.2 Linux Dist. (and I think also older ones) with the
HylaFAX package installed are vulnerable to this attack.

AFAIK the problem exists in the call of 'eval'.

I notified the S.u.S.E. team (suse.de) about that problem. Burchard
Steinbild <bs@suse.de> told me, that they have not enough time to fix that
bug for their 5.3 Dist., so they decided to just remove the script from the
file list.

After the break-in, the ISP performed the following cleanup:

  • An immediate backup of all disks was done. This backup was preserved as evidence in the event that damage was discovered that needed to be addressed.

  • The system was scanned for new SUID and SGID files. None were found.

  • Permissions on the /usr/include directory and the C compiler were changed so that only staff members could access these files and compile new programs.

  • Key programs, including /bin/ls, /bin/ps, and /bin/login, were compared with the distribution CD-ROM to determine if any had been modified. They had not been.

  • All log files were manually examined for additional suspicious activity. None was found.

  • After a week, the router rule blocking access to SplitRock was removed.

The Coroner's Toolkit did not exist at the time of Vineyard.NET's faxsurvey attack. If it had, an additional step that could have been taken would have been the creation of a "mactimes" timeline and a review of every file that had been created, modified, or accessed during the attack.

    Part VI: Appendixes