In addition to ps and kill, Unix supports a large number of lesser known tools for examining and controlling running processes. These commands can be useful for programmers and system administrators; they are also very helpful in analyzing the processes of an attacker during and after a break-in. Some of the ways you can examine or control processes include the following:
You can attach to the running process with a debugger such as gdb.
You can use the gcore command to dump the process memory map.
You can use the lsof program to list the open files in use by the program.
You can examine the process directly using the /proc process filesystem.
You can see a tree of all processes with the pstree command.
Not all of these tools are available on every version of Unix.
Strictly speaking, many of these tools will work with processes that are either running or stopped. However, if you have a rogue process on your system, you may wish to stop it with the SIGSTOP signal before examining it.
On many systems, you can use the gdb command to "attach" to a running process. If the process is running an executable that was linked with a full symbol table, you will be able to use the debugger to examine the process's variables and detailed call stack. Even if you do not have an executable that was linked for debugging, you may be able to use the debugger to determine what the process is doing.
adb and dbx are additional debuggers you can use to control processes.
Under many versions of Unix, you can use the gcore program to generate a core file of a running process. A core file is a specially formatted image of the memory being used by the process at the time the signal was caught.
Once you have a core file, you can examine it with adb (a debugger), dbx (another debugger), or gdb (yet another debugger). By examining the core file with a debugger, you can see which routines were executed, register values, and more. If you simply want to get an idea of what the process was doing, you can run strings (a program that finds printable strings in a binary file) over the core image to see which files it was referencing. If the process was running a shell such as sh or csh, the strings command will display the shell's history.
Programs that you run may also dump core if they receive one of the signals that causes a core dump. On systems without a gcore program, you can send a SIGEMT or SIGSYS signal to cause the program to dump core. This method will work only if the process is currently in a directory where it can write, it has not redefined the action to receive the signal, and the core will not be larger than the core file limits imposed for the process's UID. If you use this approach, you will also be faced with the problem of finding where the process left the core file!
The List of Open Files (lsof) command is now provided as a standard part of many Unix systems; it is available as a free download for still more systems.
 lsof Version 4.64 is available from ftp://vic.cc.purdue.edu/pub/tools/unix/lsof/ for at least the following Unix systems: AIX 4.3., 5L, and 5.1; Apple Darwin 1. and 1.4 for Power Macintosh systems; BSDI BSD/OS 4.1 for Intel-based systems; DEC OSF/1, Digital UNIX, Tru64 UNIX 4.0, and 5.; FreeBSD 4. and 5.0 for Intel-based systems; HP-UX 11.00 and 11.11; Linux 2.1.72 and above for Intel-based systems; NetBSD 1. for Alpha-, Intel-, and SPARC-based systems; NEXTSTEP 3. for NEXTSTEP architectures; OpenBSD 2. and 3. for Intel-based systems; OPENSTEP 4.x; Caldera OpenUNIX 8; SCO OpenServer Release 5.0. for Intel-based systems; SCO UnixWare 7.1.1 for Intel-based systems; Solaris 2.6, 7, 8, and 9 BETA-Refresh.
As the name implies, this command examines the kernel's table of file descriptors associated with each process and displays the name of each file that is currently opened. In addition to giving the name of each file being currently referenced, lsof reveals the name of the executable currently being run by the process and the filenames of all mapped-in shared libraries. Besides this information, current versions of lsof can report open TCP/IP connections, as well as TCP and UDP sockets that are being listened to.
When lsof is run by a user, the program restricts its output to processes that are owned by that user. When lsof is run by the superuser, the program displays output for all processes on the system.
Here is an example of the output from the lsof command:
[simsong@r2 ~] 304 % lsof COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME tcsh 81776 simsong cwd VDIR 13,2 15360 12657945 /usr/home/simsong tcsh 81776 simsong rtd VDIR 116,131072 1024 2 / tcsh 81776 simsong txt VREG 116,131072 638988 6355 /bin/tcsh tcsh 81776 simsong 15u VCHR 5,3 0t138415 7898 /dev/ttyp3 tcsh 81776 simsong 16u VCHR 5,3 0t138415 7898 /dev/ttyp3 tcsh 81776 simsong 17u VCHR 5,3 0t138415 7898 /dev/ttyp3 tcsh 81776 simsong 18u VCHR 5,3 0t138415 7898 /dev/ttyp3 tcsh 81776 simsong 19u VCHR 5,3 0t138415 7898 /dev/ttyp3 lsof 81991 simsong cwd VDIR 13,2 15360 12657945 /usr/home/simsong lsof 81991 simsong rtd VDIR 116,131072 1024 2 / lsof 81991 simsong txt VREG 13,2 106848 7618686 /usr/local/sbin/lsof lsof 81991 simsong txt VREG 13,2 76752 1984040 /usr/libexec/ld-elf. so.1 lsof 81991 simsong txt VREG 13,2 19232 634990 /usr/lib/libkvm.so.2 lsof 81991 simsong txt VREG 13,2 573888 634976 /usr/lib/libc.so.4 lsof 81991 simsong 0u VCHR 5,3 0t138415 7898 /dev/ttyp3 lsof 81991 simsong 1u VCHR 5,3 0t138415 7898 /dev/ttyp3 lsof 81991 simsong 2u VCHR 5,3 0t138415 7898 /dev/ttyp3 lsof 81991 simsong 3r VCHR 2,0 0t0 6880 /dev/mem lsof 81991 simsong 4r VCHR 2,1 0xc28649c0 6872 /dev/kmem [simsong@r2 ~] 305 %
The lsof program has too many options to list here. There are significant security issues that arise from its installation and use?specifically, lsof lists the names of files throughout the filesystem, and this information is cached in a file that is located in the home directory of the person who runs the lsof command. The lsof command can be compiled and installed with various options that minimize the privacy exposure that can result from these cache files. For details, consult the lsof documentation.
/proc is the process filesystem. It allows user programs to access aspects of a process through the filesystem interface in a relatively transparent and straightforward fashion, without having to open up the kernel's memory and wade through memory structures. It also allows direct access to the memory space of other processes, which is otherwise impossible or very difficult.
Every Unix process has an associated parent process. Normally, this information is displayed as a form similar to the display of the PPID field output by the ps command. The pstree command uses this information to draw a graph of all of the processes currently running.
During a break-in, the process tree can be very useful for understanding which processes were launched by the attacker and which are innocent processes that happen to be running on the same system.
With the -u option, the pstree command will show UID transitions?that is, when one process has a child that is executing under a separate UID. Another useful option is -a, which shows the entire command line that was executed. For a list of all the options, see the documentation.
Here is an example of the output of the pstree program:
% pstree -u init-+-arpwatch |-cron |-dhcpd |-gdomap(nobody) |-8*[getty] |-httpd---11*[httpd(http)] |-inetd-+-imapd(simsong) | `-sslwrap |-lpd |-master-+-2*[bounce(postfix)] | |-cleanup(postfix) | |-flush(postfix) | |-local(postfix) | |-pickup(postfix) | |-qmgr(postfix) | |-2*[smtp(postfix)] | |-smtpd(postfix) | |-tlsmgr(postfix) | `-trivial-rewrite(postfix) |-mountd |-moused |-named |-nfsd---4*[nfsd] |-4*[nfsiod] |-nmbd---nmbd |-ntpd |-portmap(daemon) |-pwcheck |-rpc.statd |-rwhod(daemon) |-setiathome(nobody) |-slapd |-smbd---smbd(beth) |-snmpd |-sshd-+-sshd---tcsh(simsong)---pstree | |-sshd---tcsh(simsong)---tcsh(root) | `-sshd---tcsh(simsong) |-syslogd `-usbd
The boldfaced line near the end of this output shows that init (executing as root) spawned an sshd process (executing as root). This process forks a child (still root-owned in this case) for each incoming connection. When simsong logged into this sshd connection, it started up a shell (tcsh) owned by simsong, and simsong has apparently managed to start a root-privileged tcsh shell (perhaps with /bin/su, but if you don't expect simsong to have the root password, this is cause for concern)!