In Chapter 8, you examined the characteristics of processes and how to work with them. Most modern operating systems have a process model. However, you will now look at some special features of processes as implemented in Solaris. One of the most innovative characteristics of processes under Solaris is the process file system (PROCFS), which is mounted on /proc. The state of all normal threads and processes is stored on the PROCFS. Each entry in the top-level file system corresponds to a specific process ID, under which a number of subdirectories contain all state details. Applications and system services can communicate with the PROCFS as if it were a normal file system. Thus, state persistence can be provided using the same mechanism as normal file storage.
Images of all currently active processes are stored in the /proc file system by their PID.
The internals of the PROCFS can seem a little complicated, but luckily Solaris provides a number of tools to work with the /proc file system. Here’s an example of how process state is persisted on the PROCFS: first, a process is identified—in this example, the current Korn shell for the user pwatters:
# ps -eaf | grep pwatters pwatters 310 291 0 Mar 20 ? 0:04 /usr/openwin/bin/Xsun pwatters 11959 11934 0 09:21:42 pts/1 0:00 grep pwatters pwatters 11934 11932 1 09:20:50 pts/1 0:00 ksh
Now that you have a target PID (11934), you can change to the /proc/11934 directory and you will be able to view the image of this process:
# cd /proc/11934 # ls -l total 3497 -rw------- 1 pwatters other 1769472 Mar 30 09:20 as -r-------- 1 pwatters other 152 Mar 30 09:20 auxv -r-------- 1 pwatters other 32 Mar 30 09:20 cred --w------- 1 pwatters other 0 Mar 30 09:20 ctl lr-x------ 1 pwatters other 0 Mar 30 09:20 cwd ->> dr-x------ 2 pwatters other 1184 Mar 30 09:20 fd -r--r--r-- 1 pwatters other 120 Mar 30 09:20 lpsinfo -r-------- 1 pwatters other 912 Mar 30 09:20 lstatus -r--r--r-- 1 pwatters other 536 Mar 30 09:20 lusage dr-xr-xr-x 3 pwatters other 48 Mar 30 09:20 lwp -r-------- 1 pwatters other 2016 Mar 30 09:20 map dr-x------ 2 pwatters other 544 Mar 30 09:20 object -r-------- 1 pwatters other 2552 Mar 30 09:20 pagedata -r--r--r-- 1 pwatters other 336 Mar 30 09:20 psinfo -r-------- 1 pwatters other 2016 Mar 30 09:20 rmap lr-x------ 1 pwatters other 0 Mar 30 09:20 root ->> -r-------- 1 pwatters other 1440 Mar 30 09:20 sigact -r-------- 1 pwatters other 1232 Mar 30 09:20 status -r--r--r-- 1 pwatters other 256 Mar 30 09:20 usage -r-------- 1 pwatters other 0 Mar 30 09:20 watch -r-------- 1 pwatters other 3192 Mar 30 09:20 xmap
Each of the directories with the name associated with the PID contains additional subdirectories, which contain state information and related control functions. For example, the status file contains entries that contain a structure that defines state elements including the following:
Parent process ID
Process group ID
Process pending signal set
Process heap virtual address
Process stack size
User and system CPU time
Total child process user and system CPU time
The process flags contained in the structure define specific process state characteristics, including the following:
PR_ISSYS System process flag
PR_VFORKP vforked child parent flag
PR_FORK Inherit-on-fork flag
PR_RLC Run-on-last-close flag
PR_KLC Kill-on-last-close flag
PR_ASYNC Asynchronous-stop flag
PR_MSACCT Microstate accounting on flag
PR_MSFORK Post-fork microstate accounting inheritance flag
PR_BPTADJ Breakpoint on flag
PR_PTRACE ptrace-compatibility on flag
Read the materials on docs.sun.com for more information about the proc file system.
In addition, a watchpoint facility is provided, which is responsible for controlling memory access. A series of proc tools interpret the information contained in the /proc subdirectories, which display the characteristics of each process.
The proc tools are designed to operate on data contained within the /proc file system. Each utility takes a PID as its argument and performs operations associated with the PID. For example, the pflags command prints the flags and data model details for the PID in question. For the Korn shell example, we can easily print out this status information:
# /usr/proc/bin/pflags 29081 29081: /bin/ksh data model = _ILP32 flags = PR_ORPHAN /1: flags = PR_PCINVAL|PR_ASLEEP [ waitid(0x7,0x0,0x804714c,0x7) ]
We can also print the credential information for this process, including the effective and real UID and GID of the process owner, by using the pcred command:
# /usr/proc/bin/pcred 29081 29081: e/r/suid=100 e/r/sgid=10
Here, both the effective and the real UID is 100 (user pwatters), and the effective and real GID is 10 (group staff). To examine the address space map of the target process, and all of the libraries it requires to execute it, we can use the pmap command:
# /usr/proc/bin/pmap 29081 29081: /bin/ksh 08046000 8K read/write/exec [ stack ] 08048000 160K read/exec /usr/bin/ksh 08070000 8K read/write/exec /usr/bin/ksh 08072000 28K read/write/exec [ heap ] DFAB4000 16K read/exec /usr/lib/locale/en_AU/en_AU.so.2 DFAB8000 8K read/write/exec /usr/lib/locale/en_AU/en_AU.so.2 DFABB000 4K read/write/exec [ anon ] DFABD000 12K read/exec /usr/lib/libmp.so.2 DFAC0000 4K read/write/exec /usr/lib/libmp.so.2 DFAC4000 552K read/exec /usr/lib/libc.so.1 DFB4E000 24K read/write/exec /usr/lib/libc.so.1 DFB54000 8K read/write/exec [ anon ] DFB57000 444K read/exec /usr/lib/libnsl.so.1 DFBC6000 20K read/write/exec /usr/lib/libnsl.so.1 DFBCB000 32K read/write/exec [ anon ] DFBD4000 32K read/exec /usr/lib/libsocket.so.1 DFBDC000 8K read/write/exec /usr/lib/libsocket.so.1 DFBDF000 4K read/exec /usr/lib/libdl.so.1 DFBE1000 4K read/write/exec [ anon ] DFBE3000 100K read/exec /usr/lib/ld.so.1 DFBFC000 12K read/write/exec /usr/lib/ld.so.1 total 1488K
It’s always surprising to see how many libraries are loaded when an application is executed, especially something as complicated as a shell—in the example here, leading to a total of 1488KB memory used. You can obtain a list of the dynamic libraries linked to each process by using the pldd command:
# /usr/proc/bin/pldd 29081 29081: /bin/ksh /usr/lib/libsocket.so.1 /usr/lib/libnsl.so.1 /usr/lib/libc.so.1 /usr/lib/libdl.so.1 /usr/lib/libmp.so.2 /usr/lib/locale/en_AU/en_AU.so.2
Signals are the way in which processes communicate with each other, and can also be used from shells to communicate with spawned processes (usually to suspend or kill them). We examine signals in detail in Chapter 8. However, by using the psig command, it is possible to list the signals associated with each process:
# /usr/proc/bin/psig 29081 29081: /bin/ksh HUP caught RESTART INT caught RESTART QUIT ignored ILL caught RESTART TRAP caught RESTART ABRT caught RESTART EMT caught RESTART FPE caught RESTART KILL default BUS caught RESTART SEGV default SYS caught RESTART PIPE caught RESTART ALRM caught RESTART TERM ignored USR1 caught RESTART USR2 caught RESTART CLD default NOCLDSTOP PWR default WINCH default URG default POLL default STOP default TSTP ignored CONT default TTIN ignored TTOU ignored VTALRM default PROF default XCPU caught RESTART XFSZ ignored WAITING default LWP default FREEZE default THAW default CANCEL default LOST default RTMIN default RTMIN+1 default RTMIN+2 default RTMIN+3 default RTMAX-3 default RTMAX-2 default RTMAX-1 default RTMAX default
It is also possible to print a hexadecimal format stack trace for the LWP in each process by using the pstack command. This can be useful in the same way that the truss command was used:
# /usr/proc/bin/pstack 29081 29081: /bin/ksh dfaf5347 waitid (7, 0, 804714c, 7) dfb0d9db _waitpid (ffffffff, 8047224, 4) + 63 dfb40617 waitpid (ffffffff, 8047224, 4) + 1f 0805b792 job_wait (719d) + 1ae 08064be8 sh_exec (8077270, 14) + af0 0805e3a1 ???????? () 0805decd main (1, 8047624, 804762c) + 705 0804fa78 ???????? ()
Perhaps the most commonly used proc tool is the pfiles command, which displays all of the open files for each process. This is very useful for determining operational dependencies between data files and applications:
# /usr/proc/bin/pfiles 29081 29081: /bin/ksh Current rlimit: 64 file descriptors 0: S_IFCHR mode:0620 dev:102,0 ino:319009 uid:6049 gid:7 rdev:24,8 O_RDWR|O_LARGEFILE 1: S_IFCHR mode:0620 dev:102,0 ino:319009 uid:6049 gid:7 rdev:24,8 O_RDWR|O_LARGEFILE 2: S_IFCHR mode:0620 dev:102,0 ino:319009 uid:6049 gid:7 rdev:24,8 O_RDWR|O_LARGEFILE 63: S_IFREG mode:0600 dev:174,2 ino:990890 uid:6049 gid:1 size:3210 O_RDWR|O_APPEND|O_LARGEFILE FD_CLOEXEC
In addition, it is possible to obtain the current working directory of the target process by using the pwdx command:
# /usr/proc/bin/pwdx 29081 29081: /home/paul
If you need to examine the process tree for all parent and child processes containing the target PID, this can be achieved by using the ptree command. This is useful for determining dependencies between processes that are not apparent by consulting the process list:
# /usr/proc/bin/ptree 29081 247 /usr/dt/bin/dtlogin -daemon 28950 /usr/dt/bin/dtlogin -daemon 28972 /bin/ksh /usr/dt/bin/Xsession 29012 /usr/dt/bin/sdt_shell -c unset DT; DISPLAY=lion:0; 29015 ksh -c unset DT; DISPLAY=lion:0; /usr/dt/bin/dt 29026 /usr/dt/bin/dtsession 29032 dtwm 29079 /usr/dt/bin/dtterm 29081 /bin/ksh 29085 /usr/local/bin/bash 29230 /usr/proc/bin/ptree 29081
Here, ptree has been executed from the Bourne again shell (bash), which was started from the Korn shell (ksh), spawned from the dtterm terminal window, which was spawned from the dtwm window manager, and so on.
Although many of these proc tools will seem obscure, they are often very useful when trying to debug process-related application errors, especially in large applications like database management systems.