Even if recompiling your kernel, downloading a new one, or choosing a journaling file system seems too adventuresome at this point, you can some very simple things to your Beowulf node that can increase performance and manageability. Remember that just as the kernel, with its nearly five hundred dynamically loadable modules, provides drivers and capabilities you probably will never need, so too your Linux distribution probably looks more like a kitchen sink than a lean and mean computing machine. While you may now be tired of the Linux Beowulf adage "a smaller operating system is a better operating system," it must be once again applied to the auxiliary programs often run with a conventional Linux distribution. If we look at the issue from another perspective, every single CPU instruction performed by the kernel or operating system daemon not directly contributed to the scientific calculation is a wasted CPU instruction.
The starting point for pruning your Beowulf node will be what the Linux distribution installer set up. Many distributions have options during installation for "workstation" or "server" or "development" configurations. As a general rule of thumb, "server" installations make a good starting point. Workstation configurations often have windowing systems running by default, and a myriad of background tasks to make Linux as user-friendly as possible to the desktop user. Fortunately, with Linux you can understand and modify any daemon or process as you convert your kitchen sink of useful utilities and programs into a designed-for-computation roadster. For a Beowulf, eliminating useless tasks delivers more megaflop per dollar to the end user.
The first step to pruning the operating system daemons and auxiliary programs is to find out what is running on the system. For most Linux systems there are at least two standard ways to start daemons and other processes, which may waste CPU resources as well as memory bandwidth (often the most precious commodity on a cluster).
inetd: This is the "Internet superserver". Many Linux distributions use a newer version of the program, which has essentially the same functionality called xinetd. Both programs basic function is to wait for connections on a set of ports and then spawn and hand off the network connection to the appropriate program when an incoming connection is made. The configuration for what ports inetd or xinetd listening to, as well as what will get spawned can been determined by looking at '/etc/inetd.conf' and '/etc/services' or '/etc/xinetd. conf' and '/etc/xinetd.d' respectively.
/etc/rc.d/init.d: This special directory represents the scripts that are run during the booting sequence and that often launch daemons that will run until the machine is shut down.
The file 'inetd.conf' is a simple configuration file. Each line in the file represents a single service, including the port associated with that service and the program to launch when a connection to the port is made. Below are some simple examples:
ftp stream tcp nowait root /usr/sbin/tcpd in.proftpd finger stream tcp nowait root /usr/sbin/tcpd in.fingerd talk dgram udp wait root /usr/sbin/tcpd in.talkd
The first column provides the name of the service. The file '/etc/services' maps the port name to the port number, for example,
% grep ^talk /etc/services talk 517/udp # BSD talkd(8)
To slim down your Beowulf node, get rid of the extra services in 'inetd.conf'; you probably will not require the /usr/bin/talk program on each of the compute nodes. Of course, what is required will depend on the computing environment. In many very secure environments, where ssh is run as a daemon and not launched from 'inetd.conf' for every new connection, 'inetd.conf' has no entries. In such extreme examples, the inetd process that normally reads 'inetd.conf' and listens on ports, ready to launch services, can even be eliminated.
The next step is to eliminate any daemons or processes that are normally started at boot. While occasionally Linux distributions differ in style, the organization of the files that launch daemons or run scripts during the first phases of booting up a system are very similar. For most distributions, the directory '/etc/rc.d/init.d' contains scripts that are run when entering or leaving a run level. Below is an example:
% cd /etc/rc.d/init.d % ls anacron functions kdcrotate nfslock sendmail wine apachectl gpm keytable nscd single xfs apmd halt killall ntpd snmpd xinetd arpwatch http_sanity kudzu portmap snmptrapd ypbind atd http_sanity~ lpd radvd sshd yppasswdd autofs identd netfs random syslog ypserv crond ipchains network rawdevices vncserver ypxfrd cups iptables nfs rhnsd winbind
However, the presence of the script does not indicate it will be run. Other directories and symlinks control which scripts will be run. Most systems now use the convenient chkconfig interface for managing all the scripts and symlinks that control when they get turned on or off. Not every script spawns a daemon. Some scripts just initialize hardware or modify some setting.
A convenient way to see all the scripts that will be run when entering run level 3 is the following:
% chkconfig --list | grep '3:on' syslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off lpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off mysql 0:off 1:off 2:on 3:on 4:on 5:on 6:off httpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off sshd 0:off 1:off 2:off 3:on 4:on 5:on 6:off atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off named 0:off 1:off 2:off 3:on 4:on 5:on 6:off dhcpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off gpm 0:off 1:off 2:on 3:on 4:on 5:on 6:off inet 0:off 1:off 2:off 3:on 4:on 5:on 6:off network 0:off 1:off 2:on 3:on 4:on 5:on 6:off nfsfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off random 0:off 1:off 2:on 3:on 4:on 5:on 6:off keytable 0:off 1:off 2:on 3:on 4:on 5:on 6:off nfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off ntpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off portmap 0:off 1:off 2:off 3:on 4:on 5:on 6:off sendmail 0:off 1:off 2:on 3:on 4:on 5:on 6:off serial 0:off 1:off 2:on 3:on 4:on 5:on 6:off squid 0:off 1:off 2:off 3:on 4:on 5:on 6:off tltime 0:off 1:off 2:off 3:on 4:off 5:on 6:off crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Remember that not all of these spawn cycle-stealing daemons that are not required for Beowulf nodes. The "serial" script, for example, simply initializes the serial ports at boot time; its removal is not likely to reduce overall performance. However, in this example many things could be trimmed. For example, there is probably no need for lpd, mysql, httpd, named, dhcpd, sendmail, or squid on a compute node. It would be a good idea to become familiar with the scripts and use the chkconfig command to turn off unneeded scripts. With only a few exceptions, an X-Windows server should not be run on a compute node. Starting an X session takes ever-increasing amounts of memory and spawns a large set of processes. Except for special circumstances, run level 3 will be the highest run level for a compute node.
In addition to 'inetd.conf' and the scripts in '/etc/rc.d/init.d', there are other common ways for a Beowulf node to waste CPU or memory resources. The cron program is often used to execute programs at scheduled times. For example, cron is commonly used to schedule a nightly backup or an hourly cleanup of system files. Many distributions come with some cron scripts scheduled for execution. The program slocate is often run as a nightly cron to create an index permitting the file system to be searched quickly. Beowulf users may be unhappy to learn that their computation and file I/O are being hampered by a system utility that is probably not useful for a Beowulf. A careful examination of cron and other ways that tasks can be started will help trim a Beowulf compute node.
The ps command can be invaluable during your search-and-destroy mission.
% ps -eo pid,pcpu,sz,vsize,user,fname --sort=vsize
This example command demonstrates sorting the processes by virtual memory size.
The small excerpt below illustrates how large server processes can use memory. The example is taken from a Web server, not a well-tuned Beowulf node.
PID %CPU SZ VSZ USER COMMAND 26593 0.0 804 3216 web httpd 26595 0.0 804 3216 web httpd 3574 0.0 804 3216 web httpd 506 0.0 819 3276 root squid 637 0.0 930 3720 root AgentMon 552 0.0 1158 4632 dbenl postmast 13207 0.0 1213 4852 root named 13209 0.0 1213 4852 root named 13210 0.0 1213 4852 root named 13211 0.0 1213 4852 root named 13212 0.0 1213 4852 root named 556 0.0 1275 5100 dbenl postmast 657 0.0 1280 5120 dbenl postmast 557 0.0 1347 5388 dbenl postmast 475 0.0 2814 11256 mysql mysqld 523 0.0 2814 11256 mysql mysqld 524 0.0 2814 11256 mysql mysqld 507 0.0 3375 13500 squid squid
In this example the proxy cache program squid is using a lot of memory (and probably some cache), even though the CPU usage is negligible. Similarly, the ps command can be used to locate CPU hogs. Becoming familiar with ps will help quickly find runaway processes or extra daemons competing for cycles with the scientific applications intended for your Beowulf.