System Startup
System Startup
There are three kinds of boots that administrators should be aware of. In addition to a normal reboot, which is initiated by the command
# shutdown
from a superuser shell, a reconfiguration boot involves reconstructing device information in the /dev and /devices directories, while a recovery boot involves saving and analyzing crash dump files if a system does not respond to commands issued on the console. A reconfiguration boot is commonly undertaken in older SPARC systems when new hard disks are added to the system, although this may not be necessary with newer systems, such as the E450, which have hot-swapping facilities. This kind of boot can be initiated by typing
# boot -r
at the OpenBoot monitor prompt, or by issuing the command
# touch /reconfigure
prior to issuing a shutdown command from a superuser shell. A recovery boot is a rare event on a Solaris system—although hardware failures, kernel module crashes, and incorrect kernel parameters can sometimes result in a hung system. A stack trace is usually provided if a system crash occurs, which can provide vital clues to tracking the source of any system problems using the kernel debugger (kadb).
Although Solaris has eight init states, only five are commonly encountered by administrators during normal operations. The first is run level S, which is a single-user init state used for administrative tasks and the repair of corrupted file systems, using the following command:
# /usr/sbin/fsck
Also encountered are run level 2, where the init state changes to multiuser mode for the first time, with the exception of NFS exported network resources; run level 3, where all users can log in, and all system and NFS network resources are available; run level 6, which halts the operating system and initiates a reboot; and run level 0, during which the operating system is shut down, ensuring it is safe to power down. In older SPARC systems, it is necessary to bring the system down to run level 0 to install new hardware, such as disk drives, peripheral devices, and memory modules. However, newer systems, such as the E450, are able to continue to operate in multiuser init states while disks are hot-swapped into special drive bays. This means that these machines may not have a need to enter - run level 6. Further, uptimes of many months or years are not uncommon.
The Solaris software environment provides a detailed series of run control (rc) scripts to control run level changes. In this section, we will examine each of the control scripts in turn, and highlight the improvements and innovations from the old BSD-style Solaris 1.x control scripts. Each run level has an associated rc script located in the /sbin directory, which is also symbolically linked into the /etc directory: rc0, rc1, rc2, rc3, rc5, rc6, and rcS. /sbin/rc0 is responsible for:
-
Executing all scripts in /etc/rc0.d, if the directory exists.
-
Terminating all system services and active processes, initially using /usr/sbin/killall and /usr/sbin/killall 9 for stubborn processes.
-
Syncing all mounted file systems, using /sbin/sync.
-
Unmounting all mounted file systems, using /sbin/umountall.
/sbin/rc5 and /sbin/rc6 are just symbolic links to /sbin/rc0, and do not need to be maintained separately, whilst /sbin/rc1 is responsible for executing all scripts in the /etc/rc1.d directory, if it exists. This terminates all system services and active processes, initially using /usr/sbin/killall, and /usr/sbin/killall 9 for stubborn processes. The differences between /etc/rc0 and /etc/rc1 are that the latter brings up the system into single-user mode after shutting down all processes in multiuser mode, and does not unmount any file systems.
In run level 2 state, /sbin/rc2 executes all scripts in the /etc/rc2.d directory, bringing the system into its first multiuser state. Thus, all local file systems listed in /etc/vfstab are mounted, disk quotas and file system logging are switched on if configured, temporary editor files are saved, the /tmp directory is cleared, system accounting is enabled, and many network services are initialized. These services are described in more detail in Table 5-2.
In run level 3 state, /sbin/rc3 executes all scripts in the /etc/rc3.d directory, bringing the system into its final multiuser state. These services are mainly concerned with shared network resources, such as NFS, but Solstice Enterprise Agents and other SNMP-based systems may also be started here. /sbin/rcS executes all scripts in the /sbin/rcS.d directory, to bring the system up to the single-user run level. A minimal network configuration is established if a network can be found; otherwise, an interface error is reported. Essential system file systems (such as /, /usr, and /proc) are mounted if they are available, and the system name is set.
Under Solaris 1.x, there were two main BSD-style control scripts: /etc/rc and /etc/rc.local. Typically, vendor-provided daemons were initialized from /etc/rc, while customized and locally installed daemons were executed from /etc/rc.local. For example, /etc/rc was responsible for mounting file systems, enabling quotas, adding swap space, and starting the Internet super daemon (inetd). Alternatively, /etc/rc.local was responsible for later innovations, such as web servers and authentication services, as well as printer drivers. A general rule of thumb when upgrading legacy systems from Solaris 1.x to Solaris 2.x is to cross-check all of the required services in /etc/rc, and ensure that they are enabled in either /etc/rc1.d or /etc/rc2.d, and to add any local customizations from /etc/rc.local to a System V–style startup script in /etc/rc2.d. Alternatively, shared network resource scripts can be added to /etc/rc3.d.
| Tip |
Many Solaris 1.x applications will run in binary compatibility mode under Solaris 2.x, but your software vendor should be contacted for the latest versions of third-party software. |
To the superuser on the console, the transition between run levels is virtually invisible: most daemons, whether starting in a single-user or multiuser init state, display a status message when starting up, which is echoed to the console. A sample console display during booting will look something like this:
ok boot Resetting ... SPARCstation 20 (2 X 390Z50), Keyboard Present ROM Rev. 2.4, 128 MB memory installed, Serial #6745644 Ethernet address 6:3:10:a:cc:4a HostID 5767686 Boot device: /iommu/sbus/espdma@f,400000/esp@f,800000/sd@1,0 File and args: SunOS Release 5.9 Version generic [UNIX(R) System V Release 4.0] Copyright (c) 1983-2002, Sun Microsystems, Inc. configuring network interfaces: le0. Hostname: server The system is coming up. Please wait. add net default: gateway 10.16.27.1 NIS domainname is subdomain.mydomain.com starting rpc services: rpcbind keyserv ypbind done. Setting netmask of le0 to 255.255.0.0 Setting default interface for multicast: add net 224.0.0.0: gateway server syslog service starting. Print services started. volume management starting. Starting Apache webserver...done. The system is ready. server console login:
When booting into single-user mode, there will obviously be fewer messages displayed on the console, as multiuser init state processes are not started. The single-user run level messages will appear as something like this:
ok boot -s SunOS Release 5.9 Version [UNIX(R) System V Release 4.0] Copyright (c) 1983-2001, Sun Microsystems, Inc. configuring network interfaces: le0. Hostname: server INIT: SINGLE USER MODE Type Ctrl-d to proceed with normal startup, (or give root password for system maintenance):
At this point, the password for the superuser account should be entered (it will not be echoed to the display). Assuming that the correct password is entered, the display will then proceed with another banner and a Bourne shell prompt:
Sun Microsystems Inc. SunOS 5.9 November 2001 #
After maintenance is complete, simply exit the shell by using CTRL-D, and the system will then proceed with a normal multiuser boot.
The /sbin/init daemon is responsible for process control initialization, and is a key component of the booting process. While it is not significant in many day-to-day operations after booting, its configuration for special purposes can be confusing for first-time users. In this section, we will examine the initialization of init using the /etc/inittab file, and explain in detail what each entry means. The primary function of init is to spawn processes, usually daemon processes, from configuration information specified in the file /etc/inittab in ASCII format.
| Note |
Process spawning always takes place in a specific software context, which is determined by the current run level. |
After booting the kernel from the OpenBoot monitor, init reads the system environment variables stored in /etc/default/init (for example, the time zone variable TZ), and sets them for the current run level. init then reads the /etc/inittab file (described more completely in the next section), setting the init level specified in that file by the initdefault entry. In most multiuser systems, this entry will correspond to run level 3, and the entry will look like this:
is:3:initdefault:
If the file /etc/inittab does not exist during booting, the superuser will be asked to manually enter the desired run level for the system. If this event ever occurs unexpectedly for a multiuser system, it is a good strategy to enter single-user mode (by typing s) to perform maintenance on the /etc/inittab file. Another potential problem (which is discussed later) is if /etc/inittab does contain an empty rstate value in the initdefault entry: the system will go to firmware and continuously reboot! If this occurs, exit from the operating system into the OpenBoot monitor by holding down the STOP key, and pressing A. You can now boot directly into single-user mode, and add an appropriate rstate entry to the /etc/inittab file. There are safeguards built into init, however: if the system discovers that any entry in /etc/inittab is respawning rapidly (that is, more than five times per minute), init assumes that a typographical error has been made in the entry, and a warning message is printed on the system console. init will then not respawn the affected entry until at least five minutes has elapsed since the problem was identified.
After entering a multiuser run level for the first time since booting from the OpenBoot monitor, init reads any appropriate boot and bootwait entries in /etc/ inittab. This provides for basic initialization of the operating system, such as mounting file systems, which is generally performed before users may be allowed to operate on the system.
In order to spawn processes specified in /etc/inittab, init reads each entry and determines the process requirements for the commands to be executed. For example, for entries that must be respawned in the future, a child process is created using fork(). After reading all entries and spawning all processes, init simply waits until it receives a signal to change the system’s init state (this explains why init is always visible in the process list). /etc/inittab is always reread at this point to ensure that any modifications to its specified behavior are used. In addition, init can be initialized at any time by passing a special parameter to force rereading of /etc/inittab:
# init q
When init receives a valid request to change run levels, a warning signal is sent to all affected processes, and it waits five seconds before forcibly terminating any processes that do not behave well, and then exits by sending a kill signal. Affected processes are those that will be invalid under the target init state (for example, when going from multiuser to single-user mode, daemons started in multiuser mode will be invalid). Since five seconds may not be sufficient to shut down an entire database server and close all open files, it is best to ensure that such activities precede any change of state that affects the main applications running on your system (for example, by executing the appropriate command in /etc/init.d with the stop parameter).
/sbin/init can only be executed by a superuser, as changes in the system’s init state executed by a normal user could have serious consequences (for example, using init to power down a live server). Thus, it is always wise to ensure that file permissions are correctly set on the /sbin/init binary.
Shutdown
A Solaris system is designed to stay up continuously, with as few disruptions to service through rebooting as possible. This design is facilitated by a number of key high-availability and redundancy features in Solaris, including:
-
Dual power supplies, where a secondary supply can continue to power the system if the primary power supply fails.
-
Mirroring of disk data, meaning that the system can generally continue to operate even in the face of multiple disk failure.
-
Hot-swappable disks, meaning that a faulty disk can be removed and replaced while the system is still online. The new disk can be formatted and used immediately, especially when DiskSuite is used.
-
The use of domains on E10000 systems, where maintenance performed on one “virtual” host can be performed while a second domain acts in its place.
However, there are a number of situations where a Solaris system must be halted by the superuser, such as:
-
Performing a reconfiguration boot
-
Powering down the system
Note that the drvconfig command can be used to recognize most new hardware devices, further reducing the need for rebooting. A number of different commands are available to shut down and halt a system, and which one is used depends on the specific situation at hand. For example, some commands cycle through a series of shutdown scripts that ensure that key applications and services, such as databases, are cleanly shut down. Others are designed to ensure that a system is powered down as rapidly as possible. For example, if a storm strikes out the main power system and you’re only left with a few minutes of battery backup, it might be wise to perform a rapid power down to protect equipment from further damage. We’ll investigate the following commands: init, shutdown, poweroff, halt, and reboot.
Shutting Down the System
The shutdown command is used to change a system’s state, performing a similar function to init as described previously. However, shutdown has several advantages over init, for instance:
-
A grace period can be specified, so that the system can be shut down at some future time, rather than immediately.
-
A confirmation message requires the superuser to confirm the shutdown before it proceeds. If an automated shutdown is to be executed at some future time, the confirmation message can be avoided by using the -y option.
-
Only init states 0, 1, 5, 6, and S can be reached using the shutdown command.
For example, to shut down the system to run level 5 so that the system can be moved, the following command would be used, giving 60 seconds notice:
# shutdown -i 5 -g 60 "System will be powered off for maintenance. LOGOUT NOW."
This will print the following messages at 60 and 30 seconds, respectively:
Shutdown started. Thu Jun 21 12:00:00 EST 2001
Broadcast Message from root (pts/1) on cassowary Thu Jun 21
12:00:00 EST 2001...
The system will be shut down in 1 minute
System will be powered off for maintenance. LOGOUT NOW.
Shutdown started. Thu Jun 21 12:00:30 EST 2001
Broadcast Message from root (pts/1) on cassowary Thu Jun 21
12:30:00 EST 2001...
The system will be shut down in 30 seconds
System will be powered off for maintenance. LOGOUT NOW.
Once the countdown has been completed, the following message will appear:
Do you want to continue? (y or n):
If you type y, the shutdown will proceed. If you type n, the shutdown will be cancelled and the system will remain at the current run level.
Rebooting
The reboot command is used to reboot the system, from the current run level to the default run level, and not to change to any other run level. The reboot command has several options: the -l flag can be used to prevent the recording of the system halt in the system log, which it normally attempts before halting the CPU, while the -n option prevents the refreshing of the superblock, which is performed by default to prevent damage to mounted file systems. The most extreme option is -q, which does not attempt any kind of fancy actions before shutting down.
In addition, reboot accepts the standard parameters passed to the boot command, if they are preceded by two dashes and are placed after the reboot parameters described above on the command line.
For example, to perform a configuration reboot without recording an entry in the system log, the following command could be used:
# reboot -l -- -r
Reconfiguration Boot
Performing a reconfiguration boot involves updating the hardware configuration for the system. If new hardware is added to the system, other than a disk, the system must be brought down to the hardware maintenance state (level 0) before the new device can be inserted. In addition, the system must be notified of a reconfiguration reboot by either booting from the OpenBoot PROM monitor with the command boot -r, or by creating an empty file called reconfigure in the root directory before changing to run level 0. This can be achieved by using the command touch /reconfigure. Be sure to remove the /reconfigure file after the system has been reconfigured if not rebooting!
Powering Down
The poweroff command is used to rapidly shut down the system, and switch off power (like switching to run level 5), without cycling through any intermediate run levels, and executing the kill scripts specified for those run levels. This ensures that a very fast shutdown can be achieved when emergency situations dictate that the system cannot remain live, even with the risk of data loss. For example, if a system is under a denial of service attack and the decision is made to pull the plug on the service, the halt command will do so much faster than init or shutdown. The CPU is halted as quickly as possible, no matter what the run level.
The poweroff command has several options: the -l flag can be used to prevent the recording of the system halt in the system log, which it normally attempts before halting the CPU, while the -n option prevents the refreshing of the superblock, which is performed by default to prevent damage to mounted file systems. The most extreme option is -q, which does not attempt any kind of fancy actions before shutting down.
Halting the System
The halt command is used to rapidly shut down the system, to the OpenBoot PROM monitor, without cycling through any intermediate run levels, and executing the kill scripts specified for those run levels. This ensures that a very fast shutdown can be achieved when emergency situations dictate that the system cannot remain live, even with the risk of data loss. For example, if a system is under a denial of service attack, and the decision is made to pull the plug on the service, halt will do so much faster than init or shutdown. The CPU is halted as quickly as possible, no matter what the run level.
The halt command has several options: the -l flag can be used to prevent the recording of the system halt in the system log, which it normally attempts before halting the CPU, while the -n option prevents the refreshing of the superblock, which is performed by default to prevent damage to mounted file systems. The most extreme option is -q, which does not attempt any kind of fancy actions before halting.
