Chapter 10. System Log Management and Monitoring

Whatever else you do to secure a Linux system, it must have comprehensive, accurate, and carefully watched logs. Logs serve several purposes. First, they help us troubleshoot virtually all kinds of system and application problems. Second, they provide valuable early-warning signs of system abuse. Third, after all else fails (whether that means a system crash or a system compromise), logs can provide us with crucial forensic data.

This chapter is about making sure your system processes and critical applications log the events and states you're interested in and dealing with this data once it's been logged. The two logging tools we'll cover are syslog and the more powerful Syslog-ng ("syslog new generation"). In the monitoring arena, we'll discuss Swatch (the Simple Watcher), a powerful Perl script that monitors logs in real time and takes action on specified events.

What About klogd?

One daemon you probably won't need to reconfigure but should still be aware of is klogd, Linux's kernel log daemon. This daemon is started automatically at boot time by the same script that starts the general system logger (probably /etc/init.d/syslogd or /etc/init.d/sysklogd, depending on which Linux distribution you use).

By default, klogd directs log messages from the kernel to the system logger, which is why most people don't need to worry about klogd: you can control the handling of kernel messages by editing the configuration file for syslogd.

This is also true if you use Syslog-ng instead of syslog, but since Syslog-ng accepts messages from a much wider variety of sources, including /proc/kmsg (which is where klogd receives its messages), some Syslog-ng users prefer to disable klogd. Don't do so yourself unless you first configure Syslog-ng to use /proc/kmsg as a source.

klogd can be invoked as a standalone logger; that is, it can send kernel messages directly to consoles or a log file. In addition, if it isn't already running as a daemon, klogd can be used to dump the contents of the kernel log buffers (i.e., the most recent kernel messages) to a file or to the screen. These applications of klogd are especially useful to kernel developers.

For most of us, it's enough to know that for normal system operations, klogd can be safely left alone (that is, left with default settings and startup options ? not disabled). Just remember that when you use syslog in Linux, all kernel messages are handled by klogd first.