Section 10.1. syslog

syslog is the tried-and-true workhorse of Unix logging utilities. It accepts log data from the kernel (by way of klogd), from any and all local process, and even from processes on remote systems. It's flexible as well, allowing you to determine what gets logged and where it gets logged to.

A preconfigured syslog installation is part of the base operating system in virtually all variants of Unix and Linux. However, relatively few system administrators customize it to log the things that are important for their environment and disregard the things that aren't. Since, as few would dispute, information overload is one of the major challenges of system administration, this is unfortunate. Therefore, we begin this chapter with a comprehensive discussion of how to customize and use syslog.

10.1.1 Configuring syslog

Whenever syslogd, the syslog daemon, receives a log message, it acts based on the message's type (or "facility") and its priority. syslog's mapping of actions to facilities and priorities is specified in /etc/syslog.conf. Each line in this file specifies one or more facility/priority selectors followed by an action; a selector consists of a facility or facilities and a (single) priority.

In the following syslog.conf line in Example 10-1, mail.notice is the selector and /var/log/mail is the action (i.e., "write messages to /var/log/mail").

Example 10-1. Sample syslog.conf line

mail.notice                /var/log/mail

Within the selector, mail is the facility (message category) and notice is the level of priority.

10.1.1.1 Facilities

Facilities are simply categories. Supported facilities in Linux are auth, auth-priv, cron, daemon, kern, lpr, mail, mark, news, syslog, user, uucp, and local0 through local7. Some of these are self-explanatory, but the following are of special note:

auth

Used for many security events.

auth-priv

Used for access-control-related messages.

daemon

Used by system processes and other daemons.

kern

Used for kernel messages.

mark

Messages generated by syslogd itself, which contain only a timestamp and the string --MARK--; to specify how many minutes should transpire between marks, invoke syslogd with the -m [minutes] flag.

user

The default facility when none is specified by an application or in a selector.

ocal7

Boot messages.

*

Wildcard signifying "any facility."

none

Wildcard signifying "no facility."

10.1.1.2 Priorities

Unlike facilities, which have no relationship to each other, priorities are hierarchical. Possible priorities in Linux are (in increasing order of urgency): debug, info, notice, warning, err, crit, alert, and emerg. Note that the "urgency" of a given message is determined by the programmer who wrote it; facility and priority are set by the programs that generate messages, not by syslog.

As with facilities, the wildcards * and none may also be used. Only one priority or wildcard may be specified per selector. A priority may be preceded by either or both of the modifiers, = and !.

If you specify a single priority in a selector (without modifiers), you're actually specifying that priority plus all higher priorities. Thus the selector mail.notice translates to "all mail-related messages having a priority of notice or higher," i.e., having a priority of notice, warning, err, crit, alert, or emerg.

You can specify a single priority by prefixing a = to it. The selector mail.=notice translates to "all mail-related messages having a priority of notice." Priorities may also be negated: mail.!notice is equivalent to "all mail messages except those with priority of noticeor higher," and mail.!=notice corresponds to "all mail messages except those with the priority notice."

10.1.1.3 Actions

In practice, most log messages are written to files. If you list the full path to a filename as a line's action in syslog.conf, messages that match that line will be appended to that file. (If the file doesn't exist, syslog will create it.) In Example 10-1, we instructed syslog to send matched messages to the file /var/log/mail.

You can send messages other places too. An action can be a file, a named pipe, a device file, a remote host, or a user's screen. Pipes are usually used for debugging. Device files that people use are usually TTYs. Some people also like to send security information to /dev/lp0 ? i.e., to a local line printer. Logs that have been printed out can't be erased or altered by an intruder, but they also are subject to mechanical problems (paper jams, ink depletion, etc.) and are harder to parse if you need to find something in a hurry.

Remote logging is one of the most useful features of syslog. If you specify a hostname or IP address preceded by an @ sign as a line's action, messages that match that line will be sent to UDP port 514 on that remote host. For example, the line:

*.emerg            @mothership.mydomain.org

will send all messages with emerg priority to UDP port 514 on the host named mothership.mydomain.org. Note that the remote host's (in this example, mothership's) syslogd process will need to have been started with the -r flag for it to accept your log messages. By default, syslogd does not accept messages from remote systems.

syslog has no access-control mechanism of its own: if you enable the reception of remote messages with the -r flag, your host will accept messages on UDP port 514 from any and all remote computers. See the end of this section for some advice on how to mitigate this.

If you run a central log server, which I highly recommend, you'll want to consider some sort of access controls on it for incoming messages. At the very least, you should consider tcpwrappers' "hosts access" (source-IP-based) controls or maybe even local firewall rules (ipchains or iptables).

10.1.1.4 More sophisticated selectors

You can list multiple facilities separated by commas in a single syslog.conf selector. To extend Example 10-1 to include both mail and uucp messages (still with priority notice or higher), you could use this line (Example 10-2).

Example 10-2. Multiple facilities in a single selector

mail,uucp.notice   /var/log/mail

The same is not true of priorities. Remember that only one priority or priority wildcard may be specified in a single selector.

arp -s 192.168.192.168 03:03:03:31:33:77

You may, however, specify multiple selectors separated by semicolons. When a line contains multiple selectors, they're evaluated from left to right: you should list general selectors first, followed by more specific selectors. You can think of selectors as filters: as a message is passed through the line from left to right, it passes first through coarse filters and then through more granular ones.

Continuing our one-line example, suppose we still want important mail and uucp messages to be logged to /var/log/mail, but we'd like to exclude uucp messages with priority alert. Our line then looks like Example 10-3.

Example 10-3. Multiple selectors in a single line

mail,uucp.notice;uucp.!=alert    /var/log/mail

Actually, syslogd's behavior isn't as predictable as this may imply: listing selectors that contradict each other or that go from specific to general rather than vice versa can yield unexpected results. Therefore, it's more accurate to say "for best results, list general selectors to the left and their exceptions (and/or more-specific selectors) to the right." Wherever possible, keep things simple. You can use the logger command to test your syslog.conf rules (see "Testing System Logging with logger" later in this chapter).

Note that in the second selector (uucp.!=alert), we used the prefix != before the priority to signify "not equal to." If we wanted to exclude uucp messages with priority alert and higher (i.e, alert and emerg), we could omit the = (see Example 10-4).

Example 10-4. Selector list with a less specific exception

mail,uucp.notice;uucp.!alert     /var/log/mail

You might wonder what will happen to a uucp message of priority info: this matches the second selector, so it should be logged to /var/log/mail, right? Not based on the previous examples. Since the line's first selector matches only mail and uucp messages of priority notice and higher, such a message wouldn't be evaluated against the second selector.

There's nothing to stop you from having a different line for dealing with info-level uucp messages, though. You can even have more than one line deal with these if you like. Unlike a firewall rule base, each log message is tested against all lines in /etc/syslog.conf and acted on as many times as it matches.

Suppose we want emergency messages broadcast to all logged-in users, as well as written to their respective application logs. We could use something like Example 10-5.

Example 10-5. A sample syslog.conf file

# Sample syslog.conf file that sorts messages by mail, kernel, and "other,"

# and broadcasts emergencies to all logged-in users

   

# print most sys. events to tty10 and to the xconsole pipe, and emergencies to everyone

kern.warn;*.err;authpriv.none    |/dev/xconsole

*.emerg                           *

   

# send mail, news (most), & kernel/firewall msgs to their respective logfiles

mail.*                           -/var/log/mail

kern.*                           -/var/log/kernel_n_firewall

   

# save the rest in one file

*.*;mail.none                    -/var/log/messages

Did you notice the - (minus) sign in front of the write-to-file actions? This tells syslogd not to synchronize the specified log file after writing a message that matches that line. Skipping synchronization decreases disk utilization and thus improves performance, but it also increases the chances of introducing inconsistencies, such as missing or incomplete log messages, into those files. Use the minus sign, therefore, only in lines that you expect to result in numerous or frequent file writes.

Besides performance optimization, Example 10-5 also contains some useful redundancy. Kernel warnings plus all messages of error-and-higher priority, except authpriv messages, are printed to the X-console window. All messages having priority of emergency and higher are too, in addition to being written to the screens of all logged-in users.

Furthermore, all mail messages and kernel messages are written to their respective log files. All messages of all priorities (except mail messages of any priority) are written to /var/log/messages.

Example 10-5 was adapted from the default syslog.conf that SuSE 7.1 put on one of my systems. But why shouldn't such a default syslog.conf file be fine the way it is? Why change it at all?

Maybe you needn't, but you probably should. In most cases, default syslog.conf files either:

• Assign to important messages at least one action that won't effectively bring those messages to your attention (e.g., by sending messages to a TTY console on a system you only access via SSH)

• Handle at least one type of message with too much or too little redundancy to meet your needs

We'll conclude our discussion of syslog.conf with Tables Table 10-1 through Table 10-4, which summarize syslog.conf's allowed facilities, priorities, and types of actions. Note that numeric codes should not be used in syslog.conf on Linux systems. They are provided here strictly as a reference, should you need to configure a non-Linux syslog daemon that uses numeric codes (e.g., Cisco IOS), or to send syslog messages to your log server because they're used internally (i.e., in raw syslog packets). You may see them referred to elsewhere.

10.1.1.5 Running syslogd

Just as the default syslog.conf may or may not meet your needs, the default startup mode of syslogd may need tweaking as well. Table 10-5 and subsequent paragraphs touch on some syslogd startup flags that are particularly relevant to security. For a complete list, you should refer to the manpage sysklogd (8).

In addition, note that when you're changing and testing syslog's configuration and startup options, it usually makes sense to start and stop syslogd and klogd in tandem (see the "What About klogd?" sidebar at the beginning of this chapter if you don't know what klogd is). Since it also makes sense to start and stop these the same way your system does, I recommend that you use your system's syslog/klogd startup script.

On most Linux systems, both facilities are controlled by the same startup script, named either /etc/init.d/syslog or /etc/init.d/sysklog ("sysklog" is shorthand for "syslog and klogd"). See Table 10-5 for a list of some of syslogd's flags.

-r

The first syslogd flag we'll discuss is the only one used by default in Red Hat 7.x in its /etc/init.d/syslog script. This flag is -m 0, which disables mark messages. mark messages contain only a timestamp and the string --MARK--, which some people find useful for navigating lengthy log files. Others find them distracting and redundant, given that each message has its own timestamp anyhow.

To turn mark messages on, specify a positive nonzero value after -m that tells syslogd how many minutes should pass before it sends itself a mark message. Remember that mark has its own facility (called, predictably, "mark") and that you must specify at least one selector that matches mark messages (such as mark.*, which matches all messages sent to the mark facility, or *.*, which matches all messages in all facilities).

For example, to make syslogd generate mark messages every 30 minutes and record them in /var/log/messages, you would first add a line to /etc/syslog.conf similar to Example 10-6.

Example 10-6. syslog.conf selector for mark-messages

mark.*                            -/var/log/messages

You would then need to start syslogd, as shown in Example 10-7.

Example 10-7. Invoking syslogd with 30-minute marks

mylinuxbox:/etc/init.d# ./syslogd -m 30

Another useful syslogd flag is -a [socket]. This allows you to specify one or more sockets (in addition to /dev/log for syslogd) from which to accept messages.

In Chapter 6, we used this flag to allow a chrooted named process to bounce its messages off of a dev/log socket (device-file) in the chroot jail to the nonchrooted syslogd process. In that example, BIND was running in a "padded cell" (subset of the full filesystem) and had its own log socket, /var/named/dev/log. We therefore changed a line in /etc/init.d/syslog that read as shown in Example 10-8.

Example 10-8. init.d/syslog line invoking syslogd to read messages from a chroot jail

daemon syslogd -m 0 -a /var/named/dev/log

(Note that the "daemon" function at the beginning of this line is unique to Red Hat's init script functions; the important part here is syslogd -m 0 -a /var/named/dev/log.)

More than one -a flag may be specified (Example 10-9).

Example 10-9. Invoking syslogd with multiple "additional log device" directives

syslogd -a /var/named/dev/log -a /var/otherchroot/dev/log -a /additional/dev/log

Continuing down the list of flags in Table 10-5, suppose you need to test a new syslog configuration file named syslog.conf.test, but you prefer not to overwrite /etc/syslog.conf, which is where syslogd looks for its configuration file by default. Use the -f flag to tell syslogd to use your new configuration file (Example 10-10).

Example 10-10. Specifying the path to syslogd's configuration file

mylinuxbox:/etc/init.d# ./syslogd -f ./syslog.conf.test

We've already covered use of the -r flag, which tells syslogd to accept log messages from remote hosts, but we haven't talked about the security ramifications of this. On the one hand, security is clearly enhanced when you use a centralized log server or do anything else that makes it easier for you to manage and monitor your logs.

On the other hand, you must take different threat models into account. Are your logs sensitive? If log messages traverse untrusted networks and if the inner workings of the servers that send those messages are best kept secret, then the risks may outweigh the benefit (at least, the specific benefit of syslogd's unauthenticated clear-text remote logging mechanism).

If this is the case for you, skip to this chapter's section on Syslog-ng. Syslog-ng can send remote messages via the TCP protocol and can therefore be used in conjunction with stunnel, ssh, and other tools that greatly enhance its security. Since syslog uses only the connectionless UDP protocol for remote logging and therefore can't "tunnel" its messages though stunnel or ssh, syslog is inherently less securable than Syslog-ng.

If your log messages aren't sensitive (at least the ones you send to a remote logger), then there's still the problem of Denial of Service and message forgery attacks. If you invoke syslogd with the -r flag, it will accept all remote messages without performing any checks whatsoever on the validity of the messages themselves or on their senders. Again, this risk is most effectively mitigated by using Syslog-ng.

But one tool you can use with syslog to partially mitigate the risk of invalid remote messages is TCPwrappers. Specifically, TCPwrappers' "hosts access" authentication mechanism provides a simple means of defining which hosts may connect and via which protocols they may connect to your log server. Hosts-access authentication is easily tricked by source-IP-spoofing (especially since syslog transactions are strictly one way), but it's better than nothing, and it's probably sufficient to prevent mischievous but lazy attackers from interfering with syslog.

If you're willing to bet that it is, obtain and install TCPwrappers and refer to its hosts_access(5) manpage for details. Note that despite its name, TCPwrappers' hosts access can be used to control UDP-based applications.