21.5 Handwritten Logs

Another type of logging that can help you with security is not done by the computer at all; it is done by you and your staff. Keep a log book that records your day's activities. Log books should be kept on paper in a physically secure location. Because you keep them on paper, they cannot be altered by someone hacking into your computer even as superuser. They will provide a nearly tamperproof record of important information.

Handwritten logs have several advantages over online logs:

  • They can record many different kinds of information. For example, your computer will not record a suspicious telephone call or a bomb threat, but you can (and should) record these occurrences in your log book.

  • If the systems are down, you can still access your paper logs. (Thus, this is a good place to keep a copy of account numbers and important phone numbers for field service, service contacts, and your own key personnel.)

  • If disaster befalls your disks, you can recreate some vital information from paper, if it is in the log book.

  • If you keep the log book as a matter of course, and you enter into it printed copies of your exception logs, such information might be more likely to be accepted into court proceedings as business records. This advantage is important if you are in a situation in which you need to pursue criminal or civil legal action.

  • Juries are more easily convinced that paper logs are authentic, as opposed to computer logs.

  • Having copies of significant information in the log book keeps you from having to search all the disks on all your workstations for some selected information.

  • If all your other tools fail or have possibly been compromised, holding an old printout and a new printout of the same file together and up to a bright light may be a quick way to reveal changes.

Think of your log book as a laboratory notebook, except the laboratory is your own computer center. Each page should be numbered. You should not rip pages out of your book. Write in ink, not pencil. If you need to cross something out, draw a single line, but do not make the text that you are scratching out unreadable. Keep your old log books.

The biggest problem with log books is the amount of time you need to keep them up to date. These are not items that can be automated with a shell script. Unfortunately, this time requirement is the biggest reason why many administrators are reluctant to keep logs?especially at a site with hundreds (or thousands) of machines, each of which might require its own log book. We suggest that you try to be creative and think of some way to balance the need for good records against the drudgery of keeping multiple books up to date. Compressing information and keeping logs for each cluster of machines are ways to reduce the overhead while receiving (nearly) the same benefit.

There are basically two kinds of log books: per-site logs and per-machine logs. We'll outline the kinds of material you might want to keep in each type. Be creative, though, and don't limit yourself to what we suggest here.

21.5.1 Per-Site Logs

In a per-site log book, you keep information that would be useful across all your machines and throughout your operations. The information can be further divided into exception and activity reports, and informational material. Exception and activity reports

These reports hold such information as the following:

  • Times, dates, and duration of power outages; over time, such information may help you justify uninterruptible power supplies or trace a cause of frequent problems

  • Servicing and testing of alarm systems

  • Triggering of alarm systems

  • Servicing and testing of fire suppression systems

  • Visits by service personnel, including the phone company

  • Dates of employment and termination of employees with privileged access (or with any access) Informational material

This material contains such information as the following:

  • Contact information for important personnel, including corporate counsel, law enforcement, field service, and others who might be involved in any form of incident

  • Copies of purchase orders, receipts, and licenses for all software installed on your systems; these will be invaluable if you are one of the targets of a Software and Information Industry Association (formerly the Software Publishers Association) audit

  • Serial numbers for all significant equipment on the premises

  • All MAC-level addresses for each machine (e.g., Ethernet addresses) with corresponding IP (or other protocol) numbers

  • Time and circumstances of formal bug reports made to the vendor

  • Phone numbers connected to your computers for dial-in/dial-out

  • Paper copy of the configuration of any routers, firewalls, or other network devices not associated with a single machine

21.5.2 Per-Machine Logs

Each machine should also have a log book associated with it. Information in these logs can be divided into exception and activity reports and informational material. Exception and activity reports

These reports hold such information as the following:

  • Times and dates of any halts or crashes, including information on any special measures for system recovery

  • Times, dates, and purposes of any downtimes

  • Data associated with any unusual occurrence, such as network behavior out of the ordinary, or a disk filling up without obvious cause

  • Time and UID of any accounts created, disabled, or deleted, including the account owner, the username, and the reason for the action

  • Instances of changing passwords for users

  • Times and levels of backups and restores along with a count of how many times each backup tape has been used

  • Times, dates, and circumstances of software installation or upgrades

  • Times and circumstances of any maintenance activity Informational material

This material contains such information as the following:

  • Copy of current configuration files, including passwd, group, and inetd.conf (update these copies periodically, or as the files change)

  • List of patches applied from the vendor, software revision numbers, and other identifying information

  • Configuration information for any third-party software installed on the machine

  • ls -l listing of any setuid/setgid files on the system, and of all device files

  • Paper copy of a list of disk configurations, SCSI geometries, and partition tables and information

    Part VI: Appendixes