Advanced Logging

Nobody likes looking through the Web logs generated by IIS, or any other Web server for that matter, but being able to extract useful information from them is as important as having the site running in the first place. The better the quality of your logs, the more information you can obtain.

However, there is generally a trade-off between what can be useful to record and the impact of the additional information on the performance of the server. For example, thinking about some basic information that you might record, such as the date, time, URL accessed, and the status code, it could easily be 250 or more bytes of information. On a busy server, that could be many KB every second and could even approach MB over a period of a few hours.

IIS 6 incorporates a number of improvements in the logging system, but the main one is the new binary logging format, which can have a dramatic impact on server performance.

Log File Content Improvements

IIS 6 incorporates a few content improvements to resolve limitations in the previous versions. The most significant of these is the addition of UTF-8 support for multi-byte and extended character sets.

UTF-8 Logging

With international sites, chances are that you will be dealing with extended character sets and quite possibly multi-byte characters such as Kanji. With previous versions of IIS, log files would be written using the standard ASCII format, making it difficult to identify specific pages and URLs accessed.

IIS 6 will now write log files using the UTF-8 character set. Like binary logging though, it's a global property?it's either on for the entire IIS installation or it's off.

To set the property, right-click on the machine on which you want to enable UTF-8 logging and select Properties from the pop-up list to show the IIS computer properties as seen in Figure 4.15.

Check (or uncheck) the Encode Web Logs in UTF-8 check box. You must restart IIS in order for the changes to take effect?you can either let IIS Manager do this for you, or you can do it manually at a more suitable time.

HTTP Sub-status Codes

IIS 6 now records the sub-status code for a request, as opposed to just the main status code. For example, an authorization failure because of an ACL problem would be logged as a 401 error under IIS 5 and earlier. Within IIS 6, it would be logged as a 401.3 error. This makes it significantly easier to trace problems and identify issues raised by users.

SSL Statistics

A very small change that might affect some site statistics is that IIS 6 records the number of decrypted bytes during SSL communication, rather than the IIS 5.1 and earlier method of recording only the number of bytes transferred.

Binary Log File Format

Binary logging uses a fixed binary format for the storage or log information. There are a number of benefits to this. First, it's significantly more efficient?fields such as the date and time can be stored within a couple of bytes, compared to the 10?15 used in the W3C, IIS, or NCSA formatted logs.

This also has a domino effect on the performance?IIS buffers 64KB of log information before writing it to disk. With text-based logs, that buffer will fill more rapidly. With binary logging, it will take a lot longer, meaning fewer log writes and a lower overall overhead of the IIS service.

The downside is that we can't just open the file and view the contents; it needs to be specially parsed to extract the information from it. This also means that we can't just join log files together to produce larger logs for processing without going through the conversion process first.

Binary logging is also a global property?either all sites (including FTP and HTTP sites) are logged using the binary method, or individual sites are logged using one of the various text formats.

Enabling Binary Logging

To enable centralized binary logging, you must set a property within the Metabase?it can't be done through IIS manager. The path to the property is /LM/W3SVC/CentralBinaryLoggingEnabled, which is a simple binary value (stored in a DWORD type), where true enables binary logging.

There are a number of ways you can do this, including editing the raw XML or using the Metabase Explorer tool from the resource kit. The suggested way, though, is through the adsutil.vbs script:

cscript adsutil.vbs SET W3SVC/CentralBinaryLoggingEnabled true

Once set, you will need to restart IIS, which is easier done from here if we're already at the command line:

net stop W3SVC

net start W3SVC

Once IIS has restarted, information will be placed into the binary log file, with an extension of .ibl (so that it's not recognized as a text file by default). Unless you've otherwise changed the location of your log files, they will be placed in to the %systemroot%\System32\Logfiles\W3SVC directory.

If you want to change the directory or log renewing parameters, changing the Metabase is the easiest way to change them. Look within the /LM/W3SVC tree for properties starting with 'Log'.

Parsing Binary Log Files

You can't read binary log files directly?well, you can open them in a text editor, but you probably won't make a lot of sense out of the contents.

The format of the files is basically a chain or fixed length records generated in the form of a structure within C/C++ or Visual Basic. The MSDN site has details on the exact format of these files.

A better solution is to use one of the existing tools, such as Seagate Crystal Reports, WebTrends, or Analog. The binary format is, in fact, a recognized standard called Internet Binary Logging (hence the .ibl extension) and is readable by many of the log parsing and reporting utilities.

Converting Binary Log Files

If your chosen log processing tool doesn't support the IBL format or you want to be able to browse the contents through a standard text editor, you will need to convert the binary into a text format.

The convlog file, which has been a part of Windows for some time, doesn't support the IBL format, but the Resource Kit comes with Log Parser, which can read these and many other formats. As well as being a general purpose parsing tool that can generate summary information and search for specific patterns within a log file, Log Parser can also convert files between the different formats.

Log Parser supports two output formats when converting binary files?the W3C Extended Format and the native IIS format. To use Log Parser in this way from the command line, you must use the ?c command-line option and then specify the input format (using the ?i option) and the output format (-o). For example, to convert all the entries in a binary file to W3C format, you might use

logparser ?c ?i:BIN ?o:W3C input.ibl output.log

Because Log Parser can also understand and filter the contents, we can also apply filters to the generated text file. For example, to get all the 404 errors from a file

logparser ?c ?i:BIN ?o:W3C input.ibl output.log "StatusCode==404"

Binary logs contain all the log information for all the Web sites on a single server, but you can split it into separate, site-based log files by doing two things. First, use a wildcard in your output destination, and second, switch on the multisite option.

For example, we can change our original conversion line to

logparser ?c ?i:BIN ?o:W3C input.ibl W3SVC*\output.log ?multisite:on

Check the documentation for some additional information on how to use Log Parser for parsing and converting log files.

Centralized (Remote) Logging

You can go one stage further in the centralization process with IIS 6 by configuring individual logs to be written to the same central server through a UNC share. You can do this, for example, to introduce a single server for monitoring purposes (perhaps combining it with System Monitor or Network Monitor functionality).

There is a trade-off here, though. Using this method obviously increases your network use and will have a performance hit on your IIS servers because the time taken to write to the network is longer than that to write to a local disk.

If you are using a secondary network segment to support the 'background' network chatter between your servers and domain controllers, separate from the client-side network segment, this can go some way to alleviating the problem. But be prepared to suffer some performance hit for the benefit of centralizing your logging.

To enable logging using this method, create a shared directory onto which the log files will be kept on a server. I also recommend that you create a directory within the share for each server that you expect to log in this way. Then right-click on the Web Site's folder within IIS and choose Properties. Now click Properties within the Enable Logging section and enter the UNC location of the share you want to use in the Log File Directory area.

On the server, log files will be created within the directory you specified, with a separate directory (of the form W3SVC#, where # is the site identifier) for each site.