14.2 The Cache Manager
The Cache Manager is an interface to Squid for receiving information about various components. It is accessed via normal HTTP requests with a special protocol name: cache_object. A full cache manager URL looks like cache_object://cache.host.name/info. Squid provides two easy ways to access the cache manager information: the command-line squidclient program or the cachemgr.cgi CGI program.
The squidclient utility is a simple HTTP client, with a few special features for use with Squid. For example, you can use a shortcut to request the cache manager pages. Rather than typing a long URL like this:
% squidclient cache_object://cache.host.name/info
you can use this shorter version:
% squidclient mgr:info
squidclient is a convenient way to quickly see some of the cache manager pages. It's also useful when you need to save the cache manager output to disk for later analysis. However, some pages, such as the memory utilization table, are difficult to read in a terminal window. They are really designed to be formatted as an HTML page and viewed with your web browser. In that case, you may want to use cachemgr.cgi.
To use cachemgr.cgi, you must have an HTTP server that can execute the program. You can use an existing server or install one alongside Squid if you prefer. Keep in mind that the cache manager has only weak security (cleartext passwords). If the HTTP server is on a different host, you need to add its IP address to a cache manager access list (see Section 14.2.2). You may also want to add access controls to the HTTP server so that others can't access cachemgr.cgi.
If you use Apache, I recommend making a special cgi-bin directory so you can protect cachemgr.cgi with access controls. For example, create a new directory, and copy the binary to it:
# mkdir /usr/local/apache/squid-cgi # cp /usr/local/squid/libexec/cachemgr.cgi /usr/local/apache/squid-cgi # chmod 755 /usr/local/apache/squid-cgi/cachemgr.cgi
Now, add a ScriptAlias line to Apache's httpd.conf:
ScriptAlias /squid-cgi/ "/usr/local/apache/squid-cgi/"
Finally, create an .htaccess file in the squid-cgi directory that contains access controls. To allow requests from only one IP address, use something like this:
Allow from 192.168.4.2 Deny from all
Once cachemgr.cgi is installed, simply enter the appropriate URL into your web browser. For example:
If the CGI program is working, you should see a page with four fields. See Figure 14-1 for an example. The Cache Host field contains the name of the host on which Squid is runninglocalhost by default. You can set it with the enable-cachemgr-hostname option when running ./configure. Similarly, Cache Port contains the TCP port number to which Squid listens for requests. It's 3128 by default and can be changed with the enable-cachemgr-port option. The Manager name and Password fields are for access to protected pages, which I'll talk about shortly.
Figure 14-1. The cachemgr.cgi login screen
After clicking on the Continue... button, you should see a list of all cache manager pages currently available. The following section describes the various pages, some of which are available only when you enable certain features at compile time.
14.2.1 Cache Manager Pages
This section describes the cache manager pages, in the same order in which they appear in the menu. Each section title has both the page name (for use with squidclient), followed by its description. Descriptions that contain an asterisk indicate pages that are disabled by default, unless you configure a password for them. Table 14-1 shows the table of contents and the section number for each page.
220.127.116.11 leaks: Memory Leak Tracking
This page is available only with the ./configure enable-leakfinder option and is intended for developers trying to track down memory leaks. The page shows each memory pointer being tracked and where and when it was most recently referenced. See the Squid Programmer's Guide (http://www.squid-cache.org/Doc/Prog-Guide/) for more information about Squid's leak-finder feature.
18.104.22.168 mem: Memory Utilization
The memory utilization page shows a large table of numbers. Each row corresponds to a different pool of memory. The pools have names like acl_list and MemObject. Much of this information is of interest to developers only. However, a few columns are worth mentioning here.
The impact column shows each pool's contribution to the total amount of memory allocated. Usually, the StoreEntry, MD5 digest, and LRU policy node pools take up most of the memory.
If you are a developer, you can use this page to look for memory leaks. The column labeled high (hrs) shows the amount of time elapsed since the pool reached its maximum size. A small value in this column may indicate that memory for that pool isn't being freed correctly.
You can also use this page to find out if certain features, such as netdb, the ipcache, and client_db consume too much memory. For example, the ClientInfo pool is associated with the client_db feature. The memory utilization page shows you how much memory you can save if you disable client_db in squid.conf.
22.214.171.124 cbdata: Callback Data Registry Contents
The Callback Data Registry is an internal Squid programming feature for managing memory pointers. Currently, this cache manager page doesn't provide much useful information, apart from the number of active cbdata pointers being tracked. In earlier Squid versions, the cbdata feature was implemented differently and this page provided some information to developers debugging their code.
126.96.36.199 events: Event Queue
Squid maintains an event queue for a number of tasks that must occur separately from user requests. Perhaps the most important of these is the periodic task that maintains the disk cache size. Every second or so, this task runs and looks for cache files to remove. On this page, you can see all tasks currently scheduled for execution. Most likely, you'll not find this very interesting unless you are hacking the source code.
188.8.131.52 squidaio_counts: Async IO Function Counters
This page is available only with the ./configure enable-storeio=aufs option. It shows counters for the number of open, close, read, write, stat, and unlink requests received. For example:
ASYNC IO Counters: Operation # Requests open 15318822 close 15318813 cancel 15318813 write 0 read 19237139 stat 0 unlink 2484325 check_callback 311678364 queue 0
The cancel counter is normally equal to the close counter. This is because the close function always calls the cancel function to ensure that any pending I/O operations are ignored.
The write counter is zero because this version of Squid performs writes synchronously, even for aufs.
The check_callback counter shows how many times the main Squid process has checked the done queue for completed operations.
The queue value indicates the current length of the request queue. Normally, the queue length should be less than the number of threads x 5. If you repeatedly observe a queue length larger than this, you may be pushing Squid too hard. Adding more threads may help, but only to a certain point.
184.108.40.206 diskd: DISKD Stats
This page is available only with the ./configure enable-storeio=diskd option. It provides various statistics relating to the diskd storage scheme.
The sent_count and recv_count lines are counters for the number of I/O requests sent between Squid and the group of diskd processes. The two numbers should be very close to each other and could possibly be equal. The difference indicates how many requests are currently outstanding.
The max_away value indicates the largest number of outstanding requests. Similarly, the max_shmuse counter indicates the maximum number of shared memory blocks in use at once. These two values are reset (to zero) each time you request this page. Thus, if you wait longer between requests for this page, these maximum counters are likely to be larger.
The open_fail_queue_len counter indicates the number of times that the diskd code decided to return failure in response to a request to open a file because the message queue exceeded its configured limit. In other words, this is the number of times a diskd queue reached the Q1 limit. Similarly, block_queue_len shows how many times the Q2 limit has been reached. See the descriptions of Q1 and Q2 in Section 8.5.1.
The diskd page also shows how many requests Squid sent to the diskd processes for each of the six I/O operations: open, create, close, unlink, read, and write. It also shows how many times each operation succeeded or failed. Note, these counters are incremented only for requests sent. The open_fail_queue_len check occurs earlier, and in that case, Squid doesn't send a request to a diskd process.
220.127.116.11 config: Current Squid Configuration*
This option dumps Squid's current configuration in the squid.conf format. Thus, if you ever accidentally remove the configuration file, you can recover it from the running Squid process. By saving the output to a file, you can also compare (e.g., with the diff command) the running configuration to the saved configuration. Note, however, that comments and blank lines aren't preserved.
This option reveals potentially sensitive information, so it's available only with a password. You must add a cache manager password for the config option with the cachemgr_passwd directive. See Section 14.2.2 for specifics. Additionally, these cache manager passwords aren't displayed in this output.
18.104.22.168 comm_incoming: comm_incoming( ) Stats
This page provides low-level network I/O information to developers and Squid wizards. The loop that checks for activity on file descriptors is called comm_poll( ). Over the years, this function has become increasingly complicated in order to improve Squid's performance. One of those performance improvements relates to how often Squid checks certain network sockets relative to the others.
For example, the incoming HTTP socket is where Squid accepts new client connections. This socket tends to be busier than a normal data socket because each new connection comes through the incoming socket. To provide good performance, Squid makes an extra effort to check the incoming socket more frequently than the others.
At the top of the comm_incoming page, you'll see three incoming interval numbers: one each for ICP, DNS, and HTTP. The interval is the number of normal I/O events that Squid handles before checking the incoming socket again. For example, if incoming_dns_interval is set to 140, Squid checks the incoming DNS socket after 140 I/Os on normal connections. Unless your Squid is very busy, you'll probably see 256 for all incoming intervals.
The page also contains three histograms that show how many events occur for each incoming function call. Normally, the majority of the histogram counts occur in the low values. In other words, functions such as comm_select_http_incoming( ) usually handle between one and four events.
22.214.171.124 ipcache: IP Cache Stats and Contents
The IP cache contains cached results of hostname-to-address lookups. This cache manager page displays quite a lot of information. At the top of this page you'll see a handful of statistics like these:
IPcache Entries: 10034 IPcache Requests: 1066445 IPcache Hits: 817880 IPcache Negative Hits: 6846 IPcache Misses: 200497
In this example, you can see that the IP cache contains slightly more than 10,000 entries (hostnames). Since Squid was started, there have been 1,066,445 name-to-address requests, 817,880 of which were cache hits. This is a cache hit ratio of 77%. An IP cache negative hit occurs when Squid receives a subsequent request for a hostname that it recently failed to resolve. Rather than retry the DNS lookup immediately, Squid assumes it will fail again and returns an error message to the user.
Following these brief statistics, you'll see a long list of the IP cache contents. For each hostname in the cache, Squid prints six fields:
Here is a short sample (formatted to fit the page):
Hostname Flg lstref TTL N ads.x10.com 9 110 1( 0) 126.96.36.199-OK us.rd.yahoo.com 640 -340 4( 0) 188.8.131.52-OK 184.108.40.206-OK 220.127.116.11-OK 18.104.22.168-OK www.movielodge.com 7143 -2161 1( 0) 22.214.171.124-OK shell.windows.com 10865 -7447 2( 1) 126.96.36.199-BAD 188.8.131.52-OK www.surf3.net 126810 -40415 1( 0) 184.108.40.206-OK
The list is sorted by the time since last reference. Recently referenced names are at the top of the list, and unused (about to be removed) names are at the bottom.
IP addresses are marked OK by default. An address is marked BAD when Squid receives an error or timeout during a TCP connection attempt. Subsequent IP cache requests don't return BAD addresses. If all the host's addresses become BAD, Squid resets them all back to OK.
220.127.116.11 fqdncache: FQDN Cache Stats and Contents
The FQDN cache is similar to the IP cache, except that it stores address-to-hostname lookups. Another difference is that the FQDN cache doesn't mark hostnames as OK or BAD.
Your FQDN cache may be empty, unless you enable the log_fqdn directive, use domain-based ACLs (such as srcdomain, dstdomain, srcdom_regex, and dstdom_regex), or use a redirector.
18.104.22.168 idns: Internal DNS Statistics
Squid contains an internal DNS client implementation, which is enabled by default. Disabling internal DNS with the disable-internal-dns option also disables this page. Here is some sample output:
Internal DNS Statistics: The Queue: DELAY SINCE ID SIZE SENDS FIRST SEND LAST SEND ------ ---- ----- ---------- --------- 001876 44 1 0.010 0.010 001875 44 1 0.010 0.010 Nameservers: IP ADDRESS # QUERIES # REPLIES --------------- --------- --------- 192.168.19.124 4889 4844 192.168.19.190 91 51 192.168.10.2 73 39 Rcode Matrix: RCODE ATTEMPT1 ATTEMPT2 ATTEMPT3 0 6149 4 2 1 0 0 0 2 38 34 32 3 0 0 0 4 0 0 0 5 0 0 0
The Internal DNS page contains three tables. First, you'll see the queue of unanswered queries. Unfortunately, you can't see the contents of the query (the hostname or IP address). Instead, Squid prints the ID, size, number of transmissions, and elapsed times for each query. You should see relatively few queries in the queue. If you see a lot relative to your total traffic rate, make sure your DNS servers are functioning properly.
The second table (Nameservers) shows how many queries have been sent to, and replies received from, each DNS server. Squid always queries the first server in the list first. Second (and third, etc.) servers are queried only when the previous server times out for a given query. If you see zero replies from the first address, make sure a server is actually running at that address.
Finally, you'll see a table of DNS response codes versus number of attempts. The cell for response code 0 and ATTEMPT1 should have the highest count. Response code 0 indicates success, while others are different types of errors (see RFC 1035 for their descriptions). You may see some smaller numbers for response code 0 in the columns for ATTEMPT2 and ATTEMPT3. This shows the cases when retransmitting a query, after initially receiving an error, resulted in a successful reply. Note that Squid retries only response code 2 (server failure) errors.
22.214.171.124 dns: Dnsserver Statistics
This cache manager page is available only when you use the disable-internal-dns option. In this case, Squid uses a number of external dnsserver processes to perform DNS lookups. The dnsserver program is one of a number of helper processes Squid can use. The other types of helpers are redirectors, authenticators, and external ACLs. All Squid's helpers have cache manager pages that display the same statistics. For example:
Dnsserver Statistics: number running: 5 of 5 requests sent: 3001 replies received: 3001 queue length: 0 avg service time: 23.10 msec # FD PID # Requests Flags Time Offset Request 1 6 20110 128 AB 0.293 0 www.nlanr.net 2 7 20111 45 A 0.000 0 (none) 3 8 20112 4 A 0.000 0 (none) 4 9 20113 0 A 0.000 0 (none) 5 10 20114 0 A 0.000 0 (none)
The number running line shows how many helper processes are running and how many should be running. The dns_children directive specifies how many dnsserver processes to use. The two numbers should match, but they may not if a helper process dies unexpectedly or if some processes could not be started. Recall that when you reconfigure a running Squid instance, all the helpers are killed and restarted. See the discussion in Appendix A.
The requests sent and replies received values display the number of requests sent to (and responses received from) the helpers since Squid started. The difference between these two, if any, should correspond to the number of outstanding requests.
The queue length line shows how many requests are queued, waiting for one of the helpers to become free. The queue length should usually be zero. If not, you should add more helpers to reduce delays for your users.
The avg service time line shows the running average service time for all helpers. Your particular value may depend on numerous factors, such as your network bandwidth and processing power.
The next section displays a table of statistics for the running dnsserver processes. The FD column shows the file descriptor for the socket between Squid and each dnsserver process. Similarly, the PID column shows each helper's process ID number.
The # Requests column shows how many requests have been sent to each helper. These numbers are zeroed each time you reconfigure Squid, so they many not add up to the total number of requests sent, as shown earlier. Note that Squid always chooses the first idle helper in the list, so the first process should receive the largest number of requests. The last few processes may not receive any requests at all.
The Flags column shows a few flags describing the state of the helper process. You should normally see A (for Alive) in each column. Occasionally, when the helper process is handling a request, you'll see B (for Busy).
The Time column displays the amount of time elapsed (in seconds) for the current, or last, request. Offset shows how many bytes of the response message Squid has read on the socket. This is almost always zero. Finally, the Request column shows the request that was sent to the helper process. In this case, it is either a hostname or an IP address.
126.96.36.199 redirector: URL Redirector Stats
The Redirector Stats page is available only if you are using a redirector (see Chapter 11). The format of this page is identical to Dnsserver Statistics, described earlier.
188.8.131.52 basicauthenticator: Basic User Authenticator Stats
This page is available only with the ./configure enable-auth=basic option and when you define a Basic authenticator with the auth_param basic program directive. The format of this page is identical to Dnsserver Statistics, described earlier.
184.108.40.206 digestauthenticator: Digest User Authenticator Stats
This page is available only with the ./configure enable-auth=digest option and when you define a Digest authenticator with the auth_param digest program directive. The format of this page is identical to Dnsserver Statistics, described earlier.
220.127.116.11 ntlmauthenticator: NTLM User Authenticator Stats
This page is available only with the ./configure enable-auth=ntlm option and when you define a NTLM authenticator with then auth_param ntlm program directive. The format of this page is similar to Dnsserver Statistics, described earlier, with a few additions.
The table of helper processes includes an extra column: # Deferred Requests. NTLM requires "stateful" helpers because the helper processes themselves generate the challenges. Squid receives a challenge from a helper, sends that challenge to a user, and receives a response. Squid must send the user's challenge response back to the same helper for validation. For this protocol to work, Squid must defer some messages to be sent to a helper until the helper is ready to accept them.
These helpers also have two new flags: R (reserved or deferred) and P (placeholder). The R flag is set when the helper has at least one deferred request waiting. The P flag is set when Squid is waiting for the NTLM helper to generate a new challenge token.
18.104.22.168 external_acl: External ACL Stats
This page displays helper statistics for your external ACLs. If you don't have any external_acl_type lines in squid.conf, this page will be empty. Otherwise, Squid displays the statistics for each external ACL. The format is the same as for the Dnsserver Statistics.
22.214.171.124 http_headers: HTTP Header Statistics
This page displays a number of tables containing statistics about HTTP headers. It contains up to four sections: HTCP reply stats (if HTCP is enabled), HTTP request stats, HTTP reply stats, and a final section called HTTP Fields Stats. The HTCP reply statistics refer to HTCP replies received by your cache. The HTTP request section refers to HTTP requests either sent or received by your cache. Similarly, the HTTP reply section refers to replies either sent or received by Squid.
The first three sections have the same format. Each section contains three tables: Field type distribution, Cache-control directives distribution, and Number of fields per header distribution.
The Field type distribution table shows the number of times that each header value occurs and the percentage of cases in which it occurs. For example, in Table 14-2 you can see that the Accept header occurs in 98% of HTTP requests.
Unfortunately, these (and the following) statistics are tricky because they don't correspond one-to-one for client requests. For example, Squid may report 1,416,268 Accept headers in requests but only 800,542 client requests. This happens because Squid creates more than one HTTP header data structure for each request. In the case of HTTP replies, it seems that Squid may create up to four separate header structures, depending on the circumstances.
The Cache-Control directives distribution is similar, but applies only to the values of the Cache-Control header. Table 14-3 shows some of the possible field values.
The Number of fields per header distribution table shows how many headers occur in each request or reply. Usually, you should see something like a normal distribution with a peak around 10-13 headers per request or response.
Finally, this page ends with a table labeled Http Fields Stats (replies and requests). For each header, this table shows three values: #alive, %err, and %repeat.
The #alive column shows how many instances of this header are currently stored in memory. HTTP headers are kept in memory for both active requests/responses and for completed objects stored in the memory cache.
The %err column shows the percentage of times Squid encountered an error while parsing this header. Common errors include incorrect date formats for Date, Expires, Last-Modified, and similar headers. The value -1 indicates no errors.
The %repeat column indicates the number of times that a particular header is repeated in a single request or response. These aren't errors because HTTP allows headers to be repeated.
126.96.36.199 via_headers: Via Request Headers
This page is available only with the ./configure enable-forw-via-db option. The information in this page is intended to help cache administrators understand where client requests come from. When enabled, Squid counts the number of times each unique Via header occurs in client requests.
The Via header contains a list of downstream proxies that have forwarded the request so far. When a proxy forwards a request, it should append its hostname and other identifying information to the Via header. With the information in this database, you can, in theory, reconstruct the hierarchy of proxies forwarding requests through yours.
Squid prints the Via database entries in a random order. The output may look something like this:
4 1.0 proxy.firekitten.org:3128 (squid/2.5.STABLE1) 1 1.0 xnsproxy.dyndns.org:3128 (squid/2.5.PRE3-20020125) 1751 1.0 nt04.rmtcc.cc.oh.us:3128 (Squid/2.4.STABLE6), 1.0 tasksmart.rmtcc.cc.oh.us:3128 (Squid/2.4.STABLE7) 137 1.0 reg3.bdg.telco.co.id:8080 (Squid/2.2.STABLE5), 1.0 c1.telco.co.id:8080 (Squid/2.4.STABLE6), 1.0 cache2.telco.co.id:8080 (Squid/2.4.STABLE1) 53 1.0 IS_GW_312:3128 (Squid/2.4.STABLE6) 60 1.0 proxy.kiltron.net:3128 (Squid/2.4.STABLE7) 815 1.1 DORM
In this example, Squid received 1751 requests that previously passed through two other proxies (nt04 and tasksmart). Note that only proxies add a Via header. Requests from user-agents usually don't have the header and, therefore, aren't counted in this database.
As you can see, the Via headers reveal some semiprivate information, such as hostnames, port numbers, and software versions. Please take care to respect the privacy of your users if you enable this feature.
The Via database is stored entirely in memory and is lost if Squid restarts. The database is cleared whenever you rotate the log files (see Section 13.7).
188.8.131.52 forw_headers: X-Forwarded-For Request Headers
This page is available only with the ./configure enable-forw-via-db option. It is similar to the via_headers page, except that it displays the accumulation of X-Forwarded-For headers.
X-Forwarded-For is a nonstandard HTTP header that originated with the Squid project. Its value is a list of client IP addresses. In other words, when Squid receives and forwards a request, it appends the client's IP address to this header. It is similar to Via because the header grows each time a proxy passes the request on towards the origin server.
The forw_headers output is similar to via_headers. Each line begins with an integer, followed by a header value. The integer indicates how many times that particular X-Forwarded-For value was received. For example:
1 10.37.1.56, 10.1.83.8 3 10.3.33.77, 10.1.83.8 569 184.108.40.206 21 10.65.18.200, 10.1.83.120 31 220.127.116.11 5 10.1.92.7, 10.1.83.120 1 10.3.65.122, 10.3.1.201, 10.1.83.8 2 10.73.73.51, 10.1.83.120 1 10.1.68.141, 10.1.83.8 3 10.1.92.7, 10.1.83.122
As with via_headers, this database is also stored in memory and is lost if Squid exits. The database is cleared each time you rotate Squid's log files.
18.104.22.168 menu: This Cache Manager Menu
This page simply displays a listing of the other cache manager pages. You can use it if you forget the name of a page or if you want to know if certain optional pages are available. When using cachemgr.cgi, each item in the menu is a clickable link.
22.214.171.124 shutdown: Shut Down the Squid Process*
This is one of the few cache manager functions that doesn't simply display some information. Rather, this "page" allows you to shut down Squid remotely. To allow shutdown via the cache manager, you must assign it a password with the cachemgr_passwd (see Section 14.2.2) directive in squid.conf. Without a password, the shutdown operation is disabled (but you can still use squid -k shutdown).
Because the cache manager has very weak securitypasswords are sent in cleartextI don't recommend enabling this operation.
126.96.36.199 offline_toggle: Toggle offline_mode Setting*
This is another function that allows you to control Squid, rather than simply receive information. It also requires a password (see Section 14.2.2) in order to become active.
Each time you request this page, Squid toggles the offline_mode setting. Squid reports the new setting on your screen and in cache.log.
188.8.131.52 info: General Runtime Information
This page provides a lot of basic information about the way that Squid is operating. It is a good starting point for using the cache manager and for tracking down performance problems.
At the top, you'll see the release version (e.g., Version 2.5.STABLE4) and two timestamps: the starting and current times. For example:
Squid Object Cache: Version 2.5.STABLE4 Start Time: Mon, 22 Sep 2003 03:10:37 GMT Current Time: Mon, 13 Oct 2003 10:25:16 GMT
Following that, you'll see seven different sections. The first section, Connection information, displays a few statistics about the number and rate of connections, and the number of cache clients:
Connection information for squid: Number of clients accessing cache: 386 Number of HTTP requests received: 12997469 Number of ICP messages received: 16302149 Number of ICP messages sent: 16310714 Number of queued ICP replies: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 423.7 Average ICP messages per minute since start: 1063.2 Select loop called: 400027445 times, 4.601 ms avg
The Cache information section displays hit ratio and cache size statistics:
Cache information for squid: Request Hit Ratios: 5min: 22.6%, 60min: 25.8% Byte Hit Ratios: 5min: 24.6%, 60min: 38.7% Request Memory Hit Ratios: 5min: 0.7%, 60min: 1.4% Request Disk Hit Ratios: 5min: 6.0%, 60min: 12.4% Storage Swap size: 41457489 KB Storage Mem size: 10180 KB Mean Object Size: 14.43 KB Requests given to unlinkd: 0
The Median Service Times section displays the median of various service time (or response time) distributions. You'll see a value for the last five minutes and for the last hour. All values are in seconds. Squid uses the median, rather than the mean, because these distributions often have heavy tails that can significantly skew the mean value. The output looks like this:
Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.19742 0.15048 Cache Misses: 0.22004 0.17711 Cache Hits: 0.05951 0.04047 Near Hits: 0.37825 0.14252 Not-Modified Replies: 0.01309 0.01387 DNS Lookups: 0.05078 0.03223 ICP Queries: 0.00000 0.07487