16.1 Some Common Problems

Before discussing debugging in general, I'll mention a few specific problems that commonly arise.

16.1.1 "Failed to make swap directory"

Failed to make swap directory /var/spool/cache: (13) Permission denied

This happens when you run squid -z, and the Squid user ID doesn't have write permission to the /var/spool directory. Remember that if you start Squid as root and don't add a cache_effective_user line, Squid runs as the user nobody by default. Thus, your solution may be to simply run:

# chown nobody:nobody /var/spool

16.1.2 "Address already in use"

commBind: Cannot bind socket FD 10 to *:3128: Address already in use

This message appears when the bind( ) system call fails because the requested port is already opened by another application. Usually, this happens when you try to start a second instance of Squid when the first one is still running. If you see this error message, use ps to see if Squid is already running.

Squid uses the SO_REUSEADDR socket option, so that the bind( ) call should succeed even if there are some leftover sockets in the TIME_WAIT state. If you get the message, even though Squid isn't already running, your operating system may be buggy or especially finicky. Rebooting your system is one way to get around this problem.

Another possibility to consider is that the port (e.g., 3128) is currently being used by a different application. If you suspect this, you can use the lsof program (ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof) to find which application is listening on the port. FreeBSD users can use sockstat instead.

16.1.3 "Could not determine fully qualified hostname"

FATAL: Could not determine fully qualified hostname.  Please set 'visible_hostname'

You'll see this if Squid can't figure out its own fully qualified domain name. Here is the algorithm Squid uses:

  • If you told Squid to bind the HTTP port to a specific interface address, Squid attempts a reverse DNS lookup of that address. If successful, the answer is used.

  • Squid calls the gethostname( ) function, and then attempts to resolve its IP address with gethostbyname( ). If successful, Squid uses the official hostname string returned by the latter function.

If neither technique works, Squid exits with the fatal message shown earlier. In this case, you must tell Squid its hostname with the visible_hostname directive. For example:

visible_hostname my.host.name

16.1.4 "DNS name lookup tests failed"

By default, Squid makes a few DNS queries before starting. This ensures that your DNS servers are reachable and functioning properly. If these tests fail, you'll see the following message in cache.log and/or syslog:

FATAL: ipcache_init: DNS name lookup tests failed

If you use Squid on an intranet, Squid may be unable to query its standard list of hostnames. You can specify your own hostnames with the dns_testnames directive. Squid considers the DNS test successful as soon as it receives any reply.

If you want to skip the DNS tests altogether, simply use the -D command-line option when starting Squid:

% squid -D ...

16.1.5 "Illegal character in hostname"

urlParse: Illegal character in hostname 'super_bikes.tripod.com'

By default, Squid checks the characters in the hostname part of URLs and complains if it finds nonstandard characters. According to RFCs 1034 and 1035, names must consist of the letters A-Z, the digits 0-9, and a hyphen (-). The underscore (_) is one of the most problematic characters.

Squid validates hostnames because, in some cases, DNS resolvers behave differently with respect to illegal characters. For example:

% host super_bikes.tripod.com

super_bikes.tripod.com has address

% ping super_bikes.tripod.com

ping: cannot resolve super_bikes.tripod.com: Unknown server error

Rather than return the Unknown server error message, Squid checks the hostname first. It can then tell the user when the hostname contains illegal characters.

Some DNS resolvers do work with underscores and other nonstandard characters. If you'd prefer that Squid not check hostnames, use the disable-hostname-checks option when running ./configure. If you want to allow underscores as the only exception, use the enable-underscores option.

16.1.6 "Running out of filedescriptors"

WARNING! Your cache is running out of filedescriptors

The above message appears when Squid uses up all available file descriptors. If this happens under normal conditions, you need to increase the kernel's file-descriptor limits and recompile Squid. See Section 3.3.1.

You might also see this message if Squid is the target of a denial-of-service attack. Someone may be intentionally, or unintentionally, sending Squid hundreds or thousands of requests at once. If this is the case, you can probably add a packet-filtering rule to block incoming TCP connections from the offending address(es). If the attack is distributed or using a spoofed source address, you'll have a harder time stopping it.

Forwarding loops (see Section 10.2) might also consume all of Squid's file descriptors, but only if Squid can't detect the loop. The Via header contains the hostname of all proxies that have seen a particular request. Squid looks for its own hostname in the header, and, if found, reports the loop. If, for some reason, the Via header is filtered from outgoing or incoming HTTP requests, Squid can't detect the loop. In this case, all file descriptors are quickly consumed by the same request going through Squid over and over.

16.1.7 "icmpRecv: Connection refused"

You'll see the following message if the pinger program isn't correctly installed:

icmpRecv: recv: (61) Connection refused

Most likely, the pinger program exits immediately because it doesn't have permission to open a raw ICMP socket. Because the process isn't running, Squid receives an I/O error when trying to talk to it. To alleviate the problem, go to the source directory and, as root, type:

# make install-pinger

If successful, you should find that the pinger program has the following file ownership and permission settings:

# ls -l /usr/local/squid/libexec/pinger

-rws--x--x  1 root  squid  140728 Sep 16 19:58 /usr/local/squid/libexec/pinger

16.1.8 Squid Becomes Slow After Running for Some Time

Most likely, Squid is competing with other processes, or with itself, for memory on your system. When the Squid process no longer fits entirely in memory, the operating system is forced to read and write areas of memory to and from the swap space. This has a drastic effect on Squid's performance.

To validate this theory, check the Squid process size with utilities such as top and ps. Also check Squid's own page fault counter, as described in Section Once you've identified memory consumption as the problem, try the following steps to reduce Squid's memory usage:

  1. Reduce the value of cache_mem and read Appendix B.

  2. Turn off memory pooling with this option:

    memory_pools off
  3. Reduce the size of the disk cache by lowering the size of one or more cache directories. See Section 7.1.

16.1.9 Debugging Access Controls

If you're having no luck getting your access controls to work properly, here's a little tip that might help. Edit your squid.conf file and set the debug_options line to this:

debug_options ALL,1 33,2

Then, reconfigure Squid:

% squid -k reconfigure

Now, Squid writes a message to cache.log for each client request and another for each reply. The messages contain the request method, URI, whether the request/reply is allowed or denied, and the name of the last ACL that matched it. For example:

2003/09/29 20:22:05| The request

    GET http://images.slashdot.org:80/topics/topicprivacy.gif is ALLOWED,

    because it matched 'localhost'

2003/09/29 20:22:05| The reply for

    GET http://images.slashdot.org/topics/topicprivacy.gif is ALLOWED,

    because it matched 'all'

Knowing the name of the ACL doesn't always tell you the corresponding http_access line, but it gets you pretty close. If necessary, you can replicate your acl lines and give them unique names so that a given ACL name appears on only one http_access rule.

    Appendix A. Config File Reference