9.6 Debugging Problems

HTTP interception is complicated because many different devices must all work correctly together. To help you track down problems, here's a trouble-shooting check list:

Are client packets going through the router/switch?

This should be obvious for simple networks. You can trace the cables and watch the activity lights blink. In a large, complex network, however, packets may be taking an alternate path. If your organization is large enough to have a network sniffer, you may want to observe the traffic on the link that should carry requests from web clients. A low-tech approach is to disconnect the link in question and see if it affects the client's web browsing.

Is the router/switch configured properly?

You may want to double-check your router/switch configuration. If you configured specific interfaces, did you get the right ones?

Is your new configuration actually running on the device? Perhaps the router/switch was rebooted before you could save the configuration. You may need to reboot before the changes take effect.

Can the switch/router talk to the Squid host?

Can you ping Squid from the router/switch? Most layer four interception configurations require that the device and Squid be on the same subnet. Log into the router/switch, and make sure you can ping Squid's IP address.

Does the switch/router believe that Squid is up?

Many traffic interception devices don't send traffic to Squid unless they know it's healthy. Use the debugging commands to view Squid's health status. You may find that a layer three health check (e.g., ICMP ping) is simpler than a layer four check (e.g., HTTP), and more likely to make the network device mark Squid as up.

Is Squid actually running?

Double-check that Squid is really running, especially if the system has recently been rebooted.

Are packets arriving at the Squid host?

You should be able to see intercepted TCP connections with tcpdump. Here's an example:

# tcpdump -n -i eth0 port 80

If you use WCCP, check for GRE packets coming from the router:

# tcpdump -n -i eth0 ip proto gre

If you don't see any output from tcpdump, the router/switch is probably not sending anything. In that case, return to the previous suggestions.

Note, if the device is using layer four health checks, you should see those in the tcpdump output. Health checks come from the router/switch IP address, so they should be easy to spot. If you see health checks, but no other traffic, it probably means the router/switch is interpreting Squid's reply as unhealthy. For example, the device may want to see a 200 (OK) response, but Squid returns an error, such as 401 (Unauthorized) or 404 (Not Found). You may want to run tail -f on the access.log.

Did you enable IP forwarding?

Double-check that Squid's operating system is configured to forward IP packets. If not, the host may drop intercepted packets because the destination IP address isn't local.

Did you configure the packet filter?

Make sure that the packet filter (i.e., ipfw, iptables, pf, etc.) is configured correctly. When everything is working well, you should be able to run the command periodically that displays the filtering rules and see the counters increase. For example:

# ipfw show 300 ; sleep 3; ipfw show 300

00300  86216 8480458 fwd 127.0.0.1,3128 tcp from any to any 80 in

00300  86241 8482240 fwd 127.0.0.1,3128 tcp from any to any 80 in

Note that in this example on FreeBSD, the packet and byte counters (second and third columns) are being incremented.

Is the loopback interface up and configured?

If you have a rule to forward/redirect packets to 127.0.0.1, make sure that the loopback (e.g., lo0, lo) interface is up and configured. If not, the kernel may simply skip the forward/redirect rule.

Are WCCP/GRE packets being unencapsulated correctly?

If you use WCCP, make sure that the GRE packets are being unencapsulated. If, for some reason, your system doesn't know what to do with GRE packets, it probably increments the "unknown/unsupported protocol" counter in netstat -s output:

# netstat -s | grep unknown

        46 packets for unknown/unsupported protocol

If your OS has a GRE interface, run netstat -i every so often and look for increasing packet counts:

# netstat -in | grep ^gre0

Name    Mtu Network       Address              Ipkts Ierrs    Opkts Oerrs  Coll

gre0   1476 <Link#4>                          

304452     0        0     4     0

Also, try running tcpdump on the GRE interface:

# tcpdump -n -i gre0

Can Squid talk back to the clients?

You may have a situation in which the router/switch is able to send packets to Squid, but Squid can't send packets back to the clients. This can happen if your firewall filter rules reject those outgoing packets or if Squid just doesn't have a route to the client addresses. To check for this condition, run netstat -n and look for a lot of sockets in the SYN_RCVD state:

% netstat -n

Active Internet connections

Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)

tcp4       0      0  10.102.129.246.80      10.102.0.1.36260       SYN_RCVD

tcp4       0      0  10.102.129.226.80      10.102.0.1.36259       SYN_RCVD

tcp4       0      0  10.102.128.147.80      10.102.0.1.36258       SYN_RCVD

tcp4       0      0  10.102.129.26.80       10.102.0.2.36257       SYN_RCVD

tcp4       0      0  10.102.129.29.80       10.102.0.2.36255       SYN_RCVD

tcp4       0      0  10.102.129.226.80      10.102.0.1.36254       SYN_RCVD

tcp4       0      0  10.102.128.117.80      10.102.0.1.36253       SYN_RCVD

tcp4       0      0  10.102.128.149.80      10.102.0.1.36252       SYN_RCVD

If you see this, use ping and traceroute to make sure that Squid has bidirectional communication with the clients.

Can Squid talk to origin servers?

Intercepted HTTP connections get stuck if Squid can't connect to origin servers. When this happens, netstat should show you a lot of connections in the SYN_SENT state:

% netstat -n

Active Internet connections

Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)

tcp4       0      0  172.16.102.66.5217     10.102.129.145.80      SYN_SENT

tcp4       0      0  172.16.102.66.5216     10.102.129.224.80      SYN_SENT

tcp4       0      0  172.16.102.66.5215     10.102.128.71.80       SYN_SENT

tcp4       0      0  172.16.102.66.5214     10.102.129.209.80      SYN_SENT

tcp4       0      0  172.16.102.66.5213     10.102.129.62.80       SYN_SENT

tcp4       0      0  172.16.102.66.5212     10.102.129.160.80      SYN_SENT

tcp4       0      0  172.16.102.66.5211     10.102.128.129.80      SYN_SENT

tcp4       0      0  172.16.102.66.5210     10.102.129.44.80       SYN_SENT

tcp4       0      0  172.16.102.66.5209     10.102.128.73.80       SYN_SENT

tcp4       0      0  172.16.102.66.5208     10.102.128.43.80       SYN_SENT

Again, use ping and traceroute to make sure that Squid can talk to origin servers.

Are outgoing connections being intercepted?

If Squid can ping origin servers, and you still see a lot of connections in the SYN_SENT state, the router/switch may be intercepting Squid's outgoing TCP connections. In some cases, Squid can detect such forwarding loops, and it writes a warning message to cache.log. Such a forwarding loop can quickly exhaust all of Squid's file descriptors, which also generates a warning in cache.log.

If you suspect this problem, use the squidclient program to make some simple HTTP requests. For example, this command makes an HTTP request directly to the origin server:

% /usr/local/squid/bin/squidclient -p 80 -h slashdot.org /

If this command succeeds, you should see a bunch of ugly HTML from the Slashdot site on your screen. You can then try the same request through Squid:

% /usr/local/squid/bin/squidclient -r -p 3128 -h 127.0.0.1 http://slashdot.org/

Again, you should see some HTML on your screen. If not check for error messages in cache.log. If you see forwarding loop errors, you need to reconfigure your router/switch so that it allows Squid's outgoing connections to pass without being intercepted.