HTTP interception is complicated because many different devices must all work correctly together. To help you track down problems, here's a trouble-shooting check list:
This should be obvious for simple networks. You can trace the cables and watch the activity lights blink. In a large, complex network, however, packets may be taking an alternate path. If your organization is large enough to have a network sniffer, you may want to observe the traffic on the link that should carry requests from web clients. A low-tech approach is to disconnect the link in question and see if it affects the client's web browsing.
You may want to double-check your router/switch configuration. If you configured specific interfaces, did you get the right ones?
Is your new configuration actually running on the device? Perhaps the router/switch was rebooted before you could save the configuration. You may need to reboot before the changes take effect.
Can you ping Squid from the router/switch? Most layer four interception configurations require that the device and Squid be on the same subnet. Log into the router/switch, and make sure you can ping Squid's IP address.
Many traffic interception devices don't send traffic to Squid unless they know it's healthy. Use the debugging commands to view Squid's health status. You may find that a layer three health check (e.g., ICMP ping) is simpler than a layer four check (e.g., HTTP), and more likely to make the network device mark Squid as up.
Double-check that Squid is really running, especially if the system has recently been rebooted.
You should be able to see intercepted TCP connections with tcpdump. Here's an example:
# tcpdump -n -i eth0 port 80
If you use WCCP, check for GRE packets coming from the router:
# tcpdump -n -i eth0 ip proto gre
If you don't see any output from tcpdump, the router/switch is probably not sending anything. In that case, return to the previous suggestions.
Note, if the device is using layer four health checks, you should see those in the tcpdump output. Health checks come from the router/switch IP address, so they should be easy to spot. If you see health checks, but no other traffic, it probably means the router/switch is interpreting Squid's reply as unhealthy. For example, the device may want to see a 200 (OK) response, but Squid returns an error, such as 401 (Unauthorized) or 404 (Not Found). You may want to run tail -f on the access.log.
Double-check that Squid's operating system is configured to forward IP packets. If not, the host may drop intercepted packets because the destination IP address isn't local.
Make sure that the packet filter (i.e., ipfw, iptables, pf, etc.) is configured correctly. When everything is working well, you should be able to run the command periodically that displays the filtering rules and see the counters increase. For example:
# ipfw show 300 ; sleep 3; ipfw show 300 00300 86216 8480458 fwd 127.0.0.1,3128 tcp from any to any 80 in 00300 86241 8482240 fwd 127.0.0.1,3128 tcp from any to any 80 in
Note that in this example on FreeBSD, the packet and byte counters (second and third columns) are being incremented.
If you have a rule to forward/redirect packets to 127.0.0.1, make sure that the loopback (e.g., lo0, lo) interface is up and configured. If not, the kernel may simply skip the forward/redirect rule.
If you use WCCP, make sure that the GRE packets are being unencapsulated. If, for some reason, your system doesn't know what to do with GRE packets, it probably increments the "unknown/unsupported protocol" counter in netstat -s output:
# netstat -s | grep unknown 46 packets for unknown/unsupported protocol
If your OS has a GRE interface, run netstat -i every so often and look for increasing packet counts:
# netstat -in | grep ^gre0 Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll gre0 1476 <Link#4> 304452 0 0 4 0
Also, try running tcpdump on the GRE interface:
# tcpdump -n -i gre0
You may have a situation in which the router/switch is able to send packets to Squid, but Squid can't send packets back to the clients. This can happen if your firewall filter rules reject those outgoing packets or if Squid just doesn't have a route to the client addresses. To check for this condition, run netstat -n and look for a lot of sockets in the SYN_RCVD state:
% netstat -n Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 10.102.129.246.80 10.102.0.1.36260 SYN_RCVD tcp4 0 0 10.102.129.226.80 10.102.0.1.36259 SYN_RCVD tcp4 0 0 10.102.128.147.80 10.102.0.1.36258 SYN_RCVD tcp4 0 0 10.102.129.26.80 10.102.0.2.36257 SYN_RCVD tcp4 0 0 10.102.129.29.80 10.102.0.2.36255 SYN_RCVD tcp4 0 0 10.102.129.226.80 10.102.0.1.36254 SYN_RCVD tcp4 0 0 10.102.128.117.80 10.102.0.1.36253 SYN_RCVD tcp4 0 0 10.102.128.149.80 10.102.0.1.36252 SYN_RCVD
If you see this, use ping and traceroute to make sure that Squid has bidirectional communication with the clients.
Intercepted HTTP connections get stuck if Squid can't connect to origin servers. When this happens, netstat should show you a lot of connections in the SYN_SENT state:
% netstat -n Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 172.16.102.66.5217 10.102.129.145.80 SYN_SENT tcp4 0 0 172.16.102.66.5216 10.102.129.224.80 SYN_SENT tcp4 0 0 172.16.102.66.5215 10.102.128.71.80 SYN_SENT tcp4 0 0 172.16.102.66.5214 10.102.129.209.80 SYN_SENT tcp4 0 0 172.16.102.66.5213 10.102.129.62.80 SYN_SENT tcp4 0 0 172.16.102.66.5212 10.102.129.160.80 SYN_SENT tcp4 0 0 172.16.102.66.5211 10.102.128.129.80 SYN_SENT tcp4 0 0 172.16.102.66.5210 10.102.129.44.80 SYN_SENT tcp4 0 0 172.16.102.66.5209 10.102.128.73.80 SYN_SENT tcp4 0 0 172.16.102.66.5208 10.102.128.43.80 SYN_SENT
Again, use ping and traceroute to make sure that Squid can talk to origin servers.
If Squid can ping origin servers, and you still see a lot of connections in the SYN_SENT state, the router/switch may be intercepting Squid's outgoing TCP connections. In some cases, Squid can detect such forwarding loops, and it writes a warning message to cache.log. Such a forwarding loop can quickly exhaust all of Squid's file descriptors, which also generates a warning in cache.log.
If you suspect this problem, use the squidclient program to make some simple HTTP requests. For example, this command makes an HTTP request directly to the origin server:
% /usr/local/squid/bin/squidclient -p 80 -h slashdot.org /
If this command succeeds, you should see a bunch of ugly HTML from the Slashdot site on your screen. You can then try the same request through Squid:
% /usr/local/squid/bin/squidclient -r -p 3128 -h 127.0.0.1 http://slashdot.org/
Again, you should see some HTML on your screen. If not check for error messages in cache.log. If you see forwarding loop errors, you need to reconfigure your router/switch so that it allows Squid's outgoing connections to pass without being intercepted.