Section 13.5. Checking Routing
The "network unreachable" error message clearly indicates a routing problem. If the problem is in the local host's routing table, it is easy to detect and resolve. First, use netstat -nr and grep to see whether or not a valid route to your destination is installed in the routing table.[5]
[5] netstat -nr works on most systems, but Linux administrators prefer route -n.
This example checks for a specific route to network 128.8.0.0:
% netstat -nr | grep '^128\.8\.'
128.8.0.0 26.20.0.16 UG 0 37 dnet0
This same test, run on a system that did not have this route in its routing table, would return no response at all. For example, a user reports that the "network is down" because he cannot ftp to helios.metalab.unc.edu, and a ping test returns the following results:
% ping -s helios.metalab.unc.edu 56 2
PING helios.metalab.unc.edu: 56 data bytes
sendto: Network is unreachable
ping: wrote helios.metalab.unc.edu 64 chars, ret=-1
sendto: Network is unreachable
ping: wrote helios.metalab.unc.edu 64 chars, ret=-1
----helios.metalab.unc.edu PING Statistics----
2 packets transmitted, 0 packets received, 100% packet loss
Based on the "network unreachable" error message, check the user's routing table. In our example, we're looking for a route to helios.metalab.unc.edu. The IP address[6] of helios.metalab.unc.edu is 152.2.210.81. So we check for any route to a destination that begins with 152.2:
[6] Use nslookup to find the IP address if you don't know it. nslookup is discussed later in this chapter.
% netstat -nr | grep '^152\.2\.'
%
This test shows that there is no specific route to a destination that begins with 152.2. If a route was found, grep would display it. Since there's no specific route to the destination, remember to look for a default route. This example shows a successful check for a default route on a Solaris system:[7]
[7] On a Linux system, grep for network 0.0.0.0, which Linux uses instead of the word "default" to indicate the default route.
% netstat -nr | grep def
default 172.16.12.1 UG 0 101277 dnet0
If netstat shows the correct specific route or a valid default route, the problem is not in the routing table. In that case, use traceroute, as described in the next section, to trace the route all the way to its destination.
If the routing table doesn't contain the expected route, it's a local routing problem. There are two ways to approach local routing problems, depending on whether the system uses static or dynamic routing. If you're using static routing, install the missing route using the route add command. Remember, most systems that use static routing rely on a default route, so the missing route could be the default route. Make sure that the startup files add the needed route to the table whenever the system reboots. See Chapter 7 for details about the route add command.
If you're using dynamic routing, make sure that the routing program is running. For example, the command below makes sure that gated is running:
% ps 'cat /etc/gated.pid'
PID TT STAT TIME COMMAND
27711 ? S 304:59 gated -tep /etc/log/gated.log
If the correct routing daemon is not running, restart it and specify tracing. Tracing allows you to check for problems that might be causing the daemon to terminate abnormally.
13.5.1 Tracing Routes
If the local routing table is correct, the problem may be occurring some distance away from the local host. Remote routing problems can cause the "no answer" error message, as well as the "network unreachable" error message. But the "network unreachable" message does not always signify a routing problem. It can mean that the remote network cannot be reached because something is down between the local host and the remote destination. traceroute is the program that can help you locate these problems.
traceroute traces the route of UDP packets from the local host to a remote host. It prints the name (if it can be determined) and IP address of each gateway along the route to the remote host.
traceroute uses two techniques, small TTL (time-to-live) values and an invalid port number, to trace packets to their destination. traceroute sends out UDP packets with small TTL values to detect the intermediate gateways. The TTL values start at 1 and increase in increments of 1 for each group of three UDP packets sent. When a gateway receives a packet, it decrements the TTL. If the TTL is then 0, the packet is not forwarded and an ICMP "Time Exceeded" message is returned to the source of the packet. traceroute displays one line of output for each gateway from which it receives a "Time Exceeded" message. Figure 13-2 presents a sample of the single line of output that is displayed for a gateway, and shows the meaning of each field in the line.
Figure 13-2. traceroute output
When the destination host receives a packet from traceroute, it returns an ICMP "Unreachable Port" message. This happens because traceroute intentionally uses an invalid port number (33434) to force this error. When traceroute receives the "Unreachable Port" message, it knows that it has reached the destination host, and it terminates the trace. So, traceroute is able to develop a list of the gateways, starting at one hop away and increasing one hop at a time until the remote host is reached. Figure 13-3 illustrates the flow of packets tracing to a host three hops away. The following shows a traceroute to www.internic.net from a Solaris system hanging off the Comcast network. traceroute sends out three packets at each TTL value. If no response is received to a packet, traceroute prints an asterisk (*). If a response is received, traceroute displays the name and address of the gateway that responded and the packet's round trip time in milliseconds.
Figure 13-3. Flow of traceroute packets
$ traceroute www.internic.net
traceroute to www.internic.net (207.151.159.3), 30 hops max, 40 byte packets
1 ani (192.168.0.1) 1.712 ms 1.40 ms 1.34 ms
2 10.81.130.1 (10.81.130.1) 52.01 ms 34.38 ms 118.97 ms
3 bb1-fe1-0.mtgmry1.md.home.net (24.11.248.1) 13.30 ms 100.92 ms 31.99 ms
4 c2-se9-0-10.washdc1.home.net (24.7.73.25) 118.63 ms 94.92 ms 121.10 ms
5 24.7.71.6 (24.7.71.6) 127.63 ms 26.29 ms 132.07 ms
6 p4-6-1-0.r00.plalca01.us.bb.verio.net (129.250.2.245) 186.02 ms 164.81 ms 156.44 ms
7 p16-0-0-0.r06.plalca01.us.bb.verio.net (129.250.2.161) 86.59 ms 130.28 ms 121.09 ms
8 p16-0-0-0.r04.snjsca03.us.bb.verio.net (129.250.3.162) 84.594 ms 117.42 ms 174.59 ms
9 p16-3-0-0.r01.snjsca03.us.bb.verio.net (129.250.2.63) 123.87 ms 91.39 ms 119.79 ms
10 p4-2-0-0.r00.lsanca01.us.bb.verio.net (129.250.2.26) 142.38 ms 166.11 ms 95.32 ms
11 ge-0-0-0.a02.lsanca02.us.ra.verio.net (129.250.29.116) 137.59 ms 98.28 ms 256.11 ms
12 uscisi-pl.customer.ni.net (209.189.66.66) 98.64 ms 125.03 ms 231.11 ms
13 207.151.151.2 (207.151.151.2) 192.06 ms 164.52 ms 103.30 ms
14 icann-IWC.interworld.net (206.124.230.170) 113.33 ms 145.72 ms 107.39 ms
15 * host159-3.icann.org (207.151.159.3) 99.67 ms 178.72 ms
This trace shows that 15 intermediate gateways are involved, that packets are making the trip, and that round trip travel time for packets from this host to www.internic.net is about 140 ms.
Variations and bugs in the implementation of ICMP on different types of gateways, as well as the unpredictable nature of the path a datagram can take through a network, can cause some odd displays. For this reason, you shouldn't examine the output of traceroute too closely. The most important things in the traceroute output are:
Did the packet get to its remote destination?
If not, where did it stop?
In the code below we show another trace of the path to www.internic.net. This time the trace does not go all the way through to the InterNIC.
$ traceroute www.internic.net
traceroute to www.internic.net (207.151.159.3), 30 hops max, 40 byte packets
1 ani (192.168.0.1) 1.712 ms 1.40 ms 1.34 ms
2 10.81.130.1 (10.81.130.1) 52.01 ms 34.38 ms 118.97 ms
3 bb1-fe1-0.mtgmry1.md.home.net (24.11.248.1) 13.30 ms 100.92 ms 31.99 ms
4 c2-se9-0-10.washdc1.home.net (24.7.73.25) 118.63 ms 94.92 ms 121.10 ms
5 24.7.71.6 (24.7.71.6) 127.63 ms 26.29 ms 132.07 ms
6 p4-6-1-0.r00.plalca01.us.bb.verio.net (129.250.2.245) 186.02 ms 164.81 ms 156.44 ms
7 p16-0-0-0.r06.plalca01.us.bb.verio.net (129.250.2.161) 86.59 ms 130.28 ms 121.09 ms
8 p16-0-0-0.r04.snjsca03.us.bb.verio.net (129.250.3.162) 84.594 ms 117.42 ms 174.59 ms
9 * * *
10 * * *
.
.
.
29 * * *
30 * * *
When traceroute fails to get packets through to the remote end system, the trace trails off, displaying a series of three asterisks at each hop count until the count reaches 30. If this happens, contact the administrator of the remote host you're trying to reach, and the administrator of the last gateway displayed in the trace. Describe the problem to them; they may be able to help. In our example, the last gateway that responded to our packets was p16-0-0-0.r04.snjsca03.us.bb.verio.net. We would therefore contact this system administrator and the administrator of www.internic.net.
13.5.2 Locating an Administrator
To contact a remote administrator, you must know who to contact. whois helps you locate important people. One of the most important pieces of information in a network is who is in charge at the other end. When troubleshooting a network problem, whois is a tool that helps you find this out.
whois obtains the requested information from the Internet white pages. The white pages is a database of information about responsible people that is maintained by the Internet registrars. When you request an official network number or domain name, you are asked to provide contact information, which becomes your personal record in the white pages database. Because of this, everyone who is responsible for an official network or domain is supposed to have an entry in the white pages, and that entry can be retrieved by anyone who needs to contact them.
Many Unix systems provide a whois command to query the white pages. The general form of this command is:
% whois [-h server] name
The name field is the information to be searched for in the white pages database. The server field is the name of a system containing the white pages.
In the following example, we search for contact information for the verio.net domain, which is the domain where the remote router from the traceroute example is located.
$ whois verio.net
[whois.crsnic.net]
Whois Server Version 1.3
Domain names in the .com, .net, and .org domains can now be registered
with many different competing registrars. Go to http://www.internic.net
for detailed information.
Domain Name: VERIO.NET
Registrar: MELBOURNE IT, LTD. D/B/A INTERNET NAMES WORLDWIDE
Whois Server: whois.inww.com
Referral URL: http://www.inww.com
Name Server: NS0.VERIO.NET
Name Server: NS1.VERIO.NET
Name Server: NS2.VERIO.NET
Updated Date: 13-jun-2001
>>> Last update of whois database: Tue, 17 Jul 2001 02:04:28 EDT <<<
The Registry database contains ONLY .COM, .NET, .ORG, .EDU domains and
Registrars.
[whois.inww.com]
Domain Name.......... verio.net
Creation Date........ 1996-12-07
Registration Date.... 2000-05-10
Expiry Date.......... 2001-12-06
Organisation Name.... Verio, Inc.
Organisation Address. 8005 South Chester Street
Organisation Address. Suite 200
Organisation Address. Englewood
Organisation Address. CO
Organisation Address. 80112
Organisation Address. UNITED STATES
Admin Name........... Hostmaster Verio
Admin Address........ 8005 South Chester Street
Admin Address........ Suite 200
Admin Address........ Englewood
Admin Address........ 80112
Admin Address........ CO
Admin Address........ UNITED STATES
Admin Email.......... DomainAdmin@verio.net
Admin Phone.......... 214 290 8620
Admin Fax............ .
Tech Name............ Hostmaster Verio
Tech Address......... 8005 South Chester Street
Tech Address......... Suite 200
Tech Address......... Englewood
Tech Address......... CO
Tech Address......... 80112
Tech Address......... UNITED STATES
Tech Email........... hostmaster@verio.net
Tech Phone........... 214 290 8620
Tech Fax............. .
Name Server.......... NS0.VERIO.NET
Name Server.......... NS1.VERIO.NET
Name Server.......... NS2.VERIO.NET
The query displays the name, address, and telephone number of the contacts for the domain, as well as a list of hosts providing authoritative name service for the domain. This example shows how it is supposed to work, and for substantial, well-run networks such as verio.net, it usually does. Unfortunately, many whois queries return no useful information because the white pages database is poorly maintained. If whois provides no information, try checking DNS name service. The DNS SOA record should contain a mailing address for a domain contact who may be able to point you to the right system administrator.
