9.2 How sendmail Uses DNS

The sendmail program uses DNS in several different ways:

  • When sendmail first starts, it might use DNS to get the canonical name for the local host. That name is then assigned to the $j macro ($j).[5] If DNS returns additional names for the local host, those names are assigned to the class $=w ($=w).

    [5] Prior to V8 sendmail, the canonical name was stored in the $w macro ($w) and sendmail initialized only the $j macro ($j). Beginning with V8, sendmail initializes both of those variables, among others (Section 21.1).

  • When sendmail first starts, it looks up the IP number or numbers assigned to each network interface. For each address it finds, it uses DNS to look up the hostname associated with that address.

  • When another host connects to the local host to transfer mail, the local sendmail looks up the other host with DNS to find the other host's canonical name.

  • Before accepting mail, sendmail can look up the IP address of the connecting host on various blacklist sites (Chapter 7). If that address is listed, the message is rejected.

  • To relay based on MX records (Section 7.4.4), sendmail does a lookup to determine if the connecting host is listed as an MX server for the local domain.

  • When delivering network SMTP mail, sendmail uses DNS to find the address (or addresses) to which it should connect.

  • When sendmail expands $[ and $] in the RHS of a rule, it looks up the hostname (or IP number) between them.

We discuss each of these uses later in this chapter.

9.2.1 Determine the Local Canonical Name

All versions of sendmail use more or less the same logical process to obtain the canonical name of the local host. As illustrated in the following sample program, sendmail first calls gethostname(3) to obtain the local host's name. That name can either be a short name or a fully qualified one depending on how your system is set up. If the call to gethostname(3) fails, the name of the local host is set to localhost:

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/param.h>
#include <netdb.h>
#include <stdio.h>

main( )
        char hostbuf[MAXHOSTNAMELEN];
        struct hostent *hp;

        /* Get the local hostname */
        if (gethostname(hostbuf, sizeof(hostbuf)) < 0)
                strcpy(hostbuf, "localhost");
        printf("hostname = "%s"\n", hostbuf);

        /* canonicalize it and get aliases */
        if((hp = gethostbyname(hostbuf)) =  = NULL)
        printf("canonical = "%s"\n", hp->h_name);
        while (*hp->h_aliases != NULL)
                printf("alias: "%s"\n", *hp->h_aliases);

The local hostname is then given to the gethostbyname routine to obtain the canonical name for the local host. That same routine also returns any aliases (other names for the local host). Note that, if you defined NETINET6 (NET...) when compiling (for IPv6 support), you must use getipnodebyname(3) in place of gethostbyname(3).

The short (host) name found by gethostbyname(3) or getipnodebyname(3) is assigned as the value of the $w sendmail macro. The short name, the canonical name, and any aliases are added to the class $=w.

If the DontProbeInterfaces option (DontProbeInterfaces) is undefined, or set to false, the address and hostname associated with each interface are also added to the class $=w (see Section 9.2.2).

Some old Sun and Ultrix machines are set up to use NIS where the canonical name is the short name, and a fully qualified name that should have been the canonical name appears as an alias. For such systems you must link with the BIND library (libresolv.a) when compiling this program or compiling sendmail. That library gets its information from DNS rather than from NIS. But note that V8.7 and above versions of sendmail do the intelligent thing and use the canonical name that was found in the list of aliases, if it exists.

If a good BIND library is not available, or if it is not convenient to compile and install a new version of sendmail, you can circumvent the short name assigned to the $j sendmail macro by defining $j like this:

define(`confDOMAIN_NAME', `canonical name here')

The canonical name is your site's hostname with a dot and your domain name appended.

The result of all these lookups can be viewed by running sendmail with a -d0.4 debugging switch (-d0.4). The actual DNS lookups can be watched with the -d8.8 debugging switch (-d8.8).

9.2.2 Probe Network Interfaces

After the canonical name, and any other names for the local machine, have been placed in $=w, sendmail then searches (probes) all the network interfaces to find any additional names and addresses that might also need to be added to $=w. But note that if the DontProbeInterfaces option (DontProbeInterfaces) is defined as true, this additional step is skipped. Note also that if the DontProbeInterfaces option is defined as the literal value localhost, only the loopback interface is skipped, and all the other network interfaces are included.

The list of network interfaces is obtained from your kernel using a system call appropriate for your operating system. The kernel generally returns a list composed of interface and IP address pairs. If you defined NETINET6 (NET...) when compiling, the list might contain IPv6 addresses. If you defined NETINET (NET...) when compiling, the list might contain IPv4 addresses.

For each address that is found, sendmail performs a reverse-lookup using gethostbyaddr(3) or getipnodebyaddr(3). Each lookup (if successful) will return the hostname associated with the address.

Each address and hostname is appended to the class $=w. The names and addresses added can be viewed with the -d0.4 debugging command-line switch (-d0.4), which also allows errors in this process to be printed.

9.2.3 Look Up a Remote Host's Name

When sendmail begins to run as a daemon, it creates a socket, binds to that socket, and listens for incoming SMTP connections. When a remote host connects to the local host, sendmail uses the accept(2) library routine to accept the connection. The accept(2) routine provides the IP address of the remote machine to sendmail. After that it calls gethostbyaddr(3) or getipnodebyaddr(3) to convert that IP address to a canonical hostname. The sendmail program then calls gethostbyname(3) or getipnodebyname(3) to find all the addresses for that found hostname. If the original address is not in that list, sendmail considers the address and hostname to be forgeries and records that fact in its syslog messages, its added Received: header, and its reply to the initial greeting:

(may be forged)

The sendmail program needs a valid canonical hostname for five reasons:

  • The remote hostname is compared to the local hostname to prevent sendmail from connecting to itself.

  • The remote hostname claimed in the HELO or EHLO SMTP line is compared to the canonical name. If they differ, sendmail adds text noting that difference to its SMTP reply, and adds both to the Received: header it generated.

  • The macro $s is assigned the canonical hostname as its value.

  • The canonical name is included in many log messages produced by the setting of the LogLevel (L) option (LogLevel) and is available for inclusion in Received: header (Received:) lines.

  • The canonical name is used by the various antirelay rule set checks.

9.2.4 DNS Blacklist Lookups

If you define the dnsbl feature (Section 7.2) or the enhdnsbl feature (Section 7.2.2) in your mc configuration file, you will cause sendmail to look up the IP number of each connecting site at the blackhole server you specify. If a lookup is successful and returns a match, the connection is rejected. If a lookup is successful and returns no match, the connection is accepted. If the lookup fails, the connection is either deferred or accepted, depending on the nature of the failure.

Lookups are performed using the host database type (dns). Each lookup attempts to find A records that correspond to the address looked up. Note that this is different from the usual way in which addresses are looked up. Normally, addresses are reverse-looked-up to find hostnames. But for blackhole purposes, addresses are forward-looked-up, as though they are hostnames.

9.2.5 Look Up Addresses for Delivery

When sendmail prepares to connect to a remote host for transfer of mail, it first performs a series of checks that vary from version to version. All versions accept an IP address surrounded with square brackets as a literal address and use it as is.

Beginning with V8.1, sendmail first checks to see whether the host part of the address is surrounded with square brackets. If so, it skips looking up MX records. (We'll elaborate on MX records soon.)

Beginning with V8.8, sendmail first checks to see whether the F=0 flag (F=0 (zero)) is set for the selected delivery agent. If it is set, sendmail skips looking up MX records.

If sendmail is allowed to look up MX records, it calls the res_search(3) BIND library routine to find all the MX records for the host. If it finds any MX records, it sorts them in order of cost, and lists them, placing the least expensive first. If V8 sendmail finds two costs that are the same, it randomizes the selection between the two when sorting.[6]

[6] Note that this is broken in many older versions of sendmail. Also note that when the MX record points to the local host, all MX records with a cost greater than or equal to the local host are tossed. (See $w for a description of this process.)

After all MX records are found and listed, or if no MX records were found, sendmail adds the host specified by the FallbackMXhost option (FallbackMXhost) to the end of the list. For V8.11 and earlier, the hostname, if there was one, was added to the end of the list as is. Beginning with V8.12, if a hostname is listed, MX records are looked up for it as well, and those MX records are added (in the proper sorted order) to the end of the list. By surrounding the hostname specified under V8.12 in square brackets, the behavior of earlier versions is emulated in that the hostname is added as is (surrounded in square brackets).

If there are no MX records, the original hostname becomes the only entry in the list. If, in this instance, the FallbackMXhost option adds MX records, they are added following that hostname.

The sendmail program then tries to deliver the message to each host in the list of MX hosts, one at a time, until one of them succeeds or until they all fail. Beginning with V8.8 sendmail, if a host in the list returns a 5xy SMTP code (permanent failure), the effect is to cause subsequent MX hosts to be ignored. (Connect failures are the exception, in that they continue to the next MX host as usual.) Most temporary errors cause sendmail to try the next MX record. If sendmail exhausts the MX list with neither success nor a permanent error, the temporary error will cause the message to be queued for a later attempt.

If no MX records are found, sendmail tries to deliver the message to the single original host. If all else fails, sendmail attempts to deliver to the host listed with the FallbackMXhost option.

Whether sendmail tries to connect to the original host or to a list of MX hosts, it calls gethostbyname(3) or getipnodebyname(3) to get the network address for each. It then opens a network connection to each address in turn and attempts to send SMTP mail. If there are IPv6 addresses,[7] they are tried first, then IPv4 addresses, if any. If a connection fails, it proceeds to the next address in the list until the list is exhausted. When there are no more addresses to try, the message is deferred and held in the queue for a later attempt.

[7] And if sendmail was built with the NETINET6 (NET...) compile-time macro defined.

9.2.6 The $[ and $] Operators

The $[ and $] operators (Section 18.7.6) are used to canonicalize a hostname. Here is a simplified description of the process.

Each lookup is actually composed of many lookups that occur in the form of a loop within a loop. In the outermost loop, the following logic is used:

  • If the address has at least one dot somewhere in it, sendmail looks up that address unmodified first.

  • If the unmodified address is not found and the RES_DNSRCH bit is set (the ResolverOptions option, ResolverOptions), sendmail looks up variations on the domain part of the address. The default domain is tried first (for a host in the sub sub-domain at dc.gov, that would be sub.dc.gov, thus looking up host.sub.dc.gov). If that fails, BIND 4.8 then throws away the lowest part of the domain and tries again (looks up host.dc.gov). BIND 4.9 and above use the search attribute, if given, and try that list of possible domains.

  • If the address has no dots and the RES_DEFNAMES bit is set (the ResolverOptions option, ResolverOptions), sendmail tries the single default domain (looks up host.sub.dc.gov). This is for compatibility with older versions of DNS.

Each lookup just described is performed by using the following three steps:

  • Prior to V8.12 sendmail, try the hostname with a T_ANY query that requests all the cached DNS records for that host. If it succeeds, IPv6 AAAA records, IPv4 A records, and/or MX records might be among those returned. However, success is not guaranteed because sometimes only NS records are returned. In that instance the following two steps are also taken.

  • Beginning with V8.12 sendmail, if using IPv6, try the hostname with a T_AAAA query that requests the AAAA record, then, if using IPv4, try the hostname with a T_A query that requests the A records.

  • If only NS records are returned, try the hostname with a T_MX query that requests MX records for the host.

Each query searches the data returned as follows:

  • Search for a CNAME (alias) record. If one is found, replace the initial hostname (the alias) with the canonical name returned and start over.

  • Search for an A or AAAA record (the IP address). If one is found, the hostname that was just used to query is considered the canonical address.

  • Search for an MX record. If one is found and a default domain has not been added, treat the MX record like an A record. For example, if the input hostname is sub.dc.gov and an MX record is found, the MX record is considered official. If, on the other hand, the input hostname has no domain added (is sub) and the query happens to stumble across sub.dc.gov as the MX record, the following searches are also tried.

  • If an MX record is found and no MX record has been previously found, the looked-up hostname is saved for future use. For example, if the query was for sub.dc.gov and two MX records were returned (hostA.sub.dc.gov and hostB.sub.dc.gov), sub.dc.gov is saved for future use.

  • If no MX record is found, but one was found previously, the previous one is used. This assumes that the search is normally from most to least complex (sub.sub.dc.gov, sub.dc.gov, dc.gov).

All this apparent complexity is necessary to deal with wildcard MX records (Section 9.3.4) in a reasonable and usually successful way.

9.2.7 Broken IPv6 Name Servers

The sendmail program will look up AAAA records only if it is built with the NETINET6 (NET...) compile-time macro defined. As described earlier, sendmail looks up the AAAA records first, then A records.

All name servers should return NODATA if a host is found and no AAAA records are available. But some name servers are broken and, when asked for an AAAA record, will wrongly return a temporary failure (SERVFAIL). This causes sendmail to queue the mail for later delivery.

If you have defined NETINET6 when building sendmail, and if you notice this kind of error, we have two recommendations:

  • Notify hostmaster[8] at the site that is running the broken name server. The sooner broken name servers are fixed, the cleaner the Internet will run.

    [8] Run the whois(1) program to find the email address of the administrator for the site. It should be hostmaster, but often it is not.

  • Add the WorkAroundBrokenAAAA argument to the ResolverOptions option (ResolverOptions) in your mc configuration file:

    define(`confBIND_OPTS', `+WorkAroundBrokenAAAA')

    This will cause sendmail to pretend that NODATA was returned when SERVFAIL is wrongly returned. This causes sendmail to continue with further lookups, specifically for A and MX records.

    Part I: Build and Install
    Part II: Administration
    Part III: The Configuration File