Data Collection Protocols: NetFlow Version 9 and IPFIX Export Protocols

Cisco IOS NetFlow services give network administrators access to information about IP flows within their networks. In the context of Cisco NetFlow, an IP flow is defined as a unidirectional sequence of packets between given source and destination endpoints. IP flows are highly granular; flow endpoints are identified by IP address and by transport layer application port numbers. NetFlow Services also uses the IP protocol type, type of service (ToS), and the input interface identifier to uniquely identify flows.

NetFlow flow records are exported to an external device, a NetFlow collector, and can be used for a variety of purposes, including network management and planning, enterprise accounting, departmental chargeback, ISP billing, data warehousing, user monitoring and profiling, combating denial of service (DoS) attacks, and data mining for marketing purposes.

NetFlow services, deliberately a generic term, refers to two completely different components: the NetFlow metering process and the NetFlow export protocol. Chapter 7, "NetFlow," analyzes in detail different aspects of the metering process: metering process concepts and features, configurations examples, deployment guidelines. This section covers the export protocol (NetFlow version 9 and the IPFIX protocol).

The IETF IPFIX charter is not intended to "fix" IP, nor does it define a fixed solution, as its name might imply. IPFIX, which stands for "IP Flow Information eXport," is an IETF effort to standardize an export protocol similar to NetFlow—specifically, a protocol that exports flow-related information. Actually, the IPFIX protocol specifications are largely based on the NetFlow version 9 export protocol. With the NetFlow version 9 export protocol in mind, the IPFIX section covers the NetFlow version 9 export protocol improvements added to the IPFIX protocol to make it a more robust protocol.

NetFlow Version 9 Export Protocol

A few years ago, NetFlow became the de facto IP accounting standard throughout the industry. By publishing the protocol details, Cisco encouraged third-party companies to develop their own NetFlow collectors. Therefore, many vendors have implemented the NetFlow export protocol at network elements or specific NetFlow collectors. Examples of open-source implementations of NetFlow collectors are nfdump/NetFlow sensor (http://nfdump.sourceforge.net/), Flow-Tools (http://www.splintered.net/sw/flow-tools/), and ntop (http://www.ntop.org/netflow.html).

The Template Mechanism

The basic output of NetFlow is a flow record exported from an exporter—a device (such as a router) with the NetFlow services enabled. An exporter monitors packets entering an observation point (such as a network interface) and creates flow records from these packets. The term "exporter" is introduced in the NetFlow version 9 and IPFIX RFCs and is used in this chapter for consistency with the RFCs. Several different formats for flow records have evolved as NetFlow has matured. The most recent evolution of the NetFlow flow record format is known as version 9. The distinguishing feature of the NetFlow version 9 format compared to the previous version is its template-based feature. Templates, which define collections of fields with corresponding descriptions of structure and semantics, provide a flexible design for the record format.

The important principle of the NetFlow version 9 flow export protocol is that it first exports the templates from the network elements to a NetFlow collector and then the flow records. As soon as a NetFlow collector receives the template(s), it can decode the subsequent flow records. The templates are composed of the field types, the field lengths, and a generated unique template ID. The flow records are composed of the unique template ID and the field values.

Using templates provides several key benefits:

• Because the template mechanism is flexible, it allows the definition and export of only the required flow fields to a NetFlow collector. This helps reduce the exported flow data volume and provides potential memory savings for the exporter and NetFlow collector.

• New field types can be added to NetFlow version 9 flow records without changing the structure of the export record format. With previous NetFlow versions (1, 5, 7, and 8), adding a new field in the flow record implied a new version of the export protocol format and a new version of the NetFlow collector that supported the parsing of the new export protocol format. The template mechanism in NetFlow version 9 provides an extensible design to the record format. It allows future enhancements to NetFlow services without requiring changes to the export format. Indeed, as the NetFlow metering process now can classify new traffic types such as IPv6 and MPLS, no changes are required to the NetFlow export protocol (just new templates). The only requirement is an update of the information model with the definitions of the new information elements.

• Templates sent to a NetFlow collector contain the structural information about the exported flow record fields. Therefore, even if the NetFlow collector does not understand the semantics of new data types, it can still decode the entire flow record. After the collector updates the information model with new data types, it can interpret the meaning of the flow record information elements.

To summarize, the NetFlow version 9 export protocol is a result of the previously fixed NetFlow export formats. Whenever an additional export field was required, a new export format was specified. NetFlow export version 9 overcame this limitation. Note that the IPFIX protocol is based on the NetFlow version 9 export protocol.

The Export Protocol

Without explaining all the details from the specification terminology section (RFC 3954, Cisco Systems NetFlow Services Export Version 9), let's review the most useful terms and concepts.

Cisco NetFlow incorporates the push model, in which the exporter sends flow records periodically to a NetFlow collector. The export frequency depends on the active and inactive flow timeouts, the flow expiration detection, and the flow cache full condition.

Some terms are capitalized in the NetFlow specification (RFC 3954) because they have a formal definition. Those terms keep their capitalizations in this section (see Table 3-4).

The NetFlow version 9 record format consists of a packet header followed by one or more Template FlowSets or Data FlowSets. A FlowSet is a generic term for a collection of flow records that have a similar structure. There are three different types of FlowSets:

• A Data FlowSet is composed of Options Data Record(s) or Flow Data Record(s). No Template Record is included.

• A Template FlowSet is composed of Template Record(s). No Flow or Options Data Record is included.

• An Options Template FlowSet is composed of Options Template Record(s). No Flow or Options Data Record is included.

The Template Record, sent in a Template FlowSet, specifies the type and length of the flow record-specific fields. Those flow records are called Flow Data Records and are present in Data FlowSets. In other words, the flow record template is defined in a Template Record, which is sent in a Template FlowSet, and the associated instances of flow records are sent as Flow Data Records in the Data FlowSet.

An additional record type is very important within the NetFlow version 9 specification: the Options Template FlowSet (and its corresponding Options Data Record). Rather than supplying information about IP flows, Options Template Records are used to define information about the NetFlow metering process configuration or NetFlow metering process-specific data. The information is supplied by the Option Data Records. For example, the Options Template FlowSet can report the sample rate of a specific interface, along with the sampling method used (if sampling is supported). Those sampling parameters are defined in an Options Template Record sent in a Template FlowSet, and the associated instances of records are sent in Options Data Records in subsequent Data FlowSets.

The Options Template Set lets the exporter provide additional information to the collector that would not be possible with Flow Records only. The scope, which is provided in the Options Template Set, gives the context of the reported Information Elements. One Options Template Set example is the "Metering Process statistics," which reports the statistics for the Observation Domain, defined as the scope. Another example is the "Template configuration," which reports the sampling configuration parameter(s) for the template, defined as the scope.

A Data FlowSet consists of one or more records of the same type, which are grouped in an Export Packet. Each record is either a Flow Data Record or an Options Data Record previously defined by a Template Record or an Options Template Record, respectively.

To achieve efficiency in terms of processing at the network elements when handling high volumes of Export Packets, the NetFlow FlowSets are encapsulated into UDP (RFC 768) datagrams for export to a NetFlow collector. UDP is a non-congestion-aware protocol. Therefore, when deploying NetFlow version 9 in a congestion-sensitive environment, best practice suggests setting up a connection between the network element and the NetFlow collector through a dedicated link. This ensures that any burstiness in the NetFlow traffic affects only this dedicated link. In a network where the NetFlow collector cannot be placed within one hop of the Exporter, or when the export path from the Exporter to the NetFlow collector cannot be exclusively used for the NetFlow export packets, the export path should be designed so that it can always sustain the maximum burstiness of NetFlow traffic from the network element. However, the NetFlow version 9 export protocol has been designed to be transport protocol-independent. Therefore, it supports the Stream Control Transmission Protocol–Partial Reliability Extension (SCTP-PR, RFC 3758) as an alternative to UDP. SCTP-PR establishes an association between a network element and a NetFlow collector. Within this association, SCTP-PR offers the possibility of having multiple streams, and potentially multiple streams with different levels of reliability: a fully reliable stream, a partially reliable stream, and an unreliable stream. Depending on the importance of the data to export, a different stream type could be used. Typically, the Template Records use the fully reliable stream. Within the same SCTP-PR association, the billing records might use the fully reliable stream, whereas some sampled capacity planning records might use the partially reliable stream. As a final remark, note that the exporter can export to multiple collectors using independent transport protocols.

Regarding template management, the NetFlow version 9 export protocol follows a few basic rules, due to the unreliable property of UDP:

• A newly created Template Record is assigned an unused Template ID from the NetFlow metering process. If the template configuration is changed, the current Template ID is abandoned and should not be reused until the NetFlow metering process restarts. A template ID must be unique per network element.

• After a NetFlow process restarts, the exporter should not send any Data FlowSet without sending the corresponding Template FlowSet and the required Options Template FlowSet in a previous packet or including it in the same Export Packet.

• In the event of configuration changes, the exporter should send the new template definitions at an accelerated rate.

• The exporter must send the Template Records and Options Template Records on regular basis to refresh the collector's database.

• In the event of a clock configuration change on the Exporter, the exporter should send the template definitions at an accelerated rate.

Table 3-5 describes a few field type definitions that an exporter may support, as examples. The complete series of field types is a selection of packet header fields, lookup results (for example, the BGP Autonomous System numbers or the IP subnet masks), and packet properties (for example, the packet length). The list should represent all useful flow related field types. However, for extensibility, the new field types are added to the existing list. These new field types have to be updated on the exporter and NetFlow collector field type lists while the NetFlow export format remains unchanged.

In some cases, the size of a field type is fixed by definition—for example, PROTOCOL or IPV4_SRC_ADDR. However, in other cases, they are defined as a variant type. This improves the memory efficiency in the collector and reduces the network bandwidth requirements between the exporter and the NetFlow collector. As an example, in the case of IN_BYTES, on an access router it might be sufficient to use a 32-bit counter (N = 4), while on a core router a 64-bit counter (N = 8) would be required.

NetFlow Version 9 Export Protocol Example

In this example, the exporter must send Flow Data Records composed of the protocol, number of bytes received, and number of packets received. Three Flow Data Records with the values shown in Table 3-6 are registered by the metering process and consequently must be exported.

The exporter first sends a Template Record (in a Template FlowSet) specifying the new template ID (256), the length of the FlowSet, the number of field types in the Template Record (three in this case), and each field type value and length, as shown in Figure 3-8.

Then the exporter sends the Data FlowSet specifying the associated Template ID (256), the length of the FlowSet, and the Flow Data Records, as shown in Figure 3-9. Each Flow Data Record contains the three values of the associated Template Records field type.

For further examples, refer to RFC 3954.

IPFIX

The first IPFIX discussions started during the IETF 51 meeting with a BOF (Birds of a Feather) session (an informal discussion group, scheduled on a conference program to consider a specific issue or subject) in London in August 2001. With some early IP flow requirements as an introduction, and with the presentation of different IP flow solutions (among them, NetFlow version 9 was presented), the group considered it worthwhile to create a working group to standardize the export of IP flow information. For the next IETF meeting, the Internet Engineering Steering Group (IESG) created the working group, with the following specific goals described in the IPFIX charter:

• Define the notion of a "standard IP flow." The flow definition will be a practical one, similar to those currently in use by existing nonstandard flow information export protocols that have attempted to achieve similar goals but have not documented their flow definition.

• Devise data encodings that support analysis of IPv4 and IPv6 unicast and multicast flows traversing a network element at packet header level and other levels of aggregation.

• Consider the notion of IP flow information export based on packet sampling.

• Identify and address any security privacy concerns affecting flow data. Determine technology for securing the flow information export data, such as Transport Layer Security (TLS, RFC 2246).

• Specify the transport mapping for carrying IP flow information—one that is amenable to router and instrumentation implementers and deployment.

• Ensure that the flow export system is reliable in that it will minimize the likelihood of flow data being lost due to resource constraints in the exporter or receiver and to accurately report such a loss if it occurs.

The IPFIX working group went through the process of evaluating the current existing candidate protocols to determine which protocol would serve best as the foundation for the IPFIX protocol specifications. The evaluation process required the formalized IPFIX requirements; RFC 3917, Requirements for IP Flow Information Export, fulfilled that need. The working group evaluated the following five protocols:

• Common Reliable Accounting for Network Element (CRANE), RFC 3423

• Diameter (RFC 3588)

• Lightweight Flow Accounting Protocol (LFAP), developed by Riverstone Networks

• Streaming Internet Protocol Detail Records (IPDR), developed by the IPDR open consortium (http://www.ipdr.org)

• NetFlow version 9 export protocol (RFC 3954)

After the process of determining the relative weight of the different requirements, the working group finally decided in RFC 3955, Evaluation of Candidate Protocols for IP Flow Information Export, to select the NetFlow version 9 export protocol as the IPFIX protocol basis. However, some improvements were needed: first, to address the specific goals of the IPFIX charter; second, to make the protocol more robust; and third, for the IESG to approve it.

The IPFIX Export Protocol

The IPFIX protocol maintains the same principles of separate templates and records as NetFlow version 9. It even preserves the same terminology as NetFlow version 9 (see Table 3-4), with one exception: the FlowSets are now called Sets in IPFIX. The IPFIX improvements over NetFlow version 9 are significant. They address transport protocol, security, IETF, and vendor-specific information elements, new information elements registration with the Internet Assigned Numbers Authority (IANA), time synchronization, etc. The rest of this section describes the improvements.

The biggest change in IPFIX compared to the first NetFlow version 9 is certainly that the data transfer must be a congestion-aware protocol. Therefore, the IPFIX protocol specifies that the Stream Control Transport Protocol (SCTP, RFC 2960) and that the Stream Control Transport Protocol–Partially Reliable (SCTP-PR, RFC 3758) must be implemented by all compliant implementations, and that UDP and TCP may also be implemented. SCTP-PR should be used in deployments in which Exporters and Collectors are communicating over links that are susceptible to congestion, because SCTP-PR can provide any required degree of reliability: full reliability, partial reliability, or unreliability. The IPFIX specification also mentions that "TCP may be used in deployments where Exporters and Collectors communicate over links that are susceptible to congestion, but SCTP-PR is preferred, due to its ability to limit back pressure on Exporters and its message versus stream orientation." Finally, a door is left open to use UDP as a transport protocol, but only under certain circumstances: "UDP may be used although it is not a congestion-aware protocol. However, the IPFIX traffic between Exporter and Collector must run in an environment where IPFIX traffic has been provisioned for or is contained through some other means."

There are several scenarios in which increased security is needed compared to NetFlow version 9—more specifically, in situations where the IPFIX protocol is run across the Internet:

• Disclosure of flow information data— The IPFIX messages themselves may contain valuable information for an attacker, such as active flow in the network, communications endpoints, and traffic patterns. Thus, care should be taken to confine their visibility to authorized users.

• Forgery of flow records— If flow records are used in billing and/or security applications, there are potentially strong incentives to forge exported flow records (for example, to save money, prevent the detection of an attack, or simply confuse the collector). This can be done either by altering flow records on the path or by injecting forged flow records that pretend to be originated by the original exporter.

Regarding the security of the IPFIX records transferred from the exporter to the collector, the confidentiality, integrity, and authenticity must be ensured. The IPFIX protocol specification does not mandate that all three functions be enabled at all times. However, any IPFIX-compliant implementation must support confidentiality, integrity, and authenticity with Datagram Transport Layer Security (RFC R4347).

Now that the IANA administers all the IPFIX exported information elements, the IPFIX Sets formats have been adapted so that a combination of IANA administered information element(s) and private information element(s) could coexist in the same records. The private information elements are vendor-proprietary elements—either on their way to be registered by IANA, under experimental testing, or maintained confidentially for business-related reasons.

Regarding time synchronization, the IPFIX protocol supports by default the microsecond precision of the flow start and flow stop times.

However, specific mechanisms exist in case the millisecond time precision is enough or in case the nanosecond time precision is required.

Work in Progress

In June 2006, the IPFIX charter was revised to incorporate the new documents of interest to the IPFIX working group. The new objectives of the charter are as follows:

• Develop guidelines for implementers based on experiences gained individually by implementers and jointly at interoperability testing events. The outcome should be "IPFIX implementation guidelines" and "IPFIX testing" documents.

• Develop methods and means for an efficient use of the IPFIX protocol by reducing redundancy in flow reports.

• Create an IPFIX MIB for reporting information and statistics of IPFIX metering, exporting, and collecting processes.

• Develop an effective method for exporting information about bidirectional flows (biflows), requested by the IP security community.

IPFIX References

Table 3-7 lists RFCs that cover IPFIX.

^[(*)] IETF draft, work in progress that will be published as RFC.