Introducing flow formats and their differences

There are multiple flow formats. What are the differences? Which are supported by Flowmon? Check the post to see the answers.

Flow monitoring has become the prevalent method for monitoring traffic in high-speed networks. Several standards of flow format exist and it can be tricky to choose the right one for your needs. In this article we will go through the most common flow formats, providing a basic overview of their history and differences.

Flow monitoring history

The history of flow monitoring goes back to 1996 when the NetFlow protocol was patented by Cisco Systems.  Flow data represents a single packet flow in the network with the same 5-tuple identification composed of source IP address, destination IP address, source port, destination port and protocol. Based on this, packets are aggregated into flow records that accumulate the amount of transferred data, the number of packets and other information from the network and transport layer. A typical flow monitoring setup consists of three main components:

• Flow exporter – create flow records by aggregating packet information and exports the records to one or more flow collectors (eg. Flowmon Probe).
• Flow collector – collects and stores the flow data (eg. Flowmon Collector).
• Analysis application – allows the visualization and analysis of the received flow data (eg. Flowmon Monitoring Center - native application of the Flowmon Collector).

Cisco originally developed the protocol for its products. Other manufacturers have followed such an approach and have developed more or less similar proprietary flow data formats.

Cisco standards

NetFlow v5

The first widely adopted version was NetFlow v5, which became available in 2006. NetFlow v5 is still the most common version, and it is supported by a wide range of routers and switches. However, it no longer meets the needs for accurate flow monitoring as it does not support IPv6 traffic, MAC addresses, VLANs or other extension fields.

NetFlow v9

NetFlow v9 brought several added improvements. The most important is support for templates, which allow a flexible flow export definition and ensures that NetFlow can be adapted to provide support for new protocols. Other improvements are the support for IPv6, Virtual Local Area Networks (VLANs) and Multiprotocol Label Switching (MPLS) and other features. NetFlow v9 is supported on most of the recent Cisco routers and switches.

Flexible NetFlow

Cisco still continues to improve NetFlow technology. The next generation is called Flexible NetFlow, and it further extends NetFlow v9. What Flexible NetFlow can export is highly customizable, which allows customers to export almost anything that is passing through the router.

Other vendors

jFlow, NetStream, cflowd

All standards mentioned above are similar to the original Cisco NetFlow standard. jFlow was developed by Juniper networks, NetStream by Huawei and cflowd by Alcatel-Lucent.

Independent standard

IPFIX

The proposal for IPFIX (Internet Protocol Flow Information eXport) protocol was published by the IETF in 2008. IPFIX is derived from NetFlow v9 and should serve as a universal protocol for exporting flow information from network devices to a collector or Network Management System. The IPFIX is more flexible than NetFlow and allows to extend flow data with additional information about network traffic. As an example, our Flowmon IPFIX extensions enrich the IPFIX flow data with application layer protocol metadata, network performance statistics and other information.In Cisco world IPFIX is usually referred to as NetFlow v10 and provides various extensions similar to Flowmon.

Related standards

NSEL

NSEL (NetFlow Security Event Logging) allows exporting Flow Data from Cisco’s ASA family of security devices. It has a similar format as NetFlow, but requires a different interpretation and has different use-cases - the purpose of NSEL is to track firewall events and logs via NetFlow. Unfortunately, sometimes people got confused by the terminology and consider NSEL compatible with NetFlow. In fact, with NSEL there is not enough information to provide traffic charts or support detailed drill downs and troubleshooting.

sFlow

Unlike NetFlow, sFlow is based on sampling. An sFlow agent obtains traffic statistics using sFlow sampling, encapsulates them into sFlow packets, which are then sent to the collector. sFlow provides two sampling modes - flow and counter sampling:

• Flow sampling where the sFlow agent samples packets in one direction or both directions on an interface based on the sampling ratio, and parses the packets to obtain information about packet data content
• Counter sampling where the sFlow agent periodically obtains traffic statistics on an interface

Flow sampling focuses on traffic details to monitor and parse traffic behaviors on the network while counter sampling focuses on general traffic statistics.Due to packet sampling it is however not possible to have an accurate representation of the traffic and some traffic will be missed.  Therefore, sampling can limit usage of flow data in cases like network anomaly detection. On the other hand, it can be used for top statistics or DDoS attack detection. Cisco has introduced very similar technology to sFlow which is called NetFlow Lite.

What formats does Flowmon support?

Our standalone Probe allows exporting flow data in NetFlow v5/v9 and IPFIX format. Additionally, the Probe can use the Flowmon IPFIX extension that allows enriching the flow data with additional information, such as network performance statistics (for example, Round-Trip Time, Server Response Time and Jitter) and information from the application protocols (HTTP, DNS, DHCP, SMB, E-mail, MSSQL and others).The Flowmon Collector can process network traffic statistics from various sources and flow standards, including:

• NetFlow v5/v9
• IPFIX
• NetStream
• jFlow
• cflowd
• sFlow, NetFlow Lite

Conclusion: which flow format to use?

We have introduced the most common flow formats. Although the format you can use depends on your network infrastructure, from our experience in developing high-performance network monitoring appliances, we highly recommend using NetFlow v9/IPFIX export formats, as they provide the most accurate and comprehensive information. For more detailed information about the compatible sources of flow data please refer to our web page. If you want more details on where to the flows come from, check out previous blog post.

To see how Flowmon can help to monitor your own network, check out our online demo or download a free trial