the netflow, sflow, cflow, jflow, ipfix-flow dilemma

2
The NetFlow, sFlow, Cflow, Jflow, IPFIX-Flow Dilemma Vincent Berk, October 27, 2009 It seems these days that the marketplace is saturated with flow export formats. CISCO has NetFlow, InMon has sFlow®, Juniper uses JFlow, and there are several others. Few of these manufacturers seem to release details on the inner workings of their protocols, and their subsequent benefits. What follows is an overview of flow technologies. For the NetFlow suite of protocols we most often see version 5 (supported by the majority of devices), some combined v5/v7 (the Catalysts), and some version 9 on the newer devices. Don’t be fooled by the ASA series of firewalls; they do not actually support version 9 flow exporting. Instead, these CISCO devices use NetFlow 9 to export log lines: no traffic info there! NetFlow v5 uses a static packet format (and is in this way very similar to v7), defining IPv4 IPs, protocols, ports, and millisecond precision on flow start and flow end times. Version 9 uses a dynamic format which is parsed based on a template which is sent around first. These templates are flexible and allow for expansion of the protocol in the future. (Incidentally, IPFIX is based on it also). JFlow and CFlow are the same as CISCO Netflow v5. Only NetFlow v9 supports IPv6. NetFlow defines a ‘flow’ as a unidirectional series of packets from IP A to IP B, using some protocol (TCP/UDP/ICMP/…). When the packets used either TCP or UDP, then the flow is further specified by a pair of ports; for instance, 10.20.30.40:53823->50.60.70.80:443 TCP. Often, since most communications require both sides to transmit packets, one will see NetFlow report two flows associated with every communication, accounting for the packets and bytes that went in either direction. A proper flow collector and analyzer will correlate these with each other for you, so that you can see a report of a full conversation. A proper flow collector and analyzer will correlate inbound and outbound flows with each other for you, so that you can see a report of a full conversation. The NetFlow suite of protocols (which include IPFIX, CFlow, and JFlow) is therefore a powerful source of security and network debugging information. Since each and every network communication is logged with millisecond precision you can quickly determine who communicated with whom and when. Flow information is also much more tenable than raw packet data, allowing for a much quicker first-look at the network; in other words, we can use flow data as a springboard to determine if further packet inspection is necessary. The sFlow ® protocol is a completely different animal. Easily configurable through SNMP, its primary objective is to be a statistical network monitoring tool. Lots of different performance counters can be monitored through the sFlow® protocol, and the biggest benefit of sFlow® comes from its infinite scalability in large networks under heavy loads; however this innovative statistical approach comes at a slight disadvantage in accuracy, granularity, and timing precision. To understand this, we have to take a closer look at how sFlow® measures network traffic. Unlike NetFlow, the sFlow® protocol samples every N-th packet from the traffic stream, where N can be one-in- 512, one-in-1024, etcetera. This means that some communications may slip by entirely undetected, and the sFlow® collector software will not know about them. Larger communications, such as big downloads, and online video content will stand a much bigger change of being reported, as there are many packets involved. The second drawback is the lack of accurate timestamping of the packet data. Sampled packets get forwarded as they are picked up from the datastream; however, they are not timestamped. Therefore a small amount of uncertainty is introduced as to what the exact capture time of the packet was. Although these tradeoffs render sFlow® not particularly well suited to network forensic investigations (there is

Upload: proqsys

Post on 22-Apr-2015

1.118 views

Category:

Technology


3 download

DESCRIPTION

It seems these days that the marketplace is saturated with flow export formats. CISCO has NetFlow, InMon has sFlow®, Juniper uses JFlow, and there are several others. Few of these manufacturers seem to release details on the inner workings of their protocols, and their subsequent benefits. What follows is an overview of flow technologies.

TRANSCRIPT

Page 1: The NetFlow, sFlow, CFlow, JFlow, IPFIX-Flow Dilemma

The NetFlow, sFlow, Cflow, Jflow, IPFIX-Flow Dilemma

Vincent Berk, October 27, 2009

It seems these days that the marketplace is saturated with flow export formats. CISCO has NetFlow, InMon has

sFlow®, Juniper uses JFlow, and there are several others. Few of these manufacturers seem to release details

on the inner workings of their protocols, and their subsequent benefits. What follows is an overview of flow

technologies.

For the NetFlow suite of protocols we most often see version 5 (supported by the majority of devices), some

combined v5/v7 (the Catalysts), and some version 9 on the newer devices. Don’t be fooled by the ASA series of

firewalls; they do not actually support version 9 flow exporting. Instead, these CISCO devices use NetFlow 9 to

export log lines: no traffic info there! NetFlow v5 uses a static packet format (and is in this way very similar to v7),

defining IPv4 IPs, protocols, ports, and millisecond precision on flow start and flow end times. Version 9 uses a

dynamic format which is parsed based on a template which is sent around first. These templates are flexible and

allow for expansion of the protocol in the future. (Incidentally, IPFIX is based on it also). JFlow and CFlow are the

same as CISCO Netflow v5. Only NetFlow v9 supports IPv6.

NetFlow defines a ‘flow’ as a unidirectional series of packets from IP A to IP B, using some protocol

(TCP/UDP/ICMP/…). When the packets used either TCP or UDP, then the flow is further specified by a pair of

ports; for instance, 10.20.30.40:53823->50.60.70.80:443 TCP. Often, since most communications require both

sides to transmit packets, one will see NetFlow report two flows associated with every communication, accounting

for the packets and bytes that went in either direction. A proper flow collector and analyzer will correlate these

with each other for you, so that you can see a report of a full conversation.

A proper flow collector and analyzer will correlate inbound and outbound flows with each other for you, so that you

can see a report of a full conversation.

The NetFlow suite of protocols (which include IPFIX, CFlow, and JFlow) is therefore a powerful source of security

and network debugging information. Since each and every network communication is logged with millisecond

precision you can quickly determine who communicated with whom and when. Flow information is also much

more tenable than raw packet data, allowing for a much quicker first-look at the network; in other words, we can

use flow data as a springboard to determine if further packet inspection is necessary.

The sFlow® protocol is a completely different animal. Easily configurable through SNMP, its primary objective is

to be a statistical network monitoring tool. Lots of different performance counters can be monitored through the

sFlow® protocol, and the biggest benefit of sFlow® comes from its infinite scalability in large networks under

heavy loads; however this innovative statistical approach comes at a slight disadvantage in accuracy, granularity,

and timing precision. To understand this, we have to take a closer look at how sFlow® measures network traffic.

Unlike NetFlow, the sFlow® protocol samples every N-th packet from the traffic stream, where N can be one-in-

512, one-in-1024, etcetera. This means that some communications may slip by entirely undetected, and the

sFlow® collector software will not know about them. Larger communications, such as big downloads, and online

video content will stand a much bigger change of being reported, as there are many packets involved. The

second drawback is the lack of accurate timestamping of the packet data. Sampled packets get forwarded as they

are picked up from the datastream; however, they are not timestamped. Therefore a small amount of uncertainty

is introduced as to what the exact capture time of the packet was.

Although these tradeoffs render sFlow® not particularly well suited to network forensic investigations (there is

Page 2: The NetFlow, sFlow, CFlow, JFlow, IPFIX-Flow Dilemma

some statistical uncertainty as to when a communication began, how many packets were transmitted each way,

their size, and when it ended), they are necessary to allow the virtually unlimited scalability that sFlow® offers. If

the network gets busy, it can fall back to a slower sampling rate, and keep load on the exporting device and

sFlow® collector down significantly.

A simple answer to the question: “Which flow solution is right for me?” therefore is based on the intended purpose

of the implemention. At the ISP or large enterprise level the hardware cost associated with tracking every

communication through NetFlow is substantial, and can only be justified if the NetFlow data is used for security

and network forensic analysis. If the goal is to simply get a rough overview of usage (“Who’s hogging my

bandwidth?”), the sFlow® protocol will suffice, and be much more managable, and less costly. At smaller sites,

the decision will usually be dictated by the switching and routing gear in the current network closet. Use what you

have!

Vincent Berk is the founder of ProQSys, a company that specializes in network security and analysis

software.