wide area network performance analysis methodology
DESCRIPTION
Wide Area Network Performance Analysis Methodology. Wenji Wu, Phil DeMar, Mark Bowden Fermilab ESCC/Internet2 Joint Techs Workshop 2007 [email protected] , [email protected] , [email protected]. Topics. Problems End-to-End Network Performance Analysis TCP transfer throughput - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/1.jpg)
Wide Area Network Performance Analysis Methodology
Wenji Wu, Phil DeMar, Mark BowdenFermilab
ESCC/Internet2 Joint Techs Workshop 2007
![Page 2: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/2.jpg)
2
Topics Problems End-to-End Network Performance Analysis
TCP transfer throughput TCP throughput is network-end-system limited TCP throughput is network-limited
Network Performance Analysis Methodology Performance Analysis Network Architecture Performance Analysis Steps
![Page 3: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/3.jpg)
3
1. Problems
What, Where, and How are the performance bottlenecks of network applications in wide area networks?
How to diagnose network/application performance quickly and efficiently?
![Page 4: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/4.jpg)
4
Disks Operating System
ApplicationsCPU
MEM
Disks Operating System
Network Applications
CPUMEM
Network
R/S
Router RouterCable
NIC NIC
12
3
4
5
6
78
9
Network Application Performance Factors !!!
1’2’
3’
4’
5’
6’
7’
• Network Delay• Bandwidth• Packet Drop Rate
• CPU speed• MEM Size• System Load• Disk I/O Speed • Operating System
• R/W buffer size• Disk cache size
• NIC Speed
End System
R/S
LAN
WAN
Disks Operating System
ApplicationsCPU
MEM
Disks Operating System
Network Applications
CPUMEM
Network
R/S
Router RouterCable
NIC NIC
12
3
4
5
6
78
9
Network Application Performance Factors !!!
1’2’
3’
4’
5’
6’
7’
• Network Delay• Bandwidth• Packet Drop Rate
• CPU speed• MEM Size• System Load• Disk I/O Speed • Operating System
• R/W buffer size• Disk cache size
• NIC Speed
End System
R/S
LAN
WAN
![Page 5: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/5.jpg)
5
2. End-to-End Network/Application Performance Analysis
![Page 6: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/6.jpg)
6
2.1 TCP transfer throughput
An end-to-end TCP connection can be separated into: the sender, the networks, and the receiver.
TCP adaptive windowing scheme consists of a send-window (Ws), congestion-window (CWND), and receive-window (WR). Congestion Control: congestion-window Flow Control: receive window
The overall end-to-end performance of TCP throughput is decided by the sender, the network, and the receiver, which are modeled and symbolized in the sender as Ws, CWND, and WR. Assume the round trip time RTT, the instantaneous TCP throughput at time t:
Throughput (t) = min{Ws(t), CWND(t), WR(t) }/RTT(t)
![Page 7: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/7.jpg)
7
2.1 TCP transfer throughput (cont)
If any of the three windows is small, especially when such conditions last for a relatively long period of time, the overall TCP throughput would be seriously degraded. The TCP throughput is network-end-system-limited for the
duration T, if it has:
The TCP throughput is network-limited for the duration T, if it has:
∫ ∫<<T T
S tdtCWNDtdtW0 0
)()()()( , or
∫ ∫<<T T
R tdtCWNDtdtW0 0
)()()()(
∫ ∫<<T T
S tdtWtdtCWND0 0
)()()()( , and
∫ ∫<<T T
R tdtWtdtCWND0 0
)()()()(
![Page 8: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/8.jpg)
8
2.2 TCP throughput is network-end-system limited
User/Kernel space split Network application in user space, in process context Protocol processing in kernel, in the interrupt context
Interrupt-driven operating system Hardware interrupt -> Software interrupt -> process
User space Kernel Space
send
Send buffer
Socket write
Network application
SB
SλTCP
User spaceKernel Space
receive
Receive buffer
Socket read
Network application
RB
RλTCP
(a) TCP Sender (b) TCP Receiver
![Page 9: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/9.jpg)
9
2.2 TCP throughput is network-end-system limited (cont)
Factors leading to a relatively small window of WS(t) & WR(t) Poorly-designed network application Performance-limited hardware
CPU, disk I/O subsystem, system buses, memory Heavily-loaded network end systems
System interrupt loads are too high Interrupt coalescing, Jumbo Frame
System process load are too high Poorly configured TCP protocol parameters
TCP Send/Receive buffer size TCP window scaling in high speed, long distance networks.
![Page 10: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/10.jpg)
10
2.3 TCP throughput is network-limited
Two Facts: TCP sender tries to estimate the available bandwidth in the
networks, and represents it as CWND with congestion control algorithms.
TCP assumes packet drops are caused by network congestion. Any packet drops will lead to a reduction in CWND.
Two determining factors for CWND Congestion control algorithm Network Conditions (Packet drops)
![Page 11: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/11.jpg)
11
2.3 TCP throughput is network-limited (cont)
TCP congestion control algorithm is evolving Standard TCP congestion control (Reno/NewReno)
Slow start, congestion avoidance, retransmission timeouts, fast retransmit and fast recovery
AIMD scheme for congestion avoidance Perform well in traditional networks Cause under-utilized problem in high-speed and long-distance networks
High-speed TCP variants: FAST TCP, HTCP, HSTCP, BIC, and CUBIC
Modify the AMID congestion avoidance scheme of standard TCP to be more aggressive,
Keep the same fast retransmit and fast recovery algorithm Solve the under-utilized problem in high speed and long distance networks
![Page 12: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/12.jpg)
12
2.3 TCP throughput is network-limited (cont)
With high-speed TCP variants, it is mainly the packet drops that lead to a relatively small CWND
The following conditions could lead to packet drops Network congestion. Network infrastructure failures. Network end systems.
Packet drops in Layer 2 queues due to limited queue size. Packet dropped in ring buffer due to system memory pressure.
Routing changes. When a route changes, the interaction of routing policies, iBGP, and the
MRAI timer may lead to transient disconnectivity. Packet reordering.
Packet reordering will cause duplicate ACKs to the sender. RFC 2581 suggest a TCP sender should consider three or more dupACKs as an indication of packet loss. With severe packet reordering, TCP might misinterpret it as packet losses.
![Page 13: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/13.jpg)
13
2.3 TCP throughput is network-limited (cont)
Congestion window is manipulated on the unit of Maximum Segment Size (MSS). Larger MSS entails higher TCP throughput.
Larger MSS is efficient for both networks and network end systems.
![Page 14: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/14.jpg)
14
3. Network Performance Analysis Methodology
![Page 15: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/15.jpg)
15
Disks Operating System
ApplicationsCPU
MEM
Disks Operating System
Network Applications
CPUMEM
Network
R/S
Router RouterCable
NIC NIC
12
3
4
5
6
78
9
Network Application Performance Factors !!!
1’2’
3’
4’
5’
6’
7’
• Network Delay• Bandwidth• Packet Drop Rate
• CPU speed• MEM Size• System Load• Disk I/O Speed • Operating System
• R/W buffer size• Disk cache size
• NIC Speed
End System
R/S
LAN
WAN
Disks Operating System
ApplicationsCPU
MEM
Disks Operating System
Network Applications
CPUMEM
Network
R/S
Router RouterCable
NIC NIC
12
3
4
5
6
78
9
Network Application Performance Factors !!!
1’2’
3’
4’
5’
6’
7’
• Network Delay• Bandwidth• Packet Drop Rate
• CPU speed• MEM Size• System Load• Disk I/O Speed • Operating System
• R/W buffer size• Disk cache size
• NIC Speed
End System
R/S
LAN
WAN
![Page 16: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/16.jpg)
16
Network Performance Analysis Methodology
An end-to-end network/application performance is viewed as Application-related problems,
Beyond the scope of any standardized problem analysis Network end system problems Network path problems
Network performance analysis methodology Analyze and appropriately tune the network end systems Network path analysis, with remediation of detected problems
where feasible If network end system and network path analysis do not
uncover significant problems or concerns, packet trace analysis will be conducted. Any performance bottlenecks will manifest themselves in the
packet traces.
![Page 17: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/17.jpg)
17
3.1 Network Performance Analysis Network Architecture
7 ’R/S
Router RouterCable
8
9
WAN
`
Network End System
`
BR
BREnd-to-end Path
LAN LANPacket Trace
Diagnosis Server
WAN
Network End System Diagnosis Server
Network PathDiagnosis ServerNESDS
NPDS
PTDS
NES NESDS NPDS
NES
PTDS
NESDS
NPDS
NPDS
NES
BR Border Router
![Page 18: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/18.jpg)
18
Network end system diagnosis server. We use Network Diagnostic Tool (NDT).
collect various TCP network parameters in the network end systems, and identify their configuration problems
Identify local network infrastructure problems such as faulted Ethernet connections, malfunctioning NICs, and Ethernet duplex mismatch.
Network path diagnosis server We use OWAMP applications to collect and diagnose one-way
network path statistics. The forward and reverse path might not be symmetric The forward and reverse path traffic loads likely not symmetric The forward and reverse path might have different Qos schemes
Other tools such as Ping, traceroute, pathneck, iperf, and PerfSONAR etc could be used.
3.1 Network Performance Analysis Network Architecture (cont)
![Page 19: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/19.jpg)
19
3.1 Network Performance Analysis Network Architecture (cont)
Packet trace diagnosis server Directly connect to the border router, can port-mirror any port
in the border router TCPDump, used to record packet traces TCPTrace, used to analyze the recorded packet traces Xplot, used to examine the recorded traces visually
![Page 20: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/20.jpg)
20
Step 1: Definition of the problem space Step 2: Collect of network end system information &
network path characteristics Step 3: Network end system diagnosis Step 4: Network path performance analysis
Route changes frequently? Network congestion: delay variance large? Bottleneck location? Infrastructure failures: examine the counter one by one Packet reordering: load balancing? Parallel processing?
Step 5: Evaluate packet trace pattern
3.2 Network/Application Performance Analysis Steps
![Page 21: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/21.jpg)
21
Collection of network end system information
CPU CPU speed; CPU numbers;
Memory Memory size; memory latency;
Bus Maximum bus bandwidth; Disk Maximum d isk I/O bandwidth; Hardware
NIC Maximum bandwidth; I nterrupt coalesc ing supported? TCP off loading supported? Jumbo frame supported?
Operating system
Operating system type, versions; 32bit/64bit? For Linux, identify the kernel version;
System Loads
Network applications running context; Maximum system background loads;
Network Applications
Network application traffic generation pattern; Storage system involved?
TCP Parameters
Send/Receive (Socket) buffer size; Timestamp option enabled? Window scaling option enabled? Window scaling parameter;
TCP reordering t hreshold; Congestion control algorithm; Total TCP memory size; Maximum Segment Size; SACK enabled? D -SACK enabled? ECN enabled?
Software
NIC Driver Parameter
Device driver send/recv queue size; TCP offloading enabled? Interrupt coalescing enabled? Jumbo Frame e nabled?
Table 1 Network End Systems Information
![Page 22: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/22.jpg)
22
Collection of network path characteristics
Network path characteristics Round-trip time (ping) Sequence of routers along the paths (traceroute) One-way delay, delay variance (owamp) One-way packet drop rate (owamp) Packet reordering (owamp) Current achievable throughput (iperf) Bandwidth bottleneck location (pathneck)
![Page 23: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/23.jpg)
23Traffic trace from Fermi to OEAW What happened?
![Page 24: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/24.jpg)
24Traffic trace from Fermi to Brazil What happened?
![Page 25: Wide Area Network Performance Analysis Methodology](https://reader036.vdocuments.net/reader036/viewer/2022062409/568145c2550346895db2cc4f/html5/thumbnails/25.jpg)
25
Conclusion Fermilab is working on developing a performance analysis
methodology Objective is to put structure into troubleshooting network
performance problems Project is in early stages of development
We welcome collaboration & feedback Biweekly Wide-Area-Working-Group (WAWG) meeting on
alternate Friday mornings Send email to [email protected]