re-architecting congestion management in losslessethernet · re-architecting congestion management...

Re-architecting Congestion Management in

Lossless EthernetWenxue Cheng, Kun Qian, Wanchun Jiang(CSU), Tong Zhang, Fengyuan Ren

NNS group @ Department of Computer Science and Technology, Tsinghua University

Data Center Networks

Large Scales> 10000 machines

Short Messages< 1 KB

Special ProtocolsRDMA

Small RTT< 100𝜇𝑠

High Bandwidth10/40~100/400 Gbps

Shallow Buffer< 30 MB for ToR

Packets Loss ↑

Performance ↓

Data Center Networks

Small RTT< 100𝜇𝑠

High Bandwidth10/40~100/400 Gbps

Shallow Buffer< 30 MB for ToR

Large Scales> 10000 machines

Short Messages< 1 KB

Special ProtocolsRDMA

Lossless Ethernet

Priority-based Flow Control (PFC)

XOFF XON

Upstream Port(Sender)

Downstream Port(Receiver)

PAUSE


PAUSE

XOFF XON




PAUSE

XOFF XON



Congestion Spreading

RESUMERESUME


XOFF XON



PFC IssuesCongestion Spreading & Head-of-Line Blocking

F0F1

Burst

H0

H1

H2

R0

R1

S0 S1

… H15

P2P1

P0

Congestion tree from P2 to H0 and H1.

F0 is a victim flow.

Congestion Control Schemes

F0F1

Burst

H0

H1

H2

R0

R1

S0 S1

… H15

P2P1

P0

Congestion control schemes are needede.g. QCN[IEEE 802.1], DCQCN[RoCEv2] and TIMELY[SIGCOMM 2015].

Congestion NotificationRate Adjustment

Time

Rate

Experimental Observation

Time (ms)0 3

H0H1H2…

H15

16 Messages of 64KB


(1) Congestion spreading still exists.

Sending Rate of F1

Evolution-based rate decrease is slower than PFC’s effect.


Sending Rate of F0 (Gbps)

(1) Congestion spreading still exists.(2) F0 is also victimized by CC.

PFC infects congestion detection of congestion control schemes.


(1) Congestion spreading still exists.(2) F0 is also victimized by CC.(3) Rate recovery is inadaptable to

dynamic network conditions.

Liner rate increase method and tuning parameters.

Basic Idea

Re-architecting Congestion Management

• Congestion Flows ⟷Victim Flows

Congestion Detection

• Fast Rate Decrease• Automatic Rate Increase

Rate Adjustment


Real-Congestion (P2)

∑𝑅 > 𝐶

Non-Congestion (P3)

∑𝑅 < 𝐶

Quasi-Congestion (P0)

RESUME RESUMEPAUSE

∑𝑅 ?𝐶


Quasi-Congestion (P0)1 1 1 1

RESUME RESUMEPAUSE

Real-Congestion (P2)1 1 1 11 1

Explicit Congestion Notification (ECN)• Only based on queue length• Fail to distinguish quasi-congestion and

real-congestion


Non-Paused ECN (NP-ECN)• Don’t change ECN for packets that has

been paused• Counter PN: number of packets that

has been paused



Real-Congestion (P2)Non-Paused ECN (NP-ECN)• Don’t change ECN for packets that has


has been paused

PAUSERESUMEPN=3

0 0 0PN=2PN=1PN=0





has been paused

PAUSERESUMEPN=3PN=2PN=1PN=0

1 0 0 0





has been paused

PAUSERESUMEPN=3

0 0PN=2PN=1



RESUME RESUMEPAUSE


Partially marked with ECN



has been paused



RESUME RESUMEPAUSE

Real-Congestion (P2)111111



has been paused PN=0



RESUME RESUMEPAUSE


Continuously marked with ECN



has been paused



RESUME RESUMEPAUSE


Continuously marked with ECN

Partially marked with ECNNon-Paused ECN (NP-ECN)• Don’t change ECN for packets that has


has been paused

Victim Flows

Congested Flows

Rate Adjustment

How to adjust the rates of • Congested Flows --> target?• Victim Flows --> no decrease?• Non-congested Flows

Burst = 40𝐺𝑏𝑝𝑠, F0 = 20𝐺𝑏𝑝𝑠,Reduce F1’s rate

H4

F0 = 20𝐺𝑏𝑝𝑠, Reduce F1’s rate

Rate Adjustment

How to adjust the rates of • Congested Flows è reduce to receiving rate immediately • Victim Flows & Uncongested Flows à rate increase

Rate Adjustment

How to adjust the rates of • Congested Flows è reduce to receiving rate immediately • Victim Flows & Uncongested Flows à rate increase

Receiver-Driven Rate Decrease• 𝑠𝑒𝑛𝑑𝑅𝑎𝑡𝑒 ← min 𝑠𝑒𝑛𝑑𝑅𝑎𝑡𝑒, 1 − 𝑤IJK 𝑟𝑒𝑐𝑅𝑎𝑡𝑒• No PFC & no serious throughput loss & 1 control loop

Rate Adjustment

How to adjust the rates of • Congested Flows è reduce to receiving rate immediately • Victim Flows & Non-congested Flows à rate increase

Receiver-Driven Rate Decrease• 𝑠𝑒𝑛𝑑𝑅𝑎𝑡𝑒 ← min 𝑠𝑒𝑛𝑑𝑅𝑎𝑡𝑒, 1 − 𝑤IJK 𝑟𝑒𝑐𝑅𝑎𝑡𝑒• No congestion & No PFC triggers in one control loop

Self-weighted Rate increase

• N𝑠𝑒𝑛𝑑𝑅𝑎𝑡𝑒 ← 𝑠𝑒𝑛𝑑𝑅𝑎𝑡𝑒 1 − 𝑤 +𝑀𝑎𝑥𝑅𝑎𝑡𝑒 R 𝑤𝑤 ← 𝑤 1 − 𝑤 +𝑤IST R 𝑤

• Automatic gentle-to-aggressive

Photonic Congestion Notification (PCN)

• Receiver-driven Rate Decrease

• Self-weighted Rate increase

• NP-ECN• Identify

Congested Flows • Rate Estimator

SenderSwitches

Receiver

Congestion Notification Packet (CNP)

Period T

PCN’s Benefit

Time (ms)0 3

H0H1H2…

H15

16 Messages of 64KB

No congestion tree

No serious throughput loss

Benefit

Time (ms)0 3

H0H1H2…

H15

16 Messages of 64KB



F1 is reduced in one loop

F0 is not victimized by PCN

Evaluation SetupTestbed Setup

• Dumbbell topology • Implementation on DPDK (Intel 82599)• 4 hosts (PowerEdge R530) connected to single ToR• 10Gbps

NS-3 Simulation Setup• Clos topology• 512 hosts / 32 ToRs / 16 Leafs / 8 Spines• 10Gbps / 40Gbps

Evaluations

Basic Prosperities

• Convergence• Fairness• Stability

Workbench

• Burst Tolerance• Parameter

sensitivity• Realistic

Workloads

Special Cases

• Flow Scalability• Adversarial Traffic• Multiple

Bottlenecks• Multiple Priorities• DeadlockTestbed

NS-3 Simulations

Evaluation: Large-Scale SimulationsSimulation Setup

W1: Web-server workloadW2: Hadoop cluster workload

Pod 0 Pod 7

512 hosts

Evaluation: Large-Scale Simulations

Web-server Workload

PAUSE Rate (Mbps) Flow Complete Time (ms) Flow Complete Rate (Kps)

Evaluation: Large-Scale Simulations

Hadoop Workload

PAUSE Rate (Mbps) Flow Complete Time (ms) Flow Complete Rate (Kps)

Conclusion

Evaluations on testbed and ns-3 simulation show, PCN triggers fewer PFC and achieves lower flow completion time.

Re-architecting congestion management

Proposing Photonic Congestion Notification (PCN)• NP-ECN à victim flows/congested flows• Receiver-driven rate decrease à no PFC in 1 loop• Automatic rate increase

Thanks !

[email protected]@tsinghua.edu.cn

http://126.com

http://tsinghua.edu.cn

re-architecting congestion management in losslessethernet · re-architecting congestion management...

Documents