never been kist: tor’s congestion management blossoms with ... · never been kist: tor’s...

Never Been KIST: Tor’s Congestion Management Blossoms with Kernel-Informed Socket Transport

23rd USENIX Security Symposium August 20th 2014

Rob Jansen US Naval Research Laboratory John Geddes University of Minnesota Chris Wacek Georgetown University Micah Sherr Georgetown University Paul Syverson US Naval Research Laboratory

Anonymous Communication: Tor

Tor is Slow!!! Research* ●  PCTCP: Per-Circuit TCP-over-IPsec Transport for Anonymous Communication

Overlay Networks (CCS ‘13)

●  Reducing Latency in Tor Circuits with Unordered Delivery (FOCI ‘13)

●  How Low Can You Go: Balancing Performance with Anonymity in Tor (PETS ‘13)

●  The Path Less Travelled: Overcoming Tor's Bottlenecks with Traffic Splitting (PETS ’13)

●  An Empirical Evaluation of Relay Selection in Tor (NDSS ‘13)

●  LIRA: Lightweight Incentivized Routing for Anonymity (NDSS ‘13)

●  Improving Performance and Anonymity in the Tor Network (IPCCC ‘12)

●  Enhancing Tor's Performance using Real-time Traffic Classification (CCS ’12)

●  Torchestra: Reducing interactive traffic delays over Tor (WPES ‘12)

●  Throttling Tor Bandwidth Parasites (USENIX Sec ‘12)

●  LASTor: A Low-Latency AS-Aware Tor Client (Oakland ‘12)

●  Congestion-aware Path Selection for Tor (FC ‘12) *Not a comprehensive list

Tor is Slow!!! Research*

*Not a comprehensive list

Where?

This Talk

●  Where is Tor slow? –  Measure public Tor and private Shadow-Tor networks –  Identify circuit scheduling and socket flushing problems

●  Design KIST: Kernel-Informed Socket Transport –  Use TCP snd_cwnd to limit socket writes

●  Evaluate KIST Performance and Security –  Reduces kernel and end-to-end circuit congestion –  Throughput attacks unaffected, speeds up latency attacks

Outline

●  Background

●  Instrument Tor, measure congestion

●  Analyze causes of congestion

●  Design and evaluate KIST –  Performance –  Security

Relay Overview

Relay Overview

TCP

TCP

TCP

TCP

TCP

TCP

TCP

Tor circuits are multiplexed over a

TCP transport

Relay Overview

TCP

TCP

TCP

TCP

TCP

TCP TCP

Relay Internals Kernel Input Kernel Output Tor Input Tor Output

Tor Circuits

Opportunities for traffic

management

Outline

●  Background




Live Tor Congestion - libkqtime Kernel Input Kernel Output Tor Input Tor Output

Tor Circuits


Tor Circuits

tag match tag match


Tor Circuits

tag match tag match track cells

Shadow Network Simulation

●  Enhanced Shadow with several missing TCP algorithms

–  CUBIC congestion control –  Retransmission timers –  Selective acknowledgements (SACK) –  Forward acknowledgements (FACK) –  Fast retransmit/recovery

●  Designed largest known private Tor network –  3600 relays and 12000 simultaneously active clients –  Internet topology graph: ~700k nodes and 1.3m links

Track the UID

Shadow-Tor Congestion – UIDs UID

UID Track the UID

Track the UID

Shadow-Tor Congestion – UIDs UID

UID Track the UID

Kernel Input Kernel Output Tor Input

Tor Output

Tor Circuits

Tor and Shadow-Tor Congestion

10�1 100 101 102 103 104

Time (ms)

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ulat

ive

Frac

tion

kernel in

tor

kernel out

10�1 100 101 102 103

Time (ms)

0.5

0.6

0.7

0.8

0.9

1.0

Cum

ulat

ive

Frac

tion

kernel in

tor

kernel out

Congestion occurs almost exclusively in outbound kernel buffers

Shadow-Tor Live-Tor

Outline

●  Background




Analyzing Causes of Congestion

Kernel Output Tor Output Tor Circuits

Queuing delays in kernel output buffer

Analyzing Causes of Congestion



Problem 1: Circuit scheduling

Problem 2: Flushing to Sockets

Problem 1: Circuit Scheduling


Libevent schedules one connection at a time




Tor only considers a subset of writable

circuits




Tor only considers a subset of writable

circuits

Circuits from different connections are not prioritized correctly


0.0 0.2 0.4 0.6 0.8 1.0Throughput (KiB/s)

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ulat

ive

Frac

tion

160 180 200 220 2400.00.20.40.60.81.0

DE

TER

pri+

rr

pri-

0.00.20.40.60.81.0

Shadow

0.0 0.2 0.4 0.6 0.8 1.0Throughput (KiB/s)

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ulat

ive

Frac

tion

40 60 80 100 120 140 1600.00.20.40.60.81.0

DE

TER

pri+

rr

pri-

0.00.20.40.60.81.0

Shadow

Correctly differentiated No differentiation


0.0 0.2 0.4 0.6 0.8 1.0Throughput (KiB/s)

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ulat

ive

Frac

tion

160 180 200 220 2400.00.20.40.60.81.0

DE

TER

pri+

rr

pri-

0.00.20.40.60.81.0

Shadow

0.0 0.2 0.4 0.6 0.8 1.0Throughput (KiB/s)

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ulat

ive

Frac

tion

40 60 80 100 120 140 1600.00.20.40.60.81.0

DE

TER

pri+

rr

pri-

0.00.20.40.60.81.0

Shadow

Correctly differentiated No differentiation

99.775% of any two circuits are unshared




FIFO



Worse priority traffic (high throughput flows) FIFO



Better priority traffic (low throughput flows)





Must wait for kernel to flush socket to network (blocked

on TCP cwnd)





Reduces effectiveness of circuit priority


Outline

●  Background




Ask the kernel, stupid!

●  Utilize getsockopt and ioctl syscalls

socket_space = sndbufcap – sndbuflen

tcp_space = (cwnd – unacked) * mss

sndbuflen

sndbufcap

unacked

cwnd

Kernel-Informed Socket Transport

●  Don’t write it if the kernel can’t send it; bound kernel writes by:

–  Socket: min(socket_space, tcp_space) –  Global: upstream bandwidth capacity

Solution to Problem 2




●  Choose globally from all writable circuits

Solution to Problem 1




●  Choose globally from all writable circuits

●  Try to write again before kernel starvation

KIST Reduces Kernel Congestion

10�1 100 101 102 103

Time (ms)

0.0

0.2

0.4

0.6

0.8

1.0C

umul

ativ

eFr

actio

n

vanilla

global

KIST

KIST Increases Tor Congestion

10�1 100 101 102 103

Time (ms)

0.0

0.2

0.4

0.6

0.8

1.0C

umul

ativ

eFr

actio

n

vanilla

global

KIST

KIST Reduces Circuit Congestion

0 500 1000 1500 2000Time (ms)

0.2

0.3

0.4

0.5

0.6

0.7

0.8C

umul

ativ

eFr

actio

n

vanilla

global

KIST

KIST Improves Network Latency

0.0 0.5 1.0 1.5 2.0 2.5 3.0Time to First Byte (s)

0.0

0.2

0.4

0.6

0.8

1.0C

umul

ativ

eFr

actio

n

vanilla

global

KIST

0.5 1 2 4 80.4

0.6

0.8

1.0

Outline

●  Background




Traffic Correlation: Latency

Hopper et.al. CCS’07

Goal: narrow down potential locations of the client on a target circuit



-Inject redirect or javascript -Start timer


GET


Request redirected page or embedded object


GET


-Stop timer -Estimate latency

Latency Attack | estimate – actual |

0 50 100 150 200 250Difference (ms)

0.0

0.2

0.4

0.6

0.8

1.0C

umul

ativ

eFr

actio

n

vanilla

KIST

Latency Attack num pings until best estimate

0 500 1000 1500 2000Cumulative Number of Pings

0.0

0.2

0.4

0.6

0.8

1.0C

umul

ativ

eFr

actio

n

vanilla

KIST

Traffic Correlation: Throughput

Mittal et.al. CCS’11

Goal: find guard relay of the client on a target circuit



Probe throughput of all guard relays



Correlate throughput between

exit and probes

Throughput Attack Results

�0.4�0.3�0.2�0.10.0 0.1 0.2 0.3 0.4 0.5Correlation Score

0.0

0.2

0.4

0.6

0.8

1.0C

umul

ativ

eFr

actio

n

vanilla

KIST

Conclusion

●  Where is Tor slow?

●  KIST complements other performance enhancements, e.g. circuit priority

●  Next steps –  Currently exploring various algorithmic optimizations –  Test KIST in the wild and deploy in Tor

Questions?

[email protected] robgjansen.com

github.com/robgjansen/libkqtime github.com/shadow

think like an adversary


Tor Circuits

Network Input


Tor Circuits

Split data into socket buffers


Tor Circuits

Read data from sockets into Tor


Tor Circuits

Process data (encrypt/decrypt)


Tor Circuits

Split cells into circuit queues


Tor Circuits

Circuits linked to outgoing connection


Tor Circuits

Schedule cells


Tor Circuits

Write data from Tor into sockets


Tor Circuits

Schedule data for sending


Tor Circuits

Opportunities for traffic

management

KIST Improves Network Throughput

650 700 750 800 850 900 950Throughput (MiB/s)

0.0

0.2

0.4

0.6

0.8

1.0C

umul

ativ

eFr

actio

n

vanilla

global

KIST

never been kist: tor’s congestion management blossoms with ... · never been kist: tor’s...

Documents