circuit emulation for bulk transfers in distributed storage and clouds

Setting the Mood

• "It's time to get rid of TCP/UDP protocols in DCs"

• DCs/Clouds are closed worlds, brand new technologies are OK

• with bulk transfers (BigData, ...), the business value of a TCP/UDP alternative is high

• circuits are an alternative to packets

M.Zhanikeev -- maratishe@gmail.com -- Circuit Emulation for Bulk Transfers in Dist. Storage and Clouds -- http://bit.do/marat140903 2/32...

Ethernet is the Best

.Ethernet.....

... is the cheapest and most available technology with e2esupport

• Fiber Channel (FC), SATA, etc. require expensive hardware, lowcompatibility, no e2e support

• FCoE = Ethernet, same problems, expensive hardware, no e2e support

• network virtualization is best fit for Ethernet

• disclaimer: one of proposed models will work with optical networks aswell

Ethernet is the Worst

.Ethernet.....

.... is the worst technology in terms of throughput• CSMA/CD is the biggest throughput limitation

◦ not in modern switches, but still major problem in wireless

• contention problem cannot be easily resolved

• same applies to OBS/OPS optical technologies

Ethernet Contention

Ethernet and Contention

• whaterver you do, Ethernet L2 domains cannot avoid contention

Switch Switch

Qualitatively Identical

Parallel vs Sequential (2 flows)

20 24 28 32 36 40Transfer time in contention (s)

Ethernet Switches : Basic Facts

• cut-through versus store-and-forward• cut-through is 10..15x better

• Cisco has advanced cut-through : +bytes versus routing decision tradeoff

• store-and-forward is subjected to QoS classes◦ L3 DSCP versus L2 CoS, AF, EF, BE, SBE models

Switchess : Modeling

C: Cut Through

Check, etc. Q: Queue

D: Drop QoS classes

Proposal

Proposal : Circuits

.Circuits..

... are emulations which allow for exclusive access to L2 domain byindividual parties

• circuits-over-packets emulation

• cut-through mode for each circuit is guaranteed

• highest possible throughput

• NOTE: will work with cheepest switches

• NOTE2: applies to optical networks as well (L2=lightpaths)

Implementation : 2 cases• left: book-then-send, right: separate control layer

SWITCH

Storage Node A

Storage Node B

Step 1: Book

session

Step 2: Transfer bulk

SWITCH

Storage Node A

Storage Node B

SWITCH

Bookingsegment

BulkSegment

Impl.: Centralized Case

SWITCH

Storage Node A

Storage Node B

Step 1: Book

session

• same network for booking andcircuits

• inefficient but still valid/practical

• legacy-compatible,partial implementation, etc.

Impl.: Distributed Case

SWITCH

Storage Node A

Storage Node B

SWITCH

Bookingsegment

BulkSegment

• book on one network, send on another

• legacy-incompatible• contention-sensing possible →fully distributed models

• can also use sensing andcontention control

Optimization

Optimization : Basics

• same for distributed and centralized models◦ does not matter, optimization shows the overall utility of a heuristic

• practical optimization = formulation + heuristic• given: demand matrix

• expected result: a routing table mapping demand to topology

Optimization : Basics

Optim. : OSPF → tuple notation

• OSPF is traditional in such optimizations, but too rigid for many practical cases◦ too complex for lightpaths in optical networks◦ no good heuristics for complex topologies

• OSPF notation is not very convinient1. capacity constraints2. flow preservation3. contention/congestion metrics

• alternative: tuples ... for example ⟨s, d, v, t⟩ defines demand of traffic

volume v at time t from source s to destionation d◦ this notation ismuch more flexible for several coming formulations

Optim. : Basic Tuple Notation

• nodes: source s, destination: d and others a, b, c• individual demand tupleTi = ⟨s, d, v, t⟩• lightpathλ for optical networks

• time t, can be start time, start and end of a period, etc.

• we do not care about utility so far, just the notation, but utility is obvious inmost cases

• → means results in... or leads to...

tOSPF : Traditional OSPF

Ti = ⟨s, d, v, t⟩ → ⟨s, a, b, ..., d⟩.Externals..

Using demand matrix, creates a set of per-linkweights, which define a unique route for eachdemand item.

.Internals..

Per-link capacity constraint, in/out flowconservation constraint, unstable for largetopologies and demand matrices

• s source

• d destination

• a, b, c, ... intermediatenodes on e2e paths/routes

oOSPF : Optical OSPF w/out Switching

Ti = ⟨s, d, v, t⟩ → ⟨s, λ⟩.Externals..

Using demand matrix, maps each demand item onisolated lightpath

.Internals..

Simple but inefficient because the number ofe2e lightpaths is small

• s source

• d destination

• λ a wavelength for a fixed e2elightpath from s to destination

oOSPFs : Optical OSPF with Switching

Ti = ⟨s, d, v, t⟩ → ⟨s, λs, λa, λb, ...⟩.Externals..

Using demand matrix, maps each demand item on aroute of wavelengths

.Internals..

Efficient, but suffers from the same problemsas traditional OSPF

• s source

• d destination

• λx an exit wavelength at agiven node x

Proposal : Sensing Formulation

Ti = ⟨s, d, v, t1, t2⟩ → ⟨s, λ, t⟩.Externals..

Using a matrix of loosely scheduled demand, createa schedule of sequential sessions withexlusive access to paths

.Internals..

Same approach for Ethernet (one wavelength) andoptical networks

• s source

• d destination

• t1 and t2 areuser-preferred range forthe start of a session, a valuet is picked between them

Heuristics

Centralized Case

SWITCH

Storage Node A

Storage Node B

Step 1: Book

session

• all optimization formulations exceptsensing

• very close to traditional OSPF• same problems as in OSPF

• the biggest problem is to knowdemand matrix in advance

Distributed Case

SWITCH

Storage Node A

Storage Node B

SWITCH

Bookingsegment

BulkSegment

• can be used for all formulations

• pefectly suited for the Sensingformulation

The Sensing Model• contention methods in wireless and OBS will work

◦ in practice: sensing can beSNMP-like feedback on gate's status◦ no sync among users is necessary

• same model for Ethernet (+virtual nets) and optical networks

• main advantage: the offload, no need to implement funny OSPFheuristics

Realistic Gate/Sensing Model

• an approximate view of JGNtopology

• two way = one way + ring• Gates are created at optical/ethernet border

• NOTE: already working for Ethernet

Wrapup

• circuit emulation is necessary for effective bulk transfers◦ up to 40% faster in our lab tests

• intra-DC, DC-DC, federations, etc. -- all can benefit from circuits

• circuits formulated as OSPF are bad -- a Gate/Sensing model is better• validity: worst case is the existing technology, but upper performancebound is very high

That’s all, thank you ...

[01] myself (2014)High Availability Cloud Storage...NS研

[02] Cisco (2014)LAN Switching and Wireless, CCNA Exploration Companion GuideCisco Press

[03] Cisco (2014)Cut-Through and Store-and-Forward Ethernet Switching for Low-Latency....Cisco Press

[04] NetOptics (2014)Cut-Through Ethernet Switching: A Versatile Resource for Low Latency...White Paper

[05] Cisco (2006)QoS: DSCP Classification GuidelinesRFC4594

[06] Cisco (2010)A Differentiated Services Code Point (DSCP)...RFC5865

[07] open source (current)PICA8 Project for Low Latency Virtual Networkinghttp://www.pica8.com/

Wait-n-Send Model

Bulk size per transmission

Goodput

2 potential distributions in practice

Response curve(s)

Utility of Waiting (curve)

• I called it Wait-n-SeeCurve

• source waits for some time forexclusive access --sensing and accumulating bulk

• on timeout, the current bulkis released at best effort(fallback)

circuit emulation for bulk transfers in distributed storage and clouds

circuit emulation

bulk transfers bigdata

ethernet switches

ethernet disclaimer

ethernet contentionm

ethernet l2 domains

forward cut

distributed models

Technology

brain emulation challenges

clouds, clouds, and more clouds!

clouds clouds everywhere

uvi emulation one |...

emulation unit v

ensemble emulation

host card emulation

mimo ofdm emulation

emulation – outline -...

data storage, transfers and communication in personal clouds

dhcp emulation

upscale emulation

tim o’brien emulation:

dcs emulation software

nebulapresets emulation preset

logic emulation - yonsei

8086 emulation

nctuns emulation

clouds weight the of a word. clouds clouds clouds

emulation error recovery