Technology Evaluation for Time Sensitive gyData TransportReport and status for subtask in JRA1GN3 JRA1 Workshop, 20st-22nd of November, Copenhagen

Henrik Wessing, Task leader

Time Sensitive Transport - Agenda• Motivation and objectives• Motivation and objectives

– ”Is it worth establishing circuit based paths or do we survive with the packet based solutions?”

• Technologies and layering• Technologies and layering• Delay verification through test equipment and OAM• Physical layer• OTN layer• OTN layer• MPLS-TP and PBT• Measurements • Conclusions and continued work• Conclusions and continued work

Low delay is a key property• Synchronising atomic clocks• Synchronising atomic clocks• Online music production (LOLA)

– Low Latency audio and video conferencing– Delay < 35-40 ms (for network) y ( )– Jitter < 3 ms– Characterised by many small packets

• Telemedical surgery

Picture from presentation by Claudio Allochio, GARR

– Roundtrip delay < 150 ms– Gb/s bandwidth due to no compression

• Interconnection of data centres– NREN operating as network for DC– NREN operating as network for DC.

• Banking– Stock exchange algoritms can utilise reduction of delays

in msec and usec range– Not a clear NREN task!– According to The Telegraph:

• ”… that a one millisecond advantage could be worth up to $100m a year to the bottom line of a large hedge fund.”

And more and more services emerges

Remote backup

Delay sensitiveDelay tolerant

ve

• Variety of services

3D streamingInternet surfing with

videoInternet surfing

Grid computingThin ClientsTele Health

or S

ensi

tivB

andw

idth

• Different requirements

• End to end QoS in focus

EmailMobile gaming

Ultra HD TV

Err B

End to end QoS in focus

• Delay key issues

Ultra HD TVStereoscopic TV

Video ConferencingIP Telephony

(U)HD Video StreamVideo Streaming

ndw

idth

r to

lera

nt

Online Distributed Environments

Home monitoring

Ban

Erro

r

Technologies investigatedC t k / M t t k• Campus networks / Metro networks

– Traditionally packet based (L3/L2) depending on operational use• Core / NRENs (GEANT)

P k t i itb d– Packet or circuitbased• Methodology

– Layered approach from physical layer and up– Qualifying and quantifying delay parameters

Service (IP)

Layer 2 (ETH / MPLS-TP)

L1 S b l th l (OTN ODUk)L1 Subwavelength layer (OTN - ODUk)

L0 DWDM layer

Measuring delay – External test or OAM• Delay measurements using external test equipment• Delay measurements using external test equipment• Delay measured between Maintenance association End Points

– Different levels acoording to domain

EF

35

From

M

OAM – Delay and loss statistics• Metro Ethernet Forum specifies delay statistics (MEF 10 2)• Metro Ethernet Forum specifies delay statistics (MEF 10.2)

• Frame delay (range/mean/jitter) - histograms• Frame Loss Ratio - counters• Availability

• One way FD utilising timestamps and sequence numbers (MEF 35 IA)• One way FD utilising timestamps and sequence numbers (MEF 35 IA)– Real data or as added synthetic frames– Synchroneous clocks required – (or estimated from two way FD)

• Performance Monitoring solutionsChoice depending on single or dual ended– Choice depending on single or dual ended

– Synthetic frames must match real frames• Communication messages

– Delay Measurements Message (DMM)/ Delay Measurement Response (DMR)R lt t d i bi f h t i t l• Results stored in bins for each measurement interval

PMSolution

MEGType(s)

MeasurementTechnique for Loss

PM Function(s) Mandatory or Optional

PM‐1point‐to‐point multipoint

SyntheticTesting

Single‐Ended DelaySingle‐Ended Synthetic Loss

Mandatory

PM‐2point‐to‐pointmultipoint

n/a Dual‐Ended Delay Optional

PM 3 i t t i tCounting

Si l E d d S i L O ti lPM‐3 point‐to‐pointg

Service FramesSingle‐Ended Service Loss Optional

Dedicated test equipment

• Agilent (now Ixia) N2X test solution• Main advantage: You define which packet sizes and load to test with• Main advantage: You define which packet sizes and load to test with• RFC2544 compliant• Measuring loss, signal power, delay, jitter etc.• From services to fibres• From services to fibres

Physical layer (L0)• Physical layer (L0)• Physical layer (L0)

– Propagation delay in fiber. Well defined.• 1 km approximately 5 usec

DCF adds delay May use DCM– DCF adds delay. May use DCM.– Delay in modulation formats depending on

technology in sub usec range• QPSK QAM OFDMA • QPSK,QAM, OFDMA …

– Transponders/Muxponders• ~ 5-10 usec

For medium to long range applications:

Propagation is the only significant L0 delay

Optical Transport Network (OTN) – L1• Physical Media Independent layer (L1) general• Physical Media Independent layer (L1) - general

– Mapping from higher layer to L1 – Forward Error Correction

OTN• OTN– Basically a technology to replace SDH/SONET better suited for

ethernet transport.Timestamping for Delay Measurement Message and Delay– Timestamping for Delay Measurement Message and Delaymeasurement reply (DMM/DMR)

– Vendor support: All major vendors support ODU switching

OTN (II) – Measurement scenarios• Lab measurements using Alcatel Switches• Lab measurements using Alcatel Switches

– Triangular setup• CPH - CPH -HAM

• Loops in triangle to estimate delays from

Loops intriangle

Delay contribution

1 1 GFP mapping and demapping

Total fibre propagation delayp g ymapping and FEC processing

• Expected delays– usec range (OTN mapping and processing)

Res lts sing e te nal N2X teste and OAM

2 x OTN switching latency

2 1 GFP mapping and demapping

T t l fib ti d l• Results using external N2X tester and OAM information

Total fibre propagation delay

Triangle fibre propagation delay

5 x OTN switching latency

3 1 GFP mapping and demapping

Total fibre propagation delay

OTN switchN2X tester

2 x Triangle fibre propagation delay

8 x OTN switching latency

OTN switchOTN switch OTN switchOTN switch

L2 – MPLS TP and PBT – Packet handling• MPLS TP• MPLS TP

– Two flavours: MPLS-TP and T-MPLS – No signalling protocol– Legacy Ethernet can be encapsulated in PW transported over MPLS LSP

Delay contributions– Delay contributions• Label processing• Store and forward or wire speed as no CRC• Lower priority packet may delay if in process

Payload =

FCS

l d

FCS

FCS

• PBT– Ethernet for transport purposes

VLAN Q i Q MAC i MACTPID

=802.1adframewith orwithout

FCS

I-TAG

TPID

Payload

C-TAG

TPIDPayload

TPIDPayload

FCS

16 bits

32 bits

– VLAN -> Q-in-Q -> MAC-in-MAC– Delay contributions

• MAC table lookup• Store and forward due to CRC B-DA

TPID

TPID

B-TAG

B-SA

DA

TPID

TPID

S-TAG

SA

TPID

DA

TPID

Q-TAG

SA

TPID

DA

SA

48 bits

48 bits

16 bits

16 bits

16 bits

– Dependent on packet length• Lower priority packet may delay if in process

PBT Lab scenarios

PBT switch

Simple PBTDelay in 1, 2, 3, 4 and 5 switchesResults next page

PBT switch

N2X tester PBT switch

s tc p g

PBT switchPBT switch

PBT over OTNMapping between OTN and PBTMapping between OTN and PBT- Crossconnect version- Terminate in PBT version

PBT measurements – Simple PBT

100,00

120,00

140,00

64 bytes

512 bytes

1500 bytes

60,00

80,00

0,00

20,00

40,00

• Delay in usec depending on number of PBT switches• For Jumbo frames of 9000 appr 80 usec per node (400 usec for 5 nodes)

0 1 2 3 4 5

• For Jumbo frames of 9000 appr. 80 usec per node (400 usec for 5 nodes)• Clear dependence on packet length• Processing max ethernet size packet equals 4-5 km transmission!!

For Jumbo: 16 km transmission!– For Jumbo: ~16 km transmission!

Dependence on loadp

• All numbers in usec

25

• Jumbo frames included only in text

• Vendor specific

15

20 64

512

1500

• Vendor specific• 1Gbps• Graph: Delay dependent on

10

loadDelay

Load 64 512 1500 900010 9,203 13,77 22,741 82,78520 9 208 13 779 22 743 82 787

0

520 9,208 13,779 22,743 82,78730 9,221 13,784 22,748 82,78140 9,214 13,766 22,74 82,77950 9,232 13,781 22,732 82,77360 9,233 13,781 22,732 82,776

10 20 30 40 50 60 70 80 90 100, , , ,

70 9,325 13,783 22,746 82,78780 9,439 13,795 22,74 82,79190 9,561 13,818 22,746 82,775

100 10,157 14,742 23,602 83,554

Transpacket FUSION H1 devicesTranspacket FUSION H1 devices

Allows wavelength-grade Quality of Service (QoS).– Ultra-low latency ultra-low latency variation– Ultra-low latency, ultra-low latency variation– Combined circuit and packet switching

• GST: Guaranteed traffic (circuit based)• SM: Statistically multiplexed BE traffic

1e+05

1e+06

(us)

GST SM

y p

Inputs Outputs

High P.High P.

Inputs Outputs

High P.High P. 1e+03

1e+04

1e+05

Ave

rage

pac

ket d

elay

(us

Sync problem:Packet delay variation 

Low P.

SM (Low P.)No PDVultra‐low latency, zero packet loss

Sync problem:Packet delay variation 

Low P.

SM (Low P.)No PDVultra‐low latency, zero packet loss

1e+02 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

A

Normalized lightpath load (10GE)

Measurement with GST and SM

Inputs Output

( )

GST

u a o a e cy, e o pac e oss

Inputs Output

( )

GST

u a o a e cy, e o pac e oss

GST and M

Measurement with GST and SM through 3 switches with long fibre spansProp. delay: 266 usec

No SM imSpact on GST flow. SM only inserted if ”gap” is large enough.

p ySwitching delay: 45 usecNo delay variation for GST

Conclusion and continued work• Subtask within GN3 JRA1 to identify and provide guidelines for choice between packet or circuit• Subtask within GN3 JRA1 to identify and provide guidelines for choice between packet or circuit

based transport• Relevant application

– LOLA – Online entertainment production– Telemedical applicationsTelemedical applications– Banking

• Common understanding of the different layers achieved• Delay assessment using test equipment or inline OAM functions.• Methodology -> From L0 -> L3• Methodology -> From L0 -> L3

– Physical layer– OTN transport– MPLS-TP or PBT -

• Measurements on PBT showing high dependence on packet lenghts• Measurements on PBT showing high dependence on packet lenghts.• Work to be done in future

– Conduct similar measurements with MPLS-TP– Include OTN in the loop

Verify PBT measurements with another vendor– Verify PBT measurements with another vendor– Application on top … LOLA?