Benn ThomsenMicrosoft Research

Operating Optics at scale

Data Center

1 km radius

150,000 servers

120,000 100G optical transceivers

Regional Network

70km radius

512 fibre pairs

2Pb/s

WAN

Data centers in 50+ regions

More than 70,000km of fibre

Flops

Bytes

Operating vertically: The rack

Operating vertically: DC Network

Innovating vertically: Regional Architecture

15-20 data centers in a region

Mega Data Center

Distributed data centers across an 80Km region to keep pace

with cloud growth

5

Innovating in optics

Colored optics

WAN

Regional network

(80 km links)

Regional hub

Mega data center Regional architecture 6

Gray optics

» Dual-λ, 28 Gbaud PAM4

» < 4.5 W per module

» Variable gain/tilt EDFAs

» Automatic Tunable CD compensation

Regional architecture: PAM4 QSFP28 DWDM transceiver

EDFA

L2/3

sw

itch

36

xQSFP

Lin

eca

rd

Mux/

Dem

ux EDFA

TDCMx4

QSFP28

...

DCF

QSFP28QSFP28QSFP28QSFP28

QSFP28

QSFP28

QSFP28

fiber

span

VOA

x36

ADVA SmartAmp

ColorZ: Inphi 100G QSFP28

Optimised for 80km

Storage

Networking

Disruption across the board

Gb

ps/

$

WAN capacity

TimeG

B/$

Hard disk capacity

Time

Gb

ps/

$

Data center

network

Time

IOPS/$

Flash capacity

Time

FLO

PS/$

CPUs

Time

1980s

2006

Moore’s Law for NetworkingTb

ps

(lo

gsc

ale

)

Time

~2x every two years

$

Time

Cloud network cost

64

256

1024

4096

16384

2004 2006 2008 2010 2012 2014 2016 2018

Thro

ugh

pu

t (G

b/s

)

Year

Will free scaling continue?

12.8Tbps(Broadcom, Innovium)

256x 50Gb/s PAM4

2 bits/sym6.4Tbps

256x 25Gb/s NRZ

1bit/sym

Switches are becoming edge bandwidth limited

Moore’s Law for NetworkingTb

ps

(lo

gsc

ale

)

Time

~2x every two years

$

Time

Cloud network cost

11

Gb

ps/

$

Data center network

Opportunity for photonics (shorter-term)

PCB

Switch ASIC

Optics

Fiber PCB

ASIC Optics

Fiber

CMOS Scaling limit

PCB

Multiple ASICs

Fiber

Today 16nm CMOS

On board optics In package optics Optical Network on chip

In-package optics (IPO)

Chip

SwitchASIC

Tx/Rx

PCB

1

64

TX/RX

Tx/Rx

Tx/Rx

Transceivers

OpticalSource

• Challenges• Material: Silicon, InP• Approach: Direct modulation, MZM, Ring modulator, Coherent??• Real estate, yield, operating environment• Fibre attach

70mm

Opportunity for photonics (long-term)

Fundamental challenges− All optical header processing

− Optical buffers

✓More developed technology− Higher- radix enables a flat network

Optical circuit/burst switch

Optical packet switch

An old idea made newNew goalpost

Large Scaledata-center-wide

&

Fast switchingSub nano-second

Sca

le o

f so

lutio

n

Switching speed

Slow (ms) Fast (ns)

Chip

scale

Data center

scale

Optics community

Optical switch

Mechanically-moved

mirrors (MEMS)

Network community

Bridging the last mile

Apps

Network Stack

NIC

PHY

Apps

Network Stack

NIC

PHYOptical Circuit Switch

Needs to support Burst mode CDR

Network wide synchronisation

New applications that exploit the network properties e.g. low latency

We need to consider the implications for the whole system

Optical Network Prototype

Nanosecond-granularity switching

Long, long way to go!17

Integrated Photonics: 56 Gbaud Modulator Array

Output FibreAttach

4x 56Gbaud Modulators

2x 56G Ring Modulator

5mm

Chip testing

Storage

Hot

Online

Technology: SSD, HDD

$$$$$

Cool

Read Write Occasionally

Technology: HDD

$$$

Cold

Write Once, Read Once in a Blue Moon

Technology: Tape, Blu-ray, Optical?

$

Today’s cold storage

Credit: stock.adobe.com

Hard disk drive Tape

GB/$

Hard disk capacity

Writing in Silica glass

23

Voxel structure

• Approx. ellipsoidal shape, 𝛾~1𝜇𝑚, 𝑑~5𝑢𝑚

• Volumetric nanograting

• Exhibits form birefringence

• Multi-level encoding using 𝑹𝒆𝒕 and 𝜽 to store multiple bits per voxel

𝑑

𝒏𝟏 𝒏𝟐

Transverse view

𝜽

𝛾

Voxel Writing

Arrays of voxels

Information encoded by modulating the intensity and polarisation of the laser pulses

Challenge: Scaling write speed• Modulator bandwidth• Parallelism

Reading from Silica glass

Optical microscopy:Sample

Exploit developments in consumer camerase.g. Sony 16MPixel 1000fps

Voxel Reading Read demo

A big opportunity – cross-disciplinary approach

Optics

NetworkingStorage

Hardware

28

Optics for the Cloud

Physical

Network

Storage

New optics

New hardware/software

New Media

Application New scenarios

Custom PIC

Innovation across the cloud stackAt Microsoft and with collaborators

Prototype switch

Silica

Want to know more

www.opticsforthecloud.com