1

Shigeo Urushidani

SINET3: NII’s new NetworkSINET3: NII’s new Network

National Institute of Informatics (NII)

Fall 2007 Internet2 Member MeetingSan Diego, October 8th, 2007

2

‘87 ‘88 ‘89 ‘90 ‘91 ‘92 ‘93 ‘94 ‘95 ‘96 ‘97 ‘98 ‘99 ‘00 ‘01 ‘02 ‘03 ‘04 ‘05 ‘06 2007 ～

▲2002.1

SINET3 is integrated successor network to two academic networks, SINET and Super-SINET, economically and flexibly providing rich variety of services.

SINET3 started its operations in April 2007 and completed its migration in May 2007.

▲2007.4

Packet Switching Network▲1987.1

- Growing traffic and diversified user requirements- Limited abilities of existing IP routers- New trend of end-to-end circuit services

Super-SINET

SINET3

Evolution of Japanese Academic Networks

▲1992.4

Internet backbone for more than 700 universities and research institutions

SINET

Super-high-speed environment for cutting-edge research

3IP (L3) Ethernet (L2) Lambda/Dedicated (L1)

IPv4 IPv6

Multicast

Application-based QoS

L3VPN

L3VPN

L2VPN

VPLS

L2VPN (QoS)

VPLS (QoS)

BW-specified L1VPN

On-demand

Lambda L1VPN

Multi-homing

Multicast (QoS)

Bes

t E

ffor

tH

igh

Prio

rity

QoS

-gua

rant

eed

SINET3 emphasizes four service aspects: transfer layer, virtual private network (VPN), quality-of-service (QoS), and bandwidth on demand.

Service Categories in SINET3

4

Multiple Layer Services

Ethernet network(Layer 2)

Dedicated line network(Layer 1)

IP network(Layer 3)

Past Networks

SINET3: integrated network providing all transfer layer services.

Users can freely choose best transfer layer for their applications.

It provides economical service provision and flexible network resource assignment for ever-changing and unpredictable service demands.

IP Router

EthernetSwitch

InnovativeIntegration

★ Provides all transfer layer services

★ Integrated network

User Equipment

SINET3

Cutting-edgeDevice

IP Router

EthernetSwitch

User Equipment

Cutting-edgeDevice

5

Multiple VPN Services

For collaborative research activity: closed user group environment (virtual private network: VPN) is essential for security reasons.

Users can choose from L3VPN (IP), L2VPN/VPLS (Ethernet), and L1VPN services.

* Virtual Private Network (VPN); Virtual Private LAN Service (VPLS)

IP-based VPN(L3VPN)

VPLS

L3VPN

Super-SINET

Expansion of Services & Sites

IP Router

EthernetSwitch

IP Router

L1VPN

SINET3

Analysis device

Secure Closed User

Group

6

National Institute for Fusion Science (NIFS) utilizes L3VPN services for its collaborative research with many universities and research institutions.

L3VPN (IP-based VPN)

7

L2VPN and VPLS (Ethernet-based VPNs)

SINET3 provides two types of Ethernet-based VPNs:

• Point-to-point-based VPN (L2VPN)

• Broadcast-based VPN (Virtual Private LAN Service (VPLS)).

Point-to-point-based VPN(L2VPN)

Broadcast-based VPN(VPLS)

e.g. Grid computing research e.g. Earthquake research

8

32 m

11 m

Virtual dedicated lines over shared platform form VPN among specified sites. Users can obtain protocol-free and completely exclusive environment. National Astronomical Observatory of Japan (NAOJ) utilizes L1VPN to transfer

constantly-flowing ATM cells from remote telescopes through STM-16 interfaces. On-demand capabilities will be available soon.

32 m

2.4 Gbps

e.g. e-Very Long Baseline Interferometry (eVLBI) project

L1VPN (Layer-1 VPN)

* Asynchronous Transfer Mode (ATM)

9

SINET3 provides BoD services as part of layer-1 services.

Users can specify the destinations, duration, bandwidth, and route option.

BoD server receives reservation requests, schedules accepted reservations, and triggers layer-1 path setup.

1 Gbps(13:00-14:00)

2 Gbps

(17:00-18:00)

1 Gbps

(15:00-16:00)

Bandwidth on Demand (BoD) Services

User

Web-based Interface(Destination, Duration, Bandwidth, & Route option)

On-demand layer-1 path

Layer-1 path setup trigger

SINET3

On-demand Server

10

Service Parameters of L1 BoD Services

BoD server allows users to specify destinations, duration, bandwidth, & route option via Web-based interface.

VPN Extranet Public

Connection Style + Destinations : VPN-A

: Non-VPN

: VPN-B

Pre-configuredinterfaces

Duration

- Start Time &- Finishing Time (by 15 minutes)

Bandwidth

GE

STM-16

STM-64

GE

10GE

VC-4-7v

VC-4-17v

VC-4-Av

VC-4-Bv 1 ≤ A ≤ 71 ≤ B ≤ 64

STM-64

Lambda Bandwidth-specified

VC-4 Granularity (about 150 Mbps)Route Option

- “Minimum Delay” or- “Unspecified”

11

Middleware

Layer 3 (IP)

Layer 2 (Ethernet/MPLS)

Layer 1 (TDM/Lambda)

Network Control Platform

Security

Service Control Platform

UNI,API,GUI…

User side

SINET3

Hybrid Optical and IP/MPLS Network

Dynamic Control

User-oriented Service Control Platform

Adaptive Network Control Platform

BoD

IPv6, Multicast, VPN, QoS …

- Dynamic resource control- Resilient network control- Performance monitoring

- Bandwidth on demand- Enhanced network security- Middleware/application coordination

- Multi-layer accommodation- Enriched VPN- Enhanced QoS- High availability- Flexible resource assignment- 40 Gbps (STM-256) lines

High-level Network Architecture

High-level network architecture is composed of transport network, adaptive network control platform, and user-oriented service control platform.

12

SINET3 has two-layer structure with edge and core nodes.

Edge nodes are edge layer-1 switches with layer-2 multiplexing, which are located in universities or research institutions and accommodate user equipment.

Core nodes are composed of high-end IP routers and core layer-1 switches located in public data centers.

SINET/Super-SINET

EdgeL1 SwitchWith L2 Mux

CoreL1 Switch

SINET3

IP Router

Core Node

Edge Node

: L1 (Dedicated/On-demand): L3 (IP) : L2 (Ethernet)

: L3 (IP)

Backbone Router

Backbone

SINETRouter

Super-SINET/SINET Router

Network Structure of SINET3

10GE/GE/FESTM-16

13

Has 63 edge nodes and 12 core nodes (75 layer-1 switches and 12 IP routers). Deploys Japan’s first 40 Gbps lines between Tokyo, Nagoya, and Osaka. Links form three loops in backbone to enable quick service recovery against link

and node failures and for efficient use of network bandwidth.

Network Topology of SINET3

: 40 Gbps

: 10 to 20 Gbps

: 1 to 20 Gbps

: Core Node (L1 Switch + IP Router)

: Edge Node (L1 Switch)

2.4 Gbps

10 Gbps

622 Mbps622 Mbps

Japan’s first 40 Gbps (STM256) lines

Los Angeles

New York

Hong Kong

Singapore

40 Gbps package

L1 Switch(NEC UN5000)

IP Router(Juniper T640)

14

L3 and L2 traffic are accommodated in shared bandwidth by L2 multiplexing and transferred to IP router, where each traffic is encapsulated with MPLS labels as needed.

L1 traffic is assigned dedicated bandwidth and separated from L2/3 traffic.

L2/3 (or IP/MPLS) traffic bandwidth can be hitlessly changed by LCAS to flexibly accommodate multi-layer services.

Accommodation of Multi-layer Services

L2Mux

IP Router

EthernetSwitch

Cutting-edgedevice

Shared Layer-2/3 traffic 　 　 　 　

IP/MPLS

MPLS

　 　 　 IP/MPLS traffic 　 　 　

SINET3

Layer-1 traffic

STM64/STM16

STM256/STM64

* Multi-protocol Label Switching (MPLS); Link Capacity Adjustment Scheme (LCAS)

Hitless bandwidth change by LCAS

FE/GE/10GE

GE/10GE/STM16

Edge L1 Switch

Core L1 Switch

IP Routerdata IP

data Ether

Ether data IP

data Ether

Ether

VLAN

VLAN

data IP

data EtherVLAN MPLS

L3

L2

L1

10GEFlow Control

15

L2MUX

L3VPN, L2VPN, and VPLS are logically separated by internal VLAN tags and logical routers. Each logical router exchanges different protocols for each VPN service.

L1VPN and on-demand services need GMPLS protocols to set up layer-1 paths and have separate control planes from that of IP routers.

IP Router

IPv4/IPv6L3VPN

L2VPN VPLS

L1VPN EdgeL1SW

CoreL1SW

L2VPN (L2)

L3VPN (L3)

VPLS (L2)

GMPLS Control Plane

IP/MPLS traffic

Shared Layer-2/3 traffic

IPv4/IPv6 (L3)

Accommodation of Multi-VPN Services

Layer-1 traffic

L1VPN

L1VPN

* Generalized MPLS (GMPLS)

Aggregation

data IP

data Ether

Ether

data IP

data Ether

Ether

VLAN

VLAN

data IP

data EtherVLAN MPLS

data IP MPLSor

L3

L2

L1

: Virtual routing/forwarding table

: Logical Router

16

Architecture for BoD Services

L2MUX

GMPLS control and management plane

L1SW L1SWL1SWL1SW

Layer-1BoD Server

GMPLS

BoD server receives reservation requests, schedules accepted requests, and triggers layer-1 path setup to source layer-1 switch via L1-OPS.

Source layer-1 switch sets up layer-1 path toward destination using GMPLS. BoD server changes L2/L3 traffic bandwidth by LCAS via L1-OPS as needed.

User

L2MUX

IP Router IP Router

Path setup trigger

Hitless bandwidthchange by LCAS

Destinations, Duration, Bandwidth, & Route Option

Path setup request

Scheduling

Path control

Route calculation

Resource managementF

ront

-end

IP

Ethernet

On-demand

L1-OPS

17

Path Calculation in BoD server

BoD server calculates best path for route option using two metrics for each link: delay time and available bandwidth for layer-1 services.

• For Minimum delay, route is uniquely chosen.

• For Unspecified, route that has largest available bandwidth is chosen. Available bandwidth for L1 changes depending on traffic volume of L2/L3.

Hiroshima L1SW

KyotoL1SW

OsakaL1SW

NagoyaL1SW

Tokyo １L1SW

TsukubaL1SW

SendaiL1SW

FukuokaL1SW

MatsuyamaL1SW

KanazawaL1SW

Tokyo ２L1SW

SapporoL1SW

Mon Tue Wed Thu Fri Sat Sun

L2/L3 Traffic Pattern

Lin

k B

an

dw

idth Available bandwidth for layer-1 servicesRoute for Minimum Delay

1 Gbps (VC-4-7v)

VCAT

1 Gbps (VC-4-7v)

0.6 Gbps (VC-4-4v)

0.45 Gbps (VC-4-3v)

Route for Unspecified

Route for Unspecified using VCAT

18

ServiceCriteria

IPv4/IPv6 L3VPN, L2VPN, VPLS L1VPN, On-demand

No. of users Very large Small to medium Small

Priority of availability Highest High Medium

HA function (normal) IP route recalculationMPLS protection

& Fast RerouteNone

HA function (option) - - GMPLS LSP Rerouting

Fukuoka Hiroshima Kyoto Kanazawa Hokkaido

SendaiTsukubaTokyo1NagoyaOsakaMatsuyama

Tokyo2IP route recalculation

IP

IP

MPLS

MPLS

MPLS Protection &Fast Reroute

(option)GMPLSLSP Rerouting

TDM

TDM

High-availability Networking Functions

Multiple loops easily enable multi-layer traffic to be detoured in different directions. Layer-1 switches detect link failures very quickly and inform them to neighboring

layer-1 switches and IP routers.

19

Main Features of SINET3 (Summary)

Items Features Examples

Services

Multiple Layer • L3 (IP), L2 (Ethernet), & L1 (dedicated/on-demand)

Enriched VPN • Virtual Private Network for layers 1 to 3

Enhanced QoS • Support for real-time applications

Layer-1 BoD • Support for data-intensive applications

Value-added • Network performance monitoring

Network

Hybrid Network Architecture

• Hybrid network of layer-1 switches and IP routers•75 layer-1 switches nationwide• 12 IP routers at backbone sites

High Flexibility • Flexible resource assignment to multiple layers

High Availability • Fast service recovery owing to multi-loop topology

Large Capacity • Introduction of Japan’s first STM-256(40 Gbps) lines

Technologies

NG SDH/SONET • GFP, VCAT, & LCAS

GMPLS • RSVP-TE, OSPF-TE, GMPLS-UNI, & GMPLS LSP rerouting

Logical Router • Logical routers for IPv4/IPv6, L3VPN, L2VPN, & VPLS

Advanced MPLS • MPLS-based VPN for L3VPN, L2VPN, & VPLS

Fast Detour • Multi-layer detour triggered by layer-1 switches

20

Schedule

　　　　　　　　　　　　　　　　

Year 2006 2007 2008

SINET3Operation

SINET3Services

Overlay construction

In operation (01/04/2007)

Migration

Complete (31/05/2007)

On-demand(GMPLS-based)

L1VPN(static)

IPv4/IPv6 dual stackL3VPN

L2VPN VPLS

EnhancedGMPLS

Today

SINET3 started to provide L3VPN, L2VPN, & L1VPN (static) services, as well as IPv4/IPv6 dual stack services in April 2007.

Starting VPLS services soon and layer-1 BoD services in February 2008.

21

Thank you very much!Thank you very much!

22

Backup SlidesBackup Slides

23

Multiple QoS Services

SINET3 provides QoS by identifying applications, VPNs, & physical/logical ports. Layer-2/3-based QoS has four priority classes: expedited forwarding (EF), network

control (NC), assured forwarding (AF), & best effort (BE). Layer-1-based QoS has smallest packet delay, no delay variance, & no packet

loss.

SINET/Super-SINET SINET3

Expedited

NW control

Assured

Best effort

Node

End-to-end on-demand path

Best Effort

Congestion

Network congestion affects all services

HDTV

QoS-aware

Application/VPN/port-based QoS control

UncompressedHDTV

24

Prioritizingfor IP

Prioritizingfor Ether

User Priority basedclassifier

IP Router

InternalVLAN tagIdentifier

DSCP&EXP basedclassifier

Edge L1SW

CoreL1SW

Accommodation of Multi-QoS Services

Layer-3/2-based QoS• User Priority bits of internal VLAN tags are marked at edge L2 MUX.• User Priority bits are mapped into DSCP (IP) or EXP (MPLS) bits at IP router.• There are four forwarding classes: EF, NC, AF, & BE.

Layer-1-based QoS• Layer-1 switches assign end-to-end bandwidth on demand.

L2 MUXIP VLAN

VLANBE

IP

IP MPLS

MPLS

AF

NC

EF

BE

AF

NC

EF

• Smallest packet delay• No delay variance• No packet loss

IP(L3)

Ethernet(L2)

Dedicated(L1)

Prioritymapping

for IP

Prioritymappingfor Ether

Marking User Priority bitsby identifying IP/Ethernet header

Mapping User Priority bitsinto IP DSCP or MPLS EXP bits

EtherEther

data IP

data Ether Shared Layer-2/3 traffic

Layer-1 traffic

25

Functions of BoD Server

BoD server software is composed of following function modules:• Front-end functions• Admission control and scheduling• Path calculation• Path control• Resource management

L1-OPS

GUI(Web browser) IF

OperatorIF

- User Authentication

- Session management

http(s)

GUI(Web browser) IF

Userhttp(s)

- Request Acceptance- Admission Control, Scheduling

- Database Registration

Admission Control, Scheduling

Front-end

- L1 Path Setup/Release- IP/MPLS Bandwidth Change

Path Control

- Route Selection- Link Selection

Path Calculation

- L1 Path Management- L1 Path Monitoring

Resource Management

Database

User DB Path DB Usage DB Route DB Resource DBL1SW-DB

Layer-1 BoD Server

: Function Module