1 riga, latvia april 19, 2013 uģis bērziņš, ccie #8972 emeritus chairman of the board santa...
TRANSCRIPT
1
Riga, LatviaApril 19, 2013
Uģis Bērziņš, CCIE #8972 Emeritus
Chairman of The Board
Santa Monica Networks
Internet Exchanges:how hard can it be to build a resilient infrastructure ina small country?
IXP History in Latvia
Internet Exchanges in Latvia• GIX – LATNET / LU MII, Raiņa blvd. 29 (1992/1993)
Natural peering with AS2588
• RIX – BKC, Pērses 2 (1997)Separate L2 switch, 100 Mbps Fiber, no Route server
• LIX (v1) – cooperation of 4 ISPs: Latnet, Lattelecom, Microlink, Telia Latvia (2003)Four L2 switches, 1 Gbps, no dedicated Route Server
• SMILE1 – Santa Monica Networks – non-ISP ownership (2005)Two L3 switches, Nx1Gbps, AS39626, Switch = Route Server
• LIX (v2) – cooperation of 3 ISPs: Latnet, Lattelecom, Telia Latvia (2008?)Three L3 switches, the same design as SMILE1, 10G Core
• SMILE2 – Santa Monica Networks (2010-2013)Two L2 switches, AS39626, Two Route Server, Design similar to DE-CIX
From simple things To complex… and back
LIX (v1)Lessons Learned
LIX (v1) Topology
2
1
4
3
6
5
8
7
10
9
UTIL
STAT
DUPLX
SPEED
SYSTEM
RPS
MODE11 12
CATALYST 3550
2
1
4
3
6
5
8
7
10
9
UTIL
STAT
DUPLX
SPEED
SYSTEM
RPS
MODE11 12
CATALYST 3550
2
1
4
3
6
5
8
7
10
9
UTIL
STAT
DUPLX
SPEED
SYSTEM
RPS
MODE11 12
CATALYST 3550
2
1
4
3
6
5
8
7
10
9
UTIL
STAT
DUPLX
SPEED
SYSTEM
RPS
MODE11 12
CATALYST 3550
GE
GE
GE
X X
Layer 2
STP
BPDU Filter on edge
LIX (v1) Developments
During lifetime there was increase of Core link bandwidth by using link aggregation
Layer 2 loop prevention – STP did not help
No Route Server(s) – any to any eBGP
No single point of contact
Restriction based on polytical rather than technical conditions
Platform growth potential limited
Frequent service stability issues with Link Aggregation and L2 Loops
Issues
SMILE1Lessons Learned
SMILE1 Topology
1
2
3
4
5
6
FANSTATUS
Power Supply 1 Power Supply 2
Catalyst 6500 SERIES
100-240 V~16 A
60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
INPUT100-240 V~
16 A60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
INPUT
SUPERVISOR 2
WS-X6K-S2-MSFC2/2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
SUPERVISOR 2
WS-X6K-S2-MSFC2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
STATUS
WS-X6416-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
STATUS
WS-X6416-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
1
2
3
4
5
6
FANSTATUS
Power Supply 1 Power Supply 2
Catalyst 6500 SERIES
100-240 V~16 A
60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
INPUT100-240 V~
16 A60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
INPUT
SUPERVISOR 2
WS-X6K-S2-MSFC2/2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
SUPERVISOR 2
WS-X6K-S2-MSFC2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
STATUS
WS-X6416-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
STATUS
WS-X6416-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
Layer 3
No STP needed
Switch = Route Server
SMILE1 Developments
Targets set:
• High Availability and Flexibility
• No political restrictions
• Better use of line capacity
Totally different approach than LIX
Stability of service was superior – one unplanned downtime of a single node during major power supply disaster
Use of BGP Communities
Performance – Platform limited to Nx1G
Routing Flexibility:
• eBGP Peering directly with SMILE switch
• SMILE switch selects best path – limited possibilties by SMILE customers to influence routing decision
Issues
LIX (v2) response
LIX (v2) Topology
Layer 3
No STP needed
Switch = Route Server
1
2
3
4
5
6
FANSTATUS
Power Supply 1 Power Supply 2
Catalyst 6500 SERIES
100-240 V~16 A
60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTALL
INPUT100-240 V~
16 A60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTALL
INPUT
SUPERVISOR 2
WS-X6K-S2-MSFC2/2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
SUPERVISOR 2
WS-X6K-S2-MSFC2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
STATUS
WS-X6416-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
STATUS
WS-X6416-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
1
2
3
4
5
6
FANSTATUS
Power Supply 1 Power Supply 2
Catalyst 6500 SERIES
100-240 V~16 A
60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTALL
INPUT100-240 V~
16 A60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTALL
INPUT
SUPERVISOR 2
WS-X6K-S2-MSFC2/2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
SUPERVISOR 2
WS-X6K-S2-MSFC2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
STATUS
WS-X6416-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
STATUS
WS-X6416-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
1
2
3
4
5
6
FANSTATUS
Power Supply 1 Power Supply 2
Catalyst 6500 SERIES
100-240 V~16 A
60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTALL
INPUT100-240 V~
16 A60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTALL
INPUT
SUPERVISOR 2
WS-X6K-S2-MSFC2/2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
SUPERVISOR 2
WS-X6K-S2-MSFC2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
STATUS
WS-X6416-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
STATUS
WS-X6416-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
SMILE2
SMILE2 Topology
MED transparencyAS Path transparency
SMILE2 Developments
Targets set:
• Go for 10G
• High Availability and Flexibility
• No political restrictions
• Better use of line capacity
Return to well proven DE-CIX model (L2)
Route Servers, BGP Communities
Customers fully control routing decision
Direct peerings between major players
Major players acquire small ISPs
Going commercial
Issues
Tools can be different
Thank you.