internet exchanges in asia pacific - ausnog · peering trend in japan big domestic eyeballs extend...
TRANSCRIPT
Internet Exchanges in Asia Pacific
Katsuyasu Toyama
JPNAP / APIX association
Who is Katsuyasu Toyama?
• Chief Operating Officer (2015- )• Director, General Manager of Technology Dept. (2007- 2015)
– Asia Pacific Internet Exchange (APIX) association• Chairperson
– IX Federation• Board member
– Global Peering Forum (GPF)• Board member
SVP of Internet and Mobile Services,
NTT Communications (2012-2015)2(c) 2016 INTERNET MULTIFEED CO.
Outline
• APIX association
• IXP Comparison in Asia Pacific, and Global
• IXP and Peering in Asia Pacific
• “Too much” quality of IXP operation in Japan?
3(c) 2016 INTERNET MULTIFEED CO.
APIX association
4(c) 2016 INTERNET MULTIFEED CO.
What is APIX association?
• “Asia-Pacific Internet Exchange”
– An association of Internet Exchange Providers in Asia-Pacific region
– Established in 2010, supported by APNIC
– Two meetings per year, next meeting is 14th.
– Objectives:To share information about technical, operational, and business issues and solutions regarding Internet Exchange
5(c) 2016 INTERNET MULTIFEED CO.
5 IXP associations in the world
6
Reference: Euro-IX Update at Euro-IX forum
(c) 2016 INTERNET MULTIFEED CO.
IX-F
• “The Internet Exchange Federation (IX-F) acts as a platform for affiliated Internet Exchange Point Association (IXPAs). We are working together to build a global IXP community and help the development of IXPs throughout the world.”
IX-F
AF-IX APIX Euro-IX LAC-IX
7(c) 2016 INTERNET MULTIFEED CO.
APIX members
• Membership Criteria
– Any entity, enterprise, or organization in the Asia-Pacific region,
possessing legal personality under the laws of a country or economy
where the entity resides, providing services regarding Layer 2 Internet
Exchange Points, sustaining a neutral position, and does not enforce
the purchase of transit services. The Asia-Pacific region is defined as
the economies where APNIC serves.
(from APIX bylaws)
8(c) 2016 INTERNET MULTIFEED CO.
APIX Members
• 23 IXPs, from 16 countries and economies
– Australia, Bangladesh, Cambodia, China, Hong Kong, India, Indonesia, Japan,Korea, Malaysia, Nepal, New Zealand, Singapore, Taiwan, Thailand, Vietnam
VNIXSOX
IIX
NZIX
TPIX
CNX
9(c) 2016 INTERNET MULTIFEED CO.
APIX members23 IXPs (3 members have IXPs in Australia)
– AMS-IX Hong Kong (HK)
– BBIX (JP)
– BDIX (BD)
– BKNIX (TH)
– CHN-IX (CN)
– CNX (KH)
– DIX-IE (JP)
– Equinix (HK, JP, SG, AU)
– HKIX (HK)
– IIX (ID)
– IX-Australia (AU, NZ)
– JPIX (JP)
– JPNAP (JP)
– KINX (KR)
– Megaport (AU, HK, SG, NZ)
– MyIX (MY)
– NIXI (IN)
– NP-IX (NP)
– NZIX (NZ)
– SGIX (SG)
– SOX (SG)
– TPIX (TW)
– VNIX (VN)
10(c) 2016 INTERNET MULTIFEED CO.
Topics discussed at previous APIX meeting
• Technical session– PeeringDB update (PeeringDB2.0)
– Peering and interconnection market in Philippine
– Netflix strategy for global expansion
– Yet another route server software (GoBGP)
• Administrative session– Membership fee (just decided 100 USD per year per member IXP)
– Election of the Steering Committee
11(c) 2016 INTERNET MULTIFEED CO.
Steering Committee
• Steering Committee members– Chairperson: Katsuyasu Toyama –JPNAP (JP)– Secretary: Ajai Kumar --NIXI (IN)– Treasurer: Gavin Tweedie -- Megaport (AU/HK/SG/NZ)– Meeting Arrangement: Che-Hoo Cheng --HKIX (HK)– Website Management: Brian Kim –Twitch (KR)
12Katsuyasu Che-Hoo Gavin Ajai Brian(c) 2016 INTERNET MULTIFEED CO.
IXP Comparison in Asia Pacific, and Global
13(c) 2016 INTERNET MULTIFEED CO.
Key Performance Indicators for IXPsin APAC region
1. Number of ASN
– How many networks you can potentially peer there
2. Peak traffic
– How much traffic you can potentially distribute there
3. Average traffic per ASN
= (peak traffic)/(number of ASN)
– How big eyeball and content providers gather there, or
– How much broadband internet is spread in the market
14(c) 2016 INTERNET MULTIFEED CO.
Data from:
1. Number of ASN
– PeeringDB
• except for MSK-IX (Russia) and IX.br (Brazil), which seem too few in PeeringDB
2. Peak traffic
– Their web page on traffic stats
• Unfortunately some IXPs do not disclose data. They are excluded.
3. Average traffic per ASN
= (peak traffic)/(number of ASN)
15(c) 2016 INTERNET MULTIFEED CO.
Number of ASN
189
171
135126
114
97 95
7874
7066
55 53 5142 39
35 3428 26 25 24 23 22
15 15 14 13 12 11 11 10 7 7 6 3
0
20
40
60
80
100
120
140
160
180
200
(c) 2016 INTERNET MULTIFEED CO. 16
Peak traffic
17
592
524 510
107 91.573.4 64
44 37.5 32.6 26.7 25.1 23.7 16 12.5 12 9.2 9 1.2 0.7 0.60
100
200
300
400
500
600
700
IXP Peak
HKIX 592
JPNAP Tokyo 1 524
JPIX Tokyo 510
JPNAP Osaka 107
MyIX 91.5
SGIX 73.4
JPIX Osaka 64
JPNAP Tokyo 2 44
TPIX 37.5
MegaIX Sydney 32.6
NIXI Mubai 26.7
IX-Australia VIC 25.1
(c) 2016 INTERNET MULTIFEED CO.
Average traffic per ASN
18
Gbps/ ASN
7.33
5.52
4.05 3.82
3.46
2.46
1.78 1.75 1.60 1.56 1.39 1.29 1.03 1.00
0.63 0.46 0.34 0.32 0.20 0.17 0.05 0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
(c) 2016 INTERNET MULTIFEED CO.
Comparing Asia and the rest of the world
• Europe:
– AMS-IX Amsterdam, Netherland
– DE-CIX Frankfurt, Germany
– LINX London, United Kingdom
– MSK-IX Moscow, Russia
• Latin America:
– IX.br Sao Paulo, Brazil
19(c) 2016 INTERNET MULTIFEED CO.
Scatter plot: ASN x Peak traffic
CHN-IX
npIX
MegaIX Hong Kong
BKNIX
MegaIX Singapore
NIXI Delhi
AKL-IX
BDIX
AMS-IX Hong Kong
IX-Australia VIC
NIXI Mubai MegaIX SydneyTPIX
JPNAP Tokyo 2JPIX Osaka
SGIXMyIX
JPNAP Osaka
JPIX Tokyo
JPNAP Tokyo 1
HKIX
AMS-IX
DE-CIX
LINX
MSK-IX
IX.br
0.1
1
10
100
1000
10000
1 10 100 1000Number of ASN
Peak
Tra
ffic
(G
bp
s)
20(c) 2016 INTERNET MULTIFEED CO.
Scatter plot: ASN x Traffic/ASN
npIXMegaIX Hong Kong
CHN-IXAKL-IX
MegaIX SydneyIX-Australia VIC
MegaIX Singapore
NIXI Delhi AMS-IX Hong KongBKNIX MyIX
TPIXBDIX SGIX
NIXI Mubai JPIX Osaka
HKIXJPNAP Osaka
JPIX Tokyo
JPNAP Tokyo 1
JPNAP Tokyo 2
AMS-IX
DE-CIX
LINX
MSK-IX
IX.br
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
1 10 100 1000
Traf
fic
pe
r A
SN (
Gb
ps)
Number of ASN21(c) 2016 INTERNET MULTIFEED CO.
IXP and Peering in Asia Pacific
22(c) 2016 INTERNET MULTIFEED CO.
IXP scene in Asia
• Asia
– Hong Kong, Singapore are the hubs of south east & south Asia
– Tokyo has big traffic due to broadband penetration;Tokyo is gradually attracting the attentions from Asian ISPs due to Japanese content
– Sydney, many ASN at IXP but relatively small traffic. Why?
23(c) 2016 INTERNET MULTIFEED CO.
IXP scene in Asia
• Carrier neutral IXP in Asian countries are increasing
– Thailand (BKNIX) and Mainland China (CHN-IX)
• Remote peering is slowly spread into Asia
24(c) 2016 INTERNET MULTIFEED CO.
BKNIXBangkok Neutral Internet Exchange
• Carriers or big ISPs own their “IX”, but they are layer-3 (routers)
• BKNIX is a “neutral” internet exchange in Thailand
(c) 2016 INTERNET MULTIFEED CO. 25https://bknix.co.th/en/index.php?module=service&content=1http://internet.nectec.or.th/webstats/internetmap.current.iir?Sec=intern
etmap_current
CHN-IX
(c) 2016 INTERNET MULTIFEED CO. 26
• CHN-IX has been established by ChinaCache’s initiative– Beijing, Guangzhou, Shanghai
• Initial target is ICP and “Tier-2” ISP
Remote peering
• ISPs extend their network to another country or region, to peer with ISPs there.– For instance, eyeball tries to peer with content such as video or game
• Without colos or routers in remote side
ISPsubscribers
video
game
Country or economy
R
IX
IX
R
L2 circuit
L2 circuit
27(c) 2016 INTERNET MULTIFEED CO.
remote peering
Remote peering
• Motivation or benefits
– to fulfil subscribers’ requests such as latency to servers,not by transit providers but by themselves
• This works fine if:
– “transit fee” > “circuit fee + IX port fee”
ISPgame
IXR
transittransit
L2 circuitIX port
28(c) 2016 INTERNET MULTIFEED CO.
Remote peering
• In Europe, “transit fee” > “circuit fee + IX port fee” sometimes holds
– Fibers and L2 circuits are obtained at reasonable price, becauseland lines can be built much cheaper than sea cables, and alsoL2 providers compete with each other
• In Asia
– Gradually L2 connection providers are penetrating,according to price down of sea cables
29(c) 2016 INTERNET MULTIFEED CO.
IXP competition in Japan
• Major IXPs in Japan JPNAP
JPIX
BBIX
EIE (Equinix)
– They are all commercial, and neutral internet exchanges
– Neutral, but major carriers are behind them, and only use specific IXPs
– Content providers connects all the IXPs,but small eyeballs connect one or two of them
30(c) 2016 INTERNET MULTIFEED CO.
IXP competition in Japan
JPIX JPNAP BBIX EIE (Equinix)
Commercial, Neutral Commercial, Neutral Commercial, Neutral Commercial, Neutral
KDDI group NTT/IIJ group Softbank group US company
Tokyo
ASN 126 95 78 70
Traffic 510G 524G Not Disclosed Not Disclosed
Osaka
ASN 26 28 11 11
Traffic 64G 107G Not Disclosed Not Disclosed
Features
Number of Networks is biggest in Tokyo
Largest Traffic in TokyoBiggest in Osaka
Also has IXs in Singapore and Hong Kong
US content providers
31(c) 2016 INTERNET MULTIFEED CO.
Peering Trend in JapanBig domestic eyeballs extend to abroad
• Domestic eyeballs (not carriers) extend their reach to abroad– Biglobe (AS2518) Singapore, Hong Kong, Los Angeles, San Jose
– K-opticom(AS17511) Amsterdam, Los Angeles
– For bigger eyeballs, peering cost(IX port + Leased Line) becomes
less than transit by negotiation.
• Remote peering will be morepopular in near future
32(c) 2016 INTERNET MULTIFEED CO.
“Too much” quality of IXP operation in Japan?
33(c) 2016 INTERNET MULTIFEED CO.
Internet Operation in Japan
• High reliability and high quality
– Sometimes good, sometimes bad
• Internet is “best effort” service
– usually means “do our best as far as we (service provider) can”
– in Japan sometimes “do your best until we (customer) are satisfied”
34(c) 2016 INTERNET MULTIFEED CO.
Japanese customers:strong request for service quality
• Our customer ISPs, and their customer enterprises are very severe for service quality.
• If a trouble happens and network is down, ISP orders us to report its root causes and their countermeasures immediately.
– It is strongly requested by their customers (enterprises)
End user(Enterrpise)
ISP IX$ $quaility quaility
report report
35(c) 2016 INTERNET MULTIFEED CO.
Japanese customers:strong request for service quality
• Operation center of our customer is always watching logs and traps.• If they notice anomalies in such monitoring data, they will call us to
confirm if something bad is happening or not.– For instance, if their bgp session is down, they will call us whether the
down is caused by IXP network, such as switch down, interface down, or backbone circuit is down.
• Accordingly, JPNAP built a reliable platform using optical switches and redundant ethernet switch network.– The result of competition, Japanese IXPs have adopted such redundant
network architecture.
36(c) 2016 INTERNET MULTIFEED CO.
Port-2
Port Tx
Tx
MonitorsOptical power
Intelligent Optical Switch System
Rx
RxJPNAP SW1
JPNAP SW2
CustomerRouter
Port-1
Port-2
Port Tx
Tx
Tx Rx
Automatic switching×
Rx
JPNAP SW1
JPNAP SW2
Normal state
FAILURE
Intelligent Optical Switch System
37
CustomerRouter
Intelligent Optical Switch System
tens of mili-seconds
(c) 2016 INTERNET MULTIFEED CO.
CustomerISP/ICP
B
CustomerISP/ICP
A
JPNAP: Redundant network architecture
• Backbones and Ethernet switches are redundant• Optical switch connects Customer ISP/ICP router to two Ethernet switches
• Down time could be minimum
IX backbone
38(c) 2016 INTERNET MULTIFEED CO.
CustomerISP/ICP
B
CustomerISP/ICP
A
JPNAP: Redundant network architecture
• In case of failure, automatically failover to the backup switch.
• BGP session is kept
IX backboneX X
39(c) 2016 INTERNET MULTIFEED CO.
JPNAP: Improving failure detection time
• To detect network down as soon as possible, we monitor;
– both customers’ and ours’ interface by Ping
– Link down / up traps from our ethernet switches
– bgp sessions between customers’ bgp speakers and ours
40(c) 2016 INTERNET MULTIFEED CO.
CustomerISP/ICP
B
CustomerISP/ICP
AIX backbone
MonitorEquipment
Current
Our monitoring system checks automatically
20 second at maximum by system, using SNMP polling every 10 second
System disables the port automatically
Less than 20 second to recover
Past
First alert from Ping failure or BGP down
10 minutes at maximum to detect by human,from traffic graph updated every 5 minutes
Human disables the port
More than 10 minutes to recover
JPNAP: Improving failure detection time Usually traffic fairly balanced on LAG-> We use the characteristic to detect the failure such as LINK up but frames disappear
Human eyes check the graph, and detects the
failure
mlx1->mlx2 19.8 Mbpsmlx1 e1/3 -> mlx2 e1/3: 20.4 Mbps mlx1 e1/4 -> mlx2 e1/4: 19.4 Mbps mlx1 e1/5 -> mlx2 e1/5: 544 Kbps ( 2%)mlx1 e1/6 -> mlx2 e1/6: 21.3 Mbps
System automatically checks
the balance betweenpaths
41(c) 2016 INTERNET MULTIFEED CO.
Next step?
• My personal opinion, “too much” operation should be reduced.
• But this is a kind of Japanese culture; seems difficult to change.
• Also IXPs are afraid to lose customers, they cannot step out from the current situation.
42(c) 2016 INTERNET MULTIFEED CO.
And one more topic from JPNAP…
43(c) 2016 INTERNET MULTIFEED CO.
Route server platform
• Route server platform is one of outstanding issues at IXP
– Current popular platforms: BIRD, OpenBGPd, Quagga, Cisco• BIRD: recently becoming very popular platform• OpenBGPd: multithreaded, works on BSD• Quagga: old and monolithic architecture, maintenance is poor• Cisco: expensive
– Quagga has become old and maintenance is poor.– New platform to replace Quagga is necessary.
44(c) 2016 INTERNET MULTIFEED CO.
GoBGP
• Open source BGP daemon– https://github.com/osrg/gobgp– developed by NTT Laboratories
• Go Language (developed by Google)– Using multi-core, faster processing in parallel
• API-first– GoBGP is designed to be controlled through API– Command Line interface is implemented into a software to kick the APIs
45
gobgpd
CLIgobgp
Your Software
API (gRPC)
(c) 2016 INTERNET MULTIFEED CO.
Performance comparison in laboratorywith other BGP daemons
• Much better performance than Quagga, an older and monolithic BGP daemon
• GoBGP is just a little better than BIRD, widely deployed BGP daemon
46
• Advertise 100 prefixes per peer• Without route filters
Quagga
BIRD
GoBGP
(c) 2016 INTERNET MULTIFEED CO.
GoBGP Route Server deployed at JPNAP
• Deployed at JPNAP Osaka on Aug. 3, 2016in place of Quagga route server
• So far no troubles ;-)
• Next, deploying at JPNAP Tokyo 1
Quagga BIRD GoBGP BIRD
47(c) 2016 INTERNET MULTIFEED CO.
Thank you!
ContactAPIX: sc _at_ apix.asia
JPNAP: info _at_mfeed.ad.jp48(c) 2016 INTERNET MULTIFEED CO.