hawaii ipv6 deployment experiences webcast for ipv6hawaii.org (ieee globecom 2009 ipv6 forum,...
TRANSCRIPT
Hawaii IPv6 Deployment Experiences
Webcast for IPv6Hawaii.org
(IEEE GLOBECOM 2009 IPv6 FORUM, November 30, 2009)
U. Hawaii Chief Internet Engineer
President, IPv6 Forum Hawaii
Alan Whinery
2
Acknowledgements
• Antonio Querubin – Systems Engineer, Lavanet
• Bill Becker– Network Administrator, Honolulu Community College– Pacific Center for Advanced Technology Training
• Latif Ladid– President, Global IPv6 Forum
• David Lassner– U. Hawaii VP for Info Tech
• UH ITS Network Team
3
IPv6 is not a “project”
•We don't have an “IPv6 person”, or an “IPv6 team”, or an “IPv6 initiative”.
•It is our policy to deploy IPv6 where we deploy IPv4
•As upgrades or maintenance or changes are scheduled, IPv6 is on the to-do list.
4
IPv6 is not a “project”
• Start now• Don't forklift• Consider IPv6 in the course of your design
and purchasing decisions• Work toward including IPv6 in what you do.
5
Most things support IPv6 Now
• Clients– Windows (XP,Vista,7)– Mac OS X
• Router/Switch– Cisco, Juniper, Brocade (Foundry)
• Server– Linux, Solaris, Win2003/2008, MacOS– Apache, BIND, Postfix, Sendmail
6
UH System Network• 18 Campuses and learning centers on 6
islands• ~50 separate research/outreach facilities• ISP for
– State Government– Bishop Museum
• Internet2 connector for– Dept. Of Education– NOAA NWS and NMFS
7
8
9
10
U. H. IPv6 History
• January 2004: “turned on” peerings with 3 RENs
• IPv6 Demos to Korea/Japan in 2004• ITS network engineers and Honolulu
Community College v6 enabled ever since• Addressing moved from “provider” to self in
December 2007• Turned on State-wide OSPFv3 and IPv6
addressing in 2008
11
U.H. IPv6 History
• Initial campus deployment (2004) was through Cisco 7500 running an experimental IOS load– (Juniper supported in main release from 2003)
• Distribution to user vlans was through a trunk into (inter-)campus VLANs
• Although operational DNS servers running BIND were capable early, we waited for Infoblox appliance to do IPv6 records– For user interface
12
U.H. IPv6 History
• Support for Cisco Catalyst 6500 was late in the game
• Our production, mission-critical IPv4 multicast infrastructure had us running non-standard IOS in various places
• Cisco 3550's are still not capable, but we anticipate IPv6 and other factors will drive replacement of those devices
13
Watershed Moments• Acquired address block 12/2007• (2008) Long-awaited Cisco IOS Catalyst
6500 and 3750, permitting us to “go native” on most core infrastructure
• In June 2009, we added IPv6 to our TWTC peering, which made “commodity” IPv6 viable– Prior paths to commercial sites were circuitous– Netflix “Instant” over IPv6 works well
14
Current UH IPv6 Deployment Status
• DNS Stores and answers forward and reverse IPv6 records– Forward currently served in IPv4 packets– Reverses currently served in IPv4 and IPv6
• State-wide, 99% of facility gateway routers have IPv6
• 3 campuses have deployed IPv6 in some form– Manoa, Honolulu CC and Maui CC
• UH Manoa has native v6 turned on to all user segments
15
v6 Peerings• UH
– TW Telecom– Pacific Northwest GigaPoP– DREN (Hawaii Intranet Consortium)– Hawaii Internet eXchange– AARNET (Seattle, Sydney, [LA])
• Lavanet– Hurricane Electric– Sprint– AT&T– Verizon Business– TowardEX– Hawaii Internet eXchange
16
Lava.net Status• Multiple v6 peers• network core (unicast and multicast)• public servers• DNS hosting• Lavanet web site• mail (SMTP/IMAP/POP)• NTP• most office workstations• DSL or frame-relay (on request)
17
UH Short Term Goals• DNS
– workflow• DNS• Address management
– full services in IPv4 packets– full services in IPv6 packets
• Dual stack all IP routing – Currently @ ~ 99%
• Dual-stack all public-facing services – Currently @ ~ 5%
18
UH Short Term Goals
Provide v6/SLAAC to dual stack clients Provide Tunnel endpoints for 6to4, Teredo,
ISATAP– Currently have in-state 6to4 with Lavanet– Teredo is harder to affect– ISATAP is a “good idea”
UH Medium Term Goals
• Develop single stack v6 capability• DHCP6• Translation• Proxying
• Move email infrastructure to IPv6
20
21
• Careful planning will be necessary• Training the email sysadmins will be
necessary• When email is not flowing, a crisis is
occurring
22
The Big Island Router Memory Thing ($$$)
• Problems on the south REN path from Mauna Kea Observatories on the Big Island of Hawaii
– Cisco 3750's with BGP tables of ~12,000 prefixes– Adding IPv6 overwhelmed the fixed-config
memory– We were required to re-design and buy an extra
Juniper M7i to support v6 there
• This would have occurred regardless of IPv6– This is an expense to expedite, not simply to
deploy
23
Cost• IPv6 is not value-added software
– Cisco now has “feature parity”– Juniper has stopped charging for it
• Most of our costs, Lavanet's costs are in staff time and training.
• Lavanet has participated in Opensource projects and contributed IPv6 code
• Cost can be controlled if you simply place IPv6 on your requirements list, start requiring it, and don't panic
• The Big Island router memory re-design is so far the highest-cost IPv6 deployment measure (by far).
24
List of Problems: Native IPv6 Deployment To User Networks
• • • • • • • • Honest: not a single one.
25
UH Client OS Distribution
79%
15%
1%0%0%0%0%
5%
November 2008
Windows Mac OS XLinuxSolarisHandheldsBSD/OSGame ConsolesUnclass
70%
24%
1%0%0%0%0%
5%
November 2009
Volume of HTTP GETs categorized by User-Agent
26
Out-Of-Box V6 Readiness
52%48%
V6 OOB Clients 2009
Volume of HTTP GETs
35%
65%
V6 OOB Clients 2008Volume of HTTP GETs
YesNo
27
Tunneled Forms of IPv6• Teredo (Significant incidental traffic)
– Included with Windows XP, Vista, 7– Used from behind NAT device (real/virtual)
• 6to4 (some incidental traffic)– Included with Windows/Mac OS– tunnels via well-known address (i.e. 192.88.99.1)
• ISATAP (?? traffic)– Included in Windows, some Cisco IOS– Tunnels through guessable domain name
28
Tunneled v6 In The Wild• Sources of incidental 6to4, Teredo seem to
be applications which require IPv6, e.g. P2P clients
– Teredo can be used as an indicator of NAT– There may be more insidious things present
• Setting up local tunneling services can mitigate cost and issues for tunneled clients
• Native IPv6 deployment should stop 6to4, but Teredo will persist from behind NAT
• Un-managed tunnels can represent increased attack surface and firewall by-pass.
29
UH Teredo Traffic
• All clients use one of three Teredo servers:– 207.46.48.150 (Microsoft Asia)– 213.199.162.214 (Microsoft Europe)– 65.55.158.80 (Microsoft USA)
• NAT causes Teredo traffic• Virtual machine NATs cause Teredo traffic• Exceedingly complicated• Presumably initiated by an application install
30
SNMP
• Cisco devised interim MIBs– Which persist in our current IOS
• Cisco's MIB support notes are fiction• CISCO-IETF-IP-MIB
31
Graphing v4/v6 • The old MRTG model of graphing
interface Octet-counts doesn't do per protocol accounting
• Various non-optimal things can be done– ACLs feeding counters, etc
• The following graphs were by using 8 “bpf” counters fed by individual filter expressions
– No packet was examined– Not a scalable approach
• Data represents 1 day on our TWTC v6/v4 peering
320 0 1 1 2 2 3 3 4 4 5 6 6 7 7 8 8 9 9 10 11 11 12 12 13 13 14 14 15 15 16 17 17 18 18 19 19 20 20 21 22 22 23 23
0
100000000
200000000
300000000
400000000
500000000
600000000
700000000
V4 versus V6 traffic
November 19, 2009
v4 in
v4 out
n6 in
n6 out
teredo in
teredo out
6to4 in
6to4 out
hour
bp
s
330 0 1 1 2 2 3 3 4 4 5 6 6 7 7 8 8 9 9 10 11 11 12 12 13 13 14 14 15 15 16 17 17 18 18 19 19 20 20 21 22 22 23 23
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
4000000
4500000
V6 Tunnels and Native TrafficNovember 19, 2009
teredo in
teredo out
6to4 in
6to4 out
n6 in
n6 out
hour
bp
s
340 0 0 1 1 2 2 2 3 3 4 4 4 5 5 6 6 6 7 7 8 8 8 9 9 10101011 11121212 13131414 141515 16161617 17181818 19192020 20212122 222223 23
0
1000
2000
3000
4000
5000
6000
Native IPv6 Traffic
November 19, 2009
n6 in
n6 out
hour
bp
s
35
Performance
• High-throughput transfers can exhibit throughput differences between v4 and v6
• Usually because v6 is being processor-switched (configuration?)
• v4 and v6 paths to a resource often differ.• We have been happy if it worked at all
– Now we need it to perform well
• The time for optimization is at hand.
36
Comparing v6/4 paths (UH)
v6 v40
2
4
6
8
10
12
14
Average Hops from 2504 HostsFrom ping TTL
ho
ps
v6 v40
50
100
150
200
250
300
Average RTT to/from 2504 HostsFrom ping Avg RTT
mill
ise
con
ds
37
Comparing v6/4 paths (LavaNet)
v6 v4
0
2
4
6
8
10
12
14
16
LavaNet: Avg Hops from 2804 hosts
From ping TTL
ho
ps
v6 v4
0
50
100
150
200
250
LavaNet: Avg. RTT to/from 2804 Hosts
From ping Avg RTT
mill
ise
con
ds
38
Every Firewall, ACL,etc
• Web server access controls• acls• firewall setups• PHP code to return restrict content based on
IP address• MaxMind GeoIP
– is v6 capable
39
IPv6 Deployment Scenarios
• Laissez-faire (ignoring your destiny)– Your resources will be unreachable via IPv6– external IPv6 resources will either be unreachable or
tunneled per client• You probably have significant tunneled traffic now
– Your existing IPv6 traffic will be high latency, poor performance
• Possibly without the end-users' knowledge, they will simply blame you for bad performance
• Or they will blame IPv6 and turn it off– Requires IPv4 addresses
• “I don’t believe the v6 transition is occuring”– Means “I choose denial instead of participation”
40
IPv6 Deployment Scenarios
• IPv6 Only ☺ - Client support sketchy - Translation necessary to reach IPv4 Internet + Some value in enabling v6-only servers
• Dual-stack ☺☺☺ + Client support good + No translation necessary + Serves potential v6-only groups - Requires IPv4 addresses
41
Stateless Auto-configuration (SLAAC)• Many operating systems have IPv6 turned on by
default• With SLAAC, if your router interface is using
v6, then you are too. You may use v6 without realizing it
• Your machine determines your IPv6 address, and adds it to the prefix advertised by the router
• Some OS build the RH 64 bits using the MAC address
• Others will make up random (currently only Vista and W7)– complicates address accounting/management
42
Getting a DNS Server address
• Stateless auto-configuration gets you an address and gateway
• But no DNS server• Of course, if you have DNS through IPv4,
you will learn v6 addresses through that DNS server
• Currently, the only way for a v6-only host to auto-learn the name server address is DHCPv6
• Attachments to SLAAC are proposed– RFC 5006 (IPv6 Router Advertisement Option
for DNS)
43
IPv6: Apple OSX 10.4+• On by default• Missing DHCP6• Can't specify v6 address for networked
printer, because the preferences pane for printer set-up considers a colon ‘:’ as preceding a port number (? 10.6)– Printer can, however, be specified by name
44
Apple OS X Applications
• Firefox – once required v6 “turn on”– This seems to have changed
• Safari – does browse IPv6• ping – works with separate “ping6”• traceroute – works with separate “traceroute6”• SSH client – works• telnet – works to router: fe80::209:7bff:fedc:400%en0
• email – no server to test to yet
45
IPv6: Windows XP (SP2+)• You can add it to an interface with the interfaces
“Properties” pane, just like IP(v4) or IPX/SPX or NetBIOS
• Once added, there is no GUI config, although some things can be accomplished with the command line
• Will not do DNS queries in IPv6 packets• Will receive IPv6 info from DNS in IPv4 packets• Is Ultimately doomed.
46
Windows XP Applications• Firefox – will browse IPv6• IE7 – will browse IPv6• ping – works
– Tries first address as returned by DNS
• tracert – works– Tries first address as returned by DNS
• Telnet – doesn’t appear to work• Thunderbird – no server to test to yet
47
IPv6: Windows Vista and 7
• On by default• Does DHCP6• There have been some problems
– Passing of ICMP6 messages to applications
48
Windows Vista Applications• Firefox – will browse IPv6• IE7 – will browse IPv6• ping – works
– Tries first address as returned by DNS
• tracert – works– Tries first address as returned by DNS
• Telnet – doesn’t appear to work• Thunderbird – no server to test to yet
49
IPv6: Ubuntu 8
• On by default• Does DHCP6, if you install it• Since Linux (and BSD OS) are typically
used for reference implementations, support is pretty good
50
Ubuntu Linux Applications• Firefox – will browse IPv6• ping – works as “ping6”• traceroute – works as “traceroute6”• Telnet – doesn’t appear to work
• Linux is a kernel. – Linux distributions are operating systems. They differ
as to what apps they provide for various roles. – “Distributions” means, Red Hat, Ubuntu, Suse, Debian,
Slackware, etc.
51
Steps To Dual-stack IPv6/(4) Deployment
• Get addresses• Configure routers• Configure DNS• Configure public-facing services
(web/mail/etc)• Configure clients
– Probably only necessary to the extent that you have Windows XP
52
Steps to single-stack IPv6 Deployment
• Get addresses• Configure routers• Configure DNS (in v6 only)• Configure public-facing services
(web/mail/etc)• Provide gateway to v4• Configure clients
– Need DNS server entry– Manual or DHCP
53
IVI V6 to V4 gateway
• Implementation of Internet Draft• From CERNet and 清華大學 (Beijing)• License unclear• Involves patches to out-dated kernel (2.6.18)
– Which doesn’t compile under current libc/gcc
• I have seen it work well, in February 2009, at Joint Techs, Texas A&M
54
Trying Out Your IPv6
• It’s hard to know whether you are using it.– ShowIP add-on for Firefox helps
– But it isn’t perfect
• When the OS provide resolution and connectivity– The applications still may
• Or may not
55
Perl Programming
• Can't simply handle addresses as integers anymore, without a 128-bit data type.– Math::BigInt– Net::IPv6Addr (eats RFC 1884 addr format)
56
pcap filters expressions
• tcpdump udp and 'udp[16:4] = 0x20010000'– capture packets from Teredo clients
• tcpdump net 20C0:FFEE::/32– capture for the specified 6 net
• tcpdump ip6– capture all ipv6 traffic
• tcpdump proto ipv6 – capture ipv6 encapsulated in IPv4 OR IPv6
Used to specify packet captures:with tcpdump, wireshark, ngrep, etc
57
Dirty Tricks: OK!
• You can direct the DNS AAAA record for an existing IPv4 services to a separate device– Use Apache as a transparent proxy to make it
look like the content has a v6 address– This is GREATLY simplified if the content has
an alternate name– Some scenarios/services can simply use a
different host
58
Dirty Tricks: OK!
• Nothing says that the interface or device that offers services via IPv6 is required to be the same as the one that offers those services over IPv4
59
Multicast• For IPv6 Multicast, the capabilities of Layer
2 switches become critical– Currently our v6 Multicast is only experimental– We do not have plans to migrate our multicast
applications
60
What can I reach with IPv6?
More and more.
See http://ipv6hawaii.org
“Things You Can Reach With IPv6”
61
• Hawaii IPv6 Forum– http://ipv6hawaii.org
Returning To Work On Monday