New Generation Backbone Router~ A Longevous SolutionPresented by: Furman Chang, Business Develop Department, Hauman E-Mail: furmanc@hauman.com.tw

The Old Generation NetworkFrom Humble Beginnings

The New Generation NetworkFrom Humble BeginningsThe New Generation Network is Well Beyond Critical Mass!

Why Talk About Backbone Routers?

Backbone Routers Dominate the Intelligence of InternetMaps any-any demand to point-point supply

Legacy IP RoutersPoor architectureBorrowed from embedded computer baseFunctionality incorrectly partitionedFragile and unreliable (working at edge of performance)Configurations complex due to tradeoffs and limitationsMonolithic, bloated softwareDesigned for multiprotocol LAN connectivityFocus on bells and whistles, not on stability and performanceWeak hardware baseMicroprocessors, jelly bean parts, FPGAsInefficient for forwarding IP packets

Routing Technology Revolution 1996TodayScaleTimeFor details please access BTexact Thchnologies.http://www.btexact.com/docimages/42267/42267.pdf

So What Is a Backbone IP RouterCertain minimum qualificationsCapable of switching IP datagrams: Layer 3 forwardingSymmetric any-port-to-any-port switching speedDelay-bandwidth buffering, plus congestion controlInternet scale IS-IS, OSPF, MPLS, BGP4Todays benchmarkWire-rate forwarding on all portsPerformance independent of loadSupport of class-of-service (CoS) queuing, shaping, and policingTraffic engineeringClassification and filtering at wire rate

What a Backbone Router Looks Like?Cisco GSR 12416Juniper T6406ft2ftCapacity: 320Gb/s3ft2.5ft19Capacity: 640Gb/s19

Why Are They so Hard to Build? Bottom line: inherent complexityScaling along multiple dimensionsBandwidth, packets per second# interfaces, # channels, # routes, # neighbors, # policies, #filtersUnpredictable, demanding environmentNeed for reliable, seamless interoperabilityDeep technical expertise across multipleSoftware: routing protocols, embedded systems, network managementHardware: ASIC design, board design, high-speed circuit designMechanical: power, packaging, thermal, emissionsChanging requirementsBuilding Internet routers requires a special viewpointThe network is the system, not the boxInternet routers uniquely integrate the network at scale

Building A Provider Business:Standard Services

Building A Provider Business :Smart IP ServicesService Deployment

IP Service Market ReadinessChallenges the Old IP InfrastructureService Deployment

Smart IP Services :A Longevous Solution Smart IP Services Smart IP ServicesAccountingIPv6Layer 2/3 MPLSMulticasting*tCoS/QoSSingle Operation SystemFor All Network InterfacesFor All Routing Platforms

AgendaRouter Architecture and TechnologiesMajor Router Functionalities Major Hardware of Routers Major Components of RoutersThe Evolution of Router ArchitectureCase Study A Juniper Networks ExampleArchitecture OverviewRouting EngineForwarding EngineHardware & Architecture Flexibility

Routers Position :Late 1980 ~ Early 1990To interconnect different types of LAN technologies in a multiprotocol enterprise environment.

Routers Position :Mid 1990

Switch When You Can, Route When You Must!

Routers Position :IP Service Aggregation CoreSeparate Networks

Major Functionalities of Internet RoutersRoute ProcessingRouting Table construction / maintenance / update using routing protocol.Packet ForwardingPacket ValidationDestination Address Parsing and Table LookupPacket Lifetime ControlChecksum CalculationQueuing & Scheduling (CoS/QoS)Special ServicePacket Translation, Encapsulation, Authentication, FilteringControl Plane ProcessingCommunicating with the rest of the system (Powers, Fans)Management protocols( SNMP, RMON, SMON)Other Admission control

Router Architecture and TechnologiesMajor Router Functionalities Major Hardware of RoutersOff-the-shelf ComponentsMemoryProcessorsMajor Components of RoutersThe Evolution of Router ArchitectureCase Study A Juniper Networks ExampleArchitecture OverviewRouting EngineForwarding EngineHardware & Architecture Flexibility

Why We Need Faster Router?1996199820002002100%1,000%10,000%100,000%DWDM Link speed x2/8 monthsInternet Traffic x2/1 yrRouter capacityx2.2/18 mMoores law x2/18 mDRAM access ratex1.1/18 mSource: SPEC95Int & David Miller, Stanford.To prevent routers from being the bottleneck

Why are Fast Routers Difficult to Make?CommercialDRAM x1.1/18 mMoores law x2/18 mRouter capacityx2.2/18 mDWDM Link speed x2/8 monthsAccess Time (ns)Source: Nick McKeown, Stanford.Speed of Commercial DRAM

Major Hardware of RoutersProcessorsASICNPUFPGACPUMemoryDRAM/SDRAMSRAMOff-the-shelf Components (Can buy from other vendors)CPUTransceiverMAC Chips (Ethernet and ATM SAR)SONET FramerTransceiverFramerNetwork processorCell bufferRouting tableScheduler + FQ

Major Hardware of Routers : ProcessorsForwardingRoutingServicesForwardingRouting TopologyServicesCommon ProcessorSoftware-based RouterHardware-based Router

Major Hardware of Routers : ProcessorsASIC (Application Specific Integrated Circuit)-- Hard-CodeASIP (Application Specific Instruction Processor)--(Network Processing Unit, NPU)ASIP(RISC)Co-processor-- FPGA (Field Programmable Gate Array)-- GPP (General Purpose Processor)-- Intel/AMD CISC CPU, PowerPC/MIPS/SPARC RISC CPU

Major Hardware of Routers : Processor decision considerationsSource: Niraj Shah, Understand Network PRocessor

A Compromise Solution Between Performance and FlexibilityPhysical Layer Forwarding EngineFabric InterfaceASICCPUSchedulerPacket Forwarding EngineFast PathSlow PathMemoryMemoryASIC

Major Hardware of Routers : NPU Architecture ExampleVLIW-Based NPURISC-based NPUParallel or/and Pipeline Core Processor DesignSource: Simon Stanley, Network Processors

Major Hardware of Routers : Various NPU Solutions

Major Hardware of Routers : MemoryDRAM (Dynamic RAM )(>60nsns=10-9) 1

PS. SDRAMDRAMDRAMPacket Buffer MemorySRAM (Static RAM )(4~10ns) 6(flip-flop) SRAM CPUDRAM(L2 cache) DRAMCPU 0.13m CMOS16MBSRAM0.25100Mb/s

Memory HierarchyControlDatapathSecondaryStorage(Disk)ProcessorOn-ChipCacheSecondLevelCache(SRAM)MainMemory(DRAM)TertiaryStorage(Tape)ThirdLevelCache(SRAM)

An Example: Packet buffersBufferMemoryBufferMemory40Gb/s router linecardBufferMemory10GbitsBuffer Manager Use SRAM? Fast enough random access time, but Too low density to store 10Gbits of data.

Use DRAM? High density means we can store data, but Cant meet random access time.

BufferMemory

Router Architecture and TechnologiesMajor Router Functionalities Major Hardware of Routers Major Components of RoutersInput/Output PortsProcessorSwitch FabricThe Evolution of Router ArchitectureCase Study A Juniper Networks ExampleArchitecture OverviewRouting EngineForwarding EngineHardware & Architecture Flexibility

Major Components of RoutersNetwork/RoutingProcessorSwitchFabricInput PortsOutput Ports

Major Components of Routers : Input/Output PortsLayer 1 Function : Line TerminationLayer 2 FunctionInput Port : Data-Link Protocol DecapsulationOutput Port : Data-Link Protocol EncapsulationOther Packet ProcessInput Port : Local Lookup/Forwarding/QueuingOutput Port :Buffer Management / QueuingLayer 1 Func:Line TerminationLayer 2 Func:Protocol DecapsulationLookup/Forwarding/QueuingLayer 1 Func:Line TerminationLayer 2 Func:Protocol EncapsulationBufferManagement/QueuingTo FabricInput PortFrom FabricOutput Port

Major Components of Routers : ProcessorMajor Processor TasksMaintain Routing TablePacket ProcessingDecapsulation/Encapsulation (Header Rewrite)Buffer ManagementClassificationForwarding Lookup QoS ManagementOthersAccounting (Log, Sampling)Network Management (SNMP, MIB)

EncapsulationSwitchFabricDecapsulationBuffer ManagementQoS ManagementPacket ClassifyForwarding LookupRouting TableMaintenanceProcessorINPUT OUTPUT

Major Components of Routers : Processor KernelsPattern MatchingMatching bits in packet fields (header/payload)Inputs : Regular expression pattern & packet fieldOutputs : A Boolean valueLookupLooking up data based on a key, mostly used in conjunction with pattern matching to find a specific entry in a table.Lookup type Exact Match (One-to-one) : ATM, MPLSLongest Prefix Match (Many-to-one) : IPv4, IPv6ComputationThe type of computation required for packet processing vary widelyEx. IPSec, Encryption, Decryption, Authentication, checksum, CRC valueData ManipulationAny function that modifies a packet headerEx. TTL decrement, adding tags/ header fields, replacing fields, segmentation, reassembly, fragmentationQueue ManagementThe Scheduling and storage of ingress and egress packetsControl ProcessingConsists of a number of different tasks that dont need to be performed at wire speedEx. Exceptions, table update, statistics gathering

Major Components of Routers : Processor Kernels

Major Components of Routers : Switch FabricInterconnect Input Ports to Output Ports, includes 3 modesBus All Input ports transfer data through the shared bus.Problem : Often cause in data flow congestion.Shared MemoryInput port write data into the share memory. After destination lookup is performed, the output port read data from the memory. Problem : Require fast memory read/write and management technology.CrossbarN input ports has dedicated data path to N output ports. Result in N*N switching matrix.Problem : Blocking (Input, Output, Head-of-line HOL). Max switch load for random traffic is about 59%.BusShared MemoryCrossbar

Queuing TechnologyInput QueuingOutput QueuingUsually a non-blockingswitch fabric (e.g. crossbar)Usually a fast busMemory b/w = 2RN : Number of Input/output portsR : Line Rate

Queuing Technology :Output QueuingIndividual Output QueuesCentralized Shared Memory12N12NN : Number of Input/output portsR : Line Rate

Switch Fabric : CrossbarHead-of-Line Blocking (HOL)

Switch Fabric : CrossbarVirtual Output Queue (VOQ)Require N*N BuffersN=Number of Output ports

Router Architecture and TechnologiesMajor Router Functionalities Major Hardware of Routers Major Components of RoutersThe Evolution of Router ArchitectureCase Study A Juniper Networks ExampleArchitecture OverviewRouting EngineForwarding EngineHardware & Architecture Flexibility

First Generation Routers :Single Processor, Shared BusTypically

Second Generation Routers : Multiple Processors, Shared BusBottlenecks:The shared bus still allowed only one packet at a time to move from input to output port.Route caching may not be efficient if cache not hit. The general purpose CPU in the slow path still been a bottle neck for specific traffic pattern.

RouteTableCPULineCardBufferMemoryLineCardMACBufferMemoryLineCardMACBufferMemoryFwdingCacheFwdingCacheMACBufferMemoryTypically

Third Generation Routers-Gigabit Switching Router Multiple Processors, Switched FabricBottlenecks:Switch Fabric Capacity : N * N MatrixPhysical limitation : Circuit density and number of (I/O) pinsInterconnection complexity and Power dissipationExampleC_12012(60Gbps) : 12*12C_12016(80Gbps) : 16*16C_12416(320Gbps) : 64*64C_12XXX(1Tbps) : 256*256?Slow PathHOL without VOQProblems :Distribute PFE architecture result in different performance and functionalities.

LineCardMACLocalBufferMemoryCPUCardLineCardMACLocalBufferMemorySwitched BackplaneRoutingTableFwdingTableTypically

Fourth Generation Routers-Multi-Terabit Switching RouterSwitch CoreLine cardsOptical links0.3 - 10Tb/s routers in developmentOptics inside a router for the first time

Fourth Generation Routers- Multi-Terabit Switching RouterInterface connectivitySwitch fabricconnectivityT640 T640 T640 T640 Greater than 10 Tbps5 Tbps WAN + 5 Tbps LOCALMatrix Technology

Router Architecture and TechnologiesMajor Router Functionalities Major Hardware of Routers Major Components of RoutersThe Evolution of Router ArchitectureCase Study A Juniper Networks ExampleArchitecture OverviewRouting EngineForwarding EngineData PathInternet Processor IIHardware & Architecture Flexibility

Juniper NetworksM&T Series Routers OverviewPacket Forwarding Performance per Rack InchM40M20M160M5/M101998.091999.12M40eT640T3202000.032000.092002.022002.072002.04A Growing History of Rapid InnovationIndustry's first true solution for high-performance accessIndustry's first 10G-class solution or ultra-high end accessIndustry's fastest router today. OC-768 ready.

Juniper NetworksM&T Series Routers OverviewPacketThroughputDensity

Juniper NetworksBreakthrough Density Ports per rack12416Redundant configurationsT640 redefines the core routing market

Juniper M-Series Chassis OverviewA M40e ExampleFrontRearPower Entry Modules (PEMs)PFE Clock Generators (PCGs)Routing Engines (REs)Switch Fabric Modules (SFMs)Miscellaneous Control Modules (MCSs)Flexible PIC Concentrators (FPCs)Physical Interface Cards (PICs)Craft Interface

Juniper M-Series ArchitectureRouting EnginePacketForwarding EngineSoftwareIntelligentHardwarePure Hardware(With Microcode)UpdateInternetProcessor IISwitch FabricJunos Internet SoftwareI/O CardI/O CardForwarding TableForwarding Table

Juniper M-Series Architecture :System PartitioningProblem is broken into two roughly equally complex parts that interact infrequentlyLoading of one does not affect the other, eliminating a common failure mode of legacy routersFacilitates independent hardware and software development and early software testingRE is standard off-the-shelf Intel platform, so it leverages industry advances in computer design and can be leveraged across multiple generations of FEs with no changeForwarding Engine (FE)RoutingEngine (RE)Why this partitioning is goodControl Packets OnlyAll Packets

Routing EngineFast Intel based Compact-PCI platform 768MB DRAMRouting TableForwarding TableStoragePrimary:80MB fixed flash memoryTwo Software ImagesTwo Configuration FilesMicrocodeSecondary:6.4GB IDE hard driveLog FilesMemory DumpsExternal :128MB PC Card flash driveCapacityBT Test : 450,000 Entries for Internet and MPLS VRF

Routing EngineAtomic Table UpdatesAdvantagesUpdated portion of forwarding table created separate from active tableNew portion switched into live table Single 32-bit atomic operation Done in one system clock cycleNo forwarding interruptionOther vendorsStop forwarding on all interface cards simultaneouslyUpdate table on each interface card

JUNOS Internet SoftwareCommon software across all platformsM-series and T-seriesInternet-class operating systemJUNOS 5.4 15th Major Release

Modular design for high reliability Protected memory architecture prevents one module from corrupting othersRapid software change and verificationRestart or upgrade specific module without rebooting entire chassisBest-in-class routing protocol implementations

Foundation for providing new features for servicesStandards basedMPLS VPNsIPv6

JUNOS Software Code Train3.23.33.44.04.14.24.34.45.05.15.25.35.4Flexible BandwidthPriority ServicesLayer 3 IP VPNsLayer 2 IP VPNsDoS Attack ContainmentManaged FirewallVirtual Leased LineFrame/ATM MigrationMulticast ServicesVoIP TransportPacket Sampling & CountingIPv6MPLSGraceful Restart

Packet Forwarding Engine :A M40e Example

Juniper M-Series : Shared Memory InterconnectEfficiency of memory bandwidthOne write, one readEase of multicastingOne write, number of reads

Juniper M-Series:PFE Architecture BenefitAll forwarding decisions are centralizedAll interfaces perform equally wellNew features added to Internet Processor become immediately available on every interface typeAll packet sizes are handled exactly the sameLatency through the PFE is constant across packet sizesLatency very low (< 10s)Adding additional FPCs adds additional shared memoryAvailable to any interface in the systemThere is never a possibility of memory starvation

Internet Processor II ASICIP II(Primitives):(Tree lookup)IPv4IPv6(Prefix) (Table lookup)MPLSTag(Filter instruction engine)(User-Defined Programs)(Compile)Internet Processor II ASIC

Internet Processor II ASIC : Flexible ArchitectureIP II ASIC

Internet Processor II ASIC : Enabling Smart IP ServicesIP MulticastCircuit Cross ConnectTranslational Cross ConnectRFC 2547bis VPNsGeneralized MPLSPremium FeaturesSecurityConvergence / MigrationMultiserviceVPNsSP IP ServicesService FeaturesJuniper NetworksEnablersDedicated HighSpeed AccessIP VPNsTransitServicesMPLS Traffic EngineeringLine-Rate ForwardingPacket Sampling & CountingFilter-Based ForwardingPacket Classification (CoS)Rate LimitingLine-Rate Packet Filtering

Juniper M-Series : Hardware PerformancePICMedia-Specific ASIC per interfaceFPCPacket Director ASICI/O Manager ASICSwitching Fabric ModuleDistribute Buffer Manager ASICInternet Processor II ASICAll ASICs in Packet Forwarding Engine

Source: Light Reading, March, 2001http://www.lightreading.com/testing/March 12, 2001, Juniper Networks announced winner of key and overall testing categoriesBest OverallBest OC-192cBest OC-48cBest MPLSBest IPJuniper NetworksCiscoThe M160 clearly demonstrates why Juniper has come so far in so few years This is truly the best core router available today.- David Newman

Juniper M-Series :Hardware FlexibilityFlexible Forwarding Engine All ASICs in forwarding data path are hard-coded ASICs with microcode instruction set. Intelligent HardwareIP II ASIC support various lookup by changing the components and order of primitives.JUNOSASICs Microcode can easily be re-programmed by JUNOS upgrading.Why Juniper Hardware with Flexibility?

Hard-CodeGTw\

MicrocodeGu\

ASIC

Juniper M-Series : Architecture FlexibilitySeparated Routing and Forwarding EngineRouting Engine : CPU cooperate with JUNOSForwarding Engine : ASICsRe-Programmable ASICs with Microcode Instruction SetI/O Manager ASICCan be programmed to recognize different types of frames, including IPv4, IPv6, Frame Relay, MPLS, and IPX.Distributed Buffer Manager ASICCan be programmed to look at any point in a packet header to extract forwarding information and build packet notifications.Intelligent Internet Processor II ASICsUsing millions logic to perform three primitives, not applications.Using JUNOS to chain different primitives in any order to perform various applications.Filter programs can be complied through CPU in Routing Engine.Modular Design JUNOSAccelerate the delivery of major releases. (4+ MR per year)Same software image for all platforms.Service PIC for special serviceOnly for specific traffic and wont affect common traffic flow.Why Juniper Hardware support IPv6/MPLS in any Interface?

Juniper IP Service PICsES PIC IPSec encryption up to 800-Mbps throughput rates (half duplex)1,000 IPSec tunnels or 2,000 security association (SA) pairs per PICMultilink Services PIC Aggregate throughput up to 450-Mbps, full-duplexSupports up to 128 bundles with 8 links per bundlePassive Monitoring PIC100-Kpps of monitoring performance per PICSupports 1 million recordsTunnel Services PIC IP-IP unicast tunneling. GRE unicast tunneling. PIM-SM encapsulation and de-encapsulation for locally attached hosts and rendezvous point operation.

Summary

CategoryFeatures

IP ScaleT-series & M-series: Single binary imageJUNOS seamless scale to multi-terabitIP Dependability Internet proven platformsInternet proven JUNOSIP SecurityAny port, any speed, any scaleNo compromiseIP Service RichnessAny port, any speed, any scaleNo compromise

Questions?

Thank You!