building gigabit-rate routers with th n ... - wiki.netfpga.org · pci netfpga driver (extensible)...

Building Gigabit-rate Routersith th N tFPGAwith the NetFPGA:

NICTA Tutorial at UNSWPresented by:

John W. Lockwood, Jad Naous, Glen Gibb (Stanford University)(Stanford University)

Hosted by: Lavy Libman (NICTA) and Philip Allen (UNSW)

February 6, 2008: 9am-5pmLab 343A, Electrical Engineering Building (G17)Lab 343A, Electrical Engineering Building (G17)

Kensington Campus, University of New South WalesSydney, Australia

NICTA 2008 - NetFPGA Tutorial 1 S T A N F O R D U N I V E R S I T Y

http://NetFPGA.org

What is the NetFPGA?

CPU MemoryNetworkingSoftwareSoftwarerunning on a standard PC

1GE

PCI PC with NetFPGA

A hardware

FPGA1GE

1GE

1GE

acceleratorbuilt with Field Programmable Gate Array

Memory 1GENetFPGA Board

Gate Arraydriving Gigabit network links


Introduction

Who uses the NetFPGA• Teachers• Students• Researchers

How they use the NetFPGA1. To run the Router Kit2 To build modular reference designs2. To build modular reference designs

• IPv4 router• 4-port NIC• Ethernet switch, …

3. To create new systems


Running the Router Kit

User-space development, 4x1GE line-rate forwarding

CPU MemoryOSPF BGP

My Protocoluser

PCI

kernelRouting

Table

“Mi ”

FPGA1GE

1GE

1GE

1GE

FwdingTable

PacketBuffer

“Mirror”

Memory

1GE

1GE

1GE

IPv4Router

1GE

1GE

1GE


Building Modular Router Modules

PW OSPFCPU Memory

Java GUIFront Panel

PW-OSPFVerilog

EDA Tools(Xilinx,

Mentor etc )

PCI NetFPGA Driver

(Extensible) Mentor, etc.)

1 Design

FPGA1GE

1GEIn Q

MgmtL2

ParseL3

Parse

1GE

1GE

1. Design2. Simulate3. Synthesize4. Download

Memory

1GE

1GE

1GE

Mgmt

IPLookup

ParseParse

Out QMgmt

1GE

1GE

1GE

MyBlock


1GEVerilog modules interconnected by FIFO interfaces

Creating new systems

CPU MemoryVerilog

EDA Tools(Xilinx,

Mentor etc )

PCI NetFPGA Driver

Mentor, etc.)

1. Design2. Simulate3 Synthesize

FPGA1GE

1GE

1GE

1GEMy Design

3. Synthesize4. Download

Memory

1GE

1GE

1GE

1GE

1GE

1GE

My Design

(1GE MAC is soft/replaceable)


1GE

Tutorial OutlineB k d• Background

– Basics of an IP Router– The NetFPGA Platform

• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router

• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router

Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA


Basic Operation of an IP RouterR3

AR1

R4 DA

B

R2

E

C R2FR5

R3DNext HopDestination

R5FR3ER3D


What does a router do?R3

AR1

R4 DA16 3241

Total Packet LengthT.ServiceHLenVerB E

Header ChecksumProtocolTTL

Fragment OffsetFlagsFragment ID

0 by

tes

C R2FR5

R3DNext HopDestination

O ti (if )

Destination Address

Source Address20

R5FR3ER3D

Data

Options (if any)


What does a router do?

R3

AR1

R4 D

B EB

C R2

E

CFR5


Basic Components of an IP Router

Management& CLI

S

Control PlaneR ti

Routing Protocols

& CLI

Softw

are Control PlaneRoutingTable

e

Datapathper-packet SwitchingForwarding

Table

Hardw p p

processing

ware


Per-packet processing in an IP Router

1. Accept packet arriving on an incoming link.2. Lookup packet destination address in the

forwarding table, to identify outgoing ( )port(s).

3. Manipulate IP header: e.g., decrement TTL, update header checksumupdate header checksum.

5. Buffer packet in the output queue.6 Transmit packet onto outgoing link6. Transmit packet onto outgoing link.


Generic Datapath Architecture

Lookup Update

Header ProcessingData Hdr Data Hdr

QueueLookupIP Address

UpdateHeader

QueuePacket

Forwarding

IP Address Next Hop

BufferForwardingTable

BufferMemory


CIDR and Longest Prefix MatchesThe IP address space is broken into line segments.Each line segment is described by a prefix.A prefix is of the form x/y where x indicates the prefix of all addresses in the line segment, and y indicates the length of the segment.

Th fi 128 9/16 t th li te.g. The prefix 128.9/16 represents the line segment containing addresses in the range: 128.9.0.0 … 128.9.255.255.

128 9 0 0

128.9/16

128.9.0.0 142.12/1965/8

0 232-1216

128 9 16 14


128.9.16.14

Classless Interdomain Routing (CIDR)

128 9 19/24

128 9 16/20 128 9 176/20

128.9.19/24128.9.25/24

128.9/16

128.9.16/20 128.9.176/20

0 232-1

128.9.16.14

Most specific route = “longest matching prefix”


Techniques for LPM in hardware• Linear search• Direct lookup

– Currently requires too much memory– Updating a prefix leads to many changes

Tries• Tries– Deterministic lookup time– Easily pipelinedEasily pipelined– But requires multiple memories/references

• TCAM (Ternary CAM)( y )– Simple and widely used– But low-density, high-power


– Gradually being replaced by new algorithms

An IP Router on NetFPGA

Management& CLI

S

i l l

R ti

Routing Protocols

& CLI

Softw

are

Linux user-levelprocesses

ExceptionP i Routing

Table

eVerilog on

Processing

SwitchingForwardingTable

Hardw

Verilog on NetFPGA PCI board

ware


NetFPGA Router

Function – 4 Gigabit Ethernet ports

Fully programmable– FPGA hardware

Low cost

Open-source FPGA hardware – Verilog base design

Open-souce Software– Drivers in C and C++


Drivers in C and C

NetFPGA PlatformMajor Components

– Interfaces4 Gi bit Eth t P t• 4 Gigabit Ethernet Ports

• PCI Host Interface

– Memories• 36Mbits Static RAM• 512Mbits DDR2 Dynamic RAM• 512Mbits DDR2 Dynamic RAM

– FPGA Resources• Block RAMs• Configurable Logic Block (CLBs)• Memory Mapped Registers


• Memory Mapped Registers

NetFPGA System

Browser& VideoMonitor

SoftwareCADT l

Web &Video

User Space

Linux Kernel

ClientSoftwareTools Server

Linux Kernel

PCI-ePCI

Packet Forwarding Table

NI

NetFPGA Router

VI VI VI VI

CGGGGGG

NetFPGA RouterHardware


GE

GE

GE

GE

GE

GE

(eth1 .. 2)(nf2c0 .. 3)

NetFPGA Hardware


NetFPGA System Implementation• NetFPGA Blocks

– Virtex-2 Pro FPGA– 4.5MB ZBT SRAM– 64MB DDR2 DRAM 6– PCI Host Interface– 4 Gigabit Ethernet ports

• Intranet Test PortsIntranet Test Ports – Dual or Quad Gigabit

Etherents on PCI-e

• InternetInternet – Gigabit Ethernet

on Motherboard

• ProcessorProcessor – Dual-Core CPU

• Operating System– Linux CentOS 4 4


Linux CentOS 4.4

NetFPGA Lab Setup

GEPClient Eth2 : Server

(eth1 .. 2)

Nf2c3 : Adj ServerCPU x2 Net FPGA

Dual NICGEPC

I-e

ClientGE

GE

Eth2 : Server

ServerEth1 : Local host

Nf2c3 : Adj. ServerNet-FPGAPC

INetFPGAControl SW

GE

GE

GE

GE

InternetRouter

HardwareNf2c1 : Adjacent

Nf2c2 : Local Host

Nf2c0 : AdjacentGECAD ToolsNf2c0 : Adjacent


NetFPGA Hardware Set for Demo #1

Net-FPGA GE

GEInternet

CPU x2

PCI-e

PC

VideoServer

NICGE

PCI-e GE

GE

GE

InternetRouter

Hardware

CI

Server delivers

Net-FPGA GE

GE

GE

InternetRouter

Hardware

streaming HD videothrough a GEHardware

…

through a chain of NetFPGA R

Net-FPGACPU x2

NICGE

PCI-e

PCVideo

GE

GE

GEInternet

…Routers


CI

Display

CAD Tools

GE

GE

te etRouter

Hardware

Topology of NetFPGA Routers

VideoServerServer

HDDisplayDisplay


Setup for the Reference Router

Each NetFPGA card has four ports NetFPGA

Video Server

has four ports

Port 2 connected to Cli t / S

NetFPGA

Client / Server

Ports 0 and 3 connected to NetFPGAadjacent NetFPGA cards

NetFPGA

Video Client

NetFPGA


Demo 1: Logical Topology

1 1 4 1 7 1 10 1 13 1 16 1.1.1

.1.2.3.1

.30.2

.4.1

.4.2

.6.1.3.2

.7.1

.7.2

.9.1.6.2

.10.1

.10.2

.12.1.9.2

.13.1

.13.2

.15.1.12.2

.16.1

.16.2.15.2

.17.1.5.1 .8.1 .11.1 .14.1 .18.1

.2.1.18.1

27 1

.30.1

.24.1.27.2

.21.1.24.2 .21.2

.18.2.20.1

.23.1.26.1

.29.1

.28.1

.28.2 .27.1

.25.1

.25.2.24.1

.22.1

.22.2.21.1

.19.1

.19.2


Video ClientShortest Path

Video Server

Working IP Router

• ObjectivesObjectives – Become familiar with

Stanford Reference Router

– Observe PW-OSPF re-routing traffic around a failure


Streaming Video through the NetFPGA

• Video server– Source files

/var/www/html/video

– Network URL : http://192.168.Net.Host/Video

Vid li• Video client– Windows Media Player– Linux mplayer

• Video traffic– MPEG2 HDTV (35 Mbps)( p )– MPEG2 TV (9 Mbps)– DVI (3 Mbps)– WMF (1.7 Mbps)


( p )

Step 1 – Observe the Routing TablesThe router is already

configured and running on your machines

The routing table has converged to theconverged to the routing decisions with minimum number of hops


Next, break a link …

Step 2 - Dynamic Re-routing

Break the link between videobetween video server and video client

.1.1

.1.2.3.1

.30.2

.4.1

.4.2

.6.1.3.2

.7.1

.7.2

.9.1

.6.2

.10.1

.10.2

.12.1

.9.2

.13.1

.13.2

.15.1

.12.2

.16.1

.16.2.15.2

.17.1.5.1 .8.1 .11.1 .14.

1.18.1

.2.1

Routers re-route t ffi d th

.28.2.27.1

.30.1

.25.2.24.1

.27.2

.22.2.21.1

.24.2

.19.2

.21.2.18.2

.20.1

.23.1.26.1

.29.1

traffic around the broken link and video continues

.28.1 .25.1 .22.1 .19.1

video continues playing


Integrated Circuit Technology

Full-custom Design – Complementary Metal Oxide Semiconductor (CMOS)Complementary Metal Oxide Semiconductor (CMOS)

Semi-custom ASIC Design G t– Gate array

– Standard cell

Programmable Logic Device– Programmable Array Logic

Field Programmable Gate Arra s– Field Programmable Gate Arrays

Processors


Look-Up TablesCombinatorial logic is stored

in Look-Up Tables (LUTs) – Also called

A B C D Z0 0 0 0 00 0 0 1 0Also called

Function Generators (FGs)– Capacity is limited only by

number of inputs, not complexity

0 0 0 1 00 0 1 0 00 0 1 1 1p , p y

– Delay through the LUT is constant 0 1 0 0 10 1 0 1 1

Combinatorial Logic

A

. . .1 1 0 0 01 1 0 1 0

B

CD

Z 1 1 1 0 01 1 1 1 1


Diagram From: Xilinx, Inc

Xilinx CLB Structure

Slice 0

Each slice has four outputs– Two registered outputs,

two non-registered outputs Slice 0

LUT Carry D QCE

PRE

CLR

g p– Two BUFTs associated

with each CLB, accessible by all 16 CLB outputs

CLRy p

Carry logic run verticallyLUT Carry D

QCEPRE

CLR

Carry logic run vertically – Signals run upwards– Two independent

carry chains per CLBcarry chains per CLB

Diagram From: Xilinx Inc


Diagram From: Xilinx, Inc (Courtesy Jeff Weintraub)

Field Programmable Gate ArraysDin Clk

CLB– Primitive element of FPGA

4 LUT

G4G3G2G1

G

4 LUT

F4F3F2F1

F

3 LUT

H

S

R

D Q

S

R

D Q

H1

YQ

Y

XQ

X

M

M

M

M

CLB

Routing Module– Global routing

Local interconnect

GRMLocal Routing

CLB PIP– Local interconnect

3rd Generation LUT-based FPGA

Macro Blocks– Block Memories– Microprocessor

...

... ...

Macro

I/O Block

... ...

......

Block(uP,Mem)


Pad Routing CLB Matrix I/O

NetFPGA Block Diagram

NetFPGA platform

1M

18MSR

A1P

FPGA w/provided infrastructureFo 1GE

M

AC

1GE

MAC

Virtex II-Pro 50 FPGA withuser-defined network logic* Hardware specified with

- Verilog source code

Mb

AM

1GE

PH

Y1G

EPH

Y

64MD

DR

SDR

A

ur Gigabit E E

C1G

E

MA

C1M

g- Pre-generated cores

* Software written for - Embedded PowerPCs - Soft core processors

E Y

1GE

PH

Y1P

BR2AM

Ethernet Inte

18Mb

SRA

M

Board-B1G

E

MAC (Microblaze, LEON ..)

1GE

PH

Y

erfaces

M

FIFOpacket Control, PCI

Interface

3 Gb

SATA

Board Interc

Linux OS - NetFPGA Kernel driverHostcomputer

User defined software networking applications

buffers Interface

connect


User-defined software networking applications

Details of NetFPGA

• Fits into Standard PCI slotStandard Bus : 32 bits 33 MHz– Standard Bus : 32 bits, 33 MHz

• Provides Interfaces for processing network packets– 4 Gigabit Ethernet Ports

• Allows hardware-accelerated processing


• Allows hardware-accelerated processing – Implemented with Field Programmable Gate Array (FPGA) Logic

Hardware Description LanguagesC• Concurrent– By Default, Verilog statements

l t d tlevaluated concurrently

• Express fine grain parallelismExpress fine grain parallelism– Allows gate-level parallelism

• Provides Precise Description– Eliminates ambiguity about operation

• SynthesizableG t h d f d i ti


– Generates hardware from description

Verilog Data Typesreg [7:0] A; // 8-bit register, MSB to LSB

// (Preferred bit order for NetFPGA)reg [0:15] B; // 16-bit register LSB to MSBreg [0:15] B; // 16 bit register, LSB to MSB

B = {A[7:0],A[0:7]}; // Assignment of bits

reg [31:0] Mem [0:1023]; // 1K Word Memory

integer Count; // simple signed 32-bit integerinteger K[1:64]; // an array of 64 integerstime Start Stop; // Two 64-bit time variablestime Start, Stop; // Two 64-bit time variables


From: CSCI 320 Computer ArchitectureHandbook on Verilog HDL, by Dr. Daniel C. Hyde :

http://eesun.free.fr/DOC/VERILOG/verilog-manual.html

Signal Multiplexers

Two input multiplexer (using if / else)reg y;always @*

if (select)y = a;

elsey = b;

Two input multiplexer (using ternary operator ?:)p p ( g y p )

wire t = (select ? a : b);


From: http://eesun.free.fr/DOC/VERILOG/synvlg.html

Larger Multiplexers

Three input multiplexer

reg s;always @*

begincase (select2)

2'b00: s = a;;2'b01: s = b;default: s = c;

endcaseend



Synchronous Storage Elements• Values change at• Values change at

times governed by clock Clock

Din DoutQD

Clock Transition1Clock– Clockt=0 t=1 t=2

0Clock

time• Input to circuit

Clock Event

A B CDint 0

– Clock Event• Example: Rising edge

Clock Transition

S0Doutt=0

A B

t=0– Flip/Flop

• Transfers Value From Dinto Dout on Clock event


t=0to Dout on Clock event

Finite State Machines

Copyright 2001, John W. Lockwood, All Rights Reserved

Inputs (X)Outputs (Z)=λ (X,S(t))

[Moore](S(t))=λ

[Mealy]-or-

Combinational Logic

N tQ D

S(t) δS(t+1)=(X,S(t)) State

Next

...Q D

State Storage


Synthesizable Verilog : Delay Flip/Flops

D-type flip flopreg q;always @ (posedge clk)always @ (posedge clk)

q <= d;

D type flip flop with data enableD type flip flop with data enablereg q; always @ (posedge clk)

if (enable) q <= d;q < d;



Reference Router Pipeline• Five stages

– InputMACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

– Input Arbitration– Routing Decision and Input Arbiter

packet modification– Output Queuing

O t tOutput Port Lookup

– Output• Packet-based module

interfaceOutput Queues

interface• Pluggable design

MAC CPU MAC CPU MAC CPU MAC CPU

SIGMETICS 2007 - NetFPGA Tutorial 49 S T A N F O R D U N I V E R S I T Y

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Make your own router

Objectives: Object es– Learn how to build hardware– Run the software

E l t hit t– Explore router architecture

ExecutionExecution– Start synthesis– Rerun the GUI with the new hardware

T t ti it d t ti ti ith i– Test connectivity and statistics with pings– Explore pipeline in the details page– Explore detailed statistics in the details page


p p g

Step 1 - Build the hardware

Start terminal cd toStart terminal, cd to “NF2/projects/tutorial_router/synth”

Start synthesis with “make”


Step 2 - Run Homemade Router

cd to “NF2/projects/tutorial_router/sw”

Type: “tutorial router gui pl” totutorial_router_gui.pl touse the just built router hardwarea d a e

The same interface shouldstart again


Step 4 - Connectivity and StatisticsPing any addresses

192.168.x.y where x is from 1-20 and y is 1 or 2

O th t ti ti t b iOpen the statistics tab in the Quickstart window to see some statisticssee some statistics

Explore more statistics inExplore more statistics in modules under the details tab


Step 5 - Explore Router Architecture

Click the Details tab of the Quickstart windowthe Quickstart window

This is the reference router pipeline – a canonical simple tocanonical, simple to understand, modular router pipelinep p


Step 6 - Explore Output QueuesClick on the Output

Queues module in the Details tab

The page gives configuration details

…and statistics


Buffer Requirements in a RouterBuffer size matters:

– Small queues reduce delayLarge buffers are expensive– Large buffers are expensive

Theoretical tools predict requirements– Queuing theory– Large deviation theory– Mean field theoryMean field theory

Yet, there is no direct answer.Fl h l d l t– Flows have a closed-loop nature

– Question arises on whether focus should be on equilibrium state or transient state..


Rule-of-thumb

CRouterSource Destination

• Universally applied rule-of-thumb:A router needs a buffer size: CTB ×2

2T

– A router needs a buffer size:– 2T is the two-way propagation delay (or just 250ms)– C is capacity of bottleneck link

CTB ×= 2

p y• Context

– Mandated in backbone and edge routers.A i RFP d IETF hit t l id li– Appears in RFPs and IETF architectural guidelines.

– Already known by inventors of TCP • [Van Jacobson, 1988]


[ , ]– Has major consequences for router design

The Story So Far

10,000 20# packetsat 10Gb/s

1,000,000

(1) Assume: Large number of desynchronized flows; 100% utilization(2) A : L b f d h i d fl ; <100% tili ti


(2) Assume: Large number of desynchronized flows; <100% utilization

Using NetFPGA to explore buffer size

• Need to reduce buffer size and measure occupancy

• Alas, not possible in commercial routers• So, we will use NetFPGA instead

Objective:j– Use NetFPGA to understand how large a buffer

we need for a single TCP flow.


Why 2TxC for a single TCP Flow?

Rule for adjusting WIf ACK i i d W W+1/WOnly W packets – If an ACK is received: W ← W+1/W

– If a packet is lost: W ← W/2

y pmay be outstanding


Time Evolution of a Single TCP Flow

Time evolution of a single TCP flow through a router. Buffer is < 2T*CTime evolution of a single TCP flow through a router. Buffer is 2T*C


NetFPGA Hardware Set for Demo #2

NICGE

PC

…

Net-FPGACPU x2

NICI-ePC

IVideoClient

GE

GE

GE

GE

InternetRouter Server

GEHardware deliversstreaming HD video

PCI-e

VideoServer

NICGE

PCI-e GE

HD videoto adjacent client

CPU x2


Setup for the Demo 2

Each NetFPGA card has four portshas four ports

Port 2 connected to L l H tLocal Host

Port 3 connected to AdjacentServer

adjacent Server

LocalHost

NetFPGA


Topology for Second DemonstrationRouters connected point-to-point topologyPort 3 connects to local hostPort 1 connects to adjacent neighborPorts 0 and 2 n sedPorts 0 and 2 unused

.2.1 .5.1 .8.1 .11.1

.1.1 .1.2

.4.1

.4.2

.7.1

.7.2

.10.1

.10.2

.13.1

.13.2.2.2 .5.2 .8.2 .11.2

.14.2.29.1

.14.126 2

.29.2

.16.1.16.2

.28.1

.28.2

.25.1

.25.2

.22.1

.22.2

.19.1

.19.2.17.2

.17.1

.20.2

.20.1

.23.2

.23.1

.26.2

.26.1


Enhanced Router

ObjectivesObjectives – Observe router with new modules– New modules: rate limiting, delay, event capture

Execution– Run event capture router– Run event capture router– Look at routing tables– Explore details pane– Start tcp transfer, look at queue occupancy– Change rate/delay, look at queue occupancy


Step 1 - Run Pre-made Enhanced Router

Start terminal and cd to “NF2/projects/tutorial roNF2/projects/tutorial_router/sw/”

Type “./tut adv router gui.pl”_ _ _g p

A familiar GUI should start


Step 3 - Explore Enhanced Router

Cli k th D t il t bClick on the Details tab

A similar Pipeline to the one seenthe one seen previously shows with some additions


Enhanced Router PipelineTwo modules added

1. Event CaptureMACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

to capture output queue events (writes reads

Input Arbiter

O t t P t L k(writes, reads, drops)

Output Port Lookup

Event Capture

2 Rate Limiter topOutput Queues

2. Rate Limiter to create a bottleneck MAC

TxQCPUTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

RateLimiter


MACTxQ

Step 4 - Decrease the Link RateTo create bottleneck and

show the TCP “sawtooth”, li k t i d dlink-rate is decreased.

I th D t il t b li k thIn the Details tab click the “Rate Limit” module

Check Enable

Set link rate to 1.953Mbps


Step 5 – Decrease Queue Size

Go back to the DetailsGo back to the Details Panel and click on “Output Queues”.

Select the “Output Queue 2” tab.

Change the output queues size in packets slider to 16


slider to 16

Step 6 - Start Event Capture

Click on the Event CaptureClick on the Event Capture module under the Details tab

This should start the configuration page


Step 7 - Configure Event Capture

Check Send to local h t t i thost to receive events on the local host

Check Monitor Queue 2to monitor output queueto monitor output queue of MAC port1

Check “Enable Capture” to start Event capture


Step 8 - Start TCP Transfer

We will use iperf to run aWe will use iperf to run a large TCP transfer and look at queue evolutionoo at queue e o ut o

Start a terminal and cd to“NF2/projects/tutorial_router/sw”

type “iperf.sh”


Step 9 - Look at Event Capture Results

Click on the Event Capture module underCapture module under the details tab.

The sawtooth patternThe sawtooth pattern should now be visible.


Queue Occupancy Charts


NetFPGA in the Classroom

• Stanford CS344: “Build an Internet Router”– Courseware will be available later in 2007– Students work in teams of three (2 software, 1

hardware)– Design and implement hardware and software

in 8 weeksS ft CLI PW OSPF– Software: CLI, PW-OSPF

– Show interoperability with other groups Add ne feat res in remaining t o eeks– Add new features in remaining two weeks

• Firewall, NAT, DRR, Packet capture, Data generator, …


g ,

Networked FPGAs in Research1. RCP: Congestion control

• New module for parsing and overwriting new packet• New software to calculate explicit ratesp

2. Packet Monitoring (ICSI)• Network Shunt

3. Deep Packet Inspection (FPX)3. Deep Packet Inspection (FPX)• TCP/IP Flow Reconstruction• Regular Expression Matching• Bloom FiltersBloom Filters

4. Ethane: Network security• New switch (“managed flow-table”) deployed

5 Buffer Sizing5. Buffer Sizing• Reduce buffer size and measure effect on network

performance.• Need a way to set buffer size and measure buffer


• Need a way to set buffer size, and measure buffer occupancy.

Enhance Your Router

ObjectivesObjectives – Add new modules to datapath– Synthesize and test routery

Execution– Open user_datapath.v, uncomment

delay/rate/event capture modulesS h i– Synthesize

– After synthesis, test the new system.


An aside: xemacs TipsWe will be modifying the Verilog source codeSlides show xemacs, but vim also available.xemacs:

– To undo, use ctrl+shift+'-'T l lti k t k d j t t t l– To cancel a multi-keystroke command, just type ctrl+g

– To select lines, hold shift and press the arrow keys.– To comment some selected lines, type ctrl+c+c, yp– To uncomment a commented block, move the cursor to

one of the lines inside the commented block and type ctrl+c+uctrl+c+u

– To save type ctrl+x+s– To search, type ctrl+s search_pattern


Step 1 - Open the SourceWe will modify the Verilog

source code to add event capture, rate limiter, and delay modules

We will simply comment and uncomment existing codeuncomment existing code

Open terminalpType “xemacs

NF2/projects/tutorial_router/sr/ d t th


c/user_data_path.v

Step 2 - Add wires

Now we need to add wires t t thto connect the new modules

Search for “new wires” (ctrl+s new wires) then(ctrl s new wires) then press Enter

Uncomment the wires (ctrl+c+u)


Step 3 - Connect Event Capture

Search for opl_output (ctrl+s opl_output) then press Enter

Comment the four lines above (up, shift + up + up + up + up ctrl+c+c)up, ctrl+c+c)

Uncomment the block below toUncomment the block below to connect the outputs (ctrl+s opl out, ctrl+c+u)


p _ , )

Step 4 - Add the Event Capture Module

Search for evt capture topSearch for evt_capture_top (ctrl+s evt_capture_top) then press Entere p ess e

Uncomment the block (ctrl+c+u)


Step 5 - Connect the Output Queue to the Rate Limiter

Search for port_outputs (ctrl+s ports outputs, Enter)( p _ p , )

Comment the 4 lines above (select the four lines by using shift+arrow keys, th t t l )then type ctrl+c+c)

U t th t dUncomment the commented block by scrolling down into the block and typing


the block and typing ctrl+c+u

Step 6 - Add Rate Limiter

Scroll down until you reachScroll down until you reach the next “Excluded” block

Uncomment the block containing the rate limiter instantiationsinstantiations. (scroll into the block and type ctrl+c+u)

Save (ctrl+x+s)


Step 7 - Build the hardware

Start terminal, cd to “NF2/projects/tutorial_router/synth”ter/synth”

Start synthesis with “make”Start synthesis with make


Full System Components

Softwaref2 0 f2 1 f2 2 f2 3 i tl

PCI Bus

nf2c0 nf2c1 nf2c2 nf2c3 ioctl

CPUR Q

CPUT Q

nf2_reg_grpCPU CPUCPU CPUCPU CPU

NetFPGARxQ TxQ

user data path

RxQ TxQCPURxQ

CPUTxQ

CPURxQ

CPUTxQ

MACTxQ

MACRxQ

MACTxQ

MACRxQ

MACTxQ

MACRxQ

MACTxQ

MACRxQ

NICTA 2008 - NetFPGA Tutorial 92 S T A N F O R D U N I V E R S I T YEthernet

Life of a Packet through the hardware

port0 port2192.168.2.y192.168.1.x


Router Stages Again

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

Input Arbiter

Output Port Lookup

Output Queues

MAC CPU MAC CPU MAC CPU MAC CPU


MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Inter-module Communication

Using “Module Headers”:Data WordCtrl Word

Module Hdrx Contain information such as packet

(64 bits)(8 bits)

Eth Hdr0Last Module Hdry

……p

length, input port, output port, …

IP Hdr…

00

Last word of packet0x10 Last word of packet0x10


Inter-module Communication

data

ctrlwrwr

rdy


MAC Rx Queue


Rx Queue

Eth Hdr:

Pkt length,input port = 00xff

IP Hdr:IP Dst: 192 168 2 3

Dst MAC = port 0, Ethertype = IP

0

0

IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4

Data

0

0


Input Arbiter

Pkt

PktPkt


Pkt

Output Port Lookup


Output Port Lookup1- Check input port matches

Dst MAC

5- Add output port module


output port = 40x04Dst MAC

2- Check TTL, checksum

6- Modify MAC Dst and Src

IP HdIP Hd

EthHdr: Dst MAC = 0Src MAC = x,Ethertype = IP

0

input port 0checksum

3- Lookup next hop IP &

addressesEthHdr: Dst MAC = nextHopSrc MAC = port 4,

Ethertype = IPIP Hdr:

IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4

IP Hdr:IP Dst: 192.168.2.3,

TTL: 63, Csum:0x3ac2

D t

0

0

output port (LPM)

4- Lookup

7-Decrement TTL and update

checksumData04 Lookup next hop MAC address (ARP)

checksum


Output Queues

OQ0OQ0

OQ4

OQ7


MAC Tx Queue


MAC Tx Queue

Pkt length,0xff

output port = 40x04

EthHdr: Dst MAC = nextHopSrc MAC = port 4,

Ethertype = IP0

input port = 00xff

IP Hdr:IP Dst: 192.168.2.3,

TTL: 64, Csum:0x3ab4

IP Hdr:IP Dst: 192.168.2.3,

TTL: 63, Csum:0x3ac2

Ethertype IP

0

Data0


Exception Packet

• Example: TTL = 0 or TTL = 1• Packet has to be sent to the CPU which will

generate an ICMP packet as a response• Difference starts at the Output Port lookup p p

stage


Exception Packet Path


PCI Bus


CPU CPU CPU CPU CPU CPU CPU CPU nf2_reg_grp

NetFPGA

CPURxQ

CPUTxQ

CPURxQ

CPUTxQ

CPURxQ

CPUTxQ

CPURxQ

CPUTxQ

user data path

MACTxQ

MACRxQ

MACTxQ

MACRxQ

MACTxQ

MACRxQ

MACTxQ

MACRxQ


Output Port Lookup1- Check input port matches

Dst MAC


output port = 10x04Dst MAC

2- Check TTL, checksum –

IP Hd

EthHdr: Dst MAC = 0,Src MAC = x,Ethertype = IP

0

input port 0checksum EXCEPTION!

IP Hdr:IP Dst: 192.168.2.3,

TTL: 1, Csum:0x3ab4

D t

0

0

3- Add output port module

Data0


Output Queues

OQ0OQ0

OQ1

OQ2

OQ7


CPU Tx Queue


CPU Tx Queue

Pkt length,0xff

output port = 10x04

EthHdr: Dst MAC = 0, Src MAC = x,Ethertype = IP

0

input port = 00xff

IP Hdr:IP Dst: 192.168.2.3,

TTL: 1, Csum:0x3ab4

Ethertype IP

0

Data0


ICMP Packet

• For the ICMP packet, the packet arrives at the CPU Rx Queue from the PCI Bus

• Follows the same path as a packet from the MAC until the Output Port Lookup.

• The OPL module seeing the packet is from the CPU Rx Queue 1, sets the output port directly to 0.

• The packet then continues on the same path as the non-exception packet to the Output Queues and then MAC Tx queue 0.


ICMP Packet Path


PCI Bus


CPU CPU CPU CPU CPU CPU CPU CPU nf2_reg_grp

NetFPGA

CPURxQ

CPUTxQ

CPURxQ

CPUTxQ

CPURxQ

CPUTxQ

CPURxQ

CPUTxQ

user data path

MACTxQ

MACRxQ

MACTxQ

MACRxQ

MACTxQ

MACRxQ

MACTxQ

MACRxQ


NetFPGA-Host Interaction

• Linux driver interfaces with hardware– Packet interface via standard Linux network

stack

– Register reads/writes via ioctl system call (with convenience wrapper functions)

• readReg(nf2device *dev int address unsigned *rd data)• readReg(nf2device *dev, int address, unsigned *rd_data)• writeReg(nf2device *dev, int address, unsigned *wr_data)

eg:eg:readReg(&nf2, OQ_NUM_PKTS_STORED_0, &val);



NetFPGA to host packet transfer1. Packet arrives –forwarding table sends to CPU queue

P2. Interrupt tifi 3 Driver sets upC

I Bus

notifies driver of packet arrival

3. Driver sets up and initiates DMA transfer



NetFPGA to host packet transfer (cont)

PC4. NetFPGA t f

5. Interrupt signals C

I Bus

transfers packet via DMA

completion of DMA

6. Driver passes packet to network stack


network stack


Host to NetFPGA packet transfers

PC

3. Interrupt signals

2. Driver sets up and initiates C

I Bus

completion of DMA

DMA transfer

1. Software sends packet via network sockets. Packet delivered to driver


Packet delivered to driver.


Register access

PC

2. Driver performs C

I Bus

pPCI memory read/write

1. Software makes ioctl call on network socket. ioctl passed to driver


ioctl passed to driver.


• Packet transfers shown using DMA interface

• Alternative: use programmed IO to transfer p gpackets via register reads/writes– slower but eliminates the need to deal with

network sockets


Step 8 – Perfect the Router

If interested, go back to “Demo 2: Step 1” after synthesis is done and redo the steps with your ownsynthesis is done and redo the steps with your own router.

You can also change the bandwidth and queue size settings to see how that effects the queue occupancy evolutionevolution.

To run your router:1- cd NF2/projects/tutorial_router/sw2- type “./tut_adv_router_gui.pl --use_bin

/ / /bitfiles/tutorial router bit”


../../../bitfiles/tutorial_router.bit

We’re done!We re done!

Congratulations!


AcknowledgementsNetFPGA Team : January 2007

Jianying Luo, Glen Gibb, Nick McKeown, Greg Watson, Jim Weaver, Jad Naous, Ramanan Raghuraman,

Paul Hartke John Lockwood


Paul Hartke, John Lockwood

Acknowledgements• Support for the NetFPGA project has been provided

by the following companies and institutions

Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in this


y p g pmaterial do not necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this project.

Reference on the Web

NetFPGA homepagep g

http://NetFPGA.orgp g


building gigabit-rate routers with th n ... - wiki.netfpga.org · pci netfpga driver (extensible)...

Documents