Open Source Routing, Firewalls and Traffic Shaping

Russell SutherlandComputing and Networking ServicesUniversity of Torontorussell.sutherland@utoronto.ca

Reference URLs for Tree Huggers This presentation

– http://madhaus.cns.utoronto.ca/~russ/canheit2004/

Routing

– http://www.quagga.net/– http://www.xorp.net/– http://latrc.org/

Traffic Shaping

– Linux http://tcng.sourceforge.net/– FreeBSD http://info.iet.unipi.it/~luigi/

ip_dummynet/

Packet Filtering

– Linux (iptables) http://www.netfilter.org/– FreeBSD (ipfw) http://www.freebsd.org/– OpenBSD (pf) http://www.benzedrine.cx/pf.html

Routing Chronology

1984 BSD 4.2 ships with routed (RIPv1)

1986 Fuzz Ball PDP-11 NSFNet Routers

1988 Age of dedicated routing machines– Cisco, Proteon, Wellfleet, ACC

1992 Gated Consortium Formed

1996 GNU Zebra

2002 Quagga, XORP

Quagga Routing Architecture

Modular Design

One process per protocol– bgpd, ospfd, ripd

One main controlling process– zebra

Extensible

Quagga Architecture Diagram

bgpdospfd ripd

zebra

Unix Kernel Routing Table

Quagga Routing Protocols

RIPv1, RIPv2, RIPng

OSPFv2, OSPFv3

BGP-4, BGP+

BGP route server and reflector

IPv6

Supported RFCs– 1058 RIPv1, 2453 RIPv2, 2080 RIPng– 2328 OSPFv2, 2740 OSPF for Ipv6– 1771 BGPv4, 1965, 1997, 2545 BGPv6, 2796 BGP

Route Reflection, 2858 Multiprotocol extensions, 2842 Capabilities Advertisement

Quagga Supported Platforms

GNU Linux– Debian, RedHat, SuSE, Slackware– Kernels 2.2.x - 2.4.x

FreeBSD– versions 4.x and 5.x

OpenBSD– version 3.x

NetBSD– version 1.4

Solaris– 2.6 and version 7

Hardware Requirements

CPU Intel 2.0 – 3.0 Ghz

Memory 512MB

Disks 18GB– RAID-1 (optional)– SCSI or IDE

Ethernet Interfaces– 2 x 10/100 Intel, 2 x 10/100/100 Broadcom

Redundancy– hot spare serves as backup to N production units

Scottish Economics

Router Prices– Cisco Mid-size

7204VxR, Catalyst 3550 $15k – $32k

– Extreme Alpine 3800 $31k - $38k

– Foundry BigIron 4000 $16k

– Intel 2.x Ghz server Dell 2650, IBM x335 $2.5k - $3.5k

Network Topology

Internal

External

Cogent A [100Mbps]

Cogent B [100Mbps]

C4 [1000Mbps]

Skye

Mull

Jura

Bute

McL

1. UofT A2. UofT B3. ResNet

Touchdown Network1000 Mbps

Traffic Shaper

Network Routing Policy

Three classes of traffic (based on src IP)– ResNet– UofT A– UofT B

ResNet– to (via TS) Skye to Cogent A– No C4 transit !!!

UofT A– to C4 if dst IP == C4 otherwise via Skye to Cog A

UofT B– to C4 if dst IP == C4 otherwise via Mull to CogB

Network Packet Filtering Policies

Drop all packets with– spoofed (non UofT) source IP addresses– non-routable destination addresses

0.0.0.0/8, 10.0.0.0/8, 127.0.0.0/8 169.254.0.0/16, 172.16.0.0/12, 192.168/16 etc.

– nasty tcp/udp M$ worm ports (Blaster, Welchia, etc.)

67, 68, 69, 135, 137, 139 161, 162 445, 593, 707, 1433, 1434, 3127, 4444

– non-assigned UofT subnets

Allow everything else

Network Traffic Shaping Policies

All traffic from a local Redhat ftp site to the outside world gets a 50 kbps pipe

Peer to peer traffic to and from UofT A&B gets a 256 kpbs full duplex pipe– KaZaa 1214– eDonkey 466[12]– BitTorrent 6881-6889

ResNet traffic gets conditioned by a dedicated Traffic Shaper (Packeteer)

Everything else flows freely

Routing Protocols and Configuration Jura

– runs OSPF on int. intf. with other UofT routers– runs BGP on external interface with C4 peer– contains all UofT and C4 specific routes

Mull– runs OSPF on int. intf. with other UofT routers– runs BGP on external interface with Cogent B peer– advertises UofTB routes– defaults points to Cogent B

Skye– same setup as Mull but with Cogent A– advertises UofTA and ResNet routes

Quagga Routing Configuration

Command line interface similar to Cisco IOSC4# conf tC4(config)# interface eth2C4(config-if)# description dummy interfaceC4(config-if)# ip address 10.1.2.3/24C4(config-if)# exitC4(config)# exitC4#

C4# conf tC4(config)# router bgp 328C4(config-router)# bgp router-id 10.1.1.10C4(config-router)# network 10.1.1.0/24C4(config-router)# redistribute staticC4(config-router)# neighbor 10.1.1.1 remote-as 999C4(config-router)# exitC4(config)# exitC4#

Quagga Operation

# show ip route

Codes: K - kernel route, C – connected, S – static, O -OSPF

B – BGP, > - selected route, * FIB route

S>* 0.0.0.0/0 [10/0] via 128.100.96.194, disc0

B>* 6.1.0.0/16 [20/0] via 205.211.94.97, yk0, 01w4d03h

B>* 6.2.0.0/22 [20/0] via 205.211.94.97, yk0, 01w4d03h

B>* 6.3.0.0/18 [20/0] via 205.211.94.97, yk0, 01w4d03h

# show bgp neighbors

BGP neighbor is 205.211.94.97, remote AS 549, local AS 239, external link

BGP version 4, remote router ID 205.211.94.253

BGP state = Established, up for 01w4d22h

FreeBSD ipfw Packet Filtering

Native packet filtering interface

Implemented as a multifunction user command

The packet passed to the firewall is compared against each of the rules in the firewall ruleset.

When a match is found, the action corresponding to the matching rule is performed and the search terminates.

General syntax– ipfw [rule number] action [log] body

ipfw examples

Drop all www traffic from a network– ipfw add deny tcp from 12.12.12.0/24 to www.ubc.ca 80

Drop all telnet traffic from a bad host– ipfw add deny tcp from bad.host.com to my.host.com 23

Throw away RFC 1918 networks– ipfw add deny all from 10.0.0.0/8 to any in via fxp0– ipfw add deny all from 172.16.0.0/12 to any in via fxp0– ipfw add deny all from 192.168.0.0/16 to any in via fxp0

Allow ssh– ipfw add allow tcp from any to any 22 in via fxp0 setup

keep-state

ipfw actions

allow | accept | pass | permit– Allow packets that match rule. The search ends.

deny | drop– Discard packets that match rule. The search ends.

fwd | forward ipaddr[,port]– Change the next-hop on matching pckts to ipaddr

pipe N– Pass packet to a dummynet(4) for bandwidth

limitation. [ conditionally end or continue ]

count– Update counters for all packets that match rule.

The search continues with the next rule

Traffic Control Concepts I

Set of mechanisms to condition net traffic

Examples– raise priority of some kinds of traffic– limit the rate at which traffic is sent– block undesirable traffic (same as packet filtering)

TC is done at the network interface– ingress (traffic entering an interface)

limited set of functions (classifying, dropping)

– egress (traffic leaving an interface) full range of functions available queueing

Traffic Control Concepts II

QueueingClassification Scheduling

Traffic Control Concepts III

Classification– looks at packet content and assigns each to one

or more classes.

Queueing– stuffs incoming packets into storage silos based

on class

Scheduling– transmitting packets in queues based upon

priority

Queueing and Scheduling are often combined into queuing disciplines

Traffic Control Concepts IV

Common Queueing Disciplines– simple drop tail (FIFO)

stores and emits packets in order which they arrive

– Random Early Detection (RED) starts dropping packets already before reaching

maximum queue size

– Token Bucket Filter (TBF) shapers that emits packets at a fixed rate

– Priority Scheduler (PQ) emits packets in higher priority classes before

packets in lower priority classes

– Weighted Fair Queueing (WFQ) assigns an independent queue for each flow a weight can be defined for each queue

FreeBSD Dummynet Features

Integrated with ipfw to classify packets

Can be used equally well on egress/ingress

Abstractions/features– pipes

fixed bandwidth channels variable queue size, delays, random packet loss

– queues queues of packets weighted share bandwidth of pipe they are associated with

proportionally to their weight

WF2Q+ used for queuing discipline

Dummynet Examples

Limit WWW traffic to 100Mbps ipfw pipe 1 config bw 100Mbit/s ipfw add pipe 1 ip from any to any dst-port 80

Prefer ssh to telnet traffic ipfw pipe 2 config bw 256kbit/s ipfw queue 1 config pipe 2 weight 7 ipfw queue 2 config pipe 2 weight 3 ipfw add queue 1 ip from any to any dst-port 22 ipfw add queue 2 ip from any to any dst-port 23

Rate limit each network host's upload rate ipfw pipe 3 config mask src-ip 0x000000ff bw

16kbit/s queue 8Kbytes ipfw add pipe 3 ip from 12.18.123.0/24 to any out

via xl0

Routing Policy Using ipfw

All ResNet traffic forwarded directly to Skye– ipfw add fwd $skye from $resnet to any in recv $uoft_if

Block spoofed packets– ipfw add allow all from $uoftnet to any in recv $uoft_if– ipfw add deny in recv $uoft_if

Block bad packets (M$ worms etc.)– for i in 67-69 135-139 161 162 445 593 707 4444

do– ipfw add deny udp from any to any $i– ipfw add deny tcp from any to any $i

done

C4 traffic follows specific routes from BGP

Routing Policy Using ipfw Cont.

Block all traffic to non-defined UofT addrs– ipfw add deny all from any to $uoftnet out xmit $def_if

Partition UofT A/B traffic to Skye/Mull– add fwd $skye all from $uoftA to any out xmit $def_if– add fwd $mull all from $uoftB to any out xmit $def_if

Traffic Shaping– limit RH ftp server

ipfw pipe 1 config bw 50Kbit/s ipfw add pipe 1 ip from $rhftp to any in recv $uoftif

– limit peer to peer ipfw pipe 2 config bw 256 Kbit/s ipfw add pipe 2 ip from $uoftA to any dst_port

1214,4661,4662

Linux Packet Filtering: iptables

Similar to ipfw in functionality and use

User based command line interface

Syntax– iptables rule-action table name conditions action

Very rich set of conditions and actions

Extensible modular actions

More complicated in concept than ipfw or pf

hierarchy: tables -> chains -> rules

three default tables with default policies– filter, nat, mangle

Linux iptables Anatomy Ingress

PREROUTING

QOS Ingress

INPUT ROUTING and RPDB

ContrackmangleIMQnat

Network Interface

FORWARDINPUT

manglefilter

LOCAL PROCESSES

manglefilter

REMOTE IP ADDR

Linux iptables Anatomy Egress

POSTROUTING

QOS Egress

Network Interface

nat

IMQ

LOCAL PROCESSES

REMOTE IP ADDR

mangle

contrackmanglenatfilter

OUTPUT ROUTING

OUTPUT

iptables examples

Drop all www traffic from a network iptables -A FORWARD -p tcp –dport 80 -s 12.12.12.0/24 -d

www.ubc.ca -j DROP

Drop all telnet traffic from a bad host iptables -A INPUT -p tcp -s bad.host.com -d my.host.com –-

dport 23 -j DROP

Throw away RFC 1918 networks from inside iptables -A FORWARD -s 10.0.0.0/8 -i eth0 -j DROP iptables -A INPUT -s 10.0.0.0/8 -i eth0 -j DROP iptables -t mangle -A PREROUTING -s 172.16.0.0/12 -i eth0 -j

DROP

Allow ssh and keep state iptables -A FORWARD -p tcp –dport 22 -i fxp0 -m state -–

state NEW,ESTABLISHED -j ACCEPT

Linux Routing – Multiple Tables

Multiple routing/forwarding tables

Three fixed prefined tables– local– main– default

Each table is assigned a priority number– 0 local– 32766 main– 32767 default

match is sought starting with highest priority tables (local -> main -> default)

Linux Routing Policy Database

Traditional Routing Routing Policy Database(RPDB)

Destination IP Address Destination IP AddressType of Service Source IP Address

Type of ServiceIptables FW mark

Linux Traffic Control: tc

Uses queueing disciplines for managing bandwidth

Largely concerned with data being sent rather than received.

Classless queueing disciplines– reschedule, drop or delay– applied to the bulk interface– pfifo_fast

default, can't be changed

– TBF (Token Bucket Filter) passes traffic up to a fixed rate drops the rest allows short burst in excess of fixed rate

tc: Classless qdiscs

SFQ (Stocastic Fair Queueing)– Traffic split into large number of FIFO queues, one per

flow– Traffic gets sent/serviced in a round robin fashion, giving

each flow a chance to sent its data.– Leads to fair behaviour– prevents one flow from hogging all the bandwidth– only really useful when the link is full

RED (Random Early Detection)– drops packets statistically before queues are full– leads to a congested link to slow more gracefully– helps TCP applications find their fair speed faster

tc: Classful qdiscs

Used when different types of traffic need different treatment.

CBQ (Class Based Queueing)– very complicated to set up and tune

PRIO– classify and traffic into a number of bands each

with its own priority.

u32– used as the tool to classify the traffic into sub

queues– based on actual offset of information in the IP

header

Linux: tcng

tc syntax is very complicated both in setting up the qdisc's and classification

tc qdisc add dev eth0 root handle 1:0 prio tc qdisc add dev eth0 parent 1:0 protocol ip u32 match

ip protocol 6 ff match tcp dst 50 ffff classid 1:1 tc qdisc add dev eth0 parent 1:3 handle 30: sfq tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32

match ip sport 80 0xffff flowid 1:3

tcng was created as a higher level tool– simple to configure– more natural language to set up classes and qdisc– compiles to tc or “C” – comes with a simulator

tcng: Example Input

dev “eth0” {

egress {

class (<$high>) if tcp_port == 80;

class (<$low>) if 1;

prio {

$high = class {

tbf(limit 10kB, rate 20kbps,

burst 2kB, mtu 1500B);

$low = class {

fifo(limit 30kB)

}

}

}

}

tcng: Example Output

tc qdisc add dev eth0 handle 1:0 root dsmark indices 4 default_index 0

tc qdisc add dev eth0 handle 2:0 parent 1:0 prio

tc qdisc add dev eth0 handle 3:0 parent 2:1 tbf burst 2048 limit 10240 mtu

1500 rate 2500bps

tc qdisc add dev eth0 handle 4:0 parent 2:2 bfifo limit 30720

tc filter add dev eth0 parent 2:0 protocol all prio 1 tcindex mask 0x3

shift 0

tc filter add dev eth0 parent 2:0 protocol all prio 1 handle 2 tcindex

classid 2:2

tc filter add dev eth0 parent 2:0 protocol all prio 1 handle 1 tcindex

classid 2:1

tc filter add dev eth0 parent 1:0 protocol all prio 1 handle 1:0:0 u32

divisor 1

tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u8 0x6 0xff

at 9 offset at 0 mask 0f00 shift 6 eat link 1:0:0

tc filter add dev eth0 parent 1:0 protocol all prio 1 handle 1:0:1 u32 ht

1:0:0 match u16 0x50 0xffff at 2 classid 1:1

tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u32 0x0 0x0

at 0 classid 1:2

Routing Policy Using Linux

Routing Tables– 0: from all lookup local– 100: from 142.151.0.0/16 lookup resnet– 1000: from all lookup main– 2000: from 142.150.0.0/16 lookup uoftA– 32767: from all lookup default

resnet contains a single default to syke

uoftA contains a default to skye

default contains a default to mull

main contains all the C4 routes

Linux Traffic Shaping Policy

dev eth1 {

egress {

class ( <$rhftp> ) if ip_src == 128.100.17.10;

class ( <$p2p> ) if ( (tcp_dport == 1214 ||

tcp_dport == 4661 || tcp_dport == 4662) &&

ip_src:16 == 128.100.0.0 );

class ( <$high> ) if 1 ;

htb () {

class ( rate 100Mbps , ceil 100Mbps ) {

$rhftp = class ( rate 50kbps, ceil 75kbps );

$p2p = class ( rate 256kbps, ceil 325kbps );

$high = class ( rate 90Mbps, ceil 100Mbps );

}

}

}

}

OpenBSD packet filtering

pf runs as the native packet filtering engine

similar in syntax to ipfw

traffic shaping (ALTQ) integrated with pf

BSD only supports one main routing table

pf (like ipfw) supports a forwarding action to explicitly forward a packet

URLs– www.openbsd.org– www.csl.sony.co.jp/person/kjc/software.html– www.benzedrene.cx/pf.html

Results and Conclusions

OSS Routers in service for > 18 months

Scaled easily from 1 to 3 machines

Currently running– FreeBSD 4.x, 5.x, dummynet, ipfw

Will be moving to Linux in next 3 months

Standard network monitoring via SNMP

CPU running < 40%

OSS is a viable option for policy based routing and shaping at the edge