4/23/2003 1

Immunix & Defcon:Defending Vulnerable Code From Intense Attack

Crispin Cowan, Ph.DSeth Arnold, Steve Beattie, Chris Wright

WireXand

John Viega, Secure Software

4/23/2003 2

Talk Outline

• About WireX and Immunix– Secure Linux Systems

• The Defcon Challenge– Defend vulnerable code against massive attack

• Technology Transfer– Commercial products built on this technology

4/23/2003 3

Software Security

Software security is really simple:

• Make sure you only run perfect software

Uh-oh :-)Intrusion Prevention:

– Systems that detect attack attempts in real time, and reject them

• When bugs occur, they are not exploitable– Attacker cannot exploit the bug to gain unintended

privileges

4/23/2003 4

Immunix Security Technologies

• Shipping:StackGuard: stops buffer overflows

FormatGuard: stops printf format bugs

RaceGuard: stops temp file races

SubDomain: contains vulnerable programs

4/23/2003 5

StackGuard

• Stack Smashing Problem:– Weak bounds checking

on inputs in C programs– Attacker overflows input

buffer, corrupting adjacent state to gain control of the program

– Most common target: function return address on the stack

• StackGuard:– C compiler

enhancement– Ornaments call stack to

detect corruption– Very low performance

overhead– WireX has been

shipping fully StackGuard’d system since 1999

4/23/2003 6

FormatGuard

• Format String Problem:• Sudden discovery in June 2000

– Vulnerability in WU-FTPD

– Followed by hundreds of similar vulnerabilities

• Basis: arcane %n printf format string directive– Treat corresponding argument

as an int *

– Write back number of items formatted so far

• Problem: programs that pass un-filtered user input strings direct to printf

• FormatGuard:– Similar to StackGuard– compiled defense

against printf format string vulnerabilities

– CPP macro • Counts arguments at the

call site at compile time• Compares that number

to the format string presented at run time

4/23/2003 7

RaceGuard

• Temporary File Race Problem:– Portable procedure for

temporary file creation is non-atomic

– If attacker gets in the middle, can redirect temporary file creation by privileged programs to corrupt the system

• RaceGuard: – Kernel enhancement to

detect race attacks mid-way through

– Abstract method: detect changes between stat() and open() accesses to the same file name

4/23/2003 8

Containment

• If your software is vulnerable anyway, you need to contain it so that it runs with the least privilege necessary to perform designated functionChroot: basic isolation for vulnerable programs

Immunix SubDomain: flexible confinement for vulnerable programs

4/23/2003 9

Containment WithChroot• “Change root”: makes some

subdirectory appear to be the root (“/”) directory for the calling process and its children– Available as both a shell

command and a system call

• Effect: chroot’d programs cannot affect anything outside the chroot “jail”– Limits impact of bugs in

program, e.g. chroot BIND

Benefits:Standard: Comes with most UNIX’s

Compatible: several current programs have been modified to work within a chroot jail

Fast: no performance degradation

Limitations:Work: must move copies of everything

a jailed program needs into the jail

Isolation: jailed program cannot interact at all with the rest of the system

4/23/2003 10

Containment WithImmunix SubDomain• Part of Immunix Kernel

Extension:– Specify the list of files that a

SubDomained program may access

• Effect: SubDomained programs cannot affect anything they don’t explicitly need access to– Limits impact of bugs in

program, e.g. SubDomain CGI scripts

Benefits:Flexible: SubDomained programs

can have controlled interaction with the rest of the system

Compatible: SubDomain can confine binary programs without modifications

Fast: 1% or less performance overhead

Limitations:Work: must specify “shape” of

SubDomain

4/23/2003 11

Containing PHF

• PHF: infamous vulnerable CGI script– legitimate function: database

lookup of user information

– sloppy parsing of CGI input

– can get PHF to start an xterm on an arbitrary display

• To SubDomain PHF:– Specify all the files that PHF

needs

• Effect:– access to all other files is

denied

– Including xterm :-)

Place this file in /etc/subdomain.conf/phf

/home/httpd/cgi-bin/phf {

/bin/sh x ,

/etc/ld.so.cache r ,

/etc/nsswitch.conf r ,

/lib/ld-linux.so.2 r ,

/lib/libc.so.6 r ,

/lib/libtermcap.so.2 r ,

/usr/local/bin/ph ix ,

}

4/23/2003 12

WireX Systems:Immunix Secure OS

• Linux system similar to Red Hat Linux• RPM based• All source-available programs compiled with

StackGuard and FormatGuard– PointGuard in future releases

• Kernel equipped with SubDomain and RaceGuard

• All network-accessible daemons SubDomain-profiled

4/23/2003 13

Experimentation ...

• Some real-world red teaming• Play an Immunix server in the Defcon

Capture the Flag (CtF) games• Almost no holds barred:

– No flooding– No physical attacks

• New gaming rig designed by the Ghettohackers

4/23/2003 14

Basic Defcon CtF Rules

Player Nodes

4/23/2003 15

Basic Defcon CtF Rules

Player Nodes

Score’botPolls player nodes,Looking for req. services

If all services found ...

4/23/2003 16

Basic Defcon CtF Rules

Player Nodes

Score’botPolls player nodes,Looking for req. services

If all services found,Score one point for theFlag currently on thatnode

4/23/2003 17

Basic Defcon CtF Rules

Player Nodes

Score’botPolls player nodes,Looking for req. services

If all services found,Score one point for theFlag currently on thatnode

… while each teamtries to replace others’ flags

4/23/2003 18

No Flooding

• DoS attacks are not interesting• Explicit rule against flooding attacks

– Game masters will make you stop if you are caught at it

– Goal: ensure that all teams are actually able to play

• Penalties:– Kicked out for overt DoS attacks– Pay for bandwidth with a point penalty

4/23/2003 19

Area View

4/23/2003 20

Sporting Event

Teams named funky colors

Score obfuscated

• There was an official bookie :-)

• Score broadcast on hotel cable

Immunix was white,hence “Weiss Labs”

4/23/2003 21

The Catch

• The required services are secret

• Only a few clues:– They supply us with a VMWare/Linux image

reference distribution that provides all required services

• It is also riddled with vulnerabilities

– The score’bot polls for the required services• But the score’bot stops its poll if it finds something it

doesn’t like

4/23/2003 22

The Reference Distribution

• Red Hat 6.2, unpatched

• nmap: shows nearly everything open– finger, POP, IMAP, SMTP, SNMP, Webmin ...

• Apache running as root

• CGI’s for adduser and deleteuser– Anonymous can create a user login on your

node– As any user number, including zero

4/23/2003 23

Example Services the Score’bot Wanted• Create a user• Send that user mail• Finger the user• POP in to fetch the mail• Delete the user• Note: no crypto protocols

– No proper authentication of the score’bot– Must heuristically distinguish score’bot from attacks

using behavior signatures

4/23/2003 24

Interesting Challenge

• Not just survive severe attack, but also– Protect bad code– A lot of it– Vague functional specification– Rapid deployment

• Great new game infrastructure from Ghettohackers– Interesting challenge– Engaging scoreboard

4/23/2003 25

Captain’s Meeting

• Explain the rules in detail

• Hand us the reference distribution

4/23/2003 26

Setting Up

4/23/2003 27

The Popular Strategy: Human Intrusion Detection

• Launch the reference Linux distribution

• Ad hoc patch as stuff happens

• Defend:– look for logins, I.e. non-score’bot behavior– kill them off ASAP– very labor-intensive

4/23/2003 28

The Immunix Strategy: Protect Bad Code with Immunix Tools

• Port all plausible services to Immunix 7+ distribution– Use our own fingerd, httpd, etc., up-to-date and

compiled with StackGuard and FormatGuard– Run on an Immunix kernel with SubDomain and

RaceGuard– Wrap vulnerable services & CGI’s with SubDomain

profiles to limit access to least privilege necessary

• Launch only when we were reasonably confident that the Immunix machine was configured securely

4/23/2003 29

Dealing with Logins: the SubDomain Shim

• Change adduser CGI to use a special default shell: /bin/fubush– /bin/fubush is just a hard link to /bin/bash– Restrict /bin/fubush to only the operations

needed by the score’bot

• Attackers can go ahead and create a login with uid 0 and it still won’t do them any good – They get a root shell, stuck in a tiny sandbox

4/23/2003 30

Immunix Team

4/23/2003 31

Immunix Team

Me

Chris Wright

SethArnold

Steve Beattie

• Plus 15 volunteers

4/23/2003 32

From Our Corner

4/23/2003 33

From Our Corner

JohnViega

Me

ChrisWright

SethArnold

SteveBeattie

4/23/2003 34

Mental Stress

• This is a tough game to play– Head-to-head competition with a lot of very

smart people– Real-time, continuous

• The intensity of qualifying exams– That go on for 22 hours in a 48 hour period– … set in the middle of a rave

• Hydrate or die :-)

4/23/2003 35

Rave

• Loud music• Smoking• Gawkers• Social

engineering• Periodic

“news breaks”

4/23/2003 36

Our Strategic Error

What We Did• For first 4 hours

– No server at all

– Porting services to Immunix ASAP, based largely on nmap and source inspection

• Next 4 hours– Launch Immunix server

– It’s secure, but is not making the score’bot happy

• Cost us massive points– Too focused on the science of

“can we defend Immunix?” and not enough on the game rules

What We Should Have Done• Launch reference system

immediately– Defend ad hoc like everyone

else

– Run network sniffer to determine what the score’bot wants

• Would have:– Put us over the top on points

– Learned what score’bot wants much faster

• We eventually did this

4/23/2003 37

Immunix Server Not Up Yet

6th

place

4/23/2003 38

Once Immunix Server Up …in the Score’bot’s Opinion :)

• Our score quickly rose

2nd

place

4/23/2003 39

Once Immunix Server Up …in the Score’bot’s Opinion :)

Close2nd

place

4/23/2003 40

Once Immunix Server Up …in the Score’bot’s Opinion :)

1st

place• Stayed

there most of Saturday

4/23/2003 41

Late Saturday:New Service Requirement

• With 4 hours of play to go, the score’bot changed: now it wanted Webmin– Open source web-GUI for Linux administration– Competitor to WireX’s commercial server

appliance software– Rather famously vulnerable :)

• Took us 2 hours Sunday morning to make the score’bot happy again– Lost our lead

4/23/2003 42

Some of Our Creative Attacks

Lock Out the Owner• Once we root the

machine, install a back door

• Also replace root’s login shell with /sbin/halt– Owner can’t log in to

their own machine– But we can

Spam’bot• Add user to their server• User sends spam mail to

all the other teams• Costs them penalty

points• Penalties are per

connection– Spam’bot sends 1-byte e-

mails

4/23/2003 43

Final Score: 2nd Place

Team Score Points PenaltiesOrange 54.3764 64 9.6236White(Immunix)

51.1160 55 3.8840

Brown 48.2203 90 42.7797Green 40.1943 46 5.8057Yellow(Chaos Computer Club)

22.1865 43 20.8135

Red(Naval Postgraduate School)

6.1215 17 10.8785

Blue -22.7039 30 52.7039

Purple -24.5107 5 29.5107

4/23/2003 44

Lesson: Symmetric Red Teaming Solves Rules Issues

• Everyone is both an attacker and defender

• Bad: everyone needs to learn how to attack

• Good:– Everyone should learn how attacks are done :-)– Rule fussing about how hard or easy it is for the

attacker apply to all parties -> less fussing

• Ghettohackers have designed a great game– Looking for technology transfer to Government

4/23/2003 45

Lesson: Mandatory Access Control is Not Enough • telnetd was a required service• WireX never bothered to patch a vulnerability in telnetd for Immunix– Only idiots run telnetd :-)

• Someone hacked our telnetd– Didn’t get out of the SubDomain sandbox

– Did make our telnetd stop working

– Cost us a point that round

• General case: MAC protects your system, but not your individual services

4/23/2003 46

Lesson: Resource Management is a Security Attribute

• SubDomain confined attacker logins to only run prescribed code– Including PERL

• Attacker launched a PERL fork bomb– Consumed all of real and virtual memory– While our machine is thrashing, the score’bot

passes us by– Costs us a point that round

4/23/2003 47

Lesson: Redundancy Helps When You Are Vulnerable

• Penetration attacks take a long time to recover– Must clean up state, find & fix vulnerability

• DoS attacks take a long time to recover– If machine crashes, must fsck file system; can

take 10 minutes

• Hot spare can be on-line in seconds– Heterogeneous hot spare keeps attacker from

immediately deploying the same attack

4/23/2003 48

Lesson: Redundancy is Resource-Constrained• Must have humans on watch to clean up the

compromised machine– The hot spare will not protect you for long

• Presumption that hot spare prevents repeat attacks assumes resource limit at the attacker’s end– If attacker has lots of exploits/resources, they will hack

your heterogeneous server just as quickly

• We had a hot spare, but not enough of them

4/23/2003 49

Lesson: Immunix was Impenetrable, but not Incorruptible

• No one ever “flagged” the Immunix server– Others did plant enemy flags on our reference

server (as expected)

• But they did hit the Immunix server hard enough to compromise availability– Take out one required service, and the

score’bot doesn’t award a point– We missed first place by less than 4 points out

of 55

4/23/2003 50

Immunix SecureServer Appliances

• Base: Immunix Secure OS– Appliances that defend themselves

• Turn-key installation:– WireX installer turns PC hardware into server

appliance in 5 minutes– No technical skills required

• Graphical ease of use– WireX Secure Server Manager Web-GUI

management system

4/23/2003 51

4/23/2003 52

Immunix SecureServer Solutions• Partner with HP• First product: Trend Micro Antivirus Mail scanner

http://h18000.www1.hp.com/products/servers/solutions/iis

• Websense content filter• Secure Webmail Appliance

– Customized for USTRANSCOM– Provides secure webmail access to Microsoft Exchange– Supports Microsoft Exchange/Outlook calendar events

– Partner with Secure Computing • All being demonstrated in our booth

4/23/2003 53

Summary

• WireX Immunix:– Secure Linux system with low overhead and

administrative hassle

• Defcon fun:– Couldn’t crack Immunix– Integrated undocumented code into Immunix in 12

hours

• Available as– Secure Linux system– Turn-key secure server solutions