Building Defensive Architectures Using Backdoors

Liviu IftodeDepartment Of Computer Science

Rutgers University(iftode@rutgers.edu)

2

3

Rebooting Not Always Acceptable

Service.com

4

Traditional (Eager) Recovery Approaches

• Checkpointing (process, virtual machine)

• Hot machine backups (state machine replication, primary-backup)

• Limitation: time-consuming and costly– Intrusive during failure-free execution– Require dedicated machines – Require stable storage available after the failure

5

Lazy State Recovery?

• OS and application state still in system memory

• Can we recover or repair the state lazily (after the failure occurred)?

Operating System

Mem

NIC

CPU

Disk TTY …

6

How to Access the State?

CPU and OS resources not available

7

The World Today

InternetInternet

Failure

Attacks

10:00pm EST 3:00am GMT

• Computer maintenance requires human intervention: slow and expensive

• Emails and phone calls do not scale

• Worse at planetary scale: different time zones and language barriers

• Computer maintenance requires human intervention: slow and expensive

• Emails and phone calls do not scale

• Worse at planetary scale: different time zones and language barriers

8:30am IST

8

Our Vision : Defensive Architectures

InternetInternet

Failure

Attacks

10:00pm EST 3:00am GMT

• Computer Systems perform self-defensive management cooperatively

• Access to memory possible even in presence of failures and attacks

• Operating System not involved

• Computer Systems perform self-defensive management cooperatively

• Access to memory possible even in presence of failures and attacks

• Operating System not involved

8:30 AM IST

9

Outline

• Motivation

• Backdoor Architecture– Components– Remote Repair using Backdoors– Lazy State Recovery using Backdoors

• Defensive Architectures using Backdoors

• Future Work and Conclusions

10

Backdoor

Backdoor: a hidden software or hardware mechanism, usually created for testing and troubleshooting

--American National Standard for Telecommunications

11

The Backdoor Idea

Mem NICCPU

Backdoor

Management Infrastructure

“Front door”

• Backdoor provides an alternative access to system resources– A programmable I/O device for physical platforms– A virtual machine for virtual platforms over a virtual machine monitor (VMM)

• Backdoors can be connected over a private network, specialized interconnect, or even a cellular link

• Backdoor provides an alternative access to system resources– A programmable I/O device for physical platforms– A virtual machine for virtual platforms over a virtual machine monitor (VMM)

• Backdoors can be connected over a private network, specialized interconnect, or even a cellular link

12

Mem

I/O

CPU

BD

Target System

BD

Monitor System

Mem

I/O

CPU

Backdoor Architecture

• Hardware– Programmable device with

processor and memory• Software

– OS extensions for remote healing– Firmware extensions for BD

programming

Design Principles

• Availability– Failure of target OS must not

impair BD• Nonintrusiveness

– BD operations must not involve processors of the target system

• Access control– monitor and target systems

negotiate access permissions • Integrity

– target system cannot modify the result of a BD operation

• Responsiveness– target system cannot block BD

operation

13

Backdoor Implementation

NIC CPU

CPU

Memory

BD

CPU

Memory

BD

MonitorTargetMONITOR

(Remote-R)

RECOVER/REPAIR(Remote-R/W)

14

Nonintrusive Remote Healing

• Three components – Detection, Diagnosis, and Action

• Performed nonintrusively from a remote machine– Zero-cycle monitoring and failure detection for target

system– Remote extraction of useful state from a “hung”

system for diagnosis and recovery– In-place repair of OS state of a “damaged” system

15

• Monitoring and Failure Detection– Sensor Box: system health indicators (sensors) provided by the

target OS in its memory– Sensor: <UniqueID, Update Deadline , Value>

• Repair of damaged OS State– Externalized State: OS state data that the BD can read – Remote Access Hooks: OS control data that the BD can write to

perform repairing actions

• Recovery of light-weight state– Continuation Box: fine-grain OS and application state that the BD

can transfer between systems to migrate running applications

Backdoor Software Architecture

16

Outline

• Motivation

• Backdoor Architecture– Components– Remote Repair using Backdoors– Lazy State Recovery using Backdoors

• Defensive Architectures using Backdoors

• Future Work and Conclusions

17

Failure Model

• Computer system freeze– OS bug: hang, crash, deadlock, etc.– OS damage: resource exhaustion, DoS attack – hardware: peripheral device stops responding

• Fail-stop (no erratic behavior)– memory not wiped out during failure

18

• Target OS updates progress sensors in Sensor Box • Monitor BD reads Sensor Box periodically, checks

counters– Failure = counter stalled beyond its deadline

Target OS Monitor

Sensor Box

Backdoor

<Timer interrupts><Context switches><NIC interrupts>

…

Monitoring and Failure Detection

19

Diagnosis and Repair

• Diagnosis – Inspect live OS data structures in target’s memory

through( the externalized state)– Identify damaged OS state e.g. resource exhaustion

due to memory hogging processes

• Repair – Modify target OS memory (remote access hooks) to

correct damaged state (e.g. remove memory hogging processes by “injecting” a kill signal in its process control block)

20

Backdoor Prototype• Myrinet LanaiX NIC • Modified firmware and low level GM library• Modified FreeBSD 4.8 kernel

– Sensor Box– Externalized State and Remote Memory Hooks

• Two resource exhaustion case studies – Memory exhaustion– CPU starvation

• Experimental setup– Dell Poweredge 2600 servers with 2.4 GHz dual Intel Xeon, 1GB

RAM, 2GB swap, Myrinet Lanai X NIC– Benchmark: a simple CPU-bound application

21

Effectiveness of Remote Repair

0

5

10

15

20

0 2 4 6 8 10 12 14 16

Number of memory hog processes

Exe

cutio

n tim

e (s

)

Impaired system

With remote repair

22

Repair Timeline

0 0.5 1 1.5 2 2.5 3

Time (s)

Memory pressure

Detection

Diagnosis & Repair

End of repair

Local cleanup of damaged stateRemote Repair

23

Outline

• Motivation

• Backdoor Architecture– Components– Remote Repair using Backdoors– Lazy State Recovery using Backdoors

• Defensive Architectures using Backdoors

• Future Work and Conclusions

24

Internet Services and Servers

Server 1

C1

C2

servers

Internet

25

Internet Services and Servers

Server 1

Server 2

C1

C2 se

rvice

servers

Internet

26

Service Continuation (SC)

SC2 = {“96.3FM”, “2nd song”}

GET “96.3FM”

Streaming Server 1

Streaming Server 2

C1

C2

27

Service Continuation (SC)

SC2 = {“96.3FM”, “2nd song”}

Streaming Server 1

Streaming Server 2

C1

C2

28

Service Continuation (SC)

SC2 = {“96.3FM”, “2nd song”}

Streaming Server 1

Streaming Server 2

C1

C2

29

Continuation Box (CB)

• Idea– extract “essential” state– pass it to similar application on a healthy machine

• CB encapsulates fine-grained server state associated with a client session– OS data, e.g., data in transit through IPC channels– Application data

• Application may need to cooperate with the OS!

30

Lazy Extraction of Continuation Box

Memory

BD

CPU

BD

Victim machine(crashed)

Recovery machine(healthy)

Memory

Continuation Box

Recovered State

OS

31

Recovery: What and How?

Time

Victim machine Recovery machine

2recv

recv

recv

Backdoor

3

CB = ???

1

32

Solution: Continuation Box API

Time

2

3 export()

CBimport()

recv

recv

recv

recv

Backdoor

3

Victim machine Recovery machine

log

1

33

Service Continuation Structure

Client 1

SC2

SC1

TCP/IP IPC

Front-end serverprocess

Back-end server process

Client 2

SC_APPSC_IO

34

Service Continuation API

• export SC_APP

• import SC_APP

• create_sc for a client session

• associate I/O channel with the SC

• open_sc from an I/O channel

35

A Server with Service Continuation

while (cid = accept()) { scid = create_sc(cid) if (import(scid, &{file_name, offset}) == NULL) { receive(cid, file_name) offset = 0 } fd=open(file_name) seek(fd, offset) while (read(fd, block, size) != EOF) { send(cid, block, size) offset += size export(scid, {file_name, offset}) }}

36

Case Study: Multi-tier Internet Service

Back-End

MySQL DB server

Front-End (FE)

Apache web server

Middle Tier (MT)

JBoss app. server

37

Recoverable Service

38

• Experimental setup– Dell PowerEdge 2600 servers, 2.4 GHz dual

Intel Xeon, 1GB RAM, 1Gb Ethernet– Workload modeled after TPC-W

• Fault injection in FE and MT nodes– synthetic freeze, emulated freeze by remote

OS locking, bugs inserted in network drivers

• Evaluation– Low cost, low overhead under load– Fast recovery

Recoverable Service

39

Low Cost

• Monitoring– < 1% CPU @ 100 ms sampling period (100

sensors)

• Continuation Box– API export/import < 30 us– Extraction 358 us for 10 KB CB

40

Low Overhead under Load

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

20 100 300 500 700 900 1,100

Clients

Req

ues

ts/m

in

Base

Recoverable FE

Recoverable FE+MT

41

Recovery Timeline

0 5 10 15 20 25 30

Time (ms)

Failure

Detection

Import CB

Recovery done

Recovery latencyDetection Latency

42

Outline

• Motivation

• Backdoor Architecture– Components– Remote repair using Backdoors– Lazy State Recovery using Backdoors

• Defensive Architectures using Backdoors

• Future work and Conclusions

43

Defensive Architectures Using Backdoors

• Autonomous Backdoors– BDs are programmed to execute defensive activities during

bootstrap, then “sealed”– Tamper-resistant during normal execution: OS cannot alter or stop

BD execution – BDs communicate among themselves to execute certain defensive

activities cooperatively• Hierarchical Defensive Architectures

– Defensive Computer Architecture (DCA) : Single computer system equipped with BD

– Defensive Network Architecture (DNA) : Cluster nodes equipped with BD connected over high-speed private network

– Defensive Inter-Network Architecture (DINA) : Loosely coupled DNA clusters over a Wide Area Network

44

Applications of Defensive Architectures

• Smart Watchdog for DCA– Continuously monitor the system memory– Identify and enforce OS invariants in the host memory or the I/O

system– Search for virus/worm signatures

• Continuous Remote Logging and Integrity Verification over DNA– Continuously retrieve logged data from system memory – Send it to another node in the DNA– Cooperative OS integrity verification

• Defensive News Agency over DINA– A global secure information network– Critical system controllers (routers, GRID control nodes, PlanetLab

peers, etc.) subscribe to it– Multiple DNAs publish information to the system– System propagates information of interest about the Internet,

individual networks, or hosts

45

Future Work

• Virtual Backdoor– Realization of the BD for virtual environments e.g. VMMs and PlanetLab– Enables planetary-scale system monitoring and management

• Orion : Holistic Approach to System Monitoring– Continuous non-intrusive physical memory inspection over BD– Identify memory modification patterns and correlate them to predict

unstable system states• BD Language

– BD can execute basic building blocks of defensive actions – Express complex defensive actions using the basic building blocks

• Security– Prevent malicious users from using BDs to perform remote attacks– Authenticate and verify actions before performing them

• BD over the Phone– Use cellular link to access the BD for system management operations

46

People Behind Backdoors

• Florin Sultan• Aniruddha Bohra • Pascal Gallard (INRIA/IRISA, France)• Iulian Neamtiu (University of Maryland)• Stephen Smaldone• Yufei Pan• Arati Baliga• Tzvika Chumash