Framework for Design Validation of Security ArchitecturesJeffrey Dwoskin Mahadevan Gomathisankaran Ruby Lee

Abstract

New security architectures are difficult to prototype andtest. They require interactions between hardware, op-erating systems, and applications, making them hardto simulate and monitor. We have designed and proto-typed a testing framework using a virtualization plat-form which emulates the behavior of new hardware se-curity architecture in the virtual CPU, and performsa wide range of hardware and software attacks on thesystem under test. Our testing framework significantlyspeeds up development of the testing environment andinfrastructure, and provides APIs for launching attacksand monitoring the effects of an attack on the hardwareand software layers, which is especially convenient dur-ing the design and validation phases for new hardware-software architectural solutions. We have used our test-ing framework to test the trust chain of the SP archi-tecture [1] as an example.

1 Introduction

Designers of security architectures face the challengeof testing new designs to validate the required secu-rity properties. To provide strong guarantees of pro-tection, it is often necessary and desirable to put low-level security mechanisms into the hardware which thesoftware layers can rely upon for a wide-range of ap-plications. The resulting architecture is a combinationof hardware and software components which are diffi-cult to test together. Testing must be done during thedesign time to give confidence in the architecture be-fore the complete system is built, at which point it iscostly to make fundamental changes in response to se-curity flaws. To address this problem of design-timesecurity validation, we propose a new testing frame-work for hardware-software security architectures. Ourframework provides a controlled environment to emu-late the behavior of new hardware components with afull software stack running in a virtualization environ-ment, with which coordinated security attacks can beperformed and observed.In a new architecture, security mechanisms may beadded to the processor at the lowest software-visiblelayers, yet the effects on system operation reach up intothe operating system, middleware, and application soft-

ware, and can even span networked systems. Each ofthese layers will use abstractions of the lower layers andmake assumptions about their security properties. Val-idation requires that each layer be modeled and simu-lated together to study the interactions of the compo-nents and their effects on the overall operation of thesystem. This is unlike traditional computer architecturewhere performance and power optimizations can usuallybe designed and tested with layer-specific measurementsthat focus on the new components. Instead, our frame-work must emulate all hardware and software layers si-multaneously, while coordinating simulated attacks byobserving and controlling activity at each system layer.Furthermore, the testing environment — including thehardware implementation, software stack, threat mod-els, and attack mechanisms — must be as realistic aspossible so that results are meaningful.

Our Approach

Our testing framework is composed of two components:a system under test (SUT) containing the new hard-ware architecture and software stack under considera-tion, and a testing system (TS) which coordinates mon-itoring of, and attacks on, the SUT. We build on topof existing virtualization technology, which allows us torun a full set of commodity software efficiently. In avirtualized system, a Virtual Machine Monitor (VMM)creates multiple virtual machines that run on a singlephysical host machine [2, 3, 4]. Typically the virtualmachines are nearly identical to the host machine, how-ever, we modify the VMM to emulate the new securityhardware features of the SUT. The SUT and the TS areseparate virtual machines, so that the environment ofthe SUT is as realistic as possible — it runs a commod-ity operating system (Linux), commodity user applica-tions, and any new application software using the newsecurity mechanisms.

The SUT is attacked by the TS according to the threatmodel being tested; the TS itself is not attacked. TheTS monitors hardware and software events that occur inthe SUT using hooks provided by our framework. It alsocan inject attacks at all layers of the system. An attackscript running in the TS virtual machine coordinatesevents and attacks from both hardware and softwarecomponents in the SUT.

1

Our testing framework can model real attack mecha-nisms using known penetration mechanisms. It can alsomodel unknown future attacks by more powerful adver-saries by enabling direct attacks on software and hard-ware components that may go beyond known penetra-tion methods. We do this by mapping attacks to theirimpacts on the SUT. Hence, a key advantage of our sys-tem is that it allows design-time testing assuming a verypowerful attacker to test the limits of the SUT, withoutthe need to find a specific penetration path through thesystem.The primary contributions of this work are:

• a new flexible framework for design-time testingof new hardware-software architectures for securityproperties, leveraging VMM technology;

• a realistic environment using commodity operatingsystems for testing different applications using thenew security mechanisms;

• a flexible, fast, and low-cost method for emulatingHW features in the VMM for the purpose of designvalidation — without the need for costly and time-consuming fabrication of HW prototypes;

• the ability to simulate the impact of very powerfulattackers for “future” attacks; and

• the application of our framework towards the val-idation of the security properties of the SP archi-tecture [1, 5].

The rest of our paper is organized as follows: Sec-tion 2 discusses hardware-software architectures andtheir threat models. Section 3 describes the architec-ture and implementation of our new testing framework.Section 4 describes the SP architecture, its emulation inthe testing framework, and methodologies for its test-ing. Section 5 offers results and sample security attacks.We discuss related work in Section 6 and conclude inSection 7.

2 Hardware-Software SecurityArchitectures

New security architectures can take many forms. Forthis work, we focus on hardware-software architectureswhere new hardware security mechanisms are addedto a general-purpose computing platform to protectsecurity-critical software and its critical data. The hard-ware provides strong security protection which cannotbe bypassed, and the software provides flexibility to im-plement different applications and usage scenarios, withdifferent security policies.Figure 1 shows a typical system with the addition of atrusted software application and new trusted hardware

Figure 1: Threats and Attacks on Security Architec-tures

security mechanisms added to the CPU (e.g. new in-structions, faults, registers, and hardware mechanisms).Sometimes, as shown in the figure, the OS cannot betrusted, especially if it is a large monolithic OS likeWindows or Linux. Other times, an architecture mighttrust parts of the operating system kernel (e.g. a mi-crokernel [6]), but not the entire operating system.The figure also shows the sources of attacks that weconsider in our testing framework. First, malware or ex-ploitable software vulnerabilities can allow adversariesto take full control over the operating system to performsoftware attacks. They can access and modify all OS-level abstractions such as processes, virtual memory andvirtual memory translations, file systems, system calls,kernel data structures, interrupt behavior, general reg-isters, and I/O.Second, if adversaries get physical possession of a device,they can perform hardware attacks, such as directly ac-cessing data on the hard disk, probing physical memory,intercepting data on the display and I/O buses. Verypowerful attackers may even be able to probe parts ofthe processor chip.Third, network attacks can be performed with eithersoftware or hardware access to the device, or with ac-cess to other parts of the network. Some network attackmechanisms act like software attacks (e.g. remote ex-ploits on software), while others attack the network it-self (e.g. eavesdropping attacks) or application-specificnetwork protocols (e.g. modification attacks and man-in-the-middle attacks).In order to adequately test a new security architecture,all of these attack mechanisms must be considered andtested. Our testing framework provides hooks into eachrelevant system component, and additionally allows in-formation and events at each level to be correlated to

2

emulate the most knowledgable attacker.

3 Testing Framework

The design goals of our testing framework are to create ageneric platform that can emulate and test a wide rangeof security architectures in a realistic environment. Itmust test the architecture-specific application softwarerunning on top of full commodity operating systems.The testing framework should also allow easy monitor-ing of software and hardware events, and allow modifi-cation of software and hardware state to launch attackson the system.

3.1 Architecture

A virtual machine monitor (VMM) is the software whichcreates Virtual Machines (VMs), efficiently providingan execution environment in each VM which is almostidentical to the original machine [7]. By modifying anexisting VMM, we can augment the virtual machine tohave the additional hardware features of our new secu-rity architecture in addition to those of the base ma-chine. Since we are still using efficient virtualizationsoftware, we can run commodity operating systems andapplications in our modified virtual machines.Our Testing Framework is divided into two systems, asshown in Figure 2, the System Under Test (SUT) andthe Testing System (TS), each running as a virtual ma-chine on our modified VMM. The SUT machine simu-lates the system being designed and tested for the newsecurity architecture, as if it were actually built andprogrammed. The TS machine simulates the attacker,who is trying to violate the security properties of theSUT. Through a variety of hooks in our modified VMMand guest OS, the TS has full access to the internaloperations and state of the SUT.

Figure 2: Testing Framework Design

In isolation, the SUT is meant to behave as closely aspossible to a real system which has the new securityarchitecture. It must behave as if it has all of the newsecurity primitives available in hardware, along with thecorresponding protected software for that architecture.In our current system, the SUT runs a full commod-ity operating system (Linux) as its guest OS, which isvulnerable to attack and is untrusted. However, for thepurposes of the testing framework, we add a softwarecomponent, the TS Proxy (TSP), to the SUT to sim-ulate the effect of a compromised operating system forlaunching software attacks, allowing the OS to be fullycontrollable by the TS — as if it has been compromised.It is still necessary to keep the TS as a separate virtualmachine so that the TS Proxy in the SUT need not beinvoked to launch a virtual hardware attack.For the purposes of fully exploring the design space ofnew security architectures, the TS has the additional ca-pability to be a super-attacker. The testing frameworkitself is ignorant of the threat model of the system be-ing designed, and instead enables full controllability andobservation of the SUT in both hardware and software.This makes it suitable for many purposes during the de-sign phase of a new architecture. For initial design andimplementation of the system, the TS can act as a de-bugger, able to see the lowest-level micro-architecturalbehaviors of the hardware features and all code behav-ior and data in the software stack. When testing thesupposedly correct system, the TS is the attacker, con-strained by a threat model to certain attack vectors.A particular point of elegance of our framework is thatthe threat model can be easily changed, and the setof attack tools given to the attacker adjusted for eachtest. The framework can be used for any combination ofmechanisms: access to internal CPU state of the virtualprocessor, “physical” attacks on the virtual machinehardware (e.g. hardware probes on the buses, mem-ory, or disk), software attacks on the operating system(e.g. a rootkit installed in the OS kernel), and networkattacks (e.g. interception and modification of networkpackets and abuse of network protocols and applicationdata). For example, in some cases, it might be desir-able to perform black-box testing of a new design usingonly the network to gain access to the SUT, while inother cases, white-box testing will allow the attackerknowledge about the system’s activities, such as precisetiming of attacks with hardware interrupts or break-points into the application code or observation of datastructures in memory.

3.2 Testing Framework Modules

The main components of our Testing Framework shownin Figure 2 are: the Testing System Controller (TSC),the Testing System Proxy (TSP), the Events and Attack

3

Table 1: Example Events and Attacks

Layer Events Monitored Impact of AttackProtected Application API function entry/exit, Library calls,

User authentication, Network messages,Other application-specific events.

Read/write application data structures,Trigger application API calls, Inter-cept/modify network messages, Otherapplication-specific attacks.

OS Memory access watchpoints, Virtualmemory paging, File system access, Sys-tem calls, Process scheduling, Instructionbreakpoints, Device driver access, Net-work socket access, Interrupt handler in-vocation, etc.

Read/write virtual memory, Read/writekernel data structures, Read/write filesystem, Intercept/modify syscall pa-rameters or return values, Read/writesuspended process state, Modify pro-cess scheduling, Intercept/modify net-work data, Modify virtual memory trans-lations.

Base Hardware (x86) Privileged instruction execution, Trig-gering of page faults and other in-terrupts, Execution of an Instructionpointer.

Read/write general registers, Read/writephysical memory, Trigger interrupts, In-tercept device I/O (e.g. raw network &disk accesses).

Secure ArchitectureHardware

Execution of new instructions, Trigger-ing of new faults, Accesses to new regis-ters.

Read/write new registers & state,Read/write protected memory plaintext.

module, the Hardware Emulation Module, the TestingApplication, and the Attack Scripts. The TSC and TSPare each divided into a user-level application and a com-ponent in the OS kernel.

A System Under Test is comprised of three layers,namely: Application, Operating System, and underly-ing Hardware. The Testing System should be able toboth monitor the events and modify the state in allthree of these layers. The modified VMM monitors andcontrols the hardware, while the the TSP monitors andcontrols the applications and OS.

The TS and SUT components communicate to exchangeinformation about events occurring in the SUT and toimplement attacks from the TS by modifying systemstate in the SUT. The user level components communi-cate with the kernel components through system calls,and the kernel components use signals to communicateasynchronously with the user level components. TheVMM communicates with the guest OS kernel asyn-chronously through a new virtual hardware interruptwhich we have defined.

Testing System Controller The TS Controller,running on the TS, is the aggregation point that re-ceives events from all three layers in the SUT. It re-ceives OS and Application level (software) events fromthe TS Proxy via a network channel and receives hard-ware events from the VMM through a channel from itskernel-level component. It provides APIs to the AttackScripts which can monitor or wait for specific events and

adaptively mount a coordinated attack on the SUT.

Testing System Proxy This module acts as a proxyfor the TSC running on the SUT itself. It controlsthe application to be tested, and uses its correspondingkernel-level component to control and monitor OS be-havior and the OS-level abstractions used by the testingapplication, including system calls, virtual memory, filesystems, sockets, etc. The TSC communicates with theTSP over TCP/IP on the virtual network which con-nects the VMs. Note that the TSP is used to simulatethe impact of a compromised OS in the SUT, whereasin the framework the OS itself is not actually compro-mised.

Event and Attack Module This module provideshooks to both observe and control the internal machinestate of the SUT from inside the VMM. The TS Con-troller registers with this module for the set of eventsto be monitored, and uses the APIs provided by thismodule to observe or modify (i.e., attack) the state.Events that can be watched include: interrupts/faultsspecific to the secure architecture, execution of new in-structions, execution of new behaviors (e.g. modifiedhardware interrupt handling), regular x86 interrupts,etc. Machine state that is exposed includes CPU reg-isters, physical memory, and the new registers for thesecure architecture. Note that unlike a normal attacker,this model is very powerful and can stop the machineprecisely for observation or modification before or af-ter any specific instruction execution or other detected

4

hardware event. In Figure 2, we show this module splitwithin the VMM into two separate components for eachVM to accommodate the split-VMM structure presentin some virtualization software.Hardware Emulation Module This component,added to the VMM, emulates the behavior of the newsecurity architecture to be tested. It enhances thebase micro-architecture with new instructions, func-tional units, registers, interrupts/fault behaviors andmemory access behavior. New instructions are exportedto the software layers through hypercalls, which allowapplication or OS layers to directly invoke the VMM.Testing Application The Testing Application is theactual implementation of the application software thatwould be run on the secure architecture. It shoulddemonstrate the features of the architecture and pos-sibly interact with a local user to provide access to se-crets and data which the architecture is protecting. Thetesting framework has hooks that allow the applicationto report any internal events such as function calls ordecision points. The usage of these hooks are optionaland is required only if the application level events needto be monitored for the attack script.Attack Scripts The Attack Scripts reside on the TSand specify how particular attacks are executed on theSUT. They provide step-by-step instructions for mon-itoring events and dynamically responding to them inorder to successfully launch attacks, or detect that anattack was prevented by the security architecture. Thescripts act like a state-machine, acting on hardware andsoftware events which are aggregated by the TS. Scriptscan be written to form a library of generic attacks, thatcan be used to attack any application. Alternativelythey can be specific to the behavior of the testing ap-plication, written by the user of the testing framework.The TS Controller reads and executes these scripts andimplements the communication mechanisms and controlof the SUT as needed.In addition to these components, the TS hosts and emu-lates any other trusted third parties required by the sys-tem, possibly intercepting and modifying their networktraffic as an additional source of events and attacks.By using two separate virtual machines to host the com-ponents of our testing framework, we are able to keepthe SUT as close as possible to the real security archi-tecture being tested, including all of its software com-ponents, both trusted and untrusted. We do not needto trust the SUT’s guest OS for the sake of correctnessof the emulation of new hardware security features. Infact, the SUT’s guest OS can launch real attacks on thesystem (as controlled by the TS).We also have the ability to use the separate VM tolaunch hardware attacks asynchronously from the TS;the TS can pause the execution of the entire SUT VM

and continue running the attack scripts to inject attacksat any instruction boundary.

3.3 Events and Attacks

The testing framework is designed to expose events fromthe three layers of the SUT (application software, OS,and hardware) to the TS, and to allow the TS accessto the state of the SUT to launch attacks. Table 1 listsvarious events and attacks for each layer of the system.The hardware layer is further classified into events andattacks from the base hardware (x86 architecture in ourwork) and the new emulated security architecture.The hardware layer is accessed by the TS Controllerthrough the event and attack module in the VMM,which communicates events across an inter-VM channelinside the VMM to relay events and attacks betweenthe SUT and TS. This channel is used to communicatewith the TS during execution of a single instruction orHW operation in the SUT, possibly changing the resultof that operation.The software layers are accessed through the TS Proxy,which hooks into the OS kernel through its kernel mod-ule, and to the testing application using its user-modecomponent. The TSP can function as a debugger toolreading the application’s memory and accessing its sym-bol table to map variable and function names to vir-tual addresses. The application can also optionally beinstrumented to access its state and events. The TSProxy suspends scheduling of the application’s processwhile it is communicating an event or attack to the TSController.The SUT modules in both the hardware and softwarelayers capture their relevant events and signal the TSController. The controller, following its attack scripts,can then attack the SUT based on those events. Table 2lists the API which the TS Controller exports to theattack scripts for event handling and the basic attackmechanisms.The attack mechanisms we provide are designed aroundthe impact that attacks have on the state of the SUT.The security properties and attacks considered in thethreat model of a given security architecture may in-stead focus on the method of penetration used to ac-cess that state. For our testing framework, we enablea powerful attacker without regard to how penetrationoccurs. This is preferred since (1) new attack penetra-tion methods are frequently discovered after a systemis deployed and often are not foreseen by the designer,(2) most real attacks result in or can be modeled by theimpact of attacks which we provide in Table 1, and (3)the attack scripts themselves can be restricted to modelspecific penetration methods when testing for a morelimited attacker. Thus an architecture can be tested

5

Table 2: TS Controller API for Attack Scripts

Function Descriptionh ← INIT() Initializes the TSC and returns a handle h to access the resources.FREE(h) Free the resources for the handle h.EXECUTE(h,app,params) Execute the application app on SUT with the given parameters params.EVENTADD(h,eventType) Add the eventType to watch-list.EVENTDEL(h,eventType) Delete the eventType from the watch-list.event ← WAIT(h) Blocking call that waits for any event in the watch-list to occur in

the SUT. Once an event is triggered, the SUT is paused and the TScontinues running the attack script. An application exit in the SUTalways causes a return from WAIT().

event ← WAITFOR(h,eventType) Similar to WAIT() but waits for the specified event (or applicationexit), regardless of the watch-list.

CONT(h) Execution of the SUT is resumed.ACCESSREG(h,type,r/w,buf) Reads/writes (r/w) the general registers or SP registers (type) of the

SUT to/from buf.ACCESSMEM(h,v/p,r/w,addr,sz,buf) Reads/writes (r/w) sz bytes from virtual or physical memory (v/p) of

the SUT at address addr to/from the buffer buf.INTERRUPT(h,num) Trigger a virtual hardware interrupt number num on the SUT.

against unknown attacks as well as known methods ofpenetration by considering the most powerful attacker.

3.4 Implementation

We implemented our testing framework on VMware’svirtualization platform [8], including all of the mod-ules in Section 3.2 and events and attacks at each sys-tem layer. The Hardware Emulation Module and VMMEvent & Attack Module required modifying the sourcecode of the VMware VMM. The kernel components ofthe TSP and TSC are implemented as Linux kernelmodules. The TSP application is implemented as aLinux user process and controls the execution of theTesting Application. The TSC application is imple-mented as a static library which is called by the AttackScripts. Upon initialization, the TSC connects with theTSP over the virtual network, and the TSP registerswith the Event and Attack Module via a hypercall tothe VMM.As a sample security architecture, we implement the SParchitecture, described in Section 4. Hence, the Hard-ware Emulation Module emulates the SP architectureincluding its master secrets, secure memory, and inter-rupt protection. We have also implemented a libraryof protected software for SP, which is used for a remotekey-management application as described in Section 4.4.Our Testing Application uses this library to exercise thesoftware, and in turn, the SP hardware.

4 SP Architecture and Emula-tion

For the rest of this paper, we will use the Secret Protec-tion (SP) architecture [1][5] to demonstrate our frame-work. We have chosen SP because it was specificallydesigned to be as simple as possible to facilitate veri-fication, while providing robust security utilizing bothhardware and software components. At the same time,it uses a somewhat controversial and unproven design,skipping layers in the trust chain by using hardwareto directly protect an application without trusting theunderlying operating system. To be accepted by the se-curity community, SP’s security properties need to bevalidated, demonstrating that it can protect the confi-dentiality and integrity of cryptographic keys in its per-sistent storage which in turn protect sensitive user data.Furthermore, it is important to write and test many se-cure software applications to run on SP in a realisticenvironment, where the untrusted OS can be a sourceof attacks. To the best of our knowledge no single toolexists to test hardware-software security architectureslike SP.Our testing framework first emulates SP’s hardware fea-tures using modifications to the VMM. This allows us todirectly test the primitives and guarantees made by theSP hardware itself and how those primitives interactwith software components. Additionally, we can con-sider the hardware emulation as a constant and test a

6

range of software implementations which take advan-tage of the SP hardware. We wish to test how SP hard-ware protects its software, how trusted software can bewritten to take advantage of the hardware guarantees,and how trusted software interacts with untrusted soft-ware on an SP device.

4.1 Secret Protection Architecture

In the Secret Protection (SP) architecture, the hardwareprimarily protects a Trusted Software Module (TSM),which protects the sensitive or confidential data of anapplication. Hence, a TSM plus hardware SP mech-anisms form the equivalent of the trusted computingbase (TCB) for the application. Rather than protect-ing an entire application, only the security-critical partsof an application are made into a TSM, while the rest ofthe application can remain untrusted. Furthermore theoperating system is not trusted; the hardware directlyprotects the TSM’s execution and data.Protecting the TSM’s execution requires ensuring theintegrity of its code and the confidentiality and integrityof its intermediate data. Code must be protected fromthe time it is stored on disk until execution in the pro-cessor. Data must be protected any time when the op-erating system or other software can access it. This in-cludes storage on disk, in main memory, and in generalregisters when the TSM is interrupted.To provide this protection, SP maintains its state usingnew processor registers. It assumes the processor chipto be the security boundary, safe from physical attackswhich are very costly to mount on modern processors.As shown in Figure 3, SP uses two on-chip master se-crets: the Device Root Key (DRK) and the StorageRoot Hash (SRH). For code integrity, the DRK is usedto sign a MAC (a keyed cryptographic hash) of eachblock of TSM code on disk. When a TSM is execut-ing, the processor enters a protected mode called Con-cealed Execution Mode (CEM). As the code is loadedinto the processor for execution in the protected mode,the processor hardware verifies the MAC before execut-ing each instruction. For the TSM’s intermediate data,while in protected mode, the TSM can designate cer-tain memory accesses as “secure”, which will cause thedata to be encrypted and hashed before being writtento main memory. This secure data is verified and de-crypted when it is loaded back into the processor fromsecure memory. Secure data and code are tracked withtag bits added to the on-chip caches. Additionally, theSP hardware intercepts all faults and interrupts thatoccur while in the protected mode before the OS getscontrol of the processor. SP will encrypt the contents ofthe general registers in place, and keep a hash of the reg-isters on-chip in the interrupt registers to verify beforedecryption when the TSM is resumed.

Figure 3: Secret Protection (SP) Architecture. Enlarge-ments show (a) the additional CEM hardware and (b)the application secrets protected by the TSM.

Secret data belonging to the application is also pro-tected in persistent storage by encryption using keysprotected by the TSM. SP allows a TSM to derive newkeys from the DRK using a new hardware instruction,DRK_DeriveKey. These derived keys are used by theTSM to protect the confidentiality and integrity of itspersistent data. Other software, including the OS, maynot use the DRK_DeriveKey instruction. The TSM isalso the only software that can read and write the SRHregister, using it as the root of a hash tree to protectthe integrity of this persistent secure data.

Hence, to emulate SP hardware we require the follow-ing components: new processor registers (including theprotected mode and master secrets); new instructions;hardware mechanisms for code integrity checking, se-cure memory, and interrupt protection; and new hard-ware faults which these mechanisms generate.

4.2 Emulation of the SP Architecture

A traditional VMM provides a virtual machine whichis (nearly) identical to the physical machine, matchingthe instruction set and behavior of the real CPU. It doesthis by trapping or translating privileged code, while ig-noring microarchitecture effects that do not affect pro-gram correctness, such as cache memory. Most of thetime, the VMM runs code on the physical hardware,and only emulates the components that are virtualized.In order to implement and emulate new hardware archi-tecture features, we take advantage of the VMM’s vir-tualization methods. For example, the VMM maintainsdata structures for the virtual CPU state, which we canexpand to store new security registers. The VMM thenemulates accesses that are made to those new registers.Other useful VMM behaviors include: interception of allhardware interrupts, binary translation of code, map-ping of virtual memory translations, and virtualization

7

of hardware devices.To emulate the SP architecture, a number of compo-nents must be added to the virtual machine as part ofthe Hardware Emulation Module:Protected Mode SP’s Concealed ExecutionMode [5] requires new registers which are added to thevirtual CPU, including the registers holding SP’s mas-ter secrets (DRK and SRH), mode bits, and interrupthandling registers. New SP instructions are currentlymodeled as hypercalls, where the TSM running in theSUT is able to directly invoke the emulation modulewithout going through the guest OS. Any trappinginstruction would serve this purpose. Alternatively,binary translation can be used by the VMM for TSMcode, which can be written with new unused opcodesto implement new instructions.Interrupts and SP Faults The SP architecturechanges the hardware behavior when interrupts occurwhen in protected mode. Since the VMM already needsto emulate interrupt behavior, we can simply detect thatan interrupt has occured during the protected mode andemulate the effect on the CPU, which includes suspend-ing the protected mode and encrypting and hashing thegeneral registers. To detect returning from an interrupt,the VMM inserts a breakpoint at the current instruc-tion pointer where the interrupt occurs, so that it isinvoked to emulate the return-from-interrupt behaviorof SP, which includes verifying the hash and decryptingthe general registers before resuming the TSM. Addi-tionally, new SP faults can be triggered by both hard-ware events and new SP instructions. When the emu-lated hardware generates a new fault, it first reports tothe TSC, and then translates the fault into a real x86fault, such as a general protection fault, which is raisedin the SUT causing the OS to detect the failure of theTSM.Secure Memory We change the SP abstraction ofsecure memory to work on virtual memory pages insteadof cache lines, since the VMM does not intercept cachememory behavior. While this limits the ability to modela few low-level attacks on SP (such as the behavior ofcache tags), the majority of the security properties ofthe hardware and all those of the software can still betested.We also change how the hardware determines when toaccess secure memory, so that the VMM need not trapevery individual memory access, causing performancedegradation. Rather than using new instructions toaccess secure memory and regular memory, we allowthe TSM to define regions of memory which are alwaysaccessed securely — an abstraction which fits the em-bedded SP model [9]. This simplifies writing exampleTSMs since an entire data structure or stack segmentcan be protected, and the compiler need not be modi-

fied to emit new instructions for memory accesses. TheTSM simply declares which memory ranges should beprotected upon application launch.

Code Integrity SP verifies the integrity of the TSMcode dynamically during execution. The TSM’s code issigned with a keyed hash over each cache-line of codeand the virtual address of that line, and is checked aseach cache line is loaded into the processor. We modelthis using the VMM’s binary translator to execute theTSM code. The implementation tags secure instruc-tions as code fragments in the binary translator cache.SP’s instruction signing (with keyed hashes) is similarlygenerated and stored across larger regions of code (foremulation efficiency) and saved in a separate file.

4.3 Other Architectures

While this paper focussed on testing the hardware andsoftware mechanisms of the SP architecture, our test-ing framework is by no means limited to this architec-ture. Other security architectures such as XOM [10],AEGIS [11, 12, 13] and Arc3D [14, 15] modify hard-ware in similar ways but have somewhat different goalsand assumptions from SP. However they combine hard-ware and software in ways that also make them suitablefor validation in our framework. Similarly, TPM [16]modifies hardware to protect all software layers andprovide cryptographic services. However, rather thanutilizing changes to the processor itself, TPM adds aseparate hardware chip that integrates with the systemboard. This is still compatible with our testing frame-work however requiring a different set of modificationsto the VMM to implement a virtual TPM device.

4.4 Remote Key-management TSM

SP’s Concealed Execution Mode provides code integrity,a secure execution state, and secure memory for theTSM. The TSM in turn must be written to supportapplications that use sensitive data. In software, theTSM uses the master secrets (in the DRK and SRHregisters) to protect the confidentiality and integrity ofpersistent data, and to offer interfaces to unprotectedapplications that need to use this data. It must alsomanage secure communication with the user (includingauthentication) and with third parties over the network.

A TSM can be written for many different usage sce-narios. As an example we look at the remote key-management scenario from Authority-mode SP [1]. Weuse this TSM to test various aspects of the SP ar-chitecture, from the hardware features in the virtualprocessor (which are now protecting a real TSM), tothe software and network interfaces. We demonstrate

8

how the security-critical components of the remote key-management software can be partitioned off into a TSMwhich can be used by other software. Using the frame-work, we then subject our TSM to a suite of attacks tosee if the desired security properties are preserved.For remote key-management, we consider a centraltrusted authority which owns multiple SP devices andwants to distribute sensitive data to them. The author-ity installs its remote key-management TSM on eachdevice as well as the protected data, consisting of se-crets and cryptographic keys that protect these secrets.It also stores policies for each key which dictate how itmay be used by the local user. During operation, theTSM will accept signed and encrypted messages fromthe authority to manage its stored keys, policies, anddata. It also provides an interface to the applicationthrough which the local user can request access to dataaccording to the policies attached to the keys. The TSMmust authenticate the user, check the policy, and thendecrypt and display the data as necessary.Testing SP requires testing TSM software in combina-tion with SP hardware mechanisms. The TSM pro-vides protection for the application’s critical data andis itself protected by the SP hardware. We can at-tack the robustness of the TSM’s memory usage, persis-tent storage, network protocols, and software interfaces.We thereby use our testing framework as a testbed forsecure software development in addition to the securehardware — both the software TSM and the hardwareSP features are required to provide SP’s confidentialityand integrity guarantees.

4.5 Testing Example

We now give an example of how the TSC can test an ap-plication and TSM running on the SUT. Table 3 shows,on the left, the protected application using a TSM(TSMapp) and a corresponding unprotected application(unsafeApp). The attack script is shown on the right.The attack simulates a compromised OS trying to accessconfidential data currently being used by a TSM (in thiscase, a secret key used for encrypting sensitive data) byinterrupting the TSM during its execution. The TSMappuses SP instructions to enter and exit protected mode(BEGIN_TSM and END_TSM) and to generate a key(DRK_DeriveKey). The unsafeApp is instrumented toexplicitly notify the TSP (with TSP_Notify)when itgenerates a key.The application first generates a new key, forTTP_KeyID, which it uses to encrypt data. It thensends the encrypted data over the network to the re-mote Trusted Third Party (TTP). In the first line of theattack script, the TSC launches the testing application,either using the TSMapp (where we expect the attacks to

fail) or the unsafeApp (where the attacks should suc-ceed). The script then watches for a particular event tooccur in the application, in this case the generation of akey. The application executes normally until it triggersthis event which the attack script is monitoring.

Steps 1-5 in figure 4, show the steps taken by the testingframework to detect an event. In step 1, the TSM usesan SP instruction to generate a new derived key fromthe DRK master secret. This requires making a hyper-call down to the SP HW Emulation Module to emulatethe instruction and produce results into the general reg-isters. In step 2, the SP HW notifies the Event & AttackModule of this HW event (the execution of an SP in-struction). In step 3, the event is passed up to the TSCkernel module via a virtual HW interrupt/IRQ, and theVMM freezes execution of the entire SUT. In step 4, theTSC is notified of the event via a Linux signal. In step5, the TSC returns the event to the attack script whichwas waiting for notification of this key generation event.It can now continue executing the script to perform anyhardware attacks while the SUT is frozen. These stepsare repeated for every hardware event monitored by theTSC.

Figure 4: Example Attack Sequence

The script resumes the TSM and then attacks by in-structing the TSC to generate an interrupt during theTSM’s execution. This triggers the SP HW’s interruptprotection mechanism, encrypting the general registersand secure memory, and suspending protected mode forthe TSM. Next the script, acting like a corrupted oper-ating system’s interrupt handler, attempts to read andmodify the TSM’s general registers. It overwrites thegenerated key with a known key. With SP protection, itcan only read an encrypted version of the registers, andupon resuming from the interrupt, the incorrect gen-eral register values will cause an SP fault. If the sameattack is performed using the unprotected applicationwithout SP and its TSM, the attacker would obtain thegenerated key and the user of the SUT would remainunaware of the modification to its registers, encryptingwith a key known to the attacker.

9

Table 3: Example Application and Attack Script for Eavesdropping and Spoofing Attack using Interrupt

Application with TSM (TSMapp)BEGIN_TSM· · ·Reg1 ← DRK_DeriveKey(TTP_KeyID)Ciphertext ← Encrypt(Reg1, &SecureMem, sz)END_TSMNetwork_Send(TTP, Ciphertext, sz)· · ·

Application without TSM (UnsafeApp)

// No SP protection· · ·Reg1 ← App_Generate_Key(TTP_KeyID)TSP_Notify(App_Keygen, Reg1)Ciphertext ← Encrypt(Reg1, Data, sz)Network_Send(TTP, Ciphertext, sz)· · ·

Attack ScriptEXECUTE(TSMapp, params) // or UnsafeApp// Wait for key generationEVENTADD(DRK_DeriveKey)EVENTADD(App_Keygen)EVT ← WAIT()ACCESSREG(SP, r, SPRegs)SPKey ← SPRegs.DerivedKeyAppKey ← EVT.param1// Trigger an interruptINTERRUPT(03); CONT()EVT ← WAITFOR(Interrupt)// Attack confidentiality of general registers (GRs)ACCESSREG(GR, r, GRegs)if SPKey = GRegs.R1 OR AppKey = GRegs.R1 then

print “Register Confidentiality Attack Succeeded”else

print “Register Confidentiality Attack Failed”end if// Attack integrity of general registers (GRs)ACCESSREG (GR, w, KnownKey); CONT()EVT ← WAITFOR(SPFault_All)if EVT = SPFault_Reg_Integrity then

print “Register Integrity Attack Failed”else

print “Register Integrity Attack Succeeded”end if

5 Testing of SP

As we have shown in the previous section, it is possibleto use our new testing framework to perform a widerange of tests and attacks on the SUT and the securityarchitecture it emulates. We now look more closely athow we can use the framework to validate the securityproperties of the SP architecture in particular througha range of tests.SP claims to provide a number of security properties,of particular importance are the confidentiality and in-tegrity of: master secrets (DRK & SRH), TSM execu-tion, secure data, and usage policies. We must demon-strate that each security property is enforced throughthe combination of trusted and verified TSM softwareand trusted SP hardware.The main goal of the remote key-management TSM isto enforce policies on secure data, as dictated by the Au-thority, in an SP device. The following steps establishthe trust chain of the architecture:

1. The authority binds the TSM code to the SP device

(and its master secrets, which cannot leave the de-vice) and provides secret data to the TSM on thatdevice.

2. The TSM executes in Concealed Execution Mode.

3. The TSM binds security policies and secure datato the keys derived from SP’s master secrets on thedevice.

We need to verify each of these steps by launching at-tacks to test the SP’s trust chain.In Table 4 we show a list of attacks which test theseaspects of the architecture, using our framework, theemulated SP hardware, and our TSM. Each row in thetable indicates a specific attack we have implemented,where ‘PASS’ indicates that the architecture success-fully protected against the attack.When the authority provides secret data, it needs to besure that it can only be used on the designated deviceand only using the TSM it provided which is trustedto enforce its security policies. Therefore it must notbe possible to replace that TSM with another, even

10

when an adversary has physical possession of the SPdevice. Therefore, for step 1, we attempt to replacethe TSM by installing a new DRK which signs differentTSM code, however it then cannot access the author-ity’s data which was protected by the original DRK.Since the DRK never leaves the processor chip and can-not be extracted, there is no other way to get access tothe original DRK.In step 2, the integrity of a TSM must be protected bothbefore and during execution. This involves testing SP’shardware protection of the confidentiality and integrityof all intermediate data that it generates. This inter-mediate data can be attacked while in general registersduring a processor interrupt or in secure memory at anytime while the TSM is not running. Additionally, theSP master secrets themselves must only be used by theTSM itself and not other software. The code integrityof the TSM must be protected both on disk and duringexecution.Finally, step 3 covers the proper design and implemen-tation of the TSM itself. The TSM must store cryp-tographic keys, security policies, and secure data in itspersistent secure storage, which it protects using SP’sunderlying hardware mechanisms (DRK & SRH). Wetest the confidentiality and integrity of the storage itself,plus the TSM’s use of the storage and its enforcement ofthe policies on accesses to the data. Adversaries mightattack the TSM’s persistent data offline when stored ondisk or after the TSM has loaded it into secure memory.Our system implements the SP hardware mechanisms,a full TSM providing an API to the testing application,and a suite of attacks which test both the software andhardware components using our new testing framework.This is a first step towards the complete validation ofthe design of the SP architecture together with its ap-plications, and provides a framework for further testingthe architecture with other software and attacks.

6 Related Work

Our testing-framework of a system architecture solutionhas multiple related fields of research. Our emulationof security architecture can be compared with hardwaresimulation research, our validation mechanisms can becompared with formal and hybrid verification method-ologies, and our virtualization architecture can be com-pared with systems virtualization solutions.Micro-architectural simulators like Simplescalar [17] arecycle-accurate and hence can be very useful in estimat-ing performance metrics, but they can not simulate a re-alistic software system with full commodity OS. Thus itis impossible to test the security critical interactions of asoftware-hardware security solution with such a simula-

tion architecture. Our emulation architecture, in usingthe existing virtualization technology, enables reason-able performance while allowing our SUT to providea realistic software stack. Other simulation and emu-lation environments are available, such as Bochs [18],Qemu [19], and Xen [20]. Where these environmentsprovide sufficient software performance and granularityof hardware emulation, they could be used in place ofVMware to implement our framework as designed anddescribed in this paper.The efforts by IBM [21], Intel [22] and others [23] pro-vide the functionality of a virtual TPM device to soft-ware, even when the physical device is not present. Incontrast, we not only emulate the new hardware but alsohook into the virtual device to observe and control itsbehavior for testing purposes, and study the interactionwith other hardware and software components.Another related area of research is the Formal Verifi-cation of both hardware and software, in which formalmathematics is used to write specifications for computerhardware and software, and proof techniques are usedto determine the validity of such specifications. Thecomplexity of formal verification problems range fromNP-hard to undecidable [24, 25, 26, 27]. The com-plexity of these formal verification mechanisms led tothe use of hybrid techniques [28] which uses some for-mal as well as informal techniques. Some formal meth-ods of verification are methods using theorem provers(ACL2 [29], Isabelle/HOL [30]), model checkers, satis-fiability solvers, etc. Some informal techniques used inpractice are control circuit exploration, directed func-tional test generation, automatic test program genera-tion, heuristic-based traversal, etc. The formal and hy-brid techniques try to verify the hardware and softwareseparately, unlike our holistic verification of a software-hardware system.The important distinction between our approach andthese formal verification techniques is that we try toverify the complete system while these formal tech-niques try to verify each piece by piece. The complex-ity of both specification and verification explodes ex-ponentially with addition of more and more pieces tobe tested. In our approach we model both the securitycritical hardware and software as a single entity, thus,making the verification problem solvable in an informalbut systematic and efficient way.

7 Conclusion

We have designed and implemented a virtualization-based framework for validation of new security archi-tectures. This framework can realistically model a newsystem during the design phase, and draw useful conclu-sions about the operation of the new architecture and

11

Table 4: Verifications and Attacks

# Security Property Attack Mechanism Results1 Binding TSM to Secrets Attack the persistent storage protected

by one TSM from another TSM whenthe DRK is changed.

Physical possession ofSP device

PASS

2 Confidentiality & In-tegrity of Secure Memory

Attacks secure memory outside pro-tected mode through eavesdropping,spoofing, and splicing.

Virtual/physical mem-ory access to TSMpages

PASS

Confidentiality & In-tegrity of General Regis-ters

Attacks on registers during an interruptof a TSM through eavesdropping, spoof-ing, splicing, and replay.

OS access to savedprocess context

PASS

Access HW Master Se-crets

Attack the SP master secrets (DRK &SRH) from outside protected mode

Unprotected applica-tion execution

PASS

Integrity of TSM Code Attacks on TSM code before and duringexecution through spoofing and splic-ing.

Disk access or accessto text pages in mem-ory

Not YetTested

3 Confidentiality & In-tegrity of PersistentSecure Storage

Attack the TSM’s secure storage, cre-ated using DRK and SRH, on disk orwhen loading into memory.

Disk access or accessto data pages in mem-ory

PASS

Binding Security Policiesto Keys

Attack the TSM’s enforcement of secu-rity policies on the use of keys in itspersistent storage, and the secrets thosekeys protect.

Disk access or accessto data pages in mem-ory

PASS

its software interactions. It also enables testing varioussoftware applications using the new security hardware,unlike a new hardware prototype which does not havean OS running on it.

Our framework serves as a rapid prototyping vehiclefor black-box or white-box testing of security proper-ties. It can utilize and integrate multiple event sourcesand attack mechanisms from the hardware and softwarelayers of the system under test. These mechanisms cantest both low-level mechanisms and high-level applica-tion behavior. As a result, a full range of attacks arerealizable on the hardware, operating system, and ap-plications.

Finally, we implement the SP architecture in our frame-work and test its security mechanisms thoroughly. Wedemonstrate that the full trust chain of the architectureholds up under attack, using a real TSM for remote key-management.

References

[1] J. S. Dwoskin and R. B. Lee, “Hardware-rootedTrust for Secure Key Management and Tran-sient Trust,” in ACM Conference on Computer

and Communications Security (CCS), (Alexandria,VA), pp. 389–400, October 2007.

[2] R. P. Goldberg, Architectural Principles for VirtualComputer Systems. PhD thesis, Harvard Univer-sity, 1972.

[3] R. Goldberg, “Survey of Virtual Machine Re-search,” IEEE Computer, pp. 34–45, June 1974.

[4] M. Rosenblum and T. Garfinkel, “Virtual machinemonitors: current technology and future trends,”IEEE Computer, vol. 38, no. 5, pp. 39–47, 2005.

[5] R. B. Lee, P. C. S. Kwan, J. P. McGregor, J. S.Dwoskin, and Z. Wang, “Architecture for Protect-ing Critical Secrets in Microprocessors,” in Intl.Symposium on Computer Architecture (ISCA),pp. 2–13, 2005.

[6] “OKL4 Microkernel.” Open Kernel Labs, http://www.ok-labs.com.

[7] G. Popek and R. P. Goldberg, “Formal Require-ments for Virtualizable 3rd Generation Architec-tures,” Communications of the A.C.M., vol. 17,no. 7, pp. 412–421, 1974.

[8] “VMware Workstation.” VMware Inc., http://www.vmware.com.

12

[9] J. Dwoskin, D. Xu, J. Huang, M. Chiang,and R. Lee, “Secure Key Management Architec-ture Against Sensor-node Fabrication Attacks,” inGlobecom, 2007.

[10] D. Lie, C. A. Thekkath, M. Mitchell, P. Lincoln,D. Boneh, J. C. Mitchell, and M. Horowitz, “Ar-chitectural Support for Copy and Tamper Resis-tant Software,” in Intl. Conference on ArchitecturalSupport for Programming Languages and OperatingSystems (ASPLOS), pp. 168–177, 2000.

[11] G. E. Suh, D. E. Clarke, B. Gassend, M. van Dijk,and S. Devadas, “AEGIS: Architecture for Tamper-evident and Tamper-resistant Processing,” in Intl.Conference on Supercomputing (ICS), pp. 160–171,2003.

[12] G. E. Suh, C. W. O’Donnell, I. Sachdev, and S. De-vadas, “Design and Implementation of the AEGISSingle-Chip Secure Processor Using Physical Ran-dom Functions,” in Intl. Conference on ComputerArchitecture (ISCA), pp. 25–36, 2005.

[13] G. E. Suh, C. W. O’Donnell, and S. Devadas,“Aegis: A Single-Chip Secure Processor,” IEEEDesign & Test of Computers, vol. 24, no. 6,pp. 570–580, 2007.

[14] M. Gomathisankaran and A. Tyagi, “Arc3D : A 3DObfuscation Architecture,” in High PerformanceEmbedded Architectures and Compilers (HiPEAC),pp. 184–199, Springer, 2005.

[15] M. Gomathisankaran and A. Tyagi, “ArchitectureSupport for 3D Obfuscation,” IEEE Trans. Com-puters, vol. 55, no. 5, pp. 497–507, 2006.

[16] Truscted Computing Group, Trusted Plat-form Module Specification Version 1.2 Re-vision 103, July 2007. https://www.trustedcomputinggroup.org/.

[17] T. Austin, E. Larson, and D. Ernst, “SimpleScalar:An Infrastructure for Computer System Model-ing,” Computer, vol. 35, pp. 59–67, February 2002.

[18] D. Mihocka and S. Shwartsman, “VirtualizationWithout Direct Execution or Jitting: Designinga Portable Virtual Machine Infrastructure,” in 1stWorkshop on Architectural and MicroarchitecturalSupport for Binary Translation in ISCA-35, (Bei-jing), June 2008.

[19] F. Bellard, “QEMU, a Fast and Portable DynamicTranslator,” in USENIX Annual Technical Confer-ence, FREENIX Track, pp. 41–46, 2005.

[20] P. Barham, B. Dragovic, K. Fraser, S. Hand,T. Harris, A. Ho, R. Neugebauer, I. Pratt, andA. Warfield, “Xen and the art of virtualization,”in SOSP ’03: Proceedings of the nineteenth ACMsymposium on Operating systems principles, (NewYork, NY, USA), pp. 164–177, ACM, 2003.

[21] S. Berger, R. Caceres, K. A. Goldman, R. Perez,R. Sailer, and L. van Doorn, “vTPM: Virtual-izing the Trusted Platform Module,” Tech. Rep.RC23879, IBM, February 2006.

[22] V. Scarlata, C. Rozas, M. Wiseman, D. Grawrock,and C. Vishik, Trusted Computing, ch. TPM Virtu-alization: Building a General Framework, pp. 43–56. Springer, 2008.

[23] M. Strasser, H. Stamer, and J. Molina, “Software-based TPM Emulator.” http://tpm-emulator.berlios.de.

[24] W. A. Hunt, “Mechanical Mathematical Meth-ods for Microprocessor Verification,” in Intl. Con-ference on Computer Aided Verification (CAV),pp. 523–533, 2004.

[25] S. Ray and W. A. Hunt, “Deductive Verification ofPipelined Machines Using First-Order Quantifica-tion,” in Intl. Conference on Computer Aided Ver-ification (CAV), pp. 31–43, 2004.

[26] H. Iwashita, S. Kowatari, T. Nakata, and F. Hirose,“Automatic test program generation for pipelinedprocessors,” in Intl. Conference on Computer AidedDesign (ICCAD), pp. 580–583, 1994.

[27] R. C. Ho, C. H. Yang, M. Horowitz, and D. L.Dill, “Architecture Validation for Processors,” inInt. Symposium on Computer Architecture (ISCA),pp. 404–413, 1995.

[28] J. Bhadra, M. S. Abadir, L.-C. Wang, and S. Ray,“A Survey of Hybrid Techniques for FunctionalVerification,” IEEE Design & Test of Computers,vol. 24, no. 2, pp. 112–122, 2007.

[29] S. S. Moore, “Symbolic Simulation: An ACL2 Ap-proach,” in Formal Methods in Computer-AidedDesign, pp. 334–350, 1998.

[30] T. Nipkow, L. Paulson, and M. Wenzel, Isabelle’sLogics: HOL.

13