FREE: FPGA-Reinforced Execution Enclaves in the Public Cloud

Mark Zhao*, Mingyu Gao†, and Christos Kozyrakis*

1

2

Your Data Leaks

Why Does Your Data Leak?

3

• Humans are erroneous and malicious• Key Misuse• Social Engineering• Malicious Employees (Devs, SysAdmins, Security Guards, Janitors, etc.)

• We implicitly trust a mountain of software• Hypervisor (Xen ~270k LoC)• Operating System (Linux ~17M+ LoC)• Applications• Cloud Service Provider (CSP) Infrastructure

We need to shrink the Trusted Computing Base (TCB) to improve security

A Catalyst for Hardware-Based Isolation• Reduce TCB to only HW

• Isolated HW containers called enclaves• Secure against privileged SW• e.g. SGX, TrustZone, Sanctum, XOM, Aegis, etc.

• Hardware-enforced security mechanisms• Remote Attestation• Isolated Execution• Sealing

4

Hardware

OS

App App

VMM

DRA

M

Verifier

CPU Enclaves are Not Enough• Reason 1: Complexity and Obscurity

5

CPU Enclaves are Not Enough• Reason 2: Compute is moving towards specialization

6

High PerformanceLow Performance

High Security

Low Security

The Goal

• Accelerated and trusted computation• Customizable levels of privacy and compute• Independence from CPU enclaves

7

FPGA Specialization in the Cloud

8

FREE: FPGA-Reinforced Execution Enclaves

• Open-Source framework for the secure execution of reconfigurable accelerators in the public cloud…• …without trusting privileged software• …running on commercial, unmodified FPGAs• …guaranteeing the confidentiality and integrity of FPGA code+data• Targeting both System-on-Chips and standalone FPGAs

9

Background: FPGAs in the Cloud• FPGAs: Field-Programmable Gate Arrays

• Custom accelerators on reconfigurable logic cells

• Development Cycle

10

FPGA

Host CPU

Client IP

Accel.

CSP Shell

CSP Bitstream

Shell

Accel.

Host Memory

FPGA Memory

Threat Model• Key Idea: Trust FPGA vendors, not CSP

HW/SW/Employees• Adversary can compromise:

• All off-chip communication and memory accesses• All software (incl. host OS/VMM)• The Shell• Co-tenants on FPGA logic

• We do not consider DoS attacks

11

FPGA

Host CPU

Shell

Accel.

Host Memory

FPGA Memory

Threat Model• FREE protects the integrity, confidentiality, and freshness of the

Accelerator and its data assuming trust in:• The FPGA manufacturer and physical integrity of the package• The client’s compilation tools and on-prem workstation• Attestation service

12

Manufacturer FPGA Chip On-premWorkstation

Verifier

FREE System Components• Modified Bootloader

• Hardware-based Root of Trust

• Security Kernel• Secure loading and remote attestation

• Shield• Isolated execution and sealed storage

13

Bootloader Security Kernel

Shield

Example Application: 23andMe

14

Africa

Asia

Europe

?

?

?

Alice’s DNA Classifier

Secure Compilation of Accelerator

Key Idea: Client compiles Accelerator + Shield on a secure workstation and encrypts the bitstream

15

Encrypted Accelerator

Classifier IP FREE Shield

Merges and Compiles

CSP

Sends to

AES_CEK

From Manufacture to Datacenter

16

Key Idea: Build a hardware root-of-trust• FPGA Manufacturer acts as root CA• Manufacturer provisions each FPGA w/ AESD

• Provides CSP with encrypted Bootloader• Bootloader contains {SKD, PKD}, signed by SKM

AESD

SKD

AESD

Manufactures and Provisions

Decrypts and Boots Certifies

Signs

Encrypted Bootloader

CertPK_D

{SKM, PKM}Secures

Manufacturer

Secure Boot

Key Idea: Extend Trust to Attestation Key/Security Kernel• FPGA Decrypts and runs Bootloader• Measures Security Kernel to obtain Hs

• Generates Attestation Key {SKatt,PKatt} from {Hs, SKD}• Signs PKatt with SKD

• Boots Security Kernel• Loads Shell onto FPGA

17

Bootloader

Security Kernel

Measured By

SKD

CertPK_D

Certifies

SKAtt

Generates

CertPK_Att

Certifies

Signs

Boots

Loads

CSP Shell

Remote Attestation

Key Idea: Prove authenticity of Security Kernel and boot the accelerator• 23andMe designates a trusted Verifier• Verifier challenges Security Kernel• Security Kernel sends back an attestation• Verifier checks attestation certificates• Verifier sends over Accelerator decryption key• Security Kernel loads Accelerator onto FPGA• Restrict access to programming/debug ports

18

Attests

CertPK_D CertPK_Att

Verifies

Attestation

Trusts

Secures

FPGA

Runs

Queries

Security Kernel Verifier

Designates

Decrypts

Accelerator

Overview: Shield ArchitectureKey Idea: Wraps Role logic to provide secure I/O• Shield Isolation

• Embedded Channel Encryption Key

• Shield-to-CSP Interface (Untrusted)• Runtime in host CPU• PCIe, DMA, DDR Controllers in Shell

• Customizable level of security• Memory Encryption and Integrity• Crypto Agility• Side-channel defenses (e.g. ORAM)

19

FPGAShell

Shield

Host Memory

FPGA Memory

Accel.

Trusted I/O

Untrusted I/O

CEK

Runtime

Overview: Trusted Shield-to-Accelerator Interface

• Control Registers• Read/Write commands – encrypted/authenticated with CEK

• DMA• Transfer data between host and FPGA memory• Restricted memory access via control registers

• FPGA Memory Controller• Confidentiality: AES encryption engine• Integrity: Integrity tree• Freshness: Counters• Side-Channels: ORAM

20

2121

FPGAShell

Shield

Host Memory

FPGA Memory

Accelerated, Secure Classification• 23andMe provides encrypted

commands/data to CSP• Runtime sends commands to Shield• Shield verifies command and copies data• Shield sets up memory integrity structures• Classifier runs, while Shield secures off-chip

memory accesses• Classifier finishes and encrypts result• Runtime copies and returns result

CEK

Runtime

EncCEK(Alice’s DNA)

EncCEK(”classify Alice’s DNA”)

Classifier

“classify Alice’s DNA”

Memory State

Alice’s DNA

EncCEK(Ancestry Results)

Shield

Demo Background: Cryptocurrency Mining

22

Block Header

SHA-256Nonce

< target?

Profit

FPGA

Golden Nonce

23

Insights and Asks• The Shell makes any I/O into the accelerator untrusted• Local attestation is an unsolved problem on FPGAs• Secure, on-prem compilation of accelerator is required• Manufacturer-support streamlines remote attestation

• Secure storage for asymmetric key• Hardware-based TRNG

24

Future Work• Open-Source Implementation of FREE• Evaluation on public cloud FPGAs (e.g. EC2 F1)• Security Kernel implementation on standalone FPGAs• FPGA-Specific optimizations for memory encryption

25

Conclusion

• We need hardware security• Current trusted computing models make a poor tradeoff between

performance and security• FREE enables secure, flexible, and accelerated remote computation on

FPGAs in the public cloud

26

27

Acknowledgements

Thanks!

28