SecureCloud

Joint EU-Brazil Research and Innovation ActionSECURE BIG DATA PROCESSING IN UNTRUSTED CLOUDS

https://www.securecloudproject.eu/

Description of programming model for new micro-servicesD3.4

Due date: 31 December 2018Submission date: 10 January 2019

Start date of project: 1 January 2016

Document type: DeliverableWork package: WP 3

Editor: Pierre-Louis Aublin (IMP)

Reviewer: Keiko Veronica Ono Fonseca (UTFPR)Charles B Prado (INMETRO)

Dissemination LevelPU Public

√CO Confidential, only for members of the consortium (including the Commission Services)CI Classified, as referred to in Commission Decision 2001/844/EC

SecureCloud has received funding from the European Union’s Horizon 2020 research and innovationprogramme and was supported by the Swiss State Secretariat for Education, Research and Innovation(SERI) under grant agreement No 690111.

Tasks related to this deliverable:Task No. Task description Partners involved○

T3.2 Set of reusable secure micro-services IMP∗, TUD, UniNET3.4 Templated programming model for secure micro-services IMP∗, TUD, UniNE

○This task list may not be equivalent to the list of partners contributing as authors to the deliverable∗Task leader

Contents

1 Introduction 2

2 Programming model for new microservices 32.1 TaLoS HTTP processing interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Enabling compartmentalization within an enclave using Intel MPX . . . . . . . . . . . . 3

2.2.1 Intel MPX technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2.2 Thread model and sample use-case . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2.3 Isolation Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Summary 9

1

1 IntroductionThe goal of the SecureCloud project is to enable the secure execution of big data applications withinpotentially malicious cloud environments. While the developed SecureCloud platform is generic, theproject is meant to be applicable to various use-cases. Modern distributed big data applications are usuallycomposed of microservices [12]. Hereby, each microservice contributes a specialised functionality to thebig data application. For a microservice-based big data application to be secure, it is essential that allindividual microservices, their communication, and their composition are secure [11]. The SecureCloudplatform thus provides solutions to develop, deploy, execute, compose, and manage secure big dataapplications from individual microservices.

Microservices include a wide range of requirements, among which confidentiality and integrityof the data and application logic are paramount. SecureCloud achieves confidentiality and integrityguarantees by leveraging novel trusted execution environments (TEEs) provided by modern commodityCPUs [4, 5, 2]. Nevertheless, it is necessary to ease the development cost and integration of microservicesin the SecureCloud platform. To address this challenge, the SecureCloud project proposes programmingmodels for new types of microservices.

These new programming models will enable developers of secure applications to customise existingmicroservices and facilitate the implementation of new microservices. This will enable the creation ofsecure services that are secure-by-design, without introducing vulnerabilities, e.g. by an incorrect usageof the framework API.

This deliverable describes the programming model for new microservices that makes them secure-by-design, which encompasses:

• the TaLoS [3] HTTP processing interface, which allows developers to build microservices thatmanipulate HTTP messages in a secure way (Section 2.1);

• the compartmentalization technique that allows the isolation of third-party plugins one from anotherin a secure Intel SGX enclave (Section 2.2).

2

2 Programming model for new microservicesBy leveraging the SecureCloud platform developers can produce new microservices that are secure bydesign. In this section we present two programming models: (i) the TaLoS HTTP processing interface,with which the developer can securely process HTTP messages in an enclave without leaking secrets tothe untrusted environments. This interface can for example be used to built an auditing service; and (ii) acompartmentalization technique that leverages Intel SGX enclaves and Intel MPX technology.

2.1 TaLoS HTTP processing interface

The TaLoS library, presented in D1.3, exposes an interface for custom message processing [7].This interface consists of a set of private functions, called by LibreSSL, and public functions, used

by the custom TLS processing module to register callbacks. These callbacks are called by the “private”functions. The public interface is the following:

• void tls processing register ssl read processing cb(void (*cb)(constSSL*, char*, unsigned int)): register the callback that will be called by the functionssl3 read bytes() in ssl/s3 pkt.c when data is read from the TLS connection socket;

• void tls processing register ssl write processing cb(void (*cb)(constSSL*, char*, unsigned int)): register the callback that will be called by he functiondo ssl3 write() in ssl/s3 pkt.c when data is read from the TLS connection socket;

• void tls processing register set ssl type cb(void (*cb)(const void*,const long)): register the callback that will be called by BIO int ctrl() when thecommand is BIO C SET FD. This callback is used for Squid in SSL proxy mode to differentiatethe connection between the client and the proxy from the connection between the proxy and theserver;

• void tls processing register new connection cb(void (*cb)(const SSL*)):register the callback that will be called from SSL new() in ssl/ssl lib.c when a new TLSconnection is created;

• void tls processing register free connection cb(void (*cb)(const SSL*)):register the callback that will be called from SSL free() in ssl/ssl lib.c when a TLSconnection is terminated.

Deliverable D1.3 presents, as an example, the pseudo-code for a service that uses this interface tohide a secret in a request to the untrusted web server.

By leveraging this interface the developer is able to create new microservices which will securelyanalyse and/or modify the application messages. For example, the LibSEAL library [1] builds on top ofthis interface to provide an auditing service to the SecureCloud platform.

2.2 Enabling compartmentalization within an enclave using IntelMPX

These days many services are designed to support rich third-party plugin ecosystems that a programmercan pick and choose from in order to create a custom service deployment. However, the security

3

Deliverable 3.4 Secure Big Data Processing in Untrusted Clouds

requirements of the core service may be different from those of the third party plugins. We consider sucha scenario where a programmer wishes to deploy a service in an SGX enclave, where the service consistsof separate components that need to communicate with each other, but also need to be protected fromeach other. Since all the code inside an enclave runs with the same privilege level, a vulnerability like abuffer overflow in one component allows an attacker to gain control of all the code and data handled by adifferent component.

As part of our programming model for enclave services, we provide a new isolation primitive insidethe enclave called a compartment whose memory is isolated from other compartments. We enforcesecurity policies between these different compartments inside an enclave and allow them to communicateand share data according to programmer specification. The memory isolation is enabled by a hardwaretechnology called Intel Memory Protection Extensions (MPX) that provides dedicated hardware registersand instructions to store and check bounds information. A compiler based source-to-source transformationadds the necessary instructions to the source code to load the bounds registers with the bounds informationfor different compartments and also to check whether these bounds are adhered to by each load/storeoperation.

2.2.1 Intel MPX technology

The Intel Memory Protection Extensions (MPX) technology introduced as part of the Skylakemicroarchitecture, is a set of extensions to the x86 instruction set architecture that provides hardwaresupport for memory safety in conjunction with the compiler. MPX allows developers to transparentlyretrofit legacy C/C++ applications with bounds checking. It introduced four 128-bit bounds registers tohold memory bounds information and a set of 7 new instructions that operate on these registers to loadand check bounds at runtime. It requires changes to the operating system in the form of a new exceptionhandler that allocates storage bounds as well as sending a signal to the application when bounds areviolated. At the compiler level, new transformation passes are added to insert MPX instructions to create,store and check bounds.

Consider the following snippet of pseudo-code:

1 i n t * a [ 1 0 ] / / Array o f p o i n t e r s t o o b j s2 t o t a l = 03 f o r ( i =0 ; i<M; i + + ) :4 a i = a + i / / P o i n t e r a r i t h m e t i c on a5 o b j p t r = l o a d a i / / P o i n t e r t o o b j a t a [ i ]6 v a l u e = l o a d p t r o b j p t r7 t o t a l += v a l u e / / Sum of t h e a r r a y

The program allocates an array a[10] with 10 pointers to some integers (Line 1). Next, it iteratesthrough the first M items of the array to calculate the sum of integers in the array (Lines 3-7). After MPXbounds check is applied this code transforms into the following:

1 i n t * a [ 1 0 ]2 t o t a l = 03 a b = bndmk a , a +79 / / S e t bounds f o r a4 f o r ( i =0 ; i<M; i + + ) :5 a i = a + i

4

Deliverable 3.4 Secure Big Data Processing in Untrusted Clouds

6 b n d c l a b , a i / / Lower bound check of a [ i ]7 bndcu a b , a i +7 / / Upper bound check of a [ i ]8 o b j p t r = l o a d a i9 v a l u e = l o a d p t r o b j p t r

10 t o t a l += v a l u e

First, the bounds for the array a[10] are created on line 3 (the array contains 10 pointers each 8 byteswide, hence the upper-bound offset of 79) using the bndmk instruction. Then in the loop, before thearray item access on Line 9, two MPX bounds checks are inserted to detect if a[i] overflows (Lines 6-7).Note that since the protected load reads an 8-byte pointer from memory, it is important to check ai+7against the upper bound (Line 7).

2.2.2 Thread model and sample use-case

The deployment scenario consists of disparate application components that are developed by differentstakeholders but must be deployed within an SGX enclave. Neither stakeholder is considered maliciousbut instead want to prevent bugs such as buffer overflows in one component from compromising othercomponents thereby exposing confidential data or violating data integrity.

Our sample application use-case is a web server that supports third party plugins that could performoperations on content that is being served, such as access control or content filtering to prevent attacks.The developers of a commercial third party module may wish to keep filtering rules and whitelistsconfidential, and therefore require that the web server and their own module run in isolated compartmentsand interact in limited ways.

Let us consider the nginx [10] web server which supports a rich marketplace of third party modules [9]such as traffic accounting, authentication, web application firewall, IP blocking and so on. We considerthe example naxsi [8] which is a web application firewall that analyses and filters HTTP content toprevent vulnerabilities such as SQL injection and cross site scripting.

The security policy when deploying naxsi may include the following requirements from the twostakeholders:

• naxsi source code and data such as firewall rules should not be visible to nginx.

• naxsi will need read and write access to the incoming http requests from nginx (i.e. access toobjects of the type ngx http request t) in order to analyse and block a request when needed.

• naxsi cannot communicate the HTTP request information to anyone other than the nginx moduleitself.

2.2.3 Isolation Architecture

Each component that needs to be isolated is placed inside a separate region of the address space known asa compartment. A compartment consists of the code along with a data-only stack, heap and globals. Thecode of each compartment is retrofitted with MPX bounds checks that ensure that the code loads andstores are limited to the bounds of the compartment.

In order to facilitate interaction between compartments, programmers can specify trusted entrypointscalled a callgates. Callgates have four functions:

5

Deliverable 3.4 Secure Big Data Processing in Untrusted Clouds

• the callgate checks if the security policy allows the calling compartment to invoke the targetfunction. If it is not permitted, the callgate returns without any further action;

• the callgate loads bounds registers with the bounds for the target compartment and reinstates thebounds registers before it returns;

• it checks and copies any arguments that are passed into a compartment;

• it transfers control to the requested function.

The callgates of all compartments are located in their own separate compartment, and the securitypolicy allows all compartments to make calls and pass arguments to the callgate compartment. Callgatesare part of the Trusted Computing Base (TCB) of the application and therefore must be vetted throughmanual or automated program analysis tools to ensure they are free of vulnerabilities. Control flowis protected by ensuring that any calls or jumps will target the current compartment or the callgatecompartment. Direct jumps can be checked statically, but indirect jumps need runtime checks.

Compartments can pass arguments by value that are copied by callgates, or the security policy maystipulate that compartments are placed contiguously in the address space with a shared region betweenthem that can be used to allocate memory that is shared between compartments. Complex pointerarguments that would otherwise need a deep copy within the callgate can instead be allocated within theshared compartment region to which both compartments are given read and write access.

The libc standard library also requires its own compartment with a callgate but memory allocationfunctions are duplicated for each compartment so that they allocate memory within the address space ofthe compartment.

For the nginx and naxsi usecase, the programmer provides the following specification.

1 compar tment {2 i d : C0 ,3 s r c : / tmp / nginx −x . x . xx / ,4 e n t r y p o i n t s :5 }6

7 compar tment {8 i d : C1 ,9 s r c : / tmp / n a x s i −x . xx / n a x s i s r c / ,

10 e n t r y p o i n t s : n g x h t t p d u m m y a c c e s s h a n d l e r11 }12

13 sha red − r e g i o n s {14 {15 members : ( C0 , C1 ) ,16 a c c e s s : ( C0 , r +w) , ( C1 , r +w)17 a l l o c a t e s : ( C0 , n g x h t t p r e q u e s t t , s t r u c t )18 }19 }20

6

Deliverable 3.4 Secure Big Data Processing in Untrusted Clouds

heap

globals

nginx

callgates

naxsi

libc data-only stack

heap

globals

code

Code (Initialization of libc and compartments)

nginx+naxsi shared

Enclave

(SGX ProtectedMemory)

Figure 2.1: The address space layout of sgx enclave after compartmentalization of nginx and naxsi

21 c a l l s {22 ( C0 , C1 )23 }

There are separate compartments for nginx and naxsi with an overlapping shared memory regionthat both compartments have read and write access to. naxsi registers a callback with nginx calledngx http request handler which is a single entrypoint into the naxsi compartment. Thengx http request t object is passed to this handler, and is a fairly complex data structure, sothe programmer can specify that nginx allocates this object in the shared region. In this example, it isfairly easy for the programmer to identify the component entrypoints. But this could also be automatedusing static program analysis.

Figure 2.1 shows the address space layout of the nginx and naxsi use-case after compartmentalizationbased on the following programmer specification. There are also compartments for glibc and the callgates– the programmer does not need to specify these.

2.2.4 Implementation

Given a programmer specification as shown in the nginx+naxsi case, we provide support for automatedcompartmentalization by means of an LLVM [6] transformation pass. The LLVM pass does the following:

• Adds memory bounds checks at each load, store and function call instruction.

7

Deliverable 3.4 Secure Big Data Processing in Untrusted Clouds

• Synthesizes callgates automatically that set the bounds for the corresponding compartment.

• Synthesizes the libc compartment and maps libc functions, which are memory related (e.g., mmap,malloc) to variants, which use the compartment-local heap.

• Changes all stack allocations to use the compartment-local stack; for control data (like returnaddresses) the global stack is used.

• Moves globals to the respective compartments.

At present, the LLVM transformation pass supports only two interacting compartments. We plan toextend this to support multiple interacting compartments by using a combination of static and dynamicprogram analyses. For each memory access operation, we might be able to statically determine whichcompartment that memory points to using pointer analysis. In such cases, the appropriate boundsinformation can be loaded into one of the bounds register prior to the access and the bounds checksinstructions will check this access. Where a memory access operation may target more than onecompartment, the LLVM pass will synthesize a more complex bounds check that also combines runtimetracking information.

8

3 SummaryIn this deliverable we have presented the novel programming models for new microservices designed inthe SecureCloud project. More precisely, we have presented (i) the TaLoS HTTP processing interface,which can be used by developers to arbitrarily process HTTP messages in a secure environment, to providenew services such as auditing or to prevent the exposure of sensitive data to the untrusted environments;and (ii) a new programming model for isolation of third-party plugins in Intel SGX enclaves. Thisprogramming model leverages the Intel MPX hardware technology to enforce memory isolation. Aspecial compiler instruments the microservice source code to add MPX instructions.

The TaLoS HTTP processing interface is used by the LibSEAL library, presented in Deliverable D1.3.The programming model for isolation is currently being implemented.

9

Bibliography[1] LibSEAL: Revealing Service Integrity Violations Using Trusted Execution, Porto, Portugal, 04/2018

2018. ACM.

[2] ARM Ltd. TrustZone.https://www.arm.com/products/security-on-arm/trustzone. Accessed May2017, 2017.

[3] P.-L. Aublin, F. Kelbert, D. O’Keeffe, D. Muthukumaran, C. Priebe, J. Lind, R. Krahn, C. Fetzer,D. Eyers, and P. Pietzuch. TaLoS: Secure and Transparent TLS Termination inside SGX Enclaves.Technical Report 2017/5, Imperial College London, Mar. 2017. Technical Report,https://www.doc.ic.ac.uk/research/technicalreports/2017/#5.

[4] V. Costan and S. Devadas. Intel SGX Explained. IACR Cryptology ePrint Archive, 2016.

[5] Intel. Intel Software Guard Extensions (Intel SGX) SDK.https://software.intel.com/sgx-sdk, 2016.

[6] C. Lattner and V. Adve. LLVM: A Compilation Framework for Lifelong Program Analysis &Transformation. In CGO, 2004.

[7] LSDS research group. TaLoS: Efficient TLS Termination Inside SGX Enclaves for ExistingApplications – Secure processing of TLS communications. https://github.com/lsds/TaLoS#secure-processing-of-tls-communications.Accessed December 2018, 2017.

[8] NBS system. NAXSI web application firewall for NGINX.https://github.com/nbs-system/naxsi. Accessed December 2018, 2018.

[9] NGINX. NGINX 3rd Party Modules.https://www.nginx.com/resources/wiki/modules/. Accessed December 2018,2018.

[10] W. Reese. Nginx: the High-Performance Web Server and Reverse Proxy. Linux Journal,2008(173):2, 2008.

[11] Y. Sun, S. Nanda, and T. Jaeger. Security-as-a-service for microservices-based cloud applications.In 2015 IEEE 7th International Conference on Cloud Computing Technology and Science(CloudCom), pages 50–57, Nov 2015.

[12] J. Thones. Microservices. IEEE Software, 32(1):116–116, Jan 2015.

10