The Pyramid Scheme: Oblivious RAM for Trusted Processors

Manuel Costa, Lawrence Esswood∗, Olga Ohrimenko, Felix Schuster, Sameer Wagh†

Microsoft Research

AbstractModern processors, e.g., Intel SGX, allow applications toisolate secret code and data in encrypted memory regionscalled enclaves. While encryption effectively hides thecontents of memory, the sequence of address referencesissued by the secret code leaks information. This is aserious problem because these leaks can easily break theconfidentiality guarantees of enclaves.

In this paper, we explore Oblivious RAM (ORAM) de-signs that prevent these information leaks under the con-straints of modern SGX processors. Most ORAMs area poor fit for these processors because they have highconstant overhead factors or require large private mem-ories, which are not available in these processors. Weaddress these limitations with a new hierarchical ORAMconstruction, the Pyramid ORAM, that is optimized to-wards online bandwidth cost and small blocks. It uses anew hashing scheme that circumvents the complexity ofprevious hierarchical schemes.

We present an efficient x64-optimized implementationof Pyramid ORAM that uses only the processor’s regis-ters as private memory. We compare Pyramid ORAMwith Circuit ORAM, a state-of-the-art tree-based ORAMscheme that also uses constant private memory. PyramidORAM has better online asymptotical complexity thanCircuit ORAM. Our implementation of Pyramid ORAMand Circuit ORAM validates this: as all hierarchicalschemes, Pyramid ORAM has high variance of access la-tencies; although latency can be high for some accesses,for typical configurations Pyramid ORAM provides ac-cess latencies that are 8X better than Circuit ORAM for99% of accesses. Although the best known hierarchi-cal ORAM has better asymptotical complexity, PyramidORAM has significantly lower constant overhead fac-tors, making it the preferred choice in practice.

∗Work done at Microsoft Research; affiliated with University ofCambridge, UK.

†Work done at Microsoft Research; affiliated with Princeton Uni-versity, USA.

1 Introduction

Intel SGX [21] allows applications to isolate secret codeand data in encrypted memory regions called enclaves.Enclaves have many interesting applications, for instancethey can protect cloud workloads [3, 5, 22, 46] in mali-cious or compromised environments, or guard secrets inclient devices [21].

While encryption effectively hides the contents ofmemory, the sequence of memory address references is-sued by enclave code leaks information. If the attackerhas physical access to a device, the addresses could beobtainable by attaching hardware probes to the mem-ory bus. In addition, a compromised operating systemor a malicious co-tenant in the cloud can obtain the ad-dresses through cache-probing and other side-channel at-tacks [31, 42, 44]. Such leaks effectively break the confi-dentiality promise of enclaves [7, 9, 18, 29, 35, 47, 54].

In this paper, we explore Oblivious RAM (ORAM)designs that prevent these information leaks under theconstraints of modern SGX processors. ORAMs com-pletely eliminate information leakage through memoryaddress traces, but most ORAMs are a poor fit for en-clave, because they have high constant overhead fac-tors [10, 39, 49] or require large private memories [50],which are not available in the enclave context. We ad-dress these limitations with a new hierarchical ORAM,the Pyramid ORAM. Our construction differs from pre-vious hierarchical schemes as it uses simpler buildingblocks that incur smaller constants in terms of complex-ity. We instantiate each level of the hierarchy with ourvariant of multilevel adaptive hashing [8], that we callZigzag hash table. The structure of a Zigzag hash tableallows us to avoid the kind of expensive oblivious sortingthat slowed down previous schemes [14,15,25]. We pro-pose a new probabilistic routing network as a buildingblock for constructing a Zigzag hash table obliviously.The routing network runs in n logn steps and does notbear “hidden constants”.

1

arX

iv:1

712.

0788

2v1

[cs

.CR

] 2

1 D

ec 2

017

Hierarchical ORAMs, despite being sometimes re-garded inferior to tree-based schemes due to complex re-build phases, have specific performance characteristicsthat make them the preferred choice for many applica-tions. Inherently, most accesses to a hierarchical ORAMare relatively cheap (i.e., when no or only few levels needto be rebuilt), while a few accesses are expensive (i.e.,when many levels need to be rebuilt). In contrast, intree-based schemes, all accesses cost the same. Since, inhierarchical ORAMs, the schedule for rebuilds is fixed,upcoming expensive accesses can be anticipated and thecorresponding rebuilds can be done earlier, e.g., whenthe system is idle, or concurrently [16, 37]. This way,high-latency accesses can be avoided. To reflect this, wereport online and amortized access costs. The formermeasures access time without considering the rebuild andthe latter measures the online cost plus the amortized re-build cost. Furthermore, hierarchical schemes can nat-urally support resizable arrays and do not require ad-ditional techniques compared to tree-based schemes (asalso noted in [34]).

We present an efficient x64-optimized implementationof Pyramid ORAM that uses only CPU registers as pri-vate memory. We compare our implementation with ourown implementation of Circuit ORAM [51], a state-of-the-art tree-based ORAM scheme that also uses constantmemory, on modern Intel Skylake processors. Althoughthe predecessor of Circuit ORAM, Path ORAM [50], hasbeen successfully used in the design of custom securehardware and memory controllers [11–13,30,43], CircuitORAM has better asymptotical complexity when usedwith small private memory.

Pyramid ORAM has better online asymptotical com-plexity and outperforms Circuit ORAM in practice inmany benchmarks, even without optimized schedulingof expensive shuffles. For typical configurations, Pyra-mid ORAM is at least 8x faster than Circuit ORAM for99% of accesses. Finally, although the best known hier-archical ORAM [25] has better asymptotical complexity,Pyramid ORAM has significantly lower constant over-head factors, making it the preferred choice in practice.

In summary, our key contributions are:

• A novel hierarchical ORAM, Pyramid ORAM, thatasymptotically dominates tree-based schemes whencomparing online overhead of an oblivious access;

• Efficient implementations of Pyramid ORAM andCircuit ORAM for Intel SGX;

• The first experimental results of a hierarchicalORAM in Intel SGX. Our results show that Pyra-mid ORAM outperforms Circuit ORAM on smalldatasets and has significantly better latency for 99%of accesses.

2 Preliminaries

Attacker Model Our model considers a trusted pro-cessor and a trusted client program running inside anenclave. We assume the trusted processor’s implemen-tation is correct and the attacker is unable to physicallyopen the processor package to extract secrets. The pro-cessor has a small amount of private trusted memory: theprocessor’s registers; we assume the attacker is unable tomonitor accesses to the registers. The processor uses anuntrusted external memory to store data; we also referto the provider of this external memory as the untrustedserver as this terminology is often used in ORAM liter-ature; in our implementation this is simply the untrustedRAM chips that store data. We assume the attacker canmonitor all accesses of the client to the data stored in theexternal memory. We assume the processor encrypts thedata before sending it to external memory (this is imple-mented in SGX processors [32]), but the addresses wherethe data is read from or written to are available in plain-text. We assume the processor’s caches are shared withprograms under the control of the attacker, for instancemalicious co-tenants running on the same CPU [44, 47]or a compromised operating system [54]. Hence, weassume that an attacker can observe client memory ac-cesses at cache-line granularity. This model covers sce-narios where SGX processors run in an untrusted cloudenvironment where they are shared amongst many ten-ants, the cloud operating system may be compromised,or the cloud administrator may be malicious. All sidechannels that are not based on memory address traces,including power analysis, accesses to disks and to thenetwork, as well as channels based on shared microarchi-tectural elements inside the processor other than caches,(e.g., the branch predictor [27]) are outside the scope ofthis paper.

Definitions An element e is an index-value pair wheree.key and e.val refer to its key and value, respectively.A dummy element is an element of the same size as areal element. Elements, real or dummy, stored outside ofthe client’s private memory are encrypted using seman-tically secure encryption and are re-encrypted wheneverthe client reads them to its private memory and writesback. Hence, encrypted dummy elements are indistin-guishable from real elements. Our construction relies onhash functions that are instantiated using pseudo-randomfunctions and modeled as truly random functions. Wesay that an event happens with negligible probability ifit happens with probability at most 1

2s , for a sufficientlylarge s, e.g., 80. An event happens with very high prob-ability (w.v.h.p.) if it happens with probability at least1− 1

2s .

2

Table 1: Asymptotical performance of ORAM schemes for N elements with private memory of constant size andelements of size D = Ω(logN). All logarithms use the base of 2.

SchemeBandwidth cost Server Space

Online Amortized Overhead

Binary Tree ORAM [10] (Trivial bucket) D(logN)2 +(logN)4 logNPath ORAM (SCORAM) [52] D logN +(logN)3 log logN 1Circuit ORAM [51] D logN +(logN)3 1

Hierarchical, GO [14] D(logN)2 D(logN)3 logNHierarchical, GM [15] D(logN) D(logN)2 1Hierarchical, Kushilevitz et al. [25] D(logN)2/ log logN 1

Pyramid ORAM D(logN)(log logN)2 D(logN)2(log logN)2 (log logN)2

Data-oblivious Algorithm To protect against theabove attacker, the client can run a data-oblivious coun-terpart of its original code. Informally, a data-obliviousalgorithm appears to access memory independent of thecontent of its input and output.

Let A be an algorithm that takes some input and pro-duces an output. Let ρ be the public information aboutthe input and the output (e.g., the size of the input).Given the above attacker model, we assume that all mem-ory addresses accessed by A are observable. We say thatA is data-oblivious, i.e., secure in the outlined setting, ifthere is a simulator SimA that takes only ρ and producesa trace of memory accesses that appears to be indistin-guishable from A ’s. That is, a trace of a data-obliviousalgorithm can be reconstructed using ρ .

Oblivious RAM ORAM is a data-obliviousRAM [14]. We will distinguish between virtualand real accesses. A virtual access is an access theclient intends to perform on RAM, that is, an access itwishes to make oblivious. The real access is an accessthat is made to the data structures implementing ORAMin physical memory, i.e., those that appear on the traceproduced by an algorithm.

Performance We assume that the client uses externalmemory (i.e., memory at the server) to store elements ofsize D (e.g., 64 bytes) and it can fit a constant numberof such elements in its private memory. Similar to otherwork on oblivious RAM [50, 51], we assume that D isof size Ω(logN), that is, it is sufficient to store a tag ofthe element. During each access the client can read aconstant number of elements into its private memory, de-crypt and compute on them, re-encrypt, and write themback. When analyzing the performance of executing analgorithm that operates on external memory, we are in-terested in counting the number of bits of the externalmemory the client must access. Similar to previous work

on ORAM [50,51], we call this measurement bandwidthcost.

For ORAM schemes, we distinguish between onlineand amortized bandwidth cost. The former measures thenumber of bits to obtain a requested element, while thelatter measures the amortized bandwidth for every ele-ment. This distinction is important especially for hierar-chical schemes that periodically stall to perform a reshuf-fle of the external memory before they can serve the nextrequest. However, such reshuffles happen after a certainnumber of requests are served and their cost can be amor-tized over preceding requests that caused the reshuffle.

In Table 1 we compare the overhead of PyramidORAM with other ORAM schemes that use constantprivate memory. Though Pyramid ORAM uses morespace at the server than tree-based schemes, it has bet-ter asymptotical online performance for blocks of sizeO((logN/ log logN)2). Pyramid ORAM is less efficientthan hierarchical schemes by Kushilevitz et al. [25] andGoodrich and Mitzenmacher [15]. However, as we ex-plain in Section 9, our scheme relies on primitives thatare more efficient in practice than those used in [15, 25].For example, we do not rely on the AKS sorting network.

3 Hierarchical ORAM (HORAM)

Our Pyramid ORAM scheme follows the same generalstructure as other hierarchical ORAM (HORAM) con-structions [14, 15, 17, 25, 53]. We outline this structurebelow and describe how we instantiate it using our ownprimitives, Probabilistic Routing Network (PRN) andZigzag Hash Table (ZHT), in the next sections.

Data Layout HORAM abstractly arranges the mem-ory at the server into O(logN) levels, where N is thenumber of elements one wishes to store remotely and ac-cess obliviously later. Though N does not have to bespecified a priori and the structure can grow to accom-

3

modate more elements, for simplicity, we assume N tobe a fixed power of 2.

The first two levels, L0 and L1, can store up to p ele-ments each, where p ≥ 2. The third level L2, can storeup to 2p elements and the ith level stores up to 2i−1 p.1

The last level, l, can fit all N elements (i.e., l = logN/p).Note that we only mentioned the number of real elementsthat each level can fit. In many HORAM constructions,each level also contains dummy elements that are usedfor security purposes. Hence, the size of each level is ofsome fixed size and is often larger than the number ofreal elements the level can fit. The data within each levelis stored encrypted (implicitly done by the hardware inthe case of SGX), hence, real and dummy elements areindistinguishable.

The first level L0 is implemented as a list. The searchover L0 is a simple scan. Each consequent level, Li≥1,is implemented as a hash table to ensure fast search time(e.g., a cuckoo hash table [15, 17, 25, 40] or a hash tablewith buckets [14]).

Setup During the setup, the client inserts N elementsinto the last level Ll and uploads the first l − 1 emptylevels and Ll to the server. As mentioned before, this isnot required and the client can populate the data structureby writing elements obliviously one at a time.

Access During a virtual access for an element withe.key, the client performs the following search procedureacross all levels. If level i is empty, it is skipped. If the el-ement was not found in levels 0, . . . , i−1, then the clientsearches for e.key in Li. If Li has the element, it is re-moved from Li, cached, and the client continues access-ing levels i+ 1, i+ 2, . . . but for dummy elements. If Lidoes not have the element, the client searches for e.keyin level i+1 and so on. After all levels are accessed, thecached e is appended to L0 (even if e was found in someearlier location in L0 from which it was removed).

Rebuild After p accesses, L0 contains up to p ele-ments, where some may be dummy if the same ele-ment was accessed more than once. This triggers therebuild phase: L0 is emptied by inserting its real ele-ments into L1. After the next p accesses, L0 becomesfull again. Since L1 is already full, real elements fromboth L0 and L1 are inserted into L2. In general, after ev-ery 2i p < N accesses levels 0,1, . . . , i are emptied intoLi+1. After N accesses levels 0,1, . . . , l− 1 are all full.In this case, a new table, L′l , of the same size as Ll , is

1The construction by Kushilevitz et al. [25] uses a slightly differentlayout from the one described here. In particular, it has less levelsand lower levels can hold logN× the size of the previous level insteadof 2×.

created. Elements from L0,L1, . . . ,Ll are inserted into L′land emptied, and Ll is replaced with L′l . The same pro-cess repeats for the consequent accesses. Since the re-build procedure is a deterministic function of the num-ber of accesses performed so far, it is easy to determinewhich levels need rebuilding for any given access.

The insertion of real elements from levels 0,1, . . . , i toi+ 1 has to be done carefully in order to hide the num-ber of real and dummy elements at each level. For exam-ple, knowing that there was only one real element revealsthat all past accesses were for the same element. Thedata-oblivious insertion (or rebuild) process depends onthe specific HORAM construction and the data structureused to instantiate each level. Many constructions relyon an oblivious sorting network [2, 4] for this.

Security A very high-level intuition behind the secu-rity of an HORAM relies on the following: (1) level iis searched for every element at most once during theaccesses between two sequential rebuilds of level i; (2)searches for the same element before and after a rebuildare uncorrelated; (3) the search at each level does not re-veal which element is being searched for.

Pyramid ORAM Overview HORAM’s performanceand security rely on two critical parts: the data structureused to instantiate levels L1, . . . ,Ll and the rebuild pro-cedure. In this paper we propose Pyramid ORAM, anHORAM instantiated with a novel data structure and arebuild phase tailored to it. In Section 5 we devise a newhash table, a Zigzag Hash Table, and use it to instanti-ate each level of an HORAM. To avoid using an expen-sive oblivious sorting network when rebuilding a ZigzagHash Table, in Section 5.2 we design a setup phase usinga Probabilistic Routing Network (Section 4). The com-bination of the two primitives and their small constantsresult in an HORAM that is more efficient in practicethan previous HORAM schemes.

4 Probabilistic Routing Network (PRN)

In this section, we describe a Probabilistic Routing Net-work that we use in the rebuild phase of our HORAMinstantiation. However, the network as described here,with its data-oblivious guarantees and efficiency, may beof an independent interest.

A PRN takes a table of n buckets with c slots ineach bucket. The bucket size is sufficiently small com-pared to n. For example, in Pyramid ORAM c is set toO(log logn). The input table has at most n real elementslabeled with indices 0,1, . . . ,n−1 denoting their destina-tion buckets. The labels do not have to be stored along-side the elements and can be determined using a hash

4

5

6

1

4

3

0

7

3

n

c

log n stages

7

5

1

4

0

3

6

3

7

5

6

0

3

4

1

3

0

1

3

4

5

6

7

3

Figure 1: Routing network for n = 8 elements whereeach element is indicated in the input table by its desti-nation position (e.g., the destination of the element in thefirst bucket is 5 and destinations of the two elements inthe second bucket are 6 and 7). The elements that couldnot be routed in the second stage (i.e., tagged false by thenetwork) are greyed out.

function, for example. There can be more than one el-ement in the input bucket and, since a hash function isnot a permutation function, two or more elements maybe assigned to the same destination bucket.

A PRN’s goal is to route a large subset of elements tothe correct output buckets. In the output, elements in thatsubset are associated with the tag true, others with false.Given the destination labels, the network’s output can beused for efficient element lookups. Such a lookup is suc-cessful, if an element was routed to its correct destination(i.e., has tag true).

For the security purposes of HORAM, we wish to de-sign a network that is data-oblivious in the followingsense. The adversary that observes the routing as wellas element accesses to the output learns nothing aboutthe placement of the elements in the output as long as itdoes not learn whether accesses are successful or not.

Algorithm We now describe a routing network withthe above properties. The routing network consists oflogn stages where the input at each stage, except the firststage, is the output of the previous stage. The output ofthe last stage is the output of the network. Besides thedestination bucket indices, every element also carries aBoolean tag that is set to true for every real element inthe original input. (However, depending on the use case,input elements may be tagged false, for example, if theyare dummy elements.) Once the tag is set to false, it doesnot change throughout the network. Crucially, only ele-ments that are in their correct destination buckets in theoutput are tagged true.

At each stage i, for 1 ≤ i ≤ logn, we perform a re-

partition operation between each pair of buckets with in-dices that differ only by the bit in position i (starting fromthe lowest bit). (Note that here indices correspond to in-dices of the input buckets, not the indices of the desti-nations of the elements inside.) For example, for n = 8,when i= 1 we re-partition the following pairs of buckets:(000,001),(010,011),(100,101) and (110,111) andwhen i = 3, buckets: (000,100),(001,101),(010,110)and (011,111). The re-partition operation itself movesitems between the two buckets (number them one andtwo) and works as follows: For elements with tag true,we consider the ith bit of their bucket index. If this bit isset to 0, the element goes to the first bucket, and to thesecond bucket, otherwise. As there may be more than celements assigned the same bit at position i, the networkchooses c of them at random, and retags the extra/spilledelements with false. All elements tagged false, includ-ing those that arrived with the false tag from the previ-ous stage, are then allowed to go in either bucket (notethat there is always space for all elements). The effect ofthe network is that at stage i, every element still taggedtrue will have the first i bits match that of the destina-tion bucket. Therefore, after logn stages elements willeither be tagged false, or be in the correct bucket. (Anillustration of the routing network for n = 8 is given inFigure 1.)

Bucket Partitioning We now describe how to performthe re-partition of two buckets. If the client memoryis 2c then the re-partition is trivial. Otherwise, we useBatcher’s sorting network [4] as follows. The two buck-ets are treated as one contiguous range of length 2c thatneeds to be sorted ordering the elements as follows: (1)elements with bit 0 at the ith position, (2) empty elementsand elements tagged false, and (3) elements with bit 1 atthe ith position. Sorting will therefore cause a partition,with empty and false-tagged elements in the middle, andother elements in their correct buckets.

Analysis In this section we summarize the perfor-mance and security properties of the network.

Theorem 1. The probabilistic routing network operatesin (n/2) logn×Sort(2c) steps, where Sort(c) is time tosort c elements obliviously.

The theorem follows trivially: the network has lognstages where each stage consists of repartitioning of n/2pairs of buckets.

Corollary 1. The probabilistic routing network oper-ates in O(c(logc)2n logn) steps using constant privatememory and Batcher’s sorting network for repartition-ing bucket pairs.

5

The PRN is more efficient than Batcher’s sorting net-work, that runs in O(n(logn)2) steps, at the expense ofnot being able to route all the elements. Though the net-work cannot produce all n! permutations, it gives us im-portant security properties. As long as the adversary doesnot learn which output elements are tagged true and falseit does not learn the position of the elements in the out-put. Hence, when using the network in the ORAM set-ting we will need to ensure that the adversary does notlearn whether an element reached its destination or not.We summarize this property below.

Theorem 2. The probabilistic routing network andlookup accesses to its output are data-oblivious as longas the adversary does not learn whether lookups weresuccessful or not.

Proof. The accesses made by the routing network to theinput are independent of the input as they are determin-istic and the re-partition operation is either done in pri-vate memory or using a data-oblivious sorting primitive.Hence, we can build the simulator Simroute that, givena table of dummy elements, performs accesses accord-ing to the network. The adversary cannot distinguishSimroute from the real routing even if it knows the lo-cation of the elements in the input.

During the search, the adversary is allowed to requestlookups for the elements from the original input and ob-serve the accessed buckets. For data-obliviousness, thelookup simulator is notified about the lookup but not theelement being searched. The simulator then accesses arandom bucket of the output generated by Simroute.

The accesses of the simulator and the real algorithmdiverge only in the lookup procedure. The accessesgenerated by the simulator are from a random function,while real lookups access elements according to a hashfunction used to route the elements. Since the adver-sary is not notified whether a lookup was successful ornot, it cannot distinguish whether simulator’s functioncould have been used to successfully allocate the sameelements as the real routing network. Hence, the distri-bution of lookup accesses by the simulator is indistin-guishable from the real routing network.

We defer the analysis of the number of elements thatare routed to their output bucket to the next section asit depends on how elements are distributed across inputbuckets. For example, we assume random allocation ofthe elements in the input in Pyramid ORAM.

5 Zigzag Hash Table (ZHT)

In this section we describe a data structure, called ZigzagHash Table (ZHT), that we use to instantiate the levels of

2

1

4

3

0 5 6

7

n

c

k

H2 H3H1

Figure 2: A Zigzag hash table for n = 8 elements withk = 3 tables where each table has 8 buckets of size c = 2.Elements 0–7 were inserted in order, with hash functionsdefined as: h1(1) = 4, h1(2) = h1(3) = h1(5) = h1(6) =h1(7) = 1, h1(4) = h1(0) = 5, h2(5) = h2(6) = h2(7) = 5and h3(7) = 3. Zigzag path of element 7 is defined asp7 = 1,5,3.

our hierarchical ORAM. A ZHT is a variation of mul-tilevel adaptive hashing [8] but instantiated with tablesof fixed size and buckets of size c (see Section 9 for adetailed comparison). We begin with the non-obliviousversion of a ZHT and then explain how to perform searchand rebuild obliviously.

A ZHT is a probabilistic hash table that can store upto n key-value elements with very high probability. Werefer to n as the capacity of a ZHT, that is, the largestnumber of elements it can store. ZHT consists of k ta-bles, H1,H2, . . . ,Hk. Each table contains n buckets, witheach bucket containing at most c elements. Each tableis associated with a distinct hash function hi that mapselement keys to buckets. The hash functions are fixedduring the setup, before the elements arrive.

An element e is associated with an (abstract) zigzagpath defined by the series of bucket locations in each ta-ble: h1(e.key), h2(e.key), . . . ,hk(e.key). Elements areinserted into the table by accessing the buckets along thepath, placing the element in the first empty slot. If thepath is full, the insertion procedure has failed. When

Table 2: Notation.

N Number of elements in ORAMn Capacity of a Zigzag Hash Table (ZHT)k Number of tables in a ZHTc Bucket size of tables H1, . . . ,Hk in a ZHT

6

referring to the insertion of several elements, we usethrow(A,H1, H2, . . . ,Hk) that takes elements in the ar-ray A and inserts them according to their zigzag paths.Once inserted, elements are found by scanning the pathand comparing indices of elements in the buckets untilthe element is found. (See Figure 2 for an illustration ofa ZHT.)

Definition 5.1 (Valid zigzag paths). Let p1,p2, . . . ,pn ben zigzag paths. If every element e j can be inserted into aZHT according to locations in the path p j, then the pathsare valid.

If n elements are successfully inserted in a ZHT, wesay that the zigzag paths associated with the elementsare valid. We will use ZHTn,k,c to denote a ZHT withparameters k and c sufficient to avoid overflow with highprobability when inserting n elements. We summarizethe parameters and the performance of a ZHT below.

Theorem 3 (ZHT Performance). Insertion and search ofan element in a ZHTn,k,c takes O(kc) accesses. Insertionof n elements fails with negligible probability when k =O(log logn) and c = O(log logn).

Proof. (Sketch.) The analysis follows Theorem 5.2from [1], adopted for our scenario. We note that the anal-ysis in [1] is for re-throwing elements into the same tableand not into an empty table, as is the case in our datastructure.

We refer to an element of a table as overflowed if thebucket it was allocated to already had c elements in it.

For each table Hi, we are interested in measuring ini,the number of incoming elements, and outi, the num-ber of outgoing elements. The former comprises the ele-ments that did not fit in tables 1,2, . . . , i−1. The numberof outgoing elements comprises elements that did not fitinto table i when throwing ini. Except for the first phase,the number of incoming elements into Hi is determinedby the number of outgoing elements of Hi−1. Hence, todetermine k we need to bound the number of outgoingelements at each level.

We first show that one round of throw is required tobound the number of overflowed elements to be at mostn/(2e) (Lemma 9). We then show that as long as thenumber of elements arriving at the table is less thann/(2e) only O(log logn) tables are required to fit all nelements (Lemma 11).

We note that buckets that hold at most one element(i.e., c = 1) are sufficient to bound the number of ta-bles, k, to be loglogn+O(1) [1, 8]. However, as willbe explained in the following sections, in order to in-sert elements into a ZHT obliviously we require c to beO(log logn).

As described so far, the insertion procedure of aZHT does not provide security properties one desires forORAM (e.g., observing insertion of n elements and asearch for element e reveals when e was inserted). Inthe next section, we make small alternations to the inser-tion and search procedures of a ZHT and summarize thesecurity properties a ZHT already has. In particular, wewill show that one can obliviously search for elements ina ZHT, as long as it was constructed privately. Then, inSection 5.2 we show how to insert n elements into a ZHTobliviously using small private memory.

5.1 Data-oblivious properties of ZHT

We make the following minor changes to a ZHT andits insertion and search procedures. We fill all emptyslots of a ZHT with dummy elements. Recall that inthe ORAM setting the data structure is encrypted anddummy elements are indistinguishable from real. Then,when inserting an element, instead of stopping once thefirst empty slot is found, we access the remaining bucketson the path, pretending to write to all of them. Similarly,during the search, we do not stop when the element isfound and access all buckets on its zigzag path. Hence,anyone observing accesses of an insert (or search) ob-serve the zigzag path but do not know where on the paththe element was inserted (or found). We define dummycounterparts of insertion and search procedures that sim-ply access a random bucket in each table of a ZHT, per-forming fake writes and reads.

Given the above changes, the search for n elements isdata-oblivious as long as every element is searched foronly once and assuming that an adversary does not ob-serve how the table was built.

Lemma 1 (ZHT Data-Oblivious Search). Searchingfor n distinct elements in a ZHTn,k,c is data-oblivious.

Proof. We can build a simulator that given n,k and c pro-duces a sequence of accesses to its tables indistinguish-able from locations accessed by a ZHT algorithm.

Consider an algorithm (with large private memory)that takes n elements and inserts them one by one intoan empty ZHT. If it cannot insert all n elements, it fails.Otherwise, during a search request for an element key,e.key, it accesses the locations that correspond to thezigzag path defined by e.key and the hash functions ofthe ZHT.

The simulator, Simsearch, sets up k tables with n buck-ets of size c each. For every search, the simulator per-forms a dummy search. We say that a simulator fails ifafter simulating searches for n elements, it has producedaccesses to ZHT tables that do not form n valid zigzagpaths.

7

In each case the adversary chooses n original elements.It then requests a search for one of the n elements thatit has not requested before. A ZHT algorithm is giventhe key of the element, while the simulator is invokedto perform a dummy search. In both cases, the adver-sary is given the locations accessed during the search. Ifneither the simulator nor the ZHT algorithm fail, the ac-cessed locations are indistinguishable as long as ZHT’shash functions are indistinguishable from truly randomfunctions. The failure cases of the algorithms allow theadversary to distinguish the two as failures happen at dif-ferent stages of the indistinguishability game. However,both failures happen with negligible probability due tothe success rate of a ZHTn,k,c.

Lemma 2. Searching for n distinct elements in aZHTn,k,c is data-oblivious even if the adversary knowsthe order of insertions.

Proof. The proof follows the proof of Lemma 1, ex-cept now the adversary controls the order in which theZHT algorithm inserts the elements. As before, we useSimsearch that is not given the actual elements and the ad-versary does not observe accesses of the insertion phase.

Let pi be the zigzag path of element ei. Notice that khash functions that determine zigzag paths are chosen be-fore the elements are inserted into an empty ZHT. Hence,the order in which ei is inserted in the ZHT influenceswhere on the path ei will appear (e.g., if ei is insertedfirst, then it will be placed into H1, on the other hand, ifit is inserted last it may not find a spot in H1 and couldeven appear in Hk) but not the path it is on. Since theadversary does not learn where on the path an elementwas found, the rest of the proof is similar to the proof ofLemma 1

We capture another property of a ZHT that will be use-ful when used inside of an ORAM: locations accessedduring the insertion appear to be independent of the ele-ments being thrown. Moreover, as we show in the nextlemma, the adversary cannot distinguish whether thrownelements were real or dummy.

Lemma 3. The algorithm throw(A,H1,H2, . . . ,Hk) isdata-oblivious where H1,H2, . . . ,Hk are hash tables ofZHTn,k,c and the size of A is at most n.

Proof. The adversary chooses up to n distinct elementsand requests throw. The algorithm fails if it cannot in-sert all elements in k tables. The simulator Simthrow isgiven A that contains only dummies. It accesses a ran-dom bucket in each of the k tables for every element in A.

The adversary can distinguish the two algorithms onlywhen throw fails or when Simthrow generates invalidzigzag paths for A. According to Theorem 3, this hap-pens with negligible probability.

Lemma 4. Consider throw(A,H1,H2, . . . ,Hk) where Acontains nreal ≤ n real elements and ndummy dummy el-ements, where a real element is inserted as usual whileon a dummy element each table is accessed at a randomindex. The algorithm throw remains data-oblivious.

Proof. We define an adversary that chooses a permuta-tion of nreal real elements and ndummy dummy elements,that is, it knows the locations of the real elements. Webuild a simulator that is given an empty ZHTn,k,c table.The simulator accesses a random bucket in each table ofthe ZHT on every element in A.

The adversary can distinguish between the simulatorand the throw algorithm if (1) the algorithm fails, whichhappens when the algorithm cannot insert all real ele-ments successfully or (2) the simulator generates invalidzigzag paths when performing a fake insertion for theinput locations that correspond to real elements (recallthat the simulator does not know the locations of real ele-ments). Both events correspond to the failure of the ZHTconstruction, which happens with negligible probability(Theorem 3).

5.2 Data-oblivious ZHT SetupA Zigzag hash table allows one to search elements obliv-iously if insertion of these elements was done privately.Since the private memory of the client is much smallerthan n in our scenario, we develop an algorithm thatcan build a ZHT relying on small memory and access-ing the server memory in an oblivious manner. That is,the server does not learn the placement of the elementsin the final ZHT.

The algorithm starts by placing elements in the tablesH1,H2, . . . ,Hk using throw procedure from Section 5 ac-cording to random hash functions. The algorithm thenproceeds in k phases. In the first phase, H1 is given asan input to the probabilistic routing network where thedestination bucket of every element in H1 is determinedby h1. The result is a hash table with some fraction ofthe elements in the right buckets and some overflown el-ements (tagged as false). The algorithm then pretends tothrow all elements of H1 into tables H2,H3, . . . ,Hk usingyet another set of random functions. However, only over-flown elements, that is elements that were not routed totheir correct bucket in H1, are actually inserted into ta-bles H2,H3, . . . ,Hk. For empty entries and elements thatare labeled true in H1, the algorithm accesses randomlocations in H2,H3, . . . ,Hk and only pretends to write tothem. In the end of this step every element that did notfit into H1 during the routing network appears in one ofH2,H3, . . . ,Hk at a random location. In the second phase,H2 goes through the network where destination bucketsare determined using h2, and so on. In general, every Higoes through the network using hi, then all elements of Hi

8

are thrown into tables Hi+1,Hi+2, . . . ,Hk, where only el-ements that were not routed correctly are inserted. (SeeFigure 12 in Appendix for an illustration of the oblivioussetup of a ZHT.)

The algorithm fails if an element cannot find a spot inany of the k tables. This happens either during one of thethrows of the elements or during the final phase wherenot all elements of Hk can be routed according to hk.

Performance We now analyze the parameters and theperformance of the ZHT setup and then describe the se-curity properties.

Theorem 4. Insertion of n elements into a ZHTn,k,c asdescribed in Section 5.2 succeeds with very high proba-bility if k = O(log logn) and c = O(log logn).

Proof. (Sketch.) The analysis is similar to the analy-sis in Theorem 3 with the difference that the numberof elements going into level Hi+1 is higher due to addi-tional overflows that happen during routing. In Lemma 8,we show that if n is the number of input elements inhash table Hi, then the number of overflows from rout-ing is bounded by logn× the number of overflows of athrow into a single hash table. Then, following simi-lar proof arguments as Theorem 3, we use Lemmas 10and 12 to bound the spill at each phase and show thatif k = O(log logn), then the insertion of n elements suc-ceeds w.v.h.p.

Theorem 5 (Oblivious ZHT Setup Performance). Ifoblivious ZHT insertion of n elements does not fail, ittakes O(nkc((logc)2 logn+ kc)) steps.

Proof. The initial throw takes nkc and n(k−1)c2, n(k−2)c2, . . ., nc2 steps for the subsequent throws. The com-plexity of applying a routing network to a single tableis O(c(logc)2n logn). Since the algorithm performs krouting networks, the total cost is O(kc(logc)2n logn+n(kc)2) = O(nkc((logc)2 logn+ kc)).

When used in Pyramid ORAM, a ZHT is constructedfrom all elements in previous levels of the hierarchy. Thelevels include real elements as well as many dummyelements. Hence, the initial throw of the setup algo-rithm may contain additional dummy elements. Throw-ing dummy elements does not change the failure proba-bility of the setup. However, the initial throw becomesmore expensive than that in Theorem 5 as summarizedbelow.

Corollary 2 (Oblivious ZHT Setup Performance).Oblivious ZHT insertion of n real elements and (α−1)ndummy elements takes O(nkc((logc)2 logn+αkc)), forα ≥ 1.

Security We show that an adversary that observes theZHT setup does not learn the final placement of the ele-ments across the tables.

Theorem 6. The insertion procedure of ZHTn,k,c de-scribed in Section 5.2 is data-oblivious.

Proof. An adversary is allowed to interact with the al-gorithm and a simulator as follows: choose up to n ele-ments and request the oblivious setup of ZHTn,k,c, thenrequest a search of at most n elements (real or dummy)on ZHTn,k,c. It then repeats the experiment either for thesame set of n elements or a different one.

As before, the real algorithm fails if it cannot buildZHTn,k,c. The simulator Simbuild is given an array A ofn dummy elements and parameters k and c. It sets upZHTn,k,c and runs Simthrow(A,H1, H2, . . . ,Hk). It thencalls Simroute from Theorem 2 on H1. After that thethrow of the spilled elements is imitated via Simthrow(H1,H2, H3, . . . ,Hk) from Lemma 4. It then again callsSimroute but on its own version of H2. Once finished, onevery search request from the adversary, Simsearch, that isnot aware of the actions of Simthrow, is called.

If the algorithm does not fail during the insertion phaseand the simulator produces valid zigzag paths duringSimthrow and Simsearch, then the adversary obtains onlytraces that are produced either by a hash function (i.e., apseudo-random function) or by a random function. Weshowed that ZHTthrow and Simthrow fail with negligibleprobability. The real search and Simsearch are also indis-tinguishable even if the adversary controls the order ofthe inserted elements as long as the adversary does notlearn element’s position in its zigzag path. Since thereis Simroute which simulates route, the adversary does notlearn element positions after routing. Since every ele-ment participates in the throw, we showed in Lemma 4that the adversary does not learn which elements aredummy. Hence, the adversary does not learn positionsof the elements across the tables and search can be sim-ulated successfully.

6 Pyramid ORAM

We are now ready to present Pyramid ORAM by instanti-ating the hierarchical construction in Section 3 with data-oblivious Zigzag hash tables.

Data Layout. Level L0 is a list as before while everylevel Li, 1 ≤ i ≤ l, is a Zigzag hash table with capacity2i−1 p and ki,ci chosen appropriately for Theorem 3 tohold. We denote the bucket size and table number usedfor the last level with c and k. That is, Ll is a ZHTN,k,c.

Setup. The setup is as described in Section 3, i.e., el-ements can be placed either in Ll or inserted obliviouslyone by one causing the initially empty ORAM to grow.

9

Access. The access to each level i is an oblivious ZHTsearch, if the element was not found in previous levels,or a dummy ZHT access.

Rebuild. The rebuild phase involves calling the data-oblivious ZHT setup as described in Section 5.2. In par-ticular, at level i, a new ZHT for 2i−1 p elements is setup. Then, all elements from previous levels, includingdummy, participate in the initial throw.

We first determine the amortized rebuild cost of level ifor an HORAM scheme and then analyze the overall per-formance of Pyramid ORAM.

Lemma 5. If the rebuild of level i≥ 1 takes O(2i−1 pCi)steps, for Ci > 0, then the amortized cost of the rebuildoperation for every access to ORAM is ∑

logN/pi=1 O(Ci).

Proof. Since the rebuild of level i happens every 2i−1 paccesses, the amortized cost of rebuilding level i is O(Ci).Each access eventually causes a rebuild of every level,that is, logN/p rebuilds of different costs. Hence, totalamortized cost is:

logN/p

∑i=1

O(Ci).

Theorem 7 (Performance). Pyramid ORAM incursO(kc logN) online and O(kc(logN)2) amortized band-width overheads, O(kc) space overhead, and succeedswith very high probability.

Proof. The complexity of a lookup in a ZHT is kc. Thecomplexity of a lookup in the first level is p. There arelogN/p levels. So the cost of a lookup is p+kc logN/p.Since p is usually set to logN, the access time betweentable rebuilds, that is the online bandwidth overhead, isO(kc logN).

The rebuild of level i takesO(2i−1 pkc((logc)2 log(2i−1 p) + kc)) steps sinceα = kc (see Corollary 2) and k ≥ ki and c ≥ ci, forevery i. Setting Ci to kc((logc)2 log(2i−1 p)+ kc) and pto be at least 2 in Lemma 5, we obtain:

logN/p

∑i=1

kc((logc)2 log(2i−1 p)+ kc)

= kclogN/p

∑i=1

((logc)2 log(2i−1 p)+ kc

)= kc

((logc)2

logN/p

∑i=1

(i−1+ log p)+ kc logN/p

)= kc

((logc)2(logN)2 + kc logN

)= O(kc(logN)2)

Pyramid ORAM fails when the rebuild of some level ifails, that is, the number of tables allocated for level iwas not sufficient to accommodate all real elements fromthe levels above. Since, the number of real elements intables at levels 0,1, . . . , i− 1 is at most the capacity ofthe ZHT at level i, ZHT fails with negligible probabilityas per analysis in Theorem 4. Hence, the overall failureprobability of Pyramid ORAM can be bounded using aunion bound.

Theorem 8 (Security). Pyramid ORAM is data-oblivious with very high probability.

Proof. The adversary chooses N elements of its choiceand requests oblivious insertion in Ll . It then makesa polynomial number of requests to elements in theORAM. We use Simbuild from Theorem 6 and Simsearch

from Lemma 1 for every level except L0, which simulatesnaive scanning.

Between two rebuilds the search operation at eachlevel is performed to distinct elements and every tableuses a new set of hash functions after its rebuild. Thehash functions are also different across the levels. Hence,Simsearch can simulate the search at every level indepen-dently.

Simbuild for level i takes as input the number of dummyelements proportional to the total number of elements,real and dummy, in L0,L1, . . . ,Li−1. We showed that wecan construct Simbuild even if the adversary knows theoriginal elements inserted in the table. Hence, Simbuild

succeeds when the adversary controls the elements in-serted in the ORAM as well as the access sequence.

7 Implementation

We implemented Pyramid ORAM and CircuitORAM [51] in C++ with oblivious primitives writ-ten in x64 assembly. As ORAM block size, we use thex64 architecture’s cache-line width of 64 bytes. Giventhat our assumed attacker is able to observe cache-linegranular memory accesses, this block size facilitates theimplementation of data-oblivious primitives. A blockcomprises 56 bytes of data, the corresponding 4-byteindex, and a 4-byte tag, which indicates the element’sstate (see Section 4).

To be data-oblivious, our implementations are care-fully crafted to be free of attacker-observable secret-dependent data or code accesses. We avoid secret-dependent branches on the assembly level either by em-ploying conditional instructions or by ensuring that allbranch’s possible targets lie within the same 64-bytecache line. For data, we use registers as private mem-ory and align data in memory with respect to cache-lineboundaries. Similar to previous work [36, 41], we create

10

a library of simple data-oblivious primitives using condi-tional move instructions and CPU registers. These allowus, for example, to obliviously compare and swap values.Hash functions used in Pyramid ORAM are instantiatedwith our SIMD implementation of the Speck cipher.2

7.1 Pyramid ORAM

For bucket sizes of c > 4, our implementation of Pyra-mid ORAM uses a Batcher’s sorting network for the re-partitioning of two buckets within our probabilistic rout-ing network described in Section 4.

For c = 4, we employ a custom re-partition algorithmthat is optimized for the x64 platform and leveragesSIMD operations on 256-bit AVX2 registers. The im-plementation is outlined in the following. For most op-erations, we use AVX2 registers as vectors of four 32-bitcomponents. To re-partition two buckets A and B, wemaintain the tag vectors tA and tB and position vectorspA and pB. Initially, we load the four tags of bucket Ainto tA and those of bucket B into tB. A tag indicates anelement’s state. Empty elements and elements that werespilled in the previous stages of the network are assigned-1; elements that have not been spilled and need to berouted to either bucket A or B are assigned 0 and 1, re-spectively.

pA and pB map element positions in the input bucketsto element positions in the output buckets. We refer to theelements in the input buckets by indices. The elementsin input bucket A are assigned the indices 1 . . .4, those ininput bucket B the indices 5 . . .8. Correspondingly, weinitially set pA = (1,2,3,4) and pB = (5,6,7,8).

We then sort the tags by performing four obliviousSIMD compare-and-swap operations between tA and tB.Each such operation comprises four 1-to-1 comparisons.(Hence, a total of 16 1-to-1 compare-swap-operations areperformed.) Every time elements are swapped betweentA and tB, we also swap the corresponding elements inpA and pB. In the end, tA and tB are sorted and pA andpB contain the corresponding mapping of input to outputpositions. Finally, we load the two input buckets entirelyinto the 16 available AVX2 registers (which provide pre-cisely the required amount of private memory) and obliv-iously write their elements to the output buckets accord-ing to pA and pB.

Instantiation In our implementation, the ORAM’sfirst level holds 1024 cache-lines, because for smaller ar-ray sizes scanning performs better. All other levels usebuckets of size c = 4 and the number of tables is set todlog loge of the capacity of the level. We empirically ver-ified parameters k = c = 4 for tables of size 211 and 215

2https://en.wikipedia.org/wiki/Speck_(cipher)

when re-throwing elements 223 and 220 times, respec-tively. In both cases, we have not observed spills intothe third table (in any of the levels), hence, the last twotables of every level were always empty.

7.2 Circuit ORAM

For comparison, we implemented a recursive version ofCircuit ORAM [51] with a deterministic eviction strat-egy. Circuit ORAM proposes a novel protocol to imple-ment the eviction of tree-based ORAMs using a constantamount of private memory. This makes Circuit ORAMthe state-of-the-art tree-based ORAM protocol with op-timal circuit complexity for O(1) client memory. Similarto other tree-based schemes Circuit ORAM stores ele-ments in a binary tree and maintains a mapping betweenelement indices and their position in the tree in a recur-sive position map. We refer the reader to [51] for thedetails of Circuit ORAM.

We implemented Circuit ORAM from scratch usingour library of primitives, as existing implementations3

were built for simulation purposes and only concern withhigh-level obliviousness. This makes these implementa-tions not fully oblivious and vulnerable to low-level side-channel attacks based on control flow. Similar to Pyra-mid ORAM, we use a cache line as a block for CircuitORAM. This favors Circuit ORAM since up to 14 in-dices can be packed in each block, requiring only log14 Nlevels of recursion to store the position map. Hence, thebandwidth cost of our implementation of Circuit ORAMis only 64 bytes × O(log14 N logN). Similar to PyramidORAM, scanning is faster than accessing binary trees forsmall datasets. To this end, we stop the recursion whenthe position map is reduced to 512 indices or less andstore these elements in an array that is scanned on everyaccess. We chose 512 since it gave the best performance.

Instantiation We use a bucket size of 3 with two stashparameters: 12 and 20, that is every level of recursionuses the same stash size since these parameters are inde-pendent of the size of a tree, and a bucket size of 4 andstash size 12. We note that buckets of size 2 were notsufficient in our experiments to avoid overflows. In thefigures we denote the three settings with Circuit-3-12,Circuit-3-20 and Circuit-4-12.

8 Evaluation

We experimentally evaluate the performance of PyramidORAM in comparison to Circuit ORAM and naive scan-ning when accessing arrays of different sizes, when used

3https://github.com/wangxiao1254/oram_simulator

11

0

50

100

150

2^11 2^12 2^13 2^14

CP

U c

ycle

s

Tho

usa

nd

s

Number of elements

Pyramid Circuit-3-12 Circuit-3-20 Naïve

Figure 3: Average overhead of obliviously reading ele-ments from an array of 56-byte elements. Array sizesvary from 2048 to 16384 elements (114KB–917KB).

in two machine learning applications, and when run in-side Intel SGX enclaves. The key insight is that, forthe given datasets, Pyramid ORAM considerably outper-forms Circuit ORAM (orders of magnitude) in terms ofonline access times and is still several x faster for 99.9%of accesses when rebuilds are included in the timings.

All experiments were performed on SGX-enabled In-tel Core i7-6700 CPUs running Windows Server 2016.All code was compiled with the Microsoft C/C++ com-piler v19.00 with /O2. In cases where we ran code insideSGX enclaves, it was linked against a custom SGX soft-ware stack. We report all measurements in CPU cycles.

ORAM vs. Scanning Figure 3 shows the break-evenpoint of Pyramid and Circuit ORAM compared to thebasic approach—called Naive in the following—of scan-ning through all elements of an array for each access.Our implementation of Naive is optimized using AVX2SIMD operations and is the same as the one we useto scan the first level in our Pyramid implementation.Hence, for up to 1024 elements (the size of Pyramid’sfirst level), Naive and Pyramid perform identically. Wealso set Circuit ORAM to use scanning for small arrays.Pyramid and Circuit start improving over Naive at 4096(229 KB) and 8192 elements (459 KB), respectively.

Pyramid vs. Circuit Figures 6 and 7 depict the perfor-mance of Pyramid and Circuit when sequentially access-ing arrays of 1-byte and 56-byte elements. Pyramid dom-inates Circuit for datasets of size up to 14 MB. ThoughCircuit ORAM should perform worse for smaller ele-ments than larger ones, it is not evident in the 1-byteand 56-byte experiments since our implementation uses(machine-optimized) large block sizes favoring CircuitORAM.

0

0.5

1

1.5

2

2.5

2^17 2^18 2^19 2^20

Rel

ativ

e C

PU

cyc

les

Number of elements

Pyramid Circuit-3-12 Circuit-3-20

Figure 4: Average CPU cycle overhead of obliviouslyreading 1-byte elements from arrays of size 217 to 220

(4KB–1MB) in an SGX enclave; the measurements arenormalized to the performance of Pyramid ORAM.

Intel SGX We run several benchmarks inside of anSGX enclave. Due to memory restrictions of around100 MB in the first generation of Intel SGX, we evaluatethe algorithms on small datasets. Since both Circuit andPyramid have space overhead, the actual data that can beloaded in an ORAM is at most 12 MB. Compared to theresults in the previous section, SGX does not add any no-ticeable overhead. Figures 4 and 5 show the results forarrays with 1-byte and 56-byte elements.

Variation of access overhead As noted earlier, theoverhead of accessing a hierarchical ORAM depends onthe number of previous accesses. For example, an ac-cess that causes a rebuild of the last level is much slowerthan the subsequent access (for which no rebuild hap-pens). Hence, Pyramid ORAM exhibits different onlineand amortized cost as noted in Table 1. To investigate thiseffect, we accessed Pyramid and Circuit 3N times andmeasured the overhead of every request for N = 220 andN = 225. For both schemes the measured time includesthe time to access the element and to prepare the datastructure for the next request, that is, it includes rebuildand eviction for Pyramid ORAM and Circuit ORAM,respectively. Figures 8 and 9 show the correspondingoverhead split across 99.9% of the fastest among 3N ac-cesses. As can be seen, the fraction of expensive accessesin Pyramid is very small since large rebuilds are infre-quent. For N = 220, almost 40% of all Pyramid accessesare answered on average in 16.6K CPU cycles, almost10× faster than Circuit. Except for the few accesses thatcause a rebuild, Pyramid answers 99.9% of all accesseson average in 26.8K CPU cycles, compared to 178.7Kcycles for Circuit. A similar trend can be observed for thelarger array in Figure 9: for 35% of all accesses, Pyramid

12

0

0.5

1

1.5

2

2.5

2^12 2^13 2^14 2^15

Rel

ativ

e C

PU

cyc

les

Number of elements

Pyramid Circuit-3-12 Circuit-3-20

Figure 5: Average CPU cycle overhead (normalizedto Pyramid ORAM) of obliviously reading 56-byte el-ements from arrays of size 4096 to 32768 (229KB–1.8MB) from an Intel SGX enclave.

0

0.5

1

1.5

2

2.5

2^17 2^18 2^19 2^20 2^21 2^22 2^23 2^24

Rel

ativ

e C

PU

cyc

les

Number of elements

Pyramid Circuit-3-12 Circuit-3-20

Figure 6: Average CPU cycle overhead (normalized toPyramid ORAM) of obliviously reading 1-byte elementsfrom arrays of size 217 to 224 (131KB–16MB).

is at least 10× faster than Circuit and still 8× faster for99% of all accesses, while, on average, Circuit performsbetter. We stress that for many applications the onlinecost is the significant characteristic, as expensive rebuildscan be anticipated and performed during idle times or, tosome extent, can be performed concurrently.

Sparse Vectors Dot product between a vector and asparse vector is a common operation in machine-learningalgorithms. For example, in linear SVM [23], a modelcontains a weight for every feature while a sample recordmay contain only some of the features. Since relativelyfew features may be present in a sample record, recordsare stored as (feature index, value) pairs. The dot prod-uct is computed by performing random accesses into the

0

0.5

1

1.5

2

2.5

Rel

ativ

e C

PU

cyc

les

Number of elements

Pyramid Circuit-3-12 Circuit-3-20 Circuit-4-12

Figure 7: Average CPU cycle overhead (normalizedto Pyramid ORAM) of obliviously reading 56-byte el-ements from arrays of size 211 to 224 (114KB–939MB)

0

10

20

30

40

50

60

70

80

90

100

0 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 450,000

%

CPU cycles

Pyramid Circuit-3-12

99.9

Figure 8: Cumulative distribution of access times in CPUcycles when requesting elements from an array of 220

56-byte elements (58 MB) where 99.9% of fastest ac-cesses are plotted. For example, Pyramid ORAM (Cir-cuit ORAM) answers 50% and 99% of all requests un-der 23K (159K) and 82K (340K) CPU cycles.

model based on the feature indices, in turn, revealing theidentities of the features present in the record.

We use ORAM to store and access the model vector of4-byte floating feature values in order to protect recordcontent during classification. To measure the overhead,we choose three datasets with sparse sample recordsfrom LIBSVM Tools4: news20.binary (1,355,191 fea-tures and 19,996 records), rcv1-train (47,236 fea-tures and 20,242 records) and rcv1-test (47,236 fea-tures and 677,399 records). These datasets are commondatasets for text classification where a feature is a binaryterm signifying the presence of a particular trigram. As a

4https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/

datasets/ (accessed 15/05/2017).

13

0

10

20

30

40

50

60

70

80

90

100

0 100,000 200,000 300,000 400,000 500,000 600,000

%

CPU cycles

Pyramid Circuit-3-12

99.9

Figure 9: Cumulative distribution of access times in CPUcycles when requesting elements from an array of 225

56-byte elements (1.87GB) where 99.9% of fastest ac-cesses are plotted. For example, Pyramid ORAM (Cir-cuit ORAM) answers 50% and 99% of all requests un-der 33.8K (260K) and 90K (452K) CPU cycles.

result the datasets are sparse (e.g., if the first dataset wereto be stored densely, only 0.0336% of its values would benon-zero). In Figure 10 we compare the overhead of theORAM schemes over insecure baseline when computingdot product on each dataset. The insecure baseline mea-sures sparse vector multiplication without side-channelprotection. As expected, Pyramid performs better thanCircuit, as model sizes are 5.4MB for news20.binaryand 188KB for rcv1. Comparing to the baseline, the av-erage ORAM overhead is four and three orders of mag-nitude, respectively.

Decision Trees Tree traversal is also a common taskin machine learning where a model is a set of decisiontrees (a forest) and a classification on a sample recordis done by traversing the trees according to record’s fea-tures. Hence, accesses to the tree can reveal informationabout sample records. One can use an ORAM to pro-tect such data-dependent accesses either by placing thewhole tree in an ORAM or by placing each layer of thetree in a separate ORAM. The latter approach suits bal-anced trees, however, for unbalanced trees it reveals theheight of the tree and the number of nodes at each layer.

Our experiments use a pre-trained forest for a largepublic dataset5. The forest consists of 32 unbalancedtrees with 30K to 33K nodes each. A node comprisesfour 8-byte values and hence fits into a single 56-byteblock. We load each tree into a separate ORAM andevaluate 290,506 sample records; the performance re-sults are given in Figure 11. Pyramid’s low online cost

5The Covertype dataset is available from the UCI Machine Learn-ing Repository: https://archive.ics.uci.edu/ml/datasets/

covertype/ (accessed 18/05/2017)

12 13 14

Pyramid Circuit-3-12 Circuit-3-20

1

10

100

1000

10000

100000

1000000

news20 rcv1-train rcv1-test

Rel

ativ

e o

verh

ead

to

b

asel

ine

Pyramid Circuit-3-12 Naïve

Figure 10: Sparse vector multiplication: CPU cycle over-head of ORAM solutions on three datasets normalized tothe performance of the insecure baseline running on thesame dataset.

0

2000

4000

6000

Rel

ativ

e o

verh

ead

to

bas

elin

e

Pyramid Circuit-3-12 Naïve

Figure 11: Decision trees: CPU cycle overhead ofORAM solutions normalized to the performance of theinsecure baseline.

shows again: Pyramid and Circuit incur three orders ofmagnitude overhead on average, while 99% of all classi-fications on average have only two orders of magnitudeoverhead for Pyramid.

In summary, Pyramid ORAM can give orders of mag-nitude lower access latencies than Circuit ORAM. In par-ticular, Pyramid’s online access latencies are low. Thismakes Pyramid the better choice for many applicationsin practice.

9 Related work

Oblivious RAM The hierarchical constructions werethe first instantiations proposed for oblivious RAM [14].Since then, many variations have been proposed whichalter hash tables, shuffling techniques, or assumptions onthe size of private memory [15, 17, 40, 53]. The hier-archical construction by Kushilevitz et al. [25] has the

14

best asymptotical performance among ORAMs for con-stant private memory size. It uses the oblivious cuckoohashing scheme proposed by Goodrich and Mitzen-macher [15]. In particular, the scheme relies on theAKS sorting network [2], which is performed O(logN)times for every level in the hierarchy. The AKS network,though an optimal sorting network, in simplified formhas a high hidden constant of 6,100 [39]. Moreover, un-til the hierarchy reaches the level that can hold at least(logN)7 elements, every level is implemented using reg-ular hash tables with buckets of size logN. Thus, AKSonly becomes practical for large data sizes.

Tree-based ORAM schemes, first proposed by Shi etal. [10], are a fundamentally different paradigm. Therehave been a number of follow-up constructions [26, 28,50] that improve on the performance of ORAM schemes.Most of these works focus on reducing bandwidth over-head, often relying on non-constant private memory andelements of size Ω(logN) or Ω((logN)2). We note thatsuch schemes can be modified to rely on constant pri-vate memory by storing its content in an encrypted formexternally and accessing it also obliviously (e.g., a scan-ning for every access). However, this transformation isoften expensive. For example, it increases the asymptot-ical overhead of Path ORAM, because stash operationsnow require using oblivious sorting [51, 52].

ORAMs have long held the promise to serve as build-ing blocks for multi-party computation (MPC). Hencesome recent works focus on developing protocols withsmall circuit complexity [52, 55], including the Cir-cuit ORAM scheme [51] discussed in previous sections.Some of the schemes that try to decrease circuit sizealso have small private memory requirements, since any“interesting” operation on private memory (e.g., sort-ing) increases the size. However, not all ORAM designsthat target MPC can be used in our client-server model.For example, the work by Zahur et al. [55] optimizesa square-root ORAM [14] assuming that each party canstore a permutation of the elements.

Memory Side-channel Protection for HardwareRane et al. [41] investigated compiler-based techniquesfor removing secret-dependent accesses in the code run-ning on commodity hardware. Experimentally, theyshowed that Path ORAM is not suitable in this settingdue to its requirements on large private memory, showingthat naive scanning performs better for small datasets.

Ring ORAM [28] optimizes parameters of PathORAM and uses an XOR technique on the server to de-crease the bandwidth cost. As a case study, Ren et al.estimate the performance of Ring ORAM if used on se-cure processors assuming 64-byte element size. The de-sign assumes non-constant size on-chip memory that canfit the stash, a path of the tree (logarithmic in N) and the

position map at a certain recursion level. Ascend [11] isa design for a secure processor that accesses code anddata through Path ORAM. As a result, Ascend relieson on-chip private memory that is large enough to storethe stash and a path internally (i.e., O(logN)). PHAN-TOM [30] is a processor design with an oblivious mem-ory controller that also relies on Path ORAM, storing thestash in trusted memory and using 1KB–4KB elements.Fletcher et al. [12] improve on this using smaller ele-ments, but they also assume large on-chip private mem-ory. Other hardware designs optimized for Path ORAMexist [13, 20, 43]. Sanctum [9] is a secure processor de-sign related to Intel SGX. Other than SGX, it providesprivate cache partitions for enclaves, stores enclave page-tables in enclaves, and dispatches corresponding pagefaults and other exceptions directly to in-enclave han-dlers. This largely prevent malicious software (includ-ing the OS) from inferring enclave memory access pat-terns. However, it does not provide additional protectionagainst a hardware attacker.

Recent work also addresses SGX’s side channel prob-lem in software: Shih et al. [48] proposed executingsensitive code within Intel TSX transactions in order tosuppress and detect leaky page faults. Gruss et al. [19]preload sensitive code and data within TSX transac-tions in order to prevent leaky cache misses. Ohri-menko et al. [36] manually protect a set of machinelearning algorithms against memory-based side-channelattacks, applying high-level algorithmic changes andlow-level oblivious primitives. DR.SGX [6] aims to pro-tect enclaves against cache side-channel attacks by fre-quently re-randomizing data locations (except for smallstack-based data) at cache-line granularity. For this,DR.SGX augments a program’s memory accesses atcompile time. DR.SGX incurs a reported mean over-head of 3.39x–9.33x, depending on re-randomizationfrequency. DR.SGX does not provide strong guarantees;it can be seen as an ad-hoc ORAM construction that mayinduce enough noise to protect against certain softwareattacks. ZeroTrace [45] is an oblivious “memory con-troller” for enclaves. It implements Path ORAM and Cir-cuit ORAM to obliviously access data stored in untrustedmemory and on disk; only the client’s position map andstash are kept in enclave memory and are accessed obliv-iously using scanning and conditional x86 instructions.

Routing and Hash Tables The probabilistic routingnetwork in Section 4 belongs to the class of unbuffereddelta networks [38]. The functionality of our network isdifferent as instead of dropping the elements that couldnot be routed, we label them as spilled (i.e., tagged false)and carry them to the output. Probably the closest toours, is the network based on switches with multiplelinks studied by Kumar [24]. The network assumes that

15

each switch of the first stage (input buckets in our case)receives one packet, which is not the case in our workwhere buckets may contain up to c elements. Moreover,the corresponding analysis, similar to other analyses ofdelta networks, focuses on approximating the through-put of the network, while we are interested in the upper-bound of the number on elements that are not routed.

The Zigzag hash table can be seen as a combinationof the load balancing strategy called Threshold(c) [1]and Multilevel Adaptive Hashing [8]. In the former, ele-ments spilled from a table with buckets of size c are re-thrown into the same table, discarding the elements fromthe previous throws. In Multilevel Adaptive Hashing, ktables of different sizes with buckets of size 1 are usedto store elements. The insertion procedure is the same asthat for non-oblivious ZHT, that is elements that collidein the first hash table are re-inserted into the next table.Different from ZHTs, multilevel hashing non-obliviouslyrebuilds its tables when the kth table overflows.

10 Conclusions

We presented Pyramid ORAM, a novel ORAM designthat targets modern SGX processors. Rigorous analysisof Pyramid ORAM, as well as an evaluation of an opti-mized implementation for x64 Skylake CPUs, shows thatPyramid ORAM provides strong guarantees and practi-cal performance while outperforming previous ORAMdesigns in this setting either in asymptotical online com-plexity or in practice.

Acknowledgments

We thank Cedric Fournet, Ian Kash, and MarkulfKohlweiss for helpful discussions.

References

[1] Micah Adler, Soumen Chakrabarti, MichaelMitzenmacher, and Lars Rasmussen. Parallel ran-domized load balancing. In ACM Symposium onTheory of Computing (STOC), 1995.

[2] Miklos Ajtai, Janos Komlos, and Endre Szemeredi.An O(n logn) sorting network. In ACM Symposiumon Theory of Computing (STOC), 1983.

[3] S. Arnautov, B. Trach, F. Gregor, T. Knauth,A. Martin, C. Priebe, J. Lind, D. Muthukumaran,D. O’Keeffe, David Goltzsche, David Eyers, Rudi-ger Kapitza, Peter Pietzuch, and Christof Fetzer.Scone: Secure Linux containers with Intel SGX. InUSENIX Symposium on Operating Systems Designand Implementation (OSDI), 2016.

[4] Kenneth E. Batcher. Sorting networks and their ap-plications. In Spring Joint Computer Conf., 1968.

[5] Andrew Baumann, Marcus Peinado, and GalenHunt. Shielding applications from an untrustedcloud with Haven. In USENIX Symposium on Oper-ating Systems Design and Implementation (OSDI),2014.

[6] Ferdinand Brasser, Srdjan Capkun, AlexandraDmitrienko, Tommaso Frassetto, Kari Kostiainen,Urs Muller, and Ahmad-Reza Sadeghi. DR.SGX:Hardening SGX enclaves against cache attackswith data location randomization. arXiv preprintarXiv:1709.09917, 2017.

[7] Ferdinand Brasser, Urs Muller, AlexandraDmitrienko, Kari Kostiainen, Srdjan Capkun,and Ahmad-Reza Sadeghi. Software grand ex-posure: SGX cache attacks are practical. CoRR,abs/1702.07521, 2017.

[8] Andrei Z. Broder and Anna R. Karlin. Multileveladaptive hashing. In ACM-SIAM Symposium onDiscrete Algorithms, 1990.

[9] Victor Costan, Ilia Lebedev, and Srinivas Devadas.Sanctum: Minimal hardware extensions for strongsoftware isolation. In USENIX Security Sympo-sium, 2016.

[10] Shi Elaine, Chan T-H Hubert, Stefanov Emil,and Li Mingfei. Oblivious ram with o((logn)3)worst-case cost. In Advances in Cryptology—ASIACRYPT, 2011.

[11] Christopher W. Fletcher, Marten van Dijk, andSrinivas Devadas. A secure processor architecturefor encrypted computation on untrusted programs.In ACM Workshop on Scalable Trusted Computing(STC), 2012.

[12] Christopher W. Fletcher, Ling Ren, Albert Kwon,Marten van Dijk, Emil Stefanov, Dimitrios Ser-panos, and Srinivas Devadas. A low-latency, low-area hardware oblivious RAM controller. In Sym-posium on Field-Programmable Custom Comput-ing Machines, 2015.

[13] Christopher W. Fletcher, Ling Ren, Albert Kwon,Marten van Dijk, and Srinivas Devadas. Freecur-sive ORAM: [nearly] free recursion and integrityverification for position-based oblivious ram. InInternational Conference on Architectural Supportfor Programming Languages and Operating Sys-tems (ASPLOS), 2015.

16

[14] Oded Goldreich and Rafail Ostrovsky. Softwareprotection and simulation on oblivious RAMs.Journal of the ACM (JACM), 43(3), 1996.

[15] Michael T. Goodrich and Michael Mitzenmacher.Privacy-preserving access of outsourced data viaoblivious RAM simulation. In International Collo-quium on Automata, Languages and Programming(ICALP), 2011.

[16] Michael T. Goodrich, Michael Mitzenmacher, OlgaOhrimenko, and Roberto Tamassia. ObliviousRAM simulation with efficient worst-case accessoverhead. In ACM Cloud Computing SecurityWorkshop (CCSW), 2011.

[17] Michael T. Goodrich, Michael Mitzenmacher, OlgaOhrimenko, and Roberto Tamassia. Privacy-preserving group data access via stateless oblivi-ous RAM simulation. In ACM-SIAM Symposiumon Discrete Algorithms, 2012.

[18] J. Gotzfried, M. Eckert, S. Schinzel, and T. Muller.Cache attacks on Intel SGX. In European Work-shop on System Security (EuroSec), 2017.

[19] Daniel Gruss, Julian Lettner, Felix Schuster, OlyaOhrimenko, Istvan Haller, and Manuel Costa.Strong and efficient cache side-channel protectionusing hardware transactional memory. In USENIXSecurity Symposium, 2017.

[20] Syed Kamran Haider and Marten van Dijk. FlatORAM: A simplified write-only oblivious RAMconstruction for secure processor architectures,2017.

[21] Matthew Hoekstra, Reshma Lal, Pradeep Pap-pachan, Carlos Rozas, Vinay Phegade, and Juandel Cuvillo. Using innovative instructions to cre-ate trustworthy software solutions. In Workshopon Hardware and Architectural Support for Secu-rity and Privacy (HASP), 2013.

[22] Tyler Hunt, Zhiting Zhu, Yuanzhong Xu, SimonPeter, and Emmett Witchel. Ryoan: A distributedsandbox for untrusted computation on secret data.In USENIX Symposium on Operating Systems De-sign and Implementation (OSDI), 2016.

[23] S. Sathiya Keerthii and Dennis Decoste. A modi-fied finite newton method for fast solution of largescale linear svms. Journal of Machine LearningResearch, 6:341–361, 2005.

[24] Manoj Kumar. Performance improvement insingle-stage and multiple-stage shuffle-exchangenetworks. PhD thesis, Rice University, 1984.

[25] Eyal Kushilevitz, Steve Lu, and Rafail Ostrovsky.On the (in)security of hash-based oblivious RAMand a new balancing scheme. In ACM-SIAM Sym-posium on Discrete Algorithms (SODA), 2012.

[26] Dautrich Jr Jonathan L, Stefanov Emil, and ShiElaine. Burst ORAM: Minimizing ORAM re-sponse times for bursty access patterns. In USENIXSecurity Symposium, 2014.

[27] Sangho Lee, Ming-Wei Shih, Prasun Gera, TaesooKim, Hyesoon Kim, and Marcus Peinado. Infer-ring fine-grained control flow inside SGX enclaveswith branch shadowing. In USENIX Security Sym-posium, 2017.

[28] Ren Ling, Fletcher Christopher W, Kwon Albert,Stefanov Emil, Shi Elaine, Van Dijk Marten, andDevadas Srinivas. Constants count: Practical im-provements to oblivious RAM. In USENIX SecuritySymposium, 2015.

[29] Fangfei Liu, Yuval Yarom, Qian Ge, Gernot Heiser,and Ruby B Lee. Last-level cache side-channel at-tacks are practical. In IEEE Symposium on Securityand Privacy (S&P), 2015.

[30] Martin Maas, Eric Love, Emil Stefanov, Mohit Ti-wari, Elaine Shi, Krste Asanovic, John Kubiatow-icz, and Dawn Song. PHANTOM: practical obliv-ious computation in a secure processor. In ACMConference on Computer and Communications Se-curity (CCS), 2013.

[31] Clementine Maurice, Manuel Weber, MichaelSchwarz, Lukas Giner, Daniel Gruss, Carlo AlbertoBoano, Kay Romer, and Stefan Mangard. Hellofrom the other side: Ssh over robust cache covertchannels in the cloud. In Symposium on Networkand Distributed System Security (NDSS), 2017.

[32] Frank Mckeen, Ilya Alexandrovich, Alex Beren-zon, Carlos Rozas, Hisham Shafi, VedvyasShanbhogue, and Uday Savagaonkar. Innovativeinstructions and software model for isolated execu-tion. In Workshop on Hardware and ArchitecturalSupport for Security and Privacy (HASP), 2013.

[33] Michael Mitzenmacher and Eli Upfal. Proba-bility and Computing: Randomized Algorithmsand Probabilistic Analysis. Cambridge UniversityPress, New York, NY, USA, 2005.

[34] Tarik Moataz, Travis Mayberry, Erik-Oliver Blass,and Agnes Hui Chan. Resizable tree-based obliv-ious RAM. In Financial Cryptography and DataSecurity, pages 147–167, 2015.

17

[35] Ahmad Moghimi, Gorka Irazoqui, and ThomasEisenbarth. Cachezoom: How SGX amplifies thepower of cache attacks. CoRR, abs/1703.06986,2017.

[36] Olga Ohrimenko, Felix Schuster, Cedric Fournet,Aastha Mehta, Sebastian Nowozin, Kapil Vaswani,and Manuel Costa. Oblivious multi-party machinelearning on trusted processors. In USENIX SecuritySymposium, 2016.

[37] Rafail Ostrovsky and Victor Shoup. Private infor-mation storage (extended abstract). In ACM Sym-posium on Theory of Computing (STOC), 1997.

[38] Janak H. Patel. Performance of processor-memoryinterconnections for multiprocessors. IEEE Trans.Computers, 30(10):771–780, 1981.

[39] Mike Paterson. Improved sorting networks witho(log N) depth. Algorithmica, 5(1):65–92, 1990.

[40] Benny Pinkas and Tzachy Reinman. ObliviousRAM revisited. In Advances in Cryptology—CRYPTO, 2010.

[41] Ashay Rane, Calvin Lin, and Mohit Tiwari. Rac-coon: Closing digital side-channels through obfus-cated execution. In USENIX Security Symposium,2015.

[42] Kaveh Razavi, Ben Gras, Erik Bosman, Bart Pre-neel, Cristiano Giuffrida, and Herbert Bos. Flipfeng shui: Hammering a needle in the softwarestack. In USENIX Security Symposium, 2016.

[43] Ling Ren, Xiangyao Yu, Christopher W. Fletcher,Marten van Dijk, and Srinivas Devadas. Designspace exploration and optimization of path obliv-ious RAM in secure processors. In Annual In-ternational Symposium on Computer Architecture(ISCA), 2013.

[44] Thomas Ristenpart, Eran Tromer, Hovav Shacham,and Stefan Savage. Hey, you, get off of my cloud:Exploring information leakage in third-party com-pute clouds. In ACM Conference on Computer andCommunications Security (CCS), 2009.

[45] Sajin Sasy, Sergey Gorbunov, and ChristopherFletcher. ZeroTrace: Oblivious memory primitivesfrom Intel SGX. In Symposium on Network andDistributed System Security (NDSS), 2017.

[46] Felix Schuster, Manuel Costa, Cedric Four-net, Christos Gkantsidis, Marcus Peinado, GloriaMainar-Ruiz, and Mark Russinovich. VC3: Trust-worthy data analytics in the cloud using SGX. In

IEEE Symposium on Security and Privacy (S&P),2015.

[47] Michael Schwarz, Samuel Weiser, Daniel Gruss,Clementine Maurice, and Stefan Mangard. Mal-ware Guard Extension: Using SGX to concealcache attacks. In Conference on Detection of In-trusions and Malware & Vulnerability Assessment(DIMVA), 2017.

[48] Ming-Wei Shih, Sangho Lee, Taesoo Kim, andMarcus Peinado. T-SGX: Eradicating controlled-channel attacks against enclave programs. In Sym-posium on Network and Distributed System Secu-rity (NDSS), 2017.

[49] Emil Stefanov, Elaine Shi, and Dawn XiaodongSong. Towards practical oblivious RAM. In Sym-posium on Network and Distributed System Secu-rity (NDSS), 2012.

[50] Emil Stefanov, Marten van Dijk, Elaine Shi,Christopher W. Fletcher, Ling Ren, Xiangyao Yu,and Srinivas Devadas. Path ORAM: an extremelysimple oblivious RAM protocol. In ACM Confer-ence on Computer and Communications Security(CCS), 2013.

[51] Xiao Wang, Hubert Chan, and Elaine Shi. CircuitORAM: On tightness of the Goldreich-Ostrovskylower bound. In ACM Conference on Computer andCommunications Security (CCS), 2015.

[52] Xiao Shaun Wang, Yan Huang, T-H. Hubert Chan,Abhi Shelat, and Elaine Shi. SCORAM: ObliviousRAM for secure computation. In ACM Conferenceon Computer and Communications Security (CCS),2014.

[53] Peter Williams, Radu Sion, and Bogdan Carbunar.Building castles out of mud: practical access pat-tern privacy and correctness on untrusted storage.In ACM Conference on Computer and Communi-cations Security (CCS), 2008.

[54] Yuanzhong Xu, Weidong Cui, and Marcus Peinado.Controlled-channel attacks: Deterministic sidechannels for untrusted operating systems. In IEEESymposium on Security and Privacy (S&P), 2015.

[55] S. Zahur, X. Wang, M. Raykova, A. Gascon, J. Do-erner, D. Evans, and J. Katz. Revisiting square-root ORAM: Efficient random access in multi-partycomputation. In IEEE Symposium on Security andPrivacy (S&P), 2016.

18

2

1

4

3

0 5 6

7

Ro

uti

ng

Net

wo

rk

3

2

4

0

1

H1 H2 H3H1

1

6

5

1

5 6

7

Ro

uti

ng

Net

wo

rk

3

2

4

0

H2H2 H3H1

1 5

7

6

Ro

uti

ng

Net

wo

rk

3

2

4

0

H2 H3H1

7

6

H3

2

1

4

3

0 5 6

7

H1 H2 H3

Phase 3Phase 2

Phase 1Initial throw

3

2

4

0

1 5

7

6

H1 H2 H3

Final Zigzag Hash Table

Figure 12: Oblivious Zigzag Hash Table Setup from Section 5.2. During the initial throw elements are allocated usingrandom hash functions. In phase 1 elements in H1 are routed with probabilistic routing network using hash function h1.Element 1 could not be routed to its bucket and, hence, is greyed out (tagged false by the PRN). Hence, it is throwninto the next table. In the next phases elements in H2 and H3 are routed according to h2 and h3, correspondingly.

Appendix

A Additional lemmas

Lemma 6. When throwing m elements in n buckets, eachwith capacity c, the probability that a given bucket re-ceives c or more elements is at most

(mc

) 1nc .

Proof. The probability that a bucket gets assigned c ormore elements is

m

∑i=c

(mi

)1ni

(1− 1

n

)m−i

=(mc

)1nc

m

∑i=c

c!(m− c)!i!(m− i)!

1ni−c

(1− 1

n

)m−i

=(mc

)1nc

m−c

∑i=0

c!(m− c)!(c+ i)!(m− c− i!)

1ni

(1− 1

n

)m−c−i

To prove the lemma, we show that the sum expression isat most 1. Note that as long as the first component of thesum is at most

(m−ci

), we can use the binomial formula:

m−c

∑i=0

(m− c

i

)1ni

(1− 1

n

)m−c−i

= 1

We now show that(m−c

i

)≥ c!(m−c)!

(c+i)!(m−c−i!) for i ≥ 0 andc≥ 1:

(m− c)!i!(m− c− i)!

≥ c!(m− c)!(c+ i)!(m− c− i)!

1i!≥ c!

(c+ i)!=

1(c+1) . . .(c+ i)

Lemma 7. The average number of elements that do notget assigned to a bucket during the throw of m elementsinto n buckets of size c is at most 1

nc

( mc+1

).

Proof. Let X j be a random variable that is 1 if thereis a collision on the jth inserted element, and 0 other-wise. For the first c+1 elements, Pr[X1] = Pr[X2] = . . .=Pr[Xc] = 0. For c+ 1 ≤ j ≤ n, we need to estimate theprobability that jth element is inserted in a bucket with atleast c elements. The probability of a given bucket hav-ing c or more elements after j− 1 elements have beeninserted is given in Lemma 6. Hence,

Pr[X j]≤(

j−1c

)1nc

19

Note that X j are independent. Then, expected number ofcollisions is

µ = E

[m

∑j=1

X j

]≤

m

∑j=c+1

Pr[X j]≤

m−1

∑j=c

(jc

)1nc =

1nc

m−1

∑j=c

(jc

)Using the fact that ∑

m−1j=c

( jc

)=( m

c+1

):

µ ≤ 1nc

(m

c+1

)

Lemma 8. The number of elements that are spilled dur-ing the ith stage of the routing network of size n on inputof size m ≤ n is at most the number of elements that donot fit into H1 of a ZHT when throwing m elements.

Proof. We are interested in counting the number of ele-ments whose tag changes from true to false during the ithstage of the probabilistic routing network and refer tosuch elements as spilled during the ith stage.

Let hstart be the hash function that was used to dis-tribute elements during the throw of n elements into theinput table of the network (i.e., the elements that did finda spot in the table appear in buckets according to hstart )and hend be the hash function of the network. Let || de-note a concatenation of two bit strings.

Consider stage i of the network, 1 ≤ i ≤ logn. Afterstage i, element e is assigned to a bucket a||b where aare the first i bits of hstart(e) and b are the first logn− ibits of hend(e). Now consider, a throw phase of m ele-ments into a table of n buckets using the same hash func-tion. Let us compare the number of elements assignedto a particular bucket in each case. We argue that thenumber in the former case is either the same as in thelatter case or smaller. Note that the elements that canbe assigned to a bucket is the same in both cases as ele-ments are distributed across buckets using the same hashfunction. However, some of the elements of the PRNmay have been lost in earlier stages (those that arrive tothe ith stage with the false tag) and have already beencounted towards the spill of an earlier stage. Since ele-ments tagged false are disregarded by the network duringrouting, ith stage may spill the same number of elementsas the hash table or less since there are less “live” (taggedtrue) elements that reach their buckets.

Lemma 9. Consider the following allocation of n el-ements into tables, each with n buckets of size c =log logn. The elements are allocated uniformly at ran-dom in the first table. The elements that do not fit intothe first table are allocated into the second table.

The number of elements that arrive into the secondtable is at most n/(2e).

Proof. In Lemma 7 we showed that average number ofoutgoing elements at level i is µi =

1nc

(ini

c+1

). Hence, with

in1 = n

µ1 ≤nc+1

nc(c+1)!=

n(c+1)!

≤ n2e logn

where we rely on (c + 1)! ≥ 2e logn (see below). Ifn/(2e logn) is sufficiently large (e.g., n/(2e logn) ≥ 3swhere s is the security parameter), we can use Cher-noff bounds to show that with probability exp(−µ1/3)the number of spilled elements from the first level is nomore than 2µ1. Hence, in total the number of elementsthat spill is at most:

ne logn

≤ n2e

If n is of moderate size (e.g., n≥ 210), we can use anotherform of Chernoff bounds [33, Chapter 4] to show thatwith probability 2−R the number of spilled elements fromthe first level is no more than R≥ 6µ1, where 6µ1≥ s and6µ1 ≤ n/2e.

Finally we show that for any a ≥ 10, (loga+ 1)! ≥2ea:

(loga+1)!≥√

2π(loga+1)(loga+1)loga+1

eloga+1 ≥√2π(loga+1)

(loga+1)loga+1

3loga+1 ≥√2π(loga+1)

2log(loga+1)(loga+1)

2(loga+1) log3 ≥

2ea

Lemma 10. Consider the process in Lemma 9 but afterthe first throw, the first table is routed through the rout-ing network from Section 4 and elements spilled from thenetwork are also thrown into the second table. The ele-ments of the second table are routed as well and spilledelements are thrown into the third table, and so on. Thenumber of elements that arrive into the third table is atmost n/(2e).

Proof. From Lemma 8 we know that the total overflowfrom throwing the elements and routing them through thenetwork is at most logn times the overflow from the orig-inal throw. Consider the proof of Lemma 9. The secondtable receives at most

2n logn2e logn

=ne

20

number of elements. Then the average overflow accord-ing to Lemma 7 is

1nc

( ne

c+1

)≤ nc+1

ncec+12e logn=

n2lognec+2 ≤

n4e logn

We can again use Chernoff bound to show that the num-ber of elements that arrive into the next level will beat most 2 logn factor away from the above expression.Hence, the number of elements that arrive into the thirdtable is

2n logn4e logn

=n2e

Lemma 11. Consider the following allocation of m ≤n/(2e) elements into k tables, each with n buckets ofsize c = log logn. The elements are allocated uniformlyat random in the first table. The elements that do notfit into the first table are allocated into the second ta-ble and so on, until there are no elements left. Then,k = O(log logn) is sufficient to allocate all elements.

Proof. We analyze the process using Poisson randomvariables as an approximation of the balls-and-bins ap-proximation process [33, Chapter 5]. Recall that as longas event’s probability is monotonically decreasing or in-creasing in the number of elements, the event that hap-pens with probability ε in the Poisson case has proba-bility at most 2ε in the exact case. Let Zi be a Poissonrandom variable that measures the number of elementsin a bucket after a random throw. Let α be the mean ofZi. Recall that,

Pr(Zi ≥ c)≤ e−α(eα)c

cc

Hence, the number of n buckets that will have at leastc elements is ne−α(eα)c/cc. It is known, that withvery high probability no bucket has more than O(logn)elements, hence, the number of overflow elements is(n logn)e−α(eα)c/cc. We now show that as long asα ≤ 1/(2e), we require O(log logn) rounds until no over-flow occurs. After the first throw, the overflow is

(n logn)(eα)c+1

(c+1)c+1 ≤n

2c+1e logn(1)

where with c = dlog logne, logn/(c+ 1)c+1 ≤ 1/e (seebelow). Hence, in the next round:

(n logn)(e/(2c+1e logn))c+1

(c+1)c+1 ≤ n2(c+1)2

(logn)c+2e

Then for the ith round the overflow is:

n2(c+1)i

(logn)(c+1)i−1e

Hence, after at most O(log logn) rounds the above ex-pression is less than 1.

We are left to show that as long as a ≥ 8, a/(loga+1)(loga+1) ≤ 1/e. In the following lemma we require asimilar result but for a2 in the numerator. Hence, insteadwe show that a2/(loga+1)(loga+1) ≤ 1/e:

a2

(loga+1)(loga+1) =1

2log(loga+1)(loga+1)−2loga≤

12log(loga+1)+loga(log(loga+1)−2) ≤

1loga+1

≤ 14≤ 1

e

Lemma 12. Consider the process in Lemma 11 but afterthe first throw, the first table is routed through the rout-ing network from Section 4 and elements spilled from thenetwork are also thrown into the second table. The ele-ments of the second table are routed as well and spilledelements are thrown into the third table, and so on. Then,O(log logn) rounds are still sufficient to avoid overflowin the last round.

Proof. From Lemma 8 we know that the total overflowfrom throwing the elements and routing them through thenetwork is at most logn times the overflow from the orig-inal throw. Consider the proof of Lemma 11. In Equa-tion 1, we can replace n with n logn to account for thebuckets in each round of the network. This would resultin the spill of

n2c+1e logn

in the first round and

n2(c+1)ie

in the ith round. Hence, O(log logn) rounds are still suf-ficient to fit all the elements.

21