Dynamic Searchable Encryption in Very-LargeDatabases: Data Structures and Implementation

David Cash!, Joseph Jaeger!, Stanislaw Jarecki†, Charanjit Jutla‡,Hugo Krawczyk‡, Marcel-Catalin Rosu‡, and Michael Steiner‡

!Rutgers University†University of California, Irvine

‡IBM Research

Abstract—We design and implement dynamic symmetricsearchable encryption schemes that efficiently and privatelysearch server-held encrypted databases with tens of billions ofrecord-keyword pairs. Our basic theoretical construction sup-ports single-keyword searches and offers asymptotically optimalserver index size, fully parallel searching, and minimal leakage.Our implementation effort brought to the fore several factorsignored by earlier coarse-grained theoretical performance anal-yses, including low-level space utilization, I/O parallelism andgoodput. We accordingly introduce several optimizations to ourtheoretically optimal construction that model the prototype’scharacteristics designed to overcome these factors. All of ourschemes and optimizations are proven secure and the informationleaked to the untrusted server is precisely quantified. We evaluatethe performance of our prototype using two very large datasets:a synthesized census database with 100 million records andhundreds of keywords per record and a multi-million webpagecollection that includes Wikipedia as a subset. Moreover, wereport on an implementation that uses the dynamic SSE schemesdeveloped here as the basis for supporting recent SSE advances,including complex search queries (e.g., Boolean queries) andricher operational settings (e.g., query delegation), in the aboveterabyte-scale databases.

I. INTRODUCTION

BACKGROUND. Searchable symmetric encryption (SSE) al-lows one to store data at an untrusted server and later searchthe data for records (or documents) matching a given keywordwhile maintaining privacy. Many recent works [3]–[5], [7], [9],[14], [15], [17], [19], [21] studied SSE and provided solutionswith varying trade-offs between security, efficiency, and theability to securely update the data after it has been encryptedand uploaded. These constructions aim at practical efficiency,in contrast to generic cryptographic tools like homomorphicencryption or multiparty computation which are highly securebut not likely to be efficient in practice.

Large data sizes motivate storage outsourcing, so to beuseful an SSE scheme must scale well. Existing SSE schemesemploy only symmetric cryptography operations and standard

data structures and thus show potential for practical effi-ciency, but obstacles remain. While most constructions havetheoretically optimal search times that scale only with thenumber of documents matching the query, the performance oftheir implementations on large datasets is less clear. Factorslike I/O latency, storage utilization, and the variance of real-world dataset distributions degrade the practical performanceof theoretically efficient SSE schemes. One critical source ofinefficiency in practice (often ignored in theory) is a completelack of locality and parallelism: To execute a search, most priorSSE schemes sequentially read each result from storage at apseudorandom position, and the only known way to avoid thiswhile maintaining privacy involves padding the server indexto a prohibitively large size.

CONTRIBUTIONS. We give the first SSE implementation thatcan encrypt and search on datasets with tens of billionsof record/keyword pairs. To design our scheme, we startwith a new, simple, theoretical SSE construction that uses ageneric dictionary structure to already achieve an asymptoticimprovement over prior SSE schemes, giving optimal leakage,server size, search computation, and parallelism in search. Thisstarting point can be seen as a generalization and simplificationof the more ad-hoc techniques of [3]. We show how to makethe scheme dynamic, meaning that the data can be changedafter encryption: Our scheme can easily support additions tothe data, as well as deletions via revocation lists.

Because the scheme uses a generic dictionary that itselfhas no security properties, it allows for several extensions andmodifications with only small changes to the security proofs.In particular, our implementation effort showed that disk I/Outilization remained a bottleneck which prevented scaling; sowe extend our basic construction to improve locality andthroughput. These extensions preserve privacy with slightlydifferent leakages that we analyze with formal security proofs.Below we describe the techniques behind results in more detail,starting with the new theoretical scheme that we extend later,and then compare our results to prior work.

BASIC CONSTRUCTION. Our scheme is very simple (seeFigure 2): It associates with each record/keyword pair a pseu-dorandom label, and then for each pair stores the encryptedrecord identifier with that label in a generic dictionary datastructure. We derive the labels so that the client, on inputa keyword to query, can compute a keyword-specific shortkey allowing the server to search by first recomputing thelabels, then retrieving the encrypted identifiers from the dic-

Permission to freely reproduce all or part of this paper for noncommercialpurposes is granted provided that copies bear this notice and the full citationon the first page. Reproduction for commercial purposes is strictly prohibitedwithout the prior written consent of the Internet Society, the first-named author(for reproduction of an entire paper only), and the author’s employer if thepaper was prepared within the scope of employment.NDSS ’14, 23-26 February 2014, San Diego, CA, USACopyright 2014 Internet Society, ISBN 1-891562-35-5http://dx.doi.org/

tionary, and finally decrypting the matching encrypted recordidentifiers. The only information leaked to the server by theencrypted index (other than the indexes of records matching aquery) is the number of items in the dictionary, i.e. the numberof record/keyword pairs in the data. This scheme is easy toimplement correctly (and with parallel searching) because wemake no security demands on the dictionary thus allowinginstantiations as applications demand.

EXTENSIONS FOR EXTERNAL STORAGE. To compute theresults of a keyword search with r matches, our basic schemerequires r retrievals from the dictionary for pseudorandomlabels. Assuming O(1) cost of a dictionary retrieval, this isasymptotically optimal. However, in implementations this willbe far from optimal when the dictionary is stored in externalmemory (i.e., a block device like a HDD), because eachrandom-looking retrieval will generate a disk read. This is incontrast to a plaintext system which could store all of thematches in a single contiguous area of memory.

In view of this reality we extend our scheme to use externalstorage more carefully while maintaining privacy. We firstshow how to securely “pack” related results together via apadding strategy to reduce the number of dictionary retrievals.

We found that even this modification was too slow forthe datasets we targeted, and in particular we noticed thatreal data-sets exhibit extreme variability in the number ofmatches for a keyword: There were typically many keywordsmatching very few documents, then some keywords matching asignificant fraction of the entire database. Our padding strategythus becomes unsatisfactory because the (many) keywordsmatching only a few results create a lot of padding, and thesearches that return a large number of results still trigger alarge number of dictionary retrievals.

To address this we introduce further modifications thatreplace dictionary reads with array reads when processing largenumbers of results. These modifications result in a slightlydifferent, but intuitively acceptable (and perhaps even better)leakage profile that we discuss below.

EXTENSION FOR UPDATES. We observe that our scheme easilyextends to allow for additions to the data after it has beenuploaded. We only have to arrange that the client can computethe labels for the new data to be added, which it sends to theserver for to be added to the dictionary. This requires eitherclient state or communication proportional to the total numberof keywords ever added or deleted. To support deletions wemaintain a (pseudorandom) revocation list at the server thatallows to filter out results that should be deleted; To actuallyreclaim space we must re-encrypt periodically.

OTHER APPLICATIONS. Recent constructions of SSE support-ing more complex queries [3] and multi-client settings [13]use SSE as a black-box. Thus our data structures and asso-ciated operations (including support for dynamic databases)are readily available to support terabyte-scale databases inthese much richer/complex encrypted-search settings (see endof Section II).

IMPLEMENTATION. Our implementation remains efficient ontwo orders of magnitude larger datasets than the most scalable

previous work [3], resulting in the first implementation ofSSE on terabyte-scale databases containing tens of billionsof indexed record/keyword pairs. We report on our prototypedesign and experimental results in Section V.

COMPARISON TO PRIOR WORK. In Figure 1 we compare ourbasic theoretical scheme to prior work. The basic scheme!bas generalizes and greatly simplifies an approach implicitin [3], which complicated the analysis by demanding securityproperties of the underlying data structures.

For a database with N record/keyword pairs, our basicscheme !bas produces an encrypted index of optimal sizeO(N), leaks only the size N and the matching record id’s,and processes a search with r results in optimal O(r) time,assuming O(1)-cost for dictionary retrievals. Searching istrivial to parallelize with any number of processors.

Most prior schemes leak additional information, like thenumber of unique keywords, the size of the largest numberof matches for a keyword, and so on. Some of these worksalso pad their encrypted indexes to be (worst-case) quadraticin their input size, which is totally impractical for largedatasets. A notable issue with most prior work was a difficultywith parallelism: Other than [3], parallel searching was onlyachieved by two works that needed quadratic padding. Workslike [7] required walking through an encrypted linked list andwere not parallelizable at all. See the “Ind Leak”, “Index Size”,and “Search Time” columns in Figure 1.

The only prior dynamic schemes either had an impracti-cally large index [14] or leaked the structure of the addeddocuments [15], meaning that the server learned, say, thepattern of which keywords appear in which documents as theyare added, which is a severe form of leakage compared to theusual SSE leakage of facts like the total database size. Ourdynamic extension maintains the optimal index size and onlyleaks basic size information (and not document structure, asin [15]). Unlike prior dynamic schemes, ours does not reclaimspace after each deletion - rather, we envision applicationswhere deletions are relatively rare or, more generally, wherea periodic complete re-encryption of the data is performed(re-encryption may be desirable to mitigate the leakage fromupdates with any dynamic SSE scheme).

MORE ON RELATED WORK. The notion of SSE we considerhas its origins in work by Song, Wagner, and Perrig [19].Several schemes since have improved upon the security andefficiency offered by the original schemes. The most similarto our construction is that of Chase and Kamara [5], and Cashet al [3]. Chase and Kamara also uses a dictionary, but ina different way and with an impractical level of padding forlarge datasets. Cash et al implements a scheme similar to ourbasic construction, but do not address updates nor, as we showin Section V-E, does their approach achieve the same level ofpractical scalability.

There is also a related line of work on searchable public-key encryption starting with [2], all of which do not scaledue to linear-time searching. The version of SSE we dealwith inherently leaks the identifiers of documents that match aquery, as well as when a query is repeated. It is possible to hideeven this information using private information retrieval [6] oroblivious RAM [10]. Several recent improvements to oblivious

2

Scheme Security Ind Leak Dyn.? Dyn Leak Index Size Search Time/Comm Dyn. Comm

CGKO’06-1 [7] NonAd m,N No — O(N +m) O(r), O(1) —CGKO’06-2 [7] Ad Mn No — O(Mn) O(r), O(r) —

CK’10 [5] Ad m,n,M No — O(Mn) O(r), O(r) —LSDHJ’10 [21] Ad m,n Yes no proof O(mn) O(m), O(1) O(|Wid|)

KO’12 [17] Ad(UC) n,M No — O(Mn) O(r), O(1) —KPR’12 [15] Adro m,N Yes EP(Wid) O(N +m) O(r), O(1) O(|Wid|)KP’13 [14] Adro m,n Yes minimal O(mn) O((r logn)/p), O(1) O(|Wid|+m logn)

Basic (!bas here) NonAd,Adro N No — O(N) O(r/p), O(1) —Basic Adp (!ro

bashere) Ad N No — O(N) O(r/p), O(r) —

Basic Dyn (!dynbas

,!dyn,robas

here) NonAd,Adro N Yes minimal O(N) O((r + dw)/p), O(1) O(|Wid|+m logn)

Fig. 1. Comparison of some SSE schemes. Many leakages can be replaced by upper bounds and some search times assume interaction when the originalpaper was non-interactive. Legend: In security, “Ad” means adaptive security, Adro means adaptive security in the random oracle model, and NonAd meansnon-adaptive security. Ind Leakage is leakage from encrypted database only. Search comm. is the size of the message sent from client (O(r) from the serveris inherent.) ro means random oracle model, n = # documents, N =

!

w|DB(w)|, m = |W|, M = maxw |DB(w)|, r = |DB(w)| for the query w, p = #

processors, |Wid| = # keyword changes in an update, EP(Wid) = structural equality pattern of changed keywords (see discussion at the end of Section IV), dw= the number of times the searched-for keyword has been added/deleted.

RAM move it further towards practicality [11], [20], but it isfar from clear that they are competitive with SSE schemesat scale as one must implement the plaintext searching as anabstract RAM program and then run this program underneaththe ORAM scheme without leaking information, say, viatiming.

ORGANIZATION. Preliminary definitions are given in Sec-tion II. Our non-dynamic (i.e., static) SSE constructions aregiven in Section III, and the dynamic extensions are givenin Section IV. Finally we report on our implementation inSection V.

II. DEFINITIONS AND TOOLS

The security parameter is denoted !. We will use thestandard notions of variable-input-length PRFs and symmetricencryption schemes (c.f. [16]). For these primitives we makethe simplifying assumption that their keys are always in{0, 1}!, and that key generation for the encryption schemepicks a random key. Some of our constructions will be ana-lyzed in the random oracle model [1], with the random oracledenoted H .

Our constructions will use a symmetric encryption schemethat is assumed to have pseudorandom ciphertexts underchosen-plaintext attack, which we call RCPA security. For-mally, this says than an adversary cannot distinguish an oraclereturning encryptions of chosen messages from one returninga random string with length equal to a fresh ciphertext for thechosen message.1

SSE SCHEMES. We follow the formalization of Curtmola etal. [7] with some modifications discussed below. A databaseDB = (idi,Wi)di=1 is a list of identifier/keyword-set pair-swhere idi ! {0, 1}! and Wi " {0, 1}!. When the DB under

consideration is clear, we will write W ="d

i=1 Wi. For akeyword w ! W, we write DB(w) for {idi : w ! Wi}.We will always use m = |W| and N =

!

w"W|DB(w)|

to mean the number of keywords and the total number ofkeyword/document matches in DB.

1Our constructions can be proved secure assuming a type of key anonymityproperty, but RCPA is simpler and is anyway achieved by many efficientconstructions.

A dynamic searchable symmetric encryption (SSE) scheme! consists of an algorithm Setup and protocols Search andUpdate between the client and server, all fitting the syntaxbelow. We assume that the server is deterministic, and thatthe client may hold some state between queries. A static SSEscheme is exactly the same, but with no Update protocol.Setup takes as input a database DB, and outputs a secret keyK along with an encrypted database EDB. The search protocolis between a client and server, where the client takes as inputthe secret key K and a query w ! {0, 1}! and the server takesas input EDB and the server outputs a set of identifiers and theclient has no output. In the Update protocol the client takesas input a key K, an operation op ! {add, del, edit+, edit#},a file identifier id, and a set Wid of keywords. These inputsrepresent the actions of adding a new file with identifier idcontaining keywords Wid, deleting the file with identifier id,or add/removing the keywords in Wid from an existing file. Atthe end of the Update, the server outputs an updated encrypteddatabase, and the client has no output.

We say that an SSE scheme is correct if the search protocolreturns the (current) correct results for the keyword beingsearched (i.e., DB(w)), except with negligible probability.(More precisely, this should hold for every database DB, after apolynomially unbounded number of updates. We defer detailsto the full version.) To simplify the formalism we ignore thecase where the client attempts to add a file with an existingidentifier or delete/edit with an identifier that is not presentin DB. Our protocols (and implementations) can handle thesecases cleanly.

DISCUSSION. For simplicity our formalization of SSE doesnot model the storage of the actual document payloads. TheSSE literature varies on its treatment of this issue, but in allcases one can augment the schemes to store the documentswith no additional leakage beyond the length of the payloads.Compared to others we model also modifications of documents(edit+, edit#) in addition to add and delete of complete doc-uments (add, del) as this can lead to more efficient protocolswith reduced leakage.

The correctness definition for SSE requires the server tolearn the ids of the results. One could define correctness torequire the client to learn the ids instead. The two approachesare essentially equivalent assuming that encrypted documents

3

are of fixed length.

SECURITY. Security [5], [7], [15] follows the real/ideal simula-tion paradigm and is parametrized by a leakage function L thatdescribes what a secure protocol is allowed to leak. Formally,L is a stateful party in an ideal security game, which is definedbelow.

Definition 1: Let ! = (Setup, Search,Update) be a dy-namic SSE scheme and let L be a leakage function. Foralgorithms A and S, define games Real

!A(!) and Ideal

!A,S(!)

as follows:

Real!A(!): A(1

!) chooses DB. The game then runs (K,EDB)# Setup(DB) and gives EDB to A. Then A repeatedlyrequests to engage in the Search or Update protocols, whereA picks a client input in. To respond, the game runs the Searchor Update protocol with client input (K, in) and server inputEDB and gives the transcript to A (the server is deterministicso this constitutes its entire view). Eventually A returns a bitthat the game uses as its own output.

Ideal!A,S(!): A(1!) chooses DB. The game runs EDB #

S(L(DB)) and gives EDB to A. Then A repeatedly requeststo engage in the Search or Update protocols, where A picks aclient input in. To respond, the game gives the output of L(in)to S, which outputs a simulated transcript that is given toA. Eventually A returns a bit that the game uses as its ownoutput.

! is L-secure against adaptive attacks if for all adversariesA there exists an algorithm S such that

Pr[Real!A(!) = 1]$ Pr[Ideal!A,S(!) = 1] % neg(!).

We define L-security against non-adaptive attacks in the sameway, except that in both games A must choose all of its queriesat the start, L takes them all as input, and S uses the outputof L to generate EDB and the transcripts at the same time.We also obtain adaptive and non-adaptive security definitionsfor static SSE schemes by disallowing adversary queries forUpdate.

DATA STRUCTURES. Our constructions will employ the stan-dard data structures of lists, arrays, and dictionaries. Weformalize a dictionary data type in detail because its syntaxis relevant to our security analyses. Below, when we saylabel,data, or data structure, we mean bitstring and will treatthem as such in the analysis.

An dictionary implementation D consists of four algorithmsCreate,Get, Insert,Remove. Create takes a list of label-datapairs ("i, di)mi=1, where each label is unique, and outputsthe data structure #. On input # and a label ", Get(#, ")returns the data item with that label. On input # and (", d),Insert(#, (", d)), outputs an updated data structure, that shouldcontain the new pair. On input # and ", Remove(#, ") outputsan updated data structure with the pair removed.

We define correctness in the obvious way, i.e., the outputof Get is always the data with the (unique) label it is givenas input, and that it returns & when no data with the label ispresent.

We say that a dictionary implementation is history-independent if for all lists L the distribution of Create(L)depends only on the members of L and not their order in thelist. The Create algorithm may be randomized or deterministicand satisfy history-independence. This simplest way to achieveit is to sort L first, but for large lists this will be infeasible.

We note that we only need the ability to remove data insome limited uses of dictionaries. In all settings were we needa very large dictionary, we can use an add-only version of thedata structure.

EXTENSIONS AND GENERALIZATION. Two works [3], [13]showed that data structures for single-keyword SSE can begeneralized to work for more complex SSE functionalitiesand models. Specifically, [3] shows how to extend SSE datastructures to perform boolean queries on encrypted data (viathe OXT protocol), and [13] further extends this functionalityto more complex multi-user SSE settings. As a result, allthe data structures in this paper, including their associatedoperations, their extreme efficiency and dynamic (updatable)capabilities, can be readily used to support these richer func-tional settings. All that is needed is to extend the data stored inthese data structures from simple document identifiers (in thebasic SSE case) to other associated data such as an encryptedkey in the case of multi-client SSE (a key used by clients todecrypt documents) or protocol-specific values (such as the‘y’ value in the OXT protocol from [3]). As a consequence,our advancement on the practicality and scale of SSE datastructures immediately translates into the ability to supportvery large and dynamic databases even for functionalities asinvolved as full boolean SSE search in single- and multi-clientSSE settings. We provide concrete evidence of this practicalimpact in Section V-E where we report performance numberson query execution in these complex settings.

III. STATIC CONSTRUCTIONS

Let D = (Create,Get, Insert,Remove) be a dictionaryimplementation, F be a variable-input-length PRF, and " =(Enc,Dec) be a symmetric-key encryption scheme.

BASIC CONSTRUCTION. In Figure 2 we give our first andsimplest construction, denoted !bas. To build the encrypteddatabase, Setup(DB) chooses a key K and uses it to deriveper-keyword keys for a PRF (to derive pseudorandom labels)and for encryption (to encrypt the identifiers). Then for eachkeyword w, it it iterates over the identifiers in DB(w). Foreach identifier, it computes a pseudorandom label by applyingthe PRF to a counter, encrypts the identifier, and adds thelabel/ciphertext pair to a list L. After all of the results havebeen processed it builds the dictionary # from L, whichbecomes the server’s index. It is important that L is sortedby the labels before being loaded into the dictionary, or thatthe dictionary satisfies history independence - Without one ofthese, the scheme will leak information about the order inwhich the input was processed.

To search for keyword w, the client re-derives the keys forw and sends them to the server, who re-computes the labelsand retrieves and decrypts the results.

LEAKAGE FUNCTION. The leakage function L for our firstconstruction responds to an initial startup query, and to search

4

Setup(DB)

1. K$

# {0, 1}! allocate list L2. For each w !W :

K1 # F (K, 1'w), K2 # F (K, 2'w)Initialize counter c# 0For each id ! DB(w) :

"# F (K1, c); d# Enc(K2, id); c++Add (", d) to the list L (in lex order)

Set # # Create(L)3. Output the client key K and EDB = #.SearchClient: On input (K,w),

K1 # F (K, 1'w), K2 # F (K, 2'w)Send (K1,K2) to the server.

Server: For c = 0 until Get returns &,d# Get(#, F (K1, c)); m# Dec(K2, d)Parse and output id in each m

Fig. 2. Scheme !bas.

queries, where its behavior is defined as follows. We describethe interactive stateful leakage function for the adaptive defini-tions; The non-adaptive leakage function is the obvious versionthat iterates over the queries with the adaptive leakage function.On initial input DB, L outputs N =

!

w"W|DB(w)|, saves

DB and an empty list Qsrch as state. Then, for a search inputw, L increments i, adds (i, w) to Qsrch and outputs DB(w)and a set sp(w,Qsrch), called the search pattern for w, definedby

sp(w,Qsrch) = {j : (j, w) ! Qsrch}.The search pattern indicates which other queries were also for

the keyword w, and represents the fact that our scheme willsend the same message when a search is repeated.

We deal with non-adaptive L-security first.

Theorem 2: !bas is correct and L-secure against non-adaptive attacks if F is a secure PRF and (Enc,Dec) is RCPA-secure.

Proof sketch: Without loss of generality we assume an ad-versary never repeats a query, as this obviously will not helpbecause our search protocol is deterministic. Correctness holdsbecause collisions amongst the labels produced by F occurwith negligible probability in a random function and hencealso with the PRF F . To prove non-adaptive security we mustgive a simulator that takes as input the leakage output settingup DB (which is N ) as well as the leakages on each of thequeries (which are just the sets DB(wi) of identifiers matchingthe i-th query, assuming all queries are unique). We need toproduce the view of the server, which is the EDB data structurealong with the messages from the client from each search.

The simulator iterates over the queries, choosing keys

Ki1,K

i2

$

# {0, 1}! for the i-th query, and then for eachid ! DB(wi) it computes " and d as specified in the realSetup (using Ki

1 and Ki2 as K1 and K2), adding each of

the pairs to a list L. Then it adds random label/data pairsto L (still maintained in lexicographic order) until it has Ntotal elements, and creates a dictionary # = Create(L). Thesimulator outputs EDB = # and the transcript for the i queryis (Ki

1,Ki2).

A simple hybrid argument shows that the simulator’s outputis indistinguishable from the real server view. The first hybridshows that selecting each K1,K2 at random is indistinguish-able from deriving them from K, by the PRF security of F .The next hybrid shows that the labels and data ciphertexts forunqueried keywords are pseudorandom. The only subtlety isto verify that, because the list is maintained in lexicographicorder, the relationship between the unopened result sets is notneeded. !

It is easy to prove secure (via a reduction to the abovetheorem) a version of !bas that does not maintain a sorted listL, as long as the dictionary is history-independent.

In the random oracle model we can achieve adaptivesecurity for the same L if F is replaced with the random oracleH so F (K,x) := H(K'x), and the encryption algorithm Enc,on inputs K,m ! {0, 1}!, chooses a random r ! {0, 1}! andoutputs (r,H(K'r)(m). We denote this variant !ro

bas.

Theorem 3: !robas is L-secure against adaptive attacks in

the random oracle model.

Proof sketch: This theorem is proved in a similar way to theprevious one, except that the simulator programs the randomoracle in response to adaptive queries to open the labeledciphertexts to match the query results as they are revealed.For our version of the PRF and encryption scheme above,the simulator can arrange for the random oracle responses topoint at random labels, and for the ciphertexts to decrypt tothe revealed results. The only defects in the simulation occurwhen an adversary manages to query the random oracle with akey before it is revealed, which can be shown to happen withnegligible in ! probability. !

ALTERNATIVE APPROACH TO ADAPTIVE SECURITY. Wesketch how to modify our protocol to achieve adaptive securitywithout a random oracle at the cost of extra communication.We choose the encryption scheme for the scheme to be of theone-time pad form e.g. CTR mode with a random IV. Nowinstead of sending the keys K1 and K2, the client computesthe labels and encryption pads herself and sends them to theserver, who can retrieve the labels and perform the decryption.In general the client will not know when to stop, but we caneither have the client retrieve a server-stored encrypted counterfirst, or have the server send a “stop” message when all ofthe results have been found. Note that the required additionalcommunication is proportional to the size of the result-set andcan overlap the disk access as well as the return of results.Hence, the resulting scheme should perform in practice as goodas the prior schemes.

ASYMPTOTIC EFFICIENCY. The EDB consists of a dictionaryholding N =

!

w"W|DB(w)| identifier/ciphertexts pairs.

Searching is fully parallelizeable if the dictionary allowsparallel access, as each processor can independently computeF (K1, c) and retrieve/decrypt the corresponding ciphertext.

RELATION TO [5] A prior SSE scheme by Chase and Kamaraused a dictionary, but in a crucially different way. There, asingle label was associated with the entire set DB(w), and thussecurity requires padding all of the result sets to the maximumsize. We instead associate one label with each result for akeyword (so if there are T documents with a keyword, then

5

there are T independent labels in our scheme but only 1 labelin the Chase-Kamara scheme). This allows us to avoid paddingand enable parallel searching, resulting in significant storagesavings and performance gains on large datasets.

A. Efficient extensions

We give a sequence of three schemes (denoted!pack,!ptr,!2lev, with names explained below) that exhibitthe techniques in our most practical !2lev construction.

REDUCING DICTIONARY RETRIEVALS: !pack . During asearch for w, our basic construction performs |DB(w)| re-trievals from the dictionary, each with an independent andrandom-looking tag. Even an external-memory efficient dic-tionary will perform relatively poorly when the dictionary isstored on disk.

Most prior schemes suffer from this drawback. To improvelocality we modify the basic construction to encrypt severalidentifiers in each ciphertext. Specifically, we fix a block sizeB, and when building the results list, we process B identifiersat a time and pack them into one ciphertext d, with thesame tag. We pad the last block of identifiers up to the samelength. Searching proceeds exactly as before, except the serverdecrypts and parses the results in blocks instead of individually.We denote this variant !pack.

This reduces the number of disk accesses from |DB(w)|dictionary retrievals to )|DB(w)|/B*. We can prove securityagainst non-adaptive or adaptive attacks under the same as-sumptions, but with the leakage function LB that starts byoutputting

!

w"W)|DB(w)|/B* instead of

!

w"W|DB(w)|.

We note that this leakage is incomparable to the originalleakage (see the discussion at the end of this section).

Theorem 4: !pack is correct and LB-secure against non-adaptive attacks if F is a secure PRF and (Enc,Dec) is RCPA-secure.

The proof is a simple extension of the proof for the basicconstruction. The essential observation is that the server onlyneeds to know how many of the packed blocks to createin the encrypted index. Similar to before, we can achieveadaptive security in the random oracle model or by increasingcommunication. We defer the details to the full version.

FURTHER REDUCTION VIA POINTERS: !ptr . !pack would beinefficient when returning very large sets DB(w), as the serverstill performs )|DB(w)|/B* dictionary retrievals. Making Blarge results in too much padding when the dataset containsmany keywords only appearing in a few + B documents.

We address this tension by modifying the scheme again,calling the new variant !ptr. !ptr packages the identifiers intoencrypted blocks of B as before, but it stores these blocks inrandom order in external memory and not in the dictionary(technically, we say they are stored in an array). The schemewill now use the dictionary to store encrypted blocks of bpointers to these encrypted blocks. To search, the server willretrieve the encrypted pointers from the dictionary and thenfollow them to the encrypted identifiers.

!ptr is described in Figure 3. In this scheme, the EDB con-sists of a dictionary # holding encrypted blocks of b pointers

Setup(DB)

1. K$

# {0, 1}!; allocate array A, list L2. For each w !W :

Set Ki # F (K, i'w) for i = 1, 2T # )DB(w)/B*Partition DB(w) into B-blocks I1, . . . , ITPad IT up to B entries if neededChoose random empty indices i1, . . . , iT in AFor j = 0, . . . , T (store id blocks in array)

d# Enc(K2, Ij); Store A[ij ]# dT $ # )T/b*Partition {i1, . . . , iT } into b-blocks J1, . . . , JT !

Pad JT ! up to b entries if neededFor c = 0, . . . , T $(store ptr blocks in dict.)

"# F (K1, c) ; d$ # Enc(K2, Jc)Add (", d$) to L

Set # # Create(L)3. Output the client key K and EDB = (#, A).

SearchClient: On input (K,w),

K1 # F (K, 1'w), K2 # F (K, 2'w)Send (K1,K2) to the server.

Server: For c = 0 until Get returns &,d# Get(#, F (K1, c))(i1, . . . , ib)# Dec(K2, d)For j = 0, . . . , b (ignore padding indices)

m# Dec(K2, A[ij ])Parse and output ids in m

Fig. 3. Scheme !ptr.

and an array A holding blocks of B encrypted identifiers for agiven keyword, where b and B are block size parameters to bechosen. The setup algorithm stores blocks of encrypted resultsin random locations in A, and then stores encrypted pointersto those locations in #, with labels that allow retrieval similarto the prior variants.

In the full version we show that this variant achieves thesecurity for the leakage function Lb,B which initially outputs!

w"W)|DB(w)|/B* and

!

w"W)|DB(w)|/(bB)*, which are

the number of blocks in # and A respectively, and laterleakages are just the identifiers as before.

MOST PRACTICAL VARIANT: !2lev . In real data sets thenumber of records matched by different keywords will varyby several orders of magnitude. This presents a challenge inoptimizing our variants, and we could not find a setting of Band b that gave an acceptable trade-off between index size (dueto padding) and search time. Small sets DB(w) still resulted ina large block of size B in the dictionary and a large block ofsize b in the array, while huge sets still required many retrievalsfrom the dictionary.

Thus we again modify the scheme to extend the ideasbefore, calling the new variant !2lev. The crucial difference isthat sets DB(w) can be processed and stored differently basedon their sizes, with an extra level of indirection for very largesets that explains the name. Care must be taken to do this withan acceptable form of leakage.

Below we describe the !pack variant formally. At a high

6

Setup(DB)

1. K$! {0, 1}! allocate list L, array A

2. For each w "W

Set Ki ! F (K, i#w) for i = 1, 2T ! $|DB(w)|/B%If |DB(w)| & b (small case: ids in dict.)

Pad DB(w) to b elements!! F (K1, 0); d! Enc(K2,DB(w))Add (!, d) to L

If |DB(w)| > b (medium/large cases)Partition DB(w) into B-blocks I1, . . . , ITPad IT up to B elementsChoose random empty indices i1, . . . , iT in AFor j = 1, . . . , T (store ids in array)d! Enc(K2, Ij);A[ij ]! d

If T & b (medium case: ptrs in dict.)Pad {i1, . . . , iT } to b elements!! F (K1, 0); d! ! Enc(K2, i1# · · · #ib)Add (!, d) to L

If b < T & bB (large case: ptrs in array & dict.)T ! ! $T/B%Part. {i1, . . . , iT } into b-blocks J1, . . . , JT !

Pad JT ! to B elementsChoose random empty indices i!1, . . . , i

!

T ! in AFor j = 1, . . . , T !

d! Enc(K2, Jj);A[i!j ]! dPad {i!1, . . . , i

!

T !} to b elements!! F (K1, 0); d!! ! Enc(K2, i

!

1# . . . #i!

b)Add (!, d!!) to L

3. " ! Create(L)Output the client key K and EDB = (", A).

Fig. 4. Setup for SSE Scheme !2lev.

level, it works as follows. It classifies the sets DB(w) as small,medium, or large. For small sets, it will store the identifiersdirectly in the dictionary (so no pointers are used, similar to thepacked variant !pack). For medium size sets, it will store themas in !ptr, with a block of pointers in the dictionary and thenblocks of identifiers in the array. Finally large sets are storeddifferently, with two levels of indirection: The dictionary isnow used to hold pointers that point to blocks of pointers inthe array, which point to the identifier blocks.

In !2lev we again fix parameters b and B to be sizes ofblocks in an dictionary # and array A respectively. The schemeclassifies each of the result sets DB(w) with |DB(w)| % b assmall, sets of size b < |DB(w)| % Bb as medium, and finallysets of size Bb % |DB(w)| < B2b as large. We will alwaysset b, B so that no set is larger than B2b.

Small sets fit completely in a block of the top-leveldictionary #, and are stored there. Medium sets will be storedas in the previous variant but with a single block of at mostb pointers in # and the corresponding blocks of identifiers inA. These sets consist of between b+ 1 and bB identifiers.

Finally, for large sets we store a block of at most b pointersin #. In each of the b positions pointed to in A, we store anotherblock of at most B pointers to other positions in A. Finally,these pointers point to blocks of encrypted identifiers. Figure 4describes the Setup(DB) function of !2lev in more detail.

To search, the client works as with the other variants bysending K1,K2. The server computes the label "# F (K1, 0),and retrieves d # Get(#, "), and decrypts d using K2. If it

finds identifiers here, then it outputs them and stops. Other-wise, it uses the pointers to retrieve blocks from A. If thoseblocks contain identifiers then it outputs them. Otherwise itfollows the next level of pointers to finally find the identifiers,which it decrypts and outputs.

SECURITY. We prove security for the leakage function Lm,b,B

that initially outputs m = |W| and the value

S =#

w:|DB(w)|>b

)|DB(w)|/B*+#

w:|DB(w)|>bB

$

|DB(w)|/B2%

.

The value m is the number of data items in #, and the value Sis the number of blocks in A. This is leaking S itself, which isdefined by the above sum, and not the individual summands,resulting leakage that is incomparable to our other variants andto prior schemes. On search queries L has the same leakageas before.

Theorem 5: !2lev is correct and Lm,b,B-secure againstnon-adaptive attacks if F is a secure PRF and (Enc,Dec) isRCPA-secure.

We can prove adaptive security either in the random oraclemodel or by increasing communication. We defer this simpleextension to the full version.

POINTERS VS. IDENTIFIERS. Although pointers are smallerthan identifiers in our implementations, !2lev packs the samenumber of pointers or identifiers together (b in the dictionary,or B in the array) to simplify the presentation. The actual im-plementation packs more pointers into a block than identifiers.Formally, we introduce parameters b$, B$, and modify !2lev asfollows.

• When storing identifiers in the dictionary (in the smallcase), it packs up to b of them together, but when storingpointers there it packs b$ in the same amount of space.

• When storing identifiers in the array (in the medium andlarge cases), it packs up to B of them together, but whenstoring pointers there it packs B$ together in the sameamount of space.

This causes an analogous change to the value S leaked, whichcan be calculated similarly. We defer the formal analysis(which is almost identical to that of !2lev) to the full version.

LEAKAGE DISCUSSION. The leakage functionsLB ,Lb,B ,Lm,b,B are non-standard. First consider LB , andhow it compares to L which outputs N =

!

w"W|DB(w)|.

Any input DB with m unique keywords, each with|DB(w)| % b, will be indistinguishable under LB , butmany of them will not be under L. A similar incomparabilitygoes in the other direction. We are not aware of a scenariowhere this difference is important for reasonably small B.The function Lb,B leaks strictly more information than LB

(actually Lb), but it also does not appear to be harmful.Finally, Lm,b,B leaks this type of size information and thenumber of keywords m. The number m seems to be themost useful information for an adversary, but in prior work ithas been considered acceptable. It is possible to modify thescheme to avoid leaking exactly m, say by storing blocks ofa different size in the dictionary.

7

IV. DYNAMIC CONSTRUCTIONS

We extend our static SSE constructions to support changesto the database. Our dynamic SSE constructions will consist ofa statically encrypted database EDB using any of the schemesdescribed above, and an auxiliary encrypted database EDB+

which is maintained to be of the form of a basic dictionary-based scheme. The EDB+ is initially empty and changes asupdates happen.

ADD-ONLY SCHEME: !+bas . We start with an extension of

!bas, denoted !+bas that supports additions only, meaning

add, edit+ inputs from the client during Update. !+bas is

simpler and possibly interesting in its own right.

To support additions we use a dictionary #+ which isinitially empty and to which a pair (", d) is added with eachkeyword addition; here " is a label computed from the keywordand a keyword-specific counter, and d is the encryption ofthe record id involved in the addition operation. Search for akeyword w is performed by the server by first searching # asin the static case, then re-computing all labels correspondingto w in #+. The latter labels are computed using a w-specifickey provided by the client and a running counter.

Note that addition operations involving keyword w requirethe client to know the current value of the w-specific counter.For this, the scheme maintains a dictionary $ associatingeach keyword that was ever added via edit+ or add with itscurrent counter value. $ can be stored at the client or storedat the server and retrieved by the client for performing updateoperations. We formalize a scheme !+

bas where the client storeslocally the dictionary $ and discuss below a stateless variant.We assume throughout that the client never tries to add arecord/keyword pair that is already present - it is easy, butmessy, to extend our scheme and the leakage profile to handlethis.

In !+bas, Setup(DB) is exactly as in !bas except that the

client also initializes $ to be an empty dictionary and keeps itas state, and the server initializes an empty dictionary #+ thatis stored with EDB. We next give the update protocol.

Update: We only specify the protocol with client input op !{add, edit+}. The parties work exactly the same on eithertype of operation. The client has input id,Wid. It starts byderiving key K+ # F (K, 3)2, and proceeds as follows:For w !Wid:

Set K+1 # F (K+, 1'w), K+

2 # F (K+, 2'w).c# Get($, w); If c = & then c# 0Set "# F (K+

1 , c) ; d# Enc(K+2 , id)

c++ ; Insert (w, c) into $Add (", d) to L in lexicographic order

Send L to the server.

When inserting (w, c) into $, we assume that it will overwriteany previous entry (w, ·) if it exists.

Finally, the server adds each (", d) ! L to #+. Thiscompletes the update protocol.

To complete !+bas we describe the protocol Search.

2We use input 3 for domain separation to K+ make independent of eachkeyword-specific K1 = F (K, 1"w) and K2 = F (K, 2"w).

Search: On input w, the client sets K+ # F (K, 3) andproceeds:

K1 # F (K, 1'w),K2 # F (K, 2'w)K+

1 # F (K+, 1'w),K+2 # F (K+, 2'w)

Send (K1,K2,K+1 ,K+

2 ) to the server.

Upon receiving the message, the server computes its output asfollows:

For c = 0 until Get returns &,d# Get(#, F (K1, c)) ; m# Dec(K2, d)Parse and output id in each m

For c = 0 until Get returns &,d# Get(#+, F (K+

1 , c)) ; m# Dec(K+2 , d)

Parse and output id in each m.

Intuitively, the server is repeating the search procedure from!bas twice: Once with (K1,K2) and #, and then with(K+

1 ,K+2 ) and #+.

LEAKAGE PROFILE FOR !+bas . Let us first give some intuition

for the leakage of !+bas. Initially the leakage is exactly like

!bas, where only the size of DB is leaked. Upon an edit+ oradd query, if the keywords being added were not previouslysearched for, then the server learns nothing other than numberof record/keyword pairs added (not even the if the operationwas edit+ vs. add). If, however, one (or more) of the keywordswere previously searched for, then the server can reuse its keysfrom before to detect the presence of these keywords (this typeof leakage is inherent when the keys provided to the server forsearching are deterministically generated and the same eachtime). The leakage on a search is similar to before, exceptnow for record/keyword pairs in #+ the server can recognizewhen they were added. The order for pairs in # generated atsetup time is still hidden, however.

We proceed with the formal definition of L+ for adaptivesecurity. Amongst its state, it will keep a list Q describingall queries issued so far, where an entry of Q is of the form(i, op, . . .), meaning a counter, the operation type, and then theone or more inputs to the operation.

On initial input DB, L+ creates a state consisting of acounter i# 0, an empty list Q and DB, and a set ID initializedto contain all of the identifiers in DB. Let us define the searchpattern sp(w,Q) of a keyword with respect to Q to be theindices of queries that searched for the keyword w, i.e.

sp(w,Q) = {j : (j, srch, w) ! Q}.

For an identifier id and keyword w, the add pattern of id, wwith respect to Q is the indices that added w to the documentid, i.e.

ap(id, w,Q) = {j : (j, add, id,Wid) ! Q,w !Wid}

, {j : (j, edit+, id,Wid) ! Q,w !Wid}.

Finally, we let the add pattern of keyword w with respect to Qand ID be the set of all identifiers to which w was ever added(via a add or edit+ operation) along with the indices showingwhen they were added. That is,

AP(w,Q, ID) = {(id, ap(id, w,Q)) : id ! ID, ap(id, w,Q) -= .}.

8

L+ produces outputs for the initial query, edit+ and addupdates, and search queries as follows:

• On initial input DB it saves state as defined above andoutputs N =

!

w"W|DB(w)|.

• For a search query w, L+ appends (i, srch, w) to Qand increments i. Then it outputs sp(w,Q), DB(w), andAP(w,Q, ID).

• Update queries for edit+ and add operations are han-dled similarly. For a query (edit+/add, id,Wid), L

+ firstappends (i, edit+/add, id,Wid) to Q, adds id to ID, andincrements i. It outputs |Wid| and the (lexicographicallyordered) set of search patterns

{sp(w,Q) : w !Wid}.

If any of the search patterns was non-empty, then it alsooutputs id.

While subtle in its formulation, L+ is essentially the bestpossible leakage for an SSE scheme that generates the samesearch keys on repeated searches.

In words, the search query leakage includes sp(w,Q)and DB(w) for obvious reasons. The add pattern of w,AP(w,Q, ID), is the set of id matching w added later alongwith “history” information ap(id, w,Q) indicating when theyadded. The order information represents that the server canlook at #+ and see when each id was added by rewinding andre-running searches. For updates !+

bas leaks only the size of theupdate if the added keywords have not been searched for. Ifany of them have been searched for, then the server learns that“a keyword with search pattern sp(w,Q) was added” via theset of search patterns in the update leakage. Finally it learnsthe id being updated because it has the ability to search forany of its keywords. Each of these leakage components isunavoidable for a deterministic SSE scheme, and we regardthem as minimal.

We can now state our security for the add-only scheme. Aproof will appear in the full version.

Theorem 6: !+bas is correct and L+-secure against non-

adaptive attacks if F is a secure PRF and (Enc,Dec) is RCPA-secure.

STATELESS CLIENT VARIANT. Above, the client keeps a dic-tionary $ containing one counter per keyword that is addedafter initialization. We could modify the scheme so that theclient is stateless by storing $ in encrypted form at the serverand having the client download and re-encrypt all of $ for eachupdate (note that the size of $ is as the number of distinctkeywords added via add and edit+ and not the total numberof keywords in the set W). In this variant the server will learnhow many new keywords are added each time by watching if$ grows. We leave a formal version of this scheme to the fullversion.

DYNAMIC SCHEME !dynbas . We now augment the !bas scheme

with del and edit# operations to obtain our fully dynamicscheme !dyn

bas . We will implement deletions by maintaining arevocation list and having the server discard results that havebeen revoked.

To delete a record/keyword pair (id, w) from the server’sstorage, the client will generate a pseudorandom revocationidentifier and send it to the server. During searches, the clientwill give the server a key that allows it to recompute revocationidentifiers, which it will then use to filter out deleted results.This complicates our addition protocols. To add a pair that waspreviously deleted, the protocol must “unrevoke” that pair byhaving the server delete its revocation identifier.

We now formally specify !dynbas . Setup is exactly the same

as !+bas, except that the server also initializes an empty set

Srev. As a data structure, Srev will support additions, deletions,and membership testing.

Update: We first describe how to handle client inputs

with op ! {del, edit#}. The client takes as input(del/edit#, id,Wid), and first derives a key K# = F (K, 4)3,and then computes

For w !Wid doK#

1 # F (K#, w), revid# F (K#1 , id)

Add revid to Lrev in lexicographic orderSend Lrev to the server.

The server receives Lrev and adds each revid to Srev. Thiscompletes Update for the del and edit# operations.

Next we define Update for op ! {add, edit+}. On input(add/edit+, id,Wid), the client performs a computation similarto the list L computation in !+

bas, except that it also includesthe appropriate revid values. It then awaits a response from theserver specifying which additions resulted in a true additionand which caused an “unrevocation”, and uses this informationto increment the correct counters.

In code, the client sets K# # F (K, 4) and does thefollowing:

For w !Wid:K+

1 # F (K+, 1'w) ; K+2 # F (K+, 2'w)

K#1 # F (K#, w)

c# Get($, w); If c = & then c# 0"# F (K+

1 , c) ; d# Enc(K+2 , id)

revid# F (K#1 , id)

Add (", d, revid) to L in lexicographic orderSend L to the server.

The server generates its response r ! {0, 1}|L| as follows.For the i-th pair (", d, revid) ! L in order, if revid ! Srev, itsets the i-th bit of r to 1 and deletes revid from Srev. Else, itclears that bit to 0 and adds (", d) to #. Finally, it sends r tothe client.

Now the client increments the counters for keywords corre-sponding to 0 bits in r. It processes the keywords w !Wid inorder of their labels in L. For the i-th keyword w in that order,if the i-th bit of r is 0 it computes c# Get($, w), incrementsc, and inserts (w, c) into $. This completes the update protocol.

The last component of !dynbas is the search protocol.

Search: On client input w, it sets K# = F (K, 4),K#1 =

F (K#, w), and then computes (K1,K2,K+1 ,K+

2 ) as in !+bas.

3As with K+ the input 4 is for domain separation only.

9

It sends (K1,K2,K+1 ,K+

2 ,K#1 ) to the server. The server

computes the result identifiers using the first four keys exactlyas in !+

bas, except before outputting each id it computesrevid = F (K#

1 , id) and tests if revid ! Srev. If so, it discardsid instead of outputting it.

LEAKAGE FUNCTION. We now define the leakage profileLdyn. It will maintain a list of query information Q and setof identifiers ID like L+ from above. Below we use the samedefinitions for sp, ap,AP as in L+, and define the followinganalogous patterns dp,DP for deletions:

dp(id, w,Q) = {j : (j, del, id,Wid) ! Q,w !Wid}

, {j : (j, edit#, id,Wid) ! Q,w !Wid}.

and DP(w,Q, ID) =

{(id, dp(id, w,Q)) : id ! ID, dp(id, w,Q) -= .}.

Intuitively, dp(id, w,Q) is the set of indices of queries thatdeleted w from id, and DP(w,Q, ID) is the set of identifiersform which w was deleted, along with the correspondingdeletion pattern.

• On first input DB, Ldyn initializes a counter i# 0, emptylist Q, set ID to be identifiers in DB. It saves DB, i, ID, Qas state, and outputs N =

!

w"W|DB(w)|.

• On search input w, Ldyn appends (i, srch, w) to Q, in-crements i, and outputs sp(w,Q), DB(w), AP(w,Q, ID),and DP(w,Q, ID).

• On update input (add/edit+, id,Wid), it appends(i, add/edit+, id,Wid) to Q, adds id to ID, and incrementsi. It outputs add, |Wid| and the set

{(sp(w,Q), ap(id, w,Q), dp(id, w,Q)) : w !Wid}.

Finally, if any of the sp(w,Q) are non-empty, then it alsooutputs id.

• On update input (del/edit#, id,Wid), it appends(i, del/edit#, id,Wid) to Q, adds id to ID, and incrementsi. Then it computes its output exactly as in the add/edit+

case above, except that it outputs del instead of add asthe first component.

The leakage on searches is minimal: It consists of all patternsof searches, deletions, and additions that can be derived oncethe server has the ability to search for a keyword and rewindthe database. For leakage on updates, the server will learnwhen/if that identifier has had the same keywords added ordeleted before, and also when/if the same keywords have beensearched for. This comes from observing the revid values,which will repeat every time the same identifier/keywordpair is added or deleted. Note that, if same keyword isadded/deleted from two documents, then this information isnot leaked until it is searched for (contrast this with [14] whichleaks this information always).

We prove the following theorem, as well as an adaptivevariant, in the full version.

Theorem 7: !dynbas is correct and Ldyn-secure against non-

adaptive attacks if F is a secure PRF and (Enc,Dec) is RCPA-secure.

ASYMPTOTIC ANALYSIS. To add a file the client sends onelabel/ciphertext/revid per record/keyword pair being changed.For deletions, the $ dictionary is not involved. The clientjust sends one revid per document/keyword to be deleted.Assuming the dictionaries #, #+, and the revocation list arefully read-parallel, and the number of deletions is much smallerthan the size of the EDB, each search operation continues tohave the same order run-time complexity as in the basic staticconstruction of Figure 2.

DISCUSSION AND COMPARISON TO PRIOR WORK. Ourscheme !dyn

bas is unsatisfying in some situations as it doesnot reclaim space after deletions. While this is a drawback,all known dynamic SSE schemes [14], [15], [21] have se-vere drawbacks in different dimensions, and no scheme hasachieved an ideal combination of leakage, index size, and fullfunctionality like reclaiming space.

The scheme of [21] has no security proof, and the schemeof [14] has a worst-case quadratic size encrypted index.The dynamic scheme in [15] has much more leakage thanour scheme, effectively leaking the pattern of all intersec-tions of everything that is added or deleted, whether or notthe keywords were searched for. For an example, suppose{w1, w2, w3} are added to id1, {w1, w2} are added to id2,and {w1} is added to id3. Then [15] will leak that exactly onecommon keyword was added to all three and that exactly twocommon keywords were added to the first two (but not thethird) and so on. This structural “equality pattern” is the sortof leakage that we do not leak.

Not reclaiming space allows our implementations to bemuch simpler and also gives us the flexibility to apply variousefficiency optimizations (as in section III A) to the staticscheme which seem hard to achieve when in-place updateshave to be supported. As our data structures are more compactthan prior work, the overall space requirements will be loweranyway for some number of deletes. In particular, as comparedto prior work [14] we are not forced to estimate an upper bound(by necessity, conservative) on the maximum database size.

In some settings where SSE is used as a component, theencrypted database is re-encrypted for security reasons [13]. Inthese settings we can reclaim space and combine the auxiliarydata structure with the main static data structure while re-encrypting.

APPLICATION TO !ptr,!pack,!2lev . The dynamic extensionsto !bas can be applied as-is to other variants, resulting inalmost the same leakage Ldyn. The only difference is the sizeleakage in the initial input DB, which changes according to thedifferent schemes. In our implementation in the next sectionwe consider these variants.

V. IMPLEMENTATION

We report on our implementations of !2lev and !pack

(described in Section III), with extensions for dynamic dataupdates (Section IV). The former scheme is the most efficientand scales to the largest datasets; it represents our currentprototype. The latter is a simplification of the original OXTimplementation which we introduced in [3] and is discussedhere to better illustrate the effectiveness of the ideas in !2lev

and the improvement over prior work.

10

PRACTICAL CRITERIA. Before describing our results, weenumerate some of the practical criteria that we optimize forin the !2lev prototype.

• Parallel EDB access: The server should be able to issueconcurrent access requests to EDB when processing asearch. Modern disk controllers handle thousands ofconcurrent requests and optimize disk access patterns,increasing transfer bandwidth by orders of magnitudewhen compared with sequential access. Requests areserved out-of-order but the performance benefits offsetthe additional implementation complexity.

• EDB goodput: EDB design should maximize I/O good-put, i.e., the ratio of data used by the processing of a queryrelative to the total amount of data retrieved from externalstorage. In addition to selecting an appropriate dictionary,we achieve this by setting the parameters b, b$, B,B$ in!2lev to take maximum advantage of the block device.

• Small EDB storage: The dictionary used in EDB shouldminimize storage overhead while satisfying the otherconstraints.

• Lightweight EDB updates: Update information will beindependent from the EDB and implemented in-memory.This is consistent with our envisioned scenarios whereupdates are either infrequent or periodically folded intothe main data structure via re-encryption of the entiredatabase.

INPUT DATASETS. Our implementation accepts as input bothrelational databases and document collections. The latter aremapped to relational database tables with document attributes,such as author name, creation date, etc., stored in atomiccolumns and with the document content stored in a textcolumn.

We target clear-text datasets (DBs) that consist of severaltens of billions of distinct (keyword, id) pairs. The EDBsgenerated from such datasets take several terabytes of storageand require several times more temp storage for Setup. We aimto process such datasets efficiently (Setup(DB) and Search)on medium size 64-bit x86 platforms (in our configuration, 8cores, 96GB of RAM, and / 100TB RAID volume on externalstorage box).

The constructions described in this paper and their im-plementations can be extended to support richer functionalsettings than simple keyword search, such as SSE in multi-client settings or boolean queries via the OXT protocol [3] (seeend of Section II), by storing in the EDB for each (keyword,id) pair more data than just the encrypted document id. Inthe following, we use the term tuple for the data stored per(keyword, id) pair in any of these functional settings.

ORGANIZATION. The next two subsections describe our ex-periences with the !pack prototype, which is the subset ofthe OXT implementation [3] relevant to this work, and thedesign and implementation of our !2lev (see Figure 4). Aparticular challenging issue for both prototypes was EDB gen-eration time; the Setup implementation for !2lev is discussedseparately in Section V-C. Section V-D describes how these

constructs are used to support richer functional settings, suchas OXT. Finally, Section V-E describes several representativeexperiments.

A. !pack Implementation

The discussion of the !pack implementation here is in-tended as a preamble to our presentation of !2lev in thenext subsection as it serves to motivate the optimizationsapplied to the latter construction. Our implementation of !pack

instantiates the EDB dictionary using a bucket hash table.Buckets are split in equal-size locations, which are used tostore equal-size groups of tuples created by partitioning theDB(w) sets. The location size is equal to the group size plusthe size of the attached label. Each group can be stored in anyof the free locations in the bucket determined by hashing itslabel. As usual, the hash map is over-sized to allow for allgroups to be placed successfully; empty locations are filledwith random bits to mask the total number of groups in theEDB.

Using a bucket hash for the dictionary allowed us to avoidsorting the tuples by label (as required for security) beforecreating the dictionary. This worked by ensuring the dictio-nary is history independent, meaning the output of Create(L)depends only on the members of L and not on the order theywere added to L.

The bucket hash table is stored in one large file on an ext4RAID partition of attached storage. The bucket size is set toa multiple of the RAID stripe size4, and buckets are alignedwith the pages of the underlying file system.

The two most significant drawbacks with the above con-struction are the need for oversizing the hash table, whichtranslates into a much larger EDB than needed, and the poorgoodput, as one have to retrieve an entire bucket to access agroup of tuples. In experiments with configurations and datasets similar to those described in [3], the hash table has a loadfactor of / 60% (i.e., over-sized by a factor of / 1.6) for theplacement to be successful, and goodput is / 1%, as there are96 locations per bucket.

To achieve a higher load factor (smaller EDB), we builtanother !pack prototype which uses a Cuckoo Hash (CH)table modeled after [8] for the dictionary; page size andalignment are the same as for the bucket hash dictionary inthe previous construction. Although we achieve load factors alittle over 90%, the cost of handling collisions during EDBgeneration is very high. Moreover, making the dictionaryhistory independent is much more difficult when using a CHtable and likely inefficient in our setting.

We designed a more efficient algorithm to handle collisionsduring EDB generation, which leverages the server memory,but we found its performance to be limited by its databaseaccess patterns (see Section V-E). Finally, the need to improvethe goodput motivated the design of !ptr and !2lev.

B. !2lev Implementation

In order to meet the criteria stated at the beginning of thissection and avoid the drawbacks of !pack, we developed the

4Stripe is the smallest amount of data that can be addressed within theRAID. This is functionally equivalent to a block for an individual disk.

11

!2lev construction (see Figure 4) which uses different databasepatterns to speed-up Setup, can be configured to run Setupefficiently on platforms with a wide range of internal memory,and supports much faster retrieval as a result of higher goodput.

Recall that in !2lev, the EDB consists of two data struc-tures: a dictionary # and an array A. The dictionary is againimplemented as a bucket hash, but now with exactly onelabeled location per keyword w. The bucket address andlocation label are derived from w, but the location within thebucket is selected at random to ensure history independence.A # location stores up to b tuples or b$ pointers, i.e. indicesin array A.

The second data structure is the array A whose entries arecalled tuple blocks. Setup uses tuple blocks to store tuples, ortuples and pointers, for medium or large DB(w), respectively.Each tuple block stores up to B tuples or B$ pointers, withB 0 b and B$ 0 b$ in most settings. In contrast to thedictionary #, which is a bucket hash, the array A needs not beover-sized except for the purpose of masking the total numberof tuples in EDB. Empty locations in # and A, if any, are filledwith random bits.

For all w with more than |DB(w)| > b, the tuple blocksused for DB(w) are allocated at random in the array usingan AES-based pseudorandom permutation and the tuple listin DB(w) is split into tuple blocks (see medium/large casesin Figure 4). For any given w, if the number of tuple blocksneeded to store DB(w) is larger than the number of pointersthat fit in a dictionary location, we use additional tuple blocksto store pointers (see large case Figure 4).

The dictionary # and the array A are realized as twoseparate files on the same or separate ext4 RAID partitions.The location, bucket and tuple block sizes are configurable,but for efficiency the bucket and tuple block sizes must be amultiple of the RAID stripe size. Similarly, buckets and tupleblocks must be aligned with the pages of the underlying filesystem.

In our experiments, we use a low single digit numberof tuples per location and 32KB or 64KB for buckets andtuple blocks. Pointers are 3 or 4 bytes long, depending on thesize of the array A, and tuples are between 16 and 91 byteslong, depending on the functional setting. For the documentcollections and relational databases in our experiments, thedictionary is between one and two orders of magnitude smallerthan the array.

Unpadded, the size of the dictionary leaks the approximatenumber of keywords while the size of the array leaks theapproximate number of EDB tuples. Therefore, masking thedictionary size, which is sensitive in many scenarios, is inex-pensive given its relative small size. Leaking the total numberof tuples is less sensitive, which means that the larger datastructure requires less or no padding in most common cases.

This construction has several important advantages forvery large datasets, in particular for those having multi-modaldistributions, e.g., some DB(w) sets that are very large anda very large number of very small DB(w) as commonlyencountered. For instance, for datasets of tens of millions ofdocuments, each English word that is not rare can be foundin millions or more documents. On the other hand, ISBN

or SSN values are associated with only one book or person,respectively, independent of how many books or persons thedataset describes.

!2lev can be configured to be disk-efficient in both ex-tremes. For rare keywords, configurations with small locationsizes, corresponding to a low single digit number of tuples,allow the search engine to retrieve all the relevant tuples withonly one disk access. Using a small location size helps reducethe dictionary size, potentially to less than the amount of theavailable RAM.

For the rest of the keywords, after one access (or a fewdisk accesses for very common keywords), the addresses ofall the relevant tuple blocks are known. At this point, thequery execution engine issues as many concurrent tuple blockrequests as the RAID controller can handle. After less thanthe average disk access time, because of the large numberof pending requests, tuple blocks are read at close to themaximum rate of the RAID controller. The rate at which tuplesare retrieved from storage determines the throughput of searchengine. Note that goodput is 100% when accessing tuple blocksfilled with tuples and that for frequent keywords, the goodputof a Search operation grows asymptotically to 100%.

In contrast, and by way of comparison, the !pack con-struction computes the location addresses of all their tuplegroups from the keyword value and a running counter. Thus itcan precompute a large number of group addresses and issuerequests for tuple groups immediately, i.e. no additional diskaccesses to retrieve pointers are needed. But without a prioriknowledge of DB(w) size, which is the common case, !pack

issues many more requests than necessary. Even worse, toachieve the lowest access latency for a CH-based construction,one always needs to issue two requests per expected tuplegroup, as the group can be stored in two positions (pages) inthe CH table. Finally, these disk accesses have low goodputas each bucket contains multiple tuple groups. Thus it appearsthat low I/O goodput is inherent to !pack. For large sets, thesuperior goodput of our construction (due to large tuple blocks)more than compensates for the extra initial storage access(es)for pointers.

For keywords with just a few tuples that fit in onedictionary location, the performance is the same. However,one could accelerate the performance of !2lev by storing thedictionary, which is relatively small even for large data sets,in main memory. Dictionaries used by previous work, whichuse one large bucket hash for all tuple sets, are too large forthis optimization.

The two-level !2lev construction allows for a very efficientEDB generation process. As an example, during the longestphase of EDB generation from a database with / 25 billion(w, id) pairs in the context of multi client OXT [3], whichtook 40 hours, all cores performed crypto operations at closeto 100% utilization while at the same time reading 100 millionrecords from a MySQL DB and writing to the file systemthe tuple blocks and the temp dictionary files. Overall, thetwo-level construction is much closer to our requirements thanany previous ones and the experimental results confirm ourexpectations.

12

C. EDB Generation

For our largest datasets, EDB is on the order of 2TB. ThusEDB generation time is sensitive to implementation details andis the dominant factor determining the practical scalability ofall our constructions. This section describes the parallel Setupalgorithm used in the !2lev prototype.

Before EDB generation starts we process the input filesinto an index of the form expected by !2lev. For each textcolumn ’t’ in the clear-text database table create a new ’text t’table with columns ind and word. For each clear-text recordand for each word ’xxxx’ in its text column, add the pair(id, xxxx) to ’text t’, where id is the unique identifier of theclear-text record. The number of pairs in ’text t’ is equal to thenumber of clear-text records multiplied by the average numberof words in column ’t’. At the end, we create an index on thecolumn ’word’, which is used during Setup to retrieve DB(w)for all w = (t, xxxx), where ’xxxx’ is a word in column ’t’.

Unfortunately, for our largest databases, ’table t’ is toolarge for the index to fit in RAM, which makes building theindex impractical. To overcome this obstacle, for each textcolumn ’t’ we create multiple tables ’text t nn’, such that(1) id-word pairs are somewhat evenly distributed across thenew tables, (2) all the pairs with the same ’word’ value are inthe same table, and (3) the new tables are small enough fortheir indexes to be built efficiently on our platforms. Note thatthe atomic columns of the original table can undergo a similartransformation if the number of records in the clear-text tableis too large for indexing.

EDB is generated in three phases. The first phase countsthe number of distinct keywords wi in the clear-text table andother statistics needed for sizing the dictionary # and array A(or for masking the sizes of these data structures if so desired).This phase uses full-index scans and takes a small fraction ofthe total EDB generation time.

For performance reasons, the dictionary #, realized as abucket hash, is partitioned in equal size groups of consecutivebuckets and its generation is split across the second and thirdphases. The tuple block array A is fully generated in the nextphase.

The second phase generates the tuples in DB(w), for allkeywords w = (i, val) using full-index scans on atomic col-umn i. For each text column ’t’, the newly created ’text t nn’tables are scanned. Columns are processed in parallel workerthreads, with approximately 1.5 workers per CPU core tohide database access latencies. For each value val such thatw = (i, val), the thread processing column i retrieves the allthe ids corresponding to the records with val in column iand applies a random permutation to the resultant id sequence(i.e., DB(w)). For each id in the permuted sequence, theworker generates tuple elements with the encrypted id (andthe additional tuple values rdk and y when implementing themore advanced features of the OXT protocol from [3]).

During this phase, each worker thread creates one temp fileper dictionary partition in which it stores the content of thelocations (tuples or pointers) assigned to any of the bucketsin the partition. For better performance, the location contentis buffered and appended to the partition file in 64KB datachunks. At the same time, for medium and large DB(w), the

worker threads create all the necessary tuple blocks in the arrayA (see Figure 4).

During the third phase, the dictionary # is created from thepartition files generated in the previous phase. Each partitionis constructed by a separate worker thread. Each worker threadreads the files generated by phase-two workers for its partition,merges their contents and creates the label and content of eachdictionary entry in the partition. Next, for each bucket in itspartition, the worker assigns the dictionary entries to randomlocations in the bucket, fills in empty locations with randombits, and writes the bucket to disk. For a typical census table,the dictionary file is almost two orders of magnitude smallerthan the tuple block file.

Note that for the creation of the dictionary, the file systemis accessed using append calls (to temp files in the secondphase) and sequential read calls (from the temp files in thethird phase), which are the most efficient ways to access largefiles.

However, worker threads still issue a large number ofrandom write calls during the second phase, with one call foreach tuple block generated. To reduce the disk head movementsassociated with these writes, we extended the parallel EDBgeneration algorithm to confine concurrent workers to a tupleblock window sliding across the array. As a result, tupleblocks that are closely located on the disk are generatednear simultaneously. This optimization reduces seek overheadsnoticeably for our largest EDBs.

During the third phase, threads issue another set of ran-dom writes when writing the dictionary buckets. These diskmovements generated by these writes do not represent a majorbottleneck because these writes are already clustered to thebucket hash partitions under constructions, which we alwaysselect in increasing order.

D. Complex Functional Settings

As already mentioned at the end of Section II, our con-structs can be used to support richer encrypted-search settingsthan SSE, such as those in [3], [13]. In particular, all (single-keyword) SSE schemes presented here can be used, almost‘out-of-the-box’, to instantiate the “TSet functionality” under-lying the OXT protocol in the above works. The main changeis on the size of tuples that increases in order to accommodateadditional OXT information such as the xind and y values (seeSection 3.2 of [3]), and the key to decrypt the actual documents(as required in multi-client settings [13]).

Storing larger tuples requires minor configuration changesbut no alteration of the original construct. More specifically,hash table buckets need to be made large enough to accom-modate enough entries for all the tuples to be inserted intothe table with high enough probability, i.e., without any of thebuckets overflowing.

Another challenge in more complex protocols, such asOXT, is for the server to efficiently perform a two partycomputation which takes in-order generated data by the clientand out-of-order the tuples, as retrieved from the disk bythe !pack or !2lev prototypes. Maximizing the throughput ofsuch a computation requires using complex buffer management

13

DB Name Records (w, id) pairs EDB size

CW-MC-OXT-1 408,450 143,488,496 69.6 GBCW-MC-OXT-2 1,001,695 316,560,959 99.8 GBCW-MC-OXT-3 3,362,993 1,084,855,372 242.4 GBCW-MC-OXT-4 13,284,801 2,732,311,945 903.9 GB

LL-MC-SKS-1 100,000 114,482,724 15.0 GBLL-MC-SKS-2 1,000,000 1,145,547,173 52.0 GBLL-MC-SKS-3 10,000,000 11,465,515,716 394.0 GBLL-MC-SKS-4 100,000,000 114,633,641,708 3,961.3 GB

TABLE I. DATABASES

algorithms that optimize the use of available RAM betweentokens and tuple block buffers.

E. Experimental Results

Our prototype implementation measures roughly 65k linesof C code, including test programs. Measurements reportedhere were performed on blades with dual Intel Xeon 2.4GHzE5620 processors having 4 cores each and running Linux.Storage consists of 6 1TB SAS disks configured as a RAID-0 with a 64KB stripe and attached via a 3 Gb/s to an LSI1064e SAN controller and formatted as an ext4 file system withan 8KB page size. Clear-text databases are stored in MySQLversion 5.5.

The experiments reported in this section use databasesderived from the ClueWeb Collection [18] or syntheticallygenerated by an engine trained on US-census data. The keyattributes of these databases and derived encrypted indicesare summarized in Table I. The CW-* databases were ex-tracted from the ClueWeb Collection while the LL-* databasesemulate the US-census data. Both database families containatomic type and text columns. The ClueWeb databases wereencrypted for a multi-client setting supporting conjunctions(OXT) [3] and the census database where processed for singlekeyword search (SKS), also in multi-client settings [13] (seeSection V-D).

As already mentioned, EDB generation is the dominantfactor determining the practical scalability of all our con-structions. The two plots called CW (PH) and CW (2L)

in Figure 5 show how long it takes to generate the EDBscorresponding to the four CW-* databases when using the!pack and !2lev prototypes, respectively.

The results clearly show the !2lev construction outper-forming the !pack one. The !pack prototype is consistentlyslower because its database access patterns are more expensivethan those of the !2lev prototype. For larger datasets, theperformance of the !pack prototype collapses as soon as itsRAM requirements, which are proportional with the databasesize, approach the available RAM. The !2lev prototype doesnot exhibit a similar pattern because its RAM requirementsare roughly proportional with the size of the database columnscurrently being processed.

In separate experiments with the !2lev prototype, prepro-cessing for the LL-* family of databases also proved to scalelinearly, with roughly a rate of 3µs per (w, id) pair for thelargest database and 8.9µs per (w, id) pair for the smallest one.This translates to roughly 92 hours for biggest LL-MC-SKS-4database as shown in Figure 6. This compares very favorably

0

1000

2000

3000

4000

5000

6000

7000

8000

0 5e+08 1e+09 1.5e+09 2e+09 2.5e+09 3e+09

Tim

e (m

in)

(w,id) pairs

CW (2L)CW (PH)

Fig. 5. ClueWeb09 Pre-processing: Scaling Database Size

to the experimental results published by Kamara et al [15],who report a cost of approximately 35µs per (w, id) pair on acomputing platform with similar capabilities.

10

100

1000

10000

1e+08 1e+09 1e+10 1e+11 1e+12

Tim

e (m

in)

(w,id) pairs

End-to-end

Fig. 6. LL SKS Pre-processing: Scaling Database Size

Measurements on query performance are shown in Figure 7for queries CW-* databases with varying result set sizes andfor both constructions. The graph demonstrates even moredramatic improvements for query processing compared to pre-processing as the !2lev construction outperforms the !pack

one by almost two orders of magnitude for queries returning1% of the database.Experiments with the !pack prototypereturning 13% of the database are much slower and they werenot included in the figure to improve the visibility of theexisting curves. Experiments with OXT demonstrate similarperformance gains on conjunction queries. This illustrates thateven though OXT performance seemingly is dominated byexponentiation costs (see [3] for the details), optimizing disk-resident data-structures are crucial for good performance dueto the high I/O latency costs.5

5Using highly optimized (common-base) NIST P-224 exponentiation andmulti-threading, we can achieve almost 500,000 exponentiation/sec on thementioned test bed. The storage system provided only, depending on blocksize, 500-1,500 random I/O requests/sec and single request latencies is around10ms.

14

Figure 8 shows the execution times of two queries returninga constant, i.e., independent of the size of the input dataset,result set of 10 and 10,000 record ids, respectively. The gapbetween the two lines corresponding to the !pack prototype ismuch larger than the gap between the lines for correspondingto the !2lev prototype. The difference between the disk layoutsof the two constructs help explain the difference. To retrievethe 10 or the 10,000 ids, the !pack prototype needs to accessone or one thousand hash table buckets, respectively, whichmeans it will issue one thousand times more disk accessrequests for the second query. In contrast, for the same twoqueries, the !2lev prototype needs to access one dictionaryentry in both cases and one or eleven tuple blocks, whichmeans it will issue only six times more disk access requestsfor the second query. !pack hash table buckets and the !2lev

dictionary buckets and tuple blocks are all 64KB but tuplegroups store only ten tuples in this !pack prototype whiletuple blocks store a little under one thousand tuples. Notethat since !pack and !2lev experiments are using SKS andOXT, respectively, the gap between the 2L and PH plots forexperiments returning 10 tuples is explained by the initialcomputational overhead of OXT.

100

1000

10000

100000

1e+08 1e+09 1e+10

Tim

e (m

s)

Database size as total number of (w,id) pairs

PH: 10,0002L: 10,000

PH: 1% of db2L: 1% of db

2L: 13% of db

Fig. 7. Clueweb09 SKS Query: Scaling Database Size, comparing !pack vs!2lev for various result set sizes.

1

10

100

1000

10000

1e+08 1e+09 1e+10

Tim

e (m

s)

Database size as total number of (w,id) pairs

2L: constant small (10) result set2L: constant medium-size (10,000) result set

PH: constant small (10) result setPH: constant medium-size (10,000) result set

Fig. 8. Clueweb09 SKS Query: Scaling Database Size, comparing !pack vs!2lev for constant (10 and 10,000) result set sizes.

Lastly, to illustrate how space efficient the !2lev construc-tion is, we achieve load-factors of 58.5% for the bucket hashdictionary, 91.9% for the much larger array and 91.6% overall

for our largest LL-MC-SKS-4 database. The load-factor ofthe array A is less than 100% because although all its entriesare used for tuple blocks, some of these tuple blocks storepointers or are only partially filled with tuples.

F. Comparison with Prior Implementations

The only previous work that stores the encrypted index onexternal storage is [3], which uses a construction similar toSSE-2 in [7] but adapted to external storage. It corresponds,roughly, to our !pack prototype discussed in Section V-A.The other previous works assume that the encrypted indexis stored in main memory and that access to the index datastructure is uniform (i.e., constant cost/latency). None ofthese constructions admit straightforward extensions to ’blockdevice’-resident data structures that can be practical. This isparticularly the case for constructions using linked lists, suchas SSE-1 in [7] or its extension to dynamic SSE in [15].

Recent work by Kamara et. al in [14] discusses an ex-tension of their main memory index to a storage-resident B-tree. This system suffers from using a large index (worst-case quadratic) and their achieved CPU parallelism does notautomatically translate to efficient I/O parallelism given thedifferent characteristics of storage sub-systems. The work of[14] does not measure implementation performance and it doesnot discuss how it would address the I/O challenges facedwhen building a large scalable system. In contrast, we identifyparallel I/O access from the outset as one of the most impor-tant performance requirements for scaling to large encryptedindexes. In addition, we also achieve excellent CPU parallelismduring search because we parallelize our application-level im-plementation and because a large fraction of the I/O code pathis run in parallel kernel threads. We also validate our approachwith experimental results, paramount given the intricacies ofstorage sub-systems. Finally, our construction does not requirea fixed keyword set and is asymptotically faster by log n thanthe tree construction in [14], as we use hash instead of treeindexing.

VI. CONCLUSIONS

The tension between security and performance require-ments for SSE systems pose non-trivial challenges for both thecryptographic design and the data structures needed to supportthis design, as well as for their implementation. Leakageminimization calls for randomization of data locations in theencrypted database, EDB, in order to obscure any relationsin the original clear-text database. This results in the need torandomize access to EDB elements even when these elementsare logically correlated (e.g., the set of documents containinga given keyword). This random-order access is affordable forRAM-resident EDB but becomes prohibitive for disk-residentones; on the other hand, restricting an implementation to aRAM-resident EDB means limiting the database sizes onecan support. Thus, much of the work reported here, bothat the abstract data structure level and the specifics of theirimplementation, are driven by the need to bridge over thissecurity-performance conundrum, and is intended to find abalance between randomized ordering of data, locality ofaccess and parallel processing. In particular, our two-levelscheme seems to achieve a desirable trade-off between thesecompeting requirements.

15

As a result we are able to demonstrate the practicality ofsearch on encrypted data for very large datasets (at terabytes-scale and 10s of billions of record-keyword pairs) and withstrong security guarantees. Moreover, our implementation ex-perience shows that even complex queries, as those supportedin the work of [3], that go well beyond the single-keywordsearch capability of traditional SSE schemes, can be supportedin practical ways for these very large databases. The same istrue for the complex operational settings of [13] that supportdelegation and authorization of queries to multiple clients aswell as providing query privacy from the data owner.

Regarding the security of our schemes, it is important tohighlight that while the leakage of our constructions compareswell with related work, there is non-trivial information beingdisclosed to the server about result sets (e.g., the size of thesesets and their intersections). When the server has a prioriinformation on the database and queries, various statisticalattacks are plausible as illustrated, for instance, in [12]. Tomitigate such attacks one can apply various generic maskingcounter-measures such as padding the sets DB(w) with dummyelements or batching multiple updates to obfuscate updatepatterns. Hopefully, future work will shed more light on thebest ways to design such masking techniques. In particular, oneconfronts the hard problem of how the syntactically-definedleakage can be captured in a semantic way such that for realworld data sets and query distributions one can decide howmuch and what type of masking approaches are effective.

As mentioned in the introduction, an attractive alternativeto achieve more secure solutions to the SSE problem is the useof Oblivious RAM (ORAM) for which we have seen tremen-dous progress recently in terms of practical performance.However, nobody to our knowledge has yet systematicallyassessed on how to implement leakage-free search algorithmson top of ORAM servers. Even if we would tolerate theamount of leakage equivalent to our constructions, it is notclear whether one could achieve a similar level of performancefor ORAM when considering critical practical aspects suchas parallelism and interleaving of I/O and computation asexploited in our approach. Furthermore, the extensibility ofORAM-based solutions to scenarios such as multi-client poseseven further challenges.

ACKNOWLEDGMENT

Supported by the Intelligence Advanced Research ProjectsActivity (IARPA) via Department of Interior National BusinessCenter (DoI / NBC) contract number D11PC20201. The U.S.Government is authorized to reproduce and distribute reprintsfor Governmental purposes notwithstanding any copyrightannotation thereon. Disclaimer: The views and conclusionscontained herein are those of the authors and should notbe interpreted as necessarily representing the official policiesor endorsements, either expressed or implied, of IARPA,DoI/NBC, or the U.S. Government.

REFERENCES

[1] M. Bellare and P. Rogaway. Random oracles are practical: A paradigmfor designing efficient protocols. In V. Ashby, editor, ACM CCS 93,pages 62–73, Fairfax, Virginia, USA, Nov. 3–5, 1993. ACM Press. 3

[2] D. Boneh, G. Di Crescenzo, R. Ostrovsky, and G. Persiano. Publickey encryption with keyword search. In C. Cachin and J. Camenisch,editors, EUROCRYPT 2004, volume 3027 of LNCS, pages 506–522,Interlaken, Switzerland, May 2–6, 2004. Springer, Berlin, Germany. 2

[3] D. Cash, S. Jarecki, C. S. Jutla, H. Krawczyk, M.-C. Rosu, andM. Steiner. Highly-scalable searchable symmetric encryption withsupport for boolean queries. In R. Canetti and J. A. Garay, editors,CRYPTO 2013, Part I, volume 8042 of LNCS, pages 353–373, SantaBarbara, CA, USA, Aug. 18–22, 2013. Springer, Berlin, Germany. 1,2, 4, 10, 11, 12, 13, 14, 15, 16

[4] Y.-C. Chang and M. Mitzenmacher. Privacy preserving keywordsearches on remote encrypted data. In J. Ioannidis, A. Keromytis, andM. Yung, editors, ACNS 05, volume 3531 of LNCS, pages 442–455,New York, NY, USA, June 7–10, 2005. Springer, Berlin, Germany. 1

[5] M. Chase and S. Kamara. Structured encryption and controlleddisclosure. In M. Abe, editor, ASIACRYPT 2010, volume 6477 of LNCS,pages 577–594, Singapore, Dec. 5–9, 2010. Springer, Berlin, Germany.1, 2, 3, 4, 5

[6] B. Chor, O. Goldreich, E. Kushilevitz, and M. Sudan. Private informa-tion retrieval. Journal of the ACM, 45(6):965–981, 1998. 2

[7] R. Curtmola, J. A. Garay, S. Kamara, and R. Ostrovsky. Searchablesymmetric encryption: improved definitions and efficient constructions.In A. Juels, R. N. Wright, and S. Vimercati, editors, ACM CCS 06, pages79–88, Alexandria, Virginia, USA, Oct. 30 – Nov. 3, 2006. ACM Press.1, 2, 3, 4, 15

[8] M. Dietzfelbinger, M. Mitzenmacher, and M. Rink. Cuckoo hashingwith pages. Technical Report abs/1104.5111, arXiv, 2011. 11

[9] E.-J. Goh. Secure indexes. Cryptology ePrint Archive, Report 2003/216,2003. http://eprint.iacr.org/. 1

[10] O. Goldreich and R. Ostrovsky. Software protection and simulation onoblivious RAMs. Journal of the ACM, 43(3):431–473, 1996. 2

[11] M. T. Goodrich, M. Mitzenmacher, O. Ohrimenko, and R. Tamassia.Oblivious ram simulation with efficient worst-case access overhead. InCCSW, pages 95–100, 2011. 3

[12] M. Islam, M. Kuzu, and M. Kantarcioglu. Access pattern disclosureon searchable encryption: Ramification, attack and mitigation. InProceedings of the Symposium on Network and Distributed SystemsSecurity (NDSS 2012), San Diego, CA, Feb. 2012. Internet Society. 16

[13] S. Jarecki, C. Jutla, H. Krawczyk, M. C. Rosu, and M. Steiner.Outsourced symmetric private information retrieval. In ACM CCS 13,Berlin, Germany, Nov. 4–8, 2013. ACM Press. 2, 4, 10, 13, 14, 16

[14] S. Kamara and C. Papamanthou. Parallel and dynamic searchablesymmetric encryption. In A.-R. Sadeghi, editor, FC 2013, volume 7859of LNCS, pages 258–274, Okinawa, Japan, Apr. 1–5, 2013. Springer,Berlin, Germany. 1, 2, 3, 10, 15

[15] S. Kamara, C. Papamanthou, and T. Roeder. Dynamic searchablesymmetric encryption. In T. Yu, G. Danezis, and V. D. Gligor, editors,ACM CCS 12, pages 965–976, Raleigh, NC, USA, Oct. 16–18, 2012.ACM Press. 1, 2, 3, 4, 10, 14, 15

[16] J. Katz and Y. Lindell. Introduction to Modern Cryptography (Chapman& Hall/Crc Cryptography and Network Security Series). Chapman &Hall/CRC, 2007. 3

[17] K. Kurosawa and Y. Ohtaki. UC-secure searchable symmetric encryp-tion. In A. D. Keromytis, editor, FC 2012, volume 7397 of LNCS,pages 285–298, Kralendijk, Bonaire, Feb. 27 – Mar. 2, 2012. Springer,Berlin, Germany. 1, 3

[18] Lemur Project. ClueWeb09 dataset.http://lemurproject.org/clueweb09.php/. 14

[19] D. X. Song, D. Wagner, and A. Perrig. Practical techniques for searcheson encrypted data. In 2000 IEEE Symposium on Security and Privacy,pages 44–55, Oakland, California, USA, May 2000. IEEE ComputerSociety Press. 1, 2

[20] E. Stefanov, E. Shi, and D. X. Song. Towards practical obliviousRAM. In NDSS 2012, San Diego, California, USA, Feb. 5–8, 2012.The Internet Society. 3

[21] P. van Liesdonk, S. Sedhi, J. Doumen, P. H. Hartel, and W. Jonker.Computationally efficient searchable symmetric encryption. In Proc.Workshop on Secure Data Management (SDM), pages 87–100, 2010.1, 3, 10

16