1

XML Storage and Query Processing

Yanlei DiaoUniversity of Massachusetts

Amherst

Some slide content courtesy of Donald Kossmann

2

XML Storage Alternatives

Plain Text Trees with Navigation Tuples (i.e., mapping to RDBMS)

3

Plain Text

Use XML standards to encode data Advantages:

• simple, universal• indexing possible

Disadvantages:• need to re-parse (re-validate) all the time• no compliance with XQuery data model

(collections)• not an option for XQuery processing

4

Trees XML data model uses tree semantics

• use Trees/Forests to represent XML instances• annotate nodes of tree with data model info

Examples:• Document Object Model (DOM) http://www.w3.org/DOM/• Object Exchange Model (OEM)

f1

f4

f8f7

f5 f6f3f2

5

DataGuides [Goldman & Widom 97]

Schema-based environments

Schema Datagenerates

Queries

formulates execute against

6

DataGuides [Goldman & Widom 97]

Schema-free environments:• don't know the schema in advance.• semantic heterogeneity (i.e. a mix of schemas)

DataGuidesSummarized into

Queries

formulate

App-specific TemplatesData

generate

execute against

7

Schema vs. DataGuides

A DataGuide only includes info that exists in a DB.

A schema can be a superset of any DB that conforms to it.

So, a schema defines a superset of a DataGuide.

Issues addressed in the paper:• Summarize data into DataGuides;• Use them for query formulation and optimization.

8

Object Exchange Model (OEM)

Object Exchange Model (OEM) • Each object has an id (oid) and a value (atomic or a

set of subobjects).• Each edge links an object to one of its subobjects

with a label; a subobject may have multiple parents.

Label path: a seq. of labels

Data path: an alternating seq. of labels and oids

Target set: a set of all objects reached by traversing a label path

9

Definition of a DataGuide

Conciseness: a DataGuide describes every unique label path of a source exactly once

Accuracy: a DataGuide does not encode any label path that does not appear in the source

Convenience: represented as an OEM model, like the data

A DataGuide reflects the structure of a DB; it contains no atomic values.

10

From Data to DataGuides

Creating a DataGuide is equivalent to converting an NFA to DFA! • Consider a label path (query) as a string to be accepted by

the data source and the DataGuide. • Intuition: The data source has multiple matches, so

execution is non-deterministic. But the DataGuide has only one path, so execution is deterministic.

Cost of creation• Source DB is a tree: linear• Worst-case: exponential in #. of objects and edges in the

source• Empirical results: average performance for certain datasets

is quite encouraging

11

Multiple DataGuides An OEM source may have multiple DataGuides

• A single NFA may have many equivalent DFAs. Minimal DataGuide

• Can be created using DFA minimization

Minimality may not always be desirable• Hard to maintain as the

data source changes--well known problem with DFA.

• Does not allow annotations. 22 ? ?

12

Annotations Annotation: a property of the target set of

a label path l in the data source s• Statistical information: e.g. # occurrences of l

in s• Pointers to objects reachable via l• …

Issue with minimalityAnnotation for A.C

Annotation for B.C?

13

Strong DataGuides

Each set of label paths that share a node in the DataGuide is the set of label paths that share the same target set in the source. • Label paths can be merged in the DataGuide if they

lead to the same target set. There is one-to-one correspondence between

source target sets and DataGuide objects. Creation from the data source

• A DFS algorithm that examines source target sets reachable by al possible label paths…

Maintenance uses a similar set of data structures…

14

Query Formulation & Optimization

Query formulation • Query by example: click buttons to select a

path and add value filters • Blurs the distinction between formulating a

query and browsing a query result Query optimization

• Uses the DataGuide for structural matching (e.g. A.B.C) and retrieves the target set

• Uses value indexes (e.g. B+trees) for value filters for a specific label (e.g. C.price>100)

• Intersects the two resulting sets of objects

15

XML Data Stored as Tuples

Motivation: Use an RDBMS infrastructure to store and process the XML data• query optimization• scalability• richness and maturity of RDBMS

Alternative relational storage approaches:• Map XML schema to relational schema • Generic shredding of the data (edge, binary, …)• New XML storage integrated tightly with the

relational processor

16

Relational Support for XML [Zhang et al. 2001]

Goal: relational support for path queries, including storage and query processing

Assumption: we have the DTD/schema Problem addressed: to support XML path

queries• Can we use a relational DBMS? • Shall we design a native XML store, i.e. using

novel storage and indexing techniques?

17

Representation of XML

Each XML document is parsed to a seq. of items:• Start tag• Text word• End tag

All items are numbered, from 1.

<?xml version="1.0" ?> 1<book> 2<section id=“intro” difficulty=“easy”> 3<title> 4XML 5</title> 6<section difficulty=“easy”> 7<title> 8XML 9Processing 10</title> 11<figure source=“g1.jpg”> 12<title> 13XML 14Processing 15Cost

16</title> 17</figure>  18</section> 19<figure source=“g2.jpg”> 20<title> 21Scalability 22</title> 23</figure> 24</section>25</book>

18

Element Index An Element Index (E-index) records occurrences of

each element name inside the entire collection of documents.

Each index entry in an E-index corresponds to one occurrence of the element name. It has:• document identifier, • start position of the element in the doc, i.e. position of its

start tag.• end position of the element in the doc, i.e. position of its

end tag• document level of the element in the doc, i.e. level from

the root. An E-index is sorted in increasing order of

<document id, start position, end position>.

19

Example of E-Index<?xml version="1.0" ?> 1<book> 2<section id=“intro” difficulty=“easy”> 3<title> 4XML 5</title> 6<section difficulty=“easy”> 7<title> 8XML 9Processing 10</title> 11<figure source=“g1.jpg”> 12<title> 13XML 14Processing 15Cost

16</title> 17</figure>  18</section> 19<figure source=“g2.jpg”> 20<title> 21Scalability 22</title> 23</figure> 24</section>25</book>

(1, 1:25, 1) (2, …<book>

<section> (1, 2:24, 2) (1, 6:18, 3) (2, …

<title> (1, 3:5, 3) (1, 7:10, 4) (1, 12:16, 5) (1, 20:22, 4) (2, …

<figure> (1, 11:17, 4) (1, 19:23, 3) (2, …

20

Text Index

A Text Index (T-index) records the occurrences of each text word inside the entire collection of documents, similar to E-Index.

Difference is that each index entry in a T-index contains a single word position, instead of the pair of start and end positions.

Similarly, a T-index is sorted in increasing of <document identifier, word position>.

21

Example of T-Index<?xml version="1.0" ?> 1<book> 2<section id=“intro” difficulty=“easy”> 3<title> 4XML 5</title> 6<section difficulty=“easy”> 7<title> 8XML 9Processing 10</title> 11<figure source=“g1.jpg”> 12<title> 13XML 14Processing 15Cost

16</title> 17</figure>  18</section> 19<figure source=“g2.jpg”> 20<title> 21Scalability 22</title> 23</figure> 24</section>25</book> (1, 4:4, 4) (1, 8:8, 5) (1, 13:13, 6) (2, …“XML”

“Processing” (1, 9:9, 5) (1, 14:14, 6) (2, …

“Cost” (1, 15:15, 6) (2, …

“Scalability” (1, 21:21, 5) (2, …

22

Relational Storage(a) Element-Index

1 19 23 3

1 11 17 4

2 … … …

1 12 16 5

1 3 5 3

1 20 22 4

1 7 10 4

2 … … …

1 6 18 3

1 2 24 2

2 … … …

doc_id start_pos end_pos doc_level

1 1 25 1

2 … … …

term

<book>

<book>

<section>

<section>

<section>

<title>

<title>

<title>

<title>

<title>

<figure>

<figure>

<figure>

(b) Text-Index

2 … …

doc_id word_pos doc_level

1 4 4

1 8 5

1 13 6

1 9 5

1 14 6

2 … …

1 15 6

2 … …

1 21 5

2 … …

term

“XML”

“XML”

“XML”

“XML”

“Processing”

“Processing”

“Processing”

“Cost”

“Cost”

“Scalability”

“Scalability”

23

Relational Storage (contd.)

One relation for elements, one for text words

Clustered B+trees over each table • On (term, docno)• On all columns: lead to index-only plans

24

“//section//title”

Index Scan on <section>

Index Scan on <title>

(//)l.doc_id = r.doc_id and l.start_pos <

r.start_pos and l.end_pos > r.end_pos

25

Questions

26

Outline

Storage and Query Processing• DataGuides [Goldman and Widom 97]• Relational Approach [Zhang et al. 2001]

Other Research Topics• Query Rewriting• Benchmarking• …

27

Node Identifiers XQuery Data Model Requirements

• identify a node uniquely (implementing identity)• lives as long as node lives• robust to updates

Identifiers might include additional information• Schema/type information• Document order• Parent/child relationship• Ancestor/descendent relationship• Document information

Required for indexes

28

Simple Node Identifiers Examples:

• Alternative 1 (data: trees)• id of document (integer)• pre-order number of node in document (integer)

• Alternative 2 (data: plain text)• file name• offset in file

Encode document ordering (Alternative 1)• identity: doc1 = doc2 AND pre1 = pre2• order: doc1 < doc2

OR (doc1 = doc2 AND pre1 < pre2) Assessment:

• bad: Not robust to updates• bad: Not able to answer more complex queries

29

Dewey Order

Idea:• Generate surrogates for each path• 1.2.3 identifies the third child of the second

child of the first child of the given root Assessment:

• good: order comparison, ancestor/descendent easy

• bad: updates expensive, space overhead Improvement: ORDPath Bit Encoding

O‘Neil et al. 2004 (Microsoft SQL Server)

30

Example: Dewey Order

name

name child

person

person

hobby hobby

1.1 1.2

1

1.2.1

1.2.1.1 1.2.1.2 1.2.1.3

31

XML Storage Alternatives

Plain Text Trees with Random Access Tuples (i.e., mapping to RDBMS)

32

Plain Text

Use XML standards to encode data Advantages:

• simple, universal• indexing possible

Disadvantages:• need to re-parse (re-validate) all the time• no compliance with XQuery data model

(collections)• not an option for XQuery processing

33

Trees XML data model uses tree semantics

• use Trees/Forests to represent XML instances• annotate nodes of tree with data model info

Example<f1>

<f2>..</f2> <f3>..</f3> <f4> <f7/> <f8>..</f8> </f4> <f5/> <f6>..</f6> </f1>

f1

f4

f8f7

f5 f6f3f2

34

Trees Advantages

• natural representation of XML data• good support for navigation, updates index built

into the data structure• compliance with DOM standard interface

Disadvantages• difficult to partition• high overhead: mixes indexes and data• index everything

Example: Document Object Model (DOM) • http://www.w3.org/DOM/

35

Edge Approach (Florescu & Kossmann 99)

Source Label Target

0 person 4711

0 person 666

4711 name v1

4711 child i314

666 name v2

666 child i314

Id Value

v1 Lilly Potter

v2 James Potter

v3 Harry Potter

Id Value

v4 12

Edge Table Value Table (String)

Value Table (Integer)

36

XML Example

<person id = “4711“> <name> Lilly Potter </name> <child> <person id = “314“> <name> Harry Potter </name> <age> 12 </age> </child></person><person id = “666“> <name> James Potter </name> <child idref = “314“/></person>

37

person person

Harry Potter

name

name name

person

Lilly Potter James Potter

child

314

0

4711 666

i314

<person id = “4711“> <name> Lilly Potter </name> <child> <person id = “314“> <name> Harry Potter

</name> <age> 12 </age> </child></person><person id = “666“> <name> James Potter </name> <child idref = “314“/></person>

age

12

38

Kinds of Indexes

1. Value Indexes• index atomic values; e.g.,

//emp/salary/fn:data(.)• use B+ trees (like in relational world)• (integration into query optimizer more tricky)

2. Structure Indexes• materialize results of path expressions• (pendant to Rel. join indexes, OO path indices)

3. Full text indexes• Keyword search, inverted files• (IR world, text extenders)

Any combination of the above

39

Outline

XML Storage XML Indexing Query Processing Other Research Topics

• Query Rewriting• Benchmarking• …

40

What is a Correct Rewriting

E1 -> E2 is a legal rewriting iff• Type(E2) is a subtype of Type(E1)• FreeVar(E2) is a subset of FreeVar(E1)• For a binding of free variables, either

• E1 or E2 return ERROR (possibly different errors)• Or E1 and E2 return the same result

This definition allows the rewrite E1->ERROR• Trust your vendor she does not do that for all

E1!

41

Handling Backwards Navigation

Replace backwards navigation with forward navigation

for $x in $input/a/b for $y in $input/a,return <c>{$x/.., $x/d}</c> $x in $y/b return <c>{$y,

$x/d}</c>

for $x in $input/a/breturn <c>{$x//e/..}</c> ??

Enables streaming

42

FLWR Unnesting

Traditional database techniquefor $x in $input/a/b for $x in $input/a/b,where $x/c eq 3 $y in $x/dreturn (for $y in $x/d where ($x/e eq 4) and ($x/c

eq 3) where $x/e eq 4 return $y return $y)

Problem simpler than in OQL/ODMG

• No nested collections in XML

43

XML Query Processing

Techniques vary a lot, depending on• Storage model• Indexes available• Algebra used• …

A large body of ongoing work• Research community: McHugh and Widom 1999,

Zhang et al. 2001, Bruno et al. 2002, Ghua et al. 2002, Chen et al. 2003, Paparizos et al. 2004, Jagadish 2004, … (just look at SIGMOD and VLDB proceedings in recent years!)

• Industry: IBM DB2, Oracle, SQL Server, …

44

XML Processing Benchmark

We cannot really compare approaches until we decide on a comparison basis

XML processing very broad Industry not mature enough Usage patterns not clear enough Existing XML benchmarks (Xmark, etc. )

limited Strong need for a TP benchmark