Generating a service coordination from a hybrid query

Problem statement

• Given a query in SeQUEL (resp. HSQL)

derive an executable representation

Desirable properties

• Fit to a dynamic service-based environment• Well-defined semantics• Suitable for implementation yet human readable• Preferably yields optimal execution• Not too costly to derive• Expressed in a stand-alone language

SeQUELquery

Algebraicexpression

Queryworkflow

Servicecoordination

transform transform

transform

build

Datamodel

uses

uses

Data model

• Complex values: nested tuples, sets, and basic types

• Restriction*: flat tuples of atomic complex values (≈relations)

* In terms of full support currently

Algebra

• Equi-join• Bind-join• Time-based window• Tuple-based window• Recursive selection• Recursive projection• Recursive renaming• Group• Ungroup• Union• Intersection• Set difference

• Equi-join• Bind-join• Time-based window• Tuple-based window• Selection• Projection

Formalized Supported

SeQUEL(HSQL)

• SELECT-FROM-WHERE clauses• + windows and function calls

Compatible with supported operators

Query workflow model• Formalized by a graph model

• Equivalent to algebraic expressions: expression ↔ workflow ↔ function composition

• Transformation formalized by construction rules

Service coordination

• Workflow activities are bound to– Data services– Simple and composite computation services

• Binding to data services is enabled by a registry• Binding to computation services is static

SeQUEL query

Join algebraic expression

Join workflow

Complete query workflow

Service coordination

– Identify join attributes– Create hypregraph– Produce parse tree– Traverse parse tree

Data services

• Service interface → n data operations

• Binding-pattern representation

• Example

Example query

Find friends which are no more than 3 km away from my current location (48.85889, 2.29583) considering their locations of the last 10 min, which are also over 21 years old and that are interested in art.

SELECT p.nickname, p.age, p.gender, p.emailFROM profile AS p, location [range 10] AS l,

interests AS iWHERE p.age >= 21 AND l.nickname = p.nickname AND i.nickname = p.nickname AND i.tag=’art’ AND distance(lat, lon, 48.85889, 2.29583) <= 3.0;

Identify join attributes

• Generate symbols for join attributesl.nickname = p.nickname AND i.nickname = p.nickname

Create a hypergraph

N

A

{ , , , , , , , }X =

L

G E

T SN

N L

A G E

T SN

E = {{ , , }, { , , , }, { , , }}

N L

A G E

T SN

N L

A G E

T S

N

L’

L’

N

N

N

L’

N

L’

Produce and traverse parse tree

N A S E

N LL’

N

N T S

N

consume

((NLL’ ⋈ NASE)⋈ NTS)

N

AGE

TS

L L’

Parse tree traversal

algorithm ↺algorithm ↺

SeQUEL query

Join algebraic expression

Join workflow

Complete query workflow

Service coordination

– Translate to postorder– Construct workflow by

stack-based evaluation

Construct workflow by stack-based evaluation

NLL’ NAGE ⋈ NTS ⋈((NLL’ ⋈ NASE)⋈ NTS)

Construct workflow by stack-based evaluation

location(N L L’)

profile(N A G E)

interests(N T S)

NLL’ NAGE ⋈ NTS ⋈ NLL’NAGE

NLL’⋈NAGE

NTSNLL’ NAGE ⋈ NTS ⋈⋈⋈

⋈

SeQUEL query

Join algebraic expression

Join workflow

Complete query workflow

Service coordination

– Apply rules and heuristics

Rules for additional operators

• Place windows next in sequence to data stream operations

• Selections are pushed-down– function calls may involve multiple attributes

• A single projection operation is added at the end* Additional heuristics could be considered

Example

profile

location

[time win] σdist

σage

interests

⋈

⋈

σtag

π

profile

location

interests

⋈

⋈

SeQUEL query

Join algebraic expression

Join workflow

Complete query workflow

Service coordination

– Bind workflow activities to services

Example

Dynamic binding to data services and computation services

profile

location

[time win] σdist

σage

interests

⋈

⋈

σtag

π

location

[time win] σdist

⋈ σage

π

σtag

⋈profile

interests

Workflow of data operations and query operators

Simple and composite computation services

Service provisioning

profile

location

[time win] σdist

σage

interests

⋈

⋈

σtag

π

Genericserviceinstance

operation 1

operation 2

operation n. . .

Interface

operationi(in1, in2, ..) → outputOn-demand data servicesSimple computation servicesComposite computation servicesStream data services

Data services

Stream data service

On-demand data service

subscribe(dest, time) location

profile

t1=⟨nick:Alice, coor:⟨lat,lon⟩⟩

t2=⟨nick:Bob, coor:⟨lat,lon⟩⟩

tn=⟨nick:Mike, coor:⟨lat,lon⟩⟩. . .

profile(nick:’Bob’)out = {⟨nick:Bob, age:23, sex:M,…⟩}

Simple computation service

σdistanc

eGeo-distance

⟨nick:Bob, coor⟩

input()

out = distance(lat1, lon1, lat2, lon2)

⟨nick:Bob,coor⟩out <= 3

[time win] σdist

mycoor

tuple

Composite computation service

Symmetric hash join service [⋈]

inputOp1()

Hash-index1

inputOp2()

Hash-index2

⟨inputTuple1⟩

⟨inputTuplen⟩

⟨outTuple(s)1⟩

hash probeprobe hash

⟨outTuple(s)2⟩

profile

σdist

⋈

Composite computation service

endofInput

createHIs

leftInput

uninitialized

active

terminateddestroyHIs

negTuple

insert_tuple

delete_tuple

probeRight

N

Y

Y

N

negTuple

insert_tuple

delete_tuple

probeLeft

Y

N

endofInput rightInput

N

Y

match

match

Workflow model based on Abstract State Machines (ASM)

Parallel and sequential composition, conditionals, and iteration

Coordination model

Service interoperation and communication

Comp.service A

inputOp1()

Comp.service B

inputOp1()

inputOp2()

Comp.service C. . .

Serv. inst(s)

C

A

B

inputOp1()

inputOp2()

⟨inputTupleA⟩

⟨inputTupleB1⟩

⟨inputTupleB2⟩

⟨outTupleA⟩

⟨outTupleB⟩

GYO reduction larger example

Example query

SELECT * FROM ABC, BF, BCD, DEG, CDE, BHWHEREABC.B1 = BCD.B3 and ABC.C1 = BCD.C3 and BCD.B3 = BF.B2 andBCD.C3 = CDE.C5 and BCD.D3 = CDE.D5 and DEG.D4 = CDE.D5 andDEG.E4 = CDE.E5 and BCD.B3 = BH.B6;

Attribute symbol generation

ABC(A1!, B1!, C1!)

BF(B2?, F2!)

BCD(B3?, C3?, D3!)

DEG(D4!, E4!, G4!)

CDE(C5!, D5!, E5!)

BH(B6!, H6!)

ABC.B1 = BCD.B3ABC.C1 = BCD.C3BCD.B3 = BF.B2BCD.C3 = CDE.C5BCD.D3 = CDE.D5DEG.D4 = CDE.D5DEG.E4 = CDE.E5BCD.B3 = BH.B6

ABC(A1!, B!, C!)

BF(B?, F2!)

BCD(B?, C?, D!)

DEG(D!, E!, G4!)

CDE(C!, D!, E!)

BH(B!, H6!)

Join dependences as a hypergrph

ABC(A1!, B!, C!)

BF(B?, F2!)

BCD(B?, C?, D!)

DEG(D!, E!, G4!)

CDE(C!, D!, E!)

BH(B!, H6!)

A1

CB

E

G4

D

H6

F2

✓

✓

✓

✓

✓

✓

Parse tree construction

ABC(A1!, B!, C!)

BF(B?, F2!)

BCD(B?, C?, D!)

DEG(D!, E!, G4!)

CDE(C!, D!, E!)

BH(B!, H6!)

A1

CB

E

G4

D

H6

F2

BH

BCD

ABC BF

CDE

DEG

ISOLATED:{A1}

SHARED:{B,C}

A1!B!C!

B?C?D!A1!B!C!D!ISOLATED:{F2}

SHARED:{B}

B?F2!

A1!B!C!D!F2!ISOLATED:{H6}

SHARED:{B}

B!H6!

A1!B!C!D!F2!H6!ISOLATED:{B}

SHARED:{C,D}

C!D!E!A1!B!C!D!E!F2!H6!

ISOLATED:{C}

SHARED:{D,E}

D!E!G4!A1!B!C!D!E!F2!G4!H6!

Join expression construction

BH

BCD

ABC BF

CDE

DEG

A1!B!C! B?F2! B!H6!

A1!B!C!D!F2!H6!

A1!B!C!D!E!F2!H6!

A1!B!C!D!E!F2!G4!H6! ⋈DEG⋈

CDE⋈BH ⋈

BF ⋈ABC BCD

Each node denotes a join expressionRoot: ( DEG ⋈ ( CDE ⋈ ( ( ( BCD ⋈ ABC ) ⋈ BF ) ⋈ BH ) ) )

Join expression construction

Inorder expression: ( DEG ⋈ ( CDE ⋈ ( ( ( BCD ⋈ ABC ) ⋈ BF ) ⋈ BH ) ) )

Postorder expression: DEG CDE BCD ABC ⋈ BF ⋈ BH ⋈ ⋈ ⋈Process postorder expression using a stack to generate the join workflow

Workflow generation

⋈ABC

BCD

BF

⋈BH

⋈CDE

⋈

DEG

⋈

DEG CDE BCD ABC ⋈ BF ⋈ BH ⋈ ⋈ ⋈

DEG

CDE

BCD

ABC

BCD ⋈ ABC

BF

(BCD ⋈ ABC) ⋈ BF

BH

( ( BCD ⋈ ABC ) ⋈ BF ) ⋈ BHCDE ⋈ ( ( ( BCD ⋈ ABC ) ⋈ BF ) ⋈ BH )( DEG ⋈ ( CDE ⋈ ( ( ( BCD ⋈ ABC ) ⋈ BF ) ⋈ BH ) ) )

Annexes

Join expression constructioncreateJoinTree( Node node ) {

if ( node.children() = ∅ )return node;

else {Queue q := new Queue();foreach( child ∈ node.children() )

q.add(child);while( ⌐ queue.isEmpty() ) {

Node leftNode := node;Node rightNode := createJoinTree(queue.remove());Node joinedNode := joinNodes(leftNode, rightNode);node := joinedNode;

}return node;

}}

↺ ↺