class 18 hashing -...

hashingprof. Stratos Idreos

HTTP://DASLAB.SEAS.HARVARD.EDU/CLASSES/CS165/

class 18

/33CS165, Fall 2015 Stratos Idreos 2

professors(id,name,…)

courses(id,name, profId,…)

students(id,name,…)

database

give me all students enrolled in cs165select student.name from students, enrolled, courses where courses.name=“cs165” and enrolled.courseId=course.id and student.id=enrolled.studentId

enrolled(studentId,

courseId,…) foreign key

new resL[]; new resR[]; k=0 for (i=0;i<L.size;i=i++) for (j=0;j<R.size;j++) if L[i]==R[j] resL[k]=i resR[k++]=j

nested loops

hash join

join input 1 hash table

join input 2

hash table

bucket

val (key),pos

h=f(val)bucket=h mod k

k bucketsN keys

[(val1,pos1), (val7,pos7), …]

stream input

p1 p2 p3 p4

1. read input into stream buffer, hash and write to respective partition buffer 2. when input buffer is consumed, bring the next one 3. when a partition buffer is full, write to L2 we can partition into L1-1 pieces in one pass

partition

Level2

Level 1

grace hash join

hash partitioning

then join each pair of partitionsindependently in memory

join input 1 join input 2

stream input left

p1 right hash table result

p1 left

p1 right

result

Level2

Level 1

as long as at least one of the pieces <=L1-2

grace hash join

hash partitioning

apply recursively if a partition does not fit in memory (L1-2)

today: how to create the hash table

selectR.C

selectS.F

fetchR.A

fetchS.A

maxR.D

minS.G

static hashing

hash table

bucket

updates may create long lists

can this happen anyway, i.e., even with no updates

simple solution would be to double the hash table and rehash everything…

but…

dynamic hashing

extendible hashing - linear hashing

extendible hashing

directory bucketswhen there is no more space

split individual bucketsand if needed double the directory

hash table

with directory in memory we need 1 I/O to fetch the bucket

not true from memory to caches in general

directory maps hash values to buckets

h=hash(key)

use binary form of h use X last bits to map 2^x buckets

00 01 10 11

for X=2000 001 010 011 100 101 110 111

for X=3

extending the directory …

00 01 10 11

dic. X=2buckets

00 01 10 11

dic. X=2

000 001 010 011 100 101 110 111

for X=3

buckets

00 01 10 11

dic. X=2

000 001 010 011 100 101 110 111

for X=3

buckets

00 01 10 11

dic. X=2

000 001 010 011 100 101 110 111

for X=3

buckets

00 01 10 11

dic. X=2

000 001 010 011 100 101 110 111

for X=3

buckets

00 01 10 11

dic. X=2

000 001 010 011 100 101 110 111

for X=3

buckets

00 01 10 11

dic. X=2

000 001 010 011 100 101 110 111

for X=3

buckets

00 01 10 11

dic. X=2

000 001 010 011 100 101 110 111

for X=3

buckets

00 01 10 11

dic. X=2

000 001 010 011 100 101 110 111

for X=3

buckets

newextend directory only when no split is possible

cannot split

can split

linear hashingdo not use a directory (indirection cost)

instead incrementally extend the hash table one bucket at a time

e.g.,use 2 bits of hash value (h0)

next bucket to split00

01 10 11

dictionary is implicit when insert causes

overflow somewhere split next bucket

use +1 bit of hash value

01 10 11

00 01 10 11

next bucket to split

000 001 010 011 100 101 110 111

for X=3buckets

newdictionary is

implicit

00 01 10 11

dictionary is implicit

for X=2

search: use current hash function (bits) if after split bucket ok else use next hash function (bits+1)

000 001 010 011 100 101 110 111

for X=3buckets

00 01 10 11

for X=2

keep splitting next bucket with each overflow until all buckets are split (for x=2) then restart for x=3

any problemswhat would happen

when we split bucket C

selectR.C

selectS.F

fetchR.A

fetchS.A

maxR.D

minS.G

What can we do to start working immediately? (hint: vectorized processing)

What can we do if we wait for all data to arrive? (hint: bulk processing)

symmetric hash joinL R

while there exist buffered values from L, hash and

probe HT of R

1) hash

2) probe

3) output

when buffer is empty, switch!

symmetric hash joinL R

while there exist buffered values from R, hash and

probe HT of L

2) probe1) hash

3) output

when buffer is empty, switch!

static hashing with 2 passeskeys

count size of each bucket

phase 1

hash tablekeys

phase 2

we now exactlywhat

hash tableto build

count size of each bucket

phase 1

size1 size2 size3

… sizeK

array of k slots where k the buckets we want to have

for each bucket we know its size

hash tablekeys

phase 2

we now exactlywhat

hash tableto build

size1 size2 size3

… sizeK

0 D[0]+s1 D[1]+s2 D[2]+s3

pass D and sum all counts to get offsets in a sequentially stored hash table

can be done with binary ops only

use X LSBs

h=a*key+bb=h mod K

which hash function should we use?

what happens after the join?

selectR.C

selectS.F

fetchR.A

fetchS.A

maxR.D

minS.G

select max(R.D),min(S.G) from R,S where R.A=S.A and R.C<10 and S.F>30

block operator

access patterns

select R.A, R.B, R.C, S.A, S.B, S.C from R, S where R.J=S.J and …

R.* scan

pos R.Jfetch

posR.J

ordered sparse

preparing the R join input

same for the S join input

we need the original positions so we can fetch other R columns after the join

join input R.J R.J + posR

join input S.J S.J + posS partition (=reorder)

both join inputs to join

join result posR + posS

both sides are unordered

ordered sparse

join result posR + posS

cluster on R

join result clustered on R posR + posS

fetch R payload with sequential pattern

1 2 3 4

ID + posS

cluster on S4 2 3 1

ID + posS

d fetch S payload with sequential pattern

4 2 3 1

e.g., ID + S.A

sort on ID decluster

all result columns aligned

radix declustering

both sides are unordered

Cache-Conscious Radix Decluster ProjectionsBy S. Manegold, P. Boncz, N. Nes, and M. Kersten Very Large Databases Conference, 2004

Textbook Chapter 11

Cache-Conscious Radix Decluster ProjectionsS. Manegold, P. Boncz, N. Nes, and M. Kersten Very Large Databases Conference, 2004

DATA SYSTEMSprof. Stratos Idreos

class 18

hashing

class 18 hashing -...

Documents

cs165 practice final exam...

class 5 column stores 2 - harvard...

10/17/2002cse 202 - hashing cse 202 - algorithms hashing...

high dimensional search min-hashing locality sensitive...

5 hashing hashtabellen hashing with chaining hashing with...

lecture 11 oct 6 goals: hashing hash functions chaining...

chapter 8 hashing concept of hashing static hashing...

hashing - introduction - mcmaster...

sql & intro to db architectures - harvard...

cuckoo hashing

cs165 project for fall 2019 ||...

14. hashing -...

chapter 13 abstract classes and interfaces cs165 colorado...

file organizations jan. 2008yangjun chen acs-39021 outline:...

hashing & indexing

class 2 from bits to systems - harvard...

lecture 10 sept 29 goals: hashing dictionary operations...

hashing 1 hashing. hashing 2 hashing … * again, a...

summer school on hashing’14 locality sensitive hashing

b202 hashing