![Page 1: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/1.jpg)
Intro to Database Systems
15-445/15-645
Fall 2020
Andy PavloComputer Science Carnegie Mellon UniversityAP
22 Introduction to Distributed Databases
![Page 2: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/2.jpg)
15-445/645 (Fall 2020)
ADMINISTRIVIA
Homework #5: Sunday Dec 6th @ 11:59pm
Project #4: Sunday Dec 13th @ 11:59pm
Potpourri + Review: Wednesday Dec 9th
→ Vote for what system you want me to talk about.https://cmudb.io/f20-systems
Final Exam:→ Session #1: Thursday Dec 17th @ 8:30am→ Session #2: Thursday Dec 17th @ 1:00pm
2
![Page 3: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/3.jpg)
15-445/645 (Fall 2020)
UPCOMING DATABASE TALKS
Confluent ksqlDB (Kafka)→ Monday Nov 23rd @ 5pm ET
Microsoft SQL Server Optimizer→ Monday Nov 30th @ 5pm ET
Snowflake Lecture→ Monday Dec 7th @ 3:20pm ET
3
![Page 4: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/4.jpg)
15-445/645 (Fall 2020)
PARALLEL VS. DISTRIBUTED
Parallel DBMSs:→ Nodes are physically close to each other.→ Nodes connected with high-speed LAN.→ Communication cost is assumed to be small.
Distributed DBMSs:→ Nodes can be far from each other.→ Nodes connected using public network.→ Communication cost and problems cannot be ignored.
4
![Page 5: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/5.jpg)
15-445/645 (Fall 2020)
DISTRIBUTED DBMSs
Use the building blocks that we covered in single-node DBMSs to now support transaction processing and query execution in distributed environments.→ Optimization & Planning→ Concurrency Control→ Logging & Recovery
5
![Page 6: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/6.jpg)
15-445/645 (Fall 2020)
TODAY'S AGENDA
System Architectures
Design Issues
Partitioning Schemes
Distributed Concurrency Control
6
![Page 7: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/7.jpg)
15-445/645 (Fall 2020)
SYSTEM ARCHITECTURE
A DBMS's system architecture specifies what shared resources are directly accessible to CPUs.
This affects how CPUs coordinate with each other and where they retrieve/store objects in the database.
7
![Page 8: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/8.jpg)
15-445/645 (Fall 2020)
SYSTEM ARCHITECTURE
8
SharedNothing
Network
SharedMemory
Network
SharedDisk
Network
SharedEverything
![Page 9: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/9.jpg)
15-445/645 (Fall 2020)
SHARED MEMORY
CPUs have access to common memory address space via a fast interconnect.→ Each processor has a global view of all the
in-memory data structures. → Each DBMS instance on a processor has to
"know" about the other instances.
9
Network
![Page 10: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/10.jpg)
15-445/645 (Fall 2020)
SHARED DISK
All CPUs can access a single logical disk directly via an interconnect, but each have their own private memories.→ Can scale execution layer independently
from the storage layer.→ Must send messages between CPUs to
learn about their current state.
10
Network
![Page 11: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/11.jpg)
15-445/645 (Fall 2020)
Storage
SHARED DISK EXAMPLE
11
Node
ApplicationServer Node
Get Id=101Page ABC
![Page 12: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/12.jpg)
15-445/645 (Fall 2020)
Storage
SHARED DISK EXAMPLE
11
Node
ApplicationServer Node
Get Id=200Page XYZ
![Page 13: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/13.jpg)
15-445/645 (Fall 2020)
Storage
SHARED DISK EXAMPLE
11
Node
ApplicationServer Node
NodeGet Id=101 Page ABC
![Page 14: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/14.jpg)
15-445/645 (Fall 2020)
Storage
SHARED DISK EXAMPLE
11
Node
ApplicationServer Node
Node
Update 101Page ABC
![Page 15: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/15.jpg)
15-445/645 (Fall 2020)
Storage
SHARED DISK EXAMPLE
11
Node
ApplicationServer Node
Node
Update 101Page ABC
![Page 16: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/16.jpg)
15-445/645 (Fall 2020)
SHARED NOTHING
Each DBMS instance has its own CPU, memory, and disk.
Nodes only communicate with each other via network.→ Harder to scale capacity.→ Harder to ensure consistency.→ Better performance & efficiency.
12
Network
![Page 17: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/17.jpg)
15-445/645 (Fall 2020)
SHARED NOTHING EXAMPLE
13
Node
ApplicationServer Node
P1→ID:1-150
P2→ID:151-300
Get Id=200
![Page 18: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/18.jpg)
15-445/645 (Fall 2020)
SHARED NOTHING EXAMPLE
13
Node
ApplicationServer Node
P1→ID:1-150
P2→ID:151-300
Get Id=10 Get Id=200
Get Id=200
![Page 19: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/19.jpg)
15-445/645 (Fall 2020)
SHARED NOTHING EXAMPLE
13
Node
ApplicationServer Node
Node
P3→ID:101-200
P1→ID:1-100
P2→ID:201-300
![Page 20: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/20.jpg)
15-445/645 (Fall 2020)
EARLY DISTRIBUTED DATABASE SYSTEMS
MUFFIN – UC Berkeley (1979)
SDD-1 – CCA (1979)
System R* – IBM Research (1984)
Gamma – Univ. of Wisconsin (1986)
NonStop SQL – Tandem (1987)
14
Bernstein
Mohan DeWitt
Gray
Stonebraker
![Page 21: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/21.jpg)
15-445/645 (Fall 2020)
DESIGN ISSUES
How does the application find data?
How to execute queries on distributed data?→ Push query to data.→ Pull data to query.
How does the DBMS ensure correctness?
15
![Page 22: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/22.jpg)
15-445/645 (Fall 2020)
HOMOGENOUS VS. HETEROGENOUS
Approach #1: Homogenous Nodes→ Every node in the cluster can perform the same set of
tasks (albeit on potentially different partitions of data).→ Makes provisioning and failover "easier".
Approach #2: Heterogenous Nodes→ Nodes are assigned specific tasks.→ Can allow a single physical node to host multiple "virtual"
node types for dedicated tasks.
16
![Page 23: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/23.jpg)
15-445/645 (Fall 2020)
MONGODB HETEROGENOUS ARCHITECTURE
17
Router (mongos)
Shards (mongod)
P3 P4
P1 P2
P1→ID:1-100
P2→ID:101-200
P3→ID:201-300
P4→ID:301-400
Config Server (mongod)
Router (mongos)
⋮
⋮
ApplicationServer
Get Id=101
![Page 24: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/24.jpg)
15-445/645 (Fall 2020)
DATA TRANSPARENCY
Users should not be required to know where data is physically located, how tables are partitionedor replicated.
A query that works on a single-node DBMS should work the same on a distributed DBMS.
18
![Page 25: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/25.jpg)
15-445/645 (Fall 2020)
DATABASE PARTITIONING
Split database across multiple resources:→ Disks, nodes, processors.→ Often called "sharding" in NoSQL systems.
The DBMS executes query fragments on each partition and then combines the results to produce a single answer.
19
![Page 26: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/26.jpg)
15-445/645 (Fall 2020)
NAÏVE TABLE PARTITIONING
Assign an entire table to a single node.
Assumes that each node has enough storage space for an entire table.
Ideal if queries never join data across tables stored on different nodes and access patterns are uniform.
20
![Page 27: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/27.jpg)
15-445/645 (Fall 2020)
NAÏVE TABLE PARTITIONING
21
Table1
SELECT * FROM table
Ideal Query:
Table2 Partitions
![Page 28: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/28.jpg)
15-445/645 (Fall 2020)
NAÏVE TABLE PARTITIONING
21
Table1
SELECT * FROM table
Ideal Query:
Table2 Partitions
![Page 29: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/29.jpg)
15-445/645 (Fall 2020)
NAÏVE TABLE PARTITIONING
21
Table1
SELECT * FROM table
Ideal Query:
Table2 Partitions
Table1
Table2
![Page 30: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/30.jpg)
15-445/645 (Fall 2020)
HORIZONTAL PARTITIONING
Split a table's tuples into disjoint subsets.→ Choose column(s) that divides the database equally in
terms of size, load, or usage.→ Hash Partitioning, Range Partitioning
The DBMS can partition a database physical(shared nothing) or logically (shared disk).
22
![Page 31: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/31.jpg)
15-445/645 (Fall 2020)
HORIZONTAL PARTITIONING
23
SELECT * FROM tableWHERE partitionKey = ?
Ideal Query:
PartitionsTable1101 a XXX 2019-11-29
102 b XXY 2019-11-28
103 c XYZ 2019-11-29
104 d XYX 2019-11-27
105 e XYY 2019-11-29
hash(a)%4 = P2
hash(b)%4 = P4
hash(c)%4 = P3
hash(d)%4 = P2
hash(e)%4 = P1
Partitioning Key
![Page 32: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/32.jpg)
15-445/645 (Fall 2020)
HORIZONTAL PARTITIONING
23
SELECT * FROM tableWHERE partitionKey = ?
Ideal Query:
PartitionsTable1101 a XXX 2019-11-29
102 b XXY 2019-11-28
103 c XYZ 2019-11-29
104 d XYX 2019-11-27
105 e XYY 2019-11-29
hash(a)%4 = P2
hash(b)%4 = P4
hash(c)%4 = P3
hash(d)%4 = P2
hash(e)%4 = P1
Partitioning Key
![Page 33: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/33.jpg)
15-445/645 (Fall 2020)
HORIZONTAL PARTITIONING
23
SELECT * FROM tableWHERE partitionKey = ?
Ideal Query:
PartitionsTable1101 a XXX 2019-11-29
102 b XXY 2019-11-28
103 c XYZ 2019-11-29
104 d XYX 2019-11-27
105 e XYY 2019-11-29
P3 P4
P1 P2
hash(a)%4 = P2
hash(b)%4 = P4
hash(c)%4 = P3
hash(d)%4 = P2
hash(e)%4 = P1
Partitioning Key
![Page 34: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/34.jpg)
15-445/645 (Fall 2020)
HORIZONTAL PARTITIONING
23
SELECT * FROM tableWHERE partitionKey = ?
Ideal Query:
PartitionsTable1101 a XXX 2019-11-29
102 b XXY 2019-11-28
103 c XYZ 2019-11-29
104 d XYX 2019-11-27
105 e XYY 2019-11-29
P3 P4
P1 P2
hash(a)%4 = P2
hash(b)%4 = P4
hash(c)%4 = P3
hash(d)%4 = P2
hash(e)%4 = P1
Partitioning Key
![Page 35: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/35.jpg)
15-445/645 (Fall 2020)
HORIZONTAL PARTITIONING
23
SELECT * FROM tableWHERE partitionKey = ?
Ideal Query:
PartitionsTable1101 a XXX 2019-11-29
102 b XXY 2019-11-28
103 c XYZ 2019-11-29
104 d XYX 2019-11-27
105 e XYY 2019-11-29
P3 P4
P1 P2
hash(a)%4 = P2
hash(b)%4 = P4
hash(c)%4 = P3
hash(d)%4 = P2
hash(e)%4 = P1
Partitioning Key
![Page 36: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/36.jpg)
15-445/645 (Fall 2020)
HORIZONTAL PARTITIONING
23
SELECT * FROM tableWHERE partitionKey = ?
Ideal Query:
PartitionsTable1101 a XXX 2019-11-29
102 b XXY 2019-11-28
103 c XYZ 2019-11-29
104 d XYX 2019-11-27
105 e XYY 2019-11-29
P3 P4
P1 P2
Partitioning Key
hash(a)%5 = P4
hash(b)%5 = P3
hash(c)%5 = P5
hash(d)%5 = P1
hash(e)%5 = P3
![Page 37: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/37.jpg)
15-445/645 (Fall 2020)
CONSISTENT HASHING
24
01
0.5
hash(key1)
P1
P3
P2
![Page 38: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/38.jpg)
15-445/645 (Fall 2020)
CONSISTENT HASHING
24
01
0.5
hash(key2)
hash(key1)
P1
P3
P2
![Page 39: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/39.jpg)
15-445/645 (Fall 2020)
CONSISTENT HASHING
24
01
0.5
hash(key2)
hash(key1)
P1
P3
P2
![Page 40: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/40.jpg)
15-445/645 (Fall 2020)
CONSISTENT HASHING
24
01
0.5
If hash(key)=P4
P1
P3
P4P2
![Page 41: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/41.jpg)
15-445/645 (Fall 2020)
CONSISTENT HASHING
24
01
0.5
P5
P1
P3
P4P2
![Page 42: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/42.jpg)
15-445/645 (Fall 2020)
CONSISTENT HASHING
24
01
0.5
P5
P1
P3
P4P2
P6
![Page 43: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/43.jpg)
15-445/645 (Fall 2020)
CONSISTENT HASHING
24
01
0.5
Replication Factor = 3P5
P1
P3
P4P2
P6
![Page 44: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/44.jpg)
15-445/645 (Fall 2020)
CONSISTENT HASHING
24
01
0.5
Replication Factor = 3
hash(key1)
P5
P1
P3
P4P2
P6
![Page 45: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/45.jpg)
15-445/645 (Fall 2020)
CONSISTENT HASHING
24
01
0.5
Replication Factor = 3
hash(key1)
P5
P1
P3
P4P2
P6
![Page 46: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/46.jpg)
15-445/645 (Fall 2020)
Storage
LOGICAL PARTITIONING
25
Node
ApplicationServer Node
Id=1
Id=2
Id=3
Id=4
![Page 47: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/47.jpg)
15-445/645 (Fall 2020)
Storage
LOGICAL PARTITIONING
25
Node
ApplicationServer Node
Get Id=1
Id=1
Id=2
Id=3
Id=4
Id=1
Id=2
Id=3
Id=4
![Page 48: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/48.jpg)
15-445/645 (Fall 2020)
Storage
LOGICAL PARTITIONING
25
Node
ApplicationServer Node
Get Id=3
Id=1
Id=2
Id=3
Id=4
Id=1
Id=2
Id=3
Id=4
![Page 49: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/49.jpg)
15-445/645 (Fall 2020)
Node
Node
PHYSICAL PARTITIONING
26
ApplicationServer
Get Id=1Id=1
Id=2
Id=3
Id=4
![Page 50: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/50.jpg)
15-445/645 (Fall 2020)
Node
Node
PHYSICAL PARTITIONING
26
ApplicationServer
Get Id=3
Id=1
Id=2
Id=3
Id=4
![Page 51: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/51.jpg)
15-445/645 (Fall 2020)
SINGLE-NODE VS. DISTRIBUTED
A single-node txn only accesses data that is contained on one partition.→ The DBMS does not need coordinate the behavior
concurrent txns running on other nodes.
A distributed txn accesses data at one or more partitions.→ Requires expensive coordination.
28
![Page 52: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/52.jpg)
15-445/645 (Fall 2020)
TRANSACTION COORDINATION
If our DBMS supports multi-operation and distributed txns, we need a way to coordinate their execution in the system.
Two different approaches:→ Centralized: Global "traffic cop".→ Decentralized: Nodes organize themselves.
29
![Page 53: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/53.jpg)
15-445/645 (Fall 2020)
TP MONITORS
A TP Monitor is an example of a centralized coordinator for distributed DBMSs.
Originally developed in the 1970-80s to provide txns between terminals and mainframe databases.→ Examples: ATMs, Airline Reservations.
Many DBMSs now support the same functionality internally.
30
![Page 54: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/54.jpg)
15-445/645 (Fall 2020)
Coordinator
CENTRALIZED COORDINATOR
31
Partitions
ApplicationServer P3 P4
P1 P2
![Page 55: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/55.jpg)
15-445/645 (Fall 2020)
Coordinator
CENTRALIZED COORDINATOR
31
PartitionsLock Request
ApplicationServer P3 P4
P1 P2
P1
P2
P3
P4
![Page 56: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/56.jpg)
15-445/645 (Fall 2020)
Coordinator
CENTRALIZED COORDINATOR
31
PartitionsLock Request
ApplicationServer P3 P4
P1 P2
P1
P2
P3
P4
![Page 57: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/57.jpg)
15-445/645 (Fall 2020)
Coordinator
CENTRALIZED COORDINATOR
31
PartitionsLock Request
Acknowledgement
ApplicationServer P3 P4
P1 P2
P1
P2
P3
P4
![Page 58: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/58.jpg)
15-445/645 (Fall 2020)
Coordinator
CENTRALIZED COORDINATOR
31
PartitionsCommit Request
Safe to commit?Application
Server P3 P4
P1 P2
P1
P2
P3
P4
![Page 59: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/59.jpg)
15-445/645 (Fall 2020)
Coordinator
CENTRALIZED COORDINATOR
31
Partitions
Acknowledgement
Commit Request
Safe to commit?Application
Server P3 P4
P1 P2
P1
P2
P3
P4
![Page 60: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/60.jpg)
15-445/645 (Fall 2020)
CENTRALIZED COORDINATOR
32
Mid
dle
wa
re
Query Requests
ApplicationServer P3 P4
P1 P2
Partitions
![Page 61: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/61.jpg)
15-445/645 (Fall 2020)
CENTRALIZED COORDINATOR
32
Mid
dle
wa
re
Query Requests
ApplicationServer P3 P4
P1 P2
P1→ID:1-100
P2→ID:101-200
P3→ID:201-300
P4→ID:301-400
Partitions
![Page 62: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/62.jpg)
15-445/645 (Fall 2020)
CENTRALIZED COORDINATOR
32
Mid
dle
wa
re
Query Requests
ApplicationServer P3 P4
P1 P2
P1→ID:1-100
P2→ID:101-200
P3→ID:201-300
P4→ID:301-400
Partitions
![Page 63: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/63.jpg)
15-445/645 (Fall 2020)
CENTRALIZED COORDINATOR
32
Mid
dle
wa
re
Safe to commit?
ApplicationServer P3 P4
P1 P2
P1→ID:1-100
P2→ID:101-200
P3→ID:201-300
P4→ID:301-400
Commit Request
Partitions
![Page 64: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/64.jpg)
15-445/645 (Fall 2020)
P3 P4
P1 P2
DECENTRALIZED COORDINATOR
33
ApplicationServer
Begin Request
PartitionsMaster Node
![Page 65: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/65.jpg)
15-445/645 (Fall 2020)
P3 P4
P1 P2
DECENTRALIZED COORDINATOR
33
ApplicationServer
Query Request
PartitionsMaster Node
![Page 66: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/66.jpg)
15-445/645 (Fall 2020)
P3 P4
P1 P2
DECENTRALIZED COORDINATOR
33
ApplicationServer
Safe to commit?
Commit Request
PartitionsMaster Node
![Page 67: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/67.jpg)
15-445/645 (Fall 2020)
DISTRIBUTED CONCURRENCY CONTROL
Need to allow multiple txns to execute simultaneously across multiple nodes.→ Many of the same protocols from single-node DBMSs
can be adapted.
This is harder because of:→ Replication.→ Network Communication Overhead.→ Node Failures.→ Clock Skew.
34
![Page 68: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/68.jpg)
15-445/645 (Fall 2020)
DISTRIBUTED 2PL
35
Node 1 Node 2
NETWORK
Set A=2
A=1
Set B=7
B=8
ApplicationServer
ApplicationServer
![Page 69: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/69.jpg)
15-445/645 (Fall 2020)
DISTRIBUTED 2PL
35
Node 1 Node 2
NETWORK
Set A=2
A=1A=2
Set B=7
B=8B=7
ApplicationServer
ApplicationServer
![Page 70: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/70.jpg)
15-445/645 (Fall 2020)
DISTRIBUTED 2PL
35
Node 1 Node 2
NETWORK
Set A=2
A=1A=2
Set B=7
B=8B=7
ApplicationServer
ApplicationServerSet B=9 Set A=0
![Page 71: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/71.jpg)
15-445/645 (Fall 2020)
DISTRIBUTED 2PL
35
Node 1 Node 2
NETWORK
Set A=2
A=1A=2
Set B=7
B=8B=7
ApplicationServer
ApplicationServerSet B=9 Set A=0
Waits-For Graph
T1 T2
![Page 72: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/72.jpg)
15-445/645 (Fall 2020)
CONCLUSION
I have barely scratched the surface on distributed database systems…
It is hard to get this right.
36
![Page 73: 22 Introduction to Distributed Databases · 13 Node Application Server Node Node P3 →ID:101-200 P1→ID:1-100 P2→ID:201-300. 15-445/645 (Fall 2020) EARLY DISTRIBUTED DATABASE](https://reader033.vdocuments.net/reader033/viewer/2022060920/60ac0d8e12d69d30610311f2/html5/thumbnails/73.jpg)
15-445/645 (Fall 2020)
NEXT CL ASS
Distributed OLTP Systems
Replication
CAP Theorem
Real-World Examples
37