ScaleDBTransactional Shared Disk storage engine for MySQL

Moshe Shadmon, Founder, CTO

The product

Storage Layer

MySQL-ScaleDB Interface

DBMS/Apps

Storage Engine

ScaleDB Benefits

Enables open source DBMS (eg: MySQL) to operate as clustered, high end database

Scales applications in cloud environments No need to change the applications Provides high performance Provides unified data store

Shared Disk vs. Shared Nothing

Shared Nothing Shared Disk

Masters

Slaves

SQL ServerDB2 (Unix)MySQLPostgreSQL

Oracle RACIBM - DB2, IMS (MF)ScaleDB

Shared Disk vs Shared Nothing

Shared Nothing Shared Disk

Storage Per DBMS machine Shared storage (NAS/SAN)

Scalability By data partitioning By adding CPU

Scaling complexity High Low

High Availability Master-Slave Multi-Masters

HA complexity High Low

Implemented By MySQLPostgreSQLSQL ServerDB2 (Unix)

Oracle RACIBM on mainframe (DB2/IMS)ScaleDB

Shared disk provides “out of the box” scalability and availability

Currently available with the high end commercial databases only

Server 1

OSS DBMSOSS

DBMS

ScaleDBScaleDB

VM

The Virtualized Cloud Database

Local DiskLocal Disk

OSS DBMSOSS DBMS

Storage EngineStorage Engine

MySQL Server Server 2

OSS DBMSOSS

DBMS

ScaleDBScaleDB

VM

OSS DBMSOSS

DBMS

ScaleDBScaleDB

VM

OSS DBMSOSS

DBMS

ScaleDBScaleDB

VM

OSS DBMSOSS

DBMS

ScaleDBScaleDB

VM

Shared StorageShared Storage

Shared Disk

Shared Nothing

Standby Cluster ManagerStandby Cluster Manager

Deploying ScaleDB

…

ScaleDB Cluster ManagerScaleDB Cluster Manager

Node 1

DBMSDBMS

ScaleDBScaleDB

Node 2

DBMSDBMS

ScaleDBScaleDB

Node N

DBMSDBMS

ScaleDBScaleDB

ApplicationApplicationApplication Layer

Database Layer

(Physical or VM nodes)

Storage LayerShared StorageShared Storage Shared StorageShared Storage

ScaleDB

Benefits of ScaleDB Technology

High Availability – No single point of failure

If a node fails – a different node takes over If the cluster manager fails – the standby cluster manager is

initiated to manage the cluster

Scalability No limit on the number of nodes that can join the cluster Algorithms to minimize the need to communicate

Performance Innovative, high performance, general purpose indexing

ScaleDBCluster Manager

ScaleDB

Node

ScaleDB APIScaleDB API

Transaction ManagerTransaction Manager

IndexManager

IndexManager

DataManager

DataManager

Buffer ManagerBuffer Manager

Lock Manager

Local Lock Manager

Local Lock Manager

Global Lock CoordinatorGlobal Lock Coordinator

Log ManagerLog Manager

RecoveryManagerRecoveryManager

Storage ManagerStorage Manager

Global Recovery Manager

Global Recovery Manager

Global LockManager

Global LockManager

Global BufferManager

Global BufferManager

File SystemFile System

Storage DevicesStorage DevicesStorage DevicesStorage Devices

ScaleDB’s Internal Architecture

ApplicationsApplications XMLXML

ScaleDBCluster Manager

ScaleDB

Node

Read Customer Info

Read Customer Info

File SystemFile System

Storage DevicesStorage DevicesStorage DevicesStorage Devices

Distributed Lock Manager

Index translates logical id to Row ID

Index translates logical id to Row ID

Shared DBMS lock

Shared DBMS lock

Shared Table lock

Shared Table lock

Shared Row lockShared

Row lock

DBMS XDBMS X

Table YTable Y

Block ZBlock Z

Node 1 R/W

Node 1 R/W

Node 1 R

Node 2 R/W

Node 2 R/W

Node 2 R

Node 2 R/W

H List

H List

H List

W List

Shared Disk in the Cloud

Cloud provides Scalability and HA

Scalability - Enables dynamic allocation of computing and storage

Availability – Multiple servers over the same physical data High performance

Reduces Management Overhead

Reduced tuning and partitioning support No manual slave promotion or replication

Performance / Scalability

Benchmarks

Join operations using Multi-Table index

Sysbench

TPCC

Example multiple MySQL + ScaleDB nodes on Amazon EC2

Demo

Benchmark multi-table index

0

10

20

30

40

50

60

Engine X Join ScaleDB MTI ScaleDB 2 Nodes

Queries/Sec

Sysbench Performance on EC2

775 TPS

415

685

954

1224

1494

1764

2033

Number of Nodes1

2303 TPS

1 2 3 4 5 6 7 8

Transactions Per Second

ScaleDB

Other Engine

Current & Future Work

Optimizing TPCC – soon to be published

Integrating with other environments

GA – later this year

Demo