building scale-out database solutions on sql azure

8/8/2019 Building Scale-Out Database Solutions on SQL Azure

1/28


2/28

Building Scale-out DatabaseApplicationswith SQL Azure

Lev NovikSoftware ArchitectMicrosoft Corporation


3/28

Agenda

SQL Azure --- what is it for?

A quick recap

Why scale-out?

Building scale-out solutions today

Scaling through databases

Futures


4/28

Microsoft SQL Azure

Scale on Demand Innov

Information Platform as a Service

Managed Service

( , , )ies delivered as a service Database Data Sync Reporting

, greater flexibility

and deployment

ility and fault tolerant

Database as a utility with pay as you grow scaling

-Rely on Business ready SLAs

-Enable multi tenant solutions

Manage multiple servers

Innovate with new

data applicatio

-Build cloud based data solutions

Build on existing developer skill


5/28

Provisioning Model

AccountAccount

ServerServer

DatabaseDatabase

Each accounthas zero or more servers

Establishes a billing instrument

Each serverhas one or more databasesLogical concept equal to a master DBUnit of authentication, geo-location, breportingGenerated DNS-based name

Each database has standard SQL objects

Users, Tables, Views, Indices, etcUnit of consistency


6/28

Scaling database applications

Scale upBuy large-enough server for the job

But big servers are expensive!Try to load it as much as you can

But what if the load changes?

Provisioning for peaks is expensive!

Scale-outPartition data and load across many servers

Small servers are cheap! Scale linearly

Bring computational resources ofmany to bear800 little servers is very fast

Load spikes dont upset usLoad balancing across the entire data center


7/28

Scale-out with SQL Azure Today

Elastic Provisioning of Databases

CREATE DATABASE and goNo VMs, no servers

Pay-as-you-go business modelDont need it --- DROP it

Zero Physical AdministrationBuilt-in High Availability, patching, maintenance

Database Copy, SQL Azure Data Sync


8/28

Ticket DirectSolutionSolution

Elastic scale databElastic scale dat abserviceservicePay as you grow andPay as you grow and

Easy to provision anEasy to provision anddatabasedatabase

No hardware, no m aNo hardware, no m adatabase adm inistrdatabase adm inistr

requiredrequired

Promotions, events, t icket sellingPromotions, event s, t icket selling

businesses are bursts bound bybusinesses are bursts bound bynaturenature

Capacity constraints limit businessCapacity constraints limit businessagilityagility

High costs of entry into new businessHigh costs of entr y into new business

Difficult to roll out extra capacityDifficult to roll out extra capacityquicklyquickly

Idle capacity off-bursts is costIdle capacity off-bursts is cost

prohibitiveprohibitive

Capacit

y

Time

Capacity Bursting

AverageUsage

Average UsageComput

e

#ofHrs

Challenges TodayChallenges Today


9/28

Scale-out for Multi-tenant applicati

Put everything into one DB? Too big

Create a database per tenant? Not badSharding Pattern: better

Application is already prepared for it!

T1 T2 T3 T4 T5

T6 T7 T8 T9 T10

T11 T12 T13 T14 T15

T16 T17 T18 T19 T20

T1T2 T3 T4

T5

T6 T7 T8 T9 T10

T11 T12 T13 T14 T15

T16 T17 T18 T19 T20

Ais

one DB


10/28

Sharding Pattern

Linear scaling through database

independenceNo need for distributed transactions incommon cases

Application-influenced partitioningRather than complete transparency

Local access for mostConnection routing

Query, transaction scoping

Distributed access for some

App


11/28

DIY Sharding: Problems

ProvisioningGrowing and shrinking capacity

ManagingUpgrading, patching, HA for lots ofdatabases

RoutingWhere is the directory?How to scale it and use it?

Partition ManagementSplitting and Merging, without loss ofavailability

Covered

bySQL

Azuretoday

Comingup in SQL

Azure:Federatio

ns


12/28

Distribution of Data

Partitioned

Spread across membermachines

Each piece is on onemachine (+HA)

Most of the data!

CentralizedOnly available in one place

Read and write, but not toomuch

ReplicatedCopied to all member

Data1ref

Data2ref

Data3ref

Con-fig


13/28

SQL Azure Federations: Concepts

Federation

Represents the data being shardedFederation Key

The value that determines the routing ofa piece of data

Atomic UnitAll rows with the same federation keyvalue: always together!

Federation Member (aka Shard)A physical container for a range ofatomic units

Federation Root

The database that houses federationdirector

R

Federatio

Member

AUPK=

5P

MemberAU

PK=105

P

Member

AUPK=55

5

P

(FederaCustID)


14/28

Creating a Federation

Create a root database

CREATE DATABASE SalesDBLocation ofpartition map

Houses centralizeddata

Create the federation in rootCREATE FEDERATION Orders_Fed(RANGE BIGINT)

Specify name, federation key typeStart with integral, guid types

Creates the first member, coveringthe entire range

Sa

FedOrde

(FederatioMembe


15/28

Creating the schema

Federated tables

CREATE TABLE orders () FEDERATE ON (customFederation key must be in all unique indices

Part of the primary key

Value of federation key will determine the memb

Reference tables

CREATE TABLE zipcodes ()Absence of FEDERATE ON indicates reference

Centralized tablesCreate in root database

FedOrde

(Federat ioMem

Sa

orders

Pr


16/28

Splitting and MergingSplitting a member

When too big or too hot

ALTER FEDERATION Orders_FedSPLIT (100)

Creates two new membersSplits (filtered copy) federated data

Copies reference data to both

Online!

Merging membersWhen too small

ALTER FEDERATION Orders_FedMERGE (200)

FedOrde

(Federat ioMem

Sa

orders

Pr

Membeorders

Membe

orde


17/28

Connecting and Operating

Connect to atomic unitUSE FEDERATION Orders_Fed (56) WITHFILTERING=ON

Connection routed to member containing 56

Only data with federation key value 56 isvisible

Plus reference data

Safe: atomic unit can never be split

Connect to entire federation memberUSE FEDERATION Orders_Fed (56) WITHFILTERING=OFF

Connection routed to member containing 56

All data within the member database is

Memb

AUPK=

5


18/28

Schema Distribution

Federation members can have different schemas a

in time:Temporary, while schemas are being upgradedTemporary, while customer is testing new schema on so

Permanently, because shards are different

To alter schema:Manually

Connect to each federation member (USE FEDERATION Orders(5FILTER=OFF)

Alter it (ALTER TABLE Customers )

Future: schema-distribution serviceConnect to root

Manage and apply schemas asynchronously


19/28


20/28

Sharding in SQL Azure: Beyond v1

Schema Management

Allow multi version schema deployment and managemefederation members.

Fan-out QueriesAllow single query that can process results across large federation members.

Auto RepartitioningSQL Azure manages the federated databases for you thsplits/merges based on some policy (query response tietc)

Multi Column Federation KeysFederate on enterprise_customer_id+account_id


21/28

Summary

Scale-out is the way to build cloud solutions

You can build scale-out solutions with SQL Azure toProvisioning, auto-management, pay-as-you-go are you

Sync and Database Copy help

Implement sharding pattern today --- slide into federatiotomorrow!

Federation support (2011) will make it a lot easierPartition management

Routing

On-line splitting and merging


22/28

. . , , / . . / 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks and or trademarks in the U S and or o. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market cond

, . , , a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STA

.PRESENTATION

W lk th h t


23/28

Walk-through: setup

CREATE FEDERATION Orders_Fed (RANGE BIGINT)

USE FEDERATION Orders_Fed(0) WITH FILTERING=OFF

CREATE TABLE orders(orderid bigint, odate datetime, customerid bigint,

primary key (orderid, customerid))

FEDERATE ON (customerid)

CREATE UNIQUE INDEX o_idx1 on orders(customerid, odate)

CREATE INDEX o_idx2 on orders(odate)

CREATE TABLE orderdetails(orderdetailid bigint, orderid bigint, partid bigint, customer

primary key (orderdetailid, customerid))

FEDERATE ON (customerid)

ALTER TABLE orderdetails add constraint orderdetails_fk1 foreign key(orderid,customeorders(orderid,customerid)

CREATE TABLE uszipcodes(zipcode nvarchar(128) primary key, state nvarchar(128))

W lk th h l k


24/28

Walk-through: real work

Connect to: InitialCatalog=SalesDB

get some regular work done (within customer 239)

USE FEDERATION Orders_fed(239) WITH FILTERING=ON

SELECT * FROM Orders JOIN OrderDetails ON

INSERT INTO Orders (customerid, orderid, odate) VALUES (239, 2, 5/7/2010)

get some cross-customer work done

USE FEDERATION Orders_fed(0) WITH FILTERING=OFF

DELETE from Orders WHERE odate < 1/1/2000

- Repeat for other members

-- go back to root

USE FEDERATION ROOT

UPDATE CleanupSchedule set LastCleanupDate = GETSYSTIME()

W lk th h d d !


25/28

Walk-through: ups and downs!

--Day#2 business grows!

ALTER FEDERATION Orders_Fed SPLIT AT(1000)

--Day#3 black friday!ALTER FEDERATION Orders_Fed SPLIT AT(100)

ALTER FEDERATION Orders_Fed SPLIT AT(200,300,400)

--Day#4 recession hits!ALTER FEDERATION Orders_Fed MERGE AT(100)

--Day#5 oh boy double dip.ALTER FEDERATION Orders_Fed MERGE AT(200,300,400)


26/28

SQL Azure Data Sync V1 Overview


27/28

SQL Azure Data Sync V1 Overview

On-Premises (Headquarters)

Sync

Sync

Remote OData Sync ServiceFor SQL Azure

Retail Stores

Sync

Sync

Sync

SQL AzureDatabase

Sync Sync


28/28

. . , , / . . / 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks and or trademarks in the U S and or o. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market cond

, . , , a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STA

.PRESENTATION

building scale-out database solutions on sql azure

Documents