azure - data platform

Azure Data PlatformJesus AguilarSr Technical Evangelist Microsoft

@giventocodeMicrosoft Azure

Data Platform Components

A NoSQL document store that

offers fast time to market,

differentiated querying and

tuning capabilities with

schema-less web scale

Relational Non-Relational

SQL DB (PAAS) Table Storage File/ Blob StorageDocument DBSQL VM (IAAS)

SQL Database

Starting With The BasicsSQL Database

The Basics

SQL Server database technology as a service

Fully Managed

Enterprise-ready with automatic support for HA

Designed to scale out elastically with demand

Ideal for simple and complex applications

A Server Is Not A Machine

How It Works

Client Layer - Used by application to communicate directly with SQL Database.

Services Layer – Gateway between Client layer and Platform layer.

Platform Layer – Includes physical servicers and services that support the Services layer.

Infrastructure Layer – IT administration of the physical HW and OS.

PHPWCF Data

Services

SQL Server

Applications

and Tools

ODBC ADO.NET

Tabular Data Stream (TDS)

Server Provisioning

Service head that contains databases

Connect via automatically generated FQDN (xxx.database.windows.net)

Initially contains only a master database

Log on to Microsoft Azure Management Portal

Create a SQL Database server

Specify admin login credentials

Add firewall rules and enable service access

Use Microsoft Azure Platform PowerShell cmdlets (or use REST API directly)

wappowershell.codeplex.com

Selecting the right EditionService

Tier

Performance

Level

Common App

Pattern

Performance Business Continuity

Max DB

Size

Trans. Perf.

Objective

DTUs PITR DR / GEO-Rep

Basic Basic Small DB, SQL opp 2 GB Reliability / Hr. 5 Past 7

Days

DB Copy +

Manual Export

Standard S1 / S2 Wrkgp/cloud app,

multiple concurrent

operations

250 GB Reliability / Min. 15/ 50 Past 14

Days

DB Copy +

Manual Export

Premium P1 / P2 / P3 Mission Critical, High

volume, Many

concurrent Users

500 GB Reliability / sec. 100/

200/

800

Past 35

Days

Active Geo-

replication

DemoCreating A SQL Database Server

Create And Deploy

Your Database

Create Database…

Transact-SQL

Languages

.NET Framework (C#, Visual Basic, F#) via ADO.NET

C / C++ via ODBC

Java via Microsoft JDBC provider

PHP via Microsoft PHP provider

Frameworks

OData, Entity Framework, WCF Data Services, NHibernate

Tools

SQL Server Management Studio (2008 R2 and later)

SQL Server command-line utilities (SQLCMD, BCP)

CA Erwin® Data Modeler

Embarcadero Technologies DBArtisan®

Focus on logical vs. physical administration

Database and log files automatically placed

Three high-availability replicas maintained for every database

Tables require a clustered index

Maximum database size is 500 GB

Use command, distributed transactions, distributed views

Service Broker

Common Language Runtime (CLR)

SQL Agent

SQL Profiler

Native Encryption

Enhanced Tooling

Web designers for tables, views, stored procs

Interactive query editing and execution

Visual Studio IDE for database development

Includes modern designers and projects with declarative, model-driven development

Develop and test in both connected and disconnected states

Platform targeting for both SQL Server (2005 and above) and SQL Database

Get it free with Web PI, with SQL Server 2012 and with Visual Studio 11

Database Deployment

Alternative to traditional script based approach

Dramatically simplifies deployment, migration and versioning of databases

Provides a single unit of deployment for schema (dacpac) or for schema + data (bacpac)

Supports automatic versioning of database schemas

Supports platform targeting for both SQL Server (2005 and above) and SQL Database

Build from scratch or extract from existing db

With SQL Server Data Tools

With SQL Server 2012/2014 Management Studio

With SQL Database Import/Export Service

Via sqldacexamples.codeplex.com

Secure Your DatabaseSQL Database

There Are Two Ways To Secure A Database:

Server Benefits

SQL authentication supported (No Integrated authentication)

The Admin login is similar to sa

Connect to master to administer logins

loginmanager: Server-Level security role for creating logins

dbmanager: Server-Level security role for creating databases

Database Benefits

Logins require an associated user account

The Admin login is automatically associated with dbo

The dbo has full rights in the database

Manage users with CREATE / ALTER / DROP USER commands

Add users to roles via sp_add_rolemember to grant privileges

Utilize schemas where appropriate

SQL Database Firewall

• IP Address-based access control for SQL Database

• Rules can be defined at the server and database

• No IP authorized by default

• Configurable using the SQL Database Portal and REST API

• Option to disable/enable access from applications hosted in Microsoft Azure

Application Connectivity

1. TDS (Tabular Data Stream) protocol over TCP/IP supported

2. SSL required

3. Use firewall rules to connect from outside Microsoft data center

ASP.NET EXAMPLE:

1. login: [login]@[server]

2. Idle connections

3. Long running transactions

4. DoS guard

5. Failover events

6. Throttling

7. Connection pooling and Retry logic

8. Latency introduced for updates

9. No cross-database dependencies

<connectionStrings><addname="AdventureWorks"connectionString=

"Data Source=[server].database.windows.net;Integrated Security=False;Initial Catalog=ProductsDb;User Id=[login];Password=[password];Trusted_Connection=False;Encrypt=true;"

providerName="System.Data.SqlClient"/></connectionStrings>

Elastic SQL Database – Scaling out!

• .NET Client Libraries• Management of Shards

• Data Access

SQL on IaaS

Run SQL on VM

• Run any SQL product on cloud VM

• Support for SQL Server, Oracle, MySql

• Ready to go VM images available in Gallery

• Persistent storage using attached disk in blob storage

23Microsoft Azure

SQL Database vs SQL IaaS Comparison

24Microsoft Azure

Azure Storage Architecture

“Microsoft Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM

Symposium on Operating System Principals (SOSP), Oct. 2011

Azure Files

“I wish I could go to storage and provision a cloud drive, giving it a namespace, and that drive would then be UNC-addressable by the OSes.”

Azure Files – Customer Quotes

• Setup an IaaS VM to host a File Share backed by an IaaS Disk• Write code to find the IaaS File Share from the rest of the VMs in

your service.• Write some code to provide high availability

• Handle host upgrades, node failures

• You can only access the File Share from other VMs

Sharing Files – The old way

Azure Files•

•

•

Azure Files

Azure Files

Website Served From Azure File Share

Load Balancer

Azure

VM

Azure

VM

…

Blob Storage

Blob Storage Concepts

Blob Details

Blob Details

Blob Details

Blob Containers

• Special $root container

• A container holds a set of blobs• Set access policies at the container level • Associate Metadata with Container• List the blobs in a container• Including Blob Metadata and MD5 • NO search/query. i.e. no WHERE MetadataValue = ?

• Effectively in Partition of 1• Target of 60MB/s per Blob

Enumerating Blobs

• Prefix

• Delimiter

• Include= (snapshots, metadata etc…)

Pagination

• Either set maxresultsor;

• Exceed default value for maxresults (5000)

Two Types of Blobs Under the Hood

Uploading a Block Blob

Uploading a large blob

BenefitEfficient continuation and retry

Parallel and out of order upload of blocks

Microsoft Azure

Storage

Page Blob – Random Read/Write

Shared Access Signatures

• Use short time periods and re-issue

• Use container level policy that can be deleted

• Ad-hoc

• Policy based

Ad Hoc Signatures

• Signedresource Blob or Container

• AccessPolicy Start, Expiry and Permissions

• Signature HMAC-SHA256 of above fields

• Single use URLs

• E.g. Provide URL to mobile client to upload to container

Policy Based Signatures

• Specify StartTime, ExpiryTime, Permissions

• Signedresource Blob or Container

• Signedidentifier Optional pointer to container policy

• Signature HMAC-SHA256 of above fields

• Providing revocable permissions to certain users/groups

• To revoke: Delete or update container policy

NoSQL48

Generally scales more easily

• The storage engines of NoSQL stores are designed to minimize contentions enabling higher throughput and therefore more scalable

• Lower transaction capability in NoSQL results in less contention and therefore more scalable

• Less complex query processor means that a single query can’t degrade service

• Built-in replication capability means that store can scale out which better aligns to other application tiers (e.g. websites)

• No fixed schema or lower schema requirements

49Microsoft Azure

NoSQL on Azure

• Azure Tables service is NoSQL row store

• DocumentDB born in the cloud document database (JSON) and JS (PAAS).

• HBase is a Big Data (Hadoop) NoSQL store available in HDInsight

• MongoDB is a document (JSON) store

• Cassandra is a columnar store with excellent replication

50Microsoft Azure

Tables51

Table Storage Concepts

Table Details

Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity

Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort order

Timestamp Optimistic ConcurrencyExposed as an HTTP Etag

No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double

No Fixed Schema

Querying

Purpose of the PartitionKey

Entity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries

Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction

Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountMicrosoft Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table

Partitions and Partition Ranges

DocumentDB

Key ConsiderationsNimble document database

Scalable document database as a service

Extensibility through JavaScript

Cross-Platform access

Transactions

Configurable Consistency Levels

Schema-Free

Document Indexing

Resource Model

Database

TenantCollection

Document

Attachment

User

/users/{id}

Permission

/dbs/{id}

/colls/{id}

/docs/{id}

/attachments/{id}

/permissions/{id}

Document

Service

/Addresses

/addresses

Items and Feeds (Nouns)Natively stored and served as JSON

URI addressable

Partitioned for scale-out

Replicated for HA

Granular access control

A subscription can provision 1+ Document Services

Can run in shared or dedicated mode

Each can have 1+ databases

Interaction ModelRESTful interaction over HTTP

Standard HTTP verbs & semantics

Interact using your favorite HTTP client

Built-in Support for TCP

Novel, efficient and powerful document centric query model

Javascript based sprocs/triggers /evals

POSTItem

resource TenantFeed URI

PUTItem

resource Item URI

DELETE Item URI

GET TenantFeed Or

Item URI

Create a new resource

/Execute a script

Replace an existing resource

Delete an existing resource

Read/Query an existing

resource

Update an existing resource

PATCH Item URIItem

resource

location headquarters exports

Belgium 0 1

city

Moscow

city

Athens

0

country city

Germany Berlin

1

country city

France Paris

0

headquarters exports

country city

Italy 0 1

Germany Bonn

city dealers

Berlin 0

city

Amsterdam

name

Hans

location

Azure

footprint

16 regions worldwide in 2014

Fortune 500 using Azure

>57% >250kActive websites

Greater than

1,000,000SQL Databases in Azure

>20TRILLIONstorageobjects >300 MILLION

AD users

>13 BILLIONauthentication/wk>2

MILLIONrequests/sec >1

MILLIONDevelopers

registered with

Visual Studio

Online

Get startedVisit azure.microsoft.com

SQL Database Billing Rates (As of February 2012)

Database Size Price Per Database Per Month

0 to 100 MB Flat $4.995

> 100 to 1 GB Flat $9.99

> 1GB to 10 GB $9.99 for first GB, $3.99 per additional GB

> 10 GB to 50 GB $45.954 for first 10 GB, $1.998 for each additional GB

> 50 GB to 150 GB $145.874 for first 50 GB, $0.999 for each additional GB

SQL Database Architecture