Windows Azure StorageCloud Computing Soup to Nuts

Mike BenkovichMicrosoft Corporationwww.benkoTips.com - @mbenko

btlod-72

Agenda

Storage OverviewGetting startedWorking with BlobsAdding Tables and

QueuesWorker Roles

Core Services

Caching CDN

Service Bus Reporting

Data Sync Azure Connect

Identity HPC

Additional Services

Windows Azure

Compute Storage Database

Windows Azure StorageStorage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service usesInstant Concurrency

Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet

Wrapped in API by SDKMicrosoft.WindowsAzure.StorageClient

Windows Azure Storage Abstractions

TablesStructured storage. A table is a set of entities; an entity is a set of properties.

QueuesReliable storage and delivery of messages for an application.

BlobsSimple named files along with metadata for the file.

DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.

Windows Azure Storage Account

Create account in Management Portalhttp://windows.azure.com

Geographically located with affinityYou get 2 keys…API based on REST… http://btlod.blob.core.windows.net/public/helloworld.txt

http://<account>.<BLOB|TABLE|QUEUE>.core.windows.net/...

Tools to Explore Storage Many tools are available some for free, some from 3rd parties, including:

Visual Studiohttp://myAzureStorage.cloudapp.nethttp://clumsyleaf.comhttp://cerebrata.com

These give access to our storage account

Storage Accounts

Demo 1

Windows Azure Storage AccountCan CDN Enable AccountBlobs delivered via 24 global CDN nodes

Can co-locate storage account with compute accountExplicitly or using affinity groups

Accounts have two independent 512 bit shared secret keys

100 TBs per account

Storage in the Development FabricDeveopment Storage Emulator provides a local “Mock” storageEmulates storage in cloudAllows offline developmentRequires SQL Express 2005/2008 or aboveSSMS Provides local view of storage items

http://msdn.microsoft.com/en-us/gg433135

Scalability TargetsStorage Account• SLA – 99.9% Availability• Capacity – Up to 100 TBs• Transactions – Up to 5000 requests per second• Bandwidth – Up to a few hundred megabytes per second

Single Queue/Table Partition• Up to 500 transactions (entities or messages) per second

Single Blob Partition• Throughput up to 60 MB/s

Scale Above the limits• Partition between multiple storage accounts and partitions• When limit is hit, app may see ‘503 server busy’: applications

should implement exponential back-off

Working with BlobsBinary Large Objects include resources we use in our applications like pictures, videos, html, css, etc. that may not fit best in a relational schema

Choice of API’s – REST vs. SDK wrapper To use SDK add references to Microsoft.WindowsAzure.StorageClient Create Storage Account Object Add client for blobs Work with container references and blobs

Our example…WPF Cloud Explorer Basic WPF Application Enabled drop events – iterate thru collection of

files Create a blob container and upload blobs

Cloud Explorer

Uploading Blobs

Demo 2

Table Storage ConceptsEntityTableAccount

contoso

Name =…Email = …

Name =…EMailAdd=

customers

Photo ID =…Date =…

photos

Photo ID =…Date =…

Table Details

InsertUpdate Merge – Partial updateReplace – Update entire entityUpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction

Create, Query, DeleteTables can have metadata

Not an RDBMS! Table

Entities

Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity

Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort order

Timestamp Optimistic ConcurrencyExposed as an HTTP Etag

No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double

No Fixed Schema

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

FAV SPORT

Canoeing

Querying

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

?$filter=Last eq ‘Wegner’

Purpose of the Partition KeyEntity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries

Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction

Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE) TIMESTAMP MODELYEAR

Bikes Super Duper Cycle … 2009

Bikes Quick Cycle 200 Deluxe … 2007

… … … …Canoes Whitewater … 2009Canoes Flatwater … 2006

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE) TIMESTAMP MODELYEAR

Rafts 14ft Super Tourer … 1999

… … … …

Skis Fabrikam Back Trackers … 2009

… … … …Tents Super Palace … 2008

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE) TIMESTAMP MODELYEAR

Bikes Super Duper Cycle … 2009

Bikes Quick Cycle 200 Deluxe … 2007

… … … …Canoes Whitewater … 2009Canoes Flatwater … 2006

Rafts 14ft Super Tourer … 1999

… … … …

Skis Fabrikam Back Trackers … 2009

… … … …Tents Super Palace … 2008

Partitions and Partition Ranges

Server A

Table = ProductsServer

BTable = Products[Canoes - MaxKey)

Server A

Table = Products[MinKey - Canoes)

Table PropertiesEntity can have up to 255 propertiesUp to 1MB per entity

Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)

Uniquely identifies an entityDefines the sort order

Timestamp Optimistic ConcurrencyExposed as an HTTP ETag

No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double

Loosely Coupled Workflow with QueuesEnables workflow between rolesLoad work in a queueProducer can forget about message once it is in queueMany workers consume the queueFor extreme throughput (>500 tps) Use multiple queuesRead messages in batchesMultiple work items per message

Queue

Input Queue (Work Items)Web Role

Web Role

Web Role

Worker Role

Worker Role

Worker Role

Worker Role

Queue DetailsSimple asynchronous dispatch queue No limit to queue length subject to storage limit 64kb per message ListQueues - List queues in accountQueue operations CreateQueue DeleteQueue Get/Set Metadata Clear MessagesMessage operations PutMessage– Reads message and hides for time period GetMessages – Reads one or more messages and hides them PeekMessages – Reads one or more messages w/o hiding them DeleteMessage – Permanently deletes messages from queue UpdateMessage – Clients renew the lease and contents

Queue’s Reliable Delivery

Guarantee delivery/processing of messages (two-step consumption)

Worker dequeues message and it is marked as Invisible for a specified “Invisibility Time”

Worker deletes message when finished processing

If Worker role crashes, message becomes visible for another Worker to process

Message lifecycle

Queue

Msg 1

Msg 2

Msg 3

Msg 4

Worker Role

Worker Role

PutMessage

Web Role

GetMessage (Timeout)

RemoveMessage

Msg 2Msg 1

Worker Role

Msg 2

Creating ThumbnailsWPF Uploader Blob uploads are good Can leverage in web applications Nice to have thumbnails created

Next steps…catalog of uploads Create model for table storage Add class library to be shared between

projects Define queue for work to generate thumbs Add queue message for image files

Catalog

Adding table and queue messages…

Demo 3

Worker ProcessIdeally suited for Worker Role Read queue message with work Process thumbnail Update catalog table Sleep for predetermined time

Worker Role

Process thumbnails

Demo 4

Windows Azure Storage SummaryFundamental data abstractions to build your applicationsBlobs: Files and large objectsDrives: NTFS APIs for migrating applicationsTables: Massively scalable structured storageQueues: Reliable delivery of messages

Easy to use via the Storage Client LibraryHands on Labs

Where can I get more info?• Visit my site http://www.benkotips.com

• Resources from today’s talk• Webcasts• Downloads

• Check out the rest of this series!• http://bit.ly/s2nCloud

• Ask questions on Windows Azure Office Hours• http://bit.ly/bqtWazOH

• Get the Tools & the Trial • http://aka.ms/AzureMB• http://aka.ms/AzureTrialMB

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after

the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.