Architecting Azure Solutions Microsoft

Azure Storage Overview70-534

Architecting Microsoft Azure Solutions

Two Part Presentation

Introducing Azure Storage

Cloud Application RequirementsDurable and highly availableScalable and performant Multiple concurrent usersSecureManageable Time to market

Introducing Microsoft Azure StorageCloud Storage - Anywhere and anytime

accessBlobs, Disks, Tables, Queues and Files

Highly Durable, Available and Massively Scalable

Easily build “Internet scale” applicationsMore than 35 trillion stored objects3.5+ Million requests/sec on average

Pay only for what you useExposed via easy and open REST APIsRich Client Libraries and Tools

17 Regions Worldwide in 2015

Abstractions – BlobsBlobs – Massively scalable object store in the cloud

Simple REST interface (Put, Get, Delete)Data sharing – share documents, pictures, video, music, etc.Big Data – store raw data/logs and compute/map reduce over dataBackups – data and device backups

Abstractions – DisksPersistent disks for VMs in Azure Disks are VHDs stored in Azure Page BlobsPage blobs are optimized for random I/OVM see the VHD/Blob as a disk

Reads translated to GETs, writes to PUTsBlob protected by write-leaseReads from the blob (and snapshots) still allowed

Breaking News – Premium StorageConsistent low latency SSD based with predictable IO throughputSuitable for high-performance IO-intensive database workloadsSingle digit milliseconds latenciesSupports up to 1 TB blob/disk sizeStripe up to 32 disks for a total of

32TB and more than 50,000 IOPSPremium Storage Disks work in

conjunction with a new VM series

Abstractions – TablesTables – Massively scalable NoSQL cloud store

Key/Attribute(s) store at scaleStore user, device or any type of metadata for your serviceAuto load balances partitions to meet traffic needsOData protocol (AtomPub or JSON)

Abstractions – QueuesQueues – Reliable messaging system

Reliable, low latency, high throughput messaging systemDecouple components/roles

Web role to worker role communicationAllows roles to scale independently

Implement scheduling of asynchronous tasksBuilding process/work flows

Abstractions – FilesMove on-premises applications to cloudVMs can net use an SMB share using standard file APIs and semanticsSMB 2.1 protocolVM and storage account within same regionSupports REST and SMB protocol access to same file share

Azure Storage

Blobs

Tables

Queues

Files

Share data stored in Azure Files among Azure VMs via SMB

Microsoft Azure

SMB

REST

APIREST API

Additional Services, Tools and LibrariesAzure Import/Export

Move TBs of data into and out of Azure Blobs by shipping diskSubmit and monitor jobs via REST and PortalAll disks encrypted with BitLocker

Client LibrariesSimplify interaction with REST servicesConvenience capabilities – retry logic, parallelization, SAS tokens, … .NET (Desktop, Phone, Windows), Java, C++, Android, Node.JS, Python, PHP, Ruby, IOS (Coming Soon)

ToolsPowerShell commandsCLI toolsAzCopy 3.0 – Copy blobs and disks (Tables and Files in Preview Release)

Azure Storage Architecture

Massive Scale Out & Auto Load Balancing Index Layer

Distributed Replication Layer

Blob/Disk

EndpointQueue

EndpointTable

EndpointFile

ShareEndpoint

“Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM Symposium on Operating System Principals (SOSP), Oct. 2011

Client Libraries (.NET, Java, c++, Android, Node.JS…)

SMB Client

Import / Export

REST REST REST REST SMB 2.1

Stream Layer

Partition Layer

Front End Layer

Platform Capabilities

Locally Redundant Storage (LRS)Stores 3 replicas of the data within a single zone (facility) in a single regionProvides data durability for disk, node and rack failures

Geo Redundant Storage (GRS)Stores 6 replicas of the data across two regions (3 in each region)Provides additional durability to protect data against major regional natural disasters (e.g., tornado, hurricane, fire, etc, destroying a whole region)Updates across regions are performed asynchronously

Zone Redundant Storage (ZRS)Stores 3 replicas of the data across multiple zones (facilities) within a single region or across regions Provides additional durability to protect data against zone failures (e.g., fire burning down a facility)

Durability Options

Secondary Read-Only AccessCustomers using Geo Redundant Storage can opt to have read-only access to the eventually consistent copy of the data on Secondary tenantApplications control which location they read data from

Primary endpointaccountname.<service>.core.windows.net

Secondary endpointaccountname-secondary.<service>.core.windows.net

ConsistencyAll Writes go to the PrimaryReads to Primary are Strongly Consistent Reads to Secondary are Eventually Consistent

Our client library SDK provides features for reading from the secondary

Retry options: PrimaryOnly, SecondaryOnly, PrimaryThenSecondary, etc.

Access ControlSymmetric Shared Key AuthN

Trusted service that owns the storage accounts Shared Access Signature (SAS) Delegated AuthZ

3rd party servicesMobile device applicationsRestricted access for servicesAllow client applications to directly communicate with Storage rather than scaling a proxy web service

Public (Blob service only)CDN accessBrowser based access

ConcurrencyOptimistic Concurrency

Blobs and Tables support optimistic concurrencyBased on HTTP protocol (Etag and if-match conditions)

Pessimistic ConcurrencyBlocks provide exclusive access via leasesEnable synchronization strategies

Exclusive write / shared readExclusive write / exclusive read Shared write / exclusive read

Last Writer Wins

Storage AccountCapacity – Up to 500TBObjects per second: 20,000 1KB messagesScans/Listing considered as the number of objects read by the system and not the number of objects returnedBandwidth:

GRS: Up to 10 Gbps Ingress/20 Gbps EgressLRS: Up to 20 Gbps Ingress/30 Gbps Egress

Partition Table- Up to 2,000 1KB entities/secQueue - Up to 2,000 1KB messages/secBlob - Up to 60 MB/s, IOPS 500File - Up to 60 MB/s, IOPS 1000Check Azure Storage Scalability and Perf Targets

Scalability Targets

Storage AnalyticsStorage platform incorporates rich analytics

Metrics - Monitor service health, capacity, availability and performanceLogs – Detailed logs of every operation to support troubleshooting

Analytics is not enabled by default – we recommend enabling

During testing to establish baseline metricsIn production to assist with troubleshooting

Client libraries also enable access to metrics data via strongly typed objects Client libraries also have client side logging

Designing a Cloud Scale Application

Uniformly distributed traffic across the entire user partition namespace Range based partitioning does not cater to append/prepend patternAvoid spikes

Additional accounts to get more scale than a single accountDesign application to dynamically add accounts

Throughput: Use more simultaneous outstanding IOSingle blob: Parallel page writes or block writesMultiple blobs: Upload blobs in parallel

Scale Design Principles

Failures can occurRetry on Network/Connection failuresExponential back-off on timeout and throttling failures

Regional failures can occur Design for regional redundancyRA-GRS: Allows apps to continue with read only access to eventual consistent data

Network latencies can occur Latency sensitive apps could use cache.

Resilience Design Principles

Designing a SaaS solution to deliver Incident Management capabilities at hyper scale

Significant Use CasesCRUD operations on IncidentsGet Incidents assigned to a User

Architecture RequirementsHigh availability (99.99% for reads)Scalability beyond storage account limitsProvide direct access to storage to minimize complexity and cost

Scenario and Requirements

Use the right abstraction Key Lookups at scale for structured data – TablesScans/retrieve large amount of raw data like for analytics/metrics etc. – BlobsProcess workflows, decoupling applications – Queues

Storage Design Practices

Design for queryingSpecify both PartitionKey and RowKey in your queriesConsider storing duplicate copies of entities (with different keys)Consider denormalizing your dataDon’t create a separate table for each type of entityUse compound key valuesAvoid creating hot partitions

Table Design Principles

Table storage to store incident dataPK as Incident ID, RK as emptyIncident ID breakdown (OrgId, bucket ID, GUID)Table Property backed by blob

CRUD Operations

Partition Key(Incident ID)

Row Key Title(string) 

AssignedTo(string) 

Status (string) 

Description(string)

contoso _01_8743428c-ef91-4d05-9e7c-4a2e856e813a

Support Retrieval workflow

user5 Active http://storageaccount1.blob.core.windows.net/backingcontainer/01_8743428c-ef91-4d05-9e7c-4a2e856e813a

Querying Incidents data as is - Table scan not optimalMake “assignedto” part of the pk or rkDenormalize common properties to avoid hops – Title and StatusUse Queues to get eventual consistencyKey Pattern – Write to queue first with timeout , than update primary entityWorker role then updates the secondary indexDEMO

Get Incidents Assigned to User

Partition Key(PropertyName_Value)

Row Key(Incident ID)

Title(string) 

Status (string) 

AssignedTo_user5 contoso _01_8743428c-ef91-4d05-9e7c-4a2e856e813a

Support Retrieval workflow

Active

Design the solution to be resilient to local and geographical failures

SolutionReplicate data between two regions using GRSRead from secondary region in the event of failure in first regionClient SDK’s retry policy automates this with a configuration setting

Enable application to run in read only mode until recovery/failover

High Read Availability

Enable clients to interact directly with storage to remove the need for a storage proxy – decreasing latency and reducing complexity

SolutionIssue SAS tokens to Clients for delegated authorization

Additional benefitsSAS tokens can restrict access to tenant specific data

Provide Direct Access to Storage

Solution needs to scale beyond storage account limitsSolution

Create only as many storage accounts as needed todayKeep a map of OrgId+ bucket id to a storage account nameWhen a bucket id fills up or storage account reaches limits (capacity/throughput), create a new bucket id and pick a storage account from pool for storing data

Helps with write availability for new incidents in DR scenario

Pattern – Scale beyond single storage account

Partition Key(OrgID)

Row Key(bucket id/sharding Index)

Storage Account(string) 

IsActive(bool)

contoso 00 demo1 falsecontoso 01 demo2 trueelmaramsey 00 Demo1 true

Architectural Sketch

Storage Service

Table Service

Blob Service

Queue Service

Index Manageme

nt

Background processing

Service layerProvisioning

SAS Provider

P2.3 CRUD CRUD Incidents

P3. Update Indices

P1. Register New

Customer

P2.1 Get SAS TokenClientSign up/Admin

CRUD Incidents

P2.2 Lookup

Shard

RecapDesigning a SaaS solution to deliver Incident Management capabilities at hyper scale

Design Requirements and SolutionsMulti-tenant – SAS tokens for AuthZScale across storage accounts – Shard data across storage accountsOptimize resources – SAS tokens enable direct interaction with storageHigh availability in the face of intermittent and regional failures

Read access to secondary regionAbility to create new incidents in different storage accounts

Customizable incidents – Tables storageSupport different search strategies – Secondary indexesCoordinate updates to tables + blobs – Queue based coordination

Session Objectives And TakeawaysSession Objective(s): Provide an overview of the Storage platform abstractions and platform capabilitiesReview principles and patterns in the context of a real world scenarioWhat differentiates us:Strong consistency: Easier for applications to consumeGRS: Data replicated to DC 100s of miles apartRA-GRS: Eventual consistent access to your dataDisks & Azure Files: Enables “lift & shift” of applicationsStrong Development tools and libraries:

AzCopy, PowerShell, CLI, Client libraries in various languages

Resources

DocumentationAzure Storage BlogAzure Storage Documentation (Azure.com / Docs / Storage)REST and API Documentation (MSDN)

Client LibrariesAzure Storage NugetAzure Storage Client Library Source CodeSDK’s

Architectural Guidance (Azure.Com / Docs / Storage)

Azure Storage Performance and Scalability ChecklistMonitor, Diagnose and TroubleshootingManaging Concurrency in Azure Storage ApplicationsAzure Table Storage Design Guide (Coming Soon)

Azure Storage Resources

Continue the discussion… Storage Team Round Table5pm - 6:30pm in room 8.11

Thank you…

Come visit us in the Microsoft Solutions Experience (MSE)!Look for the Cloud and Datacenter Platform area TechExpo Hall 7

For more informationWindows Server Technical Previewhttp://technet.microsoft.com/library/dn765472.aspx

Windows Server

Microsoft Azure

Microsoft Azurehttp://azure.microsoft.com/en-us/

System Center

System Center Technical Previewhttp://technet.microsoft.com/en-us/library/hh546785.aspx

Azure Pack Azure Packhttp://www.microsoft.com/en-us/server-cloud/products/windows-azure-pack

AzureImplementing Microsoft Azure Infrastructure Solutions

Classroomtraining

Exams

+

(Coming soon)Microsoft Azure Fundamentals

Developing Microsoft Azure Solutions

MOC

10979

Implementing Microsoft Azure Infrastructure Solutions

Onlinetraining

(Coming soon)Architecting Microsoft Azure Solutions

(Coming soon)Architecting Microsoft Azure Solutions

Developing Microsoft Azure Solutions

(Coming soon)Microsoft Azure Fundamentals

http://bit.ly/Azure-Cert

http://bit.ly/Azure-MVA

http://bit.ly/Azure-Train

Get certified for 1/2 the price at TechEd Europe 2014!http://bit.ly/TechEd-CertDeal

2 5 5MOC

20532

MOC

20533

EXAM

532EXAM

533EXAM

534

MVA MVA

ResourcesLearning

Microsoft Certification & Training Resourceswww.microsoft.com/learning

TechNetResources for IT Professionals

http://microsoft.com/technet

Sessions on Demandhttp://channel9.msdn.com/Events/TechEd

Developer Network

http://developer.microsoft.com

Please Complete An Evaluation FormYour input is important!TechEd Schedule Builder CommNet station or PC

TechEd Mobile appPhone or Tablet

QR code

Evaluate this session

© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.