WelcomeCloud Architectural Patterns (using Microsoft Azure tools)

Pushkar Chivate

AgendaQuick Look: Microsoft Azure as a platformOverview of patterns for cloud applicationsSpecifically Data-Management, Resiliency and Messaging patternsDemo Azure Tables, DocumentDB, Azure Service bus (Queues and Pub-Sub architecture)

What is the Cloud?IT CapacityTimeLoad ForecastAllocated IT-capacityUnder capacityOver capacityActual LoadHow the Cloud Helps?IT CapacityTimeLoad ForecastActual LoadCapacity on DemandCloud ModelsIaaS Infrastructure as a service(e.g. Azure VMs)

PaaS Platform as a service(e.g. Azure Storage, Service Bus etc.)

SaaS Software as a Service(e.g. Office 365)

Microsoft Azure is a cloud platformOn-DemandSelf-servicePay per useHighly automatedDesigned for FailureDesigned for ScaleMicrosoft AzureDesigned for Failure

Network load balancerVM1VM2VM3VM4VM5VM6VM7VM8Fault Domain 1Fault Domain 2Fault Domain 3UD1UD2Microsoft AzureDesigned for ScaleScale out

Network load balancerVM1VM2VM3VM4Azure VMs, Websites& Rethinking AppsA look at various features Azure offersArchitectural PatternAn architectural pattern is a general, reusable solution to a commonly occurring problem within a given context.Cloud PatternsData-Management PatternsIndex Table & Sharding PatternCreate indexes over the fields in data stores that are frequently referenced by query criteria. This pattern can improve query performance by allowing applications to more quickly retrieve data from a data store. Divide a data store into a set of horizontal partitions shardsMaterialized View PatternGenerate prepopulated views over the data in one or more data stores when the data is formatted in a way that does not favor the required query operations. This pattern can help to support efficient querying and data extraction, and improve performance

Command and Query Responsibility Segregation (CQRS) PatternSegregate operations that read data from operations that update data by using separate interfaces.

Cloud PatternsMessaging PatternsPipes and Filter PatternDecompose a task that performs complex processing into a series of discrete elements that can be reused. Priority Queue PatternPrioritize requests sent to services so that requests with a higher priority are received and processed more quickly than those of a lower priority .Queue-based Load Leveling PatternUse a queue that acts as a buffer between a task and a service that it invokes in order to smooth intermittent heavy loads that may otherwise cause the service to fail or the task to timeout.

Data StorageVarious data storage techniques

SQL AzureAzure Table

Blob storage

Document DBMaterialized View PatternThis pattern can be used to support efficient querying and data extraction, and improve application performance

What is a materialized view?

Materialized view is a database object that contains results of a query

Materialized ViewOrderIDAccount1A2BItemIDNameStock100Shirts100101Pants125OrderIDItemIDQty Ordered110021101321002Materialized ViewDatabase tablesItemIDNameTotal Qty Ordered100Shirts4101Pants3Denormalized data

Data refreshes periodically Materialized Views Vs Regular ViewsMaterialized ViewStored as a physical database objectIndexes on the base tables are not used, can create indexes on materialized viewFaster performanceNeeds to be updatedRegular ViewStored as a query against the base objectIndexes on the base tables are usedNo performance improvementNeed not be updatedMaterialized viewWhen underlying data is complex and difficult to query directlySimplify queries - expose data in a way that doesnt require knowledge of underlying data structureProvide access to specific subset of data (security reasons)Bridging the disjoint when using different data stores When to use materialized views?Materialized viewSnapshotsDatabase 1Table 1Materialized view 1Database 2Table 1Materialized view 2SnapshotMaterialized viewDatabases supporting Materialized viewsOracleCalled Indexed ViewsIBM DB2SQL ServerMySQLNo Native supportCan be implemented using triggersCalled Materialized Query TableCalled Materialized ViewMaterialized viewWhen not to use materialized views? The source data is simple and easy to query.

The source data changes very quickly, or can be accessed without using a view. The processing overhead of creating views may be avoidable in these cases.

Consistency is a high priority. The views may not always be fully consistent with the original dataAzure TablesWhat are Azure Tables?

Non relational entities (not RDBMS).

Tables are nothing but collection of entitiesEntities are comprised of propertiesProperties are name value pairs

Azure TablesAzure Storage AccountEmployees (Table)EntityPropertyEntityPropertyPropertyPropertyProperty Azure Table EntitiesAzure Storage AccountEmployees (Table)EntityProperty

Property

EntitiesPartition KeyRow KeyTimestamp other propertiesEach entity can have different number and type of properties

Partition Key + Row Key = Unique Identifier(No support for secondary index in Azure Table)Property

Azure Tables PartitionsEmployees (Table)Certifications (Table)(Entity)

EmployeeName=CertName=BirthDate=CertNumber=

EmployeeName=CertName=BirthDate=CertNumber=FavoriteTeam=(Entity)

CertName=EmployeeName=CertNumber=BirthDate=

CertName=EmployeeName=CertNumber=BirthDate=PK = EmployeeName

RK = CertNamePK = CertName

RK = EmployeeNameHow do you Partition the data?EmployeeName or CertName?Azure Tables PartitionsEmployees (Table)Certifications (Table)(PK) EmployeeName(E1), (RK) CertName (C1)(PK) EmployeeName(E1), (RK) CertName (C2)

(PK) EmployeeName(E2), (RK) CertName (C1)(PK) EmployeeName(E2), (RK) CertName (C3)

(PK) EmployeeName(E3), (RK) CertName (C2)(PK) EmployeeName(E3), (RK) CertName (C3)

How do you Partition the data?EmployeeName or CertName?(PK) CertName (C1), (RK) EmployeeName(E1)(PK) CertName (C1), (RK) EmployeeName(E2)

(PK) CertName (C2), (RK) EmployeeName(E1)(PK) CertName (C2), (RK) EmployeeName(E3)

(PK) CertName (C3), (RK) EmployeeName(E2)(PK) CertName (C3), (RK) EmployeeName(E3)

ORAzure TablesEmployeeNameBirthDateFavoriteTeamDavid Anderson1/1/1970Nancy Wilson4/15/1965Atlanta FalconsJohn DoeApril 1, 1989No Fixed Schema for entitiesEmployee tableAzure TableDemoAzure TablesSummary

No nice relational schema model that we get in RDBMS

Easy to work withBuilt to scaleGreat pricing

DocumentDBAzure Document DB - relatively new service

Its fully managed Document database as a service

It stores the data in JSON

The Azure Tables are extremely scalable and cheap, but if you start querying on any other attributes then you start running into problems.

Its massively scalable but fully query-able by all parts of JSON tree.

Capacity units are 10GB in size and predictable reads and writes / second.

DocumentDB

DocumentDBDemoAzure Service BusSecure messaging capabilitiesEnable loosely coupled solutionsAchieve Publish-Subscribe scenarios

Tightly Coupled

Store Front End

Store Back EndOrder processingShippingLoosely Coupled

Store Front End

Store Back EndOrder processingShippingOrder QueueLoosely Coupled

Store Front End

Store Back EndOrder processingOrder Queue

ShippingService Bus QueueDemoPublish - Subscribe

Store Front End

Store Back EndOrder processingOrder Queue

Order PrintersSubscription 1Subscription 2ShippingOrder PapersService Bus AMQPService Bus supports the Advanced Message Queueing Protocol (AMQP) 1.0.

AMQP enables you to build cross-platform, hybrid applications using an open standard protocol

Reliable: The AMQP 1.0 protocol allows messages to be exchanged with a range of reliability guarantees, from fire-and-forget to reliable, exactly-once acknowledged delivery.

Using AMQP from .Net: Endpoint=sb://[namespace].servicebus.windows.net;SharedSecretIssuer=[issuer name];SharedSecretValue=[issuer key];TransportType=Amqp

Using AMQP from Java: # servicebus.properties - sample JNDI configuration # Register a ConnectionFactory in JNDI using the form: # connectionfactory.[jndi_name] = [ConnectionURL] connectionfactory.SBCF = amqps://[username]:[password]@[namespace].servicebus.windows.netService Bus AMQP

Cloud PatternsResiliency PatternsCircuit Breaker PatternHandle faults that may take a variable amount of time to rectify when connecting to a remote service or resource. This pattern can improve the stability and resiliency of an application. Retry PatternEnable an application to handle temporary failures when connecting to a service or network resource by transparently retrying the operation in the expectation that the failure is transient. This pattern can improve the stability of the application.

Compensating Transaction PatternUndo the work performed by a series of steps, which together define an eventually consistent operation, if one or more of the operations fails. Operations that follow the eventual consistency model are commonly found in cloud-hosted applications that implement complex business processes and workflows.

SummaryWe Looked at

- Azure as Infrastructure (VMs)

- Azure as Platform -Websites- Storage - Azure Tables, DocumentDB- Messaging Service Bus, Queues, Topics, AMQP- Azure CDN SummaryNetwork load balancerVM1VM2VM3VM4SQL DatabaseAzure TablesDocumentDBService Bus QueueSubscription 1Subscription 2Worker Role 1Worker Role 2CacheReferencesMSDN (cloud architectural patterns)

azure.microsoft.com (Learning videos and documentation)

some images from www