ms cloud day - building web applications with azure storage
TRANSCRIPT
KMohamed Faizal Lead Consultant NCS Pte Ltd
Microsoft Cloud Day
Building web applications with Azure Storage
Faizal has over a decade of experience in Information Technology with a focus on enabling portals Internet amp
Intranet application development
In this session you will learn storage capabilities of Windows Azure Blobs Tables and Queues Discover how to create
storage accounts upload and retrieve blobs and blob metadata create update and query tables and create a
simple service that uses a message queue for communication
CL201 Building Web Application with Azure Storage
KMohamed FaizalLead Consultant NCS (P) Ltd
27th April 2011
About Me ndash K Mohamed Faizal
15
Windows Azure StorageBlobs Tables Queues
Scalability ndash Best Practices amp TipsQampA
Agenda
In General
Web applicationsRelational database ( SQL Server)itrsquos very difficult to design scalable SQL Server at low cost
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
CL201 Building Web Application with Azure Storage
KMohamed FaizalLead Consultant NCS (P) Ltd
27th April 2011
About Me ndash K Mohamed Faizal
15
Windows Azure StorageBlobs Tables Queues
Scalability ndash Best Practices amp TipsQampA
Agenda
In General
Web applicationsRelational database ( SQL Server)itrsquos very difficult to design scalable SQL Server at low cost
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
About Me ndash K Mohamed Faizal
15
Windows Azure StorageBlobs Tables Queues
Scalability ndash Best Practices amp TipsQampA
Agenda
In General
Web applicationsRelational database ( SQL Server)itrsquos very difficult to design scalable SQL Server at low cost
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Windows Azure StorageBlobs Tables Queues
Scalability ndash Best Practices amp TipsQampA
Agenda
In General
Web applicationsRelational database ( SQL Server)itrsquos very difficult to design scalable SQL Server at low cost
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
In General
Web applicationsRelational database ( SQL Server)itrsquos very difficult to design scalable SQL Server at low cost
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Do you have enough space to store all the files you needHow do you add more storage capacityIf a disk crashes where does your data goIs the storage block load balancedWhat if you lose your connection to the block Is it redundantAt what point do you max out your disk in terms of reading and writingHow do you evenly distribute load across all disks
Common Consideration
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
SQL ServerNetwork shareDistributed File System (DFS)Network-attached storage (NAS)Direct-attached storage (DAS)Storage area network (SAN)
Storage Options
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
SQL ServerHigh-availability technology (such as clustering mirroring or replication) your database server is likely to be a single point of failure in the system
Network shareThis cheapo solution offers no redundancy and provides no ability to scale out
Distributed File System (DFS)Using replication ensures that there are no single points of failure in this solution and that the data is held on multiple machines
Storage Options
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Network-attached storage (NAS)NAS devices can range from being pretty cheap to very expensive depending on the levels of scalability performance and redundancy that you require from the device
Direct-attached storage (DAS)Storage area network (SAN)
SAN devices support replication and are highly scalable (they scale much higher than do DAS devices) fault tolerant high performing and incredibly expensive
Storage Options
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Which can provideCost effectiveScalable Durable and Highly Available
I Need Storage
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Cloud Storage System Provides Scalable Durable and Highly Available Storage SystemAbstractions
Blobs ndash Provides a simple interface for storing named files along with metadata for the file
Tables ndash Provides structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provides reliable storage and delivery of messages for an application
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
What is Windows Azure Storage
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
High Availability
Triplicatehellip
Your data is replicated 3 or 4 times in their data centre
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Windows Azure and SQL Azure Azure Storage SQL Azure
Vision Highly scalable highly available store in the Cloud
Scalable highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational No Yes ndash but with some limitations
Analogy File System RDBMS ndash as it is
Maximum Amount of data in a single ldquodatabaserdquo
100TB 50GB (up to Oct 2010)
Price per GB per month $ 015 $ 999
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Windows Azure Storage Account
User creates a globally unique storage account name
Can choose geo-location to host storage accountldquoUS Anywhererdquo ldquoUS North Centralrdquo ldquoUS South Centralrdquo
Can co-locate storage account with compute accountReceive a 256 bit secret key when creating account
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Storage Account Capacity at Commercial Availability
Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
Storage Account Capacity
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Account
Container Blobs
Table Entities
Queue Messages
Windows Azure Data Storage Concepts
httpsltaccountgtblobcorewindowsnetltcontainergt
httpsltaccountgttablecorewindowsnetlttablegt
httpsltaccountgtqueuecorewindowsnetltqueuegt
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
DEMOCreate Storage Account
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
What is a BLOBBinary Large OBject
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
SQL ServersChallenges with cost performance and backupYour database size may grow very big
File System StorageLoad balance Cost
Current Storage Solutions
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Block BlobPage Blob
Type of Blob
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Block blobsTargeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedEfficient continuation and retrySend multiple out of order blocks in parallel and
decide the block order during commitRandom range reads possible
Size limit 200GB per blob
Windows Azure Blobs Types
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Block 1 Block 5Block 3
Block 2
File has variable sized blocksUpload blocks in parallel using PutBlockRetry failed blocksCommit the blob using PutBlockList
Block blobs
Local file
Cloud blob
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Page BlobsTargeted at random write workloadsEach blob consists of an array of pagesSize limit 1TB per blob
PageEach page range write is committed on PUTPage is 512 bytes in sizeWrite boundary aligned at multiple of 512 byteRange reads possiblePages that do not have data are zeroed out
Windows Azure Blobs Types
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Write 5K bytes - PutPageClear 2K bytes starting at offset 1K ndash ClearPageOverwrite 2K bytes starting at 2K ndash PutPageTruncate blob to 3K - SetMaxBlobSize
Page blobs
0 5120
0 5120
0 5120
0 3072
1024 3072
2048 4096
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
DEMOBlob Storage
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Every blob request must be signed with the account ownerrsquos key
Share your filesThe container must be publicShared Access Signature (SAS) ndash share pre-authenticated URLs with users
SASUse container level access as it allows access to be easily revoked
Sharing Your Files
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Shared Access Signatures Services want to distribute access to blobs but do not want to distribute their secret keyCan create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobsValid time range
st=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
httpssallyblobcorewindowsnetimagespic1jpg st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
DEMOShared Access Signatures
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Create a point in time read-only copy of a blob
Every snapshot creates a new read only point in time copy
Charged only for unique blocks or pages ie reuse blocks or pages
For reuse use WritePages or PutBlock amp PutBlockList
Restore snapshots using copy blob
Cleanup your snapshots
What Are Snapshots
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
What does unique mean
AID=1
BBID=2
Base blob = alphabetstxt
Snapshot 1
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
1 Add 2 blocks 1 22 Commit the blob3 Take a snapshot4 Charged for
bull Base blobrsquos blocks 1 2
5 Add a new block Id=36 Commit the base blob7 Charged for
bull Base blobrsquos blocks 1 2 3
8 Snapshot to create the second snapshot
9 Charged for bull Base blobrsquos blocks 1 2
3
Snapshot 2
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
What does unique mean9 Update block id 1 to
now contain the same data ldquoArdquo
10 Commit the blob11 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
12 Take a snapshot13 Charged for
bull Base blobrsquos blocks 1 2 3
bull Snapshot 1rsquos blocks 1
Snapshot 3
AID=1
BBID=2
Base blob = alphabetstxt
CCCID=3
1 snapshot=2011-04-10T1926248690267Z
AID=1
BBID=2
2 snapshot=2011-05-10T1926248690267Z
AID=1
BBID=2
CCCID=3
3 snapshot=2011-05-10T1928248690267Z
BBID=2
CCCID=3
AID=1A
bullUploadFileUploadTextUploadFromStreamUploadByteArray overwrites all blocksbullCharged for entire snapshot and base blob
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
ScenarioFrequently accessed blobsAccessed from around the world
DesireSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery
18 locations globally (US Europe Asia Australia and South America) and growingBlob service URL vs CDN URL
Windows Azure Blob URL httpsallyblobcorewindowsnetWindows Azure CDN URL httpltguidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom
Windows Azure Content Delivery Network
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
To Enable CDN Register for CDN via Dev Portal Set container images to public
Windows Azure Content Delivery Network
Windows Azure Blob Service
pic1jpg
Content Delivery Network
Edge Location
Edge Location Edge Location
GEThttpguid01vomsecndnetimagespic1jpg
httpsallyblobcorewindowsnetimagespic1jpg
httpsallyblobcorewindowsnet
httpguid01vomsecndnet
pic1jpg
Not Accessible
pic1jpg
TTL
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Provides a durable NTFS volume for Windows Azure applications
Use existing NTFS APIsEasy migration path to the cloudDurability and survival of data on application failover or hardware failure
All flushed and un-buffered writes to drive are made durable
A Windows Azure Drive is a Page BlobMounts Page Blob as an NTFS driveMounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drivesDrives can be up to 1 TB
Windows Azure Drive
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
WINDOWS AZURE TABLES
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Provides Structured StorageMassively Scalable and Durable Tables
Billions of entities (rows) and TBs of data
A storage account can contain many tables
No limit on number of entities (aka rows) in each table
Provides flexible schema
Familiar and Easy to use API
WCF Data Services - NET classes and LINQ
REST (OData Protocol) ndash with any platform or language
Windows Azure Tables
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Windows Azure TablesIs not relationalCan Not-
Create foreign key relationships between tablesPerform server side joins between tablesCreate custom indexes on the tablesNo server side Count() for example
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
TableA storage account can create many tablesTable name is scoped by accountSet of entities (ie rows)
EntitySet of properties (columns)Required properties
PartitionKey RowKey and Timestamp
Table Data Model
38
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
PartitionKey amp RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
Required Entity Properties
39
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
All entities must have the following propertiesTimestampPartitionKeyRowKey
PartitionKey + RowKey = ldquoprimary keyrdquo
Required Properties
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Not an RDBMS More on table modeling in Storage Strategies sessionTable
Create Query DeleteTables can have metadata
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Multiple CUD Operations in a single atomic transaction
Table Details
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Entity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey amp RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic Concurrency Exposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a ltname typed valuegt pairNo schema stored for a tableProperties can be the standard NET types String binary bool DateTime GUID int int64 and double
Entity Properties
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Fav Sport
Canoeing
No Fixed Schema
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
First Last Birthdate
Kim Akers 221981
Nancy Anderson 3151965
Mark Hassall May 1 1976
Querying
$filter=Last eq lsquoAkersrsquo
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple InsertUpdateDelete in same partition in a single transaction
Table ScalabilityTarget throughput ndash 500 tpspartition several thousand tpsaccount
Windows Azure monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
Purpose of the PartitionKey
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ModelYear
Bikes Super Duper Cycle hellip 2009
Bikes Quick Cycle 200 Deluxe hellip 2007
hellip hellip hellip hellip
Canoes Whitewater hellip 2009
Canoes Flatwater hellip 2006
Rafts 14ft Super Tourer hellip 1999
hellip hellip hellip hellip
Skis Fabrikam Back Trackers hellip 2009
hellip hellip hellip hellip
Tents Super Palace hellip 2008
Partitions and Partition Ranges
Server BTable = Products[Canoes - MaxKey)
Server ATable = Products[MinKey - Canoes)
Server ATable = Products
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
DEMOTable Storage
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
WINDOWS AZURE QUEUES
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Queue are highly scalable available and provide reliable message delivery
Simple asynchronous work dispatch
A storage account can create any number of queues
8K message size limit and default expiry of 7 days
Programming semantics ensure that a message can be processed at least once
Get message to make the message invisible
Delete message to remove the message
Access is provided via REST
Windows Azure Queues
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
An account can create many queuesQueue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size lt= 8 KBTo store larger data store data in blobentity storage and the blobentity name in the messageMessage now has dequeue count
Account Queues and Messages
52
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
21
11
C1
C2
Removing Poison Messages
11
21
340
Producers Consumers
P2
P1
30
2 GetMessage(Q 30 s) msg 2
1 GetMessage(Q 30 s) msg 1
11
21
53
10
20
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
11
21
2 GetMessage(Q 30 s) msg 23 C2 consumed msg 24 DeleteMessage(Q msg 2)7 GetMessage(Q 30 s) msg 1
1 GetMessage(Q 30 s) msg 15 C1 crashed
11
21
6 msg1 visible 30 s after Dequeue30
54
12
11
12
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
C1
C2
Removing Poison Messages
340
Producers Consumers
P2
P1
12
2 Dequeue(Q 30 sec) msg 23 C2 consumed msg 24 Delete(Q msg 2)7 Dequeue(Q 30 sec) msg 18 C2 crashed
1 Dequeue(Q 30 sec) msg 15 C1 crashed10 C1 restarted11 Dequeue(Q 30 sec) msg 112 DequeueCount gt 213 Delete (Q msg1)
12
6 msg1 visible 30s after Dequeue9 msg1 visible 30s after Dequeue
30
55
13
12
13
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
DEMOQueue Storage
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
SCALABILITY ndash BEST PRACTICES amp TIPS
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Know The Scalability Targets
Single QueueTable Partitionbull Up to 500 transactions (entities or messages) per second
Single Blob Partitionbull Throughput up to 60 MBs
Scale Above the limitsbull Partition between multiple storage accounts and partitionsbull When limit is hit app may see lsquo503 server busyrsquo applications
should implement exponential back-off
Storage Accountbull SLA ndash 999 Availabilitybull Capacity ndash Up to 100 TBsbull Transactions ndash Up to 5000 entities per secondbull Bandwidth ndash Up to 3 gigabits per second
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Master System
Master System
Automatic Load Balancing - Assignment
Distributed File System
BE1 BE2 BE3 BE4
FE FE FE
VIP
Master System
- Partition
- Server Load
Legend
Offload PartitionsReassign Partitions
bull Time between offload to reload is in the order of secondsbull Time to decide to load balance is in the order of minutesbull Goal is to only reassign a partition only if the system has to
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Partition Keys In Each Abstraction
bull Every blob and its snapshots are in a single partition
Blobs ndash Container name + Blob name
Container Name Blob Name Snapshot
image annarborbighousejpg
image annarborbighousejpg 2009-12-03T1526194466877Z
image annarbordenardjpg
backup annarborbighousejpg
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Partition Keys In Each Abstraction
bull Entities with same PartitionKey value are served from same partition
Entities ndash TableName + PartitionKey
Table Name PartitionKey RowKey Zipcode City
CustomerOrder Alaska Tina Fey 99501 Anchorage
CustomerOrder Alaska Sarah Palin 99501 Anchorage
CustomerOrder Washington Bill Johnson 98053 Redmond
Customers Washington Bill Johnson 98053 Redmond
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Partition Keys In Each Abstraction
bull All messages for a single queue belong to the same partition
Messages ndash Queue Name
Queue Message
jobs Message1
jobs Message2
workflow Message1
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Table Inserts Single Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Inserts (SP)
Successful Inserts (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Table GetSingle Partition (SP) vs Multiple Partitions (MP)
1 2 5 10 160
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
Successful Gets (SP)
Successful Gets (MP)
Extra Large VMs (15 Threads per VM)
Entiti
es
Sec
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
Avoid ldquoAppendPrepend Onlyrdquo Patterns For High ScaleTimestamp as Partition Keyhellip looks like an obvious choiceIt is not a single partition as time moves forward AppendPrepend onlyRequests to single partition rangeLoad balancing does not helpServer may throttle
Are Unique Partition Key Values Sufficient To Scale
Partition Key(Timestamp)
Other properties
2010-10-15 020001 hellip
2010-10-15 020011 hellip
100000 more rows hellip
2010-10-17 115958 hellip
2010-10-17 115958 hellip
80000 more rows hellip
2010-10-17 123000 hellip
2010-10-17 123001 hellipApplicationsClient Request
Server A
Request
2010-10-17 123001 hellip
Request
2010-10-17 123002 hellip
Request
2010-10-17 123003 hellip
Server B
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-
67
Questions amp Answers
- Slide 1
- CL201 Building Web Application with Azure Storage
- About Me ndash K Mohamed Faizal
- Agenda
- In General
- Common Consideration
- Storage Options
- Storage Options (2)
- Storage Options (3)
- I Need Storage
- What is Windows Azure Storage
- Triplicatehellip
- Windows Azure and SQL Azure
- Windows Azure Storage Account
- Storage Account Capacity
- Windows Azure Data Storage Concepts
- Demo
- What is a BLOB Binary Large OBject
- Current Storage Solutions
- Type of Blob
- Windows Azure Blobs Types
- Block blobs
- Windows Azure Blobs Types (2)
- Page blobs
- Demo (2)
- Sharing Your Files
- Shared Access Signatures
- Demo (3)
- What Are Snapshots
- What does unique mean
- What does unique mean (2)
- Windows Azure Content Delivery Network
- Windows Azure Content Delivery Network
- Windows Azure Drive
- Windows Azure Tables
- Windows Azure Tables (2)
- Windows Azure Tables (3)
- Table Data Model
- Required Entity Properties
- Required Properties
- Table Details
- Entity Properties
- No Fixed Schema
- Querying
- Purpose of the PartitionKey
- Partitions and Partition Ranges
- Demo (4)
- Windows Azure Queues
- Slide 49
- Slide 50
- Windows Azure Queues (2)
- Account Queues and Messages
- Removing Poison Messages
- Removing Poison Messages (2)
- Removing Poison Messages (3)
- Demo (5)
- Scalability ndash Best Practices amp Tips
- Know The Scalability Targets
- Automatic Load Balancing - Assignment
- Partition Keys In Each Abstraction
- Partition Keys In Each Abstraction (2)
- Partition Keys In Each Abstraction (3)
- Table Inserts Single Partition (SP) vs Multiple Partitions (
- Table Get Single Partition (SP) vs Multiple Partitions (MP)
- Are Unique Partition Key Values Sufficient To Scale
- Slide 67
-