a complete guide to azure storage

A complete Guide to Windows Azure Storage

Himanshu SahuMindfire [email protected]

Windows Azure Storage

Windows Azure Storage (WAS) is a scalable cloud storagesystem that has been in production since November 2008. It is used inside Microsoft for applications such as social networking search, serving video, music and game content, managing medical records, and more.


Cloud Storage – Anywhere and anytime Access

Durable and higly Available

Massively Scalable – Easily Build “internet-scale” application

Pay for what you use

Exlopsed via REST API and easy to use client libraries in .NET, JAVA, Node.js, etc

Windows Azure StorageAbstraction

Blobs

Tables

Queue


Blob

Simple Interface to store and retrieve large amount of unstructured data like files, images, videos, etc

A single blob can be hundreds of gigabytes in size, and a single storage account can contain up to 200TB of blobs if it was created on June 8th, 2012, or later and storage accounts created prior to that date can contain up to 100TB of blobs.


Blob

Concept

Windows Azure StorageBlob

Concept

Storage Account – All access to Windows Azure Storage is done through a storage account.This is the highest level of the namespace for accessing blobsAn account can have many Blob Containers

Blob Container – A container provides a grouping of a set of blobs. The container name is scoped by the account.Sharing policies are set at the container level, where a container can be set to private or to be publicly accessible. The ability to list all of the blobs within the container is also provided.


Blob

Concept

Blob – Blobs are stored in and scoped by Blob Containers. Blobs can have meta data associated with them, which are <name, value> pairs, and they are up to 8KB in size per blob. The blob metadata can be set and retrieved separately from the blob data bits.

URL format- Blobs are addressable using the following URL format: http://<storageaccount>.blob.core.windows.net/<container>/<blob>

Windows Azure StorageBlob

Concept

<configuration> <appSettings> <add key="StorageConnectionString" value="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey]" /> </appSettings></configuration>


Blob

Common uses of Blob storage:

Serving images or documents directly to a browser

Storing files for distributed access

Streaming video and audio

Performing secure backup and disaster recovery

Storing data for analysis by an on-premises or Windows Azure-hosted service


Blob

Practical Scenarios where we use Blobs:

Enterprise document store, logs, etc.

Social sharing like photos, videos, blogs.

Backups – file, database, devices, computer, etc.


Table

NoSQL datastore which accepts authenticated calls from inside and outside the Windows Azure cloud.

Used for storing structured, non-relational data.


Table

Concept


Table

Concept

Storage Account – All access to Windows Azure Storage is done through a storage account.This is the highest level of the namespace for accessing tablesAn account can have many Tables

Table – contains a set of entities. Table names are scoped by the account. An application may create many tables within a storage account.


Table

Concept

Entity (Row) – Entities (an entity is analogous to a "row") are the basic data items stored in a table. An entity contains a set of properties. Each table has two properties, “PartitionKey and RowKey”, which form the unique key for the entity.

Combined size of all of the properties in an entity cannot exceed 1MB. This size includes the size of the property names as well as the size of the property values or their types.

Windows Azure StorageTable

Concept

Property (Column) – A property is a name-value pair.

This represents a single value in an entity. Property names are case sensitive.

An entity can hold up to 255 properties. Each entity also has 3 system properties that specify a partition key, a row key, and a timestamp.

Windows Azure StorageTable

Concept

PartitionKey – The first key property of every table. The system uses this key to automatically distribute and load balance the table’s entities over many servers.

RowKey – A second key property for the table. This is the unique ID of the entity within the partition it belongs to. The PartitionKey combined with the RowKey uniquely identifies an entity in a table.

Timestamp – Every entity has a version maintained by the system. This is used for optimistic concurrency.


Table

Common uses of Table storage:

Storing TBs of structured data capable of serving web scale applications

Storing datasets that don't require complex joins, foreign keys, or stored procedures and can be denormalized for fast access

Quickly querying data using a clustered index


Queue

Scalable, reliable and persistent messaging system that enables building work flow application or decouple components.


Queue

Concept


Queue

Concept

Storage Account – All access to Windows Azure Storage is done through a storage account.

This is the highest level of the namespace for accessing queues

An account can have many Queues


Queue

Concept

Queue – A queue contains many messages. The queue name is scoped by the account.

There is no limit on the number of messages stored in a queue (the only limit is the 200TB size limit on the storage account).A message is stored for at most a week. The system will garbage collect the messages that are more than a week old.Queues can have metadata associated with them. Metadata is in the form of <name, value> pairs, and they are up to 8KB in size per queue.


Queue

Concept

Messages – Messages are stored in queues. Each message can be up to 8KB in size. To store larger data, one can store the data in Azure Blob store or Azure Table store, and then store the blob/entity name in the message. Note that when you put a message into the store, the message data can be binary. But when you get the messages back from the store, the response is in XML format, and the message data is returned as base64 encoded.


Queue

Concept

Where Azure Queues differ from traditional queue implementations is that it is not a FIFO container. This means, the message will remain in the queue until explicitly deleted. If a message is read by one process, it will be marked as invisible to other processes for a variable time period, which defaults to 30 seconds, and can be no more than 2 hours; if the message hasn't been deleted by then, it will be returned to the queue and will be available for processing again. Because of this behavior, there is also no guarantee that messages will be in any particular order.


Queue

Concept

Messages – Messages are stored in queues. Each message can be up to 8KB in size. To store larger data, one can store the data in Azure Blob store or Azure Table store, and then store the blob/entity name in the message. Note that when you put a message into the store, the message data can be binary. But when you get the messages back from the store, the response is in XML format, and the message data is returned as base64 encoded.


Queue

Common uses of Queue storage:

Creating a backlog of work to process asynchronously

Inter-Role Communication

Process/Work Flow


Windows Azure Storage Scalability and Performance Targets

Resources

http://www.windowsazure.com/en-us/documentation/services/storage/

http://www.windowsazure.com/en-us/develop/net/how-to-guides/table-services/

http://www.windowsazure.com/en-us/develop/net/how-to-guides/blob-storage/

http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx

Questions?

Thank you!

a complete guide to azure storage

Engineering

storage accounts

windows azure cloud

single storage account

windows azurehosted

blob containers blob

single blob

blob metadata

scalable cloud storage