EDITOR’S NOTE PRIVATE CLOUD ENVIRONMENT OFFERS COST SAVINGS, SCALABILITY

FOUR LESSONS ENTERPRISES CAN TAKE FROM A CLOUD STORAGE PROVIDER

DESIGN A PRIVATE CLOUD STORAGE ARCHITECTURE WITH OBJECT STORAGE

The Nuts and Bolts of Private Cloud StorageA private cloud is a more cost-effective alternative to traditional storage, but it requires administrators to learn about new storage architectures and data protection techniques.

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

EDITOR’SNOTE

Design a Private Cloud for the Ages

There’s so much talk about public cloud providers—their rates, offerings and the com-petition between them—that private clouds can sometimes seem pushed to the back-ground. But imagine the benefits to be had from designing your own self-service option, where users can set up volumes of cloud stor-age through a dashboard without having to take time and resources away from the IT staff.

Without a doubt, IT pros who have success-fully implemented private cloud storage archi-tectures know there are big benefits in terms of security and redundancy. By the time a pri-vate cloud infrastructure is established, your shop can expect a simplified infrastructure and improved enterprise-side management practices. Then there’s the bottom line. Private clouds can help organizations to trim budgets due to their use of commoditized hardware and their ability—when configured correctly—to offer on-demand scalability.

Among other things, IT pros need to provide private clouds that meet first-rate standards of elasticity, multi-tenancy and geographic aware-ness. In all those ways, private clouds can offer all the benefits of public cloud offerings.

How do you get there? You depend on good, solid information from people who have been there before. Some of the chief challenges data storage administrators can expect to encoun-ter when developing and implementing a pri-vate cloud storage architecture are deciding how much and which type of data to send to a private cloud. How will it be migrated and recalled? Does object storage factor into the equation, and do you need it?

This Drill Down will guide you through the complicated, but worthwhile, process of build-ing and using a private cloud architecture. n

Ellen O’BrienAssociate Editorial Director, SearchCloudStorage

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

BEST STORAGE FOR PRIVATE

CLOUDS

Private Cloud Environment Offers Cost Savings, Scalability

The decision to move to a private cloud storage infrastructure requires careful planning, as the transformation often involves a major IT project and can bring significant change to the way IT resources are delivered.

Beth Cohen, a senior cloud architect at Boston-based Cloud Technology Partners (cloudTP) Inc., has worked with major corpo-rations to plan and deploy private cloud infra-structures in which the company’s end users gain self-service access to IT resources such as servers and storage.

Senior writer Carol Sliwa spoke with Cohen about the nuts and bolts of creating a private cloud storage environment.

Cohen explained at length why object storage may present organizations with the best option for a private cloud storage infrastructure, and offered additional tips on the key decision points to consider when building a private cloud.

Under what circumstances should a company

think about deploying private cloud storage

vs. a traditional storage infrastructure?

There are a number of factors that companies should be thinking about. Of course, the para-mount factor is always to save money, and there are a number of ways the private cloud offers that opportunity. One is that you’re deploying on commodity hardware rather than the spe-cialized hardware that most traditional stor-age systems run on. It also allows for a greater degree of horizontal scalability. It’s architected to scale up to petabytes of data easily and transparently, so you don’t have to worry about running out of capacity. And of course, the major consideration is self-service. That’s the real buzzword for any kind of cloud, which is that you don’t have to be deploying those stor-age modules. You can just have your users go to a dashboard and sign up for it. They can set up their own volumes without worrying about

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

BEST STORAGE FOR PRIVATE

CLOUDS

going to an administrator.

What types of storage work best

with a private cloud?

The three types of storage that you typically see are traditional block storage and object store and, of course, you can have filer storage as well. What I typically see is that the images are managed under block storage or filer stor-age, and the object store is used by the applica-tions. So, for example, a retailer might store the catalog information in the object store. But the [virtual machine (VM)] images might be stored in the block storage.

What are the advantages and disadvantages

of object storage for a private cloud?

The advantages are that the storage itself is extremely cheap so you can store the typi-cal files in great quantities without worrying about expensive hardware. But there are dis-advantages. I don’t recommend object storage

for high-speed transactional databases where every little nanosecond counts. That’s not a good fit for a private cloud object store.

How essential is it for an IT shop to build a private

cloud compute infrastructure in connection with

creating a private cloud storage infrastructure?

You can build them together, although they are actually two separate concepts. I would say that, in many cases, they are built together, but I have also seen situations where they are completely separate. Typically, if you’re build-ing a compute farm, of course you do need some storage to manage the [VM] images and the instances, and typically that is cloud storage, and there are a number of different options that you have. But you can certainly build private cloud storage independent of the computes.

Do you think it’s essential to have object storage

for a private cloud?

“ I don’t recommend object storage for high-speed transactional databases where every nanosecond counts.” —BETH COHEN, senior cloud architect at cloudTP

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

BEST STORAGE FOR PRIVATE

CLOUDS

No, it’s not essential if you’re only using the private cloud for compute. But, in many ways, I think private cloud storage that’s being used for application, for data—it’s best if you use object storage. It’s far more cost efficient than block storage.

What’s the most important tip you would offer to

someone considering private cloud storage?

You really need to understand how you’re

planning to use the storage. There’s a huge dif-ference between storage that’s going to be used for managing the virtual machine instances as compared to the storage that’s going to be used for the data. The characteristics of the data are also going to have a major influence on the architecture of the storage. If it’s going to be a large number of small files, it will have one characteristic. Or, if it’s streaming media, it will have other characteristics. —Carol Sliwa

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

LESSONS FROM THE

CLOUD

Four Lessons Enterprises Can Take From a Cloud Storage Provider

Enterprises are keeping their eyes on cloud providers to get a glimpse of how they might solve some of their more vexing data center challenges. Many enterprises are being inundated with a tsunami of unstructured data that grows each day and requires longer reten-tion periods. Since cloud providers have already experienced the first waves, there are four les-sons enterprises can glean from their struggles and use to help them build their own private cloud infrastructures.

LESSON 1: ARCHITECTURE

A cloud storage provider has to cost-effectively provide storage to its subscribers, and an enterprise needs to do the same for its users. There are two keys to this architecture:

■n It is built on commodity hardware and the infrastructure often leverages internal

server-class storage. These servers and their storage are then clustered so that the capacity of each can be aggregated into a single storage pool. Expansion of capacity is done by adding another server to the cluster.

■n These storage clusters tend to have object-based file systems running on them instead of a traditional block or file system. The object nature of these designs is important as it provides many of the features enterprises are looking for when they try to solve their own unstructured data problems.

LESSON 2: UNLIMITED FILE COUNT

Most data centers today will not reach the file count limits of an enterprise network-attached storage system, but they may exceed the capabilities of a Linux NFS or Windows SMB server, which are more likely to be used

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

LESSONS FROM THE

CLOUD

for cost-effective storage of unstructured data. Similar to cloud providers, the file count concern will grow worse within enterprises as more machines and sensor equipment data needs to be captured and stored. The Inter-net of Things is creating trillions of files and objects, and those objects need to be managed and stored. An object-based storage system is designed to do just that.

LESSON 3: DATA PROTECTION

Another feature of object-based storage that should capture the attention of IT planners is the system’s ability to significantly enhance data resiliency. An object-based system under-stands data at the object (think file) level. That means if there is disk failure in the storage cluster, only the objects stored on the failed drive or node need to be restored. Additionally, recovery of that data can come from multiple nodes. This means a much faster rebuild time

when compared to traditional RAID 5 or RAID 6 data protection schemes, which are typi-cally composed of high-capacity hard drives that further contribute to the time it takes to rebuild a failed drive.

LESSON 4: INTELLIGENT DATA PLACEMENT

Cloud providers also leverage object storage to intelligently place data throughout their unstructured storage infrastructure. There are multiple implementations of this feature, but the most common is to ensure that data is placed as close to the user as possible. For example, if a cloud subscriber has four loca-tions and one location suddenly starts using a particular data set, that information can be transferred to increase performance, while an alternate copy is maintained in another loca-tion. This allows the active location to experi-ence local access performance to the data while maintaining disaster recovery (DR) status.

The file count concern will grow worse within enterprises as more machines and sensor equipment data needs to be captured and stored.

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

LESSONS FROM THE

CLOUD

Intelligent data placement can also be used within one data center. Using similar logic, an object storage system can place data on a faster media type, such as solid-state drives, when it is being actively accessed and then move it to hard disk storage when it is not.

Intelligent data placement can be leveraged for data protection and DR purposes. For example, it can make sure a set quantity of each object is stored in a user-defined number of data centers before it is considered protected. Even within a data center, it can ensure local objects are protected locally and that redundant copies are not on the same node within the

storage cluster, or even in the same rack row.As unstructured data and long-term reten-

tion continues to increase, enterprises will be faced with the same storage challenges cloud providers have had to deal with. They will need to drive down the cost to store this informa-tion, which is often achieved through the use of commodity hardware. This requires an infrastructure that can intelligently respond to drive failures and the demands of a distrib-uted workforce. A private cloud storage solu-tion provides many of these needs and can add value to almost any sized data center, not just enterprises. —George Crump

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

PRIVATE CLOUDS AND

OBJECT STORAGE Design a Private Cloud Storage

Architecture With Object Storage

The fact that data is growing isn’t a sur-prise to any IT professional. The greater chal-lenge is the inability to purge old information while the pace of growth continues. Add to that a new reality: Data generation and collection is no longer confined to just users. Devices such as smartphones and cameras are collecting data faster than humans can. As a result, it’s become widely apparent that traditional network-attached storage-based systems are unable to scale to meet these demands.

Many data centers are now looking to build their own private cloud storage architecture based on object storage. This article provides step-by-step guidance for designing an object-based, private cloud storage architecture.

A COMMON ARCHITECTURE

AS A STARTING POINT

Object storage systems assign each file an

object ID. To access an object, you provide the system with the ID and the system will retrieve it. Object storage systems are a simple flat architecture, as opposed to a more traditional POSIX file system in which data is organized in a hierarchical folder structure.

Most object storage systems start with a similar architecture, regardless of the type of data to be stored. From a hardware perspective, they tend to be nodal in nature. This means the object store is made up of a group of serv-ers, called nodes, with internal storage that is aggregated and presented as a common pool to the attaching applications. It’s the responsi-bility of this software to perform the aggrega-tion, manage the object database and maintain reliability.

It’s important to understand how the object storage system will store data. Will it repli-cate or use erasure coding? In both cases, each object is sent to a node for encoding. In most

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

PRIVATE CLOUDS AND

OBJECT STORAGE

systems, this is done on a round-robin basis so no single node becomes overwhelmed.

In the replication use case, the object is cop-ied to x number of other nodes, with varying levels of sophistication on where those other nodes might be physically located. Erasure cod-ing segments the object into parts and then distributes those parts across a large number of discrete nodes. Think of replication as simi-lar to mirroring, in which a complete copy of the object is made a number of user- or data-defined times. The system always ensures there are x number of copies available in the object store. Think of erasure coding as RAID, where an object is segmented, then a parity segment is created to rebuild the data in case a node fails.

When you talk with vendors about object storage, you’ll likely be offered two distinct choices: a vendor-specific, turnkey solution or a vendor-agnostic software solution. The former may cost more but require less time to get up and running. However, it requires the purchase of new, dedicated hardware. The software solution may be less expensive, espe-cially because some current offerings enable

you to use an existing array or storage inter-nal to the server. But these offerings require time to choose and implement because you have to research the various hardware compo-nents within your design and then integrate them into your infrastructure. There is also an increased burden on your IT shop to support a hardware-agnostic software offering.

The reality is that neither of these solutions is always better than the other; they’re just dif-ferent. The choice is largely dependent on the time and expertise your organization has avail-able to dedicate to the project.

TIME TO UNDERSTAND THE DATA

Once you understand the architecture, the next step in successfully designing a private cloud storage infrastructure is to understand the data that will live on it. Most organizations will have one of two data types: a type that supports bil-lions of very small files or a type that supports millions of very large files. In most cases, the billions of small files require random access to a large quantity of those small files. A common example is a large analytics environment where

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

PRIVATE CLOUDS AND

OBJECT STORAGE

thousands of data points are selected from the billions of files stored so that better decisions can be made. In these environments, random I/O performance is very important, which makes it a key requirement of the selected sys-tem. In many cases, solid-state drives (SSDs) are required to store the metadata needed to locate files quickly.

The other use case involves bulk data move-ment, where very large chunks of data are accessed in a sequential fashion. An example might be a media content distributor that needs to stream audio or video to a large num-ber of users. In this instance, random I/O isn’t nearly as important as bandwidth and through-put. These systems tend to have a very high number of nodes in their storage cluster and data is dispersed across all of them. In most cases, hard drive performance, as long as high quantities of drives are available, is perfectly acceptable.

In most cases, data centers don’t have a

mixture of both data types, at least to the extent that both data types are absolutely criti-cal to the business. For the rare business that requires both data types, a perfectly mixed platform doesn’t yet exist on the market. While the software may be the same, the archi-tecture layout needs to vary between these data types.

NATIVE ACCESS TO STORAGE

The next step is to understand how data is migrated to the object storage system. There are two common ways to accomplish this. First, almost all object storage systems are accessed via a Representational State Trans-fer (REST) application programming interface. This capability allows applications to directly access storage via simple GET and PUT com-mands. No longer does an application have to go through a gateway or file system or, worse, embed SCSI commands.

A logical path for most companies is to start with a gateway approach so the application can begin using the object store immediately.

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

PRIVATE CLOUDS AND

OBJECT STORAGE

Native access to storage via the application is an ideal capability. The application knows the state of data better than an external software package. In addition, the application probably understands value and can better predict when data would be needed again. The downside to application integration is that while simple, it requires access to the actual application source code and not all systems have ready access to this.

When the application source code is inacces-sible or there’s no time to modify the applica-tion, a gateway approach may be best. Many object storage systems have the ability to pres-ent a CIFS, NFS and even SCSI (Fibre Channel or iSCSI) translation layer so applications can use object storage without alteration. While this gateway approach doesn’t provide the fine-grained customization of data placement, it does provide quick access to the other attri-butes of object storage.

A logical path for most companies is to start with a gateway approach so the application can begin using the object store immediately. Then, over time, capabilities can be natively added to the application. Gateways are available from

many object storage providers, and there are a number of third-party add-ons. My suggestion would be to make separate decisions regarding your choice of object storage software and the gateway system that fits your specific needs.

FILE SIZE, INGEST RATES ARE KEY

Regardless of how the data will be sent to the object store, it’s important to understand the volume of data that will be sent to the store. This is partly affected by the size of files described above, as well as the amount of data that needs to be ingested over a given period of time. If files are relatively small and are sent to the object store on a modest basis, then the ingest rate is inconsequential. If thousands or millions of small files are sent to the object store from a variety of sources, it can become a challenge. And if a few very large files are ingested, and the speed that the ingestion occurs is important, that can be also a factor.

Thousands of small files can best be handled by a large node-count system with plenty of hard disk drives. The system, as mentioned above, may also benefit from being able to store

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

PRIVATE CLOUDS AND

OBJECT STORAGE

metadata information on SSDs. Not all object storage systems have the ability to store meta-data separately from the actual data; if this is a requirement for your specific use case, you should verify that the vendor provides it.

In the case of ingesting very large files, a high drive-count situation may not help since the data is typically received by a single node before it’s distributed across the entire object store. In other words, the bandwidth per node becomes an issue. In these situations, you may be better served by having a very fast set of hard drives or even SSDs on a few targeted ingestion nodes.

TOTAL RECALL

How data is recalled also impacts the design. With replication, the least busy node is

responsible for sending all the data, so you’re limited to the bandwidth of a single node. With erasure coding, all the nodes that have a segment of the object will send data to the requesting application. For large bulk transfers of a low number of files, erasure coding is typi-cally best. For the transfer of a high number of small files, both methods deliver about the same performance.

The framework offered here for designing a private cloud should be shared with the vari-ous vendors offering an object storage solution to you. Look for systems that are designed to handle your type of data, your data protec-tion needs, and the rate your data needs to be ingested and recalled. Solutions will vary along these lines, but there will be a few that spe-cialize in your specific requirements.

—George Crump

HOME

EDITOR’S NOTE

PRIVATE CLOUD

ENVIRONMENT OFFERS

COST SAVINGS,

SCALABILITY

FOUR LESSONS

ENTERPRISES CAN TAKE

FROM A CLOUD STORAGE

PROVIDER

DESIGN A PRIVATE

CLOUD STORAGE

ARCHITECTURE WITH

OBJECT STORAGE

THE NUTS AND BOLTS OF PRIVATE CLOUD STORAGE

ABOUT THE

AUTHORS

CAROL SLIWA is a senior writer in TechTarget’s Storage Media Group.

GEORGE CRUMP is president of Storage Switzerland, an IT analyst firm focused on storage and virtualization. The Nuts and Bolts of Private Cloud Storage is a

SearchCloudStorage.com e-publication.

Rich Castagna | VP of Editorial/Storage Media Group

Ellen O’Brien | Associate Editorial Director

Kim Hefner | Senior Managing Editor

Sarah Wilson | Site Editor

Erin Sullivan | Assistant Site Editor

Dave Raffo | Senior News Director

Sonia Lelii | Senior News Writer

Carol Sliwa | Senior Writer

Garry Kranz | Staff Writer

Linda Koury | Director of Online Design

Neva Maniscalco | Graphic Designer

Jillian Coffin | Publisher jcoffin@techtarget.com

TechTarget 275 Grove Street, Newton, MA 02466

www.techtarget.com

© 2014 TechTarget Inc. No part of this publication may be transmitted or re-produced in any form or by any means without written permission from the publisher. TechTarget reprints are available through The YGS Group.

About TechTarget: TechTarget publishes media for information technology professionals. More than 100 focused websites enable quick access to a deep store of news, advice and analysis about the technologies, products and pro-cesses crucial to your job. Our live and virtual events give you direct access to independent expert commentary and advice. At IT Knowledge Exchange, our social community, you can get advice and share solutions with peers and experts.

COVER ART: THINKSTOCK