how we leveraged emc clariion , celerra , and centera ... · emc technologies utilized and/or...
TRANSCRIPT
How We Leveraged EMC CLARiiON®, Celerra®, and Centera® Platforms to Consolidate Three Company’s
Data Systems into a Unified Storage Environment
EMC Proven Professional™ Knowledge Sharing 2008
Christopher Johnson UNIX Administrator / Storage Architect
Spheris Operations, LLC [email protected]
2008 EMC Proven Professional Knowledge Sharing 1
Table of Contents
Abstract ............................................................................................................................... 3 Spheris Profile..................................................................................................................... 3 Migration by Virtualization ................................................................................................ 5 The Corporate Goal – From Many-to-One ......................................................................... 7 When Windows Clients and UNIX Apps Overlap ............................................................. 7 Virtual Data Movers – Planning for the Unknown............................................................. 8 Windows Home Directories, ACLs, and Orphaned Files................................................... 8 Using the Celerra for Application Integration .................................................................... 9 Transitioning from Direct Attached Storage to CLARiiON Back-end Storage for Business Critical Systems ................................................................................................. 10 The Benefits of CLARiiON Scalability............................................................................ 11 Unstructured Data Retention Defines the Need for Data Archiving ................................ 12 Replication—the Logical Path to Disaster Recovery ....................................................... 14 Why EMC Technologies Were the Right Choice for Centralizing our Enterprise Storage Environment...................................................................................................................... 15 About the Author .............................................................................................................. 16
Disclaimer: The views, processes or methodologies published in this compilation are those of the authors. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.
2008 EMC Proven Professional Knowledge Sharing 2
Abstract In today’s global markets, IT administrators are challenged to consolidate data systems
of newly acquired or consolidated companies into one unified environment. This article
describes how we used EMC CLARiiON®, Celerra®, and Centera® platforms to
consolidate the disparate data systems of three companies after a series of acquisitions.
I will describe the challenges we faced as we migrated both Windows and UNIX files
servers from multiple domains onto our Celerra NAS system. Also, with so much
unstructured data within our environment requiring long-term online access, I will discuss
how Centera provided us with a great option to meet the company’s needs and service
level agreements (SLAs). With an extensive VMware virtual environment, we use our
CLARiiON disk array for its backend storage foundation. Last, I will describe how we
used our CLARiiON disk arrays to migrate our various database servers’ storage away
from localized disk systems.
EMC Technologies Utilized and/or Discussed: CLARiiON SAN – CX700, CX3-40, Mirrorview /A
Celerra NAS – NS502G, CIFS, NFS, Celerra Replicator
Centera CAS – Parity Protection, CFA, CUA, Centera Replication
Spheris Profile
Spheris is a leading, global outsource provider of clinical documentation technology and
services. We support more than 500 health systems, hospitals, and group practices
throughout the United States. Founded by doctors, Spheris solutions address the needs
of practitioners, health information directors, IT directors, and administrators. Intimate
understanding of how doctors and healthcare facilities work enables Spheris to combine
advanced technology with expert teams to deliver comprehensive solutions.
Spheris' worldwide headquarters is located in Franklin, TN, with operations centers in
Minot, ND; Sterling, VA.; Bangalore, India; Coimbatore, India; and Delhi, India.
2008 EMC Proven Professional Knowledge Sharing 3
As a technology-based company, Spheris’ IT Infrastructure provides the core foundation
needed to stay ahead of the ever changing technology landscape. The primary data
center is located within the Franklin, TN corporate headquarters, with a secondary data
center, for disaster recovery, in Sterling, VA. Spheris’ enterprise environment is truly a
“heterogeneous environment”, using products from multiple vendors such as Cisco, HP,
Dell, McData, and EMC for networking and systems needs.
The result of a corporate merger between the companies formerly known as Total eMed
and EDiX, Spheris merged the two corporate data and application platforms into a
singular, centralized data resource. With dual data centers, and corporate operations in
both Florida and Tennessee, the first task was to consolidate and eliminate overlapping
business areas. In the midst of the Total eMed/EDiX consolidation, Spheris acquired
Healthscribe, Inc. of Sterling, VA, adding another piece to the convergence puzzle.
Spheris’ acquisition of Healthscribe, Inc., provided a third data center and corporate
operations location. Following an active hurricane season in 2004, Spheris made the
decision to consolidate the FL and TN data centers with plans to make the VA data
center a standby disaster recovery site.
2008 EMC Proven Professional Knowledge Sharing 4
Migration by Virtualization Moving and consolidating a multi-million dollar center is a formidable task, particularly
when the data center houses the vital application data that provides 85% of the
company’s generated revenues. The FL data center housed between 200 - 300 physical
HP/Compaq servers and Cisco networking equipment. This included several database
and application servers, as well as the company’s primary messaging environment. With
a tight schedule, and very little room for error or business interruptions, Spheris took a
leap of faith with VMware technology.
Scaled Down Representation of Spheris’ Pre-migration WAN
We virtualized over a third of the physical servers by leveraging our CLARiiON CX700
storage array and six HP Proliant DL585 servers running VMware ESX Server. The FL
and TN data center consolidation was completed in just over half of the planned
migration time. In addition to decreasing migration time, we also reduced the physical
server footprint in the data center by nearly a two-to-one ratio.
2008 EMC Proven Professional Knowledge Sharing 5
We used a process of dual servers in a primary to secondary role since our
consolidation required relocating systems from FL to TN with minimal downtime. The
primary server was the physical server in the FL data center, while the secondary server
was a virtual server within our VMware Virtual Infrastructure in TN. Once we decided on
the best date and time for system migration, the primary server in FL was taken offline
and we renamed the secondary server, in TN, with the original primary server name.
Once the server rename was complete, we updated all of the name resolution records in
the domain name service (DNS) and Windows Internet Name Service (WINS) to allow
for proper routing to the new primary server in TN.
Scaled Down Representation of Spheris’ TN DC Post-migration
2008 EMC Proven Professional Knowledge Sharing 6
The Corporate Goal – From Many-to-One The next major initiative was to collapse and retire all of the various file servers from the
combined FL and TN data centers into a singular storage space. We decided to utilize
the Celerra NS502G network-attached-storage (NAS) device to further reduce our
physical server footprint within the data center.
Scaled Down Representation of Spheris’ File Server Environment Pre-migration
When Windows Clients and UNIX Apps Overlap We faced a significant challenge in our data centralization efforts since both Windows
clients and UNIX applications had to have access to files and folders within the same file
systems. We implemented our Celerra for use within a multi-protocol environment since
Windows and UNIX use two different methods for data security. Common Internet File
System (CIFS) for Windows and Network File System (NFS) for UNIX were the protocols
used. The Celerra was ideal since it allows access rights to its shared file systems to use
each environment’s native access protocols. In short, it allowed file access by Windows
clients through access control lists (ACLs) and UNIX applications through their user
and/or group IDs (UID / GID).
2008 EMC Proven Professional Knowledge Sharing 7
Virtual Data Movers – Planning for the Unknown Two long-term issues had to be considered before implementing the Celerra NAS for file
server consolidations: true data segregation and disaster recovery implementation.
Although the underlying file systems and CIFS servers provided logical separation of
data to users, they still used the same root file system of the physical data movers.
Virtual data movers (VDMs) were the only way to truly isolate sensitive data from our
general network users. By encapsulating each VDM into its own isolated file systems,
users are only able to access data on the CIFS servers that are within each respective
VDM. The ability to replicate the VDM, its configurations, and all of its file system data
through replication to another Celerra, either local or remote, are added benefits of the
VDM.
Windows Home Directories, ACLs, and Orphaned Files Once we began our file server consolidations to the Celerra, we were immediately able
to use many of its built-in features, such as snapshots and multi-path access to a single
directory. However, we found a few issues that had to be worked around using custom
scripts.
The first issue was dealing with the migration of Windows home directories from multiple
domains. By default, a user’s home directory is not accessible to any other user,
including administrators. We had to have each user run a custom home directory
migration script to insure that all of their personal files were moved. Although Celerra
has its own built-in home directory migration tools, we chose this route because we had
multiple domains with some overlapping user accounts.
The next issue was two-fold and only became apparent after complete server migrations
had taken place. The Celerra did not properly migrate the file data and/or its security
identifiers when we migrated files from Windows servers to a Celerra CIFS share that
were owned by an irresolvable user account. What would cause this? Files and
directories are created by either users or applications; although theoretically applications
are users, in this context we mean human users. As employees came and went, any
files that a particular user created were owned solely by that user. When the user’s
domain account was deleted it would leave any files that the user owned in an orphaned
2008 EMC Proven Professional Knowledge Sharing 8
state. This caused the orphaned files migrate without any data and/or invalid security
identifiers. Once we identified the orphaned data, we had to administratively take
ownership and re-migrate the data to the Celerra. Now that this issue has been identified
and documented, EMC has created a set of pre-migration tools that can be used to
check for and identify orphaned files on servers before their data is migrated. In the end,
we successfully migrated more than twenty physical file servers onto Celerra CIFS
servers.
Using the Celerra for Application Integration Spheris is a company born of multiple companies combined into one, our focus then
turned to consolidation of our different transcription platforms. We were able to place
files within a common directory for access by different applications using different
Uniform Naming Convention (UNC) paths, thanks to the Celerra’s ability to share
directories with multiple CIFS servers. Although our applications consolidation is still
being implemented over a period of phases, without the Celerra this process would have
been much more difficult to accomplish.
2008 EMC Proven Professional Knowledge Sharing 9
Scaled Down Representation of Spheris’ File Server Environment Post-migration
Transitioning from Direct Attached Storage to CLARiiON Back-end Storage for Business Critical Systems Centralizing our corporate file-sharing server data was a major step toward our goal of
enterprise-wide data unification. However, the core systems of almost all enterprise
environments, including Spheris, rely upon their back-end database and messaging
systems. Our business critical (BC) environment consists of both Oracle and MS SQL
database servers, in addition to MS Exchange Messaging servers.
We presented CLARiiON LUNs to each BC server to handle their long-term storage
needs to make this transition unobtrusive to our production environment. We transitioned
our BC servers away from locally attached disk sub-systems onto CLARiiON storage
LUNs through the coordinated efforts of our database administrators and infrastructure
teams.
Scaled Down Representation of Spheris’ Isolated Business Critical Environment Pre-migration
2008 EMC Proven Professional Knowledge Sharing 10
The Benefits of CLARiiON Scalability The CLARiiON SAN provided Spheris with several scalability options that were either
unavailable or cost-prohibitive to attain with direct-attached disks systems.
CLARiiON back-end storage enabled us to utilize the benefits of MS Cluster services for
some our MS SQL database servers. This technology is a great benefit from both a
redundancy and systems expense standpoint. Through database clustering we can add
multiple nodes to the cluster and keep the database online during a physical node
outage. From a cost perspective, with CLARiiON back-end storage there is only one
storage expense for the entire cluster. Also, we can easily add LUNs to the cluster’s
storage resources as the database’s storage needs increase.
CLARiiON back-end storage also allowed us to take advantage of the CLARiiON’s built-
in redundancy features. Through snapshots and clones we are able to provide online
disk recovery options, based on a particular point in time. More important are the
different replication options that are simply not available with local-attached disk sub-
systems. With options such as Mirrorview™, SAN Copy™, and RecoverPoint we can
provide both internal replication within our TN data center as well as wide-area
replication to our remote data center in VA.
2008 EMC Proven Professional Knowledge Sharing 11
Scaled Down Representation of Spheris’ Centralized Business Critical Environment Post-migration
Unstructured Data Retention Defines the Need for Data Archiving A major misconception about centralized data architecture is that it offers unlimited
storage space. Couple that with the regulations and legalities of data governance
imposed by SOX, HIPPA, and client agreements that require data retention for longer
periods of time and you can see how a long-term archiving system is the only logical
solution.
We met with several vendors who offered various archiving technologies to address our
long-term archiving needs. After much debate and research, we selected the EMC
Centera archiving solution. The Centera provided the most flexible options for archiving
data through its various integration tools with our other EMC products.
Due to accelerated time frames and overextended development resources we had to
find a way to implement our Centera into our environment without the need for
2008 EMC Proven Professional Knowledge Sharing 12
developmental resources. We expediently integrated the Centera into our enterprise
thanks to the Centera Universal Archival (CUA) Gateway and Celerra File Archiver
(CFA). With a few custom scripts and the CUA, we were able to archive specific data
files from network shared directories to the Centera prior to implementing our CFA. We
implemented the CFA to migrate files directly from Celerra file systems to the Centera
with the Celerra NAS serving as our network file sharing system of the future. Long-term,
we plan to use either multiple CFA units or DiskXtender® for NAS to migrate all
unstructured data to our Centera.
The data protection scheme that best suited our overall archival needs was another
consideration during our Centera implementation. Our initial cluster consisted of two
four-node cubes that provided a choice of data mirroring or parity protection. Both
options had their advantages and disadvantages such as faster rebuild times in a
mirrored system versus more available space in a parity system. We chose the parity
protection scheme for our environment because of the vast amount of data we had to
archive to the Centera during the initial and subsequent phases of data archiving.
Scaled Down Representation of Spheris’ Archival Topology
2008 EMC Proven Professional Knowledge Sharing 13
Replication—the Logical Path to Disaster Recovery Exploring the various options for enterprise disaster recovery caused us to consider the
different replication technologies available through our EMC back-end systems. Each
system provided solid replication options to synchronize data between our TN and VA
data centers. Both Celerra and Centera provide IP-based replication, with the ability to
define what data to replicate. The CLARiiON offers replication options to fit replication
tasks for both local and/or wide area networks.
We used VDMs and Celerra replication to set up and implement NAS disaster recovery
in phases, replicating each specific VDM as its data is determined to be critical enough
to warrant replication. Within the Centera, archived data is written to individual virtual
pools which use IP-based replication in the same manner that Celerra VDMs allow for
specific data sets to be migrated.
We are in the planning stages of moving over to true block-level replication via our
CLARiiON storage arrays, although we currently use database log shipping in our initial
phase of database replication. We are determining the best solution for our current and
long-term environment in these preliminary steps. While there are other solutions that
offer pseudo block-level replication, we plan on leveraging our EMC investment by
implementing a solution through either Mirrorview /A, SAN Copy, RecoverPoint, and/or
Replication Manager.
We have already brought our WAN-based storage infrastructure online in preparation for
our long-term disaster recovery implementation. FCIP 1620 Fiber-to-IP bridge/routers
are installed and configured in both our TN and VA data centers. These units connect to
McData fiber switches, a Connectrix EDM Director in TN and two DSM24 workgroup
switches in VA, and provide the backbone of our enterprise storage fabric.
2008 EMC Proven Professional Knowledge Sharing 14
Scaled Down Representation of Spheris’ Storage Replication Topology
Why EMC Technologies Were the Right Choice for Centralizing our Enterprise Storage Environment There are many companies with new and emerging technologies vying for a place in the
fast growing world of enterprise storage. Some of these technologies are good, others
not so good, and some that may be great but are yet untested in an enterprise
production environment.
With EMC storage technologies we knew we were not only purchasing tried-and-true
enterprise-class products, we knew we were gaining a long-term partner for our
company. We recognized the value of a mature company with the resources to research
and bring to market products that are ready to go from the moment that they are shipped
to client sites.
2008 EMC Proven Professional Knowledge Sharing 15
Couple that with a full-fledged implementation team to guide each project through the
various phases and you have a formula for success, regardless of the size of the project
in scope. These qualities cannot be easily matched by other vendors regardless of their
size. Additionally, having access to 24-hour technical support is invaluable, with a
company such as ours.
In the end, Spheris did not need another vendor to sell us the next hot thing to hit the IT
world. We needed and found a true “follow the sun” partner in EMC… and that is why
EMC Was the Right Choice for Spheris!
About the Author In my current role as a UNIX administrator, I serve as Spheris’ primary architect and
administrator of our enterprise storage environment. I have been with Spheris for over
10 years, beginning as an employee with EDiX back in July 1997. During my career with
Spheris, I have served in multiple roles including help desk, client technical support, field
engineering, network administration, and systems administration. I have watched the
company grow from the early start-up phase into becoming one of the market-leading
medical information outsourcing companies. Currently holding certifications in Microsoft,
Cisco, and EMC technologies, I have not only a wealth of Spheris-related knowledge,
but also a great foundation and understanding of the overall enterprise network and
computing environments.
2008 EMC Proven Professional Knowledge Sharing 16