backup recovery systems and architecture student guide

Backup Recovery Systems and Architecture

Student Guide

Education Services April 2013

Copyright © 2013 EMC Corporation. All rights reserved

Welcome to Backup Recovery Systems and Architecture training. Copyright © 1996, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011,2012, 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC2, EMC, Data Domain, RSA, EMC Centera, EMC ControlCenter, EMC LifeLine, EMC OnCourse, EMC Proven, EMC Snap, EMC SourceOne, EMC Storage Administrator, Acartus, Access Logix, AdvantEdge, AlphaStor, ApplicationXtender, ArchiveXtender, Atmos, Authentica, Authentic Problems, Automated Resource Manager, AutoStart, AutoSwap, AVALONidm, Avamar, Captiva, Catalog Solution, C-Clip, Celerra, Celerra Replicator, Centera, CenterStage, CentraStar, ClaimPack, ClaimsEditor, CLARiiON, ClientPak, Codebook Correlation Technology, Common Information Model, Configuration Intelligence, Configuresoft, Connectrix, CopyCross, CopyPoint, Dantz, DatabaseXtender, Direct Matrix Architecture, DiskXtender, DiskXtender 2000, Document Sciences, Documentum, elnput, E-Lab, EmailXaminer, EmailXtender, Enginuity, eRoom, Event Explorer, FarPoint, FirstPass, FLARE, FormWare, Geosynchrony, Global File Virtualization, Graphic Visualization, Greenplum, HighRoad, HomeBase, InfoMover, Infoscape, Infra, InputAccel, InputAccel Express, Invista, Ionix, ISIS, Max Retriever, MediaStor, MirrorView, Navisphere, NetWorker, nLayers, OnAlert, OpenScale, PixTools, Powerlink, PowerPath, PowerSnap, QuickScan, Rainfinity, RepliCare, RepliStor, ResourcePak, Retrospect, RSA, the RSA logo, SafeLine, SAN Advisor, SAN Copy, SAN Manager, Smarts, SnapImage, SnapSure, SnapView, SRDF, StorageScope, SupportMate, SymmAPI, SymmEnabler, Symmetrix, Symmetrix DMX, Symmetrix VMAX, TimeFinder, UltraFlex, UltraPoint, UltraScale, Unisphere, VMAX, Vblock, Viewlets, Virtual Matrix, Virtual Matrix Architecture, Virtual Provisioning, VisualSAN, VisualSRM, Voyence, VPLEX, VSAM-Assist, WebXtender, xPression, xPresso, YottaYotta, the EMC logo, and where information lives, are registered trademarks or trademarks of EMC Corporation in the United States and other countries.

All other trademarks used herein are the property of their respective owners.

© Copyright 2013 EMC Corporation. All rights reserved. Published in the USA.

Revision Date: April 2013 Revision Number: MR-1CP-BRSSCI.2

1 Module 0: Course Introduction


This course provides participants with a solid foundation in Backup and Recovery infrastructure. The course focuses on the concepts and technologies used in Backup and Recovery environments. Participants learn about backup and recovery theory, including backup methods, planning and key terminology. The course includes topics on how storage technologies work and how their features such as replication and snapshots can be used for backup. This is followed with a look into data sources at the backup client and storage node backup targets. The course finishes with backup and recovery planning and a high level look at the EMC Backup and Recovery product portfolio.



Please note that actual class agenda may vary from what is shown here due to the pace of the individual class.



The Backup Recovery Systems and Architecture course supports the Backup Recovery Systems and Architecture Exam. This exam is an associate level qualifying exam for the following EMC Proven Professional Backup and Recovery Specialty tracks: Technology Architect, Implementation Engineer and Storage Administrator. It is an alternative to Information Storage and Management as the Associate requirement for all EMC Proven Backup and Recovery tracks.

Please see the EMC Education Services website for additional information on the EMC Proven Professional program.


Copyright © 2013 EMC Corporation. Do not copy - All Rights Reserved.

This module provides an introduction to backup and recovery. We explore the reasons for performing backups, define common backup and recovery terms, and look at the flow of data in typical client/server backup and restore operations.

1 Module 1: Backup Theory

Copyright © 2013 EMC Corporation. Do not copy - All Rights Reserved. 2

In this lesson, we define backup and look at various reasons for backing up data. We define common backup and recovery terms, including recovery point and recovery time objectives and backup granularity levels.

Module 1: Backup Theory


Physical damage to a disk can result in data loss. A significant role of traditional backups was to protect against disk-drive failures and loss of online access, but today’s availability features in EMC storage systems, such as mirroring, redundancy, multipathing, and the elimination of single points of failure, can reduce risk by protecting against data loss. However, they cannot protect against other factors that can result in data loss.

People make mistakes and software failures can destroy or lose data.

Unhappy employees or external hackers may breach security and maliciously destroy data. Viruses can destroy data, impact data integrity, and halt key operations. Physical security breaches can destroy equipment that contains data and applications.

Natural and other events such as earthquakes, lightning strikes, floods, tornados, hurricanes, accidents, chemical spills, and power grid failures can cause not only the loss of data but also the loss of an entire computer facility. Offsite data storage is often justified to protect a business from these types of events.

Government regulations may require certain data to be kept for extended timeframes. Corporations may establish their own extended retention policies for intellectual property to protect them against litigation. The regulations and business requirements that drive data as an archive generally require data to be retained at an offsite location.



Backup is a copy of production data, created and retained for the sole purpose of recovering deleted or corrupted data. With growing business and regulatory demands for data storage, retention, and availability, organizations are faced with the task of backing up an ever-increasing amount of data. This task becomes more challenging as demand for consistent backup and quick restore of data increases throughout the enterprise which may be spread over multiple sites. Moreover, organizations need to accomplish backup at a lower cost with minimum resources.

Organizations must ensure that the right data is in the right place at the right time. Evaluating backup technologies, recovery, and retention requirements for data and applications is an essential step to ensure successful implementation of a backup and recovery solution. The solution must facilitate restores and recovery from backup as required by the business.



Backups are performed for three primary purposes: disaster recovery, operational restores and long term storage.

Disaster recovery addresses the requirement to be able to restore all, or a large part of, an IT infrastructure in the event of a major disaster. The backup copies are used for restoring data at an alternate site when the primary site is incapacitated due to a disaster. Based on recovery requirements, organizations use different backup strategies for disaster recovery.

Operational backup is a backup of data at a Point-in-time for the purpose of restoring data in the event of data loss or logical corruptions that may occur during routine processing. The majority of restore requests in an organization are classified in this category. An example of an operational backup is a backup taken just before a major change to a production system. This ensures the availability of a clean copy of production data if the change corrupts the production data.

Backups are also performed to address long term storage requirements. For example, an organization may be required to keep transaction records and other business records required for regulatory compliance. Depending on when the data is retrieved, the original may no longer exist.

Apart from addressing disaster recovery, long term storage, and operational requirements, backups serve as a protection against data loss due to physical damage of a storage device, software failures, or virus attacks. Backups can also be used to protect against accidents such as a deletion or intentional data destruction.



Most restores are at the file and directory level. Most common restores involve application data, email and specific files. Full system restores are rare. When planning a backup solution, it is important to evaluate the restore needs.

Data recovery or restore typically involves the recovery of individual files or directories. For this type of recovery, it is important to know, by application, the characteristics of the most common restores. For example, when do most restores occur? What is most commonly asked for? Could end-users perform their own restores? What is the age of data that is asked for? Answers to these questions impact the decisions for the backup method, backup storage media, backup retention, requirements for tiered backup storage and others.

Disaster recovery involves bringing a data center or a large part of a data center to operational state in case of disaster affecting the production site location. Data for recovery are located in offsite locations. Recovery could be from backups or replicas depending upon the chosen disaster recovery strategy. Portable media, such as tapes, sent to an offsite location could be used for recovery. Or, where data is continuously replicated to a remote site, in the event of failure, applications can continue to run using the remote replica. In another example, data backed up locally can be replicated to an offsite location by the backup application. Recovery can be from the most recent point-in-time replicated backup data.

Bare metal recovery (BMR) is the process of restoring a production server to an operational state. The backup for bare metal recovery is a backup in which all metadata, system information, and application configurations are appropriately backed up for a full system recovery. Typically, the server configuration information is gathered and backed up by a bare metal recovery application or an operating system facility, and the application and its data are backed up by a backup application. The recovery involves a combination of both backups.



A backup that has not been recovery tested cannot be trusted. In many instances, some critical data that is required to restore the servers may go missing in the backup set, which can only be identified by testing the complete recovery. It is always a good practice to test recoveries and perform trials and not wait for a disaster to occur. One of the main objectives of backups is RTO. An RTO cannot be established if the backups are never tested.

During disasters, the remote locations start the servers and load the data in their environment and get them ready. So it is important to know that the processes followed should be documented because they may be used by a different team than the original team that architected the solution. Testing data restores and documenting the procedures helps meet many of the goals.

Testing disaster recovery procedures also provides a complete view into how to get the data back and the stages involved. There can be instances where the system administrators plan to reinstall the operating system on a disaster and store the operating system software media but forget to store the license codes. Such holes will only be identified during a test and it is better to find the holes in a test rather than after a disaster.



Data sets are groups of data received from one or more backup clients. Usually one data set is generated for each file system a backup client has, but it depends on the backup configuration.

The backup window is the time a business has to back up their data, traditionally 6-8 hours in the evening or weekends, but could occur at any time. Due to the accelerating rate of data growth, backup windows for many applications are shrinking and, in some cases, nonexistent.

Staging is the process of moving a data set from one backup device to another. It’s usually used to move data from a backup disk device to a tape, in Backup to Disk solutions.

Cloning is the process of creating a copy of backup data.

Retention period is the length of time that a particular version of a backup data set is available to be restored.

The expiration date is the date when the backup or media can be overwritten.



Recovery-Point Objective (RPO): This is the Point-in-time to which systems and data must be recovered after an outage. It defines the amount of data loss that a business can endure. The RPO is not necessarily the most recent point in time. A large RPO signifies high tolerance to information loss in a business. Based on the RPO, organizations plan for the minimum frequency with which a backup or replica must be made. For example, if the RPO is six hours, backups or replicas must be made at least once in 6 hours. An organization can plan for an appropriate backup solution on the basis of the RPO it sets. For example, if RPO is 24 hours that means backups can be created on tape or disk once a day. The corresponding recovery strategy is to restore data from the last backup. Similarly for zero RPO, data can be mirrored synchronously to a remote site. As we have seen, the retention period for a backup is also derived from an RPO specified for operational recovery.

Recovery-Time Objective (RTO): The time within which systems, applications, or functions must be recovered after an outage. It defines the amount of downtime that a business can endure and survive. Businesses can optimize disaster recovery plans after defining the RTO for a given data center or network. RTO influences the type of backup media to be used. For example, if the RTO is two hours, then use a disk backup because it enables a faster restore than a tape backup. However, for an RTO of one week, tape backup will likely meet requirements. To meet a designated RTO, businesses may choose to employ a combination of different backup solutions to minimize recovery time. Some examples of RTOs and the recovery strategies to ensure data availability are listed below:

• RTO of 72 hours: Restore from backup tapes at a cold site

• RTO of 12 hours: Restore from disk or tapes at a hot site

• RTO of 4 hours: Restore from disk backup

• RTO of 1 hour: Cluster production servers with controller-based disk mirroring

• RTO of a few seconds: Cluster production servers with bidirectional mirroring, enabling the applications to run at both sites simultaneously



Backup granularity depends on business needs and the required RTO/RPO. Based on granularity, backups can be categorized as full, cumulative, and incremental. Most organizations use a combination of these three backup types to meet their backup and recovery requirements. The slide depicts the categories of backup granularity.

Full backup is a backup of the complete data on the production volumes at a certain point in time. A full backup copy is created by copying the data on the production volumes to a secondary storage device.

Incremental backup copies the data that has changed the last backup, regardless of the level of the previous backup. This is much faster because the volume of data backed up is restricted to changed data, but it takes longer to restore an entire file system or save set to a specific point in time.

Cumulative (or differential) backup copies the data that has changed since the last full backup. This method takes longer than incremental backup but is faster to restore.

Synthetic (or constructed) full backup is another type of backup that is used in implementations where the production volume resources cannot be exclusively reserved for a backup process for extended periods to perform a full backup. It is created from the most recent full backup and all the incremental backups performed after that full backup. A synthetic full backup enables a full backup copy to be created offline without disrupting the I/O operation on the production volume. This also frees up network resources from the backup process, making them available for other production uses.



The process of restoring from an incremental backup requires the last full backup and all the incremental backups available until the point of restoration. Consider an example, a full backup is performed on Monday evening. Each day after that, an incremental backup is performed. On Tuesday, a new file (File 4 in the figure) is added, and no other files have changed. Consequently, only File 4 is copied during the incremental backup performed on Tuesday evening.

On Wednesday, no new files are added, but File 3 has been modified. Therefore, only the modified File 3 is copied during the incremental backup on Wednesday evening. Similarly, the incremental backup on Thursday copies only File 5. On Friday morning, there is data corruption, which requires data restoration from the backup. The last full backup and all subsequent incremental backups are used to restore the production volume data to its previous state on Thursday evening.



A full system restore from a cumulative backup requires the last full backup and the most recent cumulative backup. Consider an example, a full backup of the business data is taken on Monday evening. Each day after that, a cumulative backup is taken. On Tuesday, File 4 is added and no other data is modified since the previous full backup of Monday evening.

Consequently, the cumulative backup on Tuesday evening copies only File 4. On Wednesday, File 5 is added. The cumulative backup taking place on Wednesday evening copies both File 4 and File 5 because these files have been added or modified since the last full backup. Similarly, on Thursday, File 6 is added. Therefore, the cumulative backup on Thursday evening copies all three files: File 4, File 5, and File 6. On Friday morning, data corruption occurs that requires data restoration using backup copies. The last full backup and the latest cumulative backup are used to restore the production volume data to its previous state on Thursday evening.



In this lesson, we defined backup and looked at various reasons for backing up data. We defined common backup and recovery terms, including recovery point and recovery time objectives and backup granularity levels.



This lesson covers the components of a client/server backup architecture and backup and recovery workflows. Then, we discuss several strategies for backup and how replication and archive can be leveraged for backup.



Due to the complexity of IT operations, organizations typically use a backup application to work within the existing framework of hardware, operating system software and network communication protocols to provide protection for the critical application data that the framework supports. With backup applications, backups can be configured and automatically performed thus implementing consistent backup and recovery procedures throughout the organization. Typically, a backup system uses a client/server architecture with a backup server, storage node and multiple backup clients.



The backup client’s most important functions are to gather the data to be backed up, send that data to a storage node and retrieve the data during a recovery. Backup clients are usually the data servers and application hosts in an IT environment. The types of data that are typically backed up include file system data and database applications. The backup client component can also be local to the backup server or a backup client can reside on another host in order to back up the data visible to that server (often known as a proxy client).



The storage node is responsible for organizing the client’s data and writing the data to a backup device. Backup devices may be directly-attached or accessible through the network. A storage node is a host that controls one or more backup devices and is, in turn, controlled by the backup server. Storage nodes also send metadata about the save sets written to the backup device during the backup to the backup server. This information is used for future backups, as well as for recoveries.

The storage node component may be integrated with the backup server and hosted on the same physical platform, it could reside on a separate host machine, or it could reside on the client itself. In a distributed architecture, there are multiple storage nodes in a backup environment, each controlled by the backup server.

Some backup applications refer to the storage node as the media server because it connects to the storage device. Storage nodes play an important role in backup planning because they can be used to consolidate backup servers.



The backup server directs and supports the client backup and restore operations. It controls the backup environment operations, policies, and the tracking of backups. The backup server depends on backup clients to gather the data to be backed up and the storage nodes for writing/reading the backup data. The backup server receives backup metadata from the backup clients and storage nodes, and stores the information in the backup catalog.

Backup metadata is defined as data about the backup, such as file names comprising the backup data set, time of backup, size, permissions, ownership, and most important, tracking information to allow locating the data to be restored. The expiration of records in the backup catalog is based on the retention period. The backup server also stores and maintains configuration information, such as supported clients, devices and when to run the backups.



Backup devices store the backup data. Backup and recovery technologies are in transition from traditional tape media to disk. Backup to disk is replacing tape and associated devices as the primary target for backup. Backup to disk systems offer major advantages over equivalent scale tape systems in terms of capital costs, operating costs, support costs and quality of service. The slide lists various backup media and their key characteristics.



When a backup process is initiated, significant network communication takes place between the different components of a backup infrastructure. The backup server initiates the backup process for different clients based on the backup schedule configured for them. The backup server coordinates the backup process with all the components in a backup configuration. The backup server maintains the information about backup clients to be contacted and storage nodes to be used in a backup operation. The backup server retrieves the backup-related information from the backup catalog and, based on this information, instructs the appropriate storage node to load the backup media into the backup devices.

Simultaneously, it instructs the backup clients to send their metadata to the backup server, and data to be backed up to the appropriate storage node. On receiving this request, the backup client sends tracking information to the backup server. The backup server writes this metadata on its backup catalog. The backup client sends the data to the storage node, and the storage node writes the data to the storage device. The storage node also sends tracking information to the backup server to keep it updated about the media being used in the backup process. After all the data is backed up, the storage node closes the connection to the backup device. The backup server writes backup completion status to the backup catalog.



A restore process is manually initiated by the client. Some backup software may have separate applications for restore operations. These restore applications are accessible only to administrators and those having recovery permissions. Upon receiving a restore request, the user opens the restore application to view the list of clients that have been backed up. While selecting the client for which a restore request has been made, the user also needs to identify the client that will receive the restored data. Data can be restored on the same client or another client given proper permissions. The user then selects the data to be restored and the specified Point-in-time to which the data has to be restored based on the RPO. Note that because all of this information comes from the backup catalog, the restore application must also communicate with the backup server.

The backup server identifies the backup media required for the restore and notifies the storage node to load the backup media. Data is then read and sent to the client that has been identified to receive the restored data. Some restores are successfully accomplished by recovering only the requested production data. For example, the recovery process of a spreadsheet is completed when the specific file is restored. In database restorations, additional data such as log files and production data must be restored to ensure application consistency for the restored data.



Several choices are available to get the data to the backup media.

• Data can be copied from the primary storage to backup storage, either disk or tape, onsite. This strategy impacts the production server where the data is located since it uses the server’s resources.

• To avoid an impact on the production application, you can mirror (or perform a point-in-time snapshot of) a production volume. For example, you can mount the mirror or snapshot on a separate proxy server and then back it up (copy) to the backup media. This option completely frees up the production server from performing backups.

• Remote backup can be used to comply with offsite requirements. A copy from the primary storage is performed directly to the backup media that is sitting on another site.

• Backup can be performed to a set of backup media which is kept onsite for operational restore requirements and then cloned to another set of media for offsite purposes. Depending upong the type of media, the backup data can be transported to the remote site or the local backups can be cloned directly to an offsite location thus removing any manual procedures associated with transporting the backup media.



Cloning and staging features of backup applications provide the ability to structure the backup and recovery process to optimize backup performance and ensure that backup data resides where needed.

In a backup environment, cloning (a.k.a. twinning, when done simultaneously with backup) is used to produce duplicate sets of backup save sets, in other words backups of backups. The clone can be used in the same way as the original backup. An example of the use of cloning is where the original backup is first written to disk media and then cloned offsite to either tape or disk for disaster recovery purposes. This provides the high performance of backing up to and restoring from disk for operational requirements, while providing disaster protection with offsite storage.

Staging is the process of transferring data from one storage medium or volume to another. At the end of the staging process, the backup resides on only one device. For example, staging can be employed to direct the initial backup to disk, then move (copy/delete) the data to a secondary storage media at a later time, thus freeing up disk space on the original backup device. The backup data resides on disk for a period of time when most operational restores of the data are expected to occur, expediting the restore process. Both backup and restore performance are faster than a direct backup to tape or other secondary media method would be.

Backup software applications implement different features to the cloning and staging process. For example, cloning could be configured to start automatically at the end of a backup or at a specified time of day so as not to run during the backup window. Automatic staging could be initiated after a backup has been on the staging device for a period of time or event-driven where it is initiated when the available space on the staging device drops below a specified percentage. Most applications also allow for the backup administrator to manually initiate the processes. Some backup software applications allow for different retention periods on the clone and/or staged backups than the retention on the original backups. This allows the copy to be used for long term storage requirements.

A key disadvantage of the either process is that more resources (additional CPU, memory, I/O bandwidth, and media) are required. Also, they add more steps to the overall backup process.



Business continuity solutions include point-in-time backups, local replication, and remote replications.

Local replication creates a replica of production data within the same array. A replica such as a split-mirror or pointer-based snapshot can be used to create a point-in-time backup. The mirror or snapshot is mounted on a backup client for backup thus freeing the production application server from performing the backup.

In remote replication, data from the production devices is copied to replica devices on a remote array. For business continuity in the event of a disaster, applications can continue to run using remote replicas. Similar to backups of local replicas, depending upon the method used to create the remote replica, it can be used as the source of the backup data.

In the next module, we take a look at the sources of data, storage systems and the various replication and continuous data protection technologies and strategies.



Data archive is a repository where fixed content is stored. It enables organizations to retain their data for an extended period of time in order to meet regulatory compliance or generate new revenue strategies. X-rays, emails and multimedia files are examples of fixed content. An archive can be implemented as an online, nearline, or offline solution:

• Online archive: A storage device directly connected to a host that makes the data immediately accessible.

• Nearline archive: A storage device connected to a host, but the device where the data is stored must be mounted or loaded to access the data.

• Offline archive: A storage device that is not ready to use. Manual intervention is required to connect, mount, or load the storage device before data can be accessed.

Archive is different from backup in that archived data remains the only copy of that data. Users can access the data in the archive but cannot change it. Backup processing can leverage an archive by archiving static data before taking a backup of the production source. This results in less data on the production source to be backed up.



This lesson covered the components of the client/server backup architecture and backup and recovery workflows. Then, we looked at several strategies for backup and how replication and archive can be leveraged for backup.



These are the key points covered in this module. Please take a moment to review them.



This module introduces you to disk architecture, storage systems and RAID. You learn about storage area networks and network attached storage. We will complete the module with a discussion of storage system features that are used in backup and recovery environments, including local and remote replication and continuous data protection technologies.

1 Module 2: Information Storage Concepts


This lesson covers the components and physical structure of mechanical disks and the factors affecting mechanical disk drive performance. Then, we will look at the characteristics and architecture of flash drives.



A mechanical disk drive uses a rapidly moving arm to read and write data across a flat platter coated with magnetic particles. Data is transferred from the magnetic platter through the R/W head to the computer. Several platters are assembled together with the R/W head and controller, most commonly referred to as a hard disk drive (HDD). Data can be recorded and erased on a magnetic disk any number of times. This section details the different components of a mechanical disk, the mechanism for organizing and storing data on disks, and the factors that affect disk performance.

Key components of a mechanical disk drive are platter, spindle, read/write head, actuator arm assembly, and controller:

A typical HDD consists of one or more flat circular disks called platters. The data is recorded on these platters in binary codes (0s and 1s). The set of rotating platters is sealed in a case, called a Head Disk Assembly (HDA). A platter is a rigid, round disk coated with magnetic material on both surfaces (top and bottom). The data is encoded by polarizing the magnetic area, or domains, of the disk surface. Data can be written to or read from both surfaces of the platter. The number of platters and the storage capacity of each platter determine the total capacity of the drive.

A spindle connects all the platters and is connected to a motor. The motor of the spindle rotates with a constant speed. The disk platter spins at a speed of several thousands of revolutions per minute (rpm). Disk drives have spindle speeds of 7,200 rpm, 10,000 rpm, or 15,000 rpm. Disks used on current storage systems have a platter diameter of 3.5” (90 mm). When the platter spins at 15,000 rpm, the outer edge is moving at around 25 percent of the speed of sound. The speed of the platter is increasing with improvements in technology, although the extent to which it can be improved is limited.



Read/Write (R/W) heads, shown, read and write data from or to a platter. Drives have two R/W heads per platter, one for each surface of the platter. The R/W head changes the magnetic polarization on the surface of the platter when writing data. While reading data, this head detects magnetic polarization on the surface of the platter. During reads and writes, the R/W head senses the magnetic polarization and never touches the surface of the platter. When the spindle is rotating, there is a microscopic air gap between the R/W heads and the platters, known as the head flying height. This air gap is removed when the spindle stops rotating and the R/W head rests on a special area on the platter near the spindle. This area is called the landing zone. The landing zone is coated with a lubricant to reduce friction between the head and the platter. The logic on the disk drive ensures that heads are moved to the landing zone before they touch the surface. If the drive malfunctions and the R/W head accidentally touches the surface of the platter outside the landing zone, a head crash occurs. In a head crash, the magnetic coating on the platter is scratched and may cause damage to the R/W head. A head crash generally results in data loss.

The R/W heads are mounted on the actuator arm assembly, which positions the R/W head at the location on the platter where the data needs to be written or read.

The controller is a printed circuit board, mounted at the bottom of a disk drive. It consists of a microprocessor, internal memory, circuitry, and firmware. The firmware controls power to the spindle motor and the speed of the motor. It also manages communication between the drive and the host. In addition, it controls the R/W operations by moving the actuator arm and switching between different R/W heads, and performs the optimization of data access.



Data on the disk is recorded on tracks, which are concentric rings on the platter around the spindle. The tracks are numbered, starting from zero, from the outer edge of the platter. The number of tracks per inch (TPI) on the platter (or the track density) measures how tightly the tracks are packed on a platter.

Each track is divided into smaller units called sectors. A sector is the smallest, individually addressable unit of storage. The track and sector structure is written on the platter by the drive manufacturer using a formatting operation. The number of sectors per track varies according to the specific drive. The first personal computer disks had 17 sectors per track. Recent disks have a much larger number of sectors on a single track. There can be thousands of tracks on a platter, depending on the physical dimensions and recording density of the platter.

Typically, a sector holds 512 bytes of user data, although some disks can be formatted with larger sector sizes. In addition to user data, a sector also stores other information, such as sector number, head number or platter number, and track number. This information helps the controller to locate the data on the drive, but storing this information consumes space on the disk. Consequently, there is a difference between the capacity of an unformatted disk and a formatted one. Drive manufacturers generally advertise the unformatted capacity —for example, a disk advertised as being 500GB will only hold 465.7GB of user data, and the remaining 34.3GB is used for metadata. A cylinder is the set of identical tracks on both surfaces of each drive platter. The location of drive heads is referred to by cylinder number, not by track number.



Earlier drives used physical addresses consisting of the cylinder, head, and sector (CHS) number to refer to specific locations on the disk, as shown in figure (a), and the host operating system had to be aware of the geometry of each disk being used. Logical block addressing (LBA), shown in figure (b), simplifies addressing by using a linear address to access physical blocks of data. The disk controller translates LBA to a CHS address, and the host only needs to know the size of the disk drive in terms of the number of blocks. The logical blocks are mapped to physical sectors on a 1:1 basis.

In figure (b), the drive shows eight sectors per track, eight heads, and four cylinders. This means a total of 8 × 8 × 4 = 256 blocks, so the block number ranges from 0 to 255. Each block has its own unique address. Assuming that the sector holds 512 bytes, a 500 GB drive with a formatted capacity of 465.7 GB will have in excess of 976,000,000 blocks.



A disk drive is an electromechanical device that governs the overall performance of the storage system environment. The various factors that affect the performance of disk drives are discussed in this section.

Disk service time is the time taken by a disk to complete an I/O request. Components that contribute to service time on a disk drive are seek time, rotational latency, and data transfer rate.



The seek time (also called access time) describes the time taken to position the R/W heads across the platter with a radial movement (moving along the radius of the platter). In other words, it is the time taken to reposition and settle the arm and the head over the correct track. The lower the seek time, the faster the I/O operation.

Disk vendors publish the following seek time specifications:

Full Stroke: The time taken by the R/W head to move across the entire width of the disk, from the innermost track to the outermost track.

Average: The average time taken by the R/W head to move from one random track to another, normally listed as the time for one-third of a full stroke.

Track-to-Track: The time taken by the R/W head to move between adjacent tracks.

Each of these specifications is measured in milliseconds. The average seek time on a modern disk is typically in the range of 3 to 15 milliseconds. Seek time has more impact on the read operation of random tracks rather than adjacent tracks. To minimize the seek time, data can be written to only a subset of the available cylinders. This results in lower usable capacity than the actual capacity of the drive. For example, a 500 GB disk drive is set up to use only the first 40 percent of the cylinders and is effectively treated as a 200 GB drive. This is known as short-stroking the drive.



To access data, the actuator arm moves the R/W head over the platter to a particular track while the platter spins to position the requested sector under the R/W head. The time taken by the platter to rotate and position the data under the R/W head is called rotational latency. This latency depends on the rotation speed of the spindle and is measured in milliseconds. The average rotational latency is one-half of the time taken for a full rotation. Similar to the seek time, rotational latency has more impact on the reading/writing of random sectors on the disk than on the same operations on adjacent sectors. Average rotational latency is around 5.5 ms for a 5,400-rpm drive, and around 2.0 ms for a 15,000 rpm drive.



The data transfer rate (also called transfer rate) refers to the average amount of data per unit time that the drive can deliver to the HBA. It is important to first understand the process of read and write operations in order to calculate data transfer rates. In a read operation, the data first moves from disk platters to R/W heads, and then it moves to the drive’s internal buffer. Finally, data moves from the buffer through the interface to the host HBA. In a write operation, the data moves from the HBA to the internal buffer of the disk drive through the drive’s interface. The data then moves from the buffer to the R/W heads. Finally, it moves from the R/W heads to the platters. The data transfer rates during the R/W operations are measured in terms of internal and external transfer rates, as shown in here.

Internal transfer rate is the speed at which data moves from a single track of a platter’s surface to internal buffer (cache) of the disk. Internal transfer rate takes into account factors such as the seek time. External transfer rate is the rate at which data can be moved through the interface to the HBA. External transfer rate is generally the advertised speed of the interface, such as 133 MB/s for ATA. The sustained external transfer rate is lower than the interface speed.



Flash drives, also referred to as solid state drives, SSDs, use semiconductor-based solid state memory (flash memory) to store and retrieve data. They are disk drives that deliver the ultra-high performance required by performance-sensitive applications.

Unlike conventional mechanical disk drives, flash drives contain no moving parts; therefore, they do not have seek and rotational latencies. Flash drives deliver a high number of IOPS with very low response times. Also, being a semiconductor-based device, flash drives consume less power, compared to mechanical drives. Flash drives are especially suited for applications with small block size and random-read workloads that require consistently low response times. Applications that need to process massive amounts of information quickly, such as currency exchange, electronic trading systems, and real-time data feed processing, benefit from flash drives.

Overall, flash drives provide better total cost of ownership (TCO) even though they may cost more on $/GB basis. By implementing flash drives, businesses can meet application performance requirements with far fewer drives. This reduction not only provides savings in terms of drive cost, but also translates to savings for power, cooling, and space consumption. Fewer numbers of drives in the environment also means less costs for managing the storage.

Flash drives have the same external physical format and connections as mechanical hard drives. Flash drives maintain compatibility in form and format with mechanical hard drives. This enables easy replacement of a mechanical drive with a flash drive.



The key components of the flash drive are I/O interface, controller and mass storage.

The I/O interface provides power and data to the flash drive.

The controller incorporates the electronics that bridge the NAND memory component to the host computer. The controller is the embedded processor that executes the firmware-level code. RAM is used as a cache in the management of data being read and written from the flash drive and for the drive’s operational programs and data. The NVRAM is used to store the flash drive’s operational software and data when the power is off.

Mass storage is an electronic hardware circuit that stores information and supports protocols for sending and retrieving the information over the hardware interface. It is an array of non-volatile NAND memory chips that retain their contents when powered off. NAND flash memory is written and read in blocks. The number and capacity of the individual chops is directly related to the flash drive’s capacity.



This lesson covered the components and physical structure of mechanical disks. Next, we discussed the factors affecting mechanical disk drive performance. Finally, we discussed the characteristics and architecture of flash drives.



This lesson introduces the major components of storage systems including hosts, connectivity and storage. We look at the logical and physical components of hosts, connectivity options, types of storage and how files are moved to and from storage.



The major system components are host computers, storage and the connectivity between the devices.



Users store and retrieve data through applications. The computers on which these applications run are referred to as hosts. Hosts can range from simple laptops to complex clusters of servers. A host consists of physical components (hardware devices) that communicate with one another using logical components (software and protocols). Access to data and the overall performance of the storage system environment depend on both the physical and logical components of a host.

A host has three key physical components: Central processing unit (CPU), Storage (such as internal memory and disk devices) and Input/Output (I/O) devices.



The logical components of a host consist of the software applications and protocols that enable data communication with the user as well as the physical components. The major logical components of a host are applications, operating system, file system, volume manager and device drivers.



An application is a computer program that provides the logic for computing operations. It provides an interface between the user and the host and among multiple hosts. Conventional business applications using databases have a three-tiered architecture — the application user interface forms the front-end tier; the computing logic forms, or the application itself is, the middle tier; and the underlying databases that organize the data form the back-end tier. The application sends requests to the underlying operating system to perform read/write (R/W) operations on the storage devices. Applications can be layered on the database, which in turn uses the OS services to perform R/W operations to storage devices. These R/W operations (I/O operations) enable transactions between the front-end and back-end tiers. Data access can be classified as block-level or file-level depending on whether the application uses a logical block address or the file name and a file record identifier to read from and write to a disk.

Block-level access is the basic mechanism for disk access. In this type of access, data is stored and retrieved from disks by specifying the logical block address. The block address is derived based on the geometric configuration of the disks. Block size defines the basic unit of data storage and retrieval by an application. Databases, such as Oracle and SQL Server, define the block size for data access and the location of the data on the disk in terms of the logical block address when an I/O operation is performed.

File-level access is an abstraction of block-level access. File-level access to data is provided by specifying the name and path of the file. It uses the underlying block-level access to storage and hides the complexities of logical block addressing (LBA) from the application and the DBMS.

An operating system controls all aspects of the computing environment. It works between the application and physical components of the computer system. One of the services it provides to the application is data access. The operating system also monitors and responds to user actions and the environment. It organizes and controls hardware components and manages the allocation of hardware resources. It provides basic security for the access and usage of all managed resources. An operating system also performs basic storage management tasks while managing other underlying components, such as the file system, volume manager, and device drivers.



A device driver is special software that permits the operating system to interact with a specific device, such as a printer, a mouse, or a hard drive. A device driver enables the operating system to recognize the device and to use a standard interface (provided as an application programming interface, or API) to access and control devices. Device drivers are hardware dependent and operating system specific.

A file system organizes data in a structured hierarchical manner via the use of directories, which are containers for storing pointers to multiple files. Apart from the files and directories, the file system also includes a number of other related records, which are collectively called the metadata. A file system should be mounted before it can be used.



Logical Volume Managers (LVMs) introduce a logical layer between the operating system and the physical storage. LVMs have the ability to define logical storage structures that can span multiple physical devices. The logical storage structures appear contiguous to the operating system and applications.

The fact that logical storage structures can span multiple physical devices provides flexibility and additional functionality including dynamic extension of file systems, host based mirroring and host based striping. The Logical Volume Manager provides a set of operating system commands, library subroutines, and other tools that enable the creation and control of logical storage.



A Volume Group is created by grouping together one or more Physical Volumes. Physical Volumes can be added or removed from a Volume Group dynamically. It cannot be shared between Volume Groups, the entire Physical Volume becomes part of a Volume Group. Each Physical Volume is partitioned into equal-sized data blocks. The size of a Logical Volume is based on a multiple of the equal-sized data blocks. The Volume Group is handled as a single unit by the LVM, a Volume Group as a whole can be activated or deactivated. A Volume Group would typically contain related information. For example, each host would have a Volume Group which holds all the OS data, while applications would be on separate Volume Groups.

Logical Volumes are created within a given Volume Group. A Logical Volume can be thought of as a virtual disk partition, while the Volume Group itself can be thought of as a disk. A Volume Group can have a number of Logical Volumes. Logical Volumes (LV) form the basis of logical storage. They contain logically contiguous data blocks (or logical partitions) within the volume group. Each logical partition is mapped to at least one physical partition on a physical volume within the Volume Group. The OS treats a LV like a physical device and accesses it via device special files (character or block). A Logical Volume can only belong to one Volume Group. However, a Volume Group can have multiple LVs and can span multiple physical volumes. It can be made up of physical disk blocks that are not physically contiguous but appear as a series of contiguous data blocks to the OS. LV can contain a file system or be used directly.

Notes:

There is a one-to-one relationship between LV and a File System.

Under normal circumstances, there is a one-to-one mapping between a logical and physical Partition. A one-to-many mapping between a logical and physical partition leads to mirroring of Logical Volumes.



Disk partitioning improves the flexibility and utilization of HDDs. In partitioning, an HDD is divided into logical containers called logical volumes (LVs). For example, a large physical drive can be partitioned into multiple LVs to maintain data according to the file system’s and applications’ requirements. The partitions are created from groups of contiguous cylinders when the hard disk is initially set up on the host. The host’s file system accesses the partitions without any knowledge of partitioning and the physical structure of the disk.

Concatenation is the process of grouping several smaller physical drives and presenting them to the host as one logical drive.



I/O devices enable sending and receiving data to and from a host.

Communication between various devices and the host takes place in the following way:

User to host communications: Handled by basic I/O devices, such as the keyboard, mouse, and monitor. These devices enable users to enter data and view the results of operations.

Host to host communications: Enabled using devices such as a Network Interface Card (NIC) or modem.

Host to storage device communications: Handled by a Host Bus Adaptor (HBA).



Connectivity refers to the interconnection between hosts or between a host and any other peripheral devices, such as printers or storage devices. The discussion here focuses on the connectivity between the host and the storage device. The components of connectivity in a storage system environment can be classified as physical and logical. The physical components are the hardware elements that connect the host to storage and the logical components of connectivity are the protocols used for communication between the host and storage.

The three physical components of connectivity between the host and storage are Bus, Port, and Cable.

HBA is an application-specific integrated circuit (ASIC) board that performs I/O interface functions between the host and storage or storage area network, relieving the CPU from additional I/O processing workload. HBAs also provide connectivity outlets known as ports. A host may have multiple HBAs. A Bus is a collection of paths that facilitates data transmission from one part of a computer to another.



A protocol is a defined format, in this case, for communication between hardware or software components. Communication protocols are defined for systems and components that are:

• Tightly connected entities – such as central processor to RAM, or storage buffers to controllers – use standard BUS technology (e.g. System bus or I/O – Local Bus).

• Directly attached entities or devices connected at moderate distances – such as host to printer or host to storage.

• Network connected entities – such as networked hosts, Network Attached Storage (NAS) or Storage Area Networks (SAN).

The popular interface protocol used for the local bus to connect to a peripheral device is Peripheral Component Interconnect (PCI). The interface protocols that connect to disk systems are Integrated Device Electronics/Advanced Technology Attachment (IDE/ATA) and Small Computer System Interface (SCSI).



PCI is a specification that standardizes how PCI expansion cards, such as network cards or modems, exchange information with the CPU. PCI provides the interconnection between the CPU and attached devices. The plug-and-play functionality of PCI enables the host to easily recognize and configure new cards and devices. The width of a PCI bus can be 32 bits or 64 bits. A 32-bit PCI bus can provide a throughput of 133 MB/s. PCI Express is an enhanced version of PCI bus with considerably higher throughput and clock speed.



An Integrated Device Electronics/Advanced Technology Attachment (IDE/ATA) disk supports the IDE protocol. The term IDE/ATA conveys the dual-naming conventions for various generations and variants of this interface. The IDE component in IDE/ATA provides the specification for the controllers connected to the computer’s motherboard for communicating with the device attached. The ATA component is the interface for connecting storage devices, such as CD-ROMs, floppy disk drives, and HDDs, to the motherboard. IDE/ATA has a variety of standards and names, such as ATA, ATA/ATAPI, EIDE, ATA-2, Fast ATA, ATA-3, Ultra ATA, and Ultra DMA. The Ultra DMA/133 version of ATA supports a throughput of 133 MB per second. In a master-slave configuration, an ATA interface supports two storage devices per connector. However, if the performance of the drive is important, sharing a port between two devices is not recommended. A 40- pin connector is used to connect ATA disks to the motherboard, and a 34-pin connector is used to connect floppy disk drives to the motherboard. An IDE/ATA disk offers excellent performance at low cost, making it a popular and commonly used hard disk.

A SATA (Serial ATA) is a serial version of the IDE/ATA specification. SATA is a disk-interface technology that was developed by a group of the industry’s leading vendors with the aim of replacing parallel ATA. A SATA provides point-to-point connectivity and revision 3.0 enables data transfer at a speed up to 6 Gb/s. A SATA bus directly connects each storage device to the host through a dedicated link, making use of low-voltage differential signaling (LVDS). LVDS is an electrical signaling system that can provide high-speed connectivity over low-cost, twisted-pair copper cables. For data transfer, a SATA bus uses LVDS with a voltage of 250 mV. A SATA bus uses a small 7-pin connector and a thin cable for connectivity. A SATA port uses 4 signal pins, which improves its pin efficiency compared to the parallel ATA that uses 26 signal pins, for connecting an 80-conductor ribbon cable to a 40-pin header connector. SATA devices are hot-pluggable, and permit single-device connectivity. Connecting multiple SATA drives to a host requires multiple ports to be present on the host.



SCSI is available in a variety of interfaces.

Serial Attached SCSI (SAS) addresses the scalability, performance, reliability, and manageability requirements of a data center while leveraging a common electrical and physical connection interface with SATA. SAS uses SCSI commands for communication and is pin compatible with SATA. SAS supports data transfer rates of 6 Gb/s. It supports dual porting, full-duplex, device addressing, and uses a simplified protocol to minimize interoperability issues between controllers and drives. It also enables connectivity to multiple devices through expanders and is commonly preferred over SCSI in high-end servers for faster disk access.

Parallel SCSI (referred to as SCSI) is one of the oldest and most popular forms of storage interface used in hosts. SCSI is a set of standards used for connecting a peripheral device to a computer and transferring data between them. Often, SCSI is used to connect HDDs and tapes to a host. SCSI can also connect a wide variety of other devices such as scanners and printers. Communication between the hosts and the storage devices uses the SCSI command set. Since its inception, SCSI has undergone rapid revisions, resulting in continuous performance improvements. The oldest SCSI variant, called SCSI-1 provided data transfer rates of 5 MB/s; SCSI Ultra 320 provides data transfer speeds of 320 MB/s.



Data created by individuals or businesses must be stored so that it is easily accessible for further processing. In a computing environment, devices designed for storing data are termed storage devices or simply storage. The type of storage used varies based on the type of data and the rate at which it is created and used. Devices such as memory in a cell phone or digital camera, DVDs, CD-ROMs, and hard disks in personal computers are examples of storage devices. Businesses have several options available for storing data including internal hard disks, external disk arrays and tapes.

Historically, organizations had centralized computers (mainframe) and information storage devices (tape reels and disk packs) in their data center. The evolution of open systems and the affordability and ease of deployment that they offer made it possible for business units/departments to have their own servers and storage. In earlier implementations of open systems, the storage was typically internal to the server.

The proliferation of departmental servers in an enterprise resulted in unprotected, unmanaged, fragmented islands of information and increased operating costs. Originally, there were very limited policies and processes for managing these servers and the data created. To overcome these challenges, storage technology evolved from non-intelligent internal storage to intelligent networked storage.



The storage device is the most important component in the storage system environment. A storage device uses magnetic or solid state media.

Tapes are a popular storage media used for backup because of their relatively low cost. However, tape has various limitations; data is stored on the tape linearly along the length of the tape. Search and retrieval of data is done sequentially, invariably taking several seconds to access the data. As a result, random data access is slow and time consuming. This limits tapes as a viable option for applications that require real-time, rapid access to data. In a shared computing environment, data stored on tape cannot be accessed by multiple applications simultaneously, restricting its use to one application at a time. On a tape drive, the read/write head touches the tape surface, so the tape degrades or wears out after repeated use. The storage and retrieval requirements of data from tape and the overhead associated with managing tape media are significant. Even with all these limitations, tape is not yet obsolete.

Optical disk storage is popular in small, single-user computing environments. Optical disks have limited capacity and speed, which limits the use of optical media as a business data storage solution. The capability to write once and read many (WORM) is one advantage of optical disk storage. A CD-ROM is an example of a WORM device. Optical disks, to some degree, guarantee that the content has not been altered, so they can be used as low-cost alternatives for long-term storage of relatively small amounts of fixed content that will not change after it is created. Collections of optical disks in an array, called jukeboxes, are still used as a fixed-content storage solution.

Disk drives are the most popular storage medium used in modern computers for storing and accessing data for performance-intensive, online applications. Disks support rapid access to random data locations. This means that data can be written or retrieved quickly for a large number of simultaneous users or applications. In addition, disks have a large capacity. Disk storage arrays are configured with multiple disks to provide increased capacity and enhanced performance.



A file system block is the smallest “container” of physical disk space allocated for data. Each file system block is a contiguous area on the physical disk. The block size of a file system is fixed at the time of its creation. File system size depends on block size and the total number of blocks of data stored. A file can span multiple file system blocks because most files are larger than the predefined block size of the file system. File system blocks cease to be contiguous (i.e., become fragmented) when new blocks are added or deleted. Over time, as files grow larger, the file system becomes increasingly fragmented.

Shown here is the process of mapping user files to the disk storage subsystem with an LVM:

1. Files are created and managed by users and applications.

2. These files reside in the file systems.

3. The file systems are then mapped to units of data, or file system blocks.

4. The file system blocks are mapped to logical extents.

5. These in turn are mapped to disk physical extents either by the operating system or by the LVM. These physical extents are managed by the disk storage subsystem.

If there is no LVM, then there are no logical extents. Without LVM, file system blocks are directly mapped to disk sectors.



This lesson covered the major components of storage systems including hosts, connectivity and storage. We will look at the logical and physical components of hosts, connectivity options, types of storage and how files are moved to and from storage.



This lesson covers the purpose of RAID, RAID array components and RAID levels, defined on the basis of striping, mirroring and parity techniques.



Mechanical hard disk drives are susceptible to failures due to mechanical wear and tear and other environmental factors that may result in data loss. An HDD has a projected life expectancy before it fails. Mean Time Between Failure (MTBF) measures (in hours) the average life expectancy of an HDD. Today, data centers deploy thousands of HDDs in their storage infrastructures. Greater the number of HDDs in a storage array, greater the probability of a disk failure in the array.

For example, consider a storage array of 1000 HDDs, each with an MTBF of 750,000 hours. The MTBF of this collection is therefore 750,000/1000 or 750 hours. This means that a HDD in this array is likely to fail at least once in 750 hours.

In 1987, Patterson, Gibson, and Katz at the University of California, Berkeley, published a paper titled “A Case for Redundant Arrays of Inexpensive Disks (RAID).” This paper described the use of small-capacity, inexpensive disk drives as an alternative to large-capacity drives common on mainframe computers. The term RAID has been redefined to refer to independent disks, to reflect advances in the storage technology. RAID storage has now grown from an academic concept to an industry standard.

RAID is an enabling technology that leverages multiple disks as part of a set, which provides data protection against HDD failures. In general, RAID implementations also improve the I/O performance of storage systems by storing data across multiple HDDs.



A RAID array is an enclosure that contains a number of HDDs and the supporting hardware and software to implement RAID. HDDs inside a RAID array are usually contained in smaller sub-enclosures. These sub-enclosures, or physical arrays, hold a fixed number of HDDs, and may also include other supporting hardware, such as power supplies. A subset of disks within a RAID array can be grouped to form logical associations called logical arrays, also known as a RAID set or a RAID group.

Logical arrays are comprised of logical volumes (LV). The operating system recognizes the LVs as if they are physical HDDs managed by the RAID controller. The number of HDDs in a logical array depends on the RAID level used. Configurations could have a logical array with multiple physical arrays or a physical array with multiple logical arrays.

Note: There can be a mix of mechanical and flash drives in the same array, though they would not be used in the same disk group. For example, multiple disk groups or pools based on type can be defined within the same array.



In hardware RAID implementations, a specialized hardware controller is implemented either on the host or on the array. These implementations vary in the way the storage array interacts with the host.

Controller card RAID is host-based hardware RAID implementation in which a specialized RAID controller is installed in the host and HDDs are connected to it. The RAID Controller interacts with the hard disks using a PCI bus. Manufacturers also integrate RAID controllers on motherboards. This integration reduces the overall cost of the system, but does not provide the flexibility required for high-end storage systems.

The external RAID controller is an array-based hardware RAID. It acts as an interface between the host and disks. It presents storage volumes to the host, which manage the drives using the supported protocol.

Software RAID uses host-based software to provide RAID functions. It is implemented at the operating-system level and does not use a dedicated hardware controller to manage the RAID array. Software RAID implementations offer cost and simplicity benefits when compared with hardware RAID. However, they have do not support all RAID levels. Software RAID affects overall system performance. This is due to the additional CPU cycles required to perform RAID calculations. The performance impact is more pronounced for complex implementations of RAID. Software RAID is tied to the host operating system hence upgrades to software RAID or to the operating system should be validated for compatibility. This leads to inflexibility in the data processing environment.



RAID levels are defined on the basis of striping, mirroring, and parity techniques. These techniques determine the data availability and performance characteristics of an array. Some RAID arrays use one technique, whereas others use a combination of techniques. Application performance and data availability requirements determine the RAID level selection.



A RAID set is a group of disks. Within each disk, a predefined number of contiguously addressable disk blocks are defined as strips. The set of aligned strips that spans across all the disks within the RAID set is called a stripe. The slide shows a striped RAID set.

Strip size (also called stripe depth) describes the number of blocks in a strip, and is the maximum amount of data that can be written to or read from a single HDD in the set before the next HDD is accessed, assuming that the accessed data starts at the beginning of the strip. Note that all strips in a stripe have the same number of blocks, and decreasing strip size means that data is broken into smaller pieces when spread across the disks.

Stripe size is a multiple of strip size by the number of HDDs in the RAID set. Stripe width refers to the number of data strips in a stripe.

Striped RAID does not protect data unless parity or mirroring is used. However, striping may significantly improve I/O performance. Depending on the type of RAID implementation, the RAID controller can be configured to access data across multiple HDDs simultaneously.

The slide shows striping in which a stripe of 192 KB is distributed over three disks with a strip size of 64 KB each. The controller writes 64 KB of data on each of the three disks, totaling 192 KB.



In a RAID 0 configuration, data is striped across the HDDs in a RAID set. It utilizes the full storage capacity by distributing strips of data over multiple HDDs in a RAID set. To read data, all the strips are put back together by the controller. The stripe size is specified at a host level for software RAID and is vendor specific for hardware RAID. When the number of drives in the array increases, performance improves because more data can be read or written simultaneously.

RAID 0 is used in applications that need high I/O throughput. However, if these applications require high availability, RAID 0 does not provide data protection and availability in the event of drive failures.



Mirroring is a technique whereby data is stored on two different HDDs, yielding two copies of data. In the event of one HDD failure, the data is intact on the surviving HDD and the controller continues to service the host’s data requests from the surviving disk of a mirrored pair.

When the failed disk is replaced with a new disk, the controller copies the data from the surviving disk of the mirrored pair. This activity is transparent to the host.

In addition to providing complete data redundancy, mirroring enables faster recovery from disk failure. However, disk mirroring provides only data protection and is not a substitute for data backup. Mirroring constantly captures changes in the data, whereas a backup captures point-in-time images of data.

Mirroring involves duplication of data — the amount of storage capacity needed is twice the amount of data being stored. Therefore, mirroring is considered expensive and is preferred for mission-critical applications that cannot afford data loss. Mirroring improves read performance because read requests can be serviced by both disks. However, write performance deteriorates, as each write request manifests as two writes on the HDDs. In other words, mirroring does not deliver the same levels of write performance as a striped RAID. Mirroring can be implemented with striped RAID by mirroring entire stripes of disks to stripes on other disks. This is known as nested RAID.



Most data centers require data redundancy and performance from their RAID arrays. RAID 0+1 and RAID 1+0 combine the performance benefits of RAID 0 with the redundancy benefits of RAID 1. They use striping and mirroring techniques and combine their benefits. These types of RAID require an even number of disks, the minimum being four.

RAID 0+1 is also called mirrored stripe and is also know as RAID 01 or RAID 0/1. The basic element of RAID 0+1 is a stripe. This means that the process of striping data across HDDs is performed initially and then the entire stripe is mirrored. In the event of a single drive failure, the entire stripe set is faulted. Normal processing can continue with the mirrors. However, rebuild of the failed drive will involve copying data from the mirror to the entire stripe set. This will result in increased rebuild times as compared to RAID 1+0 solution. This makes RAID 0+1 implementation less common than RAID 1+0.



RAID 1+0 is also known as RAID 10 (Ten) or RAID 1/0. RAID 1+0 is also called striped mirror. The basic element of RAID 1+0 is a mirrored pair, which means that data is first mirrored and then both copies of data are striped across multiple HDDs in a RAID set. In the event of a drive failure, normal processing can continue with the surviving mirror. When replacing a failed drive, only the mirror is rebuilt. In other words, the disk array controller uses the surviving drive in the mirrored pair for data recovery and continuous operation. This results in faster rebuild times for RAID 1+0 and makes it a more common solution than RAID 0+1. Data from the surviving disk is copied to the replacement disk. RAID 1+0 performs well for workloads that use small, random, write intensive I/O.

Some applications that benefit from RAID 1+0 include the following:

• High transaction rate Online Transaction Processing (OLTP)

• Large messaging installations

• Database applications that require high I/O rate, random access, and high availability

Note that under normal operating conditions both RAID 0+1 and RAID 1+0 provide the same benefits. These solutions are still aimed at protecting against a single drive failure and not against multiple drive failures



Parity is a method of protecting striped data from HDD failure without the cost of mirroring. An additional HDD is added to the stripe width to hold parity, a mathematical construct that allows re-creation of the missing data. Parity is a redundancy check that ensures full protection of data without maintaining a full set of duplicate data. Parity RAID is less expensive than mirroring because parity overhead is only a fraction of the total capacity.

Parity information can be stored on separate, dedicated HDDs or distributed across all the drives in a RAID set. The slide shows a parity RAID. The first four disks contain the data. The fifth disk stores the parity information, which in this case is the sum of the elements in each row. Think of parity as the sum of the data on the other disks in the RAID set. Each time data is updated, the parity is updated as well, so that it always reflects the current sum of the data on the other disks

In the slide, the computation of parity is represented as a simple arithmetic operation on the data. However, parity calculation is a bitwise XOR operation. Calculation of parity is a function of the RAID controller. While parity is calculated on a per stripe basis, the diagram omits this detail for the sake of simplification. Now, if one of the disk fails, the value of its data is calculated by using the parity information and the data on the surviving disks. (In this case by subtracting the sum of the rest of the elements from the parity value). If the parity disk fails, the value of its data is calculated by using the data disks. Parity will only need to be recalculated, and saved, when the failed disk is replaced with a new disk.

Compared to mirroring, parity implementation considerably reduces the cost associated with data protection. However, there are some disadvantages of using parity. Parity information is generated from data on the data disk. Therefore, parity is recalculated every time there is a change in data. This recalculation is time-consuming and affects the performance of the RAID controller.

In the event of a disk failure, each request for data from the failed disk requires recalculation of data before sent to the host. This recalculation is time-consuming, and decreases the performance of the RAID set. Hot spare drives provide a way to minimize the disruption caused by a disk failure.



RAID 5 is a very versatile RAID implementation. In RAID 5 the drives (strips) are independently accessible. It is similar to RAID 4 because it uses striping and the drives (strips) are independently accessible. The difference between RAID 4 and RAID 5 is the parity location. In RAID 4, parity is written to a dedicated drive, creating a write bottleneck for the parity disk.

In RAID 5, parity is distributed across all disks. The distribution of parity in RAID 5 overcomes the write bottleneck. The figure on the slide illustrates the RAID 5 implementation. In RAID 5, write I/O operations suffer performance degradation because of the write penalty that manifests with a parity RAID implementation. The performance degradation also occurs during recovery and reconstruction operations in the event of a disk failure. In addition, multiple disk failures within the array may result in data loss.

RAID 5 is preferred for messaging, data mining, medium-performance media serving, and relational database management system (RDBMS) implementations in which database administrators (DBAs) optimize data access.



RAID 6 works the same way as RAID 5 except that RAID 6 includes a second parity element to enable survival in the event of the failure of two disks in a RAID group. Therefore, a RAID 6 implementation requires at least four disks. RAID 6 distributes the parity across all the disks. The write penalty in RAID 6 is more than that in RAID 5; therefore, RAID 5 writes perform better than RAID 6. The rebuild operation in RAID 6 may take longer than that in RAID 5 due to the presence of two parity sets.



The chart summarizes the discussion of RAID levels included in this lesson.



A hot spare refers to a spare HDD in a RAID array that temporarily replaces a failed HDD of a RAID set. A hot spare takes the identity of the failed HDD in the array. One of the following methods of data recovery is performed depending on the RAID implementation:

• If parity RAID is used, then the data is rebuilt onto the hot spare from the parity and the data on the surviving HDDs in the RAID set.

• If mirroring is used, then the data from the surviving mirror is used to copy the data.

• When the failed HDD is replaced with a new HDD, one of the following takes place:

• The hot spare replaces the new HDD permanently. This means that it is no longer a hot spare, and a new hot spare must be configured on the array.

• When a new HDD is added to the system, data from the hot spare is copied to it. The hot spare returns to its idle state, ready to replace the next failed drive.

A hot spare should be large enough to accommodate data from a failed drive. Some systems implement multiple hot spares to improve data availability. A hot spare can be configured as automatic or user initiated, which specifies how it will be used in the event of disk failure. In an automatic configuration, when the recoverable error rates for a disk exceed a predetermined threshold, the disk subsystem tries to copy data from the failing disk to the hot spare automatically. If this task is completed before the damaged disk fails, then the subsystem switches to the hot spare and marks the failing disk as unusable. Otherwise, it uses parity or the mirrored disk to recover the data. In the case of a user-initiated configuration, the administrator has control of the rebuild process. For example, the rebuild could occur overnight to prevent any degradation of system performance. However, the system is vulnerable to another failure if a hot spare is unavailable.



This lesson covered the purpose of RAID, RAID array components and RAID levels, defined on the basis of striping, mirroring and parity techniques.



This lesson defines intelligent storage systems and discusses the benefits of intelligent storage systems versus a collection of disks in an array. Then, we will look at the components of an intelligent storage system, including the front end, cache, back end and storage.



Business-critical applications require high levels of performance, availability, security, and scalability. A hard disk drive is a core element of storage that governs the performance of any storage system. Some of the older disk array technologies could not overcome performance constraints due to the limitations of a hard disk and its mechanical components. RAID technology made an important contribution to enhancing storage performance and reliability, but hard disk drives even with a RAID implementation could not meet performance requirements of today’s applications. With advancements in technology, a new breed of storage solutions known as an intelligent storage system has evolved. The intelligent storage systems detailed in this chapter are the feature-rich RAID arrays that provide highly optimized I/O processing capabilities. These arrays have an operating environment that controls the management, allocation, and utilization of storage resources. These storage systems are configured with large amounts of memory called cache and use sophisticated algorithms to meet the I/O requirements of performance sensitive applications.



Intelligent storage systems, a collection of disks in an array, and RAID arrays, all provide increased data storage capacity. However, intelligent storage systems provide more benefits, as listed in the slide.



An intelligent storage system consists of four key components: front end, cache, back end, and physical disks. The slide illustrates these components and their interconnections. An I/O request received from the host at the front-end port is processed through cache and the back end to enable storage and retrieval of data from the physical disk. A read request can be serviced directly from cache if the requested data is found in cache.



The front end provides the interface between the storage system and the host. It consists of two components: front-end ports and front-end controllers. The front-end ports enable hosts to connect to the intelligent storage system. Each front-end port has processing logic that executes the appropriate transport protocol, such as SCSI, Fibre Channel, or iSCSI, for storage connections. Redundant ports are provided on the front end for high availability. Front-end controllers route data to and from cache via the internal data bus. When cache receives write data, the controller sends an acknowledgment message back to the host. Controllers optimize I/O processing by using command queuing algorithms.



Cache is an important component that enhances the I/O performance in an intelligent storage system. Cache is semiconductor memory where data is placed temporarily to reduce the time required to service I/O requests from the host. Cache improves storage system performance by isolating hosts from the mechanical delays associated with physical disks, which are the slowest components of an intelligent storage system.

Accessing data from a physical disk usually takes a few milliseconds because of seek times and rotational latency. If a disk has to be accessed by the host for every I/O operation, requests are queued, which results in a delayed response. Accessing data from cache takes less than a millisecond. Write data is placed in cache and then written to disk. After the data is securely placed in cache, the host is acknowledged immediately. Write operations with cache provide performance advantages over writing directly to disks. When an I/O is written to cache and acknowledged, it is completed in far less time (from the host’s perspective) than it would take to write directly to disk. Sequential writes also offer opportunities for optimization because many smaller writes can be coalesced for larger transfers to disk drives with the use of cache.



The back end provides an interface between cache and the physical disks. It consists of two components: back-end ports and back-end controllers. The back end controls data transfers between cache and the physical disks. From cache, data is sent to the back end and then routed to the destination disk. Physical disks are connected to ports on the back end. The back end controller communicates with the disks when performing reads and writes and also provides additional, but limited, temporary data storage. The algorithms implemented on back-end controllers provide error detection and correction, along with RAID functionality. For high data protection and availability, storage systems are configured with dual controllers with multiple ports. Such configurations provide an alternate path to physical disks in the event of a controller or port failure. This reliability is further enhanced if the disks are also dual-ported. In that case, each disk port can connect to a separate controller. Multiple controllers also facilitate load balancing.



A physical disk stores data persistently. Disks are connected to the back-end with either SCSI or a Fibre Channel interface. An intelligent storage system enables the use of a mixture of SCSI or Fibre Channel drives and IDE/ATA drives.



Physical drives or groups of RAID protected drives can be logically split into volumes known as logical volumes, commonly referred to as Logical Unit Numbers (LUNs). The use of LUNs improves disk utilization. For example, without the use of LUNs, a host requiring only 200 GB could be allocated an entire 1TB physical disk. Using LUNs, only the required 200 GB would be allocated to the host, allowing the remaining 800 GB to be allocated to other hosts.

In the case of RAID protected drives, these logical units are slices of RAID sets and are spread across all the physical disks belonging to that set. The logical units can also be seen as a logical partition of a RAID set that is presented to a host as a physical disk. For example, the slide shows a RAID set consisting of five disks that have been sliced, or partitioned, into several LUNs. LUNs 0 and 1 are shown in the figure.

Note how a portion of each LUN resides on each physical disk in the RAID set. LUNs 0 and 1 are presented to hosts 1 and 2, respectively, as physical volumes for storing and retrieving data. Usable capacity of the physical volumes is determined by the RAID type of the RAID set. A host will see a LUN as if it were a single disk device. The host is not aware that this LUN is only a part of a larger physical drive. The host assigns logical device names to the LUNs; the naming conventions vary by platform/OS.

The capacity of a LUN can be expanded by aggregating other LUNs with it. The result of this aggregation is a larger capacity LUN, known as a meta-LUN. The mapping of LUNs to their physical location on the drives is managed by the operating environment of an intelligent storage system.



Virtual provisioning provides more efficient allocation of storage to hosts. Virtual provisioning enables over subscription whereby more capacity is presented to the hosts than is actually available on the storage array. LUNs created using virtual provisioning are called thin LUNs to distinguish them from traditional LUNs. Thin LUNs do not require physical storage to be completely allocated to them at the time they are created and presented to a host. Physical storage is allocated to the host "on demand" from a shared pool of physical capacity.

A shared pool consists of physical disks. A shared pool in virtual provisioning is analogous to a RAID group, which is a collection of drives on which LUNs are created. Similar to a RAID group, a shared pool supports a single RAID protection level. However, unlike a RAID group, a shared pool might contain large numbers of drives. Shared pools can be homogeneous (containing a single drive type) or heterogeneous (containing mixed drive types, such as flash, FC, SAS, and SATA drives). Multiple shared pools can be created within a storage array, and a shared pool may be shared by multiple thin LUNs. Both shared pools and thin LUNs can be expanded non-disruptively as the storage requirements of the hosts grow.



LUN masking is a process that provides data access control by defining which LUNs a host can access. LUN masking function is typically implemented at the front end controller. This ensures that volume access by servers is controlled appropriately, preventing unauthorized or accidental use in a distributed environment. For example, consider a storage array with two LUNs that store data of the sales and finance departments. Without LUN masking, both departments can easily see and modify each other’s data, posing a high risk to data integrity and security. With LUN masking, LUNs are accessible only to the designated hosts.



Highlights of storage technology and architecture evolution include:

• Direct-attached storage (DAS): This type of storage connects directly to a server (host) or a group of servers in a cluster. Storage can be either internal or external to the server. External DAS alleviates the challenges of limited internal storage capacity.

• Storage area network (SAN): This is a dedicated, high-performance Fibre Channel (FC) network to facilitate block-level communication between servers and storage. Storage is partitioned and assigned to a server for accessing its data. SAN offers scalability, availability, performance, and cost benefits compared to DAS.

• Network-attached storage (NAS): This is dedicated storage for file serving applications. Unlike a SAN, it connects to an existing communication network (LAN) and provides file access to heterogeneous clients. Because it is purposely built for providing storage to file server applications, it offers higher scalability, availability, performance, and cost benefits compared to general purpose file servers.

• Internet Protocol SAN (IP-SAN): One of the latest evolutions in storage architecture, IP-SAN is a convergence of technologies used in SAN and NAS. IP-SAN provides block-level communication across a local or wide area network (LAN or WAN), resulting in greater consolidation and availability of data.

Storage technology and architecture continue to evolve, which enables organizations to consolidate, protect, optimize, and leverage their data to achieve the highest return on information assets. In the following lessons, we will take a look at each of the above technologies.



In this lesson, we defined intelligent storage systems and discussed the benefits of intelligent storage systems versus a collection of disks in an array. Then, we looked at the components of an intelligent storage system, including the front end, cache, back end and storage.



This lesson covers direct-attached storage, its architecture and benefits.



Direct-Attached Storage (DAS) is an architecture where storage connects directly to servers without a network in between. Applications access data from DAS using block-level access protocols. The internal HDD of a host, tape libraries, and directly connected external HDD packs are some examples of DAS. DAS is classified as internal or external, based on the location of the storage device with respect to the host.

In internal DAS architectures, the storage device is internally connected to the host by a serial or parallel bus. The physical bus has distance limitations and can only be sustained over a shorter distance for high-speed connectivity. In addition, most internal buses can support only a limited number of devices, and they occupy a large amount of space inside the host, making maintenance of other components difficult.

In external DAS architectures, the server connects directly to the external storage device. In most cases, communication between the host and the storage device takes place over SCSI or FC protocol. Compared to internal DAS, an external DAS overcomes the distance and device count limitations and provides centralized management of storage devices.



The host and the storage device in DAS communicate with each other by using predefined protocols such as IDE/ATA, SATA, SAS, SCSI, and FC. These protocols are implemented on the HDD controller. Therefore, a storage device is also known by the name of the protocol it supports.



DAS requires a relatively lower initial investment than storage networking. DAS configuration is simple and can be deployed easily and rapidly. Setup is managed using host-based tools, such as the host OS, which makes storage management tasks easy for small and medium enterprises. DAS is the simplest solution when compared to other storage networking models and requires fewer management tasks, and less hardware and software elements to set up and operate.



This lesson covered direct-attached storage, its architecture and benefits.



This lesson introduces SCSI, its architecture and the components of SCSI addressing.



The SCSI-3 architecture defines and categorizes various SCSI-3 standards and requirements for SCSI-3 implementations. (For more information, see Technical Committee T10 “SCSI Architecture Model-3 (SAM-3)” document from www.t10.org.) The SCSI-3 architecture was approved and published as the standard X.3.270-1996 by the ANSI. This architecture helps developers, hardware designers, and users to understand and effectively utilize SCSI. The three major components of a SCSI architectural model are as follows:

SCSI-3 command protocol:

This consists of primary commands that are common to all devices as well as device-specific commands that are unique to a given class of devices.

Transport layer protocols:

These are a standard set of rules by which devices communicate and share information.

Physical layer interconnects:

These are interface details such as electrical signaling methods and data transfer modes. Common access methods are the ANSI software interfaces for SCSI devices. The slide shows the SCSI-3 standards architecture with interrelated groups of other standards within SCSI-3.



SCSI-3 architecture derives its base from the client-server relationship, in which a client directs a service request to a server, which then fulfills the client’s request. In a SCSI environment, an initiator-target concept represents the client-server model. In a SCSI-3 client-server model, a particular SCSI device acts as a SCSI target device, a SCSI initiator device, or a SCSI target/initiator device. Each device performs the following functions:

SCSI initiator device: Issues a command to the SCSI target device to perform a task. A SCSI host adaptor is an example of an initiator.

SCSI target device: Executes commands to perform the task received from a SCSI initiator. Typically a SCSI peripheral device acts as a target device. However, in certain implementations, the host adaptor can also be a target device.

The slide depicts the SCSI-3 client-server model in which a SCSI initiator, or a client, sends a request to a SCSI target, or a server. The target performs the tasks requested and sends the output to the initiator using the protocol service interface. A SCSI target device contains one or more logical units. A logical unit is an object that implements one of the device functional models as described in the SCSI command standards. The logical unit processes the commands sent by a SCSI initiator. A logical unit has two components, a device server and a task manager. The device server addresses client requests and the task manager performs management functions. The SCSI initiator device is comprised of an application client and task management function which initiates device service and task management requests.



Each device service request contains a Command Descriptor Block (CDB). The CDB defines the command to be executed and lists command-specific inputs and other parameters specifying how to process the command.

SCSI ports are the physical connectors that the SCSI cable plugs into for communication with a SCSI device. A SCSI device may contain target ports, initiator ports, target/initiator ports, or a target with multiple ports. Based on the port combinations, a SCSI device can be classified as an initiator model, a target model, a combined model, or a target model with multiple ports. In an initiator model, the SCSI initiator device has only initiator ports. Therefore, the application client can only initiate requests to the service delivery subsystem and receive confirmation. This device cannot serve any requests, and therefore does not contain a logical unit. Similarly, a SCSI target device with only a target port can serve requests but cannot initiate them. The SCSI target/initiator device has a target/initiator port that can switch orientations depending on the role it plays while participating in an I/O operation. To cater to service requests from multiple devices, a SCSI device may also have multiple ports of the same orientation.



The SCSI devices are identified by a specific number called a SCSI ID. In narrow SCSI (bus width=8), the devices are numbered 0 through 7; in wide (bus width=16) SCSI, the devices are numbered 0 through 15. These ID numbers set the device priorities on the SCSI bus. In narrow SCSI, 7 has the highest priority and 0 has the lowest priority. In wide SCSI, the device IDs from 8 to 15 have the highest priority, but the entire sequence of wide SCSI IDs has lower priority than narrow SCSI IDs. Therefore, the overall priority sequence for a wide SCSI is 7, 6, 5, 4, 3, 2, 1, 0, 15, 14, 13, 12, 11, 10, 9, and 8. When a device is initialized, SCSI allows for automatic assignment of device IDs on the bus, which prevents two or more devices from using the same SCSI ID.



In the Parallel SCSI Initiator-Target communication, an initiator ID uniquely identifies the initiator and is used as an originating address. This ID is in the range of 0 to 15, with the range 0 to 7 being the most common. A target ID uniquely identifies a target and is used as the address for exchanging commands and status information with initiators. The target ID is in the range of 0 to 15.

SCSI addressing is used to identify hosts and devices. In this addressing, the UNIX naming convention is used to identify a disk and the three identifiers— initiator ID, target ID, and a LUN—in the cn|tn|dn format, which is also referred as ctd addressing. Here, Cn is the initiator ID, commonly referred to as the controller ID; Tn is the target ID of the device, such as t0, t1, t2, and so on; and Dn is the device number reflecting the actual address of the device unit, such as d0, d1, and d2. A LUN identifies a specific logical unit in a target. The implementation of SCSI addressing may differ from one vendor to another. Figure shows ctd addressing in the SCSI architecture.



This lesson introduced SCSI, its architecture and the components of SCSI addressing.



This lesson introduces Storage Area Networks, including a definition of SAN and its components, and an overview of fibre channel technology and interconnectivity options.



A storage area network (SAN) is a dedicated high speed network for block level data access. It carries data between servers (also known as hosts) and storage devices through fibre channel switches. A SAN enables storage consolidation and allows storage to be shared across multiple servers. It enables organizations to connect geographically dispersed servers and storage. A SAN provides the physical communication infrastructure and enables secure and robust communication between host and storage devices. The SAN management interface organizes and manages storage elements and hosts.



Fibre Channel (FC) architecture forms the fundamental construct of the SAN infrastructure. Fibre Channel is a high-speed network technology that runs on high-speed optical fiber cables (preferred for front-end SAN connectivity) and serial copper cables (preferred for back-end disk connectivity). The FC technology was created to meet the demand for increased speeds of data transfer among computers, servers, and mass storage subsystems.

Higher data transmission speeds are an important feature of the FC networking technology. FC implementations of 8 GFC (Fibre Channel) offer throughput of 1600 MB/s (raw bit rates of 8 Gb/s), whereas Ultra320 SCSI is available with a throughput of 320 MB/s. The FC architecture is highly scalable and theoretically a single FC network can accommodate approximately 15 million nodes.



A SAN consists of three basic components: servers, network infrastructure, and storage. These components can be further broken down into the following key elements: node ports, cabling, interconnecting devices (such as FC switches or hubs), storage arrays, and SAN management software.

In fibre channel, devices such as hosts, storage and tape libraries are all referred to as nodes. Each node is a source or destination of information for one or more nodes. Each node requires one or more ports to provide a physical interface for communicating with other nodes. These ports are integral components of an HBA and the storage front-end adapters. A port operates in full-duplex data transmission mode with a transmit (Tx) link and a receive (Rx) link.



SAN implementations use optical fiber cabling. Copper can be used for shorter distances for back-end connectivity, as it provides a better signal-to-noise ratio for distances up to 30 meters. Optical fiber cables carry data in the form of light. There are two types of optical cables, multi-mode and single-mode.

Multi-mode fiber (MMF) cable carries multiple beams of light projected at different angles simultaneously onto the core of the cable. Based on the bandwidth, multimode fibers are classified as OM1 (62.5μm), OM2 (50μm) and laser optimized OM3 (50μm). In an MMF transmission, multiple light beams traveling inside the cable tend to disperse and collide. This collision weakens the signal strength after it travels a certain distance — a process known as modal dispersion. An MMF cable is usually used for distances of up to 500 meters because of signal degradation (attenuation) due to modal dispersion.

Single-mode fiber (SMF) carries a single ray of light projected at the center of the core. These cables are available in diameters of 7–11 microns; the most common size is 9 microns. In an SMF transmission, a single light beam travels in a straight line through the core of the fiber. The small core and the single light wave limits modal dispersion. Among all types of fibre cables, single-mode provides minimum signal attenuation over maximum distance (up to 10 km). A single-mode cable is used for long- distance cable runs, limited only by the power of the laser at the transmitter and sensitivity of the receiver.



Optical fiber connectors terminate the end of fiber cabling and are used to join optical fibers. A Standard connector (SC) and a Lucent connector (LC) are two commonly used connectors for fiber optic cables. An SC is used for data transmission speeds up to 1 Gb/s, whereas an LC is used for higher speeds. A Straight Tip (ST) is a fiber optic connector with a plug and a socket that is locked with a half-twisted bayonet lock. In the early days of FC deployment, fiber optic cabling predominantly used LC connectors. This connector is often used with Fibre Channel patch panels.

The Small Form-factor Pluggable (SFP) is an optical transceiver used in optical communication. The standard SFP+ transceivers support data rates up to 10 Gb/s. An SFP interfaces a network device motherboard to a networking cable.



Hubs, switches, and directors are the interconnect devices commonly used in SAN.

Hubs are used as communication devices in FC-AL implementations. Hubs physically connect nodes in a logical loop or a physical star topology. All the nodes must share the bandwidth because data travels through all the connection points. Because of availability of low cost and high performance switches, hubs are no longer used in SANs.

Switches are more intelligent than hubs and directly route data from one physical port to another. Therefore, nodes do not share the bandwidth. Instead, each node has a dedicated communication path, resulting in bandwidth aggregation.

Directors have a similar function to that of switches but have higher port count and fault tolerance capabilities. Directors are deployed for data center implementations.



The fundamental purpose of a SAN is to provide host access to storage resources. The large storage capacities offered by modern storage arrays have been exploited in SAN environments for storage consolidation and centralization. SAN implementations complement the standard features of storage arrays by providing high availability and redundancy, improved performance, business continuity, and multiple host connectivity.



The FC architecture supports three basic interconnectivity options: point-to-point, arbitrated loop (FC- AL), and fabric connect.

Point-to-point is the simplest FC configuration — two devices are connected directly to each other. This configuration provides a dedicated connection for data transmission between nodes. However, the point-to-point configuration offers limited connectivity, as only two devices can communicate with each other at a given time. Moreover, it cannot be scaled to accommodate a large number of network devices. Standard DAS uses point-to-point connectivity.



In the FC-AL configuration, devices are attached to a shared loop. FC-AL has the characteristics of a token ring topology and a physical star topology. In FC-AL, each device contends with other devices to perform I/O operations. Devices on the loop must “arbitrate” to gain control of the loop. At any given time, only one device can perform I/O operations on the loop.

As a loop configuration, FC-AL can be implemented without any interconnecting devices by directly connecting one device to another in a ring through cables.

However, FC-AL implementations may also use hubs whereby the arbitrated loop is physically connected in a star topology. The FC-AL configuration has the following limitations in terms of scalability:

• FC-AL shares the bandwidth in the loop. Only one device can perform I/O operations at a time. Because each device in a loop has to wait for its turn to process an I/O request, the speed of data transmission is low in an FC-AL topology.

• FC-AL uses 8-bit addressing. It can support up to 127 devices on a loop.

• Adding or removing a device results in loop re-initialization, which can cause a momentary pause in loop traffic.



Unlike a loop configuration, a Fibre Channel switched fabric (FC-SW) network provides interconnected devices, dedicated bandwidth and scalability. The addition or removal of a device in a switched fabric is minimally disruptive; it does not affect the ongoing traffic between other devices. FC-SW is also referred to as fabric connect. A fabric is a logical space in which all nodes communicate with one another in a network. This virtual space can be created with a switch or a network of switches. Each switch in a fabric contains a unique domain identifier which is part of the fabric’s addressing scheme. In FC-SW, nodes do not share a loop; instead, data is transferred through a dedicated path between the nodes. Each port in a fabric has a unique 24-bit fibre channel address for communication.

FC-SW uses switches that are intelligent devices. They can switch data traffic from an initiator node to a target node directly through switch ports. Frames are routed between source and destination by the fabric.



Ports are the basic building blocks of a fibre channel network. Ports on the switch can be one of the following types:

• N_port: An end point in the fabric. This port is also known as the node port. Typically, it is a host port (HBA) or a storage array port that is connected to a switch in a switched fabric.

• NL_port: A node port that supports the arbitrated loop topology. This port is also known as the node loop port.

• E_port: An FC port that forms the connection between two FC switches. This port is also known as the expansion port. The E_port on an FC switch connects to the E_port of another FC switch in the fabric through a link, which is called an InterSwitch Link (ISL). ISLs are used to transfer host-to storage data as well as the fabric management traffic from one switch to another. ISL is also one of the scaling mechanisms in SAN connectivity.

• F_port: A port on a switch that connects an N_port. It is also known as a fabric port and cannot participate in FC-AL.

• FL_port: A fabric port that participates in FC-AL. This port is connected to the NL_ports on an FC-AL loop. A FL_port also connects a loop to a switch in a switched fabric. As a result, all NL_ports in the loop can participate in FC-SW. This configuration is referred to as a public loop. In contrast, an arbitrated loop without any switches is referred to as a private loop. A private loop contains nodes with NL_ports, and does not contain FL_port.

• G_port: A generic port that can operate as an E_port or an F_port and determines its functionality automatically during initialization.

The slide shows various FC ports located in the fabric.



An ISL (Inter-Switch Link) allows for two or more Fibre Channel switches to be connected together to form a single, but larger, fabric. Expansion ports (E_Ports) on an FC switch provide interswitch link (ISL) connectivity to fabric directors and switches.

ISLs are used to transfer host-to-storage data as well as the fabric management traffic from one switch to another. ISL is also one of the scaling mechanisms in SAN connectivity.

By using inter-switch links, a switched fabric can be expanded to connect hundreds of nodes.



The FC architecture represents true channel/network integration with standard interconnecting devices. Connections in a SAN are accomplished using FC. Traditionally, transmissions from host to storage devices are carried out over channel connections such as a parallel bus. Channel technologies provide high levels of performance with low protocol overheads. Such performance is due to the static nature of channels and the high level of hardware and software integration provided by the channel technologies. However, these technologies suffer from inherent limitations in terms of the number of devices that can be connected and the distance between these devices.

Fibre Channel Protocol (FCP) is the implementation of serial SCSI-3 over an FC network. In the FCP architecture, all external and remote storage devices attached to the SAN appear as local devices to the host operating system. The key advantages of FCP are as follows:

• Sustained transmission bandwidth over long distances.

• Support for a larger number of addressable devices over a network.

• Exhibits the characteristics of channel transport.



Each device in the FC environment is assigned a 64-bit unique identifier called the World Wide Name (WWN). The Fibre Channel environment uses two types of WWNs: World Wide Node Name (WWNN) and World Wide Port Name (WWPN). Unlike an FC address, which is assigned dynamically, a WWN is a static name for each device on an FC network. WWNs are similar to the Media Access Control (MAC) addresses used in IP networking. WWNs are burned into the hardware or assigned through software. Several configuration definitions in a SAN use WWN for identifying storage devices and HBAs. The name server in an FC environment keeps the association of WWNs to the dynamically created FC addresses for nodes. The figure illustrates the WWN structure for an array and the HBA.



Zoning is an FC switch function that enables nodes within the fabric to be logically segmented into groups that can communicate with each other. When a device (host or storage array) logs onto a fabric, it is registered with the name server. When a port logs onto the fabric, it goes through a device discovery process with other devices registered in the name server. The zoning function controls this process by allowing only the members in the same zone to establish these link-level services.



Multiple zone sets may be defined in a fabric, but only one zone set can be active at a time. A zone set is a set of zones and a zone is a set of members. A member may be in multiple zones. Members, zones, and zone sets form the hierarchy defined in the zoning process. Members are nodes within the SAN that can be included in a zone. Zones comprise a set of members that have access to one another. A port or a node can be a member of multiple zones. Zone sets comprise a group of zones that can be activated or deactivated as a single entity in a fabric. Only one zone set per fabric can be active at a time. Zone sets are also referred to as zone configurations.



Zoning can be categorized into three types:

• Port zoning: It uses the FC addresses of the physical ports to define zones. In port zoning, access to data is determined by the physical switch port to which a node is connected. The FC address is dynamically assigned when the port logs on to the fabric. Therefore, any change in the fabric configuration affects zoning. Port zoning is also called hard zoning. Although this method is secure, it requires updating of zoning configuration information in the event of fabric reconfiguration.

• WWN zoning: It uses World Wide Names to define zones. WWN zoning is also referred to as soft zoning. A major advantage of WWN zoning is its flexibility. It allows the SAN to be re-cabled without reconfiguring the zone information. This is possible because the WWN is static to the node port.

• Mixed zoning: It combines the qualities of both WWN zoning and port zoning. Using mixed zoning enables a specific port to be tied to the WWN of a node.

Zoning is used in conjunction with LUN masking for controlling server access to storage. However, these are two different activities. Zoning takes place at the fabric level and LUN masking is done at the array level.



VSANs enable the creation of multiple logical SANs over a common physical SAN. They provide the capability to build larger consolidated fabrics and still maintain the required security and isolation between them. The slide depicts logical partitioning of a fabric by configuring multiple VSANs. Zoning should be done for each VSAN to secure the entire physical SAN. Each managed VSAN can have only one active zone set at a time.

As depicted in the figure, VSAN 1 is the active zone set. The SAN administrator can create distinct VSANs other than VSAN 1 and populate each of them with switch ports. In the example, the switch ports are distributed over three VSANs: 1, 2, and 3—for the IT, Engineering, and HR divisions, respectively. A zone set is defined for each VSAN, providing connectivity for HBAs and storage ports logged into the VSAN. Therefore, each of the three divisions—Engineering, IT, and HR—has its own logical fabric. Although they share physical switching gear with other divisions, they can be managed individually as stand-alone fabrics.

VSANs minimize the impact of fabric wide disruptive events because management and control traffic on the SAN—which may include RSCNs, zone set activation events, and more—does not traverse VSAN boundaries. Therefore, VSANs are a cost-effective alternative for building isolated physical fabrics. They contribute to information availability and security by isolating fabric events and providing a finer degree of authorization control within a single fabric.



Traditional SAN environments allow block I/O over Fibre Channel, whereas NAS environments allow file I/O over IP-based networks. Organizations need the performance and scalability of SAN plus the ease of use and lower TCO of NAS solutions. The emergence of IP technology that supports block I/O over IP has positioned IP for storage solutions.

IP offers easier management and better interoperability. When block I/O is run over IP, the existing network infrastructure can be leveraged, which is more economical than investing in new SAN hardware and software. Many long-distance, disaster recovery (DR) solutions are already leveraging IP-based networks. In addition, many robust and mature security options are now available for IP networks. With the advent of block storage technology that leverages IP networks (the result is often referred to as IP SAN), organizations can extend the geographical reach of their storage infrastructure.

IP SAN technologies can be used in a variety of situations. Disaster recovery solutions can also be implemented using both of these technologies.



Two primary protocols that leverage IP as the transport mechanism are iSCSI and Fibre Channel over IP (FCIP).

iSCSI is the host-based encapsulation of SCSI I/O over IP using an Ethernet NIC card or an iSCSI HBA in the host. As illustrated in the slide, IP traffic is routed over a network either to a gateway device that extracts the SCSI I/O from the IP packets or to an iSCSI storage array. The gateway can then send the SCSI I/O to an FC-based external storage array, whereas an iSCSI storage array can handle the extraction and I/O natively.

FCIP uses a pair of bridges (FCIP gateways) communicating over TCP/IP which is the transport protocol. FCIP is used to extend FC networks over distances using an existing IP-based infrastructure. Today, iSCSI is widely adopted for connecting servers to storage because it is relatively inexpensive and easy to implement, especially in environments where an FC SAN does not exist. FCIP is extensively used in disaster-recovery implementations, where data is duplicated on disk or tape to an alternate site.



Fibre Channel over Ethernet, FCoE, uses the Fibre Channel protocol over Ethernet networks. FCoE enables SAN traffic to be natively transported over Ethernet networks, while protecting and extending the investment that enterprises have made in storage networks. FCoE enables organizations to continue to run Fibre Channel over the same wires as their data networks. Ethernet is used as the physical interface for carrying FC frames. Multi-function network/storage adapters are used for FC-to-Ethernet mapping.

FCoE combined with 10 Gigabit Ethernet (10 Gbps) fabrics will grant organizations the ability to consolidate their I/O, cables and adapters while at the same time increase the utilization of their servers. It combines LAN and SAN traffic over a single 10Gb Ethernet connection.

The benefits of FCoE include lower capital and operating costs, lower cooling requirements and power savings. This results in lower total cost of ownership.

FCoE enables input/output consolidation by allowing LAN and SAN traffic to converge on a single cable or link. It reduces the number of server cables, adapters and switch ports in the data center and greatly simplifies the physical infrastructure. It also reduces the administrative overhead and complexity associated with managing the data center.



This lesson introduced Storage Area Networks, including a definition of SAN and its components, and an overview of fibre channel technology and interconnectivity options.



This lesson introduces Network-Attached Storage, including a definition of NAS, its benefits, components, and file sharing protocols.



Network-attached storage (NAS) is an IP-based file-sharing device attached to a local area network. NAS provides the advantages of server consolidation by eliminating the need for multiple file servers. It provides storage consolidation through file-level data access and sharing. NAS is a preferred storage solution that enables clients to share files quickly and directly with minimum storage management overhead.

NAS also helps to eliminate bottlenecks that users face when accessing files from a general-purpose server. NAS uses network and file-sharing protocols to perform filing and storage functions. These protocols include TCP/IP for data transfer and CIFS and NFS for remote file service. NAS enables both UNIX and Microsoft Windows users to share the same data seamlessly. To enable data sharing, NAS typically uses NFS for UNIX, CIFS for Windows, and File Transfer Protocol (FTP) and other protocols for both environments. Recent advancements in networking technology have enabled NAS to scale up to enterprise requirements for improved performance and reliability in accessing data.



A NAS device is a dedicated, high-performance, high-speed, single-purpose file serving and storage system. NAS serves a mix of clients and servers over an IP network. Most NAS devices support multiple interfaces and networks. A NAS device uses its own operating system and integrated hardware, software components to meet specific file service needs. Its operating system is optimized for file I/O and, therefore, performs file I/O better than a general purpose server. As a result, a NAS device can serve more clients than traditional file servers, providing the benefit of server consolidation.

A NAS device is optimized for file-serving functions such as storing, retrieving, and accessing files for applications and clients. As shown in the slide, a general-purpose server can be used to host any application as it runs a generic operating system. Unlike a general-purpose server, a NAS device is dedicated to file-serving. It has a real-time operating system dedicated to file serving by using open-standard protocols. Some NAS vendors support features such as built-in native clustering for high availability.



NAS offers the following benefits:

• Improved efficiency: Eliminates bottlenecks that occur during file access from a general-purpose file server because NAS uses an operating system specialized for file serving. It improves the utilization of general-purpose servers by relieving them of file-server operations.

• Improved flexibility: Compatible for clients on both UNIX and Windows platforms using industry-standard protocols. NAS is flexible and can serve requests from different types of clients from the same source.

• Centralized storage: Centralizes data storage to minimize data duplication on client workstations, simplify data management, and ensures greater data protection.

• Simplified management: Provides a centralized console that makes it possible to manage file systems efficiently.

• Scalability: Scales well in accordance with different utilization profiles and types of business applications because of the high performance and low-latency design.

• High availability: Offers efficient replication and recovery options, enabling high data availability. NAS uses redundant networking components that provide maximum connectivity options. A NAS device can use clustering technology for failover.

• Security: Ensures security, user authentication, and file locking in conjunction with industry- standard security schemas.



A NAS device has the following components:

• NAS head (CPU and Memory)

• One or more network interface cards (NICs), which provide connectivity to the network. NIC uses technologies such as Gigabit Ethernet, Fast Ethernet, ATM, and Fiber Distributed Data Interface (FDDI).

• An optimized operating system for managing NAS functionality

• NFS and CIFS protocols for stack file sharing

• Industry-standard storage protocols to connect and manage physical disk resources, such as ATA, SCSI, or FC

• Storage Array

The NAS environment includes clients accessing a NAS device over an IP network using standard protocols.



NAS uses file-level access for all I/O operations. File I/O is a high-level request that specifies the file to be accessed, but does not specify its logical block address. The NAS operating system keeps track of the location of files on the disk volume and converts client file I/O into block-level I/O to retrieve data. The retrieved data is again converted to file-level I/O for applications and clients.

Most NAS devices support multiple file service protocols to handle file I/O requests to a remote file system. NFS and CIFS are the common protocols for file sharing. NFS is predominantly used in UNIX-based operating environments; CIFS is used in Microsoft Windows–based operating environments. These file sharing protocols enable users to share file data across different operating environments and provide a means for users to migrate transparently from one operating system to another.



There are two types of NAS implementations: integrated and gateway. The integrated NAS device has all of its components and storage system in a single enclosure. In a gateway implementation, a NAS head shares its storage with a SAN environment.



This lesson introduced Network-Attached Storage, including a definition of NAS, its benefits, components, and file sharing protocols.



In this lesson, we describe replication and look at the various uses for replicas. We define consistency and learn about the importance of maintaining consistency. Then, we learn about the various local and remote replication technologies and how replicas can be leveraged for backup and recovery.



Replication is the process of creating an exact copy of data. Replicas can be used for recovery and restart operations in the event of data loss. The primary purpose of replication is to enable users to have designated data at the right place, in a state appropriate to the recovery need. The replica should provide recoverability and restartability. Recoverability enables restoration of data from replicas to production volumes in the event of data loss or data corruption. It must provide minimal RPO and RTO for resuming business operations on the production volumes, while restartability must ensure consistency of data on the replica. This enables restarting business operations using the replicas. Local replication refers to replicating data within the same array or the same data center. With remote replication, data is replicated to remote sites.



Replication can be used for various purposes, including:

Alternate source for backups: Under normal backup operations, data is read from the production volumes (LUNs) and written to the backup device. This places additional burden on the production infrastructure as production LUNs are simultaneously involved in production work. A point-in-time replica of the source data can be used to perform backup operations. Another benefit of using replicas for backup is that it reduces the backup window size.

Fast recovery: In the event of a partial failure of the source, a replica can be used to recover lost data. In the event of a complete failure of the source, the replica can be restored to a different set of source devices. In either case, this method provides faster recovery and minimal RTO.

Decision-support activities such as reporting: Running reports using the data on the replicas greatly reduces the I/O burden placed on the production device.

Testing platform: A replica can be used for testing critical business data or applications. For example, when planning an application upgrade, the upgrade can be tested using the replica. Production data is not changed.

Data migration: Replication can also be used for data migration. Data migration may be performed for various reasons, such as migrating from a small LUN to a larger LUN.



Replicas can either be Point-in-Time (PIT) or continuous.

Point-in-Time (PIT), also known as a snapshot - The data on the replica is an identical image of production at some specific timestamp. For example, a snapshot of a file system is created at 4:00 PM on Monday. This replica would then be referred to as the Monday 4:00 PM Point-in-Time copy. Note that if the primary purpose of replication is to have a viable point-in-time copy for data recovery or restore operations, then the target replica should not be modified.

Continuous replica - The data on the replica is synchronized with the production data at all times. The objective with any continuous replication is to reduce the recovery point objective (RPO) to zero.



Consistency is a primary requirement to ensure the usability of the replica.

In case of file systems, consistency can be achieved either by taking the file system offline, for example, by un-mounting the file system, or by keeping the file system online and flushing host buffers before creating the copy.

A database may be spread over numerous files, file systems, and devices. All of these must be replicated consistently to ensure that the replica is restorable and restartable. Replication can be performed with the database offline or online. If the database is offline, it is not available for I/O operations. Because no updates are occurring, the replica will be consistent. If the database is online, it is available for I/O operations. Transactions to the database will be updating data continuously. When a database is backed up while it is online, changes made to the database at this time must be applied to the backup copy to make it consistent. Performing an online backup requires additional procedures during backup and restore. Consistency of an online backup can be ensured using the dependent write I/O principle where in order for a transaction to be deemed complete, a series of writes have to occur in a particular order. At the point in time when the snapshot is created, all the writes to the source devices are captured on the replica devices to ensure data consistency. Another way to ensure consistency is to make sure that writes to all source devices are held while the replica is created.



File systems buffer data in host memory to improve application response time. The buffered information is periodically written to disk. In UNIX operating systems, the sync daemon is the process that flushes the buffers to disk at set intervals. In some cases, the replica may be created in between the set intervals. Hence, the host memory buffers must be flushed to ensure data consistency on the replica, prior to its creation.

The figure on the slide illustrates flushing of the buffer to its source, which is then replicated. If the host memory buffers are not flushed, data on the replica will not contain the information that was buffered in the host. If the file system is un-mounted prior to the creation of the replica, the buffers would be automatically flushed and data would be consistent on the replica. If a mounted file system is replicated, some level of recovery such as fsck or log replay would be required on the replicated file system. When the file system replication process is completed, the replica file system can be mounted for operational use.



A database may be spread over numerous files, file systems, and devices. All of these must be replicated consistently to ensure that the replica is restorable and re-startable. Replication can be performed with the database offline or online. If the database is offline, it is not available for I/O operations. Because no updates are occurring, the replica will be consistent. If the database is online, it is available for I/O operations. Transactions to the database will be updating data continuously. When a database is backed up while it is online, changes made to the database at this time must be applied to the backup copy to make it consistent. Performing an online backup requires additional procedures during backup and restore.



Consistency of an online backup can be ensured using the dependent write I/O principle inherent in any database management system, DBMS. In order for a transaction to be deemed complete, databases require that a series of writes have to occur in a particular order. These writes would be recorded on the various devices/file systems. The figure on the slide illustrates the process of flushing the buffer from host to source; I/Os 1 to 4 must complete in order for the transaction to be considered complete. I/O 4 is dependent on I/O 3 and will occur only if I/O 3 is complete. I/O 3 is dependent on I/O 2, which in turn depends on I/O 1. Each I/O completes only after completion of the previous I/O(s). Dependent write consistency is required for protection against power outages, loss of local channel connectivity, or storage devices.



An alternative way to ensure that an online replica is consistent is to:

Hold I/O to all the devices at the same instant

Create the replica

Release the I/O

Holding I/O is similar to a power failure and most databases have the ability to restart from a power failure.

Note: While holding I/O simultaneously, one ensures that the data on the replica is identical to that on the source devices. The database application times out if I/O is held for too long.



Consistency groups operate in unison to preserve the integrity and dependent write consistency of an application distributed across multiple devices and arrays.

When a typical DBMS application updates a database, it first writes to the disk containing a log, and then it writes the data to the actual database data files. Finally, it writes again to the log volume to flag these write I/Os (log database) that are related. In a remote replication environment, data consistency cannot be ensured if one of these I/Os was remotely mirrored, but its predecessor was not remotely mirrored. This could occur, for example, in a rolling disaster where there is a communication loss that affects only a portion of the disk controllers that are performing the remote copy function.

Consistency groups can prevent this from occurring by intercepting any I/O to a disk device that cannot communicate to its remote mirror. The consistency protocol is to then suspend the remote mirroring for all devices defined to the consistency group. In this way, consistency groups prevent dependent I/O from getting out of sync, thus ensuring the integrity and consistency of the data at the remote site.



Local replication technologies can be classified based on where the replication is performed.

• Host based - Replication is performed by using the CPU resources of the host using software that is running on the host.

• Array based - Replication is performed on the storage array using CPU resources on the array via the array’s operating environment.



In LVM-based replication, the logical volume manager is responsible for creating and controlling the host-level logical volume. In LVM-based replication, each logical partition in a logical volume is mapped to two physical partitions on two different physical volumes. An application write to a logical partition is written to the two physical partitions by the LVM device driver. This is also known as LVM mirroring. Mirrors can be split and the data contained therein can be independently accessed. LVM mirrors can be added or removed dynamically.

As every write generated by an application translates into two writes on the disk, an additional burden is placed on the host CPU. This can degrade application performance. Presenting an LVM-based local replica to a second host is usually not possible because the replica will still be part of the volume group, which is usually accessed by one host at any given time.

Tracking changes to the mirrors and performing incremental synchronization operations is also a challenge as all LVMs do not support incremental resynchronization. If the devices are already protected by some level of RAID on the array, then the additional protection provided by mirroring is unnecessary. This solution does not scale to provide replicas of federated databases and applications. Both the replica and the source are stored within the same volume group. Therefore, the replica itself may become unavailable if there is an error in the volume group. If the server fails, both source and replica are unavailable until the server is brought back online.



A file system (FS) snapshot is a pointer-based replica that requires a fraction of the space used by the original FS. This snapshot can be implemented by either the FS itself or by the LVM. It uses the Copy on First Write (CoFW) principle. With this technology, original data blocks are copied to the snapshot before those data blocks are modified on the production volume.

In order to create a snapshot of the production volume, first all applications are frozen and the file system is synchronized to ensure consistency. Then, the snapshot is captured, which is indicated by the dotted volume on the slide. The snapshot captures the image of the application data at that point in time. From that point on, if a data block changes, then before the change is written to the production volume, the original block is first copied to the snapshot cache. Unchanged blocks still reside in the primary volume and are pointed to by the snapshot. Any additional writes to the changed blocks will occur on the production volume as normal. In this way, the snapshot preserves the point-in-time copy of the data, while the production volume continues to be updated.

When the snapshot is created, a bitmap and a block map are created in the metadata of the FS snapshot. The bitmap is used to keep track of blocks that are changed on the production FS after creation of the snap. The block map is used to indicate the exact address from which data is to be read when the data is accessed from the FS snapshot. Immediately after creation of the snapshot all reads from the snapshot will actually be served by reading the production FS. To read from the FS snapshot, the bitmap is consulted. If the bit is 0, then the read is directed to the production FS. If the bit is 1, then the block address is obtained from the block map and data is read from that address. Reads from the production FS work as normal.



In storage array-based local replication, the array operating environment performs the local replication process. The host resources such as CPU and memory are not used in the replication process. Consequently, the host is not burdened by the replication operations. The replica can be accessed by an alternate host for any business operations. In this replication, the required number of replica devices should be selected on the same array and then data is replicated between source-replica pairs. A database could be laid out over multiple physical volumes: and in that case all the devices must be replicated for a consistent PIT copy of the database.

Storage array-based local replication can be further categorized as full-volume mirroring, pointer-based full-volume replication, and pointer-based virtual replication. Replica devices are also referred as target devices and accessed by backup hosts.



In full-volume mirroring, the target is attached to the source and established as a mirror of the source. Existing data on the source is copied to the target. New updates to the source are also updated on the target. After all the data is copied and both the source and the target contain identical data, the target can be considered a mirror of the source. While the target is attached to the source and the synchronization is taking place, the target remains unavailable to any other host. However, the production host can access the source.

After the synchronization is complete, the target can be detached from the source and be made available for Business Continuity operations. The point-in-time (PIT) is determined by the time of detachment or separation of the source and target. For example, if the detachment time is 4:00 PM, the PIT of the replica is 4:00 PM. Note that the target device must be at least as large as the source device.

After detachment, changes made to both source and replica can be tracked at some predefined granularity. This enables incremental resynchronization (source to target) or incremental restore (target to source). The granularity of the data change can range from 512 byte blocks to 64 KB blocks. Changes are typically tracked using bitmaps, with one bit assigned for each block. If any updates occur to a particular block, the whole block is marked as changed, regardless of the size of the actual update. However, for resynchronization (or restore), only the changed blocks have to be copied, eliminating the need for a full synchronization (or restore) operation. This method reduces the time required for these operations considerably.



Like full-volume mirroring, this technology can provide full copies of the source data on the targets. Unlike full-volume mirroring, the target is made immediately available at the activation of the replication session. Hence, one need not wait for data synchronization to, and detachment of, the target in order to access it. The time of activation defines the PIT copy of source. Pointer-based, full-volume replication can be activated in either Copy on First Access (CoFA) mode or Full Copy mode. At the time of activation, a protection bitmap is created for all data on the source devices. Pointers are initialized to map the (currently) empty data blocks on the target to the corresponding original data blocks on the source. The granularity can range from 512 byte blocks to 64 KB blocks or higher. Data is then copied from the source to the target, based on the mode of activation.

In the Copy on First Access mode, data is copied from the source to the target only when a write is issued for the first time after the PIT to a specific address on the source or a read or write is issued for the first time after the PIT to a specific address on the target. Since data is only copied when required, if the replication session is terminated, the target device only has data that was copied and not the entire contents of the source at the PIT. In this scenario, the data on the target cannot be used as it is incomplete.

In Full Copy mode, the target is made available immediately and all the data from the source is copied over to the target in the background. During this process, if a data block that has not yet been copied to the target is accessed, the replication process jumps ahead and moves the required data block first. When a full copy mode session is terminated (after full synchronization), the data on the target is still usable as it is a full copy of the original data.



In pointer-based virtual replication, at the time of session activation, the target contains pointers to the location of data on the source. The target does not contain data at any time. Hence, the target is known as a virtual replica. Similar to pointer-based full-volume replication, a protection bitmap is created for all data on the source device and the target is immediately accessible. Granularity can range from 512 byte blocks to 64 KB blocks or greater.

Pointer-based virtual replication uses CoFW technology. When a write is issued to the source for the first time after session activation, original data at that address is copied to a predefined area in the array termed the save location. The pointer in the target is updated to point to the data address in the save location. Then, the new write is updated on the source. For reads from the target, unchanged data blocks since session activation are read from the source. Original data blocks that have changed are read from the save location.

The primary advantage of pointer based copies is the reduction in storage requirement for the replicas as the target only contains pointers to the data.



Most array-based replication technologies allow the source devices to maintain replication relationships with multiple targets. This can also reduce RTO because the restore can be a differential restore. Each PIT could be used for a different backup activity and also as restore points.

In this example, a PIT is created every 6 hours from the same source. If any logical or physical corruption occurs on the source, the data can be recovered from the latest PIT and at worst, the RPO will be 6 hours.

Array local replication technologies also enable the creation of multiple concurrent PIT replicas. In this case, all replicas will contain identical data. One or more of the replicas can be set aside for restore or recovery operations. Decision support activities can be performed using the other replicas.



Backups to another media or device can be performed using snapshots as the source for the backup data. This is referred to as a “live backup”. Local PIT replicas can also be used as backups to restore data to production devices. Alternatively, applications can be restarted using the consistent point-in-time copy of the data on the replicas. The choice of target depends on the consistency of the data on the target and the desired RPO (e.g., a business may create PIT replicas every 2 hours; if a failure occurs, then at most only 2 hours of data would have been lost). If a target has been written to after the creation of the PIT, such as for application testing for example, then this target may not be a viable candidate for the restore or restart.

A replica can be used to restore data to the production devices in the event of logical corruption of production devices. In this case, the devices are available but the data on them is invalid. Restore operations from a replica are incremental and provide a very small RTO. In some instances, applications can be resumed on the production devices prior to completion of the data copy. Production devices may become unavailable due to physical failures. Depending upon the replication technology used, applications can be restarted using data on the latest replica. If the production server fails, once the issue has been resolved, the latest information from the replica devices can be restored back to the production devices.

If the production device(s) fail, applications can continue to run on replica devices. A new PIT copy of the replica devices can be created or the latest information from the replica devices can be restored to a new set of production devices. With CoFW and CoFA technologies, access to data on the snapshot is dependent on the health and availability of the original source volumes. If the original source is not available, the snapshots cannot be used for backup, restore or restart.



Using snapshot and mirroring technologies enables backup software to perform backups with minimum impact to the production application servers. Off-host or server-less backup frees up resources on the production application server by off-loading the backup workload to a secondary host called a proxy client. The proxy client acts as a backup client and storage node for backing up to a conventional backup medium. When performing a server-less backup, the snapshot can also be retained on the storage disk volume as a persistent snapshot.

The slide shows an example of an off-host backup configuration known as “LAN-free with proxy client” as all data travels over the SAN. A snapshot or clone from the application server is performed. The clone is accessed for backup by the proxy server over the SAN. The backup data is sent to a Data Domain system for storage. Backup processing load is on the proxy server only.



Remote replication is the process of creating replicas of information assets at remote sites. Remote replicas help organizations mitigate the risks associated with regionally driven outages resulting from natural or human-made disasters. Similar to local replicas, they can also be used for other business operations.

Two basic modes of remote replications are synchronous and asynchronous replication. Data has to be transferred from the source site to a target site over some network. This can be done over IP networks, over the SAN, using DWDM (Dense Wave Division Multiplexing) or SONET (Synchronous Optical Network).



With synchronous replication, data is committed at both the source site and the target site before the write is acknowledged to the host. Any write to the source must be transmitted to and acknowledged by the target before signaling a write complete to the host. Additional writes cannot occur until each preceding write has been completed and acknowledged. It ensures that data at both sites are identical at all times.

Application response time is increased with any synchronous remote replication. The degree of the impact on the response time depends on the distance between sites, available bandwidth, and the network connectivity infrastructure. The distances over which synchronous replication can be deployed depend on the application’s ability to tolerate extension in response time. Typically, it is deployed for distances less than 200 KM (125 miles) between the two sites. To minimize the response time elongation, ensure that the maximum bandwidth is provided by the network at all times.



In asynchronous remote replication, a write is committed to the source and immediately acknowledged to the host. Data is buffered at the source and transmitted to the remote site later. Data at the remote site will be behind the source by at least the size of the buffer. Hence, asynchronous remote replication provides a finite (nonzero) RPO disaster recovery solution. RPO depends on the size of the buffer, available network bandwidth, and the write workload to the source. There is no impact on application response time, as the writes are acknowledged immediately to the source host. This enables deployment of asynchronous replication over extended distances. Asynchronous remote replication can be deployed over distances ranging from several hundred to several thousand kilometers between two sites.



Host-based remote replication implies that the replication is done by using the CPU resources of the host using software that is running on the host. It is performed between data centers.

Array-based remote replication is done between storage arrays and is handled by the array operating environment.



LVM-based remote replication is performed and managed at the volume group level. Writes to the source volumes are transmitted to the remote host by the LVM. The LVM on the remote host receives the writes and commits them to the remote volume group. Prior to the start of replication, identical volume groups, logical volumes, and file systems are created at the source and target sites.

LVM-based remote replication supports both synchronous and asynchronous modes of data transfer. In asynchronous mode, writes are queued in a log file at the source and sent to the remote host in the order in which they were received. The size of the log file determines the RPO at the remote site. In the event of a network failure, writes continue to accumulate in the log file. If the log file fills up before the failure is resolved, then a full resynchronization is required upon network availability. In the event of a failure at the source site, applications can be restarted on the remote host, using the data on the remote replicas. LVM-based remote replication eliminates the need for a dedicated SAN infrastructure. LVM-based remote replication is independent of the storage arrays and types of disks at the source and remote sites. The replication process adds overhead on the host CPUs. CPU resources on the source host are shared between replication tasks and applications, which may cause performance degradation of the application. As the remote host is also involved in the replication process, it has to be continuously up and available.

A significant advantage of using LVM based remote replication is that storage arrays from different vendors can be used at the two sites, e.g., at the production site, a high-end array could be used while at the target site, a second tier array could be used. In a similar manner, the RAID protection at the two sites could be different as well.



Database replication via log shipping is a host-based replication technology supported by most databases. Transactions to the source database are captured in logs, which are periodically transmitted by the source host to the remote host. The remote host receives the logs and applies them to the remote database. Prior to starting production work and replication of log files, all relevant components of the source database are replicated to the remote site. This is done while the source database is shut down.

After this step, production work is started on the source database. The remote database is started in a standby mode. Typically, in standby mode, the database is not available for transactions. All DBMSs switch log files at preconfigured time intervals, or when a log file is full. The current log file is closed at the time of log switching and a new log file is opened. When a log switch occurs, the closed log is transmitted by the source host to the remote host.

The remote host receives the log and updates the standby database. This process ensures that the standby database is consistent up to the last committed log. RPO at remote site is finite and depends on the size of log and the frequency of log switching. Available network bandwidth, latency, and rate of updates to the source database, as well as the frequency of log switching, should be considered when determining the optimal size of the log file.

Because the source host does not transmit every update and buffer them, this alleviates the burden on the source host CPU. Host-based log shipping does not scale well.



In storage array-based remote replication, the array operating environment and resources perform and manage data replication. This relieves the burden on the host CPUs, which can be better utilized for running an application. A source and its replica device reside on different storage arrays. In other implementations, the storage controller is used for both the host and replication workload. Data can be transmitted from the source storage array to the target storage array over a shared or a dedicated network. Replication between arrays may be performed in synchronous, asynchronous, or disk-buffered modes. Three-site remote replication can be implemented using a combination of synchronous mode and asynchronous mode, as well as a combination of synchronous mode and disk-buffered mode.



The data on the remote replicas can be behind the source by a finite amount in asynchronous replication, thus steps must be taken to ensure consistency. Some vendors achieve consistency by maintaining write ordering, i.e. the remote array applies writes to the replica devices in the exact order that they were received at the source. Other vendors leverage the dependent write I/O logic that is built into most databases and applications.



This lesson covered replication, including the various uses for replicas, the importance of maintaining consistency, local and remote replication technologies, and how replicas can be leveraged for backup and recovery.



This lesson covers continuous data protection, including a definition of CDP, its fundamental attributes and benefits, and an overview of how CDP works.



CDP as defined by SNIA: Continuous data protection (CDP) is a methodology that continuously captures or tracks data modifications and stores changes independent of the primary data, enabling recovery points from any point in the past. CDP systems may be block, file or application-based and can provide fine granularities of restorable objects to infinitely variable recovery points.

CDP – Fundamental Attributes:

• Data changes are continuously captured or tracked.

• All data changes are stored in a separate location from the primary storage.

• Recovery point objectives are arbitrary and need not be defined in advance of the actual recovery.



CDP provides recovery at the local and remote sites.

• I/O is sent to the appliance in one of two ways through a host based splitter or using intelligent fabric switches.

• At the local site, the CDP engine captures every I/O into the local CDP journal with I/O bookmarking to capture application events.

CDP provides local and remote replication with bandwidth reduction and WAN optimization. It provides target side processing which enables immediate read/write access of the replicated volumes for DR, application testing and failover.



CDP provides increased protection from both logical and physical errors and reduces the exposure to data loss. It facilitates fast and flexible recovery - by optimizing the point of recovery, downtime is greatly reduced. Because data is captured whenever any change is made, it is possible to recover right up to the point in time prior to corruption or loss of data.



This lesson covered continuous data protection, including a definition of CDP, its fundamental attributes and benefits, and an overview of how CDP works.



In this module, we look at the various sources of backup data including file system data and several types of databases, including Oracle, Microsoft SQL, Microsoft Exchange and Microsoft SharePoint. Then, we will look at the role of Microsoft Volume Shadow Copy Service, NDMP, and virtualization in backup. Lastly, we examine various considerations and challenges impacting client backup environments.

1 Module 3: Backup Client


Backup clients are the hosts in the backup environment that provide the data to be backed up. This data can be classified as either file system or database data. In this lesson, we look at the considerations and differences for backing up file systems and databases including the importance of ensuring the consistency of data.



Users store and retrieve data through applications. The computers on which these applications run are referred to as hosts. Hosts can range from simple laptops to complex clusters of servers. Backup clients access the data that is to be backed up and provide the data to the backup storage node.



A file is a collection of related records or data stored as a unit with a name. A file system on disk is a structure of files. File systems enable easy access to data files residing within a disk drive, a disk partition, or a logical volume. A file system needs host-based logical structures and software routines that control access to files. It provides users with the functionality to create, modify, delete, and access files. Access to the files on the disks is controlled by the permissions given to the file by the owner, which are also maintained by the file system.

A file system organizes data via the use of directories, which are containers for storing pointers to multiple files. All file systems maintain a pointer map to the directories, subdirectories, and files that are part of the file system. Some of the common file systems are FAT 32 (File Allocation Table) and NT File System (NTFS) for Microsoft Windows, UNIX File System (UFS) for UNIX and Extended File System (EXT2/3) for Linux. The file system tree starts with the root directory. The root directory has a number of subdirectories. A file system should be mounted before it can be used.

Apart from the files and directories, the file system also includes a number of other related records, which are collectively called the metadata. The metadata of a file system has to be consistent in order for the file system to be considered healthy. A metadata contains information about the file system, such as the file system type, creation and modification dates, size and layout, the count of available resources (such as number of free blocks, inodes, etc.), and a flag indicating the mount status of the file system.



Several methods are used to get the data from its source to a backup device. The most simple and common method is a file copy. Backup applications use copy commands, sometimes the very own operating system’s copy applications (such as UNIX’s tar), or modified versions of those, to copy the data. With this type of copy, the metadata includes the names and characteristics of all files, so the level of granularity for recovery is at the file level. The performance of a file copy backup is directly affected by the number of files, sizes and the general characteristics of the files system being backed up.

In certain situations, the user may want to backup the data on a Raw Device level. That means that the file system will have to be un-mounted so the copy can take place. The backup application can then use “dump” applications, such as UNIX’s dd, to perform a copy from the raw device to the backup device. This type of backup is usually faster than a file copy, but affects restore granularity. A single file restore requires the entire raw device to be restored.



Cold backup and hot backup are the two methods deployed for backup. They are based on the state of the application when the backup is performed. In a cold backup, the application is not active during the backup process. In a hot backup, the application is up and running during the backup process with users accessing their data.



Typically when backing up file system data, the data to be backed up is accessed at the file level. The backup application must have the necessary file permissions to access the data. The backup is taken as of a specific point-in-time. To ensure consistency of the backup, no changes to the data should be allowed while the backup is being created.

When performing a cold backup, files can be backed up by a backup application with some type of file copy or proprietary method as users are not currently accessing the data. The backup of data while files are opened becomes more challenging because data is actively being used and changed. An open file is locked by the operating system and is not copied during the backup process until the user closes it. The backup application can back up open files by retrying the operation on files that were opened earlier. During the backup process, it may be possible that files opened earlier will be closed and a retry will be successful. However, this method is not considered robust because in some environments certain files are always open. In such situations, the backup application or the operating system can provide open file agents. These agents interact directly with the operating system and enable the creation of copies of open files.

Backing up a file is often not enough to ensure proper recoverability in case of a failure. There are certain attributes and properties attached to a file, such as permission information, owner and other metadata, that should be backed up as well. These attributes are as important as the data itself and should be appropriately backed up by the backup application.



In some environments the use of Open File Agents is not enough. The backup of a database or database application is a good example. A database is composed of several different files which may occupy several file systems. Data in one file may be dependent upon data in another. A single transaction may cause updates to several files and these updates may need to occur in a defined order. A consistent backup of a database means that all files need to be backed up at the same “point” or state. That doesn’t necessarily mean that all files need to be backed up at the same time, but they must be in sync with each other so the database can come up consistently in case of a restore.

Consistent backups of databases can be done using a cold (or offline) method which means that the database is shut down while the backup is running. The downside is that the database will not be accessible by users.

Hot backup is used in situations where it is not possible to shut down the database. Backup is facilitated by database backup agents that can perform a backup while the database is active. The disadvantage associated with a hot database backup is that the agents can negatively affect the performance of the database application server.



Consistency is critical to ensure that a backup can restore a file, directory, file system, or database to a specific point-in-time. Consistency is a primary requirement to ensure the usability of replica device or backup.

Consistency is achieved in various ways through the storage device, application or operating system.

In case of file systems, consistency can be achieved by taking the file system offline, i.e. by un-mounting the file system, or by keeping the file system online and flushing host buffers before creating the backup to ensure that all writes are committed. No further writes are allowed to the data while the backup is being created.

A database is composed of several different files which may occupy several file systems. Data in one file may be dependent upon data in another. A single transaction may cause updates to several files and these updates may need to occur in a defined order. A consistent backup of a database means that all files need to be backed up at the same “point” or state. That doesn’t necessarily mean that all files need to be backed up at the same time, but they must be in sync with each other so the database can come up consistently in case of a restore. To create a consistent backup, either the database is shutdown or a hot backup mode is used.



This lesson covered differences and considerations of backing up file systems and databases including the importance of ensuring the consistency of data.



This lesson begins with an overview of the basic components of a relational database system. Then, we look at how data is stored and backed up in various database applications, including Oracle, Microsoft SQL, Microsoft Exchange and Microsoft SharePoint.



Basically, a relational database system is a computerized record keeping system. Its overall purpose is to maintain data and assist an end-user to collect information. The relational model uses the basic logical concept of a relation or table. The columns or fields in the table identify attributes such as First Name, Last Name, City and State. A row contains all data from a single record, such as a person whose last name is ‘Johnson’. In the relational model, every row must have a unique identification or key based on the data. In this example, the Employee Number field is the key that uniquely identifies each row in the relation.



The key to understanding a relational database management system is understanding the relationship from one table to the next.

In the example on the slide, the Master table, Table A, contains general data elements for each individual. The Salary table, Table B, contains specific data elements such as Salary and Department. If you look at the Salary table alone, you cannot determine Jim Smith’s salary or what department he works in.

By joining the two tables, a view is created called Salary Information using the Key Relation field in Table A and B. Now the end-user can extract salary and department information from the Salary Information view for Jim Smith. A relationship can span across multiple tables. Once a view is successfully created based on this relationship, it can be used just like a table. If data is modified in any of the tables, the view reflects the change immediately.

The key to establishing or creating a robust RDBMS environment is to create appropriate tables that hold appropriate data entities. With tables and views appropriately laid out, the end-user can easily extract information from the database. Designing a relational database application that works efficiently is the result of precisely following a set of guidelines or rules called ‘Normalization’.



A database index is an object to help locate or access data within a database faster.

Think of an index as a tab marker in a phonebook. DBA’s could set up an index on a specific field, such as Last_Name.

When a request is submitted for a specific name, such as Smith, the index jumps to the appropriate record.

Without the Index, the query would evaluate every record and if a table has a large number of records, this could take some time.



The ACID properties of a DBMS allow safe sharing of data. Every database transaction should test positive against the following four important properties: Atomicity, Consistency, Isolation, and Durability. It is important for Database Administrators, or DBAs, to understand these four properties when maintaining a database.

Atomicity means that when updating a database, either all or none of the update becomes available to anyone.

Consistency occurs when any values are changed within the database or instance, and they are consistent with changes to all other values.

The term Isolation refers to the safeguards used by an RDBMS application to prevent conflicts that may occur between two transactions at the same time.

Durability means the DBA can recover all committed transactions, no matter what fails, and that updates must never be lost.



Binary large object (BLOB) data is user data that can be stored inside or outside the database. BLOBs are typically images, audio or other multimedia objects. When stored outside the database, consistency when backing up must be taken into account as the database tracks the location of BLOBs stored in the file system. Backup approaches include performing cold backup of the database file system or backing up from a point-in-time snapshot of the file system.



Typically, a backup application can back up file system data only. In cold backup mode, database files are backed up with the database shut down.

To do a hot backup with the database online, the backup application typically provides a database module for the specific database. This module interfaces with the database backup utility’s API to perform the backup or restore.

The slide depicts an example of a backup of a database using the hot backup mode. In this option, the backup application’s database agent is installed on the database agent. Advantages of this approach include the ability to perform backup while the application is running and no additional primary storage is required to do the backup. Cons include a high CPU utilization on the client and backup is limited to databases supported by the backup application.



The point-in-time (PIT) copy method is deployed in environments where the impact of downtime from a cold backup or the performance resulting from a hot backup is unacceptable. A pointer-based PIT copy consumes only a fraction of the storage space and can be created very quickly. In this method of backup, the database is stopped or frozen momentarily while the PIT copy is created of the data on the storage volume. Once the snapshot is taken, a separate proxy machine mounts the snapshot as if it was a locally-attached volume. Similarly, mirroring technology could be used and the backup client host would mount the split mirror. The backup client running on the proxy then reads and backs up the volume. In this option, the proxy client machine then “owns” the backup.

The pros with this approach include:

• Fastest restore - latest backup copy is immediately available (with split-mirror approach)

• No additional CPU utilization on the database server as backup is performed by the proxy client

• Standard file system backup can be employed

• Works with databases that are not supported with backup agents by the backup application

The cons with this approach include :

• Additional primary storage is required to store the snapshot or mirror

• Additional server is required



In this option, the database is first dumped to a local “disk cache” accessed by a machine where the backup client is installed. In this option, the client machine then “owns” the backup. The backup application backs up the dump files as a standard file system backup. This option works with databases that provide an export or backup to file capability.

Pros include:

• Fast restore - backup copy immediately available on local “disk cache”

• Modest CPU utilization on database server; same as used for tape backup

• Standard file system backup

• Databases that are not supported with backup application database can be backed up as long as they provide an export or backup to file capability

Cons include:

• Additional primary storage required

• Additional server may be required

• Two step restore process



The diagram depicts the components of an Oracle database environment. In the next few slides, we explore the Oracle components in more detail.



Oracle DBMS is an object-relational database management system. An Oracle server consists of an Oracle instance and an Oracle database. The Oracle instance is software executed on the server as part of the Oracle server that stays resident in memory. An Oracle instance consists of a memory structure, called System Global Area or SGA, and background processes used by an Oracle server to manage a database.

This software enables access to data that resides on the storage array. The database consists of files that are located on any number of physical storage devices. These files are timestamp-related to each other. Losing one file results in a corrupt database. The user does not access the data directly. Requests for information from the database are processed through the instance. The Oracle processes manage all activity inbound or outbound for all database files.



An Oracle database can be divided into smaller, logical areas of space known as tablespace. An Oracle tablespace is a logical unit that consists of one or more datafiles. Except for the Oracle SYSTEM tablespace, a tablespace can be taken online or offline while the database is running. A tablespace can belong to only one database. The collection of all tablespaces creates a logical database.

Data is stored physically in datafiles. A datafile is a physical file, with a file extension of dbf, that contains actual Oracle data. A datafile can belong to only one tablespace.

The Control file is a binary file that contains metadata information about the database itself The Control file is crucial for the operation of the database and should always be part of the database backup process. Modification or removal of this file results in a corrupt database.



Datafiles consist of extents and segments; datafiles consist of one or more data blocks. Data blocks are the basic unit of storage.

An Extent is a set of data blocks that are contiguous.

A Segment consists of a number of extents that are not necessarily contiguous. A Segment could be a Table or Index. A Table or Index is also referred to as an Oracle object. The Oracle environment is made up of many objects or segments that are comprised of many extents.



All Oracle transactions are recorded in the online redo logs. Online redo logs are used to make sure all committed transactions are saved to disk and all uncommitted transactions are rolled back. This allows for automatic recovery of Oracle transactions in the event of a database failure. Each redo log file belongs to a logical entity called a Log Group. Oracle uses the redo log groups in a circular fashion through a process called Log Switching.

When the database runs in Archive Log Mode, the redo log file is archived before being written over by the log switch operation. Archive log files consume a significant amount of space over a long period of time.

Archive log files enable administrators to perform a complete point-in-time recovery of the database. In an environment where loss of any data is not acceptable, Oracle recommends that the database run in Archive Log Mode.



The System Change Number, SCN, is a sequential counter number that is incremented over time as change vectors are generated, applied, and written to the redo logs. When a user commits a transaction, the transaction is assigned a system change number which Oracle records along with the transaction's redo entries in the redo log.

The system change number is crucial to the database recovery procedure. SCNs are recorded in the redo log so that recovery operations can be synchronized.



There are two basic methods of performing database backups based on the state of the database when the backup is performed. The first method is an offline or cold backup. In an offline backup, Oracle database files and the control file are backed up while the Oracle database is shutdown. All files copied during an offline database backup are consistent to a point-in-time. A recovery is only as good as the last full offline, database backup. Any transactions that occurred after the offline backup are lost.



The second method is an online or hot backup. In this type of backup, the database is up and running during the backup process with users accessing their data. In Oracle, an online backup requires that the database operate in ARCHIVELOG mode. The backup unit for online backup is some or all Oracle tablespace(s), whereas the backup unit for offline database backup is the entire Oracle database.

During an online backup of an Oracle database, all changes made to the database, either committed or uncommitted, are stored to the redo log buffer and subsequently are written to the online redo log files. A recovery can be carried out using the archived redo logs.



If the Oracle database is configured in NOARCHIVELOG mode, no redo history is saved to the archived redo logs. Recovery operations are limited only to the last full backup and a loss of transactions may occur. The Oracle database is created for NOARCHIVELOG mode by default.

The advantage of NOARCHIVELOG mode is that recoveries are easy to perform because you only need to restore all datafiles and control files from the backup. The time taken for recovery is merely the length of time it takes to restore all files.



If the Oracle database is configured in ARCHIVELOG mode, all the redo history is maintained in the archived redo logs and the Oracle database can be recovered in full. The total recovery time is the length of time to restore the datafiles and apply all archived redo logs and online redo logs.

When in ARCHIVELOG mode, the Oracle database can be backed up while it is online. The Oracle database is protected from the loss of data due to media failure. In other words, you can restore the damaged files from backup and use the archived log files to bring the datafiles up-to-date while the Oracle database is online or offline. The Oracle database can be recovered to a specific point-in-time, the end of a specific archived redo log, or to a specific System Change Number.



In Oracle, there is a distinction between restore and recovery.

Restoring is copying the backup files from backup media to disk. Recovery is the process of applying redo logs to the datafiles to bring the database to a consistent state.



All datafiles must be synchronized first before the Oracle database can be opened. Archived redo logs and online redo logs are used to synchronize Oracle database.



Oracle Recovery Manager (RMAN) is a utility that manages Oracle database backup, restore and recovery operations. Oracle RMAN is a command-line and Enterprise Manager-based (GUI) tool providing the user with a set of easy to use processes for backing up and recovering an Oracle database environment. RMAN supports backup and restore tasks across many different operating systems. Backup applications provide agents that act as intermediaries between the backup server and Oracle Recovery Manager.



RMAN's architecture is a combination of an executable program, the RMAN utility, and background processes that interact with one or more databases. The RMAN executable manages all backup and recovery operations.

Within RMAN, the database to backup or recover is referred to as the Target Database. The Recovery Catalog is used by the RMAN utility to record and track production database backup activity. RMAN uses the information in the recovery catalog to determine how to perform requested backup and restore operations. The recovery catalog can be stored in another Oracle database or in a database control file which is backed up with the original database.

The Flash Recovery Area stores RMAN files that are related to the backup and recovery process.

The Media Management Layer (MML) is software that manages database backups to media, such as tape and other devices. Oracle's Media Management Layer (MML) API lets third-party vendors build a media manager.



When using a backup application media manager, the backup administrator may use the backup software’s interface to configure and run the backups and restores. Additionally, an Oracle database administrator may choose to manually back up or restore an Oracle database directly from RMAN using RMAN scripts. In either case, the backup application’s agent running on the Oracle server works with Oracle and RMAN to back up the Oracle database, tablespace, or datafiles.

RMAN establishes connection with the target database. Each connection starts an Oracle server process that performs the backup. After parsing the contents of the RMAN backup script, the Oracle server processes will read the proper Oracle objects, for example, a tablespace, and then pass the data to the backup application through the backup agent. The backup application stores the Oracle database objects to the appropriate backup storage device. After the Oracle backup, the backup application stores the backup metadata. RMAN also updates the recovery catalog so it knows what data is needed and where it is stored.



An RMAN channel represents one stream of data to a device type and corresponds to one server session. Most backup and recovery commands in RMAN are executed by server sessions. Each channel establishes a connection from RMAN to the target database instance by starting a server session on the instance. The server session performs the backup, restore, and recovery operations.

When you connect RMAN to a target database, RMAN allocates server sessions on the target database instance and directs them to perform the operations. The channel reads data into memory, processes it, and writes it to the output device. With RMAN, you can allocate up to 255 channels; each channel can process up to 64 files in parallel.

You can control the degree of parallelism within a job by the number of channels that you allocate. Allocating multiple channels simultaneously allows a single job to read or write multiple backup sets in parallel, with each channel operating on a separate backup set.



An Oracle RMAN backup set contains one or more Oracle datafiles or archived logs stored in RMAN-specific format. A backup set is created by RMAN’s BACKUP command and it can only be restored by RMAN’s RESTORE command.

When creating backup sets, RMAN can simultaneously read multiple files and write their blocks into the same backup set. Combining blocks from multiple files is called multiplexing. With RMAN multiplexing, multiple streams are sent to an Oracle channel as shown here in this diagram. Multiplexing is implemented by using a combination of the RMAN options FILESPERSET and MAXOPENFILES. The FILESPERSET backup parameter specifies the number of Oracle datafiles to put in each backup set. MAXOPENFILES is the maximum number of datafiles that RMAN can read from simultaneously. The level of multiplexing is the minimum of MAXOPENFILES and the number of files in each backup set (as controlled by the number of CHANNELS allocated by RMAN).



An Oracle RAC database is an active-active clustered database. It consists of a pool of independent servers that cooperate as a single system and provide improved fault resilience and a modular incremental system. Oracle de-couples the Oracle instance from the Oracle database. A clustered database is a single database that can be accessed by multiple instances. Each instance runs on a separate server in the server pool. The RAC system enables multiple Oracle instances across multiple nodes to access the same Oracle database at the same time. All the database files reside on shared disks. When additional resources are required, additional servers and instances can easily be added to the server pool with no downtime. Once the new instance is started, applications using services can immediately take advantage of it with no changes to the application or application server.



The database engine is the most crucial underlying component of SQL Server operations. Data is stored in tables in the database. SQL Server graphical administrative tools such as SQL Server Management Studio and Configuration Manager provide user access to the database via Transact SQL (T-SQL) in the application support layer. Other applications can also submit either SQL statements or queries and request that the database engine return the results in the form of an XML document.



An instance is a copy of SQL Server running on a computer. There can be multiple instances running on the same computer. The default instance takes its name from the computer’s network name.

A SQL Server database is a collection of physical objects: data and transaction log files. It is possible to create many databases within a SQL Server instance. A recommended best practice is to have a single instance with multiple production databases. Each one of the databases comprises a different set of files. Data files are OS files where SQL Server stores the database data. Filegroups are a form of logical storage. By default, all data files are placed in a single filegroup called PRIMARY. Additional filegroups can be used to simplify administrative tasks such as physical disk organization and backup/restore operations. Each database needs at least one transaction log file. Log files contain recovery information. A log file is unique to a particular database.



When a user creates a database, SQL Server creates the structure needed for the database, including an “mdb” and “ldf” set of files for the database.

Every database has a primary data file. It contains startup information for the database catalog. The database catalog contains the definition of all objects in the database as well the definition of the database itself. Its file extension is mdf. User data can be stored in the primary data file or in secondary data files (extension .ndf).

The “ldf” file is the log file supporting a database. Transactions performed through Microsoft SQL Server are logged into the log files. The log files are vital for a recovery process.

The default location for these files is C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Data

Note: EMC and Microsoft recommend that data files and log files should be placed on different disks.



Microsoft SQL Server maintains a number of system databases which are used internally by various systems and functions. Every instance contains four system databases created during the installation process. The system database names are master, model, msdb, and tempdb.

The master database contains information about the SQL Server instance. The master database records the existence and location of all other databases. The model database is a template used for creating user databases. The msdb database is used for scheduling, alerts and jobs. The tempdb holds temporary or intermediate work such as temporary tables and temporary stored procedures. It is re-created every time the Microsoft SQL Server instance is started.



SQL Server uses database objects to organize data. Typical SQL Server logical database objects are:

• Tables

• Indexes

• Views

Other components include stored procedures and triggers.

A table is a database component that looks like a spreadsheet, where each row is a record and each column is a field.

Indexes help to organize and find data. SQL commands will run faster when using index objects.

A stored procedure is a set of compiled statements basically used to improve performance.

A trigger is also a stored procedure created to automatically run when a special event takes place. For example, an update command against a particular table may call a trigger that takes action on another table.

A view combines data from one or more tables.



The basic unit of data storage for SQL Server is a page. The disk space allocated to a data file in a database is logically divided into pages. Disk I/O is performed at the page level: SQL Server reads or writes data pages. Pages in a data file are numbered sequentially, starting with 0 for the first page in the file. Each file in a database has a unique file number. To uniquely identify a page, both the file ID and the page number are required. Each page contains a 96-byte header. The header includes the page number, page type, amount of free space on the page and the allocation unit ID of the object that owns the page. Starting after the header, data rows are put on the page serially.

Extents are a collection of 8 physically contiguous pages, or 64 KB. All pages are stored in extents.

SQL Server databases have 128 pages per megabyte and 16 extents per megabyte.

Database objects and files can be grouped together in filegroups for allocation and administration purposes. The primary filegroup contains the primary data file and any other files not specifically assigned to another filegroup. One filegroup in each database is designated the default filegroup.

Log files do not contain pages; they contain a series of log records. Log files are never part of a filegroup.



Every database has a transaction log that records all transactions and the database modifications made by each transaction. The transaction log is used to guarantee the data integrity of the database and for data recovery. The transaction log supports recovery of individual transactions, recovery of all incomplete transactions when SQL Server is started, rolling a restored database, file, filegroup or page forward to a point of failure, supporting transactional replication, and supporting standby-server solutions. The transaction log is implemented as a separate file or set of files in the database. The log cache is managed separately from the buffer cache for data pages.

Each log record is identified by a log sequence number. Log records are written to the logical end of the log in serial sequence as they are created. Log records associated with a transaction are linked in a chain using backward pointers. Log records for data modifications record either the logical operation performed or they record the before and after images of the modified data.

SQL Server does not write changes to tables directly to the disk. The changes are recorded in the transaction log before they are written to the disk. This is called Write-Ahead Log (WAL). SQL Server maintains a buffer cache into which it reads data pages when data is retrieved. Data modifications are made to the copy of the page in the buffer cache. When a modification is made to a page in the buffer, a log record is built in the log cache that records the modification. Log records are written to disk when transactions are committed.

Modifications are written to disk either when a checkpoint occurs or when the buffer must be used to hold a new page. The process of writing a modified data page from the buffer cache to disk is called flushing. A page modified in the cache, but not yet written to disk is called a dirty page. The log record must be written to disk before the associated dirty page is flushed from the buffer cache to disk.



SQL Server Checkpoint is an internal and periodic operation used to flush or copy data pages from memory to disk. When a checkpoint occurs, SQL Server puts a mark on the log to show that all the pages in memory have been written to disk. The SQL Server recovery process uses this mark in the transaction log to guarantee that no earlier transaction can be used to complete a recovery.



The SQL database recovery model represents a database property that is used to control how the SQL transaction logs are managed. It determines what types of backups are supported and how recoveries can be performed. We will look at two recovery model options: simple recovery model and full recovery model. The choice of which recovery model to use is based on the requirements of the business.



Using the simple recovery model poses a risk of data loss if the database becomes damaged and must be restored because transaction logs are not protected. Data can only be restored to the point of the last backup. Because of this, a balance must be found in which the backup intervals are short enough to prevent significant data loss in the event of a failure, yet still long enough that the backup overhead does not impact the production workload of the server. The backup interval will generally be driven by the Recovery Point Objective for the service being provided; shorter RPOs mean more frequent backups, while longer RPOs reduce the impact of backups on the environment, but potentially place more data at risk.



The full recovery model provides greater protection for data than the simple recovery model.

This recovery model relies on backing up the SQL transaction logs to provide full recoverability and to prevent data loss during the periods between full or differential backups.

With Microsoft SQL Server Full Recovery Model SQL transaction logs are backed up between regular full and differential backups to provide the ability to recover data from a point in time rather than just the last backup.



You can backup entire databases, partial databases, or a set of files or filegroups. For each of these, SQL Server supports full and differential backups. A full backup contains all the data and enough logs to allow for recovering that data. A differential backup is based on the latest full backup of the data and contains all the data that has changed since the latest full backup. Each data backup includes part of the transaction log so that the backup can be recovered to the end of that backup. After the first data backup, under the full recovery model, regular transaction log backups are required. Each log backup includes the part of the transaction log that was active when the backup was created and all log records that were not backed up in a previous log backup.

Native Microsoft SQL Server tools and utilities can be used to perform backups as well as 3rd party backup applications. Backup applications use a backup agent to integrate with SQL Server to perform backups based on the Virtual Device Interface (VDI) or use snapshot technologies such as Microsoft Volume Shadow Copy Service (VSS). Backup applications can also be used to backup the backups created by the native Microsoft SQL Server tool. Backup applications typically will provide media management services and the ability to stage or clone the backups to offsite storage.

SQL Server supports backing up databases through multiple data streams called stripes. This is the number of simultaneous backup streams to be created for a given backup operation. Backup applications may store each stripe as a separate backup save set. The purpose of this feature is to speed up the rate of data transmission to better match data transfer rate capabilities between source and destination media. When performing deduplication backups, compression and multiplexing of the backup data should not be enabled.

The slide depicts an example of a SQL Server backup performed by a backup application. The backup application agent provides an intermediary layer between SQL Server and the backup server/storage node. Backup data is stored on the backup storage device, in this case a Data Domain system, where it is later replicated offsite to another Data Domain system.



Data can be restored at the database, data file(s) and data page levels. The database or files are offline for the duration of the restore and recovery operations.

Restoring is the process of copying data from a backup and applying logged transactions to the data to roll it forward to the target recovery point. The process of rolling forward uncommitted transactions, if any, and bringing the database online is known as recovery.

A restore is a multiphase process. The phases include the data copy, redo (or roll forward) and undo (roll back) phases. Data copy involves copying all the data, log and index pages from one or more full backups and, optionally, differential backups and then resetting the contents of the affected database, files or pages to the time that they were captured by the backups. Redo is process of redoing logged changes to the data in the roll forward set to being the data forward in time. Log backups are processed as they are restored, starting with the log that is contained in full backups. The goal of roll forward is to return the data to its original state at the recovery point as specified. Data is always rolled forward to a point that is redo consistent. All the data has been rolled forward to a point at which undo can occur. The undo phase will rollback uncommitted transactions, if required. After the database is consistent, recovery brings the database online.



In order to understand how Exchange works, first you need to understand how a Client/Server messaging architecture works. This type of architecture distributes the load between the client and the server, with the clients sending the requests and the server servicing them. This provides for a more secure environment and less network congestion because the client is not constantly querying the server for new mail. The diagram depicts the flow of a message from the sending client to a receiving client.



There are several messaging components that ultimately make up the Exchange Store.

An Information Store is the database where all the mail passed through the Exchange server is stored. There are Private stores which hold personal mail, folders, calendars, etc. and Public stores which hold all items put into the Public Folder structure. Each store created, whether public or private, is made up of one database file that contains its contents. The extension for this file is .edb.

Storage Groups contain one or more databases and are the method for organizing Information Stores. They are also the level at which transaction logging takes place; hence, all databases in a given Storage Group are linked to the same common set of transaction log files.

A single storage group can contain multiple individual stores (public or private) and there can be multiple storage groups per server.



Exchange transaction log files record the operations before the committed transactions are written to the database file. The transaction logs are a historical recording of all activity that happens on the server. In the event of a server or database crash, the logs maintain transactional consistency, ensuring that all pending database transactions not committed to the database are able to be “replayed”. In short, they reflect not what has happened to the database, but what will happen to it. So, as long as the database files remain intact, you can use the log files to recover the database up to the last committed transaction. This makes the transaction logs arguably the most important system files for Exchange recovery. Transaction log files have an extension of .log.

The checkpoint file is a pointer that indicates which transaction log was last successfully written to the data file. The extension for this file is .chk.

A mailbox is a folder for storing electronic mail.



Most enterprises typically deploy Microsoft Exchange Server in some form of high-availability configuration. In Exchange Server 2007, this typically involves the use of Windows clusters, Exchange Server 2007 replication, or both.

Exchange 2007 single copy cluster (SCC) is a clustered mailbox server that uses shared storage in a failover cluster configuration to allow multiple servers to manage a single copy of the storage groups. In an SCC, an Exchange 2007 Mailbox server uses its own network identity, not the identity of any node in the cluster. This network identity is referred to as a clustered mailbox server. If the node running a clustered mailbox server experiences problems, the clustered mailbox server goes offline for a brief period until another node takes control of the clustered mailbox server and brings it online.

This process is known as failover. The storage hosting the clustered mailbox server's storage groups and databases is hosted on shared storage that is available to each possible host node of the clustered mailbox server. As the failover occurs, the storage associated with the clustered mailbox server is logically detected from the failed node and placed under the control of the new host node of the clustered mailbox server.



With Exchange 2007, data replication features are built directly into Exchange. These features allow Exchange data to be replicated from one Exchange server to another. Exchange 2007 offers 3 levels of data replication: LCR, CCR and SCR.

Local Continuous Replication (LCR) is a mechanism by which data is mirrored to a second disk by Exchange. It is a single‐server solution that uses built‐in asynchronous log shipping and log replay technology to create and maintain a copy of a storage group on a second set of disks that are connected to the same server as the production storage group.

CCR (Cluster Continuous Replication) enables the copying of data between nodes of an Exchange cluster. This adds hardware redundancy as well as data failure resilience. CCR is a high availability feature that combines the asynchronous log shipping and replay technology built into Exchange Server 2007 with the failover and management features provided by Windows Cluster service. It uses both an active and a passive node to run the Exchange Server. The node that is currently running a clustered mailbox server is called the active node; the node that is not running a clustered mailbox server but is part of the cluster and the target for continuous log shipping is called the passive node. Storage is not shared between the active and passive instances. There are two copies of independent logs and databases.

Standby Continuous Replication (SCR) adds a host, typically at a remote site, to the protection scheme so that should a site failure occur, operations could continue from the DR site with nearly up-to-date data to the SCR node. Note that the source is not required to be a cluster.



Exchange 2007 supports the following backup methods:

• Legacy streaming backup using the Exchange Extensible Storage Engine (ESE) API. Streaming backups are supported for active storage groups.

• Microsoft Volume Shadow Copy Service (VSS). Exchange-aware VSS backups are supported for both active and passive storage groups. The VSS backup method is required for Exchange 2007 CCR passive node backup and recovery. When using VSS, 3rd party backup applications must use the Exchange VSS Writer to backup the files.

These backup types/levels are supported:

• Full – Backs up selected databases and all necessary log files. Databases and transaction logs are backed up. Log files older than the checkpoint are deleted (truncated) after the backup completes.

• Copy – A copy backup is the same as a full except that the log files are not truncated.

• Incremental - All transaction logs are backed up since the last full or incremental; logs are then truncated.

• Differential – All transaction logs are backed up since the last full or incremental, but not truncated.



Exchange 2007 supports recovery to an Exchange Recovery Storage Group (RSG). RSG is a type of storage group in the Exchange server that allows you to mount a copy of a mailbox store (database) onto a production Exchange server. You can then recover data, such as individual mailboxes, from the restored mailbox store while the production store is still running.



Exchange 2010 does not use the concept of Storage Groups. The databases are uncoupled from mailbox servers and are now managed globally, which means databases no longer share log streams and continuous replication operates at the database level.

In Exchange 2010, the Exchange store is a storage platform that provides a single repository for managing multiple types of information in one infrastructure. Primary components of the Exchange store are:

• Mailbox databases containing the data and information that comprise mailboxes. A mailbox database is stored as an Exchange database (.edb) file.

• Public folder databases contain the data and other information that comprise the public folders.

The physical file structure on the Exchange store contains three types of files: Exchange database files (.edb), transaction log files (.log) and checkpoint files (.chk). The Exchange database files are the repository for mailbox data. They’re accessed by the Extensible Storage Engine (ESE) directory and have a B-tree structure designed for quick access. Each database has its own set of transaction logs.

Exchange 2010 mailbox databases can be moved to and mounted on any other Exchange 2010 mailbox server in the same organization.



Exchange transaction logging ensures that an Exchange database can be reliably restored to a consistent state after any sudden stop of the database. Logs are also used when restoring online backups.

Before changes are made to a database file, Exchange writes the changes to a transaction log file. After the change is safely logged, committed transactions can then be written to the database file. Exchange manages the caching of database pages; physically writing the changes to the database file is a low priority during normal operations. When a database is shut down normally, all outstanding data is written to the database files. After normal shutdown, the database file set is considered consistent and Exchange detaches it from its log stream. This means that the transaction logs aren’t required to start the database files.

Cached changes aren’t lost if the memory cache is destroyed such as in the case of a sudden stop of a database. When the database restarts, Exchange scans the log files, and reconstructs and applies any changes not yet written to the database file. The checkpoint file tracks how far Exchange has progressed in writing logged information to the database files. When restarting from a “dirty shutdown” state, all existing transaction logs from the checkpoint forward must be present before the database can be mounted. If not available, the database must be repaired.



A database availability group (DAG) is the base component of the high availability and site resilience framework in Exchange 2010. It replaces these features from previous Exchange versions: cluster continuous replication (CCR), local continuous replication (LCR), standby continuous replication (SCR), single copy cluster (SCC) and clustered mailbox servers.

A DAG is a group of up to 16 mailbox servers that host a set of databases. The DAG provides automatic database-level recovery from failures that affect the individual databases, such as disk failure or server failure. Any server in a DAG can host a copy of a mailbox database from any other server in the DAG. A single mailbox server in a DAG can only host one copy of any particular mailbox database. However, a single mailbox server in a DAG can host active and passive copies of different mailbox databases.

DAG uses the built-in continuous replication feature to replicate mailbox databases among servers in the DAG. After servers are added to a DAG, replicated database copies can be added incrementally and Exchange 2010 switches between these copies automatically to maintain availability. Each server works with the other servers in the DAG to provide automatic recovery from failures that affect mailbox databases.



Exchange 2010 supports VSS-based backups only. To backup and restore Exchange 2010, an Exchange-aware application that supports the VSS Writer for Exchange 2010 must be used, such as Windows Server Backup with the VSS plug-in, Microsoft System Center Data Protection Manager, or a 3rd party Exchange-aware VSS-based application. The VSS plug-in that ships with Exchange 2010 can be used to back up volumes containing active mailbox database copies or standalone mailbox databases; it cannot be used to backup volumes containing passive mailbox database copies. To back up passive mailbox database copies, requires either Microsoft System Center Data Protection Manager or a 3rd party Exchange-aware VSS-based application. A VSS restore cannot be performed directly to a passive mailbox database copy; the restore can be performed to an alternate location and then copied.

With Exchange 2010, the Recovery Storage Group is replaced by a recovery database. The recovery database (RDB) is a special type of mailbox database that allows you to mount a restored mailbox database and extract data from the restored database. This provides the ability to recover data from a backup without disturbing user access to the current data. The recovery database can be used to recover a mailbox and to recover specific email items.



We have taken a look at the protection afforded by Exchange 2010 continuous replication and DAG. Even when employing these features, traditional point-in-time backups may still be required by organizations to ensure sufficient protection for critical Exchange data. All costs associated with various scenarios must also be considered. The slide lists some factors to consider when planning for Exchange backup and recovery.



Please take a moment to read the key differences between Exchange server 2007 and 2010 as shown on the slide.



Microsoft SharePoint is a family of products that interact with Microsoft SQL server and Internet Information server, IIS, to provide a web-based engine and a platform for deploying a wide range of business services. The common solutions using SharePoint are collaboration, content management, enterprise search, and web portals.



SharePoint employs a flexible, service-oriented architecture.

A SharePoint farm is a set of servers comprised of databases, applications, and web services that together provide a SharePoint solution.

SharePoint web applications provide content in a SharePoint farm. Web applications are typically the interfaces through which users interact with SharePoint. Multiple web applications can be created on a server farm. Web applications are commonly independent of each other, having their own application pools, and can be restarted independently in IIS.

A site collection provides a grouping of SharePoint sites. Each web application will typically have at least one site collection. All of the sites within a site collection share a common navigational design. Site collections may be associated with their own content databases, or they may share a content database with other site collections in the same web application.

A site contains web pages and related assets, such as lists and libraries, all hosted within a site collection. A site delivers common features or content to users. A list is a collection of pieces of information that have the same properties. A library is a list where each item in the list refers to a file stored in SharePoint. SharePoint comes with templates for creating sites and several pre-defined lists and libraries.



The display on this slide illustrates the hierarchy within a SharePoint farm environment. A SharePoint farm can contain multiple web applications as shown on this slide. Each web application contains one or more site collections. A site collection contains one or more sub-sites or websites that have the same owner and share administration settings. Lists, libraries, and other sites are stored within sites.



A SharePoint server is an individual server that runs the operating system and application software required to perform roles or to provide services for the SharePoint farm. For example, Web- Front-End servers, application servers, and database servers.

A Web-Front-End (WFE) server provides services directly to end users and handles all web traffic from end users.

SQL server – SharePoint server is an application that is built on Microsoft SQL Server database engine. Most of the content and settings in a SharePoint server are stored in Microsoft SQL relational databases.



A SharePoint configuration database contains data about farm settings, such as the databases used, Internet Information Services (IIS), websites or web applications, solutions, Web Part packages, site templates, default quota, and blocked file types. A SharePoint farm can only have one set of configuration databases.

A SharePoint content database stores all site content. All the data for a specific site resides in one content database and each web application can contain many content databases. Each site collection can be associated with only one content database, although a content database can be shared with many site collections.



The three-tier roles of a Microsoft SharePoint Server farm include the web server role, application server role, and database server role.

Servers with the web server role host web pages, web services, and web parts that are necessary to process requests served by the farm. Requests are directed to the appropriate application servers.

Application server roles are associated with services that can be deployed to physical computers. Service applications are services that are shared across sites within a farm. For example, Search and Excel services. Some services can be shared across multiple farms. Services, such as the Search service and the User Profile applications, may include multiple application components and/or multiple databases.

A server hosting databases has the database server role. In SharePoint, all databases can be deployed to a single server or grouped by roles and deployed to multiple database servers.



A SharePoint Server platform can operate entirely from a single machine or it can be scaled up to be managed across many servers. The three-tier roles of a Microsoft SharePoint Server farm can all be deployed on a single server or distributed across many servers.

A SharePoint standalone farm is a configuration in which all SharePoint services run on one server.

In contrast, a SharePoint distributed farm refers to a group of SharePoint servers that are joined to form a SharePoint distributed farm. In this configuration, different servers provide different SharePoint roles and services. Sets of roles can be hosted by a single server or multiple servers in a distributed farm. A SharePoint distributed farm can be scaled to meet higher demands for availability, usage, and security.



Microsoft SharePoint Server 2010 provides two backup systems: farm and granular.

The farm backup system as depicted in the diagram on this slide, starts a Microsoft SQL Server backup of content and service application databases, writes configuration content to files, and also backs up the Search index files and synchronizes them with the Search database backups. Both full and differential backups are supported. Full backups backup the entire system. Differential backups create a backup of all data that has changed since the last full backup. Components in the farm that can be selected for backup include the farm, web applications, and shared and non-shared services and service applications.

With the granular backup system, a user can back up a site collection or export a site or list.



This lesson covered the basic components of a relational database system and discussed how data is stored and backed up in various database applications, including Oracle, Microsoft SQL, Microsoft Exchange and Microsoft SharePoint.



This lesson provides an overview of Microsoft Volume Shadow Copy Service, VSS, technology including the components of the VSS framework. Then it covers the role of each of these components when performing a backup.



Volume Shadow Copy Service (VSS) is a Microsoft framework for creating consistent point-in-time copies of data known as Shadow Copies. VSS provides the backup infrastructure for Microsoft Windows XP, Windows Server 2003 and all later releases of the Microsoft Windows operating system.

VSS allows third-party backup applications to perform volume backups of data for applications, like Exchange and SQL Server, file systems or system files, including files currently open and active. By backing up the shadow copy (or snapshot), files that are currently open for writing can be backed up because the backup application is actually backing up a copy of the file. VSS produces clean (uncorrupted) snapshots of a volume by enabling applications to flush partially committed data from memory in a coordinated fashion with the snapshot requestor and the hardware. This ability prevents “torn writes” – the occurrence of the system not being able to complete the write of a block of data to disk.



VSS is designed to address common backup issues for Windows XP, Server 2003, and later versions. However, earlier versions of Windows still require Open File Manager.

Applications that are running frequently need to keep files open in exclusive mode. This prevents backup programs from copying them.

There is a finite time requirement to open, backup, and close a file. If an application keeps multiple files open for updates, it is possible that files copied to the backup media may not all reflect the same application state.

To overcome the problems listed on the slide, a backup program could require the suspension or termination of all running programs to ensure file accessibility and integrity of data. Service interruption is not tolerated in many environments.

Prior to VSS, each storage vendor provided a volume capture mechanism. Therefore, each backup vendor needed to support multiple implementations:

Device vendors that supported volume capture did not support the coordination of running applications to freeze data on disk.

If there are multiple vendor disks on a system and there is no mechanism to coordinate data freeze, the image created for each volume may not be consistent with each other.

Few conventional disk vendors support volume capture so DAS drives cannot participate in data capture through the use of snapshots. This particularly affects the backup of system state.



The VSS solution enables developers to create applications that can be effectively backed up by any vendor’s backup application. One of the most important features of VSS is its support for the point-in-time copy method. This method is endorsed by Microsoft as VSS offers support for its applications and operating systems.

Note that this is the only backup interface for some applications, such as SharePoint, DPM, and the Vista and Server 2008 operating systems.

VSS is like mission control for snapshots. It enables context for backup through the application, OS, and the hardware.

A snapshot is taken once each component provides a green flag. This ensures consistency and usability of the snapshot.

VSS helps to create consistent point-in-time copies so that the backups are restartable snapshots of the systems and applications. To reduce the impact on production servers, VSS captures off-host backups in conjunction with VSS hardware providers.

VSS provides the option to capture copies of open files that would otherwise be locked and skipped during the backup process.



VSS coordinates the actions of the various components required to create a consistent shadow copy of the data that you want to backup. The actions to coordinate are between the backup application, the service or application that contains the data, and the I/O subsystem to get a consistent snapshot of the data. If your operating system, applications, backup software, and SAN manufacturer all support VSS, you can create flexible storage solutions that can easily be protected without the need to stop servicing clients.

There are three components in the VSS framework: the Requestor, the Writer and the Provider.



A requestor is an application that uses the VSS API to request the services of the Volume Shadow Copy Service to create and manage shadow copies of one or more volumes. VSS works with the requestor to gather information to properly save applications, services and other files on the file system. Requestor roles include accepting user requests to make shadow copies, managing the cataloging and archival of shadow copies, and managing the lifetime of shadow copies. A backup application is an example of a requestor.



The requestor maintains control over VSS backup and restore by generating COM events through the VSS Requestor API. The requestor maintains its state information in a file called the Backup Components Document (BCD). This document contains information about which components were explicitly included in a backup or restore operation and state information about the backup or recovery operation.



A writer is an application-specific component. A writer provides details of the data to the requestor, such as the location of the data and the method of backup and restore.

Writers differentiate VSS from other snapshot solutions in that there is a communications structure between the requestor and writer governing how the snapshot should take place and how the snapshot should be recovered.

A writer is available and active only if its application or service is also available and active on the system. If a service or application is present on a system but is not active, information from its writer will not be available. Consequently, writers can appear or disappear from backup to backup.

Examples of Microsoft applications and services having writers include Active Directory, Windows Registry, Event Log Writer, IIS, and MSDE. Other companies may develop VSS writers for their applications. Applications that are not VSS-aware do not have a specific writer, however, backups can still make use of the default writer (system writer) to back up the NTFS file system on which the application resides.



The writer helps in discoverability, data consistency, post-backup processing, and restore processing.

Discoverability means that VSS writers make it possible for a requestor to discover the location of an application’s data stores and requirements for their backup and restore.

Data Consistency means that writers ensure that the application’s data on disk is in a consistent state prior to the creation of the snapshot.

In post-backup processing, writers may perform clean-ups like truncating logs after the backup is complete.

In restore processing, some applications may require preparation prior to a restore, such as releasing handles or dismounting databases.

The writer is responsible for freezing and thawing the application’s data during creation of a snapshot. Writers provide application state to the requestor in the Writer Metadata Document, an XML file.

Freeze Event: A writer is part of an application’s code that listens for requests from the Volume Shadow Copy Service to freeze the data for which it is responsible. Once the application’s data is frozen (quiesced), the writer generates an XML file containing information about the location of the data and the method that should be used to backup and restore the data. This information is used by the requestor during backup and recovery.

Thaw Event: After a snapshot is created, a writer is responsible for thawing the data that was previously frozen. In other words, normal disk activity is resumed.

Examples of Microsoft applications and services having writers include Active Directory, Windows Registry, Event Log Writer, IIS, and MSDE. Other companies may develop VSS writers for their applications.



A provider is responsible for the creation and maintenance of the shadow copy. Once the provider has received the XML file describing the application data, it uses an I/O interceptor to intercept write requests and copy blocks of data from the source volume to the virtual volume before allowing the write operation to proceed.

A snapshot of a volume does not actually make a copy of the data. Instead, pointers are used in conjunction with copies of blocks that are modified in order to keep a record of the state of the data at the time of the snapshot.

There are two types of providers, software and hardware.

• A software provider is a layer of software above the operating system file system software which intercepts write requests and copies blocks of data to a virtual volume before passing the request along to the operating system.

• For a hardware provider, the work of creating the shadow copy is performed by the host bus adapter, RAID controller, or NAS device.

Windows XP, Server 2003, and 2008/2008 R2 come with a system software provider which is responsible for intercepting changes to local disks for which there are writers.

Many storage hardware vendors, including EMC, have developed hardware providers for their storage arrays.



Initially, the virtual volume contains no data. It is just disk space allocated in the event that it is needed. VSS utilizes a technique called copy-on-write to manage the snapshot.

In the copy-on-write technique, once the virtual volume is established, any write request destined for the source volume is intercepted. Before the request is allowed to continue, the contents of the sectors about to be overwritten on the source volume are copied to the virtual volume. After the data is copied, the write operation is allowed to proceed, changing the contents of the source volume. Subsequent writes to the same block are not captured and stored.

When the snapshot is read, as when a backup is performed, sectors on the source volume which have not been modified are accessed and read directly. For sectors that have been modified since the snapshot occurred, the virtual volume is read. Hence, if the source volume has not changed between the time of the snapshot and the time of the backup, all data is read from the source volume.



The Volume Shadow Copy Service (VSS) manages the creation of a point-in-time shadow copy, or snapshot, of a disk volume or set of data. Let us look at how they work together to create a snapshot.

1. The requestor, such as a backup application, requests VSS to create a snapshot of a particular volume or data set.

2. VSS notifies the application-specific writer to prepare its data for making a shadow copy. The writer creates an XML description of the backup components and defines the restore method. The writer prepares the data by completing all open transactions, rolling transaction logs and flushing caches. VSS then directs the writer to temporarily freeze requestor I/O write requests for the time required to create the shadow copy. VSS flushes the file system buffer and then freezes the file system, which ensures that file system metadata is written and that the data is written in a consistent order.

3. VSS notifies a provider to create and maintain the shadow copy until it is no longer needed. A point-in-time copy of the complete volume mapping is created using XML.

4. Once the snapshot has been successfully created, VSS thaws the file system and instructs the writers to resume normal activities, or thaw. VSS provides the location information for the shadow copy back to the requestor.

The requestor uses the snapshot to create the backup. The backup is as of the point in time that the snapshot is taken. The snapshot can also be backed up to secondary storage through a proxy client by either using an existing snapshot or with the use of temporary snapshots.



VSS configuration options are accessible through Window’s Disk Management console. You can access the Disk Management Console by entering the DISKMGMT.MSC command at the Windows Run prompt. Use the Disk Management console to verify that the volume you plan on running VSS is formatted as NTFS. Note that VSS does not support FAT or FAT-32 volumes.

Select the volume on which to store shadow copies and display its properties (Properties -> Shadow Copies -> Settings). Windows automatically limits the amount of space that can be consumed by shadow copies, although shadow copies require a minimum capacity of 300 MB. By default, the virtual volume is configured to be 10% of the size of the source volume. You can configure the properties such that Windows does not limit the space. With enough free disk space, VSS can maintain up to 64 versions of each file. As disk space starts to run low, Windows will automatically delete older shadow copies to make room for new data.

By default the virtual volume is located on the same partition as the source volume. To improve performance, it is suggested that you change the location of the virtual volume to a different partition. For VSS SYSTEM save sets, the virtual volume is located on the system drive. For systems such as Active Directory, a large amount of data exists and 300MB free space may not be sufficient, therefore the system drive should be provisioned with plenty of free disk space. Optionally, for better backup performance, the virtual volume can be located other than the system drive. However, some applications may not recommend this.

Note: Do not enable Shadow Copies in Properties. Request and schedule backups through the backup application acting as the requestor.



This lesson covered the purpose and components of Microsoft Volume Shadow Copy Service, VSS, and the role of VSS components when performing a backup.



This lesson covers the characteristics of file servers and backing up from a file server. We look at NDMP, its components and the backup challenges that NDMP addresses. Then, we describe several NDMP backup implementations.



File sharing refers to storing and accessing files over a network. In a file sharing environment, a user who creates the file (the creator or owner of a file) determines the type of access to be given to other users (read, write, execute, append, delete, and list) and controls changes to the file. When multiple users try to access a shared file at the same time, a protection scheme is required to maintain data integrity and, at the same time, make this sharing possible.



The use of NAS heads impose a new set of considerations on the backup and recovery strategy in NAS environments. NAS heads use a proprietary operating system and file system structure supporting multiple file-sharing protocols.

In application server-based backup, the NAS head retrieves data from storage over the network and transfers it to the backup client running on the application server. The backup client sends this data to a storage node, which in turn writes the data to the backup device. This results in overloading the network with the backup data and the use of production server resources to move backup data.



In serverless backup, the network share is mounted directly on the storage node. This avoids overloading the network during the backup process and eliminates the need to use resources on the production server. In this scenario, the storage node, which is also a backup client, reads the data from the NAS head and writes it to the backup device without involving the application server. Compared to the previous solution, this eliminates one network hop.



With the adoption of NAS devices by the industry, several challenges were realized. The NAS backup methodologies did not allow centralized storage management infrastructures to be utilized, i.e., a single management console to control enterprise wide storage. Most NAS devices run on proprietary operating systems designed for very specific functionality and therefore do not generally support “Open System” management software applications for control; differing data storage formats between the storage arrays; or differing security structures on the two most common network file systems, NFS and CIFS.

Backups implemented via one of the common protocols would not effectively backup any data security attributes on the NAS device that was accessed via a different protocol. For example, CIFS LAN backup, when restored, would not be able to restore NFS file attributes and vice-versa. With a dual accessed file system, NFS and CIFS gave rise to the concern that if the file system was corrupted and there was no formal, independent methodology for recovering it, then the permissions and rights of the file system could be compromised on recovery and neither protocol would understand the other’s schema. Therefore, when pre-NDMP backups were performed, the image on tape was that of the specific protocol used to perform the backup.

This environment is growing more complex and requires backup solutions that support backup of all platforms on which the enterprise's mission-critical data is stored.



NAS backup challenges are addressed with the Network Data Management Protocol, NDMP. NDMP is both a mechanism and protocol utilized on a network infrastructure to enable the control of backup, recovery, and transfer of other data between NDMP-enabled primary and secondary storage devices. TCP/IP is the transport protocol.

XDR is the data output language where all data is read from and copied back to disparate operating systems and hardware platforms without losing the data integrity. The NFS file system and Microsoft use XDR to describe its data format. By enabling this standard on a NAS device, the proprietary operating system ensures that the data storage format conforms to the XDR standard and therefore allows the data to be backed up and restored without file system structure loss with respect to different rights and permission structures, as in the case of dual accessed file systems.

Major advantages of NDMP backup include:

• Management and control through the data management or backup application

• Ability to avoid sending backup data over the network

• Ability to back up both NFS and CIFS attributes

Additionally, the ability to backup the filesystem from a block-level representation can provide a significant performance benefit, particularly in the case of dense file systems.



NDMP is a standard protocol implemented on the NAS device that responds to backup software requests for backup and restore functions. In traditional backup methods, NDMP backups only use the LAN for metadata. The actual backup data is directly transferred to the local backup device by the NAS device.

Compliance with NDMP standards ensure that the data layout format, interface to storage, management software controls, and tape format are common irrespective of the device and software being used.



The NDMP infrastructure is comprised of NDMP clients and NDMP servers. NDMP clients are client machines with NDMP backup software installed. The primary NDMP servers are the hardware systems that have access to the file systems containing the data to be backed up, such as blades. The final component is the secondary server, which is the hardware system that has access to the backup storage.



There are several different NDMP configuration models. Each one of the models target specific user needs and applications. In all the scenarios, the backup server and the NAS device are NDMP-compliant. The backup application controls the backup/restore process and handles file and scheduling information.

Backup of host data can be done at the file level and at the volume level (Volume Based Backup). Backups can be saved to a tape library unit, to a tape device physically attached to a blade or data mover, or to a virtual library created in a blade, known as NDMP2D. NDMP backups can also be saved to a virtual tape library and special purpose devices, such as an Avamar server.

Although NDMP backups are agnostic, the data can only be restored to a system that recognizes the original data format. For example, a NetApp NDMP backup cannot be restored directly to a Celerra as the data formats for the two systems are incompatible.



In NDMP, backup data is sent directly from the NAS head to the backup device, while metadata is sent to the backup server. In this model, network traffic is minimized by isolating data movement from the NAS head to a locally attached backup device. Only metadata is transported on the network. This backup solution meets the strategic need to centrally manage and control distributed data while minimizing network traffic.



In an NDMP three-way file system, data is not transferred over the public network. A separate private backup network is established between all NAS heads and the “backup” NAS head. This prevents any data transfer on the public network in order to avoid any congestion or affect production operations. Metadata and NDMP control data are still transferred across the public network. NDMP 3-way is useful when there are limited backup devices in the environment. It enables the NAS head to control the backup device and share it with other NAS heads by receiving backup data through NDMP.



Depending upon the backup software vendor, there are two additional NDMP features that are supported:

• Direct Access Restore or DAR

• Dynamic Drive Sharing or DDS

DAR is the ability to keep a track of tape position for individual files in NDMP backups so that the tape server can seek directly to the file during restore. Without DAR support, a single file restore requires reading through the entire index. Another form of DAR is the Directory DAR or DDAR, which is an improved version. DDAR supports directory-level DAR by restoring all the content under a particular directory.

DDS enables tape drives within individual tape libraries to be shared between multiple NAS devices and/or storage nodes in a SAN. By allowing storage nodes to write data to all available drives, more drives can be assigned per backup group in comparison to an environment whereby drives are dedicated to specific servers. As a result, DDS maximizes library utilization, enables backups and recoveries to be completed sooner, and increases library ROI.



Volume-Based Backup (VBB) is an EMC-specific type of NDMP backup mechanism that backs up data blocks at a volume level, rather than at a file level. VBB reads a set of data blocks in a more efficient manner than the traditional file-based backups. VBB works only with EMC-qualified vendor backup software.

VBB can be used for a full or an incremental backup. It also supports two types of restores, full destructive restore, which delivers the best possible write performance, and file-level restore. The VBB backup type must be specified to invoke the VBB functionality before a backup can be initiated.



This lesson covered the characteristics of file servers and backing up from a file server. We discussed NDMP, its components and the backup challenges that NDMP addresses. Then, we described several NDMP backup implementations.


Copyright © 2013 EMC Corporation. Do not copy - All Rights Reserved. 105

Module 3: Backup Client

In this lesson, we look at the various forms of virtualization including virtual memory, networks, servers and storage, and characteristics of a virtual infrastructure. Lastly, we describe the options for backing up VMware virtual machines.


As storage networking technology matures, larger and complex implementations are becoming more common. The heterogeneous nature of storage infrastructures has further added to the complexity of managing and utilizing storage resources effectively. Specialized technologies are required to meet stringent service level agreements and to provide an adaptable infrastructure with reduced cost of management. The virtualization technologies discussed in this module provide enhanced productivity, asset utilization, and better management of the storage infrastructure.

Virtualization is the technique of masking or abstracting physical resources, which simplifies the infrastructure and accommodates the increasing pace of business and technological changes. It increases the utilization and capability of IT resources, such as servers, networks, or storage devices, beyond their physical limits. Virtualization simplifies resource management by pooling and sharing resources for maximum utilization and makes them appear as logical resources with enhanced capabilities.



As previously mentioned, virtualization has been in use for many years. Here are some examples of virtualization:

• Virtual memory

• Virtual networks

• Virtual servers

• Virtual storage



Virtual memory makes an application appear as if it has its own contiguous logical memory independent of the existing physical memory resources.

Since the beginning of the computer industry, memory has been and continues to be an expensive component of a host. It determines both the size and the number of applications that can run on a host.

With technological advancements, memory technology has changed and the cost of memory has decreased. Virtual memory managers (VMMs) have evolved, enabling multiple applications to be hosted and processed simultaneously.

In a virtual memory implementation, a memory address space is divided into contiguous blocks of fixed-size pages. A process known as paging saves inactive memory pages onto the disk and brings them back to physical memory when required. This enables efficient use of available physical memory among different processes. The space used by VMMs on the disk is known as a swap file. A swap file (also known as page file or swap space) is a portion of the hard disk that functions like physical memory (RAM) to the operating system. The operating system typically moves the least used data into the swap file so that RAM will be available for processes that are more active. Because the space allocated to the swap file is on the hard disk (which is slower than the physical memory), access to this file is slower.



Network virtualization creates virtual networks whereby each application sees its own logical network independent of the physical network. A virtual LAN (VLAN) is an example of network virtualization that provides an easy, flexible, and less expensive way to manage networks. VLANs make large networks more manageable by enabling a centralized configuration of devices located in physically diverse locations.

Consider a company in which the users of a department are separated over a metropolitan area with their resources centrally located at one office. In a typical network, each location has its own network connected to the others through routers. When network packets cross routers, latency influences network performance. With VLANs, users with similar access requirements can be grouped together into the same virtual network. This setup eliminates the need for network routing. As a result, although users are physically located at disparate locations, they appear to be at the same location accessing resources locally. In addition to improving network performance, VLANs also provide enhanced security by isolating sensitive data from the other networks and by restricting access to the resources located within the networks.

A virtual SAN/virtual fabric is a recent evolution of SAN and conceptually, functions in the same way as a VLAN.



Server virtualization enables multiple operating systems and applications to run simultaneously on different virtual machines created on the same physical server (or group of servers). Virtual machines provide a layer of abstraction between the operating system and the underlying hardware. Within a physical server, any number of virtual servers can be established; depending on hardware capabilities. Each virtual server seems like a physical machine to the operating system, although all virtual servers share the same underlying physical hardware in an isolated manner. For example, the physical memory is shared between virtual servers but the address space is not. Individual virtual servers can be restarted, upgraded, or even crashed, without affecting the other virtual servers on the same physical machine.

With changes in computing from a dedicated to a client/server model, the physical server faces resource conflict issues when two or more applications running on these servers have conflicting requirements (e.g., need different values in the same registry entry, different versions of the same DLL). These issues are further compounded with an application’s high-availability requirements. As a result, the servers are limited to serve only one application at a time, as shown in Figure. On the other hand, many applications do not take full advantage of the hardware capabilities available to them. Consequently, resources such as processors, memory, and storage remain underutilized.

Server virtualization addresses the issues that exist in a physical server environment. The virtualization layer, shown on the slide, helps to overcome resource conflicts by isolating applications running on different operating systems on the same machine. In addition, server virtualization can dynamically move the underutilized hardware resources to a location where they are needed most, improving utilization of the underlying hardware resources.



Storage virtualization is the process of presenting a logical view of the physical storage resources to a host. This logical storage appears and behaves as physical storage directly connected to the host. Throughout the evolution of storage technology, some form of storage virtualization has been implemented. Some examples of storage virtualization are host-based volume management, LUN creation, tape storage virtualization, and disk addressing (CHS to LBA).

The key benefits of storage virtualization include increased storage utilization, adding or deleting storage without affecting an application’s availability, and non-disruptive data migration (access to files and storage while migrations are in progress). Figure illustrates a virtualized storage environment. At the top are four servers, each of which has one virtual volume assigned, which is currently in use by an application. These virtual volumes are mapped to the actual storage in the arrays, as shown at the bottom of the figure. When I/O is sent to a virtual volume, it is redirected through the virtualization at the storage network layer to the mapped physical array.



In traditional data centers, there is a tight relationship among particular computers, disk drives, and network ports and the applications they support. The Virtual Infrastructure allows us to break those bonds. We can dynamically move resources where they are needed, and move processing where it makes most sense. Also it detaches the operating system and its applications from the hardware they run on.

To understand the transformative value Virtual Infrastructure brings to backup and disaster recovery, it is important to briefly cover the four core properties of Virtual Infrastructure.

Partitioning: Partitioning enables consolidation of multiple applications and operating systems on the same machine, which drives up server utilization. This lowers capital costs and provides significant operational savings that will help fund your disaster recovery plan thereby making disaster recovery more affordable.

Hardware Independence: Virtual machines are hardware independent and can run on any x86 hardware without requiring any changes or modifications. This property significantly accelerates recovery by simplifying system startup and configuration (for recovery) at the disaster recovery site. It also minimizes the complexities, slowness and uncertainties of using traditional recovery mechanisms such as system images, bare-metal restore, and error prone tape recovery. You can also use any server hardware for recovery at your disaster recovery site, thus making it possible to avoid the cost of purchasing identical brand new servers for recovery.



Encapsulation: Encapsulation means that an entire server–operating system image, application, data, configurations, and state–is now simply stored as a file on disk. This encapsulation property transforms and simplifies tasks such as server migration, backup and recovery, replication and disaster recovery server provisioning. These tasks can instead be treated as a simple data migration, file copy or file export activity. There is no need to build an image from scratch or use multiple complex tools for recovery of system state and configuration.

Isolation: Changes or instability in one virtual machine are completely isolated from other virtual machines on the same host. In the Virtual Infrastructure environment, you can run disaster recovery tests on the actual disaster recovery hardware without impacting the ability to recover production virtual machines if your production site fails during a disaster recovery test. You can also eliminate idle hardware at your disaster recovery site by simultaneously running a test-dev or batch program workload, enabling you to maximize the utilization of your IT assets.



Virtual Machine File System (VMFS) is a high performance cluster file system that allows virtualization to scale beyond the boundaries of a single system. Designed, constructed, and optimized for the virtual server environment, VMFS increases resource utilization by providing multiple virtual machines with shared access to a consolidated pool of clustered storage. This simplifies virtual machine provisioning by efficiently storing the entire machine state in a central location.

In the figure on the slide, each of the three physical servers has two virtual machines (VMs) running on it. The arrows pointing to the virtual machine disks (VMDKs) are logical representations of the association between and allocation of the larger VMFS volume, which is made up of one large logical unit number (LUN). A virtual machine sees the VMDK as a local SCSI target. The virtual disks are really just files on the VMFS volume, shown in the illustration as a dashed oval. Each physical server stores its virtual machine files in a specific subdirectory on the VMFS file system. When a VM is operating, VMFS has a lock on those files so that other physical servers cannot update them. VMFS ensures the VM cannot be opened by more than one physical server in the cluster.

It simplifies Disaster Recovery, because VMFS stores a VM’s files in a single subdirectory, disaster recovery, testing and cloning are greatly simplified. The entire state of the VM can be remotely mirrored and easily recovered in the event of a disaster. And with automated handling of virtual machine files, VMFS provides encapsulation of the entire VM so that it can easily become part of a DR solution.



Virtualization makes it easy to copy, clone and replicate system resources—all essential to system and application recovery. This property significantly benefits the provisioning and recovery processes of a disaster recovery scenario whether it is backup and recovery or remote disaster recovery using replication.

The conceptual diagram provides an example of how a data center’s time to recovery was improved from over 40 hours in their prior physical-to-physical recovery scenario to 4 hours in a virtual-to-virtual recovery process. Because complex operating system files can now be stored on shared storage, transferring them to the recovery site is dramatically simpler. You can employ a wide variety of replication and backup technologies to copy systems and data offsite, an intrinsic component of a disaster recovery plan.



At the simplest level, virtual machines run guest operating systems which in turn run applications, which in turn provide an IT service to the business. Virtual machines are encapsulated on the storage, are isolated from one another and not dependent on the same hardware being available at the failover location. It is this service that must be protected from a BC/DR point of view. The figure on the slide illustrates how the inbuilt properties of virtual machines combined with a replicated architecture allows resources to be mapped from one virtual environment to another and at the same time enables failover of an IT service. Storage replication forms the foundation of the recovery process. Storage is replicated at the failover site, then presented to the virtual infrastructure architecture at the failover location, at which point the virtual machines can then be activated.



The VMware vSphere solution is used to build virtual server and desktop infrastructures that improve the availability, security and manageability of mission-critical applications. vSphere virtualizes servers, storage and networking, allowing multiple operating systems and their applications to run independently in virtual machines while sharing physical resources.

VMware ESXi installs directly on top of the physical server and partitions it into multiple virtual machines that can run simultaneously, sharing the physical resources of the underlying servers. Each virtual machine represents a complete system with processors, memory, networking, storage and BIOS and runs an operating system and applications.

In addition to server consolidation capabilities, the various VMware vSphere editions offer advanced capabilities such as high availability, live migration, power management, automatic load balancing and more.

VMware vCenter Server management software enables centralized management of multiple vSphere hosts.



There is flexibility in how backup applications provide a VMware backup solution. Two ways are shown here: guest and VMware image backup. Data centers can leverage these different VMware backup options to create a backup environment that meets their individual backup requirements.

With a guest backup, the virtual machine is treated like a normal client with the backup agent installed on it.

Using VMware vStorage APIs for Data Protection (VADP), backup software can backup virtual machines without using backup agents inside the virtual machines – the virtual machines being backed up are not impacted.



With the guest backup option, backup client software is installed on the individual virtual machines. Backup configuration for this method is identical to that of a physical machine. Backup agents are installed on the virtual machines and send their backup data to the backup application for storage.

The main advantages of VMware guest backup is that it lets backup administrators leverage identical backup methods for physical and virtual machines. Machines with database applications can use the backup agents for the respective databases thus ensuring consistent, online backups. There is no requirement for advanced scripting or VMware software knowledge and it means unchanged day-to-day procedures for backup. However, since each virtual machine has a separate backup client installed, ESX servers with a large number of virtual machines may experience a strain on resources, especially memory, if all machines are backed up at the same time.



vStorage APIs for Data Protection, VADP, enables backup products to do centralized, efficient, off-host backups of vSphere virtual machines. Introduced in vSphere 4.0, VADP replaces the VMware Consolidated Backup Framework, VCB, for virtual machine backups. VADP integrates directly with backup applications; no additional software is required to be downloaded and installed. With VADP, backup applications can backup and restore entire virtual machine images across SAN or local area networks. Incremental backups and restores are supported through the use of change block tracking. Also, file-level backup and restore of virtual machines is available, depending on the capabilities of the backup application.



Benefits of integrating VADP with backup applications include the ability to perform backup and restores without installing agents on individual virtual machines. Backups can be performed from a centralized location thus offloading backup processing from the ESX hosts. Because backups are non-disruptive to the virtual machines being backed up, they can be performed any time. This provides greater flexibility in scheduling backups and increased backup window times.



VMware Image Backup uses a virtual or physical machine as a proxy server to handle backup processing. Using VMware vStorage APIs for Data Protection (VADP), backup software can back up VMware virtual machines without using backup agents inside the virtual machines. This is accomplished by creating a snapshot of the virtual machine, then backing up that snapshot. No downtime for the virtual machine is required. Backup processing can be offloaded from the ESX hosts, depending upon the method of deploying the proxy. File-level backups and restores are supported with image level backups.



This lesson covered the various forms of virtualization and discussed the characteristics of a virtual infrastructure. Then, we discussed the options for backing up VMware virtual machines.



In this lesson, we look at the considerations and challenges facing a client backup environment. Then we will look at considerations for several special backup client situations including large clients, clients in different time zones, remote clients and desktop laptop clients.



When developing a backup plan for a particular client or group of clients it is important to match client characteristics with the backup application’s capabilities and features. Make sure that the backup software supports the client environment and is suitable for the type and amount of data.

Consider recovery time objectives when planning the level of granularity to employ for backups. Also ensure that recovery resources are available and recovery plans are tested.



It is important to plan the configuration of backup jobs before running initial backups. A basic rule for configuring scheduled backups is to keep it simple. Creating a standard backup client configuration can significantly decrease risk and increase resource utilization.

In most cases, clients with similar dataset, retention, and schedule requirements can be included in the same backup group. Note that separate groups may be required when backing up certain databases, large file systems and clients in different time zones - more on this later.

There may be situations where you want to configure one or more specific datasets for a client. For example, create more than one dataset for a client in order to back up subsets of data on different days of the week. You may also need to split a client’s data into more than one dataset definition in situations where there is not enough memory for all client data to be backed up as one dataset or there is too much data traveling over a LAN for backup.

Make sure the retention policy selected is sufficient to retain the backup data to accommodate the RPO for the client. Remember that all clients will not have the same RPO requirements – but will be different depending upon the criticality of the data to the organization.



Proper scheduling of backups is one of the most important factors influencing system reliability and availability. Schedule backups to run outside of any application and backup software maintenance schedules, periods of time of high network usage by production systems and peak production processing times.

Backups should complete within the window of time allocated for backups daily. When planning, allow for times when backups may take longer than usual. The actual amount of time required to back up a client depends on the total number of files, total amount of database and file system data, the chosen backup level (granularity) and the hardware performance characteristics of the client.



Large file systems or clients with large files may not be able to back up their data serially. Depending upon the client, database and/or backup application, there are several suggestions for working with large clients (large files, large file systems).

Use multistreaming where supported, for example, database management systems, NDMP. Multistreaming is defined as the process of sending multiple backup jobs from a single client. Multistreaming creates a different job or session for each file system or multiple streams for a database. In general, use the number of streams to match backup target storage.

Consider breaking up a client’s backup into multiple, separate save set backups. Stagger the backups so that they do not all start at the same time.

Reduce the amount of data to be backed up. Consider employing a backup application that uses deduplication technology. Another option is to archive old data or data that is not regularly accessed to a data archive: backup only active data.

Convert clients into storage nodes.

Backup only changed data. Reduce the frequency of full backups if RTO is not a significant requirement.

In environments with extremely large file systems containing many small files, consider employing a product that uses block tracking technology such as SnapImage.



Backing up many desktop/laptop computers has the potential to significantly impact operational efficiency and backup storage capacity.

One of the challenges with backing up desktop and laptop clients is determining when to back up the clients. Normally backup windows are at night; however, desktop and laptop clients will need to be backed up during the day when the computers are connected to the network. This requirement must be considered when assessing the amount of interruption that backing up these clients will cause to production applications and systems.

To best manage storage capacity, back up only user files and folders. The dataset should exclude data that is common to all computers and also data that can be easily recreated from other sources. For example, back up critical work files that are not replaceable. Do not back up operating system data and program files that can be reinstalled.

For desktop and laptop clients in relatively the same geographic area, use a single backup group to backup the clients with the same dataset and backup schedule. If there are a significant number of clients in a location, consider backing up to their own storage node/backup server.

If laptop and desktop users have the facility to perform ad-hoc backups, it is critical that they understand any limitations on the amount of data and the types of data to be backed up, and when backups can be run.



Backing up clients in different time zones has the potential to significantly impact operational efficiency as well. Be aware that when these clients are available for backup, it may not be the optimum time for backups to take place at the central backup facility. Consider backing up these clients to a dedicated local storage node/backup server.



It is important to backup all remote locations in an organization in addition to centralized data center backups to effectively meet backup and disaster recovery objectives. Local data center backups typically do not have bandwidth and security related concerns as all the data backed up is local to the data center and there is no worry about security as the data does not travel outside the data center.

A distributed environment brings in its own set of challenges that need to be addressed with a unique set of technologies. The primary concerns of a distributed environment in a backup context are bandwidth, security, remote deployment, and disaster recovery.

Since traditional backup applications use large amounts of bandwidth to backup their clients, with the bandwidth limitations for remote environments (like banks which have hundreds of branch offices but have few data centers to manage their storage and backups), it is important to have a solution that uses minimal bandwidth and completes the backups on time. The larger the amount of data to be backed up and the narrower the “pipe”, the more likely that the backup will not complete in a reasonable length of time. Also, the possibility of network outages during backups are an important factor to consider. For example, how does a backup application respond to a dropped connection?

As data might travel via public networks (like with sales teams that are spread across the country and have many employees that work from home), it is important that the data travels securely.

Backup planning must include disaster recovery plans for remote offices as well as centralized data centers. If backup servers/storage nodes are deployed in remote locations, consider offsite storage of backup copies or replication to offsite storage.



Where it is anticipated that the WAN connection between a remote office and a central backup facility, consider placing a backup storage node/backup server in the remote location. This is especially important where RTO is a primary factor. This decision must be weighed against other considerations including the time required to replicate or clone local backup data offsite for disaster recovery purposes and the additional expense of deploying, administering and maintaining the remote backup servers/storage nodes and backup media.



This lesson covered the considerations and challenges relating to client backups, and several special backup client situations including large clients, clients in different time zones, remote clients and desktop laptop clients.



This lesson describes factors impacting client backup and restore performance, including network considerations such as bandwidth and latency, TCP retransmissions, link errors and duplex mismatches.



The backup client may be the source of backup performance issues and may not be capable of sending the data fast enough to the backup storage node. Factors impacting client performance may include one or more of the following:

• High CPU utilization, slow CPU, slow I/O performance: The backup client process doesn’t work well because the processing power of the server is compromised. Backups can require significant CPU resources. The CPU and I/O subsystem must support production as well as backup and restore activity.

• Backup software running on the backup client can require significant memory resources as well. Ensure that backup processing does not negatively impact client production activities. If possible, a large page file/swap space may help increase performance of backups as well as applications running on the client.

• Simultaneously backing up multiple guest machines on an ESX server will slow down each of the clients due to limitations of CPU, memory and/or I/O throughputs.

• Filesystem fragmentation / seek time: In cases where file system fragmentation is high, the backup client will suffer to get all the data together before sending it over the network.

• Operating system and the data to be backed up: Operating systems perform differently under different scenarios / applications. Also the data itself could be the bottleneck. A backup of a large number of files usually takes longer than backing up a smaller number of files with the same overall size, due to all of the open file calls that need to be made as well as indexing/lookups for backup catalog operations.



Network performance between the client and backup server may impact performance. For example,

• Two ends of a connection are not matched. For example, where data nodes are set up to auto-negotiate the network connection and the switch to which the nodes are connected is set up to run at 100 mbps at full duplex. In this case, network performance can be significantly degraded.

• Across a WAN with latency, the WAN bandwidth can limit the overall backup performance.

• Encryption and compression may require significant amounts of resources on the backup client and can affect backup performance. However, the affects of performing software compression may be outweighed by the requirement of sending less data over the network where the network is a performance bottleneck. Do not perform software compression on backup data when it is sent to a device performing hardware compression.

Other factors include activities happening concurrently to backups that limit the client I/O capabilities, other network traffic, and slow random seek performance of the client. For example, performance may drop with clients backing up many files on SATA drives.



The network connection between the backup client and the LAN is critical since it will ultimately determine how much data can be backed up or restored within time constraints.

Network technologies generally require the CPU and operating system to carry much of the burden of managing low-level protocol and handshaking. TCP/IP is primarily designed for messaging around relatively small blocks of information.

High-speed networks can be overwhelmed by two cached disk-array connections and two to six tape libraries operating in full streaming mode. You should also consider other types of traffic happening concurrently with the backup process. Backup data streaming across the LAN affects the network performance of all systems connected to the same network segment as the backup server. Environments that back up many logical disks to many tape libraries will be constrained by even the fastest network technologies.



One of the factors that is key to backup performance is the network speed and most importantly, available bandwidth. When considering network speed, you should also consider how much of the bandwidth is actually available not just in terms of theory, but practically. That’s because LANs are based on the TCP/IP protocols which place a large overhead on the communication.

The smaller the number of hops and the shorter the path, the better the performance.



Many companies use the LAN for the backup data path during their backup process. Depending on how large the environment is, and the impact the backup operation puts on production, a separate LAN can be built for backups. Separating production traffic from backup is done not only to enhance the performance of the backup process but also to protect the required performance of the production network from being negatively impacted by the backup traffic.

Frequently enough, a company might already have a management LAN in place which can also be used for backups. When a separate backup network is used, make sure that the backup data flows via the backup network and not the regular production network.



TCP Offload Engine (TOE) cards can be employed in a server to offload the burden of managing low level protocol and handshaking operations from the host’s CPU to a processor on the TOE card. By doing this, the CPU is freed up and the network can perform better since there is a processor exclusively managing TCP/IP operations. These cards are also often called “mini-HBAs”.



Network bandwidth is the information carrying capacity of a medium. In terms of digital signals, it typically references the amount of data that can be transmitted from one point to another in a given amount of time, usually measured in seconds. Bandwidth typically ranges from 56Kbps (kilobits per second) for dial-up or fractional T1 connections up to 10Gbps (gigabits per second) for high capacity connections. Most network cards (NICs) today support 1Gbps and 100Mbps.

Latency in an IP network can cause unacceptable levels of performance. It is important to understand the factors that contribute to latency, in an attempt to minimize them as much as possible. Some amount of latency is always encountered, despite any efforts made, as the act of transmitting impulses (electrical or optical) along a communications medium (copper or fiber) experiences impedance or resistance from that medium. Damaged media, such as a kinked cable, can also cause signal loss or degradation which results in delays or retransmissions.

Probably the most significant source of latency is network congestion.



Retransmissions are the result of a TCP packet being sent from one device to another and the sending device not receiving an acknowledgement from the receiving device. Retransmissions can be caused by a number of network problems.

Each time a TCP Transport Retransmission occurs, the sending device must resend the packet that was lost. Before resending this packet, the sending device must wait a sufficient time period to allow the original packet to reach the receiving device, the receiving device to process the packet and an acknowledgement to make it back to the sender. The sending device must also take into account any network delays that could occur during this process, so as not to resend a packet that is just slow, not lost.

An abundance of retransmissions can severely impact the rate at which data can be moved from one device to another. When troubleshooting network slowdowns, retransmissions are among the first things to look for. When designing a NAS solution, ensure that your design minimizes the chance of retransmissions occurring by validating planning for compatible speed/duplex settings, and checking for link errors. Buffer overflows are typically diagnosed and corrected after the implementation.

Three common causes of TCP retransmissions are physical link errors, duplex mismatches and buffer overflows.



Windows

With netstat –e, discards and errors are displayed. Discards and errors are an indication of a link error.

Linux

When invoked with the –i flag, netstat displays statistics for the network interfaces currently configured. The MTU and Met fields show the current MTU and metric values for that interface. The RX and TX columns show how many packets have been received or transmitted error-free (RX-OK/TX-OK) or damaged (RX-ERR/TX-ERR); how many were dropped (RX-DRP/TX-DRP); and how many were lost because of an overrun (RX-OVR/TX-OVR).

If link/transmission errors are seen on the output of the netstat command, check the network cables and connections to ensure that they are operating correctly. If the physical connection is operating correctly, retransmissions and errors could be the result of a duplex mismatch or buffer overrun.



A speed or duplex mismatch occurs when one node is operating with settings that do not match the other node. A speed mismatch often results in no connectivity. A duplex mismatch typically causes performance degradation of the network, intermittent connectivity, and data link errors.

Autonegotiation is an attempt to reduce speed/duplex mismatches. Theoretically, if the two nodes can negotiate the speed and duplex setting at which they operate, the mismatches should be resolved. Typically, autonegotiation techniques for NICs, switches, and routers don’t operate as expected. Incompatibility often exists between vendor implementations of autonegotiation. Manually setting the speed/duplex on one node, and using autonegotiation on the other does not resolve the problem. This often results in a half duplex setting at the auto-negotiated side.

Duplex mismatches are often caused by the use of hubs. When hubs are used, the transmission protocol used is CSMA/CD (Carrier Sense Multiple Access/collision Detect) which supports half duplex transmission. When two nodes try to transmit at the same time, a collision occurs.

Speed/duplex settings are configured at each node in the network; client NIC, NAS device, and all switches/routers. Each NIC card, NAS Device, and switch vendor has specific interfaces and commands to check and configure the settings.



Client restore performance varies widely depending upon average file size of the data being restored, client processing speed, RAM, I/O subsystem capability and network connection between the client and the backup server/storage node.



This lesson covered factors impacting client backup and restore performance, including network considerations.



In this module, we look at backup and recovery from the perspective of the storage node, including the various protocols used when writing data and the advantages and disadvantages of the various types of backup storage media.

1 Module 4: Backup Storage Node


In this lesson we will look at the various components of a backup storage node.



A storage node is a host machine in the backup environment. It is primarily an I/O machine, receiving lots of data from backup clients and writing it out to tape or disk. Storage node also refers to the backup application software running on a host machine. A storage node is an I/O machine. By far, the majority of the work done is I/O related: a lot of data in from backup clients and out to tape or disk. A fast CPU is important but secondary.

There are several points to consider for the backup/recovery solution:

If the length of the window and the amount of data that must be moved is known, then the required data movement rate can be estimated. The intrusive nature of backups may cause memory contention on the host, require tuning, and perhaps remounting of the file system. Base the capacity of CPUs required on that which will support the required data movement.

The host machine may be used only as a backup storage node or it may be hosting other applications as well. It is important to consider this when planning the backup environment. Finally, if the machine is also a production server, we must not forget that the I/O subsystem must support disk activity for production work as well as backup and restore activity.



A bus is very likely to be the ultimate bottleneck of the system. The only way to know is to test. A higher bus clock speed is better but no guarantee of faster relative performance. Also, the placement of the HBA/NIC on the board could be critical. Higher end systems have multiple buses.



When deciding upon the amount of memory for a storage node, be sure to consider any applications, other than backup, that are running on the host. There should be minimal swapping or paging during backup processing.



One of the factors that is key to backup performance is the network speed and most importantly, available bandwidth. When considering network speed, you should also consider how much of the bandwidth is actually available not just in terms of theory, but practically. That’s because LANs are based on the TCP/IP protocols which place a large overhead on the communication.

The smaller the number of hops and the shorter the path, the better the performance.



Most backup software products offer scalable solutions with a central point of administration. This facilitates the administration of the backup infrastructure as opposed to environments with multiple backup servers which are difficult to manage because each has a separate catalog to maintain.

The backup infrastructure can easily scale by adding more storage nodes. This offloads the backup load from other storage nodes while still providing one single point of administration. Placement of storage nodes can be critical to overall performance of backup/recovery solution.



One very important piece of information needed to size the solution is the annual growth rate. This could come directly from the business or extrapolated from historical information.

When using tape as the destination storage device, sizing is usually limited only by the number of slots available inside the tape library, making it easy to add cartridges when needed. The number of cartridges needed and the number of cartridges that have to remain inside the tape library to satisfy common restore requests are also important to know so that the tape library can be designed to be an adequate size.

On the other hand, when disk is used as the primary destination storage device for backups, sizing must be done appropriately. The issue is not only to have enough storage space to do the backup, but also to ensure that most restores requested operationally are satisfied while the required data set is still on disk and before it is moved to tape. This is influenced by the RTO requirements.



This lesson covered the components of a storage node in a backup environment.



In this lesson we will review the various protocols used for writing backup



FTP is a client/server protocol that enables data transfer over a network. An FTP server and an FTP client communicate with each other using TCP as the transport protocol. FTP, as defined by the standard, is not a secure method of data transfer because it uses unencrypted data transfer over a network. FTP over Secure Shell (SSH) adds security to the original FTP specification.

A DFS (or NFS) is a file system that is distributed across several hosts. A DFS can provide hosts with direct access to the entire file system, while ensuring efficient management and data security. In a client/server model that uses DFS, the clients mount remote file systems that are available on dedicated file servers. Example of standard client/server file sharing protocols are NFS for UNIX and CIFS for Windows. NFS and CIFS enable the owner of a file to set the required type of access, such as read-only or read-write, for a particular user or group of users.



NFS is a client/server application that enables a computer user view, and optionally stores and update files on a remote computer as though they were on the user's own computer. It uses Remote Procedure Calls (RPC) to communicate between computers.

The user's system requires an NFS client to connect to the NFS server. Since the NFS server and client use TCP/IP to transfer files, TCP/IP must be installed on both systems.

Using NFS, the user or system administrator can mount all or a portion of a file system (which is a portion of the hierarchical tree in any file directory and subdirectory). The portion of the file system that is mounted (designated as accessible) can be controlled using permissions (e.g., read-only or read-write). NFS uses Network Information Service for domain name resolution.



CIFS is a client/server application protocol that enables client programs to make requests for files and services on remote computers over TCP/IP. It is a public, or open, variation of Server Message Block (SMB) protocol. The CIFS protocol enables remote clients to gain access to files that are on a server. CIFS enables file sharing with other clients by using special locks. File names in CIFS are encoded using unicode characters. CIFS provides the following features to ensure data integrity:

• It uses file and record locking to prevent users from overwriting the work of another user on a file or a record

• It runs over TCP

• It supports fault tolerance and can automatically restore connections and reopen files that were open prior to interruption. The fault tolerance features of CIFS depend on whether an application is written to take advantage of these features. Moreover, CIFS is a stateful protocol because the CIFS server maintains connection information regarding every connected client. In the event of a network failure or CIFS server failure, the client receives a disconnection notification. User disruption is minimized if the application has the embedded intelligence to restore the connection. However, if the embedded intelligence is missing, the user has to take steps to reestablish the CIFS connection.

Users refer to remote file systems with an easy-to- use file naming scheme:

\\server\share or \\servername.domain.suffix\share.



Backup devices store the backup data. Backup and recovery technologies are in transition from traditional tape media to disk. Backup to disk is replacing tape and associated devices as the primary target for backup. Backup to disk systems offer major advantages over equivalent scale tape systems in terms of capital costs, operating costs, support costs and quality of service.



This lesson covered the several file sharing protocols used when writing backup data: FTP, CIFS and NFS.



This lesson looks at physical tape as a medium for backup storage.



Tapes have traditionally been used extensively for backup operations. Tape stored locally can be used for operational backups and restores. Backups for disaster recovery use tape stored off-site. Tape media may also be employed for long term storage of backup data.

Tape drives are used to read and write data from and to a tape cartridge. Several types of tape cartridges are available. They vary in a number of factors including size, capacity, density, length and supported speed. Tape drives are referred to as sequential access devices because the data is written or read sequentially.

Typical backup operations used by almost all businesses are full backups (usually weekly), incremental backups (usually daily), and/or cumulative backups.



The physical tape library provides housing and power for a number of tape drives and tape cartridges, along with a robotic arm or picker mechanism. The backup software has intelligence to manage the robotic arm and entire backup process. Tape drives read and write data from and to a tape. Tape cartridges are placed in the slots when not in use by a tape drive. Robotic arms are used to move tapes around the library, such as moving a tape drive into a slot. Another type of slot called a mail or import/export slot is used to add or remove tapes from the library without opening the access doors because opening the access doors causes a library to go offline. In addition, each physical component in a tape library has an individual element address that is used as an addressing mechanism for moving tapes around the library. When a backup process starts, the robotic arm is instructed to load a tape to a tape drive. This process adds to the delay to a degree depending on the type of hardware used, but it generally takes 5 to 10 seconds to mount a tape. After the tape is mounted, additional time is spent to position the heads and validate header information. This total time is called load to ready time, and it can vary from several seconds to minutes. The tape drive receives backup data and stores the data in its internal buffer. This backup data is then written to the tape in blocks. During this process, it is best to ensure that the tape drive is kept busy continuously to prevent gaps between the blocks. This is accomplished by buffering the data on tape drives. The speed of the tape drives can also be adjusted to match data transfer rates.



In an environment without multiplexing, only one stream of data is written to a physical tape device at any given time. Multiplexing enables more than one save stream to write to the same device at the same time. This allows the device to write to the volume at the collective data rate of the save streams, up to the maximum data rate of the device. Tape drive streaming is recommended from all vendors in order to keep the drive busy thus optimizing the tape drive’s throughput.

If you do not keep the drive busy during the backup process, performance suffers. Multiple streaming helps to improve performance drastically, but it generates one issue as well. The backup data becomes interleaved; thus, the recovery times may be increased.

Multiplexing is primarily used in physical tape drives to keep it streaming and avoid the “shoe shining” effect. This setting should be avoided when backing up to a deduplication device since it decreases the algorithm’s ability to recognize data patterns.



Tape is a good storage media when factors such as portability, capacity, and low cost are considered.



Tape is a good storage media when factors such as portability, capacity, and low cost are considered. Data access in a tape is sequential, which can slow backup and recovery operations. Drives and tape cartridges are also susceptible to failure and lack protection such as that provided by RAID in disk media. A backup tape may be corrupted but the administrator would not know it until the time came to run a restore and then it is too late.

Backups implemented using tape devices involve many hidden costs. Tapes must be stored in locations with a controlled environment to ensure preservation of the media and prevent data corruption. Additional costs are involved in offsite storage of media and media management. Physical transportation of the tapes to offsite locations also adds management overhead. The traditional backup process using tapes is not optimized to recognize content, so the same content could be stored many times. Tapes are also susceptible to wear and tear. Frequent changes in these device technologies lead to the overhead of converting the media into new formats to enable access and retrieval. Traditional long term storage solutions store content offline and offsite where it is not readily accessible; tapes must be located and transported to a location where they can be read before a restore can take place.

The approach of performing weekly full backups followed by incrementals can be very problematic in the case of a full restore. Restoring the full backup and then appling all of the incremental backups can require many volumes. This can take a very long time, considering the time required to find the tapes, mount them, position and read the data.



This lesson covered the characteristics of tape storage and discussed the advantages and disadvantages of tape as backup storage. We also looked at the components of a physical tape library.



In this lesson we take a look at disk as the backup medium.



Backup to disk (B2D) refers to a solution where backup data is written to a file system. Backup applications use disk to improve data backup and restore performance and reliability. The random access capabilities of disk enable any file to be restored quickly. In addition, B2D comes with RAID to protect data from corruption and loss. Media for B2D includes ATA and SATA disks.

The addition of disk-based backup into a tape backup environment is fairly straightforward, although some parameters in the backup application may need to be set to direct the backup to a file system on disk.

In backup to disk, backup storage disks and disk storage systems can be direct-attached, SAN or LAN attached. The slide shows an example of LAN backup to disk. When a backup takes place, files on the clients are transmitted over the network to the storage node. The backup device file system (a network share point) appears as a folder on the storage node. The storage node then sends the data to the backup device to be written to disk. Metadata is transmitted to the backup server from the client and from the storage node (not shown). When a restore takes place, the data is transmitted over the network from the backup device to the storage node, and from the storage node to the client.



Backup to disk systems offer ease of implementation, reduced costs, and improved quality of service. The operating cost advantages and operational benefits of disk over tape can yield substantial ROI characteristics. The relative price of disk vs. tape and the availability of lower cost drives have dramatically reduced the cost-benefit gap between disk and tape and has made disk-centric backup practical for many companies.

In addition to performance benefits in terms of data transfer rates, disks offer faster recovery when compared to tapes. Disk storage systems also offer clear advantages due to their random access and RAID protection capabilities. For example, because tape is a sequential-access medium, it is not possible to perform both a backup and a restore operation using the same tape at the same time. Disk is a random-access medium and does not suffer from the same simultaneous read/write limitations as tape. With disk, it is possible to perform a backup and a restore operation simultaneously, when needed.

Reliability is also a strength of disk. The use of RAID sets means that data is spread across multiple disks. Disks also may require less mechanical operations than tape and experience less degradation of the media over time. Typically, there is no manual intervention required when accessing data on a disk volume as opposed to locating and transporting an off-site tape for recovery.

Because backups and restores run faster with disk, full backups may be created more frequently, which in turn improves RPO.



The slide lists some factors to consider when using disk as the backup medium.



An important part of data management is to ensure that the right data is in the right place at the right time (when it’s needed). Tape continues to play a role in protection architecture. Backup data is regularly moved from disk to tape for longer term retention and offsite storage. At the same time, disk is the first “line of defense” for the fast and reliable recovery of critical information.

The majority of restores are front-loaded. That is, the vast majority of restores are requested in the first 5 to 10 days since the data was backed up. As time goes by, the requests for restores tend to taper off. In many backup environments, backup to disk is used as a staging area where production data is backed up first to disk before transferring or staging the backup data to tapes at a later time. This strategy has the potential of enhancing both backup and restore performance as opposed to backing up directly to tape. For the period of time when most operational restore requests occur, backup data is stored on disk. As requests for the data wind down, the backup data is moved off-site to tape for disaster recovery or long term storage purposes. Some backup products allow for backup images to remain on the disk for a period of time even after they have been staged.



This lesson covered benefits and considerations when using disk for backup storage. Then, we discussed the data management strategy to ensure that the right data is in the right place at the right time by implementing D2D2T.



This lesson covers the components of a virtual tape library (VTL) and looks at factors to consider when using VTL for backup storage.



Virtual tape libraries (VTLs) are offered as single purpose appliances used for backup and recovery. For backup software, there is no difference between a physical tape library and a virtual tape library. Virtual tape libraries provide virtual tape library emulation capability that makes physical disks in the VTL appear as tapes to the backup application. This feature enables organizations to easily integrate VTL into the existing backup infrastructure since the virtual tape library looks like and behaves like a physical tape library. No changes to the backup software are required to implement.



A virtual tape library (VTL) has the same components as that of a physical tape library except that the majority of the components are presented as virtual resources. VTL uses disks as backup media and emulates tape behavior. An emulation engine is a dedicated server with a customized operating system that makes physical disks in the VTL appear as tapes to the backup application. Emulation software has a database with a list of virtual tapes, and each virtual tape is assigned a portion of a LUN on disk. A virtual tape can span multiple LUNs if required. File system awareness is not required while using backup to disk because virtual tape solutions use raw devices. Unlike a physical tape library, which involves mechanical delays, in a virtual tape library access is almost instantaneous.

Virtual tape library appliances may offer a number of features that are not available with physical tape libraries. These include multiple emulation engines configured in an active cluster configuration and replication over an IP network to a remote site. Virtual tape libraries also may enable copying of a virtual tape to physical tapes in a physical tape library.



The slide lists some factors to consider when using VTL as the backup medium. Compared to physical tape, virtual tape offers better single stream performance, better reliability, and random disk access characteristics. Backup and restore operations are sequential by nature, but they benefit from the disk’s random access characteristics because they are always online and ready to be used, improving backup and recovery times. Virtual tape does not require the usual maintenance tasks associated with a physical tape drive, such as periodic cleaning and drive calibration. In addition, virtual tapes do not require any additional modules or changes on the backup software when transitioning from tape.



This lesson covered the components of a virtual tape library and discussed some factors to consider when using VTL as the backup medium.



In this lesson, we review the different types of data deduplication and some considerations for selecting clients and data to use deduplication technology for backups.



Data deduplication can be defined as the process of finding and eliminating duplication within sets of data with the goal of increased efficiency.

The deduplication process uses well understood concepts such as cryptographic hashes and content addressed storage. For backup, unique segments of data are stored along with metadata needed to reconstitute the original data set. Data can be processed for deduplication using software or through a dedicated appliance.

In the picture on the slide, we see 3 data sets, each different, but with some common internal similarities represented by the segment colors.

By running a deduplication algorithm, these internal similarities are detected and identified as common segments. These common segments represent the essential set of unique information reduced to its minimal size. It would be possible to rebuild the original data sets from the unique segments given the metadata or pointers (e.g., which data set, order within the data set).

Its important to view deduplication as a feature, not a product. It emerges across the backup portfolio to solve different customer problems.



Data deduplication reduces storage needs by identifying duplicate or redundant data. The level at which data is identified as duplicate affects the amount of redundancy or commonality that is identified. Levels of deduplication include:

• File level deduplication helps organizations reduce storage needs for file servers by identifying duplicate files within hard disk volumes and providing an efficient mechanism for consolidating them. The most common implementation of single instance storage is at the file level. With this method, a single change in a file results in the entire file being identified as unique. As shown in the example, if there were 5 versions of a file in a backup environment, the 5 files in their entirety are stored. However, when the same file is presented for backup time and time again, it is backed up only the first time.

• Fixed block deduplication, also called fixed length deduplication, is commonly employed in snapshot and replication technologies. This method breaks a file into fixed length sub-objects. However, even with small changes to the data, all fixed length segments in a dataset can change despite the fact that very little of the dataset has actually changed.

• Variable block level deduplication uses an intelligent method of determining segment size that looks at the data itself to determine repeatable boundary points. Variable block level deduplication yields a greater granularity in identifying duplicate data, eliminating the inefficiencies of file level and fixed block level deduplication. With variable block level deduplication, a change in a file results in only the variable-sized block containing the change being identified as unique. Consequently, more data is identified as common data, and in the case of backup, there is less data to store as only the unique data is backed up.

Smaller block sizes typically yield better deduplication ratios, up to the point where pointer and formula storage overhead become significant. Fixed-length deduplication is more effective for deduplication with block-aligned data; variable-length deduplication is more effective for unstructured, unaligned data.



Employing deduplication with backup and recovery enables organizations to realize several advantages:

Retain backups onsite longer using less disk: Keep backups onsite longer using less disk though deduplication that delivers a 10-30 times reduction in data to be stored compared to traditional methods. Other benefits include the elimination of tape for operational recovery and fast, reliable disk restores.

Replicate smarter: When replicating (cloning) deduplicated backup data from source deduplication storage to target only deduplicated data travels over the existing networks for up to 99% bandwidth efficiency. The benefit: enabling fast, cost-effective disaster recovery.

Recover reliably from disk with continuous fault-detection and self-healing technologies, ensuring data recoverability to meet SLAs.



Backup can be an inefficient process that involves repetitively moving mostly the same data again and again. Deduplication dramatically reduces the amount of redundancy in backup storage.

This chart shows the amount of deduplication storage vs. traditional storage over time and dramatically shows the impact deduplication has on backups.

There are two points that are important to note here:

• The first point is that the effect grows over time—the more redundant data that is stored, the greater the degree of deduplication effect between the amount stored by traditional backups, the light area, and the amount of capacity used for deduplication storage, which is the dark area on the bottom.

• Secondly, these numbers are based on a typical backup policy schedule of a full backup on a weekly basis; the amount of data reduction varies primarily on the basis of that policy and how long that data is kept. So the retention policy will guide the degree of deduplication more than any other factor.

One thing is clear—the impact is significant.



The use of data deduplication for backups can significantly improve efficiency and reduce costs related to backup storage capacity and remote replication bandwidth requirements. An investment in data deduplication solutions is easy to rationalize. Deduplication is one of the few IT solutions that cuts costs quickly and improves service levels and compliance. With it, organizations can reduce storage expenses without compromising data protection.



Source-based deduplication uses client software agents to identify repeated sub-file data segments at the source (client), so only new, unique segments are transferred across the network and stored to disk during backup operations.

Because source-based deduplication stores only a single copy of sub-file data across sites and servers, it is a good fit for:

• Remote offices

• Branch offices

• Datacenter LANs

• VMware environments



In target-based deduplication, the backup application sends native data to a target storage device. The data is deduplicated once it reaches the target.

Target-based deduplication is a good fit for larger accounts with significant backup issues. In addition to requiring less storage for backups, a key benefit of target-based deduplication is the ability to provide transparency to the backup application.



When multiple similar systems are backed up to the same destination, there exists the potential for much redundancy within the backed up data. However some types of data, clients and situations do not yield the best deduplication results. The slide lists factors to consider when determining the clients and data in the backup environment for deduplication backups.



During this lesson, we looked at the different types of data deduplication and considerations for selecting clients and data for deduplication backups.



The objectives for this lesson are shown here. Please take a moment to read them.



Cloud computing or cloud technology is a model for enabling efficient and convenient on-demand access to a shared pool of computing resources. These resources include networks, servers, storage, and applications.

The “as a service” model represents a new way of resource delivery in IT. Just as virtualization ushered in faster and more robust services, it is now having a similar effect when applied to servers and storage. Server and storage environments can be easily provisioned, expanded, contracted, decommissioned, and repurposed, yielding extreme flexibility and elasticity.



Cloud computing should have these essential characteristics.

On-demand self-service. Consumers can use cloud services as required, without any human intervention.

Broad network access. Cloud services can be accessed via the network from a broad range of client platforms, such as desktop computer, laptop, mobile phone and thin client. This enables accessing of services from anywhere.

Resource pooling. IT resources, compute, storage, network, are pooled to serve multiple consumers. Resources are shared among multiple clients and are dynamically assigned and reassigned based on demand. Typically a consumer has no knowledge of the exact location of the provided resources.

Rapid elasticity is the ability to expand or reduce allocated IT resources dynamically, quickly and efficiently without service interruption.

Measured service means that consumers are billed based on their usage of cloud resources.



Increased dependency on online information leads to on-demand, reliable, secured, and speedy access to petabytes of information which is further growing exponentially. Businesses must align themselves to accommodate this astonishing growth much faster than ever, which requires multi-fold increases in capacity or capability on the fly. The procurement and provisioning of these resources typically take a long time, which may impact the service levels demanded by the customer. Many organizations deploy server, storage, and network virtualization in a regional basis, or within a datacenter, which results in discrete virtual computing environments leveraged by various departments of an organization across the globe. Virtualization improves resource utilization, however this advantage can be envisaged only for limited periods because of its discrete implementation, beyond which organizations have to scale up their expenditure to add new resources.

These long standing pain points can be outdated with the emergence of cloud computing. Cloud computing brings up a new generation of computing which enables an organization to extend virtualization beyond its enterprise datacenter by aggregating IT resources scattered across the globe. Location independent virtual images of aggregated resources can be created and assigned dynamically on-demand with a metering service to monitor and report resource consumption. Cloud computing allows self service requesting empowered by a fully automated request fulfillment process in the background. Organizations can build their own cloud by pooling and virtualizing distributed resources, as well as hiring computing resources from cloud service providers, and pay based on resource usage such as CPU hours used, amount of data transferred, and gigabytes of data stored.



From a business perspective, periodic upgrades of computing resources has become a necessity to deliver better and faster in the market. Organizations may need to rapidly expand their business which may force the organization to add new servers, storage devices, network bandwidth, etc. Critical business data must be protected and should be available to the intended user, which requires data security and a disaster recovery infrastructure. As the capital expenditure rises, the risks associated with the investment increase. For small and medium size businesses, this may be a big challenge, which eventually restricts their business growth. It may not be sensible or affordable every time to purchase new applications which are only needed for a limited period of time.

Instead of purchasing new resources, cloud resources are hired based on pay-per-use without involving any capital expenditures. Cloud service providers offer on-demand network access to configurable computing resources, such as networks, servers, storage, and applications. Demand for computing resources can be scaled up or down with minimal management effort or service provider interaction. Cloud service provider’s expertise can be leveraged to store, protect, backup, and replicate data empowered by the most advanced technologies which otherwise would cost more.



Cloud services are used and paid for as needed, upon demand. Consumers can save money because there is no capital expenditure required. Also, an organization’s expenses for running a data center, such as cooling, power and management, are reduced by employing cloud resources. Cloud can reduce the time required to provision and deploy new applications and services, thereby allowing organizations to respond more quickly to changing business demands. Because resources used by an organization can grow or shrink dynamically, organizations can easily scale up or down. With cloud computing, there is the ability to ensure application availability at various levels depending upon policy and priority. The use of redundant resources and clustered software encompassing multiple datacenters in different geographical regions provide fault tolerance and availability.



Cloud computing can be classified into three deployment models, private, public, and hybrid, which provide a basis for how cloud infrastructures are constructed and consumed.

In a Private Cloud, infrastructure is deployed and operated exclusively for an organization or enterprise. This model offers the greatest level of security and control. It may be managed by the organization or by an independent third party and may exist on-premise or off-premise at a hosting facility. Many enterprises, including EMC, offer private cloud systems.



A Public Cloud infrastructure is available to the public or to many industry groups or customers. It is owned by the organizations promoting and selling cloud services. This can also be thought of as an “on-demand” and a “pay-as-you-go” environment where there are no on-site infrastructure or management requirements; however, the environment is no longer within the customer’s perimeter. Popular examples of public clouds include Amazon’s Elastic Compute Cloud (EC2), Google Apps, and Salesforce.com.

A community cloud is one shared among several organizations and supports a specific community with shared concerns. It may be managed by the community organizations or by a third party. This option is more expensive than a public cloud but may offer more control and a higher level of privacy, security and/or policy compliance. It also offers organizations the potential of a wider pool of resources than that in a private cloud.



Hybrid Cloud is a composition of two or more clouds, private and public. Each cloud retains its unique entities. Clouds may be federated or bound together by technology, enabling data and application portability. Hybrid cloud is prevalent for several reasons. For example, many organizations have an existing private cloud infrastructure and may need to extend their capability, or often the benefits of combining both private and public clouds may be a more efficient model to handle an unexpected surge in the application workload.

Most large organization CIOs are holding off on putting their computing requirements on public cloud, but they are leading to develop private clouds. In this case, critical customer data can be restricted within an organization’s private cloud; however, management and monitoring applications can run on public cloud. The customer gets updates from the public cloud and can send queries. Both the clouds remain partitioned from each other, however, together they form a hybrid cloud.



This lesson covered the characteristics of cloud technology and its benefits. Then, we reviewed the three cloud deployment models which provide a basis for how cloud infrastructures are constructed and consumed.



This module examines the various factors to be considered in backup and recovery planning. The module concludes with an overview of the EMC Backup and Recovery product portfolio. Then, students are given the opportunity to use what they have learned in the course to develop a proposed solution to address a sample company’s backup and recovery concerns.

1 Module 5: Backup and Recovery Planning


In this lesson, we review the various factors to consider when planning a backup environment including looking at an enterprise’s current backup environment and the characteristics of the data to be backed up, the affect of compression and encryption on backups, and the influence of business needs and service level objectives on retention periods and backup planning.

Module 5: Backup and Recovery Planning 2


The best solution depends on many factors. They include, but are not limited to: costs, business requirements, application requirements, recovery point objective (RPO), recovery time objective (RTO).

To understand how to develop an effective backup architecture, you need to first look at the total amount of capacity that has to be backed up, and then look at the types of applications involved. With that in mind, you need to make choices as to what needs to be backed up, how often it should be backed up, and how fast recovery needs to be if it is needed. Finally, the connectivity needs to be determined, whether it’s SAN based, LAN based, or some combination of the two.

Talking specifically about backing up to disk, one of the biggest challenges that a user faces is sizing the solution. Gaining the best value from a backup-to-disk solution may involve assessing the current backup retention, as well as the backup frequency.



This slide lists some considerations for designing a backup and recovery solution.



An important topic to be covered is ”What do I really need to backup?”

It is very important to study the environment to discover the type and amount of data to backup. Some data can be very important for some businesses but not relevant for others. That means different businesses have different needs.

Besides discovering the type of data to backup, it’s necessary to decide when and where to perform the backups. Each type of data may require a different backup frequency, different backup media and have other unique requirements.



The key to a successful solution is to understand the current IT infrastructure.

To do that, you must consider many factors such as policies, business requirements, existing backup infrastructure and actual issues that may need to be solved. It is important to understand the customer’s environment, specifically, the applications in the environment, which data needs to be backed up, amount of data to be backed up, and location of the data: storage platforms, sites.

The slide shows the typical elements of an IT infrastructure that should be considered.



The slide continues from the previous slide, showing the typical elements of an IT infrastructure that should be considered.

An EMC Backup Assessment service should also be considered. This service helps the customer understand the whole infrastructure by providing reports that show what the issues and limitations are.



Multiple operating system platforms, many types of data to be backed up, where that data is stored and the available target for back up all add to the complexity of the backup environment. It is important to understand the variables involved and the complexity they bring to the backup infrastructure. The heterogeneity of the customer environment is what makes the backup operations complex. The variables include the storage arrays in the environment, which can be Symmetrix DMX, CLARiiON, Celerra, Centera, or storage arrays from other vendors.

They can have one or more applications including: databases - like SQL, Oracle, and DB2; email applications - like Exchange; and many custom applications that reside on any operating system including Windows, Solaris, Linux and others that need to be backed up periodically. These applications are configured to access data based on SAN, LAN, iSCSI and other topologies, and are used again to move data across to the target devices, using the same or different topologies. The type of target devices, which may include physical tape libraries, virtual tape libraries and disk targets, also add complexity to the backup environment.

All of these variables need to be considered when setting the customer service level objectives on backup RPOs and RTOs .



This slides shows a typical mix of applications. In order to have a successful backup implementation, it is important to understand the operational characteristics of each of the applications in the environment.

For example, Tier 1 applications may need to be recovered within a matter of seconds, or revenues could be impacted. This is particularly true in businesses that have revenues tied to system uptime. Note that the RTO is measured in seconds, and the RPO goal is the very last transaction. Also, in this particular case, there is no window of time during which backups can occur, so leveraging online backups to a point-in-time copy will make sense.

E-mail in this situation is similar to Tier 1, with minor differences in the recovery objectives as well as the backup times.

In considering the other applications, their requirements are a lot less stringent. As a consequence, the backup and recovery strategies employed with a backup scenario will be architected differently than the Tier 1 and e-mail applications.

Creating this kind of spreadsheet gives clarity to the requirements of a backup implementation.



Data can be classified as structured or unstructured based on how it is stored and managed. Structured data is organized in rows and columns in a rigidly defined format so that applications can retrieve and process it efficiently. Structured data is typically stored using a database management system (DBMS).

Data is unstructured if its elements cannot be stored in rows and columns, and is therefore difficult to query and retrieve by business applications. For example, customer contacts may be stored in various forms such as sticky notes, e-mail messages, business cards, or even digital format files such as .doc, .txt, and .PDF. Due to its unstructured nature, it is difficult to retrieve using a customer relationship management application. Unstructured data may not have the required components to identify itself uniquely for any type of processing or interpretation. Businesses are primarily concerned with managing unstructured data because over 80 percent of enterprise data is unstructured and requires significant storage space and effort to manage.



What data will be backed up and the characteristics of that data are important considerations affecting the backup process. Different types of data have different backup requirements and require different backup strategies. Some of the factors to consider are listed here.

• Location: Many organizations have dozens of heterogeneous platforms supporting complex solutions. Consider a data warehouse environment that uses backup data from many sources. The backup process must address these sources in terms of transactional and content integrity. This process must be coordinated with all heterogeneous platforms on which the data resides. Some of the issues are how the backups for subsets of the data are synchronized and how these applications are restored.

• Size: Backing up large-size files may use less system resources than backing up an equal amount of data comprising a large number of small-size files. The backup and restore operation takes more time when a file system contains many small files. Backing up a large amount of data that consists of a few big files may have less system overhead than backing up a large number of small files.

• Number: Large numbers of files to backup affects both backup and restore performance. For example, it will take longer to search through a file system for changed files and more entries are created in the backup catalog which will increase time in a restore to locate the backed up data set.

• Type: The type of data may affect the selection of backup media and/or backup method. Is it file system, database, email, or rich media? What is the initial commonality of data? How often does the data change? What is the daily change rate? Is the data already compressed or encrypted?



Many organizations have dozens of heterogeneous platforms that support a complex application. Consider a data warehouse where data from many sources is fed into the warehouse. To capacity plan, back up, restore, and recover these complex applications can easily involve hundreds or thousands of files scattered across dozens of heterogeneous systems. These systems may not be in a single physical location. Portions of the application may have differing backup schedules.

Managing business continuance for such an application is a big challenge for the application owner, but consider that a storage administrator may have to manage hundreds of these complex applications.

The key issues are:

• How the backups for subsets of the data are synchronized.

• How these applications are restored.

• How these applications are recovered.

• File sizes and the number of files.



Data compression is the process of encoding information using fewer bits than the original representation. The primary goal and benefit of compression is to reduce the amount of storage media consumed by data. Depending upon where the compression takes place there could be additional benefits including sending less data across the network to backup storage and improving performance through less I/O with backup media. The algorithm used to accomplish compression depends on the backup software or device used. Two popular algorithms are the LZO (Lempel-Ziv-Oberhumer) and the LZW (Lempel-Ziv-Welch).

Compression can be performed at two levels, hardware or software. Hardware compression is compression that is done at the hardware level, such as in a tape drive or in some type of hardware expansion board. It’s usually done at the backup device, so the data travels from the backup client through the LAN uncompressed, and is compressed right before it’s written to the backup media. Tape drive hardware compression is very efficient and uses no additional CPU cycles on any component of the backup infrastructure.

Software compression is compression that is done at the host level, using CPU cycles and memory of the host, such as in a backup client. Software compression puts a large overhead on the host due to the processing requirements to accomplish compression. When software compression takes place at the backup client level, data travels compressed through the LAN, thus reducing the overall bandwidth taken, and is then written to the backup media. Compressed data must be decompressed in order to be used and this may also entail additional processing.

To effectively use compression, it is important to understand the characteristics of the data. Some data, such as application binaries, do not compress well. Data such as text can compress very well, while other data like JPEG and ZIP files are already compressed. Files that are already compressed have a tendency to get larger when they are compressed again. It is often recommended to turn off compression for data that compresses poorly. Compressed files are not good candidates for



deduplication technologies. If employing software compression, don’t use hardware compression, and vice versa. Understanding data compression can help determine how much storage capacity is required for backup data.



Encryption is the process of transforming data using an algorithm to make it unreadable to anyone except if possessing special knowledge, such as a key. More and more organizations today face challenges regarding the security of their production data. There are various ways that encryption can be used in a backup environment. For example, several organizations have offsite tapes that are used for long term retention and disaster recovery. The tapes must be securely transported and stored, and administrators need to make sure the data in these tapes can only be read by authorized parties. This is done by using encryption methods. Another example is to secure the flow of backup and recovery data between backup client and storage node or when replicating backup data.

Many backup software products offer encryption of the backup data at the backup client before it is sent for backup. There are also appliances which work in the backup/restore data flow. Backup data is encrypted/decrypted by these boxes when data is written to or read from the backup device. Referring to the example above, the customer may choose to encrypt only those tapes that will be sent offsite. Encryption may also be performed at the storage node in order to store non-encrypted data in encrypted format or by the application that creates the data.

When employing encryption, you need to be aware of the load that encryption and decryption can place on the CPU on the host or device doing the processing. Encryption of stored data may also add to the amount of storage space required.

Another point to consider is that encryption mechanisms rely on a key management system. That means that data can only be recovered when the appropriate key is used. Therefore, backups are only as secure as the key management system itself: keys must be managed and protected.

It is not recommended to send encrypted data to deduplication backup targets as initial deduplication ratios will be very low (every block is unique). Subsequently, deduplication will be realized, however, when the same encrypted data is backed up over and over again.



Business needs determine backup requirements. The amount of data loss and downtime that a business can endure are the primary considerations when developing a backup plan. Depending upon recovery requirements, organizations will use different backup strategies for disaster recovery, operational recovery and for long term storage. Some important decisions to consider before deciding on a backup strategy are listed on the slide.



The reasons for performing backup as well as the amount of data loss the business can tolerate influences how long backup is retained. Backups for operational restore purposes are generally kept for shorter periods of time.

For example, users may request to restore an application’s data from its operational backup copy for up to one month. However, the organization may need to retain the backup taken at the beginning of each month for a longer period of time because of internal policies or regulatory requirements. Storing backups requires storage space; the longer that backups are kept, the more space is required. Therefore, it is important to define the retention periods based on a analysis of all restore requests in the past and the allocated budget.



The Service Level Objectives for the enterprise play a crucial role in deciding what to backup, when to backup, how long to retain and others. For example, some organizations have policies to backup every server in their data center but not backup the desktops as the desktop users are expected to store business critical data on the server shares.

Recovery Time Objectives allow for planning disaster recovery. When an application server crashes, the operating system, the configuration information and the applications need to be installed before restoring the data or reconnecting to the storage volumes. The recovery time objectives allow the users to plan and test the disaster recovery solutions.

The Recovery Point Objectives allow the backup architects to decide how often to backup the data. If data loss of no more than 4 hours is acceptable, then the backup architect should ensure that backup starts every 4 hours and most importantly, the previous backup completes in less than 4 hours. If the backups take longer than 4 hours, then they would need to re-architect their backup environment, and ensure that the backups complete within 4 hours.

Disaster recovery planning involves not only sending tapes to offsite locations or storing data in offsite disk storage, but also to periodically test to ensure that the data can be restored within the acceptable time period.



As organizations grow, the amount of data that needs to be backed up increases, which stretches the backup windows. As they move towards 24X7 availability, applications no longer have the downtime that was used for backup purposes. This eliminates the slow periods of time which were used for backup windows.

The only option available for backups in these scenarios is to speed up the backups and complete them in a shorter period of time. To increase the backup performance, the bottlenecks need to be identified. Typical bottlenecks in backup include tape performance and networks. Using a high speed network, like a Fibre Channel network or GigE network, helps push the data faster, but tape performance may still be a bottleneck. As an alternative, consider a D2D2T strategy to speed up initial backups and, where tape is required for off-site storage, run cloning or staging separate from the backup schedule. Also, consider deduplication technologies to transfer less backup data over networks and store on disk.



This lesson covered the many factors to consider when planning a backup environment.



This lesson includes an overview of EMC Data Protection Advisor and its role in managing the backup environment. We will also discuss the importance of testing the backup and recovery plan.



With today’s complex backup environments, there are often multiple protection methods and even multiple backup applications. It is hard to find information such as which clients were successfully backed up last night across all backup servers and across the enterprise. Similarly, for all successful backups, how many backups completed within the acceptable backup window and how many did not? What are the reasons for backup failures? When was the last time that this client successfully backed up?

Having such information helps to verify that backups meet the service level objectives of the organization. Knowing which clients were the slowest in the enterprise and having a list of clients that have failed backups for over a week, allows the backup administrators to concentrate on where to improve performance and which clients need immediate attention.



There are many challenges in today’s backup and recovery environments no matter which backup product is in use.

First, unprecedented data growth is overtaxing traditional data protection approaches and technologies, leading to increased uncertainty about protection status.

Second, multiple protection methods leads to data silos. Whether it’s different types of replication or multiple backup applications—or both—this creates silos that make it difficult to know what is protected and if it’s properly protected. Lack of a single view leads to a lack of visibility into the overall protection environment.

Lastly, there’s the shift from physical to virtual server infrastructure. The processes and technologies that worked well in a physical environment may not be the most effective in a virtual one.

This all leads to increasing challenges and uncertainty for protecting business-critical data as well as added cost and complexity.

Data protection management is the process of managing all assets and software involved in protecting a company’s critical assets. Businesses need to know that their critical application data is protected, that they can prove it’s protected and that if there are issues, they can find them and fix tem quickly. Collecting and unifying the data needed to effectively manage the data protection environment can be an expensive, time consuming manual effort and never thorough enough to cover all elements in the environment.



Data protection management is the process of managing all assets and software involved in protecting a company’s critical assets. Businesses need to know that their critical application data is protected, that they can prove it’s protected and that if there are issues, they can find them and fix tem quickly. Collecting and unifying the data needed to effectively manage the data protection environment can be an expensive, time consuming manual effort and never thorough enough to cover all elements in the environment.

Data Protection Advisor is a data protection management offering that moves companies out of the red stove-piped/manual effort area into the green area with data protection management that is fully automated with a single view and drives an environment that’s efficient, risk managed, and simplified.



EMC Data Protection Manager, DPA, provides a single view of the entire infrastructure through data discovery, analysis, and reporting that leverages this data for key backup management functions. DPA incorporates backup solutions, replication technologies, virtual environments, tape/VTL storage, SAN and NAS systems, and the business applications protected by the infrastructure.

A key component of DPA is the proactive analysis engine, which, along with its powerful central data store, drives the solution’s monitoring, alerting, troubleshooting, optimization, capacity planning, and reporting capabilities. The proactive analysis engine constantly monitors incoming data against a variety of customizable rules, looking for failures, threshold exceptions, and developing conditions.

DPA transforms volumes of disparate data into actionable business information that allows companies to lower costs through the improved use of their infrastructure, avoiding unnecessary purchases and reducing manual effort; improves compliance and lowers risk through better visibility and assurance that critical data is protected; and reduces complexity with a single console to provide an integrated, automated view.

This cross-environment visibility enables both service providers and end users to monitor the health of the infrastructure. Service providers can use DPA to constantly monitor their entire infrastructure to ensure service levels for each tenant are being met, the infrastructure is optimally utilized, and there is sufficient capacity based on growth expectations. With DPA, service providers can offer data protection as-a-service and put their tenants directly in control. End-users can see exactly what’s being done and how protected they are.



A primary capability of DPA is its ability to measure and prove data protection. With DPA’s automated monitoring, analysis, and reporting across the backup and recovery infrastructure, replication technologies, file servers, and virtual environments, businesses are able to more effectively manage SLAs while reducing costs and complexity. DPA enables single-click verification of service levels and empowers application owners and management to verify compliance.

DPA’s enhanced analysis engine drives services predictability by providing real-time analytics. The analysis engine looks across the entire environment, providing end-to-end visibility, as well as enabling unified, cross-domain correlation analysis which provides higher-level decision support based on defined policies. The analysis engine can identify trends, such as capacity growth, and enable administrators to plan future protection costs.

Visibility is crucial to protecting data in dynamic virtualized environments and ensuring compliance with SLAs. DPA tracks virtual data movement to provide confidence that virtualized critical applications are protected. Using VMware support provides a comprehensive understanding of the protection of virtual environments.

DPA is built at cloud scale, with distributed architecture offering multi-tenant data protection management and significant cost reduction by supporting thousands of tenants on a single DPA instance without performance degradation. DPA also combines chargeback or show-back reporting for backup and replicated data.



DPA’s user interface and streamlined navigation enables rapid access to this vital business information. Dashboards and comprehensive list of reports are available to IT administrators, service providers, and stakeholders. All have the ability to quickly search on a key word to identify the report or reports they are looking for, as well as to perform a categorical search based on the infrastructure components they wish use for reporting.

DPA provides customizable views based on access privileges or preferences. Viewing the most important, yet environment-wide, information on one screen enables better operational decisions based on the broad data set.



A backup that has not been recovery tested cannot be trusted. In many instances, some critical data that is required to restore the servers may go missing in the backup set, which can only be identified by testing the complete recovery. It is always a good practice to test recoveries and perform trials and not wait for a disaster to occur. One of the main objectives of backups is RTO. An RTO cannot be established if the backups are never tested.

During disasters, the remote locations start the servers and load the data in their environment and get them ready. So it is important to know that the processes followed should be documented because they may be used by a different team than the original team that architected the solution. Testing data restores and documenting the procedures helps meet many of the goals.

Testing disaster recovery procedures also provides a complete view into how to get the data back and the stages involved. For example, consider a situation where the system administrators plan to reinstall the operating system during a disaster and find that they can restore the operating system software media but that they forgot to store the license codes. Such holes will only be identified during a test. It is better to find the holes in a test rather than during a disaster or when the CEO requires a file to be restored!



This lesson covered an overview of EMC Data Protection Advisor and its role in managing the backup environment. We also looked at the importance of testing the backup and recovery plan.



This lesson includes an overview of disaster recovery terms, challenges and considerations.



The word “disaster” in the Business Continuity context is defined by the Disaster Recovery Journal as “A sudden, unplanned catastrophic event causing unacceptable damage or loss. 1) An event that compromises an organization’s ability to provide critical functions, processes, or services for some unacceptable period of time 2) An event where an organization’s management invokes their recovery plans.”

Disaster recovery: This is the coordinated process of restoring systems, data, and the infrastructure required to support key ongoing business operations in the event of a disaster. It is the process of restoring a previous copy of the data and applying logs or other necessary processes to that copy to bring it to a known point of consistency. Once all recoveries are completed, the data is validated to ensure that it is correct.

Disaster restart: This is the process of restarting business operations with mirrored consistent copies of data and applications.



Typically a disaster does not happen in an instant. A complete disaster is usually the culmination of a series of events which happen over a period of time. The period of time during which these events are occurring is an extremely critical period for remote replication solutions. During this period, data corruption could be occurring to the production applications because some components may be working while others are not. Remote replication will faithfully replicate the data corruption unless some corrective action is taken by the technology. Even if there is no data corruption on the primary devices, the replication may cause data corruption because some I/Os may get replicated while others do not, causing data integrity issues on the replica. Remote replication technologies must ensure data consistency throughout this period of time.



Disaster recovery planning involves people, processes and technology. The technology components of remote replication by themselves are complex. Thus, disaster restart and return home procedures tend to be complex as well. It is extremely important to test DR and return home procedures under controlled conditions so that when a disaster does occur one can resume operations at the more site in as efficient a manner as possible. The procedures can be tweaked and eventually finalized. The finalized procedures should be documented and made available at all the sites.



Once the remote devices have been made accessible and a local replica has been created then operations can be started at the remote site using the remote devices. The specific steps necessary to restart the operation are dependent on the specific application. Applications could be spread out over many servers, file systems and could be using one or more relational databases. Organizations should have extensively documented and tested their DR procedures. Once a disaster is declared, these documented procedures should be executed.



Production operations can be resumed at the primary site after the components that were damaged in the disaster are fixed. Before resuming operations at the primary site, a number of actions need to be taken. Typically, return home procedures would have been tested and documented. Thus, if would be a matter of executing the various documented steps. The steps are obviously unique for each case.

Some of the actions that should be found in most return home procedures are listed on the slide. One would perform an orderly shutdown of all applications. This will ensure that the data on the replica devices are all consistent. A local replica should be created in case of any data corruption during the return home process. The data on the remote replicas should be restored to the original source volumes at the primary site. Then applications can be resumed at the primary site.



Backup applications typically back up the data and expect the client component to be present to initiate a restore activity. However, in a true disaster you cannot expect the servers to be up and running with the operating system and the backup client. In large scale natural disasters like earthquakes and fires, you cannot expect the servers to be intact. You will have to rebuild the servers from scratch. Even operating system crashes and hardware failures may require rebuilding the servers from scratch.

Server rebuilding involves many stages requiring the current configuration information along with the list of applications and data volumes to bring back the system correctly. You will also need the license codes for the applications to activate the software. To rebuild the server, the operating system should be installed first using the operating system media and applicable license codes along with the network and configuration information. Once the operating system is installed, it needs to be updated with the correct service packs and patches. Then, the backup application client is installed before starting to restore the applications and data.



This lesson covered an overview of disaster recovery planning: terms, challenges and considerations.



In this lesson we will take a look at the EMC’s Backup and Recovery strategy and portfolio: EMC Data Domain, EMC Avamar and EMC NetWorker.



The challenges faced by the backup administrators vary from ever-shortening backup windows to growing data needs in the environment. Multiple operating systems and applications with multiple service packs and patches along with the different storage arrays, network topologies, and target devices -- all add to the complexity.

There is no one backup solution available in the market that supports all operating system flavors and their versions or all databases and applications in all environments. Most backup solutions support major operating systems and major applications and provide the flexibility to backup non supported applications via the cold backup method and using scripts. It is important to find a solution that meets the organization’s requirements today and scales for future requirements.



The ever-increasing amount of data for backup presents a challenge to organizations facing the demands of shorter backup windows, quicker restore responses, consistent backups of remote sites and corporate and regulatory requirements. EMC’s backup and recovery products help organizations to meet these challenges through software and disk-based technologies. This slide shows how the EMC portfolio addresses the different needs and requirements in backup and recovery environments.



EMC Data Domain provides disk-based deduplicated storage for a variety of data movers and applications. Data Domain’s deduplication stores only unique sub-file objects, thus greatly reducing the amount of storage needed.

Data Domain systems are qualified with leading enterprise backup software and archiving applications and easily integrate into existing storage infrastructures. A Data Domain storage system can receive data from multiple applications and provide deduplication across data from each.

EMC Data Domain also supports replication for disaster recovery, encryption and data shredding for increased security, and compression for added capacity. Retention is also available to satisfy compliance policies.



The Data Domain storage system provides inline data deduplication during the backup process. A Data Domain appliance is a storage system with shelves of disks and a controller. It is optimized, first to backup and second to archive. Because only unique data is stored on the Data Domain system, the amount of disk storage required to store backup data is dramatically reduced. After backup, deduplicated data can be replicated from one Data Domain system to another providing a safe, tape-free disaster recovery solution.

Data can be transferred into the Data Domain storage system target using NFS, CIFS, VTL or through an optimized protocol such as Data Domain Boost. When using DD Boost, much of the deduplication processing is shifted to backup application components.

The example displayed on the slide shows a simple backup deployment using Data Domain as the backup target. Deduplicated backup data is then replicated from the backup target to an offsite Data Domain system for disaster recovery.



One of the most conventional approaches to deduplication competing with Data Domain is using what’s known as a post process deduplication. In this architecture, data is stored to a disk before deduplication, and then after it’s stored, it’s read back internally, deduplicated and written again to a different area.

Although this approach may sound appealing, seeming as if it would allow for faster backups and the use of less resources, it actually creates problems:

First, more disk is needed to store both the raw data temporarily and the deduplicated data. Post Process deduplication also has an impact on speed because post process deduplication systems are usually spindle-bound. There are typically three or four times more disks in a post-process configuration than you’ll see in a Data Domain deployment.

An inline approach is also much simpler. If data is all filtered before it’s stored to disk, then it’s just like a regular storage system: it just writes data; it just reads data. There’s no separate administration involved in managing multiple pools, some with deduplication, some with regular storage, managing the boundary conditions between them. Any less administration in the storage system is always better. So by being simpler and smaller to provision, an in-line approach, and especially a CPU-centric in-line approach, will always be more attractive.



This slide shows an example of where Data Domain is a good fit. The environment has a very large number of servers with a large and growing amount of backup data. Storage costs are increasing as the amount of data increases.

Data Domain uses deduplication to reduce the amount of physical storage required. More data can be stored on the same amount of disk since redundant data is stored only once. Data Domain can be easily implemented into existing environments since it interfaces with many common products and applications.



Avamar’s unique global, source data deduplication technology eliminates the unnecessary transmission of redundant backup data – only unique data, identified at a sub-file level, is sent over the network and stored for backup. This defuses the enormous amount of data growth both in core data centers and in remote offices. Avamar deduplication is realized at the source (backup client) and also globally, across sites and servers. As a result of deduplicating data in this fashion, Avamar technology can shrink the amount of time required for backup, as well as substantially reduce network utilization for backups. This data reduction also alters the fundamental economics of disk versus tape, allowing companies to cost-effectively utilize disks at a cost that is equal to, or less than, tape.



Avamar provides specialized database client software plug-ins for backing up and restoring databases. Supported databases and applications include Microsoft Exchange, Microsoft SQL Server, Microsoft SharePoint, Lotus Domino, IBM DB2, Oracle, SAP with Oracle and Sybase.



Data Domain systems can be used as storage for Avamar backup data. Backup data is sent directly from the client to the Data Domain system using DD Boost technology. Backups can then be managed through the Avamar system. This can provide faster backup and recovery, especially for large, active databases. Data Domain integration is supported for DB2, Microsoft Exchange VSS, Hyper-V VSS, Microsoft SQL Server, Microsoft SharePoint VSS, Oracle, SAP with Oracle, Sybase and VMware image backup and restore.

Backups, restores, and replication between Data Domain devices of Avamar backup data stored on Data Domain systems are configured, managed and reported in Avamar. DD Boost software is installed with the Avamar client software.



Avamar is ideally suited for protecting clients in VMware environments by reducing the amount of backup data within and across the virtual machines. Avamar provides the flexibility of implementing a VMware backup in two ways. Avamar agents can be installed in the virtual machines for VMware guest level backups. When integrated with VMware VADP, vStorage APIs for Data Protection, image level backups and restores can be performed using an Avamar proxy client.

VMware backups can be centrally configured, scheduled and managed with Avamar Administrator. Avamar Administrator also has the ability to browse the virtual machines in the environment and display information for each machine as shown on the slide.



Shown on the slide is an example of a site where Avamar is the backup solution. A data center has a growing backup environment. Because of this, backups are not able to finish in the backup window. Also, network bandwidth is limited.

Avamar features client-side deduplication. This means that less data is sent over the network relieving bandwidth usage. Deduplication solves the problem of the growing backup environment by reducing the amount of capacity required. Backup times also decrease.



NetWorker works within the existing framework of the hardware, operating system, software, and network communication protocols to provide protection for critical business data by centralizing, automating, and accelerating backup and recovery operations across an enterprise. EMC NetWorker helps organizations control costs by bringing management and reporting of the entire backup environment into one central solution. NetWorker leverages this centralized, broad protection to bridge the gap between traditional backup and new backup technologies, providing a common platform for backup to tape, backup to disk, snapshot management and replication management.

Traditionally, the industry used tape backups that follow a one-size-fits-all strategy. However, tapes are challenged to meet service-level requirements. EMC NetWorker provides the capability to use both tapes and disks, as well as other types of media, for backup. It provides better recoverability from tape backups by leveraging a future-proof open tape format to recover from damaged tape media. NetWorker enables simultaneous-access operations to a disk volume for both reads and writes, as opposed to a single operation with tapes. NetWorker supports backup to both private and public cloud configurations. Cloning and staging capabilities from one media to another are supported. . NetWorker maintains security to avoid recovery of data by unauthorized users. Recoverable data can include files, directories, file systems, or application data.

Industry-leading deduplication capability is provided via integration with EMC Avamar and EMC Data Domain. Implementing deduplication in NetWorker is extremely flexible. A single NetWorker server can manage both traditional and deduplication backups. Deduplication backups can be enabled for both existing and new NetWorker clients, and a single client physical host can be configured for both deduplication and traditional backups.



EMC NetWorker application modules act with third-party applications, together with NetWorker, to provide a comprehensive data storage management system. NetWorker modules allow applications to be backed up in a consistent state.

NetWorker application modules fully integrate with the third-party vendor-specific APIs, eliminating the need to develop or maintain custom backup and recovery scripts. They provide fast, online, automated, and reliable granular backup and recovery for popular database, messaging, content, and ERP applications.

Available NetWorker application modules include the modules listed here.



NetWorker integrates with Avamar to provide source-based deduplication backups. This method of backup identifies and stores only unique, variable-sized, sub-file data objects. Redundant data is identified at the source (client) machine and only unique data is sent over the network to the deduplication node and stored on disk. Backup data consumes significantly less network and backup storage resources as only unique data objects are stored. NetWorker deduplication with Avamar takes advantage of reliable disk storage for primary backup storage with the potential benefits of reducing backup and recovery times.

The major components of the NetWorker deduplication environment are the NetWorker deduplication node, client, and storage node. The NetWorker deduplication node, an Avamar server that has been configured for NetWorker deduplication, stores deduplicated client backup data. A NetWorker client resource is configured for deduplication backup of the identified computer and save sets. Primary processing for NetWorker deduplication is performed by the client software. The NetWorker storage node stores hash ID metadata that is used for recovery of data stored on the deduplication node and to enforce NetWorker retention policies.

NetWorker provides a central point for configuring and managing backups, both traditional and deduplication, through the use of NetWorker workflows and administrative, backup, and recovery interfaces. Backup specifications are defined in NetWorker using NetWorker resources. NetWorker Management Console is used to monitor Avamar server system events and report on deduplication backup activity.



NetWorker can take advantage of Data Domain deduplication in several ways.

Data Domain can be used for backup storage as a virtual tape library and as a disk type device. In this environment, backup data is sent to the Data Domain system which then deduplicates and stores the data. NetWorker and the Data Domain system are each managed separately. Any deduplication within the Data Domain system is not visible to NetWorker. If data is replicated in Data Domain, the replicated data is not visible to NetWorker.

When integrated with Data Domain through the use of the NetWorker Data Domain device type, DD Boost technology is used. This significantly increases backup performance by performing deduplication processing either at the storage node or the backup client.

With Data Domain integration, NetWorker provides the configuration and management of backups and restores through the use of NetWorker interfaces. NetWorker can also monitor Data Domain hardware, detect Data Domain capacity utilization, and receive alert notifications. Cloning of data (replication) stored on Data Domain is managed by NetWorker. These cloning enhancements also enable replicated data at a remote Data Domain facility to be used to make a clone of the data to tape.



NetWorker provides support for backup and recovery of VMware virtual clients in two ways: guest level backups and image level backups with VADP, vStorage APIs for Data Protection.

With a guest backup, the virtual machine is treated like a normal backup client with the NetWorker client software installed on the host virtual machine. When integrated with VADP, vSphere virtual machines can be backed up without using backup agents inside the virtual machines. The NetWorker client software is installed on a proxy machine, which depending upon the configuration method used, can be either a physical or virtual machine. With this method, the virtual machines being backed up are not impacted by backup processing.

Backups and restores of VMware virtual machines can be centrally configured, scheduled and managed with NetWorker interfaces. Additionally, NetWorker can automatic ally discover the virtual machines in a vCenter environment and display information for each virtual machine in tabular or graphical format as shown here.



This slide shows an example of an environment where NetWorker solves a number of problems. Backups consist of large files that are distributed in multiple remote locations. Backup to a central server is problematic because of networking constraints. The customer does not want to have to manage multiple backup environments at each remote office.

NetWorker provides the ability to deploy multiple storage nodes at each of the remote offices. In this way, large files do not have to be sent over large distances. Backups at the remote sites can all be managed centrally using NetWorker’s administrative interface.



This lesson covered an overview of EMC’s Backup and Recovery strategy and portfolio: EMC Data Domain, EMC Avamar and EMC NetWorker.



The purpose of the course exercise is to use what you have learned in this course to design a backup and recovery solution that addresses the company’s concerns described above. Justify how your solution will ensure that the company’s needs are met.



These are the key points covered in this course. Please take a moment to review them.


backup recovery systems and architecture student guide

Documents