emc business continuity for microsoft exchange enabled by ... · pdf filecontents 4 emc...

EMC Business Continuity for Microsoft ExchangeEnabled by EMC® Celerra®, Replication Manager,

VMware HA, DRS, and SRM

Reference Architecture

EMC NAS Product Validation

Corporate Headquarters Hopkinton, MA 01748-9103

1-508-435-1000 www.EMC.com

Copyright © 2009 EMC Corporation. All rights reserved.

Published May 2009

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All other trademarks used herein are the property of their respective owners.

EMC Business Continuity for Microsoft Exchange Enabled by EMC Celerra, Replication Manager, VMware HA, DRS, and SRM


P/N H6312

2 EMC Business Continuity for Microsoft Exchange Enabled by EMC Celerra, Replication Manager, VMware HA, DRS, and SRM


Contents

Chapter 1 Solution Overview........................................................................................................................... 7 Executive summary ......................................................................................................................... 8 Introduction ..................................................................................................................................... 8 Audience ......................................................................................................................................... 8 Business benefits ............................................................................................................................. 8 Technology challenges .................................................................................................................... 9 System configuration..................................................................................................................... 10

Storage architecture ..............................................................................................................11 Network architecture ............................................................................................................11 Virtualization layer ...............................................................................................................12 Application architecture........................................................................................................12 High availability and failover ...............................................................................................12

Conclusion..................................................................................................................................... 13 Related documents ........................................................................................................................ 14

Chapter 2 Storage Sizing and Scaling............................................................................................................ 15 Introduction ................................................................................................................................... 16 Reference configuration ................................................................................................................ 16

Workload ..............................................................................................................................16 Test duration .........................................................................................................................16 Server configuration .............................................................................................................17 Storage configuration............................................................................................................18

Applying the reference .................................................................................................................. 19 Workload evaluation.............................................................................................................19 Server memory sizing ...........................................................................................................19 Storage sizing .......................................................................................................................19 Conclusion ............................................................................................................................20

Chapter 3 Virtualization Options ................................................................................................................... 21 Introduction ................................................................................................................................... 22 VMware Distributed Resource Scheduler (DRS).......................................................................... 22 VMware High Availability (HA) .................................................................................................. 23 VMware vCenter SRM.................................................................................................................. 24 Conclusion..................................................................................................................................... 24

EMC Business Continuity for Microsoft Exchange Enabled by EMC Celerra, Replication Manager, VMware HA, DRS, and SRM 3


Contents



Tables

Table 1 Solution advantages........................................................................................................9 Table 2 Schedule of LoadGen tests ...........................................................................................16 Table 3 Server hardware............................................................................................................17 Table 4 Server software.............................................................................................................17 Table 5 Application file or activity types and locations ............................................................19 Table 6 Exchange environment downtime during HA and DRS failover .................................23 Table 7 Recovery time during SRM failover.............................................................................24

EMC Business Continuity for Microsoft Exchange Enabled by EMC Celerra, Replication Manager, VMware HA, DRS, and SRM 5 Reference Architecture

Tables



Chapter 1 Solution Overview

This chapter presents these topics:

Executive summary ......................................................................................................................... 8 Introduction ..................................................................................................................................... 8 Audience ......................................................................................................................................... 8 Business benefits ............................................................................................................................. 8 Technology challenges .................................................................................................................... 9 System configuration..................................................................................................................... 10 Conclusion..................................................................................................................................... 13 Related documents ........................................................................................................................ 14


Solution Overview

Executive summary

Messaging is one of the mission-critical applications for most businesses today. Microsoft Exchange Server 2007 is the leading messaging platform for customers ranging from large enterprises to branch offices to small business units. The Exchange messaging environments are growing in complexity and user requirements have become increasingly demanding. Additionally, the manner in which Exchange is used to support business operations has changed. It is now even more business-critical than ever before. Therefore, it is important to design the Exchange environment for optimum performance and high availability.

An increasing number of companies are exploring ways to virtualize Exchange environments to reduce costs, increase availability, add flexibility, and use resources more efficiently. The validated solution uses VMware Infrastructure 3 to build a three-node, high-availability (HA) cluster of ESX servers to host the Exchange environment. This cluster was also enabled with VMware Distributed Resource Scheduler (DRS) to balance the workloads across all three ESX servers. The production Exchange site is protected by using VMware vCenter Site Recovery Manager (SRM) along with EMC® Celerra Replicator™ and EMC Replication Manager for high availability and automatic failover.

Introduction

Exchange is considered to be the backbone of messaging environments and mailbox sizes are growing at a rapid pace. As the number of users, type of files, and departments grow, so does the requirement for a scalable, reliable, and cost-effective Exchange architecture to meet the new business requirements. While IT administrators prepare to meet growth and architecture changes, they are challenged by issues related to power and cooling, efficient asset utilization, security and provisioning needs, and business continuance. Customers understand that unmanaged growth is not viable and are tasked to return measurable benefits from data center consolidation and virtualization. A virtualized data center infrastructure enables the customer to efficiently deploy an Exchange messaging environment.

Audience

This document is intended for internal EMC personnel, EMC partners, and customers.

Business benefits

This document provides findings discovered during testing that enable you to store, protect, and back up e-mail messages more efficiently. It simplifies e-mail lifecycle management, accelerates backup and restore processes, protects your operations in case of unplanned site failure, and reduces costs. You can also efficiently manage an Exchange environment without requiring valuable time from end users or overburdening IT resources.

To manage the growth of the messaging environment, the IT administrator needs to:

• Maintain uptime

• Restore databases quickly

• Maximize server and storage utilization and deliver optimal system performance

• Manage upgrades and migrations



Solution Overview

• Reduce the demands on limited IT resources and budgets

This solution addresses each of these challenges by using tested and proven solutions.

Table 1 provides the details about the benefits of the solution.

Table 1 Solution advantages

Benefits Details

Maintain service levels Ensure that critical and revenue-generating Microsoft applications are available and running at peak performance

Reduce costs Minimize the cost of server and information management

Reduce physical footprint Replace two physical servers with one VMware server in this architecture, saving power and space

Reduce risk Offer a reference architecture that includes tested and proven configurations that improve performance and scalability

Back up and restore at the storage group level

Offer backup and restore capabilities with NetWorker® Module for Microsoft Applications

Schedule backups • Automate full or incremental backup jobs

• Enable a quick view of backup and restore job history

• Support schedule and incremental backup

Provide site resiliency • Replicate storage with Celerra Replicator

• Automate site failover with VMware vCenter SRM

Maintain business continuity Ensure that the messaging environment is highly available for businesses

Technology challenges

E-mail management is difficult, particularly in distributed environments where skills vary from site to site. For many businesses, incomplete backup and recovery along with insufficient disaster recovery strategies are the norm. These businesses may face the additional challenge of insufficient resources to research, test, and validate the best possible storage solution for their Exchange environment. EMC Proven™ Solutions enable you to start from a known reference configuration, and then customize it for your needs by examining the various methods to accomplish common tasks in a database environment. This includes options for:

♦ Server virtualization

♦ Data backup and recovery

♦ Site replication and protection


Solution Overview

System configuration

VMware Infrastructure 3 enables guest operating systems to connect to the storage by using several methods. In this reference architecture, virtual machines (VMs) reside in VMware vStorage Virtual Machine File System (VMFS) data stores on the ESX server. The ESX servers connect to Celerra® iSCSI LUNs using ESX’s built-in iSCSI initiator. Figure 1 provides an overview of the three-node ESX cluster architecture.

Figure 1 System overview

A cluster is a collection of ESX hosts and associated VMs with shared resources and a shared management interface. When you add a host to a cluster, the host’s resources become part of the cluster’s resources. When you create a cluster, you can choose to enable it for DRS, HA, or both. For this configuration, a VMware cluster was created by using three ESX servers. Three resource pools were created from the cluster for the VMs. Both HA and DRS were enabled for the resource pool. Five VMs that run the Exchange environment were balanced across the ESX servers. In case of any physical ESX server failure, the VMs on the failed server will be automatically restarted on the surviving ESX server. VMware vCenter SRM is a disaster recovery framework that integrates with the various EMC replication software products such as Celerra Replicator to automate the failover process of the VMware vStorage VMFS data stores. The Exchange mailbox VM is protected with VMware vCenter SRM and Replicator. Replication Manager is used for crash-consistent replication of all Exchange mailbox VMFS data stores.



Solution Overview

You can use EMC NetWorker to back up the Exchange environment by using the EMC NetWorker Module for Microsoft Applications (NMM). This solution can back up the Exchange environment to Advanced Technology-Attached (ATA) drives on the same Celerra system that hosts the Exchange data and VMs.

Storage architecture

The validated solution can use the storage through the CIFS and iSCSI protocols. The CIFS protocol provides access to the storage area that is used for database backups. The iSCSI protocol provides data storage to all VM operating systems (OSs), Exchange databases, and logs. The underlying architecture of each storage area is based on the anticipated workload that is applied to that area. These areas can have vastly different I/O performance, space, and protection requirements. Therefore, you must maintain these areas separately.

iSCSI area – VM operating systems, Exchange databases, and logs

To accommodate a range of potential implementations and to enable several array-based functions, the validated solution specifies that the primary application database must reside on the Celerra iSCSI storage. You must size this area based on the requirements of the specific database application. For VMware DRS to work, the VM OSs must reside on the Celerra by using iSCSI.

CIFS area – Database backups

Database backups are critical to the administration of the production applications. The CIFS share hosted on Celerra provides the storage space to accommodate full and incremental backups.

Network architecture

System-wide network design and architecture are outside the scope of this solution. This section presents recommendations for proper functionality that are consistent with industry-accepted best practices and compatible with the existing network infrastructure and policies.

Switches

In any highly available solution, you must have multiple network paths between each component in the system so that the failure of a component does not disrupt communication. The solution architecture illustrated in Figure 1 defines a set of logical networks that are recommended for the solution. The networks do not provide failover protection. Therefore, the networks should be designed such that they are highly available on their own.

The validated solution uses Gigabit Ethernet (GbE) switches.

Virtual local area networks

When you create highly available networks, the requirement to have multiple switches on multiple networks can quickly become costly. Virtual local area networks (VLANs) allow each physical network switch to maintain multiple virtual networks that are logically isolated from one other. This solution was validated by using VLANs within a highly available network and can be implemented with logical separation by using VLANs, or with physical separation using distinct physical switches.

EMC Celerra NX4

EMC Celerra NX4 contains two Data Movers. The Data Movers can operate independently. They can also operate in the active/passive mode, with the passive Data Mover serving as a failover


Solution Overview

device for the active Data Mover. In this solution, the Data Movers operate in active/passive mode.

The NX4 Data Mover has four network ports. Figure 2 on page 12 shows the ports on the rear of an NX4 Data Mover.

Figure 2 EMC Celerra NX4 Data Mover ports

Ports cge0 and cge1 handle the storage traffic. Port cge2 is used for the backup-to-disk traffic and port cge3 is left open for future requirements.

The Data Mover supports several types of link aggregation for IP traffic. However, in this configuration, no link aggregations or Ethernet channels were configured.

Note: As a best practice, dedicate the Data Mover network ports connected to the storage network to storage traffic. However, if the ports are not heavily used, they can be shared with non-storage network traffic. EMC recommends monitoring the network to avoid bottlenecks.

Virtualization layer

The validated solution uses multiple virtual components to save space in the data center and also to reduce power and cooling costs. As with all other layers in the solution stack, ensure that the impact of any virtualization technology on the performance, availability, and recoverability of the system is well understood before implementing it in a production environment. The validated solution uses advanced VMware features such as VMware High Availability clusters, VMware Distributed Resource Scheduler, and SRM. Chapter 3 Virtualization Options provides more information about the virtualization layer.

Application architecture

The validated solution uses four Exchange storage groups on a single mailbox server. The Exchange storage group database files, log files, and the Exchange server guest OS reside on separate EMC Celerra iSCSI LUNs. The Exchange server role is divided into two virtual machines. The Exchange Hub Transport server and Client Access server reside on a virtual machine, and the mailbox server resides on another virtual machine.

High availability and failover

The validated solution provides protection at the storage layer, connectivity layer, ESX server layer, and application layer.



Solution Overview

Storage layer

Celerra can have multiple Data Movers to provide high availability and load balancing. In the Microsoft Exchange Server 2007 EMC Celerra solution, the primary and standby Data Movers provide seamless failover capabilities for the Celerra storage. This minimizes the end-user disruption during routine Celerra maintenance such as upgrading the data access in real time (DART) OS.

The RAID disk configuration on the Celerra backend provides protection against hard disk failure.

Connectivity layer

The solution configuration also includes separate network interface cards (NICs) at the source of each I/O path, separate network infrastructure (cables, switches, routers, and so on), and separate ports.

ESX server layer

The ESX servers have redundant power supplies and network connections as protection against hardware failures. In the validated solution, the ESX servers are clustered by using VMware HA and DRS clustering, thus ensuring high availability and greater performance.

Application layer

The Exchange application is protected by implementing redundant functionality and disaster recovery plans. The domain controller needs to be available for the Exchange application to work. Therefore, the protected site and recovery site have one domain controller to ensure that the domain controller is protected against site failure. Similarly, Hub Transport servers and Client Access servers are installed at the protected and recovery sites. To protect the Exchange mailbox server, a disaster recovery plan is implemented with VMware SRM along with Celerra Replicator and Replication Manager.

Conclusion

This document provides a reference architecture for business continuity solutions based on Microsoft Exchange Server 2007 with EMC Celerra. Several technologies, products, and considerations are presented as working together to create a system that can meet performance, reliability, availability, and recovery targets. However, a customer environment can have different needs. With this in mind, each of the subsequent chapters discusses the reference configuration for a specific functionality, along with other options that can be used by the customer.

Exchange is a highly configurable environment. This document attempts to provide a framework to implement the solutions that address the inherent complexity of many environments and retain the ease of use that characterizes the EMC Celerra platform.


Solution Overview



Related documents

The following documents, located on Powerlink®, provide additional, relevant information. Access to these documents is based on your login credentials. If you do not have access to the following documents, contact your EMC representative:

♦ EMC Solutions for Microsoft Exchange 2007 EMC Celerra Unified Storage Platforms — Applied Best Practices Guide

♦ EMC Replication Manager Administrator’s Guide

Chapter 2 Storage Sizing and Scaling


Introduction ................................................................................................................................... 16 Reference configuration ................................................................................................................ 16 Applying the reference .................................................................................................................. 19


Storage Sizing and Scaling

Introduction

Understanding sizing and scaling is critical to deploy this solution to Exchange environments. Sizing systems for Exchange workload is a complex task that deserves a great deal of thought and planning to ensure that systems behave in a manner that is acceptable to the various stakeholders.

Reference configuration

This section outlines one configuration that is validated with the solution modules presented here. This solution is an example of a system that can be implemented. However, this solution is not an answer to all customer issues. Exchange solutions can vary drastically based on individual customer requirements. Each customer environment must be evaluated, analyzed, and sized individually. The configuration presented here can be used as a reference point so that the interactions between solution modules can be explored.

Workload

LoadGen is a Microsoft tool that simulates Exchange traffic. LoadGen enables you to test how an Exchange server responds to an e-mail load. Specifically, it helps you determine if the Exchange server can handle an e-mail load as expected.

Test duration

The test was run with Store, Backup, and Protect modules concurrently for 48 hours. The test was run for 48 hours to relate it to a typical production environment. The day-to-day operations in a typical production environment were scheduled in the test environment to study the performance.

Test user profiles

This solution is targeted for commercial and midsize enterprises with the following Exchange characteristics:

♦ User mailbox size for testing: 250 MB

♦ Exchange user count: 1,000 Exchange users

♦ User I/O profile for testing: Heavy and light users. Table 2 describes the test schedule for different user I/O profiles.

Table 2 Schedule of LoadGen tests

Day Test User profile Start time Test duration

1 Master LoadGen1 Light 6 A.M. 48 hours

1 Master LoadGen2 Heavy 8 A.M. 10 hours

2 Master LoadGen2 Heavy 8 A.M. 10 hours




Server configuration

The ESX server has 32 GB of memory and four physical processors. The Exchange mailbox virtual machine is configured with four virtual processors and 16 GB memory. The Exchange Client Access, Hub Transport Server and domain controller are configured with two virtual processors and 4 GB memory.

Table 3 lists the hardware used for this solution.

Table 3 Server hardware

Name Quantity Description

2U server 4 • Two 3 GHz Intel Xeon dual-core processors

• 32 GB memory

• One 60 GB SCSI disk

• Two onboard 10/100/1,000 Mb Ethernet NICs

• Four additional 10/100/1,000 Mb Ethernet NICs

1U server 2 A dual-processor with a speed of 2.8 GHz and 4 GB RAM

Gigabit Ethernet switch 1 Copper GbE ports

EMC Celerra NX4 2 • Two Data Movers

• Four GbE network connections

• Two FC shelves

• One ATA shelf

• Twelve 300 GB (15k) FC disks and five system disks

• Fifteen 1,000 GB SATA disks

Table 4 lists the software used for the Exchange solution.

Table 4 Server software

Name Quantity Description

Windows 2003 Server Enterprise Edition, 64-bit SP2

3 Exchange Server

Windows 2003 Server Enterprise Edition, 32-bit SP2

2 AD server

VMware ESX 3.5.0 (123630) 3 ESX server OS

Microsoft Exchange Server 2007 Enterprise Edition 64-bit SP1

3 Exchange Mailbox/Hub Transport/Client access server

EMC NetWorker® 7.5 1 NetWorker server installed on the virtual machine residing on ESX

EMC NetWorker Module for Microsoft Applications 2.1

1 Installed on Exchange Server



Storage configuration

The Microsoft Exchange Server 2007 binary files reside on RAID 5 five-spindles file systems. Database files and log files reside on EMC Celerra Unified Storage Platform file systems. Each type of file resides on a separate file system. The file system for the backup-to-disk (B2D) files resides in two RAID 5 seven-disk storage pools.

The Exchange server contains four databases and four storage groups (based on the guideline of one database file per storage group). Each database file resides in its own iSCSI LUN. The file system is striped across two RAID 1 sets (four disks). The database log files reside in their own iSCSI LUN. The file system resides in a single RAID 1 set (two disks).

Figure 3 outlines how storage is provisioned in the validated solution.

Figure 3 Storage layout




Table 5 on page 19 lists each file or activity type and indicates where it resides or where it is recorded.

Table 5 Application file or activity types and locations

File or activity type

Location Number of iSCSI LUNs

RAID level

Celerra storage pool

Total number of physical disks

Type

Database log files

NX4 4 RAID 1 Logs 4 FC

Database files

NX4 4 RAID 1 Database 8 FC

OS files NX4 5 RAID 5 Clar5_performance 5 FC

B2D database files

NX4 NA RAID 5 clarata_archive 10 ATA

Applying the reference

This configuration provides a reference point when sizing a deployment in a real environment. By understanding how the real workload relates to the test workload, and how changes to the test system impact that workload, you will understand how changes will impact the real-world environment. This changes and results indicated here will not translate exactly into another environment. This must be used as a reference to take decisions that impact customers.

Workload evaluation

The first step in any sizing exercise is to evaluate the workload, or plan the workload, which will be handled by the system. It is important to understand how the workload will translate into storage requirements so that the I/O performance capacity can be determined. There are numerous ways to accomplish this task, which will vary based on the environment.

Server memory sizing

In Exchange environments, memory and I/O are intrinsically linked. A large amount of memory allows web pages to be served from memory instead of disk, which is very efficient.

Storage sizing

Factors such as metrics of capacity need to be considered when sizing database storage. Disks have two primary metrics of capacity: storage capacity and performance capacity. Storage capacity is typically measured in gigabytes (GB) and performance capacity is typically measured as Input/Output Operations per Second (IOPS). Both must be considered when designing a storage system. It is easy to determine the storage capacity in most environments. However, the performance capacity is often overlooked as it is difficult to discern.

When examining a disk, determine the performance capacity. After this number is known, it is easy to evaluate the storage capacity. EMC Solutions for Microsoft Exchange 2007 EMC Celerra Unified Storage Platforms — Applied Best Practices Guide, available on Powerlink, gives you more information on this.





Conclusion

The reference configuration can be used as a starting point to evaluate the data storage options available in your environment. This configuration is intended only as a reference and should not be substituted for evaluation and testing of the actual environment and workload.

Chapter 3 Virtualization Options


Introduction ................................................................................................................................... 22 VMware Distributed Resource Scheduler (DRS).......................................................................... 22 VMware High Availability (HA) .................................................................................................. 23 VMware vCenter SRM.................................................................................................................. 24 Conclusion..................................................................................................................................... 24


Virtualization Options

Introduction

Many companies are exploring ways to virtualize Exchange environments to reduce costs, increase flexibility, and use resources more efficiently. When you virtualize the Exchange environment on Celerra, you can:

♦ Consolidate and contain Exchange servers without compromising on performance

♦ Optimize the storage and network infrastructure

♦ Ensure high availability and business continuity

♦ Enable dynamic pooling and sharing of resources

♦ Allow effective capacity management and automation

The validated solution uses VMware Infrastructure 3 to allow a physical system to host more than one logical system. This is intended as a proof of concept and not as a requirement. This section describes the different VMware virtualization features used in this solution.

VMware Distributed Resource Scheduler (DRS)

VMware Distributed Resource Scheduler (DRS) dynamically allocates resources to enforce resource management policies while balancing resource usage across multiple ESX hosts. VMware DRS continuously monitors utilization across the resource pool and intelligently aligns resources with application needs.

Figure 4 VMware DRS

DRS allows the system to:

♦ Dynamically allocate IT resources to the highest priority applications. Create rules and policies to prioritize how resources are allocated to virtual machines.

♦ Give IT autonomy to business organizations. Provides dedicated IT infrastructure to business units while still achieving higher hardware utilization through resource pooling.




♦ Empower business units to build and manage virtual machines within their resource pool while giving central IT control over hardware resources.

VMware High Availability (HA)

VMware HA continuously monitors all physical servers in a resource pool and restarts virtual machines affected by server failure. The tested environment has three ESX servers that are being monitored by VMware HA.

Figure 5 VMware HA

VMware HA also does the following:

♦ Monitors and detects virtual machines for guest OS failures and automatically starts virtual machines after user-specified intervals.

♦ Detects server failures automatically, by using a heartbeat on servers.

♦ Restarts virtual machines almost instantly without human intervention on a different physical server within the same resource pool.

♦ Continuously monitors and selects the optimal physical servers within a resource pool on which to restart virtual machines (if used in conjunction with VMware DRS).

Table 6 shows the time taken to recover from a physical ESX server failure by using VMware HA and DRS features:

Table 6 Exchange environment downtime during HA and DRS failover

Failed ESX server Exchange Server RPC operations downtime

Server downtime

ESX hosting Exchange mailbox

5 minutes 10 seconds 4 minutes 2 seconds

ESX hosting the Exchange HUB/CAS and AD/DC

0 seconds 40 seconds

EMC Business Continuity for Microsoft Exchange Enabled by EMC Celerra, Replication Manager, VMware HA, DRS, and SRM 23





VMware vCenter SRM

VMware vCenter SRM extends the VMware vCenter Server interface with array-based replication tools. SRM simplifies and automates the key elements of disaster recovery such as creating disaster recovery plans, testing the plans, and executing failover in a data center disaster scenario. The benefits of using SRM include:

♦ Accelerating recovery with automated process

♦ Ensuring reliable recovery through automation and easier testing

♦ Taking control of your disaster recovery plans

The validated SRM failover shows that the process of failover during a disaster is fully automated with minimal user intervention compared to the Microsoft Exchange SCR disaster recovery solution or Celerra iSCSI disaster recovery solution. Table 7 shows the recovery time taken by VMware SRM during the failover. The Exchange Server recovery time denotes the actual downtime for the Exchange mailbox server.

Table 7 Recovery time during SRM failover

SRM recovery execution time Exchange Server recovery time

15 minutes 49 seconds 8 minutes 56 seconds

Conclusion

Virtualization provides opportunities to reduce costs, increase flexibility, and use resources efficiently. Many of the components in this solution can be virtualized depending on your specific needs. However, no components in this solution explicitly require virtualization to be considered part of the validated solution.

emc business continuity for microsoft exchange enabled by ... · pdf filecontents 4 emc...

Documents