microsoft exchange® server 2007 on systems vmware® esx · by deploying exchange 2007 server on...

Virtualizing Microsoft®

Exchange® Server 2007 on Dell™ Systems Running VMware® ESX A Dell Technical White Paper

End‐to‐End Solutions Engineering

By Ramesh Radhakrishnan and Tom King

Dell Product Group

May 2009

Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX

Page ii

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

© 2009 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.

Dell, the DELL logo, EqualLogic, and the DELL badge are trademarks of Dell Inc.; Microsoft, Outlook, and Windows are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries.


Page 1

Table of Contents Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX .......................................... i

Executive Summary ....................................................................................................................................................... 2

Introduction ................................................................................................................................................................... 2

Virtualized Exchange Server 2007 Building Block .......................................................................................................... 3

Solution Architecture ................................................................................................................................................. 3

Deployment Best Practices ........................................................................................................................................ 4

Network Setup and best practices ......................................................................................................................... 5

Storage Best Practices ........................................................................................................................................... 5

Performance Evaluation of Virtualized Mailbox Role .................................................................................................... 5

Test Methodology .................................................................................................................................................... 5

Scalability Testing .................................................................................................................................................... 6

Conclusion ..................................................................................................................................................................... 9

References ..................................................................................................................................................................... 9

Appendix A................................................................................................................................................................... 10


Page 2

Executive Summary This paper provides an overview of deploying Exchange 2007 Server in a virtualized environment using

VMware 3.5 Update 3 on Dell PowerEdge Servers and EqualLogic iSCSI storage array. In this paper a solution

building block is proposed for 4000 mailbox users to deploy in an ESX environment running on Dell hardware.

Various mailbox configurations of Exchange Server 2007 are tested to understand the sizing requirements for

Exchange Server 2007 in a virtualized environment and validate the building block.

Introduction There is a growing interest in deploying Exchange Server 2007 in virtualized environments with the recent

announcement regarding change in support policies for virtualized Exchange deployments1. In a virtualized

environment, Exchange users can consolidate multiple Exchange Server Mailbox roles on a single physical

server to take advantage of unused CPU cycles or available memory. Additional Exchange Server roles (Hub

Transport or Client Access Server) can be hosted on virtual machines (VM) and consolidated on the same

hardware to reduce costs. Benefits of running Exchange in a virtualized environment will depend on the

customer environment. Some of the advantages of virtualizing Exchange 2007 Server are:

• Cost savings thru server hardware consolidation

• Reducing performance bottlenecks by moving Mailbox VMs to systems with available CPU cycles

• Easier management of Exchange systems and infrastructure

In this paper a sample configuration is presented which is tested and validated to meet the performance and

stability requirements of an enterprise Exchange deployment. This sample configuration is scalable and can

be used as a building block for larger scale deployments. By deploying Exchange 2007 Server on ESX

environment hosted on Dell Servers and EqualLogic iSCSI SAN, organizations can reduce the management

complexity and improve utilization of hardware resources by consolidating on fewer hardware resources.

To provide sizing guidelines for Exchange Server 2007 the Mailbox role was deployed in a Virtualized

environment and tested using multiple Mailbox configurations. The trending data was used to understand

the CPU and memory requirements for Exchange Server 2007 running within a VM and how it differed

compared to a physical environment. For consolidation purposes, the performance of deploying Exchange

Server 2007 Mailbox roles in multiple VMs on a single server is also evaluated.

This white paper is intended for customers, IT architects, storage administrators and IT administrators who are evaluating, planning, or deploying Exchange 2007 Server on VMware ESX 3.5 platform and require guidance on available options and best practices. The goals of this white paper are to introduce customers to Dell’s building block for deploying Exchange 2007 Server on VMware ESX and simplify the decision making process for the customer when deploying Exchange 2007 in Dell hardware running ESX hypervisors.

1 Microsoft Support Policies and Recommendations for Exchange Servers in Hardware Virtualization Environments, http://technet.microsoft.com/en‐us/library/cc794548.aspx


Page 3

Virtualized Exchange Server 2007 Building Block This section presents an overview of the building block configuration for an Exchange Server 2007

deployment which uses VMware ESX to create a virtualized environment. The building block presented in

Figure 1 was tested using LoadGen tool to simulate a 4000 heavy user workload and it comfortably met the

performance requirements recommended by Microsoft for good quality of service. The Performance

Evaluation section describes how this building block configuration performed at the server, storage and

application level when LoadGen was used to simulate up to 4000 users.

Figure 1 – Architecture of Building Block for Exchange 2007 on ESX 3.5

Solution Architecture In this section the core components that make up the Dell|VMware Virtualized Exchange Building Block are

detailed. The architecture can be broken into four primary layers, each coexisting and working together to

provide an optimized platform for messaging deployment targeting small to mid‐size organizations or

businesses. The individual layers of the building block are –

• Layer 1: Dell PowerEdge Servers

• Layer 2: Dell|EqualLogic iSCSI SAN Storage

• Layer 3: VMware Infrastructure 3

Vmware ESX

W8OS

Active Directory VM

W8OS

CAS/Hub Transport VM

Virtualized Storage Environment

Clients

PC6248 PC6248

W8OS

Exchange Mailbox VM

EqualLogic PS6000XV

Dell PowerEdge R710

EqualLogic PS6000XV

Workload LAN

iSCSI Fabric

Networking Legend


Page 4

• Layer 4: Windows Server 2008 and Microsoft Exchange Server 2007

Layer 1: Selecting the right server for a virtualized Exchange environment requires careful consideration since

it could negatively impact the cost and performance of the overall solution. The proposed building block uses

the Dell PowerEdge R710 server with dual quad‐core processors (Intel Xeon 5500 series). The R710 server is

optimized for virtualization solutions with its enhanced memory and I/O features. Microsoft recommended

guideline should be followed when determining the CPU and memory resources being assigned to the

individual machines running the Exchange 2007 Roles. [1]

Layer 2: The storage solution used for the virtualization building block is based on the Dell EqualLogic 6000

Series iSCSI SAN. EqualLogic storage arrays provide a cost‐effective virtualized storage solution that offers

excellent performance for messaging environments. The number and size of volumes for Exchange 2007

mailbox databases and logs were based on best practices recommended by Dell and Microsoft and depends

on the number and size of user mailboxes. When planning storage architecture for Exchange 2007 the

Exchange 2007 Storage Requirements calculator is a valuable tool. [2]

Layer 3: VMware Virtualization Infrastructure 3 (VI3) was used as the virtualization foundation to create the

virtualized messaging environment. VI3 abstracts hardware resources (CPU, memory, storage and network)

to components that can serve multiple virtual machines providing greater hardware utilization. Using a

virtualized environment allows the hardware to run multiple virtual machines that host the individual pieces

of the Exchange 2007 solution. Multiple virtual machines can be used to run the Exchange 2007 Mailbox role,

each hosting a small number of users to allow more flexibility to the administrator in terms of migrating VMs

across physical servers. Some solution may use a larger VM configuration and host a larger number of users

per VM while still taking advantage of virtualization features like decoupling from hardware and increasing

hardware utilization levels.

Layer 4: The final layer consists of the guest OS running in the VMs and the application itself. The virtual

machines run Windows Server 2008 OS editions and Exchange Server 2007 roles are installed on the guest

VMs. Additional infrastructure server such as Active Directory server may be virtualized as well to increase

server consolidation. Exchange 2007 server best practices should be followed when implemented in a

virtualized environment as well to maximize performance and solution stability [3].

Additional consideration should be made for storage configuration as iSCSI connections can be established at

the guest OS level or it may be configured at the ESX host level. Similarly, network design should be planned

out ahead so that enough network devices are available for both workload LAN and storage fabric to meet

the performance requirements of the messaging solution.

Deployment Best Practices In this section, we discuss a few of the best practices and proceedures that should be followed when

implementing the network and storage infrastructure for the virtualized Exchange 2007 deployment.


Page 5

Network Setup and best practices Although not required or necessary, it is better to isolate the iSCSI traffic from the remaining virtual machine traffic. This allows more fine‐grained monitoring and control over the networks’ differing configuration, e.g. Jumbo Frames for iSCSI traffic versus “normal” MTU settings for the remaining traffic.

Further, each network should be divided with multiple physical paths for load balancing and failover capabilities. In our test bed, we used one PowerConnect 6248 for virtual machine network and management traffic connected to one pair of NICs on the ESX server, and another PowerConnect 6248 for iSCSI traffic connected to another pair of NICs from a dual‐port PCIe card on the ESX server. In highly mission‐critical installations, these networks can be further divided to ensure no single point of failure.

ESX provides highly flexible tools when configuring separate networks. See Appendix A, Figure 6 for details of how the network topology was setup using the ESX networking management tool.

Storage Best Practices Throughout each set of tests, each mailbox server was provided 80% of its allocated space for the mailbox database and 20% of its allocated space for the mailbox logs. Each mailbox server’s iSCSI target was not shared with other virtual machines. None of these particular iSCSI targets housed the virtual machines VMDK files.

The ESX array used RAID 10 across 14 disks with two hot spares. Prior testing using Jetstress showed that RAID 10 offers the best performance in similar environments.

Our testing covered three different storage access configurations:

‐ iSCSI initiator on guests with MPIO ‐ iSCSI initiator on ESX with VMFS file systems ‐ iSCSI initiator on ESX with Raw Device Maps (RDM)

If any external systems need access to the database and log files, using RDM is advantageous. Additionally, using RDM allows switching the file system to be accessed with a guest‐based iSCSI initiator utilizing a multi‐path I/O driver without reformatting. On the other hand, VMFS offers added high availability features such as integration with EqualLogic ASM/VMware editions to create VSS‐aware snapshots and backups.

Whereas using the ESX 3.5 iSCSI initiator allows only one session per target, using the guest‐based initiator allowed load balancing through MPIO. However, our tests showed there was no performance gain in doing so due to the increased overhead and added layers between the guests and the iSCSI targets.

Performance Evaluation of Virtualized Mailbox Role This study evaluates the scalability and performance characteristics of running Exchange Server 2007 in a VMware ESX environment. The performance of running Exchange Server roles (Mailbox, Hub Transport and CAS) in a virtual machine is compared against running them in a physical server to measure the virtualization overhead associated with a messaging workload.

Test Methodology


Page 6

The sizing of the VMs (CPU cores and memory) is based on Dell and Microsoft recommended guidelines for a

heavy user Mailbox profile and prior testing done at Dell for physical Exchange deployments. The test

configuration is monitored for bottlenecks by measuring Server, Storage and Exchange related counters

within the Virtual Machine (VM) and also at the host ESX level.

Microsoft Exchange Load Generator (LoadGen) tool (version 08.02.0045) is used to simulate user workloads

and evaluate performance of the Exchange Server 2007 server configurations. For sizing recommendations,

performance is evaluated at multiple levels—application, server, and storage. Performance is measured at

the application level by using Exchange related metrics measured by Exchange LoadGen. Microsoft Reliability

and Performance monitor is used to measure server and storage level performance metrics. Esxtop is used

to collect CPU and memory usage from the ESX server.

Table 1 lists events that were measured and thresholds that were used to determine if a configuration

performed as expected.

Table 1: Counters Used to Evaluate Performance

Performance Counter Expected Value (MS Recommended)

SERVER PERFORMANCE

Exchange VM: Processor(_Total)\% Processor Time Average less than 75%

ESX : Esxtop ‐ Physical Cpu(_Total)\% Processor Time Average less than 75%

APPLICATION PERFORMANCE LoadGen 95th Percentile Send Mail Latency Less than 1000 ms

MS ExchangeIS: RPC Averaged Latency Average less than 25 ms

MS ExchangeIS Mailbox: Messages Queued for Submission

Average less than 50

STORAGE PERFORMANCE Database Disks: Average Sec/Read Less than 20 ms

Database Disks: Average Sec/Write Less than 100 ms

Log Disks: Average Sec/Write Less than 10 ms

Scalability Testing

LoadGen was used to simulate 1000 to 4000 users and tested against the solution building block to study the scalability of the proposed solution configuration. Table 2 describes the VM configuration in terms of CPUs and memory that was allocated to the VM. These configurations are based on the resources that would be recommended for a physical deployment to satisfy similar number of users and workload.


Page 7

Table 2 ‐ Exchange Server 2007 Mailbox VM Configuration

1000 User VM 2000 User VM 3000 User VM 4000 User VMCPUs 2 vCPU 2 vCPU 4 vCPU 4 vCPU Memory 2GB(OS)+5GB 2GB(OS)+10GB 2GB(OS)+15GB 2GB(OS)+20GBHardDisk (OS) 20GB 20GB 20GB 20GB

Figure 2 illustrates the performance and scalability of the proposed architecture. The primary metrics used to compare performance were the host server’s processor utilization and the 95th‐percentile response time for the LoadGen SendMail task. The proposed architecture building block is observed to provide good quality of service as the 95th percentile’s average latency is less than 500 milliseconds and still has ample CPU headroom to handle spikes in the workload.

Figure 2 ‐ Scalability of Exchange 2007 Server in ESX Environment

Figure 3 illustrates two of the RPC specific counters that are commonly monitored to evaluate the health and performance of the messaging environment. RPC averaged latency represents the time in milliseconds that it takes Exchange (store.exe) to process the last 1024 packets received. It represents the shortest amount of time that clients have waited for a response from the server. The RPC averaged latency is well below the threshold of 25 ms for 4000 users indicating that the server is able to process requests in a reasonable amount of time.

The Messages Queued for Submission performance counter indicates the number of messages in the mailbox store that are ready to be sent and are waiting to be submitted to a transport server. The queued messages are observed to be well within the recommended threshold of 50 indicating a healthy configuration.


Page 8

Figure 3 ‐ Application Level Performance

Figure 3 illustrates the observed disk subsystem performance observed for the proposed solution building block using EqualLogic iSCSI Storage Arrays. Disk subsystem bottlenecks that usually manifest as high disk latencies can cause performance bottlenecks that lead to poorly performing Exchange deployments.

Figure 4 ‐ Storage Level Performance

The disks read and write latencies are within the 20 millisecond threshold that is recommended for optimal performance of the Exchange 2007 deployment. The mailbox database disk latencies are observed to increment as the number of users are increased from 1000 to 4000, but still remain well within the 20 ms threshold. The storage subsystem was designed to support 4000 heavy users and took in to account both the capacity requirements and the performance requirements for this workload. A RAID 10 configuration was used for both database and log disks to meet the performance requirements of the LoadGen heavy workload profile.

Finally, in Figure 5 we compare the performance of the virtualized Exchange 2007 solution to a physical deployment to quantify the impact of virtualization. SendMail latency, CPU loads and RPC latency is


Page 9

compared for virtual and physical deployments of 1000‐3000 users. SendMail latency is plotted on the primary axis and CPU Load and RPC latency is plotted on the secondary axis. The send mail latency is observed to increase from 64 ms for 1000 users to 94 ms for 3000 users for both physical and virtual Exchange deployments. There is practically no difference in user perceived latencies for either deployment. There is an increase in CPU load for the virtualized Exchange deployment and this is an artifact of both virtualization overhead and running additional Exchange roles on the server. The increase however is minimal, does not affect end user performance as observed from the SendMail latencies and leaves adequate CPU headroom to handle any spikes in the workload.

Figure 5 ‐ Comparing Performance of Virtualized Building Block to Physical Exchange Deployment

Conclusion This study provides an overview of best practices and using a building block approach to deploy Microsoft Exchange Server 2007 on Dell Servers running VMware ESX. Using the Exchange 2007 solution building block proposed in this paper, IT administrators can design, configure and deploy Exchange Server 2007 using Dell servers, EqualLogic storage and VMware ESX virtualization software.

References [1] Planning Processor Configurations, Microsoft TechNet http://technet.microsoft.com/en‐us/library/aa998874.aspx

[2] Exchange 2007 Mailbox Server Role Storage Requirements Calculator http://msexchangeteam.com/files/12/attachments/entry438481.aspx


Page 10

[3] Dell Exchange Server 2007 Solutions http://www.dell.com/exchange

Appendix A

Figure 6 ‐ Network topology for Virtualized Building Block


Page 11

Figure 7 ‐ Storage Configuration Setup and Details

microsoft exchange® server 2007 on systems vmware® esx · by deploying exchange 2007 server on...

Documents