designing a bulletproof exchange 2007 architecture j. peter bruzzese co-founder of cliptraining...

Designing a BulletproofExchange 2007 Architecture

J. Peter BruzzeseCo-Founder of ClipTrainingMCSE/MCT/MCITP: Messaging for Exchange 2007

Who is J. Peter Bruzzese?

• MCSE, MCT, MCITP: Messaging 2K7

The Purpose of this Discussion

• What does ‘designing a bulletproof architecture’ mean?– Includes best practices– Understanding features– Security– Centralization and Consolidation– Virtualization

• ESX vs Hyper-V

Agenda for this Discussion

• Why Exchange 2007 Is Better Architecture• Active Directory Preparation• Evaluate and Plan Your Server Deployment• Managed Content Settings, Journal and Transport

Rules• Reviewing High Availability Options• Reviewing Disaster Recovery Solutions• Planning for Unified Messaging• Security Concerns• Scalability for Mission Critical Exchange

Pre-Exchange 2007 - Very Limited

Primary limitations:– I/O footprint

– Non Paged Pool Memory

– Database size

– Disaster Recovery Scenarios

– 32-Bit Architecture is the limiting factor

Exchange 2007 Dramatically Increases Opportunities

Primary Influences:– 64-Bit Architecture

– More available memory

– Less I/O footprint through redesign and architectural options with memory and storage

– Revised Exchange Service Architecture

– Built-in D/R options for easier management and less complex infrastructures

Active Directory Preparation

• Behind the scenes: Schema and Configuration– Method:

Install Exchange and it happens automaticallyRun switches like /PrepareAD to manually handle the preparations

• Physically: Remove any preconfigured site links if possible. Let the Knowledge Consistency Checker (KCC) handle the creation of your replication topology.

What are Server Roles?

• Common practice to deploy servers in dedicated roles on Exchange Server 2000/2003– Installs all code– Larger footprint, unnecessary services & features

installed, less secure

• Exchange Server 2007 formally defines server roles– Installs only required code, smaller footprint, more

secure and management interfaces change based on server role

Server Roles

• One server can have more than one role installed • Can’t co-exist: Cluster Mailbox and Edge roles• Required roles in an Org: Mailbox, CAS, Hub Transport

– Single server deployments: install all three required roles

• Optional: Edge Transport, Unified Messaging

EdgeEdge

Hub Transport ServerHub Transport Server

Client Access ServerClient Access Server

MailboxMailbox

Unified MessagingUnified Messaging

Roles: Mailbox Server

• Hosts user mailboxes and public folders

• Provides MAPI access to Outlook clients– Outlook MAPI clients DO NOT connect to CAS server

• Co-exists with Hub Transport, Client Access Server, and Unified Messaging roles

• Clustered Mailbox Server does not co-exist with any other role

MAILBOX SERVERMAILBOX SERVER

Roles: Client Access Server (CAS)

• Equivalent of 2003/2000 Front-End servers• Provides clients access using OWA, Exchange

ActiveSync, Outlook Anywhere, and POP3/IMAP4• Distributes Offline Address Book (OAB) provides

Availability services and AutoDiscover connection info for Outlook 2007 clients

CLIENT ACCESS SERVERCLIENT ACCESS SERVER

MAILBOX SERVER

MAILBOX SERVER

OWA / IMAP4 /

POP3

OWA / IMAP4 /

POP3

OUTLOOK ANYWHEREOUTLOOK

ANYWHERE

ACTIVESYNCACTIVESYNC

Roles: Hub Transport Server

• Routes mail within Exchange Organization to/from Mailbox servers, other Hub Transport servers, and to Edge Transport servers / smarthosts

• Can be configured to route external mail outside Org

– Edge Transport server not a *requirement*

• Uses Site and Site Link info in AD to route internal messages

MAILBOX SERVER

MAILBOX SERVER

HUB TRANSPORT

HUB TRANSPORT

HUB TRANSPORT

HUB TRANSPORT

Roles: Edge Transport Server

• Managed SMTP Gateway• Typically sits in perimeter networks• Not member of AD• Routes mail in/out of Exchange Organization• Applies messaging hygiene (anti-spam/anti-virus)

filtering agents and organizational policies

EdgeEdge

HUB TRANSPORT SERVER

HUB TRANSPORT SERVER

Roles: Unified Messaging

• Concept: Universal Inbox – email, voicemail, fax

• Outlook Voice Access– Access mailbox,

address book, calendar over the phone

• AutoAttendant

Managed Content Settings

• Managed Content Settings are applied to content in a particular folder or entire mailbox

• Messages can be expired based on when they’re delivered to the mailbox or when moved to a particular folder.

• Specifies Retention Settings (expire messages, take action) and Journaling actions for that content

Message Journaling

• Requirements: legal compliance• Journaling happens at Transport• Granular: per mailbox (previous versions = Store-

based)• Standard Journaling: per mailbox Store, per server• Per-recipient or distribution list journaling: all

messages to and from recipients and senders on a journaling-enabled mailbox

• Premium Journaling: rules-based, available in Enterprise Edition Only– Scope: internal/external/global

Transport Rules

• Apply messaging policies• Transport Rule Agent runs on Hub Transport servers• Edge Rules Agent runs on Edge Transport servers• Together they provide a mechanism to apply policy-based

rules to all messages– Inappropriate content

– Confidential or sensitive information

– Ethical Walls /Conflict of interest situations (e.g. brokers & analysts)

– Redirecting messages

– Applying disclaimers

High Availability Options

• Local Continuous Replication (LCR)

• Cluster Continuous Replication (CCR)

• Single Copy Cluster (SCC)

• With SP1– Standby Continuous Replication (SCR)– Windows 2008 Support

Other DR Features

• Database Portability: Store from one server can be mounted on another server– Only restriction: Store needs to be from a

server in the same Exchange Organization– After mounting Store on another server,

modify user account settings:move-mailbox –configurationonly

– AutoDiscover automatically redirects Outlook 2007 clients

DR Features (cont.)

• Recovery Storage Group– Can be created and used using shell– Not visible in console– Recover Stores from Exchange Server 2007,

Exchange Server 2003 SP1 or later, Exchange 2000 SP3 or later

• Supports restores from VSS backups

Traditional Approach: Multi-Site Disaster Recovery

One Way Data & Service

Replication to DR Site

Double The Servers (High $$$)

(10) Infrastructure Servers

(10) Mailbox Servers (10) Mailbox Servers

(10) Infrastructure Servers

PRODUCTION SITE DR SITE

* Source: Unisys* Source: Unisys

Planning for Unified Messaging

• Server? We encourage 4x Processor Cores and at least 2GB of RAM (show you why in next slide)

• What about your legacy PBX?– Consult the Telephony Advisor from Microsoft for

supported VoIP Gateways, PBXs and IP-PBXs

• If you have a legacy PBX… try a VoIP Gateway the PBX is functional.

• If you are starting fresh… go with an IP-PBX

UM Metrics with 1/2/4 Cores

The View from Above

Security Concerns

• Permissions and Roles within Exchange

• Using Transport Rules

• Authentication options

• Anti-spam (for the Edge and Hub Transport servers)

• Anti-virus

• Hosted Solutions

• Microsoft Forefront

Bulletproof Design Thinking

• Centralize

• Consolidate

• Virtualize

Centralize

• Exchange Servers Only Deployed in Mission Critical Locations

• Fewer or No Remote Site Servers

• Increased Control and Security


Consolidate

• Fewer Servers

• Less Attack Surface

• Better Resource Usage

• Higher User Density

• Reduced Cost per User

• Green ITUnderutilized Servers

Higher Server Utilization


•Virtualized Infrastructure Server

• CAS, HUB, GC

•Virtualized Infrastructure Server

•EDGE, ISA

•Virtualized MBX Servers

Virtualize

• Fewest Physical Servers

• Least Attack Surface

• Optimized Resource Usage

• Highest User Density

• Lowest Cost per User

• Green IT* Source: Unisys* Source: Unisys

An Example of Bulletproof Design

Exchange Server 2003Exchange Server 2003

Previous Email Environment Previous Email Environment - 30K Users- 30K Users

62 servers …No redundancy…no DR62 servers …No redundancy…no DR

Deployment

Solution also includes:• Disaster recovery (CCR)• Collaboration (SharePoint)

Exchange Server 2007Exchange Server 2007

(2) 24 dual core ES7000s(2) 24 dual core ES7000s

New ApproachNew Approach - - 42K users42K users


ESX vs Hyper-V

• Many people ask which virtualization solution is better.

• Our friends at Unisys had the chance to perform benchmark testing in the Microsoft lab to find out. We thank them for allowing us to show their results here.

• Used LoadGen: simulation tool used to measure the impact of MAPI, OWA, IMAP, POP and SMTP clients on Exchange.


VMware Testing Information

• VMware ESX 3.5

• Microsoft Windows 2008

• Microsoft Exchange 2007

• Microsoft LoadGen

• 8 Virtual Machines

• 24,000 through 56,000 Heavy Users (MAPI)


VMware Test Information

• 17 load generator systems and 1 master to drive the tests

• Heavy Action profile: Outlook 2007 MAPI-Connected• 250 MB mailbox size• Test duration 8 hours• Simulated 8 hour day• Tasks per User per Day = 132• No Distribution Lists• No Contacts• No External Recipients


Hyper-V Testing Information

• Hyper-V

• Microsoft Windows 2008

• Microsoft Exchange 2007

• Microsoft LoadGen

• 4 Virtual Machines

• 12,500 Average Users per VM

• 50,000 Users Total * Source: Unisys* Source: Unisys

VMWare Processor Utilization

* Shows average for all 8 VMs during the steady state (after initial user logons)

Average VM Processor Utilization

0%

20%

40%

60%

80%

24k 32k 40k 48k 56k

Users


VMWare Disk IO

• IOPS/User = 0.16

• IOPS/User remained the same for all tests (24k through 56k users)

• Avg. Disk sec/Read was .006 with 24k users and .008 with 56k users

• Avg. Disk sec/Write was .001 for 24k through 56k users


ESX Host Processor Utilization

ES7000 Model 7405R % Processor Time

0

20

40

60

80

100

120

0 1 2 3 4 5 6 7 8

Hours into the test

24k

32k

40k

48k

56k


Hyper-V Testing Results

MAPI Tests

• Single VM – 15% average CPU utilization on 1 Mailbox VM

• Two VM’s – 28% average CPU utilization on each of 2 Mailbox VM’s

• Three VM’s – 49% average CPU utilization on each of 3 Mailbox VM’s

• Four VM’s – 60% average CPU utilization on each of 4 Mailbox VM’s

• No problems with disk latency – 6 to 8 ms

• No problem with LoadGen task latencies

Hyper-V Cluster Testing

CCR Cluster Tests• Single active / passive VM

– 30% CPU utilization on VM on active node

– 26% CPU Utilization on Passive VM

• Two active / passive VM’s

– 55% CPU utilization per VM on active node

– 32% CPU Utilization on Passive VM

• No problems with disk latency (6 - 8 ms for EDB files)

• No Copy Queues (1 - 2 per SG)


Setup Parameters Hardware– ES7000/one Dual Core with 8 sockets and 48 GB RAM

– Four HBA´s with 2 Gbit connection each

– HP EVA 8000 with 80 spindles for the test

– LoadGen Clients virtualized on HP Servers

Setup Parameters LoadGen– 50 MB initial mailbox size (Storage Contraints)

– 8 hour working day

– No dynamic DL´s

– No external mailflow

Testing Results

Hyper-V @ MTC Munich


Setup Hyper-V–4 logical cores and 20 GB RAM per VM–Pass through discs–One LUN for every 2,000 users

Setup Parameters Exchange 2007–Two Mailbox servers, each configured as

HUB/CAS/MBX–Clean Active Directory setup on VM´s

Testing Results



Testing Results

Test run with 10,000 average users per VM– CPU and RPC Latency spike during logon– System proceeded to normal state after 15 minutes– Average CPU utilization around 18% per VM– Average 8 Messages/Sec, 480/Min, 28,800/Hr

Test run with 10,000 heavy users per VM– CPU and RPC Latency spike during logon– System proceeded to normal state after 15 minutes– Average CPU utilization around 26%– Average 13 Messages/Sec, 780/Min, 46,800/Hr



Testing Results

Test with 10,000 very heavy users per VM

–Average CPU utilization around 35% per VM

– Average 17 Messages/Sec, 1,020/Min, 61,200/Hr

– Average RPC Latency ~ 8 ms

– 5 Megabyte traffic per second on NIC´s



VMWare or Hyper-V?

• Depends:– If you matured into the virtualization space

believing in a certain solution– VMWare has Vmotion (although Hyper-V has

Live Migration coming in Server R2)

• However, I believe the preceding slides show that Performance is not necessarily a factor in the decision.

High Availability and Virtualization

• Microsoft says:– We don’t recommend you use hypervisor-

provided clustering• No Live Migration• No Vmotion

– We DO recommend CCR for high availability

– http://technet.microsoft.com/en-us/library/cc794548.aspx

Summary

• Designing a Bulletproof Exchange Architecture involves the following:– Knowing Best Practices– Understanding Features– Knowing Your Options– Centralizing, Consolidating and Virtualization– Virtualization Saves You A Great Deal…

Virtualization combined with Disaster Recovery may save your company.

Q & A

• Email me @– [email protected]

• Watch my training clips @– www.cliptraining.com– www.youtube.com/cliptraining – www.exclusivelyexchange.com

• Read my blog @– http://weblog.infoworld.com/enterprisewindows/

designing a bulletproof exchange 2007 architecture j. peter bruzzese co-founder of cliptraining...

Documents

server roles

server role slide

role mailbox server

secure exchange server

org edge transport server

content slide

transport rules

edge roles