Lotus Domino consolidation to Linux on System z - Project "Big Green" experience and more

Barbara Sannerud, Elsie Ramos, Mike Wojton

© 2009 IBM Cooperation

IBM System z

2

Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

* Registered trademarks of IBM CorporationThe following are trademarks or registered trademarks of other companies.

* All other products may be trademarks or registered trademarks of their respective companies.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license there from. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.InfiniBand is a trademark and service mark of the InfiniBand Trade Association.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.

The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.DB2*Domino*DS8000FICON*GDPS*Geographically Dispersed Parallel SysplexHiperSockets

IBM*IBM (eServer)IBM (logo)Lotus Notes*MQSeries*System z*System z9*

System z10System z10 Business ClassWebSphere*z9*z10z10 BC

zSeries*z/VM*

Trademarks

3

Leveraging IBM Domino® on IBM System z®

Values of System zDomino on System z and TCO

IBM Domino Migrations to Linux® on System zApplication and Mail Server MigrationMigration Setup and Strategy

Benchmarks and CustomersBenchmark resultsCustomer implementations

Agenda

4

Leveraging IBM Domino on IBM System z

140 million seats of Lotus Notes® sold worldwide

Used by more than 46,000 companies around the world

Lotus Notes and Domino are supported by over 10,000 IBM Business Partners worldwide

FORTUNE Global 500: More than half of the Global 100 corporations use Lotus Notes and Domino software

5

Reliability of System z is Unmatched

Five nines availabilityDesigned for MTBF* of decadesDomino can enjoy reliability of System z

Concurrent operationsError correcting codeParts replacementPhone homeSparingError captureRecovery routinesFully integrated into life cycle

In sum, unique design points to provide near zero down time

Reliable Operations

Error preventionTechnologyDesignTest

Error detectionInstantaneousError domainData capture Recovery

FenceResetFault tolerance

Problem determinationProblem correlationProblem isolation

Corrective MaintenanceMaintenanceService personnelPartsProblem mgt

Change managementHardwareDrivers

Measurements & analysis RAS

ProductFailure analysis

* Mean Time Between Failure

6

General z benefits• Linux can take advantage of System z reliability, scalability, availability• Excellent performance and dependability• Improved failover at low cost• Virtualization

System z with Linux performs well with multiple Domino partitions in a single LPAR

• With Domino partitioning and multi-processors• Domino infrastructure scales well

You can add resources (CPU/Memory/IO) instead of adding servers to grow your environment

• Reduce scale out and associated costs• Reduce labor• Reduce tooling

Balancing of system workloadsIncrease in utilization through virtualization

• DPARs and LPARs are individually managed on System z. • Portable solution given Domino code base

• Reduce networking• Non disruptive upgrade paths

Why Use Domino On System z?

7

One instance of a Domino server is called a Domino partition (DPAR)

You can run multiple DPARs in different LPARs on a single processor

You can run multiple DPARS spread across more than one processor

Each DPAR is independent of other DPARs, with its own address spaces and files

DPARS can easily be moved from one image to another

Use TCP/IP to communicate and transfer data

Domino also makes use of multiple processors with multiple threads and processes • The Domino main server address space has a pool of physical threads for

separate tasks, and multiple tasks execute concurrently

Architecture of Domino Aligns with System z

8

Domino mainframe users get high availability, reliability and scalabilityScale - A single System z system can host many Domino servers on Linux images

System z channels can support a quarter million IO devices Limited number of open files on Intel®Support for thousands of open files per Linux process A System z system can support tens of thousands of Notes users

Run multiple DPARs on a single LPARDPARs can scale to support thousands of usersAdd more DPARs, if needed

Scale IO to support throughput you need, and handle backup demandsScale with Capacity on Demand - low cost option

System z Domino users can upgrade as needed when capacity limits are reachedScale out quickly with Linux IFLs*

Linux guests can be added quickly giving users a highly flexible environment Resources can be shared among multiple Linux images - processor, memory, storageAsynchronous network I/O

* Integrated Facility for Linux

Scaling Helps Address Growth

9

Domino Version 8.5 is native 64-bit version for Linux on System z

Domino enjoys the Quality of Service on System z

We have seen limitations running Domino on x86• Must be rebooted to recover crashed Domino servers

• Doesn't grow vertically, grows horizontally

• There are memory management issues

• I/O rates for backup can be an issue on Intel

• Reliability, availability and scalability on System z is superior

• Higher people costs in distributed environments

Domino On System z – A Better Choice

10

System z, application and mail server segregation provides the flexibility needed to prioritize Domino workload characteristics.

• Domino has variable workloads—can be I/O and processor heavy• There is no one slow period• System must be able to handle multiple peaks in workload• Virtualized mainframe environment ideal

General Virtualization benefits• You can bring down, recycle, or recover a single DPAR on System z and not affect

another Domino server – Dynamically adjust LPARs without stopping Domino Servers– Run multiple DPARs in a single LPAR

VM support for scheduling, automation, performance monitoring

Applications might need different resource prioritization schemes. • System z LPARs, and VM guests can be managed to direct system resources where

needed

Intrinsic System z Virtualization Values

11

Security of z platform • Common Criteria, virus resistant, protected address spaces, etc.

Logging• System z offers comprehensive logging and auditing• Lotus Domino ID Vault to automatically recover IDs, reset passwords

Support for SSL hardware-based encryption• Domino Web browser clients can use hardware based encryption

Use of VLANS• A VLAN allows a physical network to be divided administratively into

separate logical networks which operate independently of each other.Enterprise disaster recovery• System z Geographically Dispersed Parallel Sysplex™ GDPS®-based

backup, restore, and disaster recovery tools and processes • Flexible HA solution can be delivered without GDPS

Logging, Security are Other Advantages

12

Domino supports clustering and failover across different hardware, and different operating systems Multiple database replicas are created on Domino servers

• Databases changes are synchronized across replicas Domino clustered servers can be deployed on the same mainframe using different LPARs or Linux guests This offers more flexibility when scheduling system maintenance HiperSockets™ or VLAN communication can be used on System z

Active/Passive Cluster – Two LPARs Configuration on One Machine running Linux

ACTIVE LPAR

z/VM®

This configuration uses 4 DPARS, 2 active and 2 passive. In the event of a failure of the active DPARs, the passive DPARS take over.

PASSIVE LPARz/VM

DPAR 1 DPAR 2 DPAR 3 DPAR 4System z

Failover

Failover

Virtualimages

Domino Clustering for High Availability

13

Configuration used for sizing

Microsoft configuration came from Microsoft

We did not use Sharepoint or OCS (22-8 servers)

For a complete on premise collaboration solution supporting 12,000 users Microsoft® suggests using 22 Wintel Servers

Microsoft Configuration

14

TCO: 3 Years Per User Cost

Microsoft Exchange on fourteen x86 Servers $ 12,557,473 $ 1,046

Domino on one z10 BC™ with 6 IFLs $ 4,286,997 $ 357

Savings with Domino on System z Linux $ 8,270,476 $ 689Prices are in USD. Prices may vary in other countries.Assumes 12,000 users

Email, Calendaring, and Collaborative Application on System z is 1/3 the Cost of x86 and Saves $8M+ over 3 years

0

2.000.000

4.000.000

6.000.000

8.000.000

10.000.000

12.000.000

14.000.000

Microsoft Exchange® onfourteen x86 Servers

Domino on one z10™ with 6IFLs

Labor

Software Service &Support

Software

Hardware Support

Hardware Cost

Incremental FloorSpace & Equipment

Power & Cooling

15

Summary of Benefits:The per-user cost is 1/3 of x86 MS Exchange solution due to significant savings on labor, power and cooling, floor space, HW /SW support

Microsoft Exchange on fourteen x86 servers

Domino on one z10 BC with 6 IFLs

Hardware cost $ 267,000 $ 339,000

Software

$ 6,067,986Office (includes Outlook),

Exchange 2007, CAL Enterprise

$1,931,790 Domino for Linux on System z:

z/VM, Linux, Domino, Lotus Notes

Hardware Support $ 93,450 $ 201,312

Software Service & Support $ 5,125,551 $ 1,364,601

Power and Cooling $ 40,366 $ 294

Labor $ 900,000 $ 450,000Incremental floor space & equipment $ 63,120 $ 0

TCO: 3 years $12,557,473 $ 4,286,997

Per user cost $ 1,046 $ 357

Email, Calendaring, and Collaborative Application on System z is 1/3 the Cost of x86 (Details for Previous Chart)

Prices are in USD. Prices may vary in other countries.

16

Domino on Linux on System z

Many application consolidation opportunities on zLinuxIFLs provide significant cost incentives for customersAvoid server sprawl and its cost effects Reduce management and tooling as well as storage Run at under a Watt per MIPS - less cooling, less floor space impact

We decided to do this for ourselves…

17

IBM Domino Migrations to Linux on System z

Dynamic InfrastructureImprove service. Reduce cost. Manage risk.

IBM will consolidate and virtualize thousands of servers onto approximately 30 IBM System z mainframesSubstantial savings expected in multiple dimensions: energy, software and system support costs Major proof point of IBM’s ‘Project Big Green’ initiativeThe virtualized environment will use 80% less energy and 85% less floor spaceThis transformation is enabled by the System z sophisticated virtualization capability

* Results will vary based on several factors including # of servers and work load types ** IBM Global Asset Recovery Services for reuse, recycling and/or reclamation

By leveraging new IBM System z10™ . . .Number of machines could be cut by about halfEven greater savings in energy, floor space, software and support costs

Linux on System z is matching the attributes of a dynamic infrastructure - exploiting the outstanding virtualization, automation, availability and security capabilities of the System z

18

Overview IBM Domino Deployment

Domino infrastructure moving to Linux on System z

Part of Project Big Green

Application servers

Mail servers

Infrastructure servers

Designed to save• Space

• Energy

• Money

19

120+ Domino Application servers (DPARs) migrated Over 37,000 applications migrated

• Critical business applications – i.e. payroll, Executive applications

Migrated Domino application servers with additional IBM middleware –Lotus Enterprise Integrator, WebSphere® MQSeries® and DB2®

Configuration • 7 Shared z/VM LPARs on IBM System z9® Enterprise Class(z9® EC)

• 32 Linux Guests

• 30 TB of Storage (SAN, FICON®)

Application Server Migrations to Linux on System z

20

Mail Server Migrations to Linux on System z

Target 60K users in North America to Linux on System z in 2009• 4 Mail Cluster (failover) Servers migrated

– 5300+ users

• Ongoing Primary Mail Servers migrations

– 4 migrated with 10.000+ users

• Configuration for servers migrated

– 3 VM LPAR on System z9 EC

– 4 Linux Guests

– SLES10

– 8.5TB+ storage

Japan mail servers migrated to Linux on System z

EMEA to begin migrations soon

21

Migration Setup

Linux Build−

Cloning of new Linux guests

−

SLES 10

Lotus Domino Server Build−

UNIX® environment setup (security,.profile, crontab, id setup etc.)

−

Lotus Domino software install, templates

−

Backups install

−

Scanmail

−

Domino 7x (application servers)

−

Domino 8.02x (mail servers)

22

Migration Strategy

Server Migration• Move entire server “as is” to Linux on System z

– Server already “consolidated” prior to migration

• Server name remains the same

• Migration transparent to the end users

– Only change is IP address

• Migrate application servers using ftp

• Migrate mail servers using replication

23

APPSRV1

APPSRV2

APPSRV3

APPHUB1

APPSRV4

APPSRV5

APPSRV6

APPSRV7

MailServer1

MailServer2

Linux Guest 1 Linux Guest 2 Linux Guest 1

LPAR1 LPAR2

Sample Domino Server Configuration

24

Benchmarks and Customers

25

102K NRPC R6 Mail Benchmark

1 Linux kernel – SLES 10 under z/VM 5.3

6 DPARs running 8.5• Comparison to R7 50K benchmark – 8 DPARs

IBM System z10 Enterprise Class• Domino LPAR - 12 IFLs, 48 GB

– Multiple benchmarks, SysPlexs running on the same box

• DS8000™ – 3 LPARs – 27K IOs per second

• DPAR Network at memory speeds

Cost per user started to degrade over 80K users

Around 1.5 Million Domino transactions every 10 minutes

1000000

1200000

1400000

1600000

Total Domino Transaction10 Minute Intervals

Transactions

Domino Linux for System z Scalability

85000

90000

95000

100000

105000

Connected and Active 15 Minute Users10 Minute Intervals

Connected Active 15

26

Domino for Linux on System z Scalability

Running with z/VM is a recommended environment for Linux since Domino v6.5

• 10% overhead for first OS guest (not DPAR), 1-2% for each guest after that

Scale your Infrastructure Up! Not out!• Lower cost running larger DPARs

– 50K users in 6 DPARs was almost 20% more costly than 50K in 3 DPARs

• Less server instances– Less images to administer and manage with allowing upgrades

• What happens today in a distributed environment if a server is out of resources?

– Purchase new hardware, install new OS, install new Domino server, migrate workload off of original server

– With System z, upgrade hardware (if needed), update LPAR/guest.– Let your infrastructure fit your messaging environment, instead of

fitting your messaging environment into your infrastructure

51 K Users

00.010.02

0.030.040.05

Processor Seconds per active 15 minute User

6 DPARs 3 DPARs

27

IT Services provider customer example

IT Services provider, delivers a high-level of quality of service to its customers at competitive prices. The high-availability Domino solution, a Domino cluster on 2 System z9 EC, fully addressed this business need.

• IBM z/VM V5.2• Novell SUSE Linux Enterprise Server V9 and V8 • Lotus Domino Enterprise Server V7.0.3 • 4,500 registered users• Several Domino Enterprise Server V6.5 on SLES 8 guests in the same z/VM

for collaborative and archive of Lotus Domino data.Customer was running out of capacity due to growth

• By upgrading from old IBM eServer™ zSeries® 990 (z990) to System z9, customer gained extra capacity for the same IFL number.

28

Migrated to System z resulted in less servers than previous platform• Less Server to upgrade• Backups complete in overnight window• Multiple systems consolidated to 2 Linux for System z guest each running 4 DPARS

– DPARs are a mixed of mail, administration, name servers

Running older 32-bit code• Supporting around 8TB of data on each Linux guest• Just under 14K production users split between the two Linux guest

Customer was running out of capacity due to growth• Solution, they added in an additional IFL to their z/VM. Reconfigure the Linux guest

for the new capacity. DONE! No need to acquire new hardware for a new server, build a new OS and Domino server, then migrate users/workload off of existing box with the constraint.

• When the Customer upgraded their existing z9 to a z10, they were able to turn off (and stop paying for) the additional IFL. Again, no changes were needed in the Domino Administration area to support this virtual reconfiguration.

HealthCare Customer Example

29

IndustryBanking/Financial RetailTransportationMedicalGovernmentInsuranceIndustryEnergyIT Service providersHealthCare

GEOsAmericasEMEAAP

Production/Pilots/Proof of ConceptsCustomers who have deployed Domino on Linux for System z or who are looking to start a pilot:

30

Best of Breed• Hardware availability – While others talk about as being as good as the mainframe,

System z is the mainframe that sets the bar

• Virtualization and lowest overhead cost

Most Robust IO infrastructure of any Domino Platform• System z can support/backup significantly more I/O than any other Domino platform

Best Power/Cooling performance of any Domino Platform• Just stand behind System z and others• Heat generated is power consumed

Largest Scalability of any Domino Platform• Consolidate more DPARS/LPARs of any Domino Platform onto a single footprint

Reason Why Domino on Linux on System z makes Sense

31

Lower TCO costLess DPARs/LPARS to manage is less overhead for Administration and Support

System z allows your infrastructure to support Domino, not have Domino fit into your infrastructure

Dynamic UpgradesCapacity on Demand

Dynamic Upgrades

Domino benefits from Vertical scalabilityNot only are does less servers reduce your TCO overhead, it will reduce the actual cost of running your Domino environment.

Scaling vertically first takes less processor cycles than scaling horizontally

Customers and IBM is doing it todayIBM is moving Domino workloads to System z because of a better TCOCustomers are looking at System z for improved TCO benefits

Reason Why Domino on Linux on System z makes Sense

32

Linux on System z provides a great consolidation and simplification opportunity for Lotus Domino workload

Get Started and contact your IBM or Business Partner representative for a sizing.

33

Presenters & Contacts

Web pages• Linux on System z• Lotus Domino on Linux on System z• Lotus Domino and Lotus Notes

White papers• Extend the Value of the Mainframe for Collaboration• Consolidation of Lotus Domino and Lotus Notes to Linux on System z

Articles• IBM Lotus Domino, Linux, virtualization, scalability: No longer mutually exclusive terms

(Benchmarks and results of early adopters)

• Virtualized System z Brings Green Computing

Barbara Sannerud (sannerud@us.ibm.com) System z Competitive project Office

Elsie Ramos (ramos@us.ibm.com) Project Manager for ECM

Mike Wojton (mwojton@us.ibm.com) ATS for Domino on System z in Americas

Armelle Creuzet (armelle_creuzet@fr.ibm.com) ATS for Domino on System z in Europe

Contacts & Resources

ZSP03175-USEN-00