Best Practices for Testing in Virtual Environments

Martin Klaus

Agenda

Virtualization Basics

Why is Virtualization such a big trend?

Taking Advantage of Virtual Labs for Testing and QA

Architecting Virtual Labs

Best Practices for Testing in Virtual Labs

Virtual Labs at VMware

Performance Testing Considerations

Summary and Next Steps

Virtualization Basics

What is Virtualization?

Virtual MachinePhysical Machine

Virtualization packages hardware + OS + applications into a portable virtual machine package

Key Properties of Virtual MachinesPartitioning Run multiple operating systems on one physical machine Divide system resources between virtual machines

Isolation Fault and security isolation at the hardware level Advanced resource controls preserve performance

Encapsulation Entire state of the virtual machine can be saved to files Move and copy virtual machines as easily as moving and

copying files

Hardware-Independence Provision or migrate any virtual machine to

any similar or different physical server

Run several operating systems on a single machine.

Physical – Hosted – Server-based Virtualization

Create shared pools of resources to optimize your infrastructure.

Why is Virtualization such a big trend?

CapacityOn Demand

Hypervisor

Management

VirtualInfrastructure

AutomateSeparate Consolidate Aggregate

Self-Managing Datacenter

Server Consolidation

Test and Development

Computing Clouds

Hypervisor Hypervisor

Management

Hypervisor

Management

VirtualInfrastructure

Automation

Liberate

CapEx Savings

OpEx Savings

Business Agility

Platform for Business Critical Applications

CapacityOn Demand

Self-Managing Datacenter

Server Consolidation

Test and Development

Computing Clouds

Business-Critical Application Momentum

>100,000 companies worldwide virtualize their datacenter

54% deploy enterprise applications on virtual infrastructure

46% standardizing on virtual infrastructure

63% plan to virtualize more than 50% of their x86 infrastructure in 3 years

“In a recent ESG survey, nearly half of the current users consider themselves to be running mission critical applications on virtual machines.”ESG Research, December 2007

Virtualizing critical applications in production

92% of our customers are confident that their mission-critical apps run well on VI

Source: VMware customer survey, October 2007, sample size 500

Percent customers running these applications in production

MSFT Exchange 36%

MSFT SharePoint 54%

MSFT SQL Server 62%

ORCL Application Server 31%

ORCL Database 32%

IBM Domino 37%

IBM DB2 20%

IBM WebSphere 44%

All Databases 42%

Taking Advantage of Virtual Labs for Testing and QA

Common Test Lab Challenges

Inefficient provisioning of test environments

Setup, teardown, reset and configuration time

Process, time and budget dependency on IT

Customer Example: Average Machine Utilization

Maintenance Tasks – Non-test related tasks such as Imaging, Installing, Cleanup, or Locked for Triage.

Testing Tasks – Test execution

0%

10%

20%

30%

40%

50%

60%

70%

80%

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

Maintenance Tasks Testing Tasks

On Average, test execution is only 10% of the total

machine utilization

Image Machine Install Software Run Tests Await Triage Release for new test

20 Minutes 5 Minutes 20-120 Minutes 1-12 Hours!!

Common Test Lab Challenges

Inefficient provisioning of test environments

Setup, teardown, reset and configuration time

Process, time and budget dependency on IT

Constraints of physical hardware

Test matrix limited by available lab resources

Little asset sharing leads to server sprawl

Customer Example: Physical Test Lab Challenges

Resources locked until test run complete

Every task must go through imaging and product install

No reuse of common configurations

Remote access issues

Requires hardware specific Images

Consumes lots of power and space

Common Test Lab Challenges

Inefficient provisioning of test environments

Setup, teardown, reset and configuration time

Process, time and budget dependency on IT

Resource requirements

Test matrix limited by available equipment

Little asset sharing leads to server sprawl

Collaboration of development and testing teams

Reproducing environments to troubleshoot problems

Customer Example: Before Virtual Test Lab

“Before” Virtual Test Lab

Number of provisioning requests and slow IT response time (days)

High temperatures and power consumption of 120 server spread across multiple labs

Significant number of machines reaching the end of their life cycle causing unexpected downtime

Time and complexity required to manage and maintain system configurations

Inability to provide service for remote users without manual intervention

Customer Example: Virtual Test Lab

“After” Virtualizing theTest Lab

7 Blade Servers running 20-50 VMs each

Supports 120 concurrent lab users in multiple locations

Increased lab capacity by an additional 200 virtual machines

Improved IT response time from days to minutes with a self-service portal

Fewer machines and dramatically reduced hardware replacement cost

Customer Example: Test Automation

Infrastructure developed over 6 years

Roughly 100 unique users per day and over 300 per month

Support team of 8 full-time engineers

Mostly automated GUI testing but also includes programmatic SDK + API tests

Averaging roughly 86% increase in testing year

over year

Annual Release Cycle – Launch in Fall

010000

20000300004000050000

600007000080000

Feb Mar Apr May Jun Jul Aug Sep

Test Executions by Release

06 Release 07 Release 08 Release

Architecting Virtual Labs

How Virtual Lab Infrastructure Works

Virtual Lab Portal Image Storage Library

Virtual InfrastructureVirtualized Server Pool

Automated Virtual Lab

LAN/SAN

QA Engineer

User selects a multi-machine configuration, clicks deploy.

Virtual machines stored in the imagelibrary are deployed on the virtualizedserver pool

Once deployed, user directly interacts with the machines, as if sitting at each console

Elapsed Time: 1-2 minutes

Virtualized Server Pool Sizing

Infrastructure Requirements

Server-based virtualization typically achieve consolidation ratios of 10:1 or higher

QA and test labs often achieve 25:1 or more

Use 4-way servers for higher VM density

Basic sizing formula:

Users * Machines Consolidation Ratio

Number of VirtualizedServers needed =

Storage Sizing

Image Library

Efficient use of storage through linked clones and delta disks

Recommendations

Storage need depends on the number of base images + length of linked clones chain

Allocate several large VMFS data stores (300GB - 500GB each) for initial VM storage

Minimum 200GB disk space for media storage

Storage Server

A B C

A’ B’ C’

A’’ B’’ C’’

A’’’ B’’’ C’’’A’’’ B’’’ C’’’

WinXP base disk

Win2k3base disk

Test Automation in Virtual Labs

Use Test Automation Tools

Most vendors have plug-ins and integrations with virtual lab automation products

Build your own automation scripts specific to your environment and processes, e.g. automated regression testing after check-in

Capture Test Environments

Snapshot virtual machines if tests fail – linked clones reduce storage overhead

Discard virtual machines if tests succeed

Best Practices for Testing in Virtual Labs

Problem: Test Matrix Management

Test matrix is becoming too difficult to manage

Host operating system, browser, database, application server, localization, JRE, etc

Trade-off between test matrix, release schedules and quality must be made

Solution: Using Templates and Configurations

Create multi-level templates

Level 1: OS

Level 2: OS + Patches

Level 3: OS + Patches + Apps

Tests are downloaded to Level 3 configurations

Build VM Configurations from Templates

Preserve “Gold master” images

Instantiate templates in multi-tier configurations

Image library stores all test environment configurations

Selectively deploy, combine or split VM configurations for combinatorial use cases

Problem: Test Resource Collisions

Different types of testing cannot be executed simultaneously in same environment

User Acceptance Testing

Regression and Integration Testing

Installation and Upgrade Testing

Test Environment

Solution: Clones and Network Fencing

Deploy multiple identical environments simultaneously

Linked clones and delta disks reduce storage costs

Regression and Integration Testing

Installation and Upgrade Testing

Configuration AOriginal

Configuration CClone of A

Configuration BClone of A

ESX Server Pool

User Acceptance Testing

Problem: Modified Test Environments

Environment Impact Tests

Installation, Upgrade and Patch testing

Test Data Management

Maintain test data through data destructive test cycles

Snapshots and Revert Points

Capture to Library and Checkout of Library

Solution: Store Application State

Pristine Data State

Test Execution

Unknown Data State

Revert to Snapshot orCheckout Library Configuration

Problem: Not-Reproducible Defects

Non-determinism is source of many bugs

Race conditions in multithreaded code

Application behavior dependent on user input, network, etc

Non-reproducible defects get dismissed and shipped

Post-mortem debugging

Applications don’t always crash

Post mortem represents a system snapshot at the time of the crash but doesn’t capture behavior leading up to the crash

Solution: Record and Replay

Record and replay whole VM executioni.e., all threads, all processes, and operating system

Instruction exact

Recorded session encapsulated in a file

User

Network

Disk

Record and Replay Benefits

Instruction exact recording of VM executionInexpensive enough to use pervasively (won’t miss bug)

Use “go-live” to evaluate what-if scenarios

Replay bug over and over

Records all asynchronous data (user input, network, etc)Minimal execution dilation (won’t mask bug)

Memory leaks

Race conditions

Minimal probing effects during debuggingRecord bug with low overhead

Attach debugger while replaying

Control/inspect program in usual ways: breakpoints, watch points, single stepping, etc.

Virtual Labs at VMware

How VMware QA uses virtualization

Virtual Development Environments

VDE contains IDE, compilers, build tools and system paths

Switch between VDE to get different version build tools

Quickly provisioning VDE for new hires and contractors

Virtual Test Environments

VTE contains deployment environment and application build

Manual and Automated Testing team access VTEs

We also use test and build automation tools

Development and QA Collaboration

Code check-ins trigger automated smoke testing

Continuous integration testing against main branch

Virtual Lab Automation in Development and Test

Build System

SQA System

Lab Manager

QA/ Dev

1. Daily build produced by build System

2. VMware Lab Manager SOAP API used to checkout reference platforms, install build, and download and run latest smoke test scripts

3. Bugs are sent to dev team and filed in bug tracking system using LiveLinks

4. All results are checked into the Lab Manager Library. Leases used to keep library clean.

5. Fixes checked into build system for next day’s build

Tests

Fixes

Performance Testing Considerations

Performance Expectations for Virtual Infrastructure

Expect CPU overhead to be in the order of 5-20%

This is what we see consistently in partner benchmarks

Expect Storage latency overhead to be negligible

ESX Layer adds about 0.2ms per I/O, and is capable of

>100,000iops

NFS and iSCSI protocols often provide sufficient performance, but

can have slightly higher CPU overhead

VMware ESX Scales Out Well

Adding VMs scales well: over 95% scaling factor

Single VM, CPU Intensive Performance

VMware ESX 3.x compared to Native SPECcpu results covered by O.Agesen and K.Adams Paper Websphere results published jointly by IBM/VMware SPECjbb results from recent internal measurements

Measuring the Performance of DB Virtualization

ThroughputDelivered

Minimal Overheads

Throughput scales with the size of the VM

Number of virtual CPUs in Database/Guest

Java Virtual Machines

Java on VMwareJava on Linux

Tot

al S

yste

m T

hrou

ghpu

t (o

ps/s

)

Tot

al S

yste

m T

hrou

ghpu

t (o

ps/s

)

31 2 4 5 6 7 8

Java Virtual Machines

• 3-Tier client-server Java benchmark measuring num. of business transactions per sec.• Intel Xeon 3.2 GHz, 2 GB RAM, VMware ESX 3.0, BEA LiquidVM 1.1, BEA JRockit R27.3, BEA WLS 9.2 MP2, RHEL 4.0.• Each VM allocated 1 vCPU and 1 GB vMem. JVM -Xmx=800MB, 135 MB live data.

31 2 4 5 6 7 8

Application Tier Scales Better in Virtual Machines

Summary and Next Steps

Building your own Virtual Test Lab

Implement proof of concept prior to broader deployment

Get your IT department involved

Virtualize one or two QA test lab servers to gain experience

Scale out your virtual lab over time

Online Resources

VMware Web site: http://www.vmware.com/solutions/vla/

White papers, Technical articles, community forums, blogs, customer references

Free Tools

VMware Converter, VMware ESXi, VMware Server

Conclusion

Virtualize your test lab

Start by leveraging existing servers and local storage

Import physical test configurations as virtual machines

Use test automation tools

Included automated provisioning and capture of test environments in your tests

From 2007 Customer Survey

Time is the biggest item “saved” through virtual lab automation

50% increase in defects found prior to production/shipment