ASF CloudStack – QA StrategyApache Cloudstack Collaboration ConferenceDec 1, 2012

Sudha R Ponnaganti, Sr Manager, SQA Engineering, Citrix Systems Inc.,

Email:sudha.ponnaganti@citrix.com

QA Strategy - Introduction

Need for QA strategy

Goals for current presentation Transparency

Expected outcome Community participation

Superior Quality Releases

Expected release cycles

Agenda

Releases and QA Overview

QA strategy

Future

Community participation

Q & A and Feedback

Releases and QA Overview

Releases and QA overview

QA Activity Major Releases ASF CS Minor Releases ASF CS

Installation and Upgrade validation

New feature validation

Regression validation Automated

Integration validation

Performance and scalability testing

Solution validation

Security review

Recovery validation

Usability validation

L10N validation

Defect validation

Documentation review RN only

Continuous Release Cycles

Currently we are doing minor and major releases Cycles are longer

Time based vs. feature based release discussion

There is a need for

Continuous integration

Continuous validation

Continuous Integration

Every check-in gets tested as and when it is submitted

Builds upfront quality faster

Should take less than 5 minutes to complete

Below is the process that is being worked up on currently

Jenkins kicks off a job after each

checkin

That job spins up a devcloud appliance

devcloud checks out the commit that kicked off

the job;

devcloud builds cloudstack

devcloud adds itself as a host

devcloud runs the smoke test

Continuous Validation

Run all the tests on nightly builds

With current structure, below are the sequence of tests that run nightly

RAT ( release

audit tool)

Compile + junit

Docs + apidocs

Package building

Install tests Smoke testsRegression

Tests

QA Strategy

Installation and Upgrade Validation Compatibility Matrix

Manual Validation

Priority Type Type of Dist Base Upgrade V OS Networking Network Devices Storage Guest VMs Browsers Features to Cover

Xen KVM Vmware OVM Bare Metal

P1 Open Source ASF CS 4.1 CentOS 6.3 6.0.2+HF, 6.1 RHEL 6.2 5.1 2.2.2 N/A Advanced/Basic

VR, F5, SRX,

Netscaler VPX ISCSI, NFS, Local

Template: CentOS5.6PV 64bit,

CentOS5.5PV 64bit, IE8, FF 10, Chrome, SafariBVTs

P1 ASF CS 4.1 Ubuntu 12.04 6.0.2+HF, 6.1 RHEL 6.2 5.1 2.2.2 N/A Advanced/Basic

VR, F5, SRX,

Netscaler VPX ISCSI, NFS, Local

Template: OVM System.img

ISO: OVM System.img IE8, FF 10, Chrome, SafariBVTs

Multinode install

P1 Multiple MS ASF CS 4.1 CentOS 6.3 6.0.2+HF, 6.1 RHEL 6.2 5.1 2.2.2 N/A Advanced/Basic

VR, F5, SRX,

Netscaler VPX

(NS10,build 54.7.nc) ISCSI, NFS, Local

Template: CentOS5.6PV 64bit,

CentOS5.5PV 64bit,

Windows2008R2 64bit, IE8, FF 10, Chrome, SafariBVTS

P1 ASF CS 4.0 ASF CS 4.1 CentOS 6.3 6.0.2+HF, 6.1 RHEL 6.2 5.1 2.2.2 N/A Advanced/Basic

VR, F5, SRX,

Netscaler VPX ISCSI, NFS, Local

Template: CentOS5.6PV 64bit,

CentOS5.5PV 64bit, IE8, FF 10, Chrome, SafariBVTs

P1 ASF CS 4.0 ASF CS 4.1 Ubuntu 12.04 6.0.2+HF, 6.1 RHEL 6.2 5.1 2.2.2 N/A Advanced/Basic

VR, F5, SRX,

Netscaler VPX ISCSI, NFS, Local

Template: CentOS5.6PV 64bit,

CentOS5.5PV 64bit, IE8, FF 10, Chrome, SafariBVTs

Compatibility Matrix ASF 4.1- Supported Vs Tested

Upgrade

Fresh

Hypervisors

Installation and Upgrade Validation

Install and Deployment validation Specifics:

Install and deployment Validation need to be done in context of release artifacts to be in alignment with voting process.

Step by step instructions are posted here.

https://cwiki.apache.org/CLOUDSTACK/cloudstack-40-test-procedure.html

New Feature and Regression Validation

New Feature Validation Regression Validation

• QA team writes test cases for new

features based on FS.

• QA posts test plans to community to get

feedback

• QA automates and executes new test

cases for current release. Adds TCs to

regression for future releases

• We can provide template if you need to

add test cases.

• You can find test suites on wiki

• There are around 3000 + test cases in

the regression suite currently.

• This suite includes Automated and

Manual tests

• QA reviews overall feature list and

identifies regression coverage based on

impact for each release – not sustainable

• To be successful in open source env,

need to have 100% automated coverage

New Features and Regression Validation

Various Frameworks are available based on Junit, Python, Selenium to automate test cases

A combination of all of these provide comprehensive coverage

Unit Tests BVT Tests Regression Suite

Other Integration Suites ( EC2 and

S3)UI Automation

New Features and Regression Validation

• Unit Test framework ( JUnit )

• Provides early feedback and fixes issues close to source

• How to Write Unit tests - https://cwiki.apache.org/CLOUDSTACK/unit-

testing-101.html

• Currently all unit tests running with 100% pass rate

• Runs on every build

• Code coverage results are posted here

New feature and Regression Validation BVTs and Functional Automation - Marvin

600+ test cases running on multiple configurations and covering multiple hypervisors

How to write functional automated TC : https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html

BVT

Hypervisor Configuration

Xen ( multiple versions) Advanced, Basic

VM ware Advanced, Basic

KVM Advanced, Basic

OVM Planning

Functional Regression

Hypervisor Configuration

Xen ( multiple versions)Advanced, Advanced - Netscaler, EIP/ELB Rules, Basic with and

without SG

VM ware Advanced, Advanced - Netscaler, EIP/ELB Rules, Basic w/o SG

KVM Advanced, Advanced - Netscaler, EIP/ELB Rules, Basic with and

w/o SG

OVM Planning

New features and Regression Validation

Multiple test pods are being setup to run suites in parallel– around 15 test pods are required

Porting to ASF CS env has been in progress. BVTs are done.

Prasanna and Edison are covering more detailed talk on Automation on Sunday morning.

New Features and Regression Validation Integration frameworks

Currently we are in the process of submitting test suite that covers EC2 Query API validation – JCloud framework

Around 40 APIs are automated In the process of covering all APIs

Failed test cases are being fixed

Need to automate complex scenarios.

Details on test implementation are posted below https://cwiki.apache.org/confluence/display/CLOUDSTACK/EC2+Qu

ery+API+Test+Suite+Automation

New Features and Regression Validation

UI Automation

Selenium

Faster and Easy

Compatible with python backend

Plan to Deliver Smoke Tests by – Early FY 13

POC and initial submits – coming soon

Performance and Scalability Testing

Native Tool - Simulator

Simulator - mock hypervisor and will react to every command sent to it by the MS similar to a regular hypervisor - easy to setup and run

Simulator and Perf resources: http://wiki.cloudstack.org/pages/viewpage.action?pageId=1344953

2 papers have been published Covering Basic Zone

Advanced Zone

Performance Use cases

• Time taken to propagate security group rules in basic zone • Deploy VM time, CPU, load averages and # of DB connections• Steady state CPU, load average and # of DB Connections• Restart MS measures: Time to stop MS, Start MS and rebalance

with 30K Hosts across 4 MS (agent load size to 500 and 1000)• Restart MS measures : with Host in maintenance mode• Bandwidth usage collection frequency for routers• Router Status poll metrics in RVR env

Performance Testing

What Next Holistic Coverage

Router

Storage

Network

Realistic Scale Testing

Security Validation

Security validation has been important aspect of ASF CS releases

All FS should include security considerations so that QA can identify and cover these areas

Security Coverage done included the following along with manual validation and other security reviews

Solution Validation

o Current solution Validation use cases include

o reference architecture deployment and

o additional complex deployment scenarios.

o Test plans are posted on wiki

o There is no need to adopt these right away in to Cloudstack but will monitor if there is a need in future

Recovery, Usability and L10N

Recovery scenarios, Usability Reviews/ L10N - Not prioritized Jessica Wang has been doing L10N review along with few other

contributors If you have expertise in these areas, do participate in QA forums

Documentation review Focused validation at the end of the cycles Watch for QA requests for review

Defect Triage and Validation

You can participate by testing product logging defects here ( ASF JIRA)

Always log defects when you start discussion on mailing lists

Add automation for defects that you validate. Tests can be checked in to appropriate suites based on type of test –unit/smoke/regression suites

Participate in Bug fest activities

Future

Future - Status

Some improvements that QA has been doing Q3/ Q4 FY ‘12

Automation

Functional – BVTs and Regression (Python framework) - goal 700+ TCs

Unit Test Framework ( Junit ) – Add TCs for ongoing releases + improve legacy by adding TC for patches

Integration automation frameworks (ec2 and s3) –implementation and make it part of regression suite ( Jcloud Test framework)

Future - status

• TCM Implementation and loading test cases –Address Test Case gaps ( total inventory) – Currently 3000 test cases exist. Goal is to get to 5000

• Performance White paper – Use cases collected and release V3 with known use cases – backlog has been built.

• Tools Static code analyzer – In progressCode coverage – Cobertura Code inspection -

Future - status Infrastructure improvements

Test Env improvements ( Fresh Install Automation for Functional testing)

Acquire external devices and standardize test env

Build infrastructure improvements

Continuous integration implementation

Continuous validation

Standardized certifications

New versions of HV, NS and other integrations

Standardized templates

Test plan/Test case / Defect writing Guidelines, Test case execution reports, baseline metrics

Future - status

Initiatives for FY ‘13

Move towards enterprise quality standards Unit Test Coverage + Automation – 70% + Code coverage Regression coverage – 90% + coverage New features – 100% coverage Defects regressed through automation Install/Upgrade Automation UI automation - selenium Standardize QA coverage for enterprise readiness

Regression New functionality Automation Install/upgrade ( compatibility matrix automation) Usability Performance ( tools and use case expansion) Reliability ( longevity under load) L10N ( internal or vendor based)

Future - status

508 ( Accessibility)

Integration

Security

Real world use cases

Install Upgrade automation

Infrastructure improvements to run automation across

multiple configurations

Community Participation

Community Participation

Scheduled QA Meetings on weekly basis – Will be publishing meeting details for CS 4.1 and expect it to be starting soon. Everyone is welcomed to join and participate in QA Activities for ASF CS 4.1 release

David has planned IRC Sessions to drive Quality as a continuous process. Please do participate and help to drive this continuous test process - timings 1800 UTC in #cloudstack-dev on Wednesdays

ASF Cloudstack Dev mailing list is great way to share your ideas and contributions cloudstack-dev@incubator.apache.org

All Dev, Build and QA documents are posted here Defects can be logged here

Q & A

ASF CloudStack – QA StrategyApache Cloudstack Collaboration ConferenceDec 1, 2012

Sudha R Ponnaganti, Sr Manager, SQA Engineering, Citrix Systems Inc.,

Email:sudha.ponnaganti@citrix.com