apache cloudstack qa strategy

Download Apache Cloudstack QA Strategy

Post on 18-Jul-2015




1 download

Embed Size (px)


PowerPoint Presentation

ASF CloudStack QA StrategyApache Cloudstack Collaboration ConferenceDec 1, 2012

Sudha R Ponnaganti, Sr Manager, SQA Engineering, Citrix Systems Inc.,Email:sudha.ponnaganti@citrix.com

1QA Strategy - IntroductionNeed for QA strategyGoals for current presentationTransparency Expected outcomeCommunity participation Superior Quality ReleasesExpected release cycles2AgendaReleases and QA OverviewQA strategy Future Community participationQ & A and Feedback

Releases and QA Overview

Releases and QA overview

Continuous Release CyclesCurrently we are doing minor and major releases Cycles are longerTime based vs. feature based release discussion

There is a need for

Continuous integration Continuous validationContinuous IntegrationEvery check-in gets tested as and when it is submittedBuilds upfront quality faster Should take less than 5 minutes to completeBelow is the process that is being worked up on currently

Continuous ValidationRun all the tests on nightly buildsWith current structure, below are the sequence of tests that run nightly

There is a planned Test session to achieve full compliance to the above model Please participate in that session Look for discussion on dev-cloudstack mailing listQA StrategyInstallation and Upgrade Validation

Compatibility Matrix Manual Validation

Installation and Upgrade ValidationInstall and Deployment validation Specifics:Install and deployment Validation need to be done in context of release artifacts to be in alignment with voting process.Step by step instructions are posted here. https://cwiki.apache.org/CLOUDSTACK/cloudstack-40-test-procedure.html

Participate and install on various configurations during QA cycles Gives early indications of issues rather than waiting till voting is called for.

New Feature and Regression ValidationNew Feature ValidationRegression ValidationQA team writes test cases for new features based on FS.QA posts test plans to community to get feedbackQA automates and executes new test cases for current release. Adds TCs to regression for future releases

We can provide template if you need to add test cases. You can find test suites on wikiThere are around 3000 + test cases in the regression suite currently.This suite includes Automated and Manual testsQA reviews overall feature list and identifies regression coverage based on impact for each release not sustainableTo be successful in open source env, need to have 100% automated coverage

Submit automated / manual ( unit tests and functional ) test cases as part patch / new plugin / feature / integration. New Features and Regression ValidationVarious Frameworks are available based on Junit, Python, Selenium to automate test casesA combination of all of these provide comprehensive coverage

New Features and Regression Validation

Unit Test framework ( JUnit )Provides early feedback and fixes issues close to sourceHow to Write Unit tests - https://cwiki.apache.org/CLOUDSTACK/unit-testing-101.htmlCurrently all unit tests running with 100% pass rateRuns on every build Code coverage results are posted here

Participate in Unit Test activity planned on Dec 02, to improve coverage. New feature and Regression ValidationBVTs and Functional Automation - Marvin600+ test cases running on multiple configurations and covering multiple hypervisorsHow to write functional automated TC : https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html

New features and Regression ValidationMultiple test pods are being setup to run suites in parallel around 15 test pods are requiredPorting to ASF CS env has been in progress. BVTs are done. Prasanna and Edison are covering more detailed talk on Automation on Sunday morning.

New Features and Regression ValidationIntegration frameworksCurrently we are in the process of submitting test suite that covers EC2 Query API validation JCloud frameworkAround 40 APIs are automatedIn the process of covering all APIsFailed test cases are being fixedNeed to automate complex scenarios. Details on test implementation are posted below https://cwiki.apache.org/confluence/display/CLOUDSTACK/EC2+Query+API+Test+Suite+Automation

Any other tools from community are most welcome. But higher priority is to beef up existing suites. New Features and Regression ValidationUI AutomationSeleniumFaster and Easy Compatible with python backendPlan to Deliver Smoke Tests by Early FY 13 POC and initial submits coming soon All frameworks are available in ASF CS repo Make use of them either to validate your features or contribute to existing automation.Performance and Scalability TestingNative Tool - SimulatorSimulator - mock hypervisor and will react to every command sent to it by the MS similar to a regular hypervisor - easy to setup and runSimulator and Perf resources: http://wiki.cloudstack.org/pages/viewpage.action?pageId=13449532 papers have been published CoveringBasic ZoneAdvanced Zone

Simulator is great way to get started on performance validation Performance Use casesTime taken to propagate security group rules in basic zone Deploy VM time, CPU, load averages and # of DB connectionsSteady state CPU, load average and # of DB ConnectionsRestart MS measures: Time to stop MS, Start MS and rebalance with 30K Hosts across 4 MS (agent load size to 500 and 1000)Restart MS measures : with Host in maintenance modeBandwidth usage collection frequency for routersRouter Status poll metrics in RVR envResults are posted here Performance TestingWhat NextHolistic Coverage RouterStorageNetworkRealistic Scale Testing

Post performance considerations when you submit features. Would help QA to focus on areas of concern. Make use of simulator for your validation Security Validation

Security validation has been important aspect of ASF CS releasesAll FS should include security considerations so that QA can identify and cover these areas Security Coverage done included the following along with manual validation and other security reviews

There is focused Security session review that to understand full scope and coverage Solution Validation Current solution Validation use cases include reference architecture deployment and additional complex deployment scenarios. Test plans are posted on wiki

There is no need to adopt these right away in to Cloudstack but will monitor if there is a need in future

Recovery, Usability and L10NRecovery scenarios, Usability Reviews/ L10N - Not prioritized Jessica Wang has been doing L10N review along with few other contributorsIf you have expertise in these areas, do participate in QA forums

Documentation review Focused validation at the end of the cyclesWatch for QA requests for reviewDefect Triage and ValidationYou can participate by testing product logging defects here ( ASF JIRA)Always log defects when you start discussion on mailing listsAdd automation for defects that you validate. Tests can be checked in to appropriate suites based on type of test unit/smoke/regression suitesParticipate in Bug fest activities

Testing is great way to learn product. If you are new to CS, you can start off by participating in testing activities or fixing defects. FutureFuture - StatusSome improvements that QA has been doing Q3/ Q4 FY 12Automation Functional BVTs and Regression (Python framework) - goal 700+ TCsUnit Test Framework ( Junit ) Add TCs for ongoing releases + improve legacy by adding TC for patchesIntegration automation frameworks (ec2 and s3) implementation and make it part of regression suite ( Jcloud Test framework)Future - statusTCM Implementation and loading test cases Address Test Case gaps ( total inventory) Currently 3000 test cases exist. Goal is to get to 5000Performance White paper Use cases collected and release V3 with known use cases backlog has been built. Tools Static code analyzer In progressCode coverage Cobertura Code inspection -

Future - status Infrastructure improvements Test Env improvements ( Fresh Install Automation for Functional testing)Acquire external devices and standardize test envBuild infrastructure improvements Continuous integration implementation Continuous validation Standardized certifications New versions of HV, NS and other integrationsStandardized templates Test plan/Test case / Defect writing Guidelines, Test case execution reports, baseline metrics

Future - statusInitiatives for FY 13 Move towards enterprise quality standards Unit Test Coverage + Automation 70% + Code coverageRegression coverage 90% + coverageNew features 100% coverageDefects regressed through automationInstall/Upgrade AutomationUI automation - seleniumStandardize QA coverage for enterprise readinessRegressionNew functionalityAutomationInstall/upgrade ( compatibility matrix automation)UsabilityPerformance ( tools and use case expansion)Reliability ( longevity under load)L10N ( internal or vendor based)

Future - status508 ( Accessibility)IntegrationSecurityReal world use casesInstall Upgrade automation Infrastructure improvements to run automation across multiple configurations

Community ParticipationCommunity ParticipationScheduled QA Meetings on weekly basis Will be publishing meeting details for CS 4.1 and expect it to be starting soon. Everyone is welcomed to join and participate in QA Activities for ASF CS 4.1 releaseDavid has planned IRC Sessions to drive Quality as a continuous process. Please do participate and help to drive this continuous test process - timings 1800 UTC in #cloudstack-dev on Wednesdays ASF Cloudstack Dev mailing list is great way to share your ideas and contributionscloudstack-dev@incubator.apache.org

All Dev, Build and QA documents are posted here Defects can be logged here

We would not be