saic-f qa internal process (draft ) sudha chudamani qa team, frederick national lab jan 2, 2013

SAIC-F QA Internal Process (DRAFT )SAIC-F QA Internal Process (DRAFT )

Sudha ChudamaniSudha Chudamani

QA Team, Frederick National LabQA Team, Frederick National Lab

Jan 2, 2013Jan 2, 2013

Frederick National Laboratory for Cancer Research

QA Release Process MatrixQA Release Process Matrix

Release to QA Expected from Development team Expected from QA team Time to

Test (in

Weeks)

Release Notes

List of Bugs fixed

List of System Improvements

List of new

features

MST / BST RT ST PT 508

TAG/Build/ Iteration

x x x x 1-2 W

Milestone Release

(Limited)

x x x 1-2W

Release Candidate (RC)

x (Cumulative)

2-4 W

Post Release Patch release

or Hot Fix

x x TBD (determined by NCI

PO)

TBD (determined by

NCI PO)

x TBD

LEGENDS:MST / BST – Milestone Specific Test / Build specific TestRT – Regression TestST – Security Test (Appscan)PT – Performance Test508 – Section 508 Compliance


QA Project Initiation PhaseQA Project Initiation Phase

– QA project needs are evaluated for skills and resources

– Contract awarded if sub-contractor is chosen

– Else SAIC-F internal QA resource is allocated

– QA Resource / POC, reviews the Project schedule, Project requirements etc., and prepares:

• Deliverables:

– Prepares QA schedule

– Creates / Updates Test Plan

» Types of testing (Installation, Performance, Interface testing, Section 508 etc.,)

– Sets up, Clean up & Maintain JIRA, HPQC etc.


QA Project Testing (TAG/Build/Iteration) Phase QA Project Testing (TAG/Build/Iteration) Phase

– Tag/Build/Iteration Testing Phase

• QA POC reviews requirements and writes Test Cases

– Test cases are written for functionality planned to be delivered for the TAG prior to receiving TAG, which includes test cases that cover:

» New Functionality

» Feature / Functionality Enhancements

» Defect Fixes

» Non-functional test cases, such as performance tests, security tests, or conformance

• Before the first tag is

• delivered, QA creates the test environment

• Once TAG is delivered to QA

– QA verifies the technology stack conformance (for the first tag)

– QA reviews the release notes and deployment guides

– QA installs the application on the QA tier

(cont.)


QA Project Testing (TAG/Build/Iteration) Phase QA Project Testing (TAG/Build/Iteration) Phase (cont.) (cont.)

• Once TAG is delivered to QA (cont.)

– Executes Smoke tests to accept the TAG

– Executes the Test cases planned for the TAG

– Reports Defects

– Test Cases are written for next TAG

Note: Prefer to run 508 compliance and performance tests on the MR as well

• The above steps are repeated for each TAG.


QA Project Milestone Testing PhaseQA Project Milestone Testing Phase

– Milestone Testing Phase

• QA POC writes / updates Test Cases

– Test cases are written for testing entire feature / feature enhancements delivered for the Milestone prior to receiving TAG, which includes.

» Test cases that cover new features and enhancements

» Defect Fixes

» Limited Regression Test cases around the feature / defect are identified and updated if needed

• Once Milestone TAG is delivered to QA

– QA reviews the release notes and deployment guides

– QA installs the application on the QA tier

– Executes Smoke tests to accept the Milestone TAG

– Executes the planned Test cases

– Reports Defects

– Plan and writes Test Cases for the next TAG.


QA Project Release Candidate (RC) Testing QA Project Release Candidate (RC) Testing PhasePhase

– When all the planned features for the release are developed, a RC tag is cut by the development team.

• RC tag should not contain any new features / functionalities

• No show stopper bug

• Can contain defect fixes that are found in previous testing phases

– QA will deploy (or request Systems team) the RC on the QA tier

– Execute the tests planned for the Release

• Execute all the Test cases developed for the release

– Test cases written during TAG/Build/Iteration testing phase

– Test cases written during Milestone testing phase

• Execute the full Regression Tests

• Execute the Performance tests

• Execute the Section 508 compliance tests

• Retest all the bug fixes - If new defects are found, they will get fixed and a new RC tag will be cut


QA Project Release Candidate (RC) Testing QA Project Release Candidate (RC) Testing Phase - ContinuedPhase - Continued

– On successful completion of RC testing, AppScan is requested to be run on QA tier (by the Systems Team)

– QA POC documents the Test Report and submits the report to SAIC-F QA Manager for final review and recommend to (NCI QA Lead) promote the RC to Stage tier. The Package submitted should include the following artifacts at the least:

• Test Plan

• Test Report in EPLC format

• Test cases document

• Test Execution Results

• Outstanding Defects list and their severity (The outstanding defects and their severity need to be reviewed and approved by PO)

• Performance Test Report

• AppScan report

• 508 Compliance report


QA STAGE tier Testing PhaseQA STAGE tier Testing Phase

– Once RC is approved by the NCI QA Lead, SAIC-F QA Manager will send the sign-off to the Project Manager to be deployed to STAGE tier

• QA Executes Smoke Tests

• AppScan is scheduled

– If AppScan reports High / Medium vulnerabilities, Dev team fixes the vulnerabilities and a RC is cut and deployed to QA environment where complete Regression Test is performed. Exceptions apply.

– Stage Production Test Report is submitted by QA POC. This again goes through the Approval process before it is signed-off.

• Test Report in EPLC format

• Smoke Test cases document

• Test Execution Results

• Outstanding Defects list and their severity level

• AppScan report


QA ProcessQA Process

– Application Release is Deployed to PRODUCTION tier.


Proposal to improve testing efficiencyProposal to improve testing efficiency

– Challenges/Risks

• Every project uses Release Terminologies differently

– Proposed Resolution

» Create and publish (If one does not exist) a Release Process document which should include a Glossary of Terminologies and share with PMs, Developers and QA.

» Should be shared with New Employees / Contractors as part of orientation.

• Frequent releases to QA


» QA should not accept any unplanned TAG without approval from NCI/SAIC-F QA Management

» One exception - if the previous TAG failed installation or Smoke Test, this again needs approval from QA Management.

(cont.)


Proposal to improve testing efficiency (cont.)Proposal to improve testing efficiency (cont.)

– Challenges/Risks (cont.)

• Test Maintenance effort not planned for in the project schedule


» QA should ensure that this activity is included in the Project Schedule

» This activity involves Updating Regression Tests, Organizing HPQC and JIRA, Updating Master Test Plan with planned activity for the upcoming release etc.

• Too Many Release Candidates (RC Tag)


» Reduce Regression errors in RC by:

» Implementing Regression Automation

» Execute full Automated Regression Tests early on in the testing phase.

» If no automation implemented, Instead of Limited manual Regression Testing in Milestone phase, run full regression tests.

» For the first RC TAG, enough time should be provided to execute a complete regression tests and all release test cases.

(cont.)


Proposal to improve testing efficiency (cont.)Proposal to improve testing efficiency (cont.)

– Challenges/Risks (cont.)

• Appscan process

– Time consuming

» QA runs initial Appscan in local environment

» Share the Test Script with Systems

– Inconsistent results

» Have a way to update the Appscan Test Scripts for new features or pages rather than re-writing a complete new script.

» Document the Configuration settings that can be shared across projects

• Number of hardware and software platforms to be considered


End

saic-f qa internal process (draft ) sudha chudamani qa team, frederick national lab jan 2, 2013

Documents

tag qa

qa schedule

qa tier

qa teamtime

updates test cases test

planned test cases

test environment

milestone tag