cbiit quality assurance process preston wood nci cbiit government quality representative (gqr)...

CBIIT Quality Assurance Process

Preston WoodNCI CBIIT Government Quality Representative (GQR)

January 2014

RS

CBIIT Quality Assurance (QA) Process (1)

Agenda

• Goals for Quality Assurance• Quality Assurance Process• QA Artifacts• Support for Iterative Development• QA Process Innovations

2


Goals for Quality Assurance

• Requirements are verified• Test Approach is appropriate• Test coverage is adequate• Cost and schedule is understood• Testing can be completed on time• Audit capability is ensured• Confidence in releasing the application

3


4


QA ArtifactsShould be specified in early stage of the project.

Test Plan• Defines the test approach • Outlines scope of tests • Determines what time frame the project can be

completed– Schedule in P6 to determine cost/time impact to project

• Identifies risks going into the project, and mitigation plan• Requires sign-off and for any modification.

5


Test Cases

• Step by step directions for the tester• Contains verifications• Part of comprehensive test approach• Are living documents and updated.

6


Requirements Traceability Matrix (RTM)

• Identifies each requirement specifically• Identifies and matches test cases to requirements• Shows level of planned test coverage for identified

requirements.• Is a living and updated document.

7


QA Entrance Checklist• Vital to a repeatable QA process• Ensures not missing components to conduct testing:

– Tier is provisioned and code is operational– List of features implemented, and known issues– Unit test report from Development– Deployment documentation– Installation guide – Test case review– RTM is complete– Tag accepted by QA (smoke test prior to formal tests)– QA Test Plan reviewed and signed-off

Note: “No” is a possibility for an individual item, but has to be approved

8


Test Report

• Conveys what was tested• Contains variances on tests conducted• Announces what new issues were found, fixed, or still

exist• Categorizes defects to show problem areas of the app• Identifies risks to the project going forward, and

mitigation plan• Requires sign-off and is final.

9


Updated Requirements Traceability Matrix (RTM)

• Includes additional test cases• Shows completion of tests• Shows final level of test coverage for identified

requirements.

10


QA Exit Checklist•Vital to a repeatable QA process•Ensures not missing components to complete the tier:

• Source code tag received• List of defects fixed, features implemented• Test cases and execution reporting• Deployment documentation• Installation guide• Test cases list for Appscan• Appscan and 508 compliance passed

Note: “No” is a possibility for an individual item, but has to be approved

11


Audit and Confidence

The QA artifacts give us confidence in releasing the software to the next tier or Production because we have full audit capability to know:

– What was tested– How it was tested– Why it was tested– Where it was tested– When it was tested– Who tested it

12


And we know,

• Issues that are fixed,• Issues that still exist, and• What we are going to do about them.

13


The most important aspect of this process is that it is repeatable.

14


Grand Assumptions being made:

•All requirements are known from the start•No new functionality is introduced after testing begins.

What if new functionality is introduced while testing? – Not known before Test Plan written – Not a phased approach

15


Challenges for QA:• Test approach is now incomplete and unable to fully

address the new functionality.• Scope of testing may increase• Project may exceed schedule or cost• Project risks change and/or increase• Risk mitigation plan is obsolete• Project could be open to failure• Audit capability compromised• New project expectations may exceed QA capability

16


A ripple effect will occur in the community if any challenge goes unanswered.

Q: How do we address these challenges to reestablish confidence in releasing the application to Production?

A: Implement support for IterativeDevelopment

17


18

QA Support for Iterative Development

Iterative Testability Assessment

1. New functional requirements are identified

2. QA conducts requirements testability assessment

3. QA determines if updates to test plan needed:• Test Approach change needed?• Test Scope and/or Environment change needed?• Test Schedule and Milestones change needed?


If all questions are “no” (typically bug fixes only), then

• New test cases are created• RTM is updated as usual• Testing resumes without changes to the Test Plan.

19


If any question is “yes”, QA advises project group of the impact to the project.

The project determines a cost assessment of the impact.

Should the impact and cost be deemed acceptable,

1. The test plan is edited accordingly to account for the new requirements,

2. RTM is updated with new requirements and test cases

3. GQR, GPM, and GS give sign-off on the edits.

4. Then the new tag is accepted, and testing resumes for the new iteration.

20


21


Benefits of the Iterative Testability Assessment

•Requirements are properly verified•Audit capability is ensured•Confidence in releasing the application is restored

because…

22


Benefits of the Iterative Testability Assessment

•New requirements were analyzed and documented.•Test Approach was updated and comprehensive.•New timeline and cost for QA was assessed and accounted for.•Correct environment to test was confirmed.•Test coverage was appropriate.

So testing can be completed on time and on budget.

23


Challenges to QA Support for Iterative Development

•Iterative development can be cost intensive– Restarting QA multiple times.– New functionality may invalidate previous verifications.– Completed tests may need to be re-run.

•Finding the balance point, cost analysis– More iteration/costs vs features to the community– Streamlining the sign-off process

• new JIRA Workflow Process

– Tracking the project in P6

24


QA Innovations and Focus on Individual Sub-Project Needs

• QA Process Modification – Sub-Projects deploying to Production tier before end of

FY’13. – Critical fix deployment

• Funding running out• Test Plan waived

– assume previous test approach would be adequate

• Form Builder 4.0.3.1 critical fix– deployed from Dev to Production tier in two weeks

25



• Implemented QA support for iterative development processes– QA can properly support sub-projects with multiple

iterations containing new requirements– EPLC process can be maintained

26



• QA Process Improvement– Review and grooming of QA artifact templates– One test plan– Research for a new appscan policy proposal– Improve workflow, transparency of process

• Adopt new JIRA workflow to track projects

27


Recent QA Process Innovations:

• Review and Approval of Test Cases in HP Quality Center– Review, comments, approval

• Electronically approve QA Artifact PDF– For projects that request that level of tracking

• QA Document Generation by HP Quality Center– Test Case spreadsheet (alt. workflow for test case review)– RTM (needs requirement tracking in HP-QC)

28


29


Reference material:

CBIIT QA wiki site

https://wiki.nci.nih.gov/display/CBIITQA

CBIIT Engineering wiki site

https://wiki.nci.nih.gov/display/CBIITeng/CBIIT+Engineering+SOPs

30

https://wiki.nci.nih.gov/display/CBIITQA





Questions

31

cbiit quality assurance process preston wood nci cbiit government quality representative (gqr)...

Documents

repeatable qa process

qa smoke test

quality assurance requirements

formal tests qa test

test cases step

implemented test cases

additional test cases

appropriate test coverage