test process updated

7/31/2019 Test Process Updated

1/18

www.ccpu.com Confidential and Proprietary

SQA Test Process for Femtotality


2/18

2www.ccpu.com Confidential and Proprietary

Verification Process

Unit TestingCover the functional aspects of acomponent.Ensure that all the error/failurescenarios are handled properly.Integration with other components.FACT test cases verification

SIT (System integration Testing)Check-in VerificationDaily nightly build RegressionComprehensive release testingincluding NE-SM IntegrationSanity testing on the nightly build

End-to-endTesting withCustomerspecificEquipmentHand heldDevicesRANCore

LAV (Live AirVerification)End-to-end,RF-relatedfunctionalityandperformanceHome Trials

Verification andAcceptance TestLive Trials

Development QACustomer

AcceptanceIOT

SanityEndto-End SystemTestingFunctionalRegressionPerformanceLoadStressRecoveryUsabilityDeploymentInstallationUser documentationUpgrade RollbackSOAKMPS


3/18


QA Process

Feature SystemsRequirementSpecification

QA testplan templateReview testplanMeeting minutes

FSRS QA Test Plan

Approve testplanTestplan repository

Testplan version controlClearquest Database

RequirementsTracking

Testcase Results

Master Control

RequirementsTracibility Matrix

Feature Acceptance Test(FACT)

Test Start through TestFirst Pass(TS-TF)

Test First Pass throughTest Done(TF-TD)

First Customer Shipmentthrough

General Availability(FCS-GA)


4/18


QA Test Plan

1. The format of QA test plans are derived from a test plan template to ensureconsistency in all test plans and allows for imports into the test casesdatabase.

2. The template includes required sections such as:a. feature description

b. product coveragec. test methodologyd. restriction/limitationse. test bed set-upf. appendix for required meeting minutes of test plan review


5/18


QA Entrance Criteria example (1 of 3)

Marketing/Program Management1. Project Plan/PRD is complete and has been updated

Hardware1. 0D version (or other approved version) of hw available to QA2. Hardware version control in place

Software1. All Functional Specifications complete and have been updated to accurately reflect the

implementation, including diagnostics2. All Design Specification complete and have been updated to be sync with the implementation3. All Code complete and frozen Code changes allowed only for bug fixing but not features4. Software version control in place5. Unit test plan (s) and SIT test plan (s) are reviewed and approved by QA6. 100% unit tests executed and passed7. Unit level test plan(s) reviewed and approved by QA


6/18


QA Entrance Criteria (2 of 3)

FACT

1. FACT test plan(s) reviewed and approved by Dev/QA

2. First Pass Test cases ~70% (if less than 70% the feature need to go back todevelopment for UT and IIT)

3. 100% FACT tests executed and 95% passed or deferred (as approved by QA)since start or since a revert back to start

4. No more than 1 crash/week averaged since FACT start or since last revert.5. 72 hours of soak time on the QA load

6. No P1 bugs open without QA agreement

7. No P2 bugs open without QA agreement

8. Less than 50 P3 and P4 bugs open


7/187www.ccpu.com Confidential and Proprietary

QA Entrance Criteria (3 of 3)

QA

1. Test Strategy document completed and approved by all relevant groups (SWDevelopment, QA, System Eng where applicable)

2. Test Plan(s) completed and approved and entered into database

3. TC execution plan in place

4. Cross- referencing between test cases to requirements in place

Tech Publication

1. Documentation updates will be available for QA testing, Deployment,Upgrade/Rollback and CLI Guide should be available at the Test start.

2. Online help needs to be included in all major drops to QA
http://www.google.co.in/imgres?imgurl=http://www.trinergy.co.th/en/articles/wp-content/uploads/2009/08/02.jpg&imgrefurl=http://www.trinergy.co.th/en/articles/%3Fp%3D775&h=319&w=552&sz=62&tbnid=DtjrbfYZuYqrJM:&tbnh=77&tbnw=133&prev=/search%3Fq%3DTektronix%2BG35%2Bpicture%26tbm%3Disch%26tbo%3Du&zoom=0&q=Tektronix+G35+picture&hl=en&usg=__QZwecNQ6isoLt4GYzYTfXgHn8WM=&sa=X&ei=F2fTTarADIGbtwfF99msCg&ved=0CCIQ9QEwAw



Feature Acceptance Test (FACT)

1. The purpose of the FACT is to ensure the new feature and release meets aminimum level of quality before it enters formal QA testing.

a. The FACT test plan is jointly created by QA and Development who makeup the FACT team. It is essential that test cases are chosen carefully tocover the feature and its interaction with the existing code and otherdepended features.

b. QA supplies equipment and is responsible for execution of FACT as theacceptance criteria

2. A branching and drop strategy for multiple drops to QA is agreed betweendevelopment and QA.

a. Each drop must have a FACT plan to get formally released to QA testing.

3. When the code base passes FACT and meets the other QA entrance criteria, QAtesting begins.

a. Development is responsible for passing FACT and meeting the handoffdates to QA.
http://www.google.co.in/imgres?imgurl=http://www.trinergy.co.th/en/articles/wp-content/uploads/2009/08/02.jpg&imgrefurl=http://www.trinergy.co.th/en/articles/%3Fp%3D775&h=319&w=552&sz=62&tbnid=DtjrbfYZuYqrJM:&tbnh=77&tbnw=133&prev=/search%3Fq%3DTektronix%2BG35%2Bpicture%26tbm%3Disch%26tbo%3Du&zoom=0&q=Tektronix+G35+picture&hl=en&usg=__QZwecNQ6isoLt4GYzYTfXgHn8WM=&sa=X&ei=F2fTTarADIGbtwfF99msCg&ved=0CCIQ9QEwAw



QA Test Cycles

There are 3 cycles of testing and each cycle has its own entry and exit criteria:Test Cycle Goal

TS to TF

(TestReady to TestFirstPass)

100% of test-cases are executed andretested to achieve a 95% passrate. The first pass rate should

be more than 80% per feature.

TF to TD

(TestFirstPass to TestDone)

Bug fix validation and regression areexecuted to achieve a 99% passrate

All soaks with call model should pass

and all agreed KPI/KCIs shouldbe met.

FCS to GA

(First Customer Shipment or FOA

to General Availability)

Product acceptance in customer

environment



QA Test Cycles: TS to TF

Test Start (TS) to Test First Pass (TF)Goal1. Before TF, all the test cases from the test plan(s) will be executed.2. Failed test cases will be re-run to achieve 95% pass rate.

Exit Criteria:1. All test cases will be executed at least once2. First Pass rate at 70%3. 95% pass rate4. No P1s5. < 10 P2s, all documented and accepted by QA6. < 50 P3/P4s7. < 2 crashes/week between TS to TF (from last revert, if necessary)8. 72 hours of uninterrupted soak on the soak/capacity test bed



QA Test Cycles: TF to TD

Test First Pass (TF) to Test Done (TD) Goal:

1. Validate bugs and retest failed test cases.2. A Regression set of test cases will be re-executed3. 98% of test cases should pass

Exit Criteria1. All the test cases executed/addressed2. Pass 98% of all the test cases3. Compliance matrix will be provided.4. Test cases are updated in test Database with updated test procedure5. No crash in last two weeks of all testing (as recorded in the test

report)6. 0 P1, 0 P2s, no more than 30 P3/P4s7. 72 hours of uninterrupted soak on the soak/capacity test bed8. All the resolved bugs must be in closed state9. Cross-functional group agreement of release notes covering all open

bugs with assessment of customer impact



QA Test Cycles: FOA/CUR to GA

First Customer Shipment for FOA to General Availability (GA) Goals

1. Product acceptance in customer environment

Exit Criteria1. The product is released to the customer.

2. All required customer bugs fixed3. Regression tests successful4. 2 week acceptable soak of final release



Rational Clearquest: Bug and Testcase Tracking

1. Rational Clearquest is our bug tracking system

a. Integrated with Clearcase software version control

b. Code check-ins are cross-linked to bugs

2. Rational Clearquest is our QA testcase database tracking system

a. The database tracks test results: Pass/Fail

b. Testcase failures are cross-linked to Clearquest bugs



Bug Priority & Severity Definitions

Priority Effect on Scheduling of Fixes Criteria

1 - Product in QA: Resolve

Immediately Must be fixed

for next build of code to SQA

- Product in field: patch or

maintenance release

candidate for immediate

delivery.

- Blocks testing progress

- Change poses major risk

to large amounts of

code.

- Unacceptably hinders

customer use of

product.

2 - Product in QA: High Attention

Try to fix for next build to QA.

- Product in field: Fix for next

release to field.

- Blocks a limited area of

testing

- Change poses some risk,

but not for major

destabilization

- Important function is not

working for customer

- Trivial problem, but makes

us look silly.

3 - Product in QA: Normal Queue -

Try to get a fix in before final

QA Cycle check in.

- Product in field: Schedule for fix

in an upcoming release to

field.

- Blocks testing of only that

small, specific item of

functionality.

- Doesnt hinder use of the

product for required

purpose.

4 - Product in QA or the field: Low

Priority - When all higher

priority bugs are fixed.

- Doesnt meet any of

above criteria.

Severity Criteria Side-effect on FixScheduling

1 - Box or board crash or hang

- Major function not working, no

workaround

- Large Resource leak (memory,

buffers)

- Memory corruption

Fix before lesser

Severities of

same Priority

2 - Major function not working,

workaround

- Minor function not working, no

workaround

- Significant performance issue

Fix before lesser

Severities of

same Priority

and after higher

Severities of

same Priority

3 - Minor function not working,

workaround

- Operation inconvenient or

inconsistent

- Cosmetic imperfection

Fix last of this

Priority


15/18


Unit Test Process

Entry Criteria:1. Final reviewed version of FRS, FSRS document. This provides feature level

requirements.2. Final reviewed version of FAS document. This provides the agreed-upon feature

requirements, architecture and external interfaces for the feature.3. Reviewed version of FS document. This provides component level specification for the

feature.4. Reviewed version of OAM&P document. User Interface document related to feature

reviewed.5. Reviewed version of DS document that defines the design for the component.6. Compiled and running software on target hardware for the component / feature.7. Code review for the software (Under Test) conducted and review comments

incorporated.

Exit Criteria:1. 100% success on passing the entire unit tests for the component. Unit test cases and

results should be archived as quality record.2. Purify memory leak and coverage results should be recorded in Project notebook.

3. Project notebook should be updated with unit test results.4. Problems found during unit tests are fixed and submitted in Clearcase.5. Number of problems found during unit testing should be recorded in Project Notebook.6. Component level regression testing should be conducted and anomalies should be

corrected.7. Component in ready to check-in in feature branch.


16/18


System Integration Test (SIT) Process

Check-in Verification1. Basic sanity check of the software components functionality as well as end-to-endverifications of the overall system functionality.

2. New software component, features or a bug fix is verified in an end-to-end systemenvironment before check-in to code base in SIT branch.

3. The developer fills a check-in request form after the software component, feature or a bug fixhas been unit tested, integrated with other software components and tested by the developer.

4. The check-in request form contains pertinent details regarding the check-in and its testing.

5. Once the check-in form is reviewed and approved by the software manager, the new softwareis put to test in an end-to-end SIT test bed.6. After the SIT verification and testing the view is merged with rest of the code in SIT branch.

The merged code is then checked-into SIT.

Daily nightly build Regression1. Daily Smoke Test: maintains the quality of software and ensure that previous days check-ins

did not break the system in any way.2. Nightly build regression test plan ensures that all aspects of the system have been verified.

Release Testing and Regression1. Regression Testing: to ensure that new release does not break features from previous

release.2. New Feature Testing3. Capacity and Performance Testing


17/18


Streams/Drops

CIT = Check In Test

(includes smoke test & P1 of FACT)Coding

Unit Test FACT = Feature Acceptance Test (section of QA Feature TestPlan)

Functional Test

Integration Test

Stress Test

CIT

Release Branch LOCK FACT QA Feature Test

Feature B Drop2

Feature C

Feature B Drop1

Feature A

Release Branch FACT A FACT CFACT B (1) FACT B (2)

QA Feature Test A QA Feature Test C

QA Feature Test B (1) QA Feature Test B (2)

QA Regression

QA System Test

QA Load & Soak Test

QA LAV & Performance Test

Design

TestPlan

merge

QA Feature TestPlan

final

merge

FACT

Complete

Design Test

Complete

High Level View

Bundle #1 Bundle #2

Complete

CheckIn synchronized

w hen ALL features

in bundle are ready, so

FACT start is synchronized

NOTES:

1.Design owns milestones up to FACT Complete.

2.Joint cooperation & learning between Design & QA

during FACT.

3.EMS-NE functionality must be available for FACT to

start, but CheckIn of NE or EMS can precede

indpedendently.

4.CRs written for FACT failures/deferrals. 95% pass

required for FACT complete.5.End-to-end equipment required for FACT. (details to

be worked)

6. A feature could have multiple Drops (e.g. IRSHO).

7. Features are bundled together. All features within a

bundle are synchronized so checkin & FACT starts

with same image (this way Integration is with pre-

existing features + features within bundle)

8. QA regression test etc. picks up the load once

FACT complete for all features within bundle (tbd, this

could be changed).


18/18


Thank You

test process updated

Documents