express functional testing approach- doing more with less

Express functional testing approach-doing more with less

Anuj Magazine

anuj.magazine@citrix.comhttp://anujmagazine.blogspot.com

Twitter: anujmagazine

© 2012 Citrix | Confidential – Do Not Distribute2

Have you…• Ever been asked to test a project

with no or very less or not to your satisfaction requirements specifications ?

• Ever been involved in testing a project which had no test cases or a very little time to create them ?

• Ever been involved in testing a project in which features/implementation changes dynamically ?

© 2012 Citrix | Confidential – Do Not Distribute3

Citrix Globalization Services

© 2012 Citrix | Confidential – Do Not Distribute4

Group’s key challenge

© 2012 Citrix | Confidential – Do Not Distribute5

Traditional G11N Functional testing process

G11N Testing Cycle

© 2012 Citrix | Confidential – Do Not Distribute6

Situations where Traditional process didn’t scale

Start-up projects Maintenance projects

Typical Shortcomings

• Less Base Language test cases or No Base Language test cases at all

• The “Rapid product changes” syndrome

• Difficulty in seamlessly keeping-up with Product team’s milestones

• Difficulty in dealing with situations when there are “no second chance” of finding bugs

• Regression testing not scientific enough

© 2012 Citrix | Confidential – Do Not Distribute7

The basis of Express functional testing approach

© 2012 Citrix | Confidential – Do Not Distribute8

What kind of change we were looking to see ?

• Something that can help us deal with the rapid product changes• Something that can help us find bugs soon and early• Something that can help us make Regression testing more scientific• Something that can help us deal with reducing dependency on EN(base

language) test cases• Something that can help us seamlessly merge with product team’s

milestones

© 2012 Citrix | Confidential – Do Not Distribute9

We had

• Less documentation

• Lesser base test cases

• Less time at hand

• Less chances of timely bugs

• Less preparedness to deal with chances

We wanted to

• Find more bugs

• Become more responsive to project changes

• Enhances our chances of meeting internal milestones.

• Not increase human resources infinitely

In short, we wanted to do more with less

© 2012 Citrix | Confidential – Do Not Distribute10

The basis of Express functional testing approach

Test caseTest case

© 2012 Citrix | Confidential – Do Not Distribute

The essence of Express functional testing

11

SBTM: Session Based Test Management

© 2012 Citrix | Confidential – Do Not Distribute12

The Three Core ideas

• Basic source of tests-

a code check-in• Refined approach to testing-

express test ideas generation• Automating functional tests –

reuse the existing code

12

Explaining the basic source of tests- a code check-in

© 2012 Citrix | Confidential – Do Not Distribute Citrix Confidential - Do Not Distribute

AnalysisVerification

Track Daily check-ins

Assess the functional impact

Generate test ideas at runtime

Note the build, results

Report bug if any issue

Perform regression tests

Results are updated in the

Tracker

Elements of the Express functional testing

© 2012 Citrix | Confidential – Do Not Distribute

Components of a check-in

15

© 2012 Citrix | Confidential – Do Not Distribute

“check-in Summary”

Actual Source Code Changes…

No.of Lines in the Code that were affected by the change

© 2012 Citrix | Confidential – Do Not Distribute

check-in types

17

• check-ins related to Feature Changes

• check-ins related to Bug Fixes

• check-ins related to Document Changes

• check-ins related to Changes in Automation script

© 2012 Citrix | Confidential – Do Not Distribute18

Check-in types

check-in for feature change

Using check-in Summary and the other details in the

source code we could derive ideas for tests

© 2012 Citrix | Confidential – Do Not Distribute

© 2012 Citrix | Confidential – Do Not Distribute

Some challenges

20

• Evolving the skill level of the team.

• Enhance the Developer skills and mix it appropriately with tester mindset to create an irresistible engineering combination.

• Build rapport/communication with the development team.

• Building the buy-in for the change in the approach.

• Imbibing the necessary Sense of urgency within the team.

Express test ideas generation/testing

© 2012 Citrix | Confidential – Do Not Distribute Citrix Confidential - Do Not Distribute

AnalysisVerification

Track Daily check-ins

Assess the G11n impact

Generate test ideas at runtime

Note the build, results

Report bug if any issue

Perform regression tests

Results are updated in the

Tracker

Elements of the Express functional testing

© 2012 Citrix | Confidential – Do Not Distribute

Challenges in absence of test cases

• Building accountability in testing.

• Establishing the focus into specific areas.

• Knowing how much is tested and how much more needs to be tested.

• Designing the tests and running them almost at the same time.

© 2012 Citrix | Confidential – Do Not Distribute

Session based testing

• What is a session?

“A session is an uninterrupted block of reviewable, chartered test effort”• Typical sessions last for 90 minutes

© 2012 Citrix | Confidential – Do Not Distribute

Session

Bugs Investigation/Reporting

Test Design/Execution Session Setup

Test design and execution means scanning the product and looking for problems

Bug investigation and reporting is what happens once the tester stumbles into behavior that looks like it might be a problem

Anything else testers do that makes the first two tasks possible, including configuring equipment, reading manuals, or writing a session report

© 2012 Citrix | Confidential – Do Not Distribute

Session Report

Automating tests- Reusing the existing code

© 2012 Citrix | Confidential – Do Not Distribute Citrix Confidential - Do Not Distribute

How the Automation Process Evolved….• Understanding Architecture• Execution on EN Setup

• Scheduler• Automatic analysis of test results• Auto e-Mail test Result

ANALYSIS

SUPPORT FOR NON-EN PLATFORMS

ENABLE UNICODE SUPPORT

OPTIMIZATION

• Execute Auto BVT on JA, FR, DE, ES, SC, RU languages

• Input non-ASCII characters as Input Data • (Multi Language Data)

© 2012 Citrix | Confidential – Do Not Distribute

• Make the scripts UTF enabled

• # -*- coding: utf-8 -*-

Citrix Confidential - Do Not Distribute

Enable the scripts to input non-ascii characters

© 2012 Citrix | Confidential – Do Not Distribute30

Multi Language data“ß証吞暴予禄 дю眀äöñé가앜刓訞一么@!23”

Results summary

© 2012 Citrix | Confidential – Do Not Distribute

More Functional bugs per release

32

If further colors are needed, please contact Creative Services

No. of EN issues

No. of I18N issues

No. of L10N issues

No. of Doc issues

0

20

40

60

80

100

120

Project K Project N

© 2012 Citrix | Confidential – Do Not Distribute

Early, Timely detection/fixing of Functional bugs

33

If further colors are needed, please contact Creative Services

Itr 1 Itr 2 Itr 3 UI Itr RC Itr0

20

40

60

80

100

120

Project K Project N

© 2012 Citrix | Confidential – Do Not Distribute

“It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change”

The crux of Express functional testing approach

© 2012 Citrix | Confidential – Do Not Distribute

Images Sources:• http://www.cleapublicrelations.com

• http://www.c3-ssi.com

• http://www.linkbuildr.com

• http://www.torwars.com

• http://www.branchpastor.com

• http://blog.workinsports.com

• http://www.121box.com

• http://www.march-hare.com

• http://www.larter.ca

References

Other Credits:

• Collègues: Rubina Gonsalves and Shivaraj Shet

• http://www.satisfice.com/sbtm/

• http://satisfice.com/articles/sbtm.pdf

Work better. Live better.

Anuj Magazine

anuj.magazine@citrix.com

http://anujmagazine.blogspot.com

Twitter: anujmagazine