Express functional testing approach-doing more with less
Anuj Magazine
[email protected]://anujmagazine.blogspot.com
Twitter: anujmagazine
© 2012 Citrix | Confidential – Do Not Distribute2
Have you…• Ever been asked to test a project
with no or very less or not to your satisfaction requirements specifications ?
• Ever been involved in testing a project which had no test cases or a very little time to create them ?
• Ever been involved in testing a project in which features/implementation changes dynamically ?
© 2012 Citrix | Confidential – Do Not Distribute3
Citrix Globalization Services
© 2012 Citrix | Confidential – Do Not Distribute4
Group’s key challenge
© 2012 Citrix | Confidential – Do Not Distribute5
Traditional G11N Functional testing process
G11N Testing Cycle
© 2012 Citrix | Confidential – Do Not Distribute6
Situations where Traditional process didn’t scale
Start-up projects Maintenance projects
Typical Shortcomings
• Less Base Language test cases or No Base Language test cases at all
• The “Rapid product changes” syndrome
• Difficulty in seamlessly keeping-up with Product team’s milestones
• Difficulty in dealing with situations when there are “no second chance” of finding bugs
• Regression testing not scientific enough
© 2012 Citrix | Confidential – Do Not Distribute7
The basis of Express functional testing approach
© 2012 Citrix | Confidential – Do Not Distribute8
What kind of change we were looking to see ?
• Something that can help us deal with the rapid product changes• Something that can help us find bugs soon and early• Something that can help us make Regression testing more scientific• Something that can help us deal with reducing dependency on EN(base
language) test cases• Something that can help us seamlessly merge with product team’s
milestones
© 2012 Citrix | Confidential – Do Not Distribute9
We had
• Less documentation
• Lesser base test cases
• Less time at hand
• Less chances of timely bugs
• Less preparedness to deal with chances
We wanted to
• Find more bugs
• Become more responsive to project changes
• Enhances our chances of meeting internal milestones.
• Not increase human resources infinitely
In short, we wanted to do more with less
© 2012 Citrix | Confidential – Do Not Distribute10
The basis of Express functional testing approach
Test caseTest case
© 2012 Citrix | Confidential – Do Not Distribute
The essence of Express functional testing
11
SBTM: Session Based Test Management
© 2012 Citrix | Confidential – Do Not Distribute12
The Three Core ideas
• Basic source of tests-
a code check-in• Refined approach to testing-
express test ideas generation• Automating functional tests –
reuse the existing code
12
Explaining the basic source of tests- a code check-in
© 2012 Citrix | Confidential – Do Not Distribute Citrix Confidential - Do Not Distribute
AnalysisVerification
Track Daily check-ins
Assess the functional impact
Generate test ideas at runtime
Note the build, results
Report bug if any issue
Perform regression tests
Results are updated in the
Tracker
Elements of the Express functional testing
© 2012 Citrix | Confidential – Do Not Distribute
Components of a check-in
15
© 2012 Citrix | Confidential – Do Not Distribute
“check-in Summary”
Actual Source Code Changes…
No.of Lines in the Code that were affected by the change
© 2012 Citrix | Confidential – Do Not Distribute
check-in types
17
• check-ins related to Feature Changes
• check-ins related to Bug Fixes
• check-ins related to Document Changes
• check-ins related to Changes in Automation script
© 2012 Citrix | Confidential – Do Not Distribute18
Check-in types
check-in for feature change
Using check-in Summary and the other details in the
source code we could derive ideas for tests
© 2012 Citrix | Confidential – Do Not Distribute
© 2012 Citrix | Confidential – Do Not Distribute
Some challenges
20
• Evolving the skill level of the team.
• Enhance the Developer skills and mix it appropriately with tester mindset to create an irresistible engineering combination.
• Build rapport/communication with the development team.
• Building the buy-in for the change in the approach.
• Imbibing the necessary Sense of urgency within the team.
Express test ideas generation/testing
© 2012 Citrix | Confidential – Do Not Distribute Citrix Confidential - Do Not Distribute
AnalysisVerification
Track Daily check-ins
Assess the G11n impact
Generate test ideas at runtime
Note the build, results
Report bug if any issue
Perform regression tests
Results are updated in the
Tracker
Elements of the Express functional testing
© 2012 Citrix | Confidential – Do Not Distribute
Challenges in absence of test cases
• Building accountability in testing.
• Establishing the focus into specific areas.
• Knowing how much is tested and how much more needs to be tested.
• Designing the tests and running them almost at the same time.
© 2012 Citrix | Confidential – Do Not Distribute
Session based testing
• What is a session?
“A session is an uninterrupted block of reviewable, chartered test effort”• Typical sessions last for 90 minutes
© 2012 Citrix | Confidential – Do Not Distribute
Session
Bugs Investigation/Reporting
Test Design/Execution Session Setup
Test design and execution means scanning the product and looking for problems
Bug investigation and reporting is what happens once the tester stumbles into behavior that looks like it might be a problem
Anything else testers do that makes the first two tasks possible, including configuring equipment, reading manuals, or writing a session report
© 2012 Citrix | Confidential – Do Not Distribute
Session Report
Automating tests- Reusing the existing code
© 2012 Citrix | Confidential – Do Not Distribute Citrix Confidential - Do Not Distribute
How the Automation Process Evolved….• Understanding Architecture• Execution on EN Setup
• Scheduler• Automatic analysis of test results• Auto e-Mail test Result
ANALYSIS
SUPPORT FOR NON-EN PLATFORMS
ENABLE UNICODE SUPPORT
OPTIMIZATION
• Execute Auto BVT on JA, FR, DE, ES, SC, RU languages
• Input non-ASCII characters as Input Data • (Multi Language Data)
© 2012 Citrix | Confidential – Do Not Distribute
• Make the scripts UTF enabled
• # -*- coding: utf-8 -*-
Citrix Confidential - Do Not Distribute
Enable the scripts to input non-ascii characters
© 2012 Citrix | Confidential – Do Not Distribute30
Multi Language data“ß証吞暴予禄 дю眀äöñé가앜刓訞一么@!23”
Results summary
© 2012 Citrix | Confidential – Do Not Distribute
More Functional bugs per release
32
If further colors are needed, please contact Creative Services
No. of EN issues
No. of I18N issues
No. of L10N issues
No. of Doc issues
0
20
40
60
80
100
120
Project K Project N
© 2012 Citrix | Confidential – Do Not Distribute
Early, Timely detection/fixing of Functional bugs
33
If further colors are needed, please contact Creative Services
Itr 1 Itr 2 Itr 3 UI Itr RC Itr0
20
40
60
80
100
120
Project K Project N
© 2012 Citrix | Confidential – Do Not Distribute
“It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change”
The crux of Express functional testing approach
© 2012 Citrix | Confidential – Do Not Distribute
Images Sources:• http://www.cleapublicrelations.com
• http://www.c3-ssi.com
• http://www.linkbuildr.com
• http://www.torwars.com
• http://www.branchpastor.com
• http://blog.workinsports.com
• http://www.121box.com
• http://www.march-hare.com
• http://www.larter.ca
References
Other Credits:
• Collègues: Rubina Gonsalves and Shivaraj Shet
• http://www.satisfice.com/sbtm/
• http://satisfice.com/articles/sbtm.pdf
Work better. Live better.
Anuj Magazine
http://anujmagazine.blogspot.com
Twitter: anujmagazine