webinar: going agile? don't forget about testing
TRANSCRIPT
September 25, 2015
Introduction to QASymphony for [INSERT COMPANY NAME]
#AgileTransformation
Agile Transformation: People, Process and Tools to Make
Your Transformation Successful
#AgileTransformation
The audio for this webinar is delivered through your computer. There is no dial-in number. Make sure your speakers are turned up or use a pair of headphones.
Going Agile? Don’t Forget About Testing
#AgileTransformation
HOW TO WATCH THIS WEBINAR
• Audio for this webinar is delivered through your computer. Make sure your speakers are turned up or use a set of headphones.
• If your audio quality is choppy, it could be your internet connection.
• You can customize your webinar viewing experience by increasing, decreasing or minimizing the size of the widgets on your screen.
• If you have questions, enter them in the widget on the right.
2
#AgileTransformation
• This webinar will be recorded and available on-demand tomorrow. You will get an email when it is available.
• Join the conversation on Twitter using the hashtag #AgileTransformation
• Use the Q&A widgets to ask questions during the webinar.
• At the end of the webinar, you will be asked to take a short survey.
HOUSEKEEPING
#AgileTransformation
NEXT WEBINAR
#AgileTransformationRobust test management platform purpose-built to help agile
teams accelerate software testing
ABOUT QASYMPHONY
#AgileTransformation
ABOUT GALLOP
Analyst Recognitions
Awards
Thought Leadership
Certifications
Partnership with Leading Tools
Vendor
2,000+ resources worldwide
FY2015-16 turnover of USD 85mn that grew at a CAGR of ~58% over
the last 3 years
Currently serving 80 of the Forbes
Global 2000 Companies
Project Cignificance aims to empower lives with free
and quality education
CSR Initiatives
Independent Testing Services
Company in North America
Publicly Listed & CMMi L5 certified
5 continents, 10+ countries
Global presence
500+ Satisfied
Customers
58%+ Revenue Growth
Gallop Thought Leadership
Total Quality Management
5 Patents*
Testing IP
400+ Testers in North America1200+ Testers at the Global Delivery
Center in India
Certifications : ISTQB, HP, IBM, CSQA, CEH, CISSP and more
Reduced Cost of Quality
*Pending
#AgileTransformation
OTHER WEBINARS & RESOURCES
WWW.QASYMPHONY.COM/RESOURCES
#AgileTransformation
PRESENTERS
Kevin DunneVP, Strategy and Business Development
QASymphony@KevinDunneQA
Sujit NairAssociate Director with the Enterprise Solutions Group
Gallop@Sujit_N
#AgileTransformation
WHY COMPANIES ARE GOING AGILE
There are a number of microeconomic and macroeconomic trends pushing teams towards adopting agile, some of the most prominent include:
App EconomyInstant feedback and
“needed it yesterday”mentality
Cloud AdoptionAbility (and expectation)
for frequent releases
Best of BreedIntegrating several tools,
increasing complexity
Startup CompetitionFast moving startups with
funding disrupting markets
Open SourceAbility to build quickly on
existing components
Digital DisruptionShift from traditional
channels to mobile & web platforms
#AgileTransformation
WHAT DOES AGILE LOOK LIKE?
Agile differs from Waterfall in creating shorter, more integrated cycles that generate code more quickly, allowing teams to react to changes as they come and reduce risk and waste
Source: http://www.kenwayconsulting.com/wp-content/uploads/2012/03/Agile-v-Waterfall.jpg
#AgileTransformation
Making a successful Agile transformation is not as simple as selecting an agile process and moving forward, it requires thoughtful planning across process, as well as people and tools
WHAT DO WE NEED TO CONSIDER?
Process
● What works well in my current process? What is not working well?
● What do I value - time to market, cost, quality, something else?
● What restrictions are in place to keep us from adopting agile? Which can we move?
People
● What valuable skills do my current employees have? Which are they lacking?
● Is my organization equipped to deal with uncertainty?
● Can my people adopt new skillsets needed by agile? Mindsets?
Tools
● What tools are critical to my process? Which can be replaced to assist in transformation?
● What new tools and categories of tools will be required?
● How can I empower smaller teams to use and leverage tools?
Thank You
#AgileTransformation
PROCESS
#AgileTransformation
WHAT AGILE MEANS FOR TESTING
Agile development means more frequent development cycles, often with more regular code delivery.
What does this mean for testing?1. Testers must be involved and aligned with the entire
development process2. Testing will happen much more regularly, in smaller
increments3. Testing will follow development more closely, in tandem
with coding
#AgileTransformation
HOW TO OPTIMIZE TESTING FOR AGILE
Get Testers Involved EarlyEmbed testers within product or scrum teams, making sure they have a seat at the table in
planning discussions and provide insights around quality and testing considerations
Embrace Frequent, if not Continuous, TestingAs agile promotes frequent code merges, builds, and deployments, employ a process that allows
the most vital testing to be identified and executed efficiently
Decrease the Time from Design to TestWith 2-3 agile sprints, we do not have weeks (or even days) to define and automate test cases,
adopt a process for manual or automated tests that allows tests to closely follow designs
#AgileTransformation
GALLOP & AGILE TESTINGS
DLC
Requirements Gathering
Application Design Development Testing
Test Requirement Analysis
Test Planning & Strategy Test Design Text
Execution
Analyse PlanDesign
BuildTest
Deploy
Analyse PlanDesign
BuildTest
Deploy
Peo
ple
Development Lead QA Lead
Project ManagerDevelopers Testers
Functional Automation
Developer ArchitectSCRUM Master
Sprint Team
Pro
cess
Tool
s
Product Owner
Pro
cess
Test Management Defect Management Automation
Service Test
High Focus on Compliance Focus on Outcomes
Process Heavy Project Based Delivery Lean Processes Product Based Delivery
Waterfall to Agile Transformation
#AgileTransformation
Quality Assurance to Quality Engineering – The Transition
Manual Testing Skills
Specific Testing Skills
Regression Automation
Performance Testing
Manual Test Cases
Black Box Testing
Prolonged Cycle Time
UI Testing
Standard Third Party Tool Sets
Transform to QE Skill-Set
Optimization by Cross Training
Early Functional Automation
Performance Engineering
Automated Test Case Generation
White Box Testing
Reduced execution cycles
Database, Services & Component Testing
Open Source Tools & Commercial Tools
Quality Assurance
(From)
Quality Engineering
(To)
Tran
sition
Courtesy by Lisa Crispin/Brian Marick
Agile Testing Quadrants
#AgileTransformation
GALLOP’s AGILE ASESSMENT FRAMEWORK
Discover and Document
Analyze and Benchmark
Roadmap & Implementation
Approach
Goals, Constraints, Timelines, Risks, Key
Focus Areas
Applications, Technology, Platform,
Automation Framework,
Automated scripts
Agile Processes, Metrics, Tools Strategy, Test
Assets, Skills, Org Map, Functional Testing, Test Environments, Test Data
Performance Goals, Scripts & Reports
Assess Performance Test Strategy, Performance
Engineering, Monitoring and related processes
Evaluate functional testing assets, maturity
of QA processes & benchmark based on agile maturity model
Analyze Tool Utilization, Licensing,
Integration opportunities
Review Test Automation Strategy, Framework maturity,
test scripts & coverage
Design the roadmap based on the target
operating model and prioritized key focus
areas
Identify & Prioritize the key focus areas
Review the analysis and findings with PoC & Key
Stakeholders
Design the Target Operating Model
Deliverable
s
Assessment Plan Test Assets Inventory
Target Operating Model
Agile QA Test Maturity
Implementation Roadmap
Interim Test Assessment Report
Final Test Assessment Report
#AgileTransformation
Discover and Document
Create Test Assets Inventory and Review with the POC
Elicit the existing processes, tools, test assets related to functional testing, test automation, performance testing and also the team skills to understand the AS-IS state and provide comprehensive documentation
Understand and Document Test Assets
Understand Organization Setup and Day in the life
of QA
Understand the existing processes
and test assets
Collect process and project artifacts
S.No Test Assets Inventory Overview Asset Type Details
1 Agile / SCRUM Guide Guidelines for working with SCRUM methodology, techniques
Process Docs 1
2 Test Management Tool Test Management Tool Tools 13 Automation Tool QTP Tool Overview Tools 14 Performance Testing Performance engineering scripts Non-functional Scripts 2005 Manual Test Cases for App1 Functional Test Cases for App1 Functional Test Case 12406 Manual Test Cases for App2 Functional Test Cases for App2 Functional Test Case 6737 Automated Test Scripts Automated Functional Test Scripts for App1 & 2 AutomatedTestScripts 350
8 Environments Environments - Dev, QA, Pre-Production, Production, Demo, Training
Environments 5
9 Test Org Roles Team structure and R&R Roles List 1
Organization Map Template
Test Assets InventoryTest Assets
Inventory Template
Resource Skills Inventory Template
#AgileTransformation
Analyze and Benchmark
Design Target Operating Model
Overall Agile Test Maturity
Customized Assessment Framework built using industry standard models such as SAFe, TPI, Agile & DevOps, ITIL, TMMi that helps to benchmark and design target operating model for transformation.
Conduct AssessmentMap to the Industry Benchmarks and Standards
Analyze the Test Assets Inventory from the
previous phase
Conduct interviews with
stakeholders
Analysis of defects and artifacts
Conduct gap analysis
Assessment Questionnaire
#AgileTransformation
Develop Roadmap
Deliverable
s
Based on the score, the recommendations are categorized into Quick wins, and Medium & Long Term initiatives
Design the roadmap aligned to client’s objectives, that helps in prioritizing the initiatives and balances the risks & impact, along with timelines
Focus Areas Reccomendations Deliverables Transformation Score Ease of Implementation
ROI
Agile Processes Define Key Agile ProcessesAgile FrameworkSCRUM Kit
Deploy in wave 1 6 2 3
Tools Integration Integrate JIRA with ZenDesk Integrate workflows in tools Deploy in wave 2 2 1 2
Agile Test CoEEstablish a Flexible Org. design with specialized and generalists roles in QA
TCoE Org. DesignTeams fully aligned to TCoE 3 1 3
Performance TestingEstablish Performance Capability for prioritized applications
Application Performance Analysis ReportPerformance Scripts
Deploy in wave 3 6 2 3
Enabler Establish effective Process feedbackProcess Improvement ProcessProcess Improvement guidelinesProcess Improvement template
Deploy Process ClinicsIdenitfy Process Implementation Champions
6 2 3
Test Management Tool Estabslish Test Management ToolTool Feasibility Analysis reportTool Implementation Guide
Deploy in wave 3 4 2 2
Test Automation Establish Test Automation Capability Automation Feasibility AnalysisTest Suite, Scripts, data is made ready
6 2 3
Analyze the recommendations based on ease of Implementation and ROI
#AgileTransformation
Case Study
Deliverable
s» Client is a leading provider of face to face marketing services
ranging from Pre-Event Planning, On-Site Execution, Extending the Experience to Audio Visual Solutions with a presence in North America, Europe and Asia
» Key Requirement: Functional, Regression and Compatibility Testing of four applications
» Application Environment: SharePoint and .NET Technology» Testing Types: Functional, Regression and Compatibility Testing
» Delayed receipt of user stories in every sprint» Lack of clarity in user story requirements» Parallel sprints, each with a rigid schedule» Lack of standardized QA processes and best practices» Unstable Test Environment with configuration and access issues» Critical defects leaked into post-production after regression
testing
Key Challenges
» Reduced RAI (Requirement Ambiguity Index) from 35% in sprint 1 to 12% in sprint 3 by improving the clarity of user stories
» Increased test coverage by leveraging Gallops Requirements Testing Framework
» Reduced post-production defects by 40%» Met aggressive sprint schedules with the help of reusable
process templates» Introduced industry standard Agile and QA best practices to
ensure that every sprint stays on schedule
Key Benefits» Leveraged Gallop Requirement Testing Framework to improve
requirements review process and hence the quality of user stories
» Mandated the use of sprint demos to hasten understanding of functionality being added in every sprint
» Worked with client team to resolve test environment issues» Introduced reusable process templates to accelerate and
standardize QA processes across applications and sprints» Published detailed test execution reports for the compatibility
matrix defined to enable easier and fast sprint sign-off
Solutions Provided
About the Client
Thank You
#AgileTransformation
TOOLS
#AgileTransformation
A NEW BREED OF TOOLS
Automation is undergoing even more dramatic changes than manual testing, with a new breed to tools:
● Open source: to provide lower cost and allow more members to contribute automation
● Cross Platform: to accommodate for multi channel experience across web, mobile, & desktop
● DevOps Capable: reliable, with ability to be run quickly and in parallel to allow for fast, frequent builds
#AgileTransformation
OPEN SOURCE AUTOMATION
Every category of automation tool is undergoing pressure from open source options:
Unit Testing
● jUnit● nUnit/xUnit
UI Testing
● Selenium● Protractor
Load/Performance
● jMeter● Gatling
Build
● Jenkins● Travis
Orchestration
● Puppet● Chef
Configuration Management
● Docker● Selenium Grid
#AgileTransformation
Early Automation Approach in Agile
Write Acceptance Test Cases
Decision to Automate
Test
Define Acceptance Test Criteria
Yes
Prototype Rinse & Refine Integrate Testing
Iterative design
Design/Update proto types for the Business
Components at UI and API level
Develop/Update the code, User Interface controls and
Unit test cases
Unit testing, integration testing and do bug fixes
No
User Story
Perform Manual Testing
Coordinate each other for developing Automation suite
Execute API level automation scripts on
continuous integration
environment till Acceptance criteria is
met
Execute end-to-end automation scripts on
developed UI
Design Pseudo codes/Classes for API
Level Automation
Build API Level automation scripts with negative and
positive scenarios
Execute API level automation scripts on every
component of developed code
Design Keywords for UI Level automation
Build automation scripts using Keywords for UI Level
automation
Capture UI Objects properties and integrate
with UI automation scripts
API Level Automation Testing
UI Level Automation Testing
Application Code Development
Perform goal based Exploratory testing
Adopt Hybrid Risk Based Testing for Test
optimization
Thank You
#AgileTransformation
PEOPLE
#AgileTransformation
MOVING FROM SILOS TO EMBEDDED TESTERS
Siloed Testers Embedded Testers
#AgileTransformation
A NEW BREED OF TESTER
Testers in the past were able to work independently, focusing on discrete issues with less collaboration
Agile testers do not have that luxury, they must be much more:1. Collaborative, as they might be a single tester working
inside a product team full of developers & analysts2. Adaptable, as they might be required to wear 4 or 5 hats
in any given day3. Well versed, as they may be reassigned to many different
projects within one organization
#AgileTransformation
HOW TO DEVELOP TESTERS FOR AGILE
Testers in Agile should focus on learning a broader set of skills, rather than going deep in one specific area:
Team Skills
● Collaborating in diverse teams
● Managing conflict across teams
● Hiring and retaining top talent in fast growing companies
● Creating a team culture of trust and buy in
Technology Skills
● Understanding and contributing to application code
● Understanding and contributing to automation (functional, unit testing)
● Expanding knowledge of additional testing types (performance, security, etc.)
● Expanding knowledge of build & release pipeline
Managerial Skills
● Motivating teams under high pressure
● Estimating and planning in agile organizations
● Managing agile project on both time and budget
#AgileTransformation
Q&A
Questions?
Thank You
#AgileTransformation
PLEASE TAKE OUR SHORT SURVEY
Thank you