end-to-end testing: the place to be - stickyminds · of software testing - a guide to the tmap®...
TRANSCRIPT
T2 Concurrent Session 5/5/2011 9:45 AM
"End-to-End Testing: The PLACE to Be"
Presented by:
Ruud Teunissen Polteq Test Services B.V.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Ruud Teunissen An international test consultant at POLTEQ Test Services BV, Ruud Teunissen has performed several test functions in a number of IT projects: tester, test specialist, test consultant, and test manager. Ruud participated in the development of the structured testing methodology TMap®—Test Management Approach. Together with Martin Pol and Erik van Veenendaal, Ruud is co-author of several books on structured testing, including Software Testing: A Guide to the TMap® Approach.
End-to-End TestingThe P.L.A.C.E. to be!
Ruud TeunissenPolteq Test Services BVq
The Netherlands
SDLC…
acceptance testrequirements
Focus on “single” department
wishes, laws,opportunities.
.
expectations
systemtest
unit andintegration
test code
design
systemintegration
test
Focus on “single” application
Focus on “single” interfaces
Focus on single department and its processes
Billing
Financial System
Supplies Logistics
Landscape – Yesterday, Today, …
Product & Services Admin.
Retail Stores
Billing System
User Applications
Context• Limited # of suppliers (own IT department)• Single environment• Limited # of interfaces
End-to-end testing?
• After system / system integration / acceptance test……we feel uncomfortable…
…we’re experiencing a 'broken chain' itch…p g
• Extended System (Integration) TestingF h fl f d
AppApp
– Focus on the flow of datathrough a (small) chain of applications
– Focus on integrated functional testingin a production like environment
Business & landscape evolve
• Business processes span across multiple applications• Boundaries between applications are blurring
• Integrated functional testing …what about ‘non functionals’?– Performance – 1+1=3– Security!– Usability?
BillingSBillingS
BanksBanks
BillingSBillingS
BanksBanks
• Example:– Customer passes away– Covered by one business process– Touching multiple apps
Product & Services Admin.
Product & Services Admin.
CustomersCustomers
User ApplicationsUser Applications
SystemSystem
Product & Services Admin.
Product & Services Admin.
CustomersCustomers
User ApplicationsUser Applications
SystemSystem
Business & landscape evolve
• We boldly go where no man has gone before……we’re going beyond interface testing & our
application!
• The flow of data becomes a transaction flow• The chain of applications becomes a chain of systems
End-to End Risk Analysis & TestingA small change in one component or system can have a major
impact on several business processes & systems
End-to-end testing – the seedling sprouts
E d t E d T t I tiEnd-to-End Test Innovations• Software Oriented Architecture• Sourcing• Cloud Computing
End-to-end test innovationsInnovation 1: Service Oriented Architecture
• Business processes implemented by means of services• Applications can be constructed quickly and differently
! Service is not necessarily designed for you
E2E testing to verify that the service is fit for purpose
! Changes happen frequent and unexpectedly
Every change in a service requires E2E regression testing
Service Oriented ArchitectureEnd-to-end testing required at different levels
‘Internal’ with ‘external’ sys\tems
On services that ‘form’ an application
external sys\tems
SOA systems with legacy systems
Applications withother applications
Service Oriented ArchitectureWeb services can be located anywhere
internet
• Functional and ‘non functional’ E2E testing!– security, interoperability, reliability, performance,
scalability
Functional and ‘non functional’ End-to-end testing required for acceptance testing and
continuous regression testing
End-to-end test innovationsInnovation 2: Sourcing (includes outsourcing)
• Software developed on various platforms atvarious locations by various suppliers
• Business process requires ALL software to work p qtogether!
! Acceptance at application level is insufficientE2E testing is required for acceptance
! Suppliers lack E2E testing capabilities! Suppliers lack E2E testing capabilitiesProvide E2E test cases to be included in ‘System testing’! Different rhythms of development
Each delivery requires E2E regression testing
End-to-end test innovationsInnovation 3: Cloud computing
• Software runs on various platforms at variouslocations and is accessed on various platforms from various locations
! How about your influence on interoperability, reliability, performance, scalability, security, …?
End-to-end testing required for ‘non functionals’
! Especially in combination with SOA and sourcing!E d t d t ti i d f ( ti l) t End-to-end testing required for (operational) acceptance
testing of cloud computing
Billing System
Banks
ExternalCompany
Financial System
Supplies Logistics
Landscape
Billing System
Banks
ExternalCompany
Financial System
Supplies Logistics
Product & Services Admin.
Customers
Products
Retail Stores
System
Tax Department
Product & Services Admin.
Customers
Products
Retail Stores
System
Tax Department
User Applications
Internet
Web browsers
Mailing system
User Applications
Internet
Web browsers
Mailing system
Challenges of successful End-to-end Testing
• Responsibility – Who “owns” the End-to-End chain?• Funding – Not part of any project…• Scope – Every application? Internal & External?
? “ ” ? ll b ?• Perspective – IT? “Any” customer? All business processes?
• Resources – testers and architects, developers, business, …
• End-to-End test management• End-to-End test environment• End to End test tooling• End-to-End test tooling• End-to-End test data• End-to-End test scripts
• Issue management
E2E testing: How to deal with it!
E2E
Test ManagementTest ScriptsTest Data
Business Processes
E2ETest EnvironmentTest ToolsIssue Management
acceptance test
systemIntegration
TestData
TestScripts
systemtest
unitintegration
test
Integrationtest Test
EnvironmentTestTools
E2E testing: Test Management
• Perspective – Scope – Funding – Not part of projects! Seperate funding is required…
E2E testing: Scope & Perspective
• Different view on E2E process – Same applications– Customer– Business– Financial Department– ……
E2E testing: Test Management
• Perspective – Scope – Funding – Not part of projects! Seperate funding is required…
• Stakeholders– Internal & External
• Communication• Overall planning• Knowledge
Operations
ITDept
Project AAcceptance
Business
E2E Test Manager
g– Resources– Acquire & Maintain
• Change control• Release Management
Project B
…
TD / Banks /…
E2E testing: Test Management
• E2E Risk Analysis– “Helicopter view” on business & application landscape– Define the critical chain of business processes and p
application flows– “What if” thinking helps – more than ever!
• Resources– Testers and architects, business analysts, developers,
hardware & network specialists, business people, et cetera
E2E testing: Test Scripts
• Starting point a.k.a. test basis– Business, its processes and the applications supporting it
• Test Scripts– Focus on overall “functionality”– Include “non-functionals”– Reuse existing scripts– High level in detail– Expected ‘in between’ results
to enable easy fault localizationto enable easy fault localization
– Designed for test automation
E2E testing: Test Scripts
• Actors – who – Customer, client, end-user, application management, …
• Process flow – how and what– Sequence, interfaces, online, batch, human actions, …
• Critical issues - why– Status of the client and/or system, products, complaints,
…
Test Design Techniques• Test Design Techniques– Classification Tree– Data Flow Test– Process / Data Cycle Test– State Transition Test– Decision Table Test
E2E testing: Test Environment
• Controlled• Applications, software, …
– “production like”p
E2E testing: Test Environment
• Virtualization to enable feasible, ‘cheap’ E2E test environments
E2E testing: Test Environment
• Stubs & Drivers– Well known, have been “in development” for ages– Now need to be implemented at a different levelp
• Applications & Systems• Internal & External
• Mocks– "How do I create tests for a Web Service when there is
no Web Service to write tests against?“g– Create a simulation or approximation of the Web Service
before the actual Web Service goes live
E2E testing: Test Data
• Production?• Production like!
• System Test• Acceptance Test
• Synchronisation– Time travel?
• Syntax• Semantics• Data pollution• Live cycle• Error handling• Traceability• Access• Security
• Test datamanagement
• Security• Timeliness• Speed• Regression
E2E testing: Test Tools
• Test automation to enable frequent execution of E2E regression test
• Test tools to enable automated (services) testing( ) g
• Standard interfaces to enable easy (services)testing in complex environments
wsdl, xsd
E2E testing: Issue Management
• Registering an issue is not enough!– You need to drill down in the E2E chain…
• Determine the root cause of the problemand
Define the required resolution
• Including “funding”• Taking care of all the
different roles & responsibilities
E2E testing: How to deal with it!
Business Processes
E2E
Test ManagementTest ScriptsTest Data
acceptance test
systemIntegration
TestData
TestScripts
E2ETest EnvironmentTest ToolsIssue Management
systemtest
unitintegration
test
Integrationtest Test
EnvironmentTestTools
Software is built by various suppliers, designed for various purposes, runs on various hardware platforms in various locations, is used by various clients, again, in various locations.
Cloud Internet
SaaS
E2E Testing in the current context:An integrated, production like, automated, continuous running, representative end-to-end test that covers the
full application landscape
‘Enablers’ for End-to-end testing
• Business process & application knowledge– “Helicopter view” on the business & application
landscape
• E2E Test management– Define the critical chain of business processes and
application flows and how to test it!
• Test environmentTest environment– The soil to grow in
• Test data management– As essential as water
• Test tools & automation– Too many plants to manually grow
Questions?
Ruud TeunissenPolteq Test Services BV – The Netherlands
[email protected]://www.polteq.com
About the speaker
Ruud TeunissenPolteq Test Services, The Netherlands
In the testing world since 1989, Ruud Teunissen has held numerous test functions in different organizations and projects: tester, test specialist, test consultant, test manager, etcetera. Ruud is co-author of Software Testing - A Guide to the TMap® Approach and is a frequent speaker at (inter)national conferences and workshops. He was a member of the
itt f Q lit W k E d program committee for Quality Week Europe and EuroSTAR. Ruud is currently manager operations and International Test Consultant at Polteq Test Services BV.
TMap® is a registered trademarks of Sogeti Nederland BV