Empowering Open Source Testing

FramewoksJaroslaw Kutylowski

Andegon GmbH, Germany

Europe’s Premier Software Testing EventStockholmsmässan, Sweden

WWW.EUROSTARCONFERENCES.COM

“Testing For Real, Testing For Now”

Contact: andagon GmbH Augustinusstr. 11d50226 Frechen

Tel.: +49 (0)2234 - 998 000 00 info@andagon.com Fax: +49 (0)2234 - 998 001 www.andagon.com

Empowering Open-Source Testing Frameworks

… by combining them with commercial testmanagement

Jaroslaw Kutylowski at EuroSTAR 2009

Why consider this combination?

Project with established open-source test automation• Started small in large corporate• Agile development, high amount of automated testing• Tools chosen to fit technically to automation task

Then• Project data becomes important & gains visibility• QA data becomes important• Cost efficiency is important

Test automation alone isn‘t sufficient anymore

Situation analysis

Why open-source was fitting the task?

• Instant availability (no procurement, just install …)

• Test automation tools excellent for their job(JMeter, Selenium, FitNesse)

• Higher acceptance of open source with technical staff (partially for cultural reasons)

• Possibility to extend the tool

• Cost efficiency

What open-source lacked?

Test automation(purely technical) SUT

Test abstraction

(e.g. FitNesse,

Cucumber)

Test management

open-source test automation tools

Missing in used open-source tools

Abstraction from technical details of test automation

• Binding tests to business processes and requirements

• Access to high-level information for analysts and management

• Common interface for high-level information on

• different types of tests (functional, performance)

• different automation tools

• User-friendly GUI

Great effort put into abstraction in tools like FitNesse & Cucumberbut that‘s still too technical

What open-source lacked?

What open-source lacked?

Management of the test lifecycle

• Versioning of tests and test results, history

• Structuring of tests into logical compounds

• Integration with ALM tools for release management

Reporting

• Easy analysis of test results (especially performance)

• Automatic versioning of historic test results

• Management reporting

• Different abstraction levels necessary (project managers, top management …)

• Reporting bound to high-level objects (requirements, …)

What open-source lacked?

What commercial could provide additionally?

• Focus on test management not on automation

• Vendor at hand for customization (e.g. reporting)

• Vendor guarantees for operational reliability

• Accessibility, ease of use

Vendor is primary contact for management

Open-source tools remain focused on technical aspects

Technical teams care for tool hosting & customization

Options for empowering open-source tools

Rollout scenario – options

Extend the open source test tools with custom management functionalities

Only viable if:• Company is willing to invest in own solution• Careful consideration of costs is necessary• Risk in investing in own development project must be calculated

Pro Contra

Solution tailored to one’s needs

Availability of resources, know-how

No vendor lock-in Cost-efficiency?

Can evolve over time Time-efficiency

Switch to a commercial automation tool with management functionalities

Only viable if:• Commercial tool for automation & management can be found• Script migration is usually only possible for small projects

Pro Contra

One integrated solution Migration of all test scripts

Vendor lock-in

Rollout scenario – options

Combine existing open-source tools with a management suite

Only viable if:• Supply chain and company policies support the mix• Appropriate test management tool is found

Pro Contra

Best-of-breed solution foreach category

Can be well integrated

Significantly less vendor-lock in

Rollout scenario – options

Technical aspects of integration

Integration concept

• Test automation definition remains defined and executed in open-source tool

• Management information, i.e. requirements, test description, test results, release information in commercial tool

• Two possible GUI concepts

• Test automation integrated in test management

• Test automation contains hooks for test management

Integrating open-source tools

Easier than integrating commercial tools

• Source code available

• Usually loose licensing model, changes don‘t have to go back to community

• Integration with GUI changes inside of test automation tool possible

• Technical users work with usual test automation GUI

• GUI provides additional functions for connecting with test management

Integrating open-source tools

Obstacles

• Good GUI integration necessary

• Different GUI types – web-based, desktop application

• Different technologies and programming languages

• Partially no documented interfaces for execution/results

Complex data export

Problematic error handling

Integrating open-source tools

Summary

Benefits of combined solution

• Higher acceptance of tools among all users (technical staff & management)

• Easy migration paths

• Supports small and large teams

• Cost efficient

Technical integration

• Large possibilities due to open code, though large investments necessary

Thank you for your attention!

Questions?