Software testing research

Ossi TaipaleNovember 2011Lappeenranta University of Tech.

Contents

Background Research process Research methods Research projects, research objectives and

results: Basic research of software testing, ANTI-

project Reference model of software testing,

MASTO-project Software testing in the cloud, STaaS-

project Intended quality, STX-project

Research co-operation

Background, why testing

During the last decades productivity of Finland has been higher than in other EU countries, but now the productivity in the product and service sectors is slowing down meaning lower growth rate of GDP.

The key to increase productivity is in the

hands of industry and service sectors that produce or apply ICT.

Productivity can be increased by creating tangible products or service products.

Testing enables product creation.

Background, software testing research

Top-down approach:

1. Years 2004–2007 ANTI-project: Basic research of software testing, affecting factors.

2. Years 2008–2011 MASTO-project: Software testing strategy and management. Theses: Software Testing Strategy and Management and Software Testing in the Cloud. The reference model of software testing ISO/IEC 29119 Software Testing Standard. 1. Vocabulary, 2. Testing Processes, 3. Reporting. Part 4. Methods is under development. Ossi Taipale is a member of the WG26.

3. Years 2011-2014, new STX-project: Testing level and ”Testing Spice”.

Software testing research projects

Top down approach in the research projects:

ANTI-project: Basic researchof SW testing, 2004 - 2007

MASTO-project: Reference modelof SW testing, 2008 - 2011

STX-project: Intendedquality,2011 - 2014

ISO/IEC 29119: SW testing standard

ISO/IEC and IEEE :29119 and Testing Spice,15504, IEEE1012,25010

SW testing in the cloud

Background, software testing definition

We use the definition: Software testing consists of verification and validation.

By testing, we try to answer two questions: are we building the product right and are we building the right product?

Verification answers the question: are we building the product right? Basic verification methods include inspections, walkthroughs, and technical reviews. Checklists are used as the tools of verification.

Validation answers the question: are we building the right product? Validation activities can be divided into unit testing, integration testing, usability testing, function testing, system testing, and acceptance testing.

Research process

Ontology Beliefs of the essence of phenomena under investigation

Selection of standards and constructs: e.g. ISO/IEC 12207, 15504 and 29119

Research philosophy

EpistemologyBeliefs of the suitable method

Concrete selections of the study: Delphi, survey, and grounded theory methods

Empirical studyBackground

assumptions

Research methods

Mixed methods approach consisting of quantitative and qualitative analyses (Hirschheim 1985).

The results of the quantitative and qualitative analyses are triangulated to increase the validity of the study. The principle of triangulation means that more than one method, observer or data set is used in a study to complement each other and to verify the findings (Denzin 1978).

Research methods

The objective of the Delphi method is to achieve the most reliable consensus of opinion of a group of experts. Delphi method can be used in finding good arguments about an ongoing process (Directing the study).

The survey method is used to gather information about feelings, motivations, plans, and beliefs. Tools for gathering such information are usually questionnaires or interviews (Fink & Kosecoff 1985).

Grounded theory (Strauss & Corbin 1990). The objective of a qualitative study is rather to find or reveal facts than to prove existing theorems. Strauss and Corbin (1990) define qualitative research as any kind of research that produces findings not arrived at by means of statistical procedures or other means of quantification.

Some results of the ANTI-project (2004-2007)

Hypotheses for improvement, process view:• Testing ought to be adjusted according to the business

orientation of the OU. Product oriented organizations should adopt a formal planned testing process. Service oriented organizations should adopt a flexible testing process that allows appropriate coordination with their clients.

• Enhanced testability of software components. Consider testability as an important factor in the selection of components. Review the testing process of your suppliers.

• Efficient communication and interaction between development and testing seems to reduce costs and improve quality. Product oriented OUs could develop straightforward processes because knowledge transfer is predictable. Service oriented OUs could develop innovative solutions because knowledge transfer varies according to the customer.

• The early involvement of testing seems to reduce costs and improve quality. The planning of testing is more straightforward in product oriented than in service oriented OUs.

• The risk-based testing strategy helps in avoiding ad hoc decisions on testing, because the decision on what to test is based on a predefined testing strategy.

Some results of the ANTI-project (2004-2007)

Hypotheses for improvement, knowledge management view:

• Business orientation affects the testing organization. In product oriented OUs, the organization model ought to support repetitive testing tasks that enable development of a deeper expertise. In service oriented OUs, the organization model ought to support adaptation to the processes of the customers that demand broader expertise.

• Business orientation affects the knowledge management strategy. OUs ought to adjust their knowledge management strategy according to the business orientation.

• Business orientation and the knowledge management strategy affect outsourcing. Codification of knowledge enables testing outsourcing.

• Identifying and avoiding barriers and using enablers improve knowledge transfer.

Some results of the ANTI-project (2004-2007)

Associations between testing outsourcing and knowledge management:

• Outsourcing seems to be more effective when independent testing agencies have enough domain knowledge.

• Outsourcing verification tasks is more difficult than outsourcing validation tasks.

Publications of the ANTI-project (2004-2007)

1. Finding and Ranking Research Directions for Software Testing, EuroSPI, Budapest, Taipale, O., K. Smolander, H. Kälviäinen. 2. Cost Reduction and Quality Improvement in Software Testing, SQM, Southampton, UK, Taipale, O., K. Smolander, H. Kälviäinen. 3. A Survey on Software Testing, SPICE Conference , Luxembourg, SPICE ,Taipale, O., K. Smolander, H. Kälviäinen (2006). 4. Factors Affecting Software Testing Time Schedule, ASWEC, Sydney, Australia, Taipale, O., K. Smolander, H. Kälviäinen (2006). 5. Improving Software Testing by Observing Practice, ISESE, Rio de Janeiro, Brazil, Taipale, O., K. Smolander. 6. Observing Software Testing Practice from the Viewpoint of Organizations and Knowledge Management, ESEM, Madrid, Spain, Taipale, O., K. Karhu, K. Smolander. 7. Triangulating Testing Schedule Over-runs from Knowledge Transfer Viewpoint, Lappeenranta University of Technology, Research Report 104, Taipale, O., K. Karhu, K. Smolander . 8. Outsourcing and Knowledge Management in Software Testing, EASE, Staffordshire, UK, K. Karhu, O. Taipale, K. Smolander.

Jussi Kasurinen

Overview on MASTO-project (2008-2011)

MASTO

The project was a 3-year academic joint project conducted by LUT and Aalto University, funded by Tekes (Technology and Innovation Agency of Finland) and a group of companies and organizations.

The focus on the project was on software testing in different organizational levels. MASTO at LUT focused on test

process and organizational aspects.

Aalto focused on the testing work and project-level aspects.

Researcher exchange, Jussi Kasurinen – Lund, Sweden, Leah Riungu-Kalliosaari – Limerick, Ireland.

Software-producing organization

ProjecesePof these

Organizational level, one

company has one set of these

Project-level work, one company may

have several of these

Collection of qualitative and quantitative data, 65 interviews

Round: Type of interview

Interviewee role in the organization

Interview themes

1st round: Semi-structured interview, 12 OUs

Software designer or people responsible for software design and architecture.

Design and development methods, Testing strategy and -methods, Agile methods, Standards, Outsourcing, Perceived quality

2nd round: Structured survey, additional semi-structured questions, 31 OUs, including 1st.

Manager, test manager or project leader responsible for development project or test process of a product.

Test processes and tools, Customer participation, Quality and Customer, Software Quality, Testing methods and -resources

3rd round: Semi-structured interview, 12 OUs, same as 1st.

Tester or people responsible for doing testing in the development project.

Testing methods, Testing strategy and –resources, Agile methods, Standards, Outsourcing, Test automation and –services, Test tools, Perceived quality Customer in testing

4th round: Semi-structured interview, 10 OUs, some new.

Manager, test manager or project leader responsible for development project or test process of a product.

Test policies, Test strategies, Test plans, Testing work, Software architecture, Delivery models, New software development concepts

ISO/IEC 29119 Test process

Test strategy constructs (and layers)

Journal article “Software Test Automation in Practice: Empirical Observations”

Conference paper: “Exploring Quality Concepts in Software Organizations”

Conference paper: “Test Case Selection and Prioritization: Risk or Design-based?”

Conference paper “Analysis of Problems in Testing Practices”

Conference paper: “A Study on Agility and Testing Processes in Software

Organizations”

Covered in article “Software Test Automation in Practice: Empirical

Observations”

Conference paper “How Test Organizations Adopt New Testing Practices and

Methods?”

Conference paper “A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC 29119 Test

Standard”

Testing standard and publications

According to ISO/IEC 25010 (Software product quality), the quality in the software product is a composition of several quality attributes

These quality attributes define the quality objectives for software

Software Product Quality

Stability

Reliability

Efficiency

Ope

rabi

lity

Security

Compa

tibilit

yM

aintainability

Trans

fera

bility

Quality and quality objectives

• All at least somewhat important

• Only in 9 out of 248 (3,6%) assessments “not important”

• Perceived quality, not absolute quality!

Quality characteristics

• Trust between the customer and developer.– Specifications are distributed freely, things are discussed and

more information is available if needed.

• Existing conformance with the ISO/IEC 29119 model concepts.– Workflow has a feedback system which is used, plans and made

decisions are assessed and changes can be made.

• Elaboration of quality – Everyone in the organization knows what kind of quality is

preferred and tries to achieve that.

Important things that affect perceived quality (Survey 2009)

• End-product criticality– Obviously a rocket control system is tested more thoroughly

than a mobile game, but preferred quality comes from the domain, not criticality.

• Development method– An agile product can be as high quality as a “traditionally” made

product.– Open source is not necessarily better (or worse) than closed

software.

• Outsourcing– In large organizations, outsourcing does not seem to affect

perceived quality.– In smaller organizations it may, but not always.

Things that do not affect the perceived quality that much even if they seem to have

a big influence

• Organizations do not tend to try out new ideas.• Sporadic development is done when the inconveniences

overcome acceptable losses.• Even if the test process feedback is collected, it is often

neglected if the process is “good enough”.

How do the organizations develop their test process

Combination of Test Improvement model (TIM) maturity levels and ISO/IEC 29119 processes.

Two results: General maturity and conformance with the standard model. Process improvement objectives to develop test process.

Processes from ISO/IEC 29119

Maturity levels from TIM

Individual assessment of each process area, development ideasGeneral

maturity/conformance estimation

Self assessment framework

Publications from the MASTO project (2008 -2011)

1. Test Case Selection and Prioritization: Risk-Based orDesign-Based? Jussi Kasurinen, Ossi Taipale and Kari Smolander, ESEM2. A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC 29119 Test Standard, Jussi Kasurinen, Per Runeson, Leah Riungu and Kari Smolander, EuroSPI3. Software Test Automation in Practice: EmpiricalObservations, Jussi Kasurinen, Ossi Taipale, Kari Smolander, AiSE4. Exploring Perceived Quality in Software Organizations, Jussi Kasurinen, Ossi Taipale, Jari Vanhanen and Kari Smolander, IEEE5. Analysis of Problems in Testing Practices, Jussi Kasurinen, Ossi Taipale and Kari Smolander, APSEC6. A Study on Agility and Testing Processes in Software Organizations, Vesa Kettunen, Jussi Kasurinen, Ossi Taipale, and Kari Smolander, ISSTA7. How Test Organizations Adopt New Testing Practices and Methods? Jussi Kasurinen, Ossi Taipale and Kari Smolander, TAICPART8. Exploring the Perceived End-Product Quality in Software-Developing Organizations, accepted for publication in International Journal of Information System Modeling and Design, IGI Global, Jussi Kasurinen, Ossi Taipale, Jari Vanhanen and Kari Smolander.

Software Testing in the Cloud (2010-2013)

Leah Riungu-Kalliosaari

Objectives and background

Empirical study aimed at understanding how organizations can successfully use the cloud for testing

Cloud computing: A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, servers, storage, application and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction (NIST definition).

STaaS: A model for software testing whereby testing of an application is provided as a service across the internet.

Cloud-based testing is provided on-demand and billed on pay-per-use basis, so that the user pays only for the resources they have used.

Initial observations

The trend towards cloud services is growing Traditional desktop applications online software services

e.g. SaaS, StaaS According to a 2009 Gartner report, cloud computing

changes the way software is delivered and used. This change means that traditional licensing of software seems to loose ground where as hiring of software services seems to be on the rise.

Mixed reactions Generally positive Positive – due to e.g. cost reduction, flexibility,

performance, access to global markets Negative – due to concerns regarding e.g. security, domain

knowledge, test data management Neutral – mainly explorative approach

Implications on Testing

With the cloud becoming more common, the quality expectations are growing.

New issues and complexities surrounding these services will need to be addressed. Testing will become more important Need for testing increases

Testing is seen as arena of cloud computing where the barriers to entry are relatively low. Testing in the cloud simply entails the use of computing resources and models in the cloud to perform testing. Cloud-based testing is provided on-demand and billed on pay-per-use basis, so that the user pays only for the resources they have used.

1a. SaaS software

3. Testing the cloud

2. Testing environments in the cloud

1b. Non-SaaS software

1. The system or application under test is available online

2. Testing infrastructure and platforms are hosted in the cloud (Including crowdsourcing/Human as a Service-(Haas))

3. Testing of the cloud itselfFacets of testing in the cloud

Testing in the Cloud

A roadmap towards testing in the cloud

Develop understanding of cloud computing Cloud computing is increasingly becoming a feasible choice for testing.

Carry out pilot projectsWe recommend that organizations carry out pilot projects that enable them to fully explore the potential benefits.

Come up with elaborate strategiesCloud testing vendors as well as testing and quality assurance consulting firms will be called upon to offer advice and direction.

Enhance team interaction and prepare for complexitiesOrganizations need to be prepared for additional testing brought about by the complexities and new requirements for cloud-based applications and systems.

Enhance co-operation between research and industryThese include, among others, application issues e.g. the types of applications best suited for testing in the cloud; management issues like how to organize the human resources for cloud-based testing (e.g. crowdsourcing); and legal and financial issues such as the management of test data across different global jurisdictions and how to device appropriate pricing models.

Current research problem

To better understand how cloud based services can be achieved with intended quality Identify the most important quality requirements for cloud-

based services Provide recommendations for achieving overall intended

quality of cloud-based services Difference between intended and achieved quality What are the tradeoffs, and what is the optimal balance.

Publications from the Testing in Cloud – project (2010-2013)

1. Research Issues for Software Testing in the Cloud, Leah Muthoni Riungu, Ossi Taipale, Kari Smolander, 2nd IEEE International Conference on Cloud Computing Technology and Science.

2. Software Testing as an Online Service: Observationsfrom Practice, Leah Muthoni Riungu, Ossi Taipale, Kari Smolander, Third International Conference on Software Testing, Verification, and Validation Workshops.

3. Testing in the Cloud: Exploring the Practice, IEEE Software, Leah Riungu-Kalliosaari, Ossi Taipale, Kari Smolander.

Software testing and development for intended quality, STX (2011-2014)

Tero Pesonen

Objective

To show how software development, software testing and intended quality depend on one another. Traditional software development and service models Emerging XaaS (Everything as a Service) architectures,

technologies and service models.

The project results help the participating companies in improving the efficiency of their quality management and software testing and hence the efficiency of their software development as a whole. Testing techniques Testing as a service

Research Problem

OU’s are evaluated through an assessment framework

Intended Software Quality

ISO/IEC 25010, Software Product Quality

Software Testing

ISO/IEC 29119, Testing Spice, IEEE Std 1012 StaaS

Software Development

•Software ProductsISO/IEC 12207ISO/IEC 15504•New ServicesSaaS

Research Problem The framework used as a ”grid” through which each OU

is assessed. Intended quality vs. achieved quality. Top-down and bottom-up analysis (testing work project

management organizational level). The assessment framework is defined through

international software engineering standards. We need to anchor the assessment to a specific, concrete

view on what quality, software testing and software development are.

Based on standards, the model remains coherent throughout the assessment.

Project results and usability; contribution to standardization work.

The results include improvement proposals for the OU’s.

Data collection methods include interviews and surveys.

Work division, entire project

3 year project 12 months / work package; WP 1 starts 8/2011

WP1: Requirements for intended quality Identifies the quality requirements and attributes that the

OU's perceive as key for achieving the intended quality. Defining intended quality, the importance of different

quality characteristics in achieving it, traditional vs. SaaS WP2: Building the intended quality

Achieved vs. Intended quality vs. the software development process

Action research WP3: Assessment

How the OU’s can improve their quality management and software testing to achieve the intended quality more efficiently?

Contents of work package (WP) 1

Study units are OU's. What are the quality requirements, and hence the quality

attributes, that the OU perceives as key for achieving the intended quality (level & characteristics)?

Are some quality attributes more important than others when determining whether the targeted quality has been achieved? Do OU's or projects differ here?

Does SaaS development emphasize different kind of targeted quality (level, characteristics) to traditional software development?

Data collection for the first qualitative study September – November / 2011.

Additional data from other projects also incorporated. Data analysis may produce first leads. Second data collection round will constitute quantitative

studies based on round 1 observations.

Publications from the STX project (2011-2014)

1. Observations on eBusiness implementation capabilities in heterogeneous business networks", 11th IFIP International Conference on e-Business, 2011, Kaunas, Lithuania Pesonen T., Smolander K.