Noelia Ortiz, MME, CSSGB, CQA

1

Agenda What is Cloud Computing?

Understanding the Responsibilities of the Test Manager

Performing the Risk Assessments

Testing Cloud

2

Part 1

3

What is Cloud Computing?

Is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

– NIST

4

Essential Characteristics On-demand self-service

Customers can configure computer facilities without human-interaction with the service supplier.

Services are easily available and can be obtained directly over the Internet.

5

Essential Characteristics Broad network access

Services are offered in a network.

Services can be obtained from different sources (e.g., PCs, laptops, tablets, mobile phones).

6

Essential Characteristics Resource pooling

Multiple customers share the supplier’s infrastructure through a rental model.

Computer resources: storage, computing capacity, memory, network bandwidth, and virtual environments.

7

Essential Characteristics Rapid elasticity

Services can be configured and released quickly and most of the time in an elastic fashion.

Offers the capability of quickly scaling up and down.

Customers can obtain services at any moment and in any desire quantity.

8

Essential Characteristics Measured service

Systems check and optimize the use of the underlying infrastructure.

The usage of the following is measured:

Storage Capacity

Computing Capacity

Bandwidth

Active user accounts

Provides transparency for the supplier and customer.

9

Benefits of Cloud Computing

10

Risks of Cloud Computing

CLOUD RISKS

Vendor Longevity

Support

Change Control

Security Privacy

Availability

Regulatory Compliance

Data Mobility

11

Types of Service Models

12

Types of Service Models Layers of the service growth model:

Hardware: Equipment (servers, network resources, storage).

Virtualization: Software that makes it possible to create multiple or different environments, based on the hardware. The software runs in an environment that is not physically present.

Platform: Runtime environment in which software can run (.NET, PHP, Apache)

Application: The software for the customer.

13

Types of Service Models No service purchased

In-House Outsourced

Application

Platform

Virtualization

Hardware

14

Types of Service Models IaaS

The customer gets access to computing power, storage, networks and other basic computer facilities and puts together their own infrastructure.

The customer does not manage the cloud but does decide which operating system run on it, the amount of storage, and the applications that are rolled out on it.

15

Types of Service Models IaaS

In-House Outsourced

Application

Platform

Virtualization

Hardware

16

Types of Service Models PaaS

The customer uses the programming languages and tools that are supported by the PaaS supplier.

The customer has no control or management over the infrastructure but does have complete control over the applications and some of the configuration of the platform environment and makes their choices for developing or buying them.

17

Types of Service Models PaaS

In-House Outsourced

Application

Platform

Virtualization

Hardware

18

Types of Service Models SaaS

Customer uses applications from a supplier that are running on cloud infrastructure.

These applications are accessible through a simple interface - such a web browser - for everybody, at any time, in any place and with any device.

The customer has limited influence to the configuration.

19

Types of Service Models SaaS

In-House Outsourced

Application

Platform

Virtualization

Hardware

20

Types of Service Models

21

Implementation Models

Public Cloud

Community Cloud

Private Cloud Hybrid Cloud

22

Implementation Models Private Cloud

Exclusively used by one customer.

The cloud can be managed by the customer or by an external party and can be located in-house or in an external location.

23

Implementation Models Community Cloud

Shared by a number of customers and supports a specific community that has shared interests (security requirements, policies, standards).

Management is done by the customers themselves or is outsourced.

The infrastructure can be located in-house at one of the customer’s premises or at an external location.

24

Implementation Models Public Cloud

The cloud infrastructure is publicity accessible.

The owner is the organization that provides the service.

Hybrid Cloud

Consists of two or more independently operating clouds (private, community, public) that are combined to make the exchange of data and applications possible.

25

Part 2

26

Risk Assessment With the different stakeholders, the chance of

something going wrong with the information system is discussed along with the impact this can have.

A large likelihood of failure with significant consequences represents a high risk and requires a lot of test effort.

27

Risk Assessment

System

ComponentRisk Group Risk

Classification

Failure Rate Impact Risk Category

Security

Performance

Manageability

Legislation and

regulation

28

Obtaining Agreement and Information from the Supplier At this stage, a lot of testing activities are deployed

from which information and support from the supplier is needed.

Additional agreement with the supplier in those cases where the supplier’s standard terms and conditions do not fully provide them.

Service specifications

Test environments

Guidelines for logging issues

29

Obtaining Agreement and Information from the Supplier Test Manager needs to watch out:

Where are the infrastructure configuration?

Where can the manuals be

found?

What are the guarantees on availability?

How is the customer

informed about changes in the

service?

How are issues logged?

What are the guarantees on

security?

30

Obtaining Agreement and Information from the Supplier Additional terms and conditions:

Supplier Audits (security or

process quality)

Support with implementing

the service

Periodic supplier evaluations and

reviews of service

31

Test Manager Responsibilities Selection

To determine the suitability of a service the test manager should perform the following:

Put forward cloud related aspects

Determine completeness and controllability of the selection criteria

Assess services and suppliers

Issue a selection advice

32

Test Manager Responsibilities Implementation

Implementation consists of all activities that are needed to take the service into production.

The steps for the implementation consist on the following:

Executing the risk assessment

Setting up the test strategy

Executing the test measure

33

Test Manager Responsibilities Production

The tasks of the test manager continues after a successful implementation.

The test manager is responsible to perform the following tasks once the Cloud is in production:

Ensure continuity as changes are implemented

Monitor the quality of the service

Review the original selection criteria

34

Testing with the help of Cloud Many testing services are offered under the name

“Testing as a Service” (TaaS).

TaaS = Test outsourcing

Test outsourcing can be handled with the use of the “Test Outsourcing Governance Approach (TOGA®).

35

Part 3

36

Performance Risks With cloud computing, more communication is done

over the Internet, which makes performance risk more significant.

37

Performance Risks Example:

Performance of a travel-booking website that is hosted in the cloud

What if the clients have to wait too long to book a flight?

An important performance risk would be the Internet connection.

38

Performance RisksRisks

Response times result in problems Performance decreases due to changes by supplier

Processing capacity (throughput) is insufficient

Scaling (up/down) does not suffice

Upload/download speed (bandwidth) is insufficient

User experiences quality of video or audio as low

Other customers affect performance User requirements for performance are not clear

Performance is not sufficient on all types of devices

Executing performance tests is not possible

Performance varies due to Internetconnection

Performance test are not representative

Performance deteriorate over time User demands change over time39

Security Risks There is a lot of regulation on information security and

privacy.

Sharing IT resources with other customers produces security issues.

One of the advantages of the cloud is that the services can be obtained everywhere and on all kinds of platforms. This results in a new vulnerability.

40

Security RisksRisks

Building insufficiently protected against break-in

Service is insufficiently robust against attacks by hackers

Authentication is insufficient:-Other customers gain access-Unauthorized people gain access-Customer gains access to other customers’ data

Data is lost:-Storage device error-Scaling-Procedural errors-Software errors

Authorization is insufficient:-Administrators on the supplier’s are not sufficiently restricted from accessing client data

No access to data because of a business incident on the supplier side

There are too many people that can access everything (on the supplier side)

Security is not up to date

41

Availability and Continuity Risks When a service is not available at a given moment, it

immediately has a disruptive effect on the business processes that use it. (i.e. Losing Internet connection, a device error in the service itself)

Depending on the risk, specific test measures are taken for risk mitigation.

Dependency on the Internet is an important risk.

42

Availability and Continuity RisksRisks

Connection to the Internet is disrupted:-At the supplier-At the interfacing systems (customer)-At the user devices (customer)

Bankruptcy of the service supplier threatens the continuity of business processes

The Internet connection is disrupted at other locations around the world

There is no backup plan

The business process is disrupted by problems with migration:-Missing data after migration-Data changed during migration-Transaction lost during migration

Responsibility for continuity failure is not clear because multiple suppliers are involved

Data has become unreadable:-Because of hardware failure-Because of loss of encryption key

There are insufficient agreements in place about availability

43

Functionality Risks Choosing a SaaS product is very similar like selecting

an “off-the-shelf” software.

Therefore, the risks are much like the risks found in selecting and implementing an “off-the-shelf” software.

A service often has to integrate with other systems. The risk is that the business processes do not function properly anymore.

44

Functionality RisksRisks

Service and business processes do not align:-Service does not meet all requirements of the business processes-Using the service results in the need for business process changes-Service provides insufficient means for configuration

Customization:-Is needed but not possible-Does not function properly on supplier side-Does not function properly on customer side-Is not robust when changes are not implemented

Quality of the service is inadequate(bugs)

Services do not match the technical infrastructure of the customer

Service is not according to the supplier’s description

Migration problems

Repairing user errors is not possible The manuals are inadequate

Configuration is not done correctly Changes are made to the service 45

Maintainability Risks The supplier performs different forms of service

maintenance. As a result, the customer has no control over maintenance.

When customers are not informed of changes to the service, it imposes a maintainability risk.

46

Maintainability RisksRisks

Documentation from supplier is no longer correct after the service is modified

There is insufficient support from the supplier’s help desk

Business process descriptions are not up to date

There are problems with repairing error situations

Test documentation from supplier is insufficient

Environments are not available for the following purposes:- Testing the service- Testing performance- Testing the E2E business processes

There is insufficient experience with cloud computing

IT landscape is not fully under in-house control

There is no change procedure

47

Legislation and Regulations Risks Cloud computing is a worldwide market that does not

consider national borders, and as a result resides under more than one jurisdiction.

The user of the services will have to obey different regulations.

48

Legislation and Regulations RisksRisks

Processing and storing data do not comply with some laws:-Laws in home country-Laws in other countries in which the data can reside

There is no agreement with the service provider about safe handling of data

Countries have conflicting laws Reliability of a regime in a country is questionable

Legislation is unclear Third parties gain access to company data by citation or investigation (jurisdiction)

It is not clear where data is stored, which causes uncertainty about legal risks

Legal issues arise when the service is down

Legislation changes49

Suppliers and Outsourcing Risks These risks mostly are related to supplier dependency.

It is important to determine up front what happens to the continuity of the service in the case of a business conflict.

50

Suppliers and Outsourcing RisksRisks

Bankruptcy of supplier threatens continuity of business process

There are multiple suppliers (multivendor) who shrink responsibilities toward each other

Supplier shut downs the service Supplier does not support the correct functioning of the service with test results

There is ambiguity in the contract about the following:-Continuity-Performance-Security-Defects-End of contract

There is no test set available to determine the correct functioning of the service

Customer is dependent on one supplier Service ownership is handed over to another party 51

Part 4

52

Test Measures In order to address the risk, test measures are needed.

Test Management

Test Specifications

Test Execution

Testing cloud services is a pick and mix of test measures depending on the risks.

The testing approach varies between service models (IaaS, PaaS, SaaS).

53

Test Measures The starting point is the outcome of the product risk

assessment of the service.

The result is a list of relevant risks with an assigned classification (high/medium/low).

This list aids the test manager to identify the necessary test measures.

54

Testing during selection The supplier selection process is divided into the

following steps:

Include cloud-related aspects

Determine completeness and controllability of selection criteria

Assess services and suppliers

Issue selection advise

55

Testing during selection Include cloud-related aspects

When the decision is made to go into the cloud, new risks are introduced.

This may lead to certain choices in the selection process such as choosing a private service rather than a public service.

The decision to not choose a cloud solution can be also be drawn after the risks are assessed.

56

Testing during selection Determine completeness and controllability of

selection criteria

Selecting a cloud service is very similar to selecting a software package.

Evaluate service capabilities

Evaluate the supplier

Criteria need to be defined for the selection process.

Criteria must be assessable or testable.

57

Testing during selection Assess services and suppliers

Selection criteria for a service

The required quality needs to be known.

How business requirements are translated into requirements and acceptance criteria for the cloud service?

Selection criteria for a supplier

Supplier properties need to be assessed.

To what extent is the supplier of a candidate service reliable?

58

Testing during selection Assess services and suppliers

Inspect specifications and terms.

Ask for references and certificates.

Perform audits and inspections.

Proof of concept (dynamic testing).

Simulate E2E business processes.

59

Testing during selection Issue selection advise

During the selection process, not all risks identified are related to IT.

Risks related to cost and legal can also be present.

60

Testing during selection Issue selection advise

During the selection advise the following questions are addressed:

What are the risks with implementing the intended cloud solution?

To what extent are business processes supported?

What are the risks in choosing a particular service?

How much customization is needed to implement this particular service and which risks occur as a result?

What are the risks in choosing a particular supplier?

61

End-to-End Testing (E2E) E2E is the broadest possible test in which the end

results are taken as a starting point.

Is a system test of more that one system.

Ensures data integrity is maintained between various system components and systems.

62

End-to-End Testing (E2E) System Integration Test vs. E2E

The E2E testing focuses on: Business processes

“System picture” (how the systems and services are connected and information flow)

Exact functionality of the systems and services

Integration Test E2E

Mainly focuses on interfaces between systems

Tests the total combination of systems and business processes

63

End-to-End Testing (E2E) Creating E2E test cases

Describe all test actions in the test case. Example: A test starts in a certain system and after passing through a

number of systems and services ends in another system with a certain result.

Determine the data that represents the test case and the expected result of the test case.

Determine the configuration and base data needed in all systems and services through which the E2E test case passes.

Describe the expected intermediate results of the test case on all interfaces between the systems and services.

64

Testing Performance Known performance tests could be applied to test the

cloud but not all of them.

For example, stress test is not possible in most cases as it jeopardizes the stability of the service for the other customers.

To determine whether a service meets the performance requirements, acceptance criteria need to be defined.

65

Testing Performance Load Test

The tester performs actions on the service as a user would and determines whether the service reacts with sufficient speed and whether errors do or do not occur.

Possible objectives to perform the load test. Average load (number of users)

Peak load

Many users logging simultaneously

Load during working hours

Increased load on the Internet

66

Testing Performance Load Test

The fact that the customers do not know about each other’s load on the service is a complicating factor.

The load on the service by another customer can lead to different metrics from the same test at different times.

It is recommended: Obtain information from the supplier about the load statistics

of other customers.

Execute the test on real time.

67

Testing Performance Stress Test

A stress test determines the behavior of a service beyond the peak load.

This test will show what happens when the service load increases more quickly or when a sudden, unexpected peak occurs.

The stress is conducted by increasing the load until the moment something happens.

68

Testing Performance Stress Test

During the stress test execution, the tester needs to stay within the service’s terms of use to avoid impacting other customers.

It is recommended to test the system at the edge of the guaranteed load to determine what would happen with the service.

69

Testing Performance Endurance Test/Volume Test

Some of the causes for performance decreasing can often result from memory overflow (long files, memory leaks) or files becoming fragmented.

After a certain volume is processed, performance drops or stops completely.

This test is performed by inputting as much volume as possible in a short period of time to get results quickly.

70

Testing Performance Endurance Test/Volume Test

This testing approach brings risks as customer can easily move beyond the officially permitted load.

71

Testing Performance Testing Elasticity and Manual Scalability

Testing elasticity is a new aspect of performance testing.

The objectives of these tests are to determine whether the performance of the service meets the requirements across the entire load spectrum and whether the appointed service capacity scales with the service load.

72

Testing Performance Testing Elasticity and Manual Scalability

This test consists on executing a load test with the load increasing beyond the scaling boundary (scaling up) and then the load decreasing below the scaling boundary (scaling down).

The following errors could possibly emerge in testing elasticity: Automatic scaling up does not perform as required

Automatic scaling down does not perform as required

During scaling functional problems occur

73

Testing Security Security is predominantly about information security.

ISO 27001 empathizes three aspects of information security:

Data confidentiality

Integrity of data

Availability of data

74

Testing Security Testing Encryption

This test consists on verifying whether or not encryption is activated.

Most modern systems tests tools are able to test messages with and without encryption, and by comparing them, can determine if the data stream is encrypted.

75

Testing Security Testing Authorization

Functional tests are used to test authorization.

The test basis is an authorization table in which it is shown what someone with a specific role is and is not allowed to do.

User GroupAdding Users

Deleting Users

Manage Projects

Filling In Time

Sheets

Prove Time Sheets

Generate Monthly

Totals

Administration X X X X X X

ProjectManager

X X X X X

Standard User X X

76

Testing Security Testing Authorization

Test the cells by performing the actions with the specific role and checking whether it is or is not possible.

Try to gain access to data that only people with the highest authorization are permitted to access.

Try to gain access to data that only people with administrator permission are permitted to access.

77

Testing Security Testing security robustness against Internet attacks

Security risks are often exposed by hackers through directed attacks. Directory traversal – Read and/or write in directories other than

those allowed.

XML external entity attack – Include extra (bad) data in an XML file

SQL injection – Request and/or change data by manipulating SQL queries.

Testing robustness against attacks requires in-depth knowledge of and experience with test tools.

78

Testing Security Testing log files and audit trails

An important security measure is to track all changes in data and attempts to access data.

Testing is done using functional tests.

Execute changes and check whether they are registered in the log files or audit trail.

79

Testing Security Testing security patch routines

Security systems and other software need to be up to date with the latest security patches.

Testing these routines requires checking whether the proper procedures are in place and whether there is evidence that these procedures are followed.

80

Testing for Manageability The essence of manageability is how to keep the

service and the processes around it working together.

Manageability and maintainability are closely related and is about the ease that a service can be modified to keep it in production.

81

Testing for Manageability Manageability is tested using a checklist.

Manageability checks for documentation are as follows: Is documentation available?

Is documentation kept up to date?

Are changed versions sent to the current stakeholders?

Various specifications are part of manageability and are subject to testing.

82

Testing for Manageability Specifications on the supplier side

Interface specifications

Specifications for customer resources

Platform specifications

Infrastructure specifications

Specifications on the customer side Infrastructure specifications (IaaS)

Systems specifications

Architecture documentation

83

Testing for Manageability User documentation

An instruction manual is needed.

Documentation is needed on how the service needs to be configured, used, and maintained by the customer.

Test environments availability

It is important to have the availability of test environment in order to test the service before it goes live.

84

Testing for Manageability Test documentation

There are two categories:

Customer test documentation

Supplier test documentation

Test documentation is mostly used, for example, to execute regression testing and retesting for patches and infrastructural changes.

85

Testing for Manageability Incident management procedure

Preferably the supplier has an incident management procedure in which both the supplier and customer agree if and when the incident is resolved.

Certain incidents are resolved by the customer.

For example, changing the service configurations.

86

Testing for Manageability Change procedure and version control

The customer needs to set up a change control procedure.

Some changes can be performed by the customers themselves and others the supplier needs to be involved.

Continuous integration requires well-organized version control at the supplier side as well as at the customer side.

87

Testing for Manageability Change procedure and version control

The supplier is responsible to notify the customer which versions of documentation are active for which version of the service.

The customer needs to control the version of all interfacing systems along with all documentation.

It is important to know which software version and which test basis version is used.

88

Testing for Manageability Maintainability of software

For IaaS and PaaS, the customer is responsible for the maintainability of the software.

Poor maintainability results in recurring errors and parts of the service not functioning after changes and patches.

89

Testing for Availability/Continuity Availability is about a part of the IT landscape (such as

a service).

Continuity is about the process that uses it.

The continuity of a service is key:

How often does a disruption occur?

How fast it is resolved?

What damage has this disruption caused?

90

Testing for Availability/Continuity Testing failover

When executing a failover test, the worst-case scenario is assumed.

The failover test shows that the interruption of a service stays within acceptable or agreed-upon boundaries.

The service needs to be fully running to mimic a realistic situation.

91

Testing for Availability/Continuity Testing failover

For a failover test to be successful a number of things need to work properly:

IP addresses are switched

Load balancers stop trying to connect to the failed process

Backups are restored

Machines can be restarted after a power cut

Networks are up and running after a failure

Data is not lost or corrupted

92

Testing for Availability/Continuity Testing failover

Checklist specifically for SaaS continuity after failover switch:

Has the configuration been disturbed?

Is the failure even noticed?

Does the automatic failover start to work?

Are there any transactions lost?

Is there any data lost?

Does the audit trail work properly?

Is performance back to normal?

93

Functionality Testing Some of the functional test objectives are:

Does the service fit the business processes?

Do the business processes fit the service?

Is the service quality sufficient?

Is the service user friendly?

Is the service configuration done correctly?

Do interfaces between the service and other systems work properly?

Does everything work after changes?

94

Functionality Testing Testing service quality

The customer can perform an acceptance test on the service to get an impression of service quality.

Test scenarios are designed and executed for the most important processes for which the service will be used.

95

Functionality Testing Testing user-friendliness

How many screens does a user have to go through to make an appointment?

Is the tab order intuitive? Is there sufficient help functionality?

These can be tested implicitly or explicitly in the functional tests activities.

A manual or work instruction can act as a test basis and at the same time be a test object.

End users can work with the service in the development environment to learn how it works.

96

Functionality Testing Testing interfaces to other systems

Four aspects are differentiated when testing interfaces: Technique – How is data exchange organized physically?

Syntax – How is data structured and what are the syntax rules?

Semantics – Are uploaded or downloaded data processed correctly by the receiving service or receiving system?

Nonfunctional aspects (security, performance) – Any problem in these areas?

97

Functionality Testing Testing service configuration

Several things can go wrong with the configuration:

Design of the configuration is incorrect

Configuration errors

Service configuration does not work properly

Peer review is the most common method to verify the configuration.

Configuration can be tested using a functional test.

98

Functionality Testing Multi-platform testing

One of the advantages working in the cloud is the ability to access the service using different resources (laptop, tablet, mobile).

A complete functional test of the platform is not feasible.

Choices can be made based on:

What platforms and versions are used the most?

New platforms

Focus on the most-used paths through the service

99

Functionality Testing Testing for regression

When something changes in the IT landscape (the service or something else) there is always the chance that not everything will work correctly.

When changes are performed, it is important to determine where and how extensive the regression testing should be. A risk analysis will determine the depth of the testing.

Regression testing is most likely to be performed on functionality.

100

Testing Migration Different migration scenarios are:

Transfer into the cloud, where the applications remain the same. The data is only moved to another location.

Transfer to SaaS. Data from the existing application needs to be migrated to the new service.

Transfer from one SaaS to another SaaS.

Transfer out of the cloud.

101

Testing Migration The following general acceptance criteria will serve as

a basis for the migration test objectives:

Interruption to business process is minimal.

All migrated data can be tracked.

All data is converted correctly.

Defects in data before migration do not cause problems during migration.

No unnecessary data is migrated to the service.

102

Testing Migration The test robustness depends on the following factors:

How robust is the service for unexpected contents in the imported data?

How important is the migrated data for the proper functioning of the service?

How extensive are the changes in the way in which data is used?

103

Testing Migration Migration testing must contain the following:

The most important business processes. Process-oriented test, turned to the (possibly altered) user

process.

The correct import of data (is the data correct?).

The potentially changed way in which data is used.

The correct build-up of data history (correct order, with correct date and time).

104

Testing MigrationData Migration in IaaS and PaaS Data Migration in SaaS

1. There is the possibility that the data is moved with the hardware (such as setting a private cloud). No data migration is occurring. However, E2E and regression tests are needed.

2. When there is a new infrastructure and the data is copied to the cloud, a test needs to be executed to make sure the data transfers correctly.

1. When migrating data to SaaS, a data conversion is needed.

2. It can be performed by fully rebuilding the data by using the new (SaaS) application to manually enter the data. This method takes a lot of time.

3. A conversion software tool can be used.

105

Testing Migration The migration to a new test environment should be

also considered:

Transfer and migrate test data (the same steps as with production data).

Configure interfaces to other test systems.

Organize access to the new environments.

Configure administration for the new test environments (performed by supplier).

Test the test environment.

106

Testing due to legislation and regulations Organizations need to take legislation and regulations into

account when configuring the IT landscape.

There are legislations for:

Financial accountability

Pharmaceuticals

Medical care

Telecoms

On the most common legislations is Privacy legislation.

107

Testing in production Continuity in production in the case of changes

Services and services environments are always subject to change.

The following changes can occur once the service is live:

Changes in the service:

Announced changes – E2E regression test on the high-risk parts of the service

Retrospective information about changes – E2E regression test on the high-risk parts of the service

No information – Regularly perform the E2E regression testing on the parts with the highest risk

108

Testing in production Changes in other systems:

E2E regression test

Regression test targeting the changes

Changes at the supplier:

Reevaluate the supplier against selection criteria

Changes in the business processes:

E2E test with revised test cases to take into account the new procedures

Changes in connected resources:

E2E test can uncover problems in the interaction of the resources with the service in a test environment

109

Thank You!

110