testing cloud services: saas, paasand iaas€¦ · testing cloud services: saas, paasand iaas...
Post on 25-May-2020
41 Views
Preview:
TRANSCRIPT
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 1
Testing Cloud Services: SaaS, PaaS and IaaS
Martin Pol
Jeroen Mengerink
Agenda
• Introduction Cloud computing
• Challenges Risks
• Solutions Test measures
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 2
Objectives
• Learn how to
– Cope with Cloud services
– Detect the (new) risks
– Mitigate these risks
By using
• Your skills
• Heuristics
• The Book
• Discussions and exchange experiences
In the cloud?
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 3
What can be done in the cloud?
searching, recording, accounting, paying, writing,reviewing, tracking, calculating, developing, listening,analyzing, transmitting, learning, controlling,purchasing, testing, alarming, changing, updating,deleting, accessing, rejecting, correcting, studying,booking, receiving, tracing, protecting, deciding,managing, teaching, facilitating, identifying, copying,removing, demonstrating, checking, showing,selecting, subscribing, unsubscribing, sharing,mailing, communicating, reading, playing, working,meeting, gambling, shopping, storing, crosschecking, retrieving, configuring, sketching, saving,accelerating, enhancing, creating, growing, checkingin, checking out, finding out, reaching, denying,talking, designing, making, verifying, measuring
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 4
EmailSurfTransfer
Develop and Test
Operate and Manage Store
EmailSurfTransfer
Develop and Test
Operate and Manage Store
storage claim
80% unused
redundancy limitations
environmentally unfriendly
management overheadcosts for innovation
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 5
EmailSurfTransfer
Develop and Test
Operate and Manage Store
standard software bandwidth
internet technologySOA
virtualization
searching, recording, accounting, paying, writing,reviewing, tracking, calculating, developing, listening,analyzing, transmitting, learning, controlling,purchasing, testing, alarming, changing, updating,deleting, accessing, rejecting, correcting, studying,booking, receiving, tracing, protecting, deciding,managing, teaching, facilitating, identifying, copying,removing, demonstrating, checking, showing,selecting, subscribing, unsubscribing, sharing,mailing, communicating, reading, playing, working,meeting, gambling, shopping, storing, crosschecking, retrieving, configuring, sketching, saving,accelerating, enhancing, creating, growing, checkingin, checking out, finding out, reaching, denying,talking, designing, making, verifying, measuring
EmailSurfTransfer
Develop and Test
Operate and Manage Store
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 6
US: National Institute of Standards and Technologyhttp://www.nist.gov
Essential characteristics
On-demand service
Self service provisioning, pay-per-use
No human interaction
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 7
US: National Institute of Standards and Technologyhttp://www.nist.gov
Essential characteristics
On-demand service
Broad network access
Standard mechanisms over networks
“Any” client
US: National Institute of Standards and Technologyhttp://www.nist.gov
Essential characteristics
On-demand service
Broad network access
Resource pooling
Multi-tenant
Storage, processing, memory, virtual machines, …
Location independent
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 8
US: National Institute of Standards and Technologyhttp://www.nist.gov
Essential characteristics
On-demand service
Broad network access
Resource pooling
Rapid elasticity
Rapid scale in and out
“Any quantity” at any time
US: National Institute of Standards and Technologyhttp://www.nist.gov
Essential characteristics
On-demand service
Broad network access
Resource pooling
Rapid elasticity
Measured service
Controlled resource use
Transparency, pay-per-use
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 9
Service models
• Nocloud
• Infrastructure as a Service
• Platform as a Service
• Software as a Service
Application
Platform
Virtualization
Hardware
CloudInternal
Deployment models
• Public
• Private
• Community
• Hybrid
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 10
US: National Institute of Standards and Technologyhttp://www.nist.gov
Essential characteristics
On-demand service
Broad network access
Resource pooling
Rapid elasticity
Measured service
Deployment models
– private cloud
– community cloud
– public cloud
– hybrid cloud
Service Models
Software as a Service
Platform as a Service
Infrastructure as a Service
What is “done” in the cloud?
>500
PrivateHybrideCommunity
IaaS, PaaS, DaaS, SaaS
Taas
*aaS
Data CentreData Management
Business processes
Consumer
Public
SaaS
Surf and mailAppsSocial mediaDropboxGoogle servicesSpotifyPicasaGames……………
<500 employees
Public
*aaS
MailStorage
Infrastructure
CRM
Finance
Business processes
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 11
ISBN 978-1-937538-38-5
Theoretical Basis
What is in the book?
1. What is cloud computing?2. The changing role of the test manager3. What to test?4. How to test?
Page xx
What do you see as the two main risks of involving the cloud?
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 12
Continuity
Privacy
Multi platform
Legislation
Cyber crime
Impact organisation
Standards
143143
Continuity
Privacy
Multi platform
Legislation
Cyber crime
Impact organisation
StandardsPerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 13
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 14
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Page 37
Other customers
YOUROperational Profile
YOUROperational Profile
YOUROperational Profile
PLUS
YOUROperational Profile
PLUS
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 15
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Page 40
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 16
Everything over the web
The idea:“it’s safe”The idea:“it’s safe”
Home ground for hackers
Home ground for hackers
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
ManageabilityManageability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Page 40
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 17
Bring Your Own Device
No free choice of device.
No free choice of device.
Endlesspossibilities.
Endlesspossibilities.
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 18
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Page 42
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Internet connection lost@ supplier
@ user
@ other systems
‘Off line” does not work
Information is lost
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 19
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 20
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Page 44
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Mismatchservice <> business process
Functionality is changed
Insufficient usability
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 21
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Page 46
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 22
Backup and recovery
Taken care of.Taken care of.
Who will support me?
Who will support me?
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Page 46
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 23
Updates, patches, fixes, …
Planned and controlled
Planned and controlled
Do I have a choice?
Do I have a choice?
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
ManageabilityManageability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & regulationsLegislation & regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 24
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & regulationsLegislation & regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Where is my data?
And is that OK?
In house.In house.
Somewhere…Somewhere…
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & regulationsLegislation & regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 25
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & regulationsLegislation & regulations
Suppliers & outsourcingSuppliers & outsourcingRis
ks
Ris
ks
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & regulationsLegislation & regulations
Suppliers & outsourcingSuppliers & outsourcingRis
ks
Ris
ks
Page 49
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 26
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & regulationsLegislation & regulations
Suppliers & outsourcingSuppliers & outsourcingRis
ks
Ris
ks
Vendor lock in
No agreements
Supplier of the supplier of the supplier …
Supplier is taken over
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 27
Testing?
Check
Review
Monitor
Interview
Proof of concept
Testing!
Check
Review
Monitor
Interview
Proof of conceptTestenProef
IntakeInterviewProof of concept
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 28
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
TestenProefIntake
InterviewProof of concept
Selection
Implementation
Production
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 29
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 30
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 31
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
sPerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Architecture
From “individual” risksto
“individual” test measures
Architecture
From “individual” risksto
“individual” test measures
Page 47
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 32
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
sPerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Selection Criteria
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
CompletenessControllableFor serviceFor supplier
Spec’s and termsReferences……
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 33
How do you select the service?
Page 61-64
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 34
“Inspiration List”
CRITERION PRIOFunctionalDo the service and the specific business processes align?Does the service fit well in the E2E business process?Is the service sufficiently adaptable to specific requirements?Are many adjustments needed?Is customization possibleIs (a lot of) customization needed?Are the required platforms supported?Are “het nieuwe werken” and BYOD supported sufficiently?Is it possible to connect / integrate the service with the other systems?Are sufficient manuals and/or courses available?ImplementationIs the impact on current activities acceptable?Is a feasible route for migration towards the service available?
“Inspiration List”
CRITERION PRIOSupportAre changes in the service announced beforehand?Are sufficient test facilities available around the service (test environment, test tooling, testware, access to infrastructure, …)?Are there sufficient support facilities?Is it clear how incidents can be reported?Are incidents resolved fast enough?PerformanceAre response times low enough?Is the number of possible simultaneous users high enough?Is bandwidth sufficient?Is sufficient potential for growth available?Is the actual use charged correctly?
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 35
“Inspiration List”
CRITERION PRIOSecurityAre adequate authorization and authentication possibilities in place?Is the physical security of the service locations sufficient?Is the support access security of the service sufficient?Is mutual access security between customers sufficient?Are data changes traceable?Is data storage for the service reliable?Is deleting data in the service reliable?Is security of the connection to the service sufficient?Are security options for the customer sufficient?Does the supplier have security certificates? (for example SAS 70 type II)?AvailabilityIs the level of availability for the service sufficient?Are back-up / fail-over / disaster-recovery provisions sufficient?
“Inspiration List”
CRITERION PRIOLaw and regulationsDoes the data location comply to all legal requirements?Does the data processing comply to all legal requirements?Do the terms contain parts that are conflicting to the duties of the customer?SupplierIs clear what happens when the contract ends, or in case of bankruptcy or conflict?Is a good helpdesk available?Does the supplier have experience in:- Offering this particular service?- Offering services in general?- Developing services?- The customer’s field?- Developing, testing and supporting services (know how)?Do methods used by supplier align with those of the customer (if relevant)?
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 36
“Inspiration List”
CRITERION PRIOSupplierIs quality assurance arranged?Is the supplier ahead in its field?Is the size of the supplier in accordance with the expectations of the customer?Does the supplier have a good reputation (are there references)?Is providing services the core business of the supplier?Does the supplier have opportunities for future expansion?Does the supplier speak the same language?Is transparency and flexibility of the supplier sufficient?
Proof of Concept
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Dynamic testingMore suppliersTime boxingRepresentative
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 37
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
sPerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Known measurestuned and tweaked
New measures developed
Use:
– Your skills
– Heuristics
– The Book
– Your peers
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 38
Suppose your payroll system moves into the Cloud.
What would you do?
Load Testing
YOUROperational Profile
YOUROperational Profile
YOUROperational Profile
PLUSACTUAL MOMENT
YOUROperational Profile
PLUSACTUAL MOMENT
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 39
Operational profile
Page 73
Performance testing
• Test cases aimed at specific bottlenecks
• Including cloud aspectsin test cases
• Test setup for a
performance test
• Representative?
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 40
Stress Testing
Yes, you can!Yes, you can!
Definitely NOT!Definitely NOT!
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
Elasticity
Load and stress.Load and stress.
Load and elasticity.Load and elasticity.
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 41
How would you test elasticity?
load
load test – ‘up’
extend?
200 charged
100charged
no
yes
path test
99
100
101
boundary values
‘up’tc 1: use=99, pay 100tc 2: use=100, pay 100tc 3: use=101, pay 200
‘down’tc1: use=101, pay 200tc2: use=100, pay 100tc3: use=99, pay 100
boundary values
load test – ‘down’
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 42
load
load test – ‘up’
extend?
200 charged
100charged
no
yes
path test
99
100
101
boundary values
‘up’tc 1: use=99, pay 100tc 2: use=100, pay 100tc 3: use=101, pay 200
‘down’tc1: use=101, pay 200tc2: use=100, pay 100tc3: use=99, pay 100
boundary values
load test – ‘down’
• (Automatic) scaling up or down does not perform as required
• At scaling moments functional problems emerge
• Insight in use based costs is not sufficient
ISO 27001 aspects:
• Confidentiality of the data and the accompanying risk that unauthorized people can view the data
• Integrity of data and the accompanying risk that data is altered or lost unintentionally
• Availability of data and the accompanying risk that data (and services) is not available when it is required
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 43
ISO 27001 aspects:
• Confidentiality of the data and the accompanying risk that unauthorized people can view the data
• Integrity of data and the accompanying risk that data is altered or lost unintentionally
• Availability of data and the accompanying risk that data (and services) is not available when it is required
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
• Who has access to the data?
• Can the user trust that the data is correct?
• Can the user gain access to the data at all times?
• Security at:
– Network
– Supplier
– User
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
resTesting security robustness against Internet
attacks - Directory traversal. Read and/or write in
directories other than those allowed.- XML external entity attack. Include extra
(bad) data in an XML file.- SQL injection. Request and/or change data
by manipulating SQL queries. - Cross-site scripting (XSS). Transfer data to
other websites without the user knowing.- Session manipulation. Skip steps or
validation in a session.
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 44
• Security at:
– Network
– Supplier
– User
• Encryption
• Authentication and authorisation
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
IDaaS
• Security at:
– Network
– Supplier
– User
• Encryption
• Authentication and authorisation
• Test logs and audit trails
• Security Audits
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
IDaaS
Experts
Security patch routines
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 45
• Completeness and correctness of specifications and manuals
– Supplier
– User
• Availability of test environments
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
Interface specifications
Supported platforms
Business process specs
User manuals
How do you setup your environment including the
external service?
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 46
Manageablity of test environments
• Everything in the cloud
Manageablity of test environments
• Link all current environments to the service
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 47
Manageablity of test environments
• Link Production to the real service
• Link other environments to a MOCK SERVICE(or another instance of the service)
• Completeness and correctness of specifications and manuals
– Supplier
– User
• Availability of test environments
• Management of:
– Defects
– Changes
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 48
Defect Management
Incident
Supplier resolves it
Client resolves it
Incident not resolved
Test
Change work process
Change configuration
Custom solution
Service not selected
Terminate use of service
Workaround work instruction
Test
Test
Test
Test
Migrate and test
• Completeness and correctness of specifications and manuals
– Supplier
– User
• Availability of test environments
• Management of:
– Defects
– Changes
• Maintainability of the software
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 49
• Role of system architecture
• Monitoring and Logging
• Guarantees and SLA’s
• Test fail-over mechanism
• Test online/offline
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
Fail-over testing
A: disrupted
B: active
A: active
B: inactive
A is disrupted
B takes over service
A: inactive
B: active
disru
ptio
nin
A e
nded
no ch
ange
A is d
isrupte
d
no ch
ange
A: active
B: disruptedB is disrupted
A takes over service
B is
dis
rupte
dno c
hange
dis
ruptio
nin
B e
nded
no c
hange A: disrupted
B: disrupted
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 50
Online – Offline
Use case testing.Global testing.
Use case testing.Global testing.
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
Online – Offline
Use case testing.Global testing.
Use case testing.Global testing.
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Off line tests focussed on problems:
• Work continues, based on out-of-date information, and
this information could be changed in the cloud during the
offline period.
• The users are not aware that they are working (partly)
online (and are lead to believe differently).*
• Synchronization conflicts arise because data is changed
locally as well as in the cloud.
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 51
Online – Offline
Use case testing.Global testing.
Use case testing.Global testing.
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
Off line test cases:
• End the connection and check whether the users can see
that they are working offline.
• Disrupt the connection (for instance, a port or a certain
type of IP traffic) and check whether problems arise.
• Check whether changes that are made offline find their
way to the cloud when online status is regained.
• Check whether conflicts between offline and cloud data
are handled robustly (which is in fact a functional
requirement).
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing caused byLegislation & RegulationsTesting caused byLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 52
How does functional testing change with services?
Functional test objectives
• Does the service fit the business processes and vv?
• Is the service quality sufficient (number of bugs)?
• Is the service sufficiently user friendly?
• Is the service configuration done correctly?
• Does supplier customization function properly?
• Does customer customization function properly?
• Do interfaces work properly?
• Are platforms properly supported?
• Does everything work after changes (is there no regression)?
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 53
Any device – any platform
Multiplatform testing.
Multiplatform testing.
Multiplatform testing.
Multiplatform testing.
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing caused byLegislation & RegulationsTesting caused byLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
3997 distinct Android devices
http://opensignal.com/reports/fragmentation.php
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 54
Internet Explorer 6
Internet Explorer 7
Internet Explorer 8
Firefox 3.5
Firefox 3.6
Firefox 4
Safari 4
Safari 5
Chrome11
Opera11
Windows XP
Windows Vista
Windows 7
Windows 2003 Server
Windows 8
Windows CE
Linux
Unix
Mac OS Lion
Mac OS Snow Leopard
iOS
Android
Operating systems
Browsers
Multi-platform testing
Devices
Computer
Mobile phones
Tablet
PC
Macintosh
SUN
NOKIA …
Samsung …Windows Mobile
iPhone ...
…
MOTOROLA …
Blackberry …
ASUS ...
…
Internet Explorer 6
Internet Explorer 7
Internet Explorer 8
Firefox 3.5
Firefox 3.6
Firefox 4
Safari 4
Safari 5
Chrome11
Opera11
Windows XP
Windows Vista
Windows 7
Windows 2003 Server
Windows 8
Windows CE
Linux
Unix
Mac OS Lion
Mac OS Snow Leopard
iOS
Android
Operating systems
Browsers
Multi-platform testing
Devices
Computer
Mobile phones
Tablet
PC
Macintosh
SUN
NOKIA …
Samsung …Windows Mobile
iPhone ...
…
MOTOROLA …
Blackberry …
ASUS ...
…
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 55
Any device – any platform
Multiplatform testing.
Multiplatform testing.
Multiplatform testing.
Multiplatform testing.
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing caused byLegislation & RegulationsTesting caused byLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Scenarios
• Transfer into the cloud, applications remain the same– data moved to another
location
• Transfer to SaaS– data migrated to new service
• Transfer from one to another SaaS– similar
• Transfer out of the cloud.– similar
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 56
Data conversion
• Testing conversion rules
• Testing conversion on input data
• Testing if any data is lost
• Testing ongoing transactions
Existingsystems
Extraction Conversion Import
Conversion software
Service
• Rounding (totals incorrect)• Field lengths (truncation)• Totals (information lost)• Date and time conversions
what is 08-09-11?
• Audit trail, check sums
• E2E business scenario’s
Other aspects
• Cleaning data defects
– solved before migration
– no problems during migration
• Testing security aspects
– during and after migration
– not TOO much data migrated
• Testing performance
– speed (how long does it take?)
– volume (capacity sufficient?)
– stability at full volume
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 57
Example: email to the cloud
• Tools migrate existing emails to the cloud
• Low risk:
– migrating one or some mailboxes and executing a limited testing
– if successful: implementation for all mail boxes
• High risk:
– no emails lost in migration?
– formatting of the emails still correct?
– all attachments still there?
– all attributes migrated (priorities, timestamps, flags, …)?
Legal importance of email
reading, forwarding, replying, check on contents
Legislation + Regulations
=
Test basis
Incidental testing.Incidental testing.
Compliancy testing.Compliancy testing.
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
Sarbanes Oxley
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 58
Where is my data stored?
– nothing, or hardly anything, to be found on this subject
– service stores data outside the borders of permitted countries additional measures?
– service stores data within the borders of permitted counties okay
data owner is responsible for ensuring that the protection of personal data is at the required level wherever it is held
Checking for legislation and regulations
• List where data that is stored in the cloud
• Find the requirements that are applicable to this data
• Check supplier terms with customer’s requirements
• Perform (external) audit for high risk
• Test manager provides advice, management decides
Legal support needed for high risk
Example. A supplier of a storage service claims to be the owner of the intellectual capital of all data stored at their facilities. It is highly unlikely that this is compatible with the interests of the organization that is the actual owner of the data.
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 59
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
sLegal issues – threats
Example: Dropbox
Compliance with Laws and Law Enforcement Requests; Protection of Dropbox's Rights.
• We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of Dropbox or its users; or (d) to protect Dropbox’s property rights. If we provide your Dropboxfiles to a law enforcement agency as set forth above, we will remove Dropbox’s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 60
Is testing in production necessary?
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 61
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
s
ContinuousEnd-to-End Testing
Continuous Change
Continuity
Privacy
Multi platform
Legislation
Cyber crime
Impact organisation
Standards
Check
Intake
Monitor
Interview
Proof of concept
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 62
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
sPerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
MaintainabilityMaintainability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Testing starts early: in selectionScope of testing is widened
Testing continues in production
Testing starts early: in selectionScope of testing is widened
Testing continues in production
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due to Legislation & RegulationsTesting due to Legislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during Selection
Te
st M
ea
su
res
Te
st M
ea
su
res
PerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
ManageabilityManageability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Questions?Questions?
www.polteq.com
martin.pol@polteq.comjeroen.mengerink@polteq.com
Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014
© Polteq 63
Performance TestingPerformance Testing
Security TestingSecurity Testing
Manageability TestingManageability Testing
Availability & Continuity TestingAvailability & Continuity Testing
Functional TestingFunctional Testing
Migration TestingMigration Testing
Testing due toLegislation & RegulationsTesting due toLegislation & Regulations
Testing in ProductionTesting in Production
Testing during SelectionTesting during SelectionT
es
t Me
as
ure
sT
es
t Me
as
ure
sPerformancePerformance
SecuritySecurity
Availability & ContinuityAvailability & Continuity
FunctionalityFunctionality
ManageabilityManageability
Legislation & RegulationsLegislation & Regulations
Suppliers & OutsourcingSuppliers & OutsourcingRis
ks
Ris
ks
Thank you!Thank you!
www.polteq.com
martin.pol@polteq.comjeroen.mengerink@polteq.com
top related