1. TM PM Tutorial 4/30/13 1:00PM High-flying Cloud Testing Techniques Presented by: Ruud Teunissen Polteq Test Services BV Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 904-278-0524 sqeinfo@sqe.com www.sqe.com

2. Ruud Teunissen An international test consultant at Polteq Test Services BV, Ruud Teunissen has performed several test functions in a number of IT projects: tester, test specialist, test consultant, and test manager. Ruud participated in the development of the structured testing methodology TMapTest Management Approach. Together with Martin Pol and Erik van Veenendaal, Ruud is coauthor of several books on structured testing, including Software Testing: A Guide to the TMap Approach.

3. High Flying Cloud Testing Techniques Ruud Teunissen Polteq Test Services BV The Netherlands 1

4. searching, recording, accounting, paying, writing, reviewing, tracking, calculating, developing, listening, analyzing, transmitting, learning, controlling, purchasing, testing, alarming, changing, updating, deleting, accessing, rejecting, correcting, studying, booking, receiving, tracing, protecting, deciding, managing, teaching, facilitating, identifying, copying, removing, demonstrating, checking, showing, selecting, subscribing, unsubscribing, sharing, mailing, communicating, reading, playing, working, meeting, gambling, shopping, storing, cross checking, retrieving, configuring, sketching, saving, accelerating, enhancing, creating, growing, checking in, checking out, finding out, reaching, denying, talking, designing, making, verifying, measuring 3 Develop and Test Operate and Manage Email Surf Transfer Store 4 2

5. Develop and Test Email Surf Transfer redundancy, proliferation limitations 80% unused storage claim environmentally unfriendly Operate and Manage Store 5 Develop and Test Email Surf Transfer SOA internet technology virtualization standard software Operate and Manage bandwidth Store 6 3

6. searching, recording, accounting, paying, writing, reviewing, tracking, calculating, developing, listening, Develop and transmitting, learning, controlling, Email analyzing, Test Surf purchasing, testing, alarming, changing, updating, Transfer deleting, accessing, rejecting, correcting, studying, booking, receiving, tracing, protecting, deciding, managing, teaching, facilitating, identifying, copying, removing, demonstrating, checking, showing, selecting, subscribing, unsubscribing, sharing, mailing, communicating, reading, playing, working, meeting, gambling, shopping, storing, cross checking, retrieving, configuring, sketching, saving, Operate and Manage accelerating, enhancing, creating, growing,Store checking in, checking out, finding out, reaching, denying, talking, designing, making, verifying, measuring 7 4

7. Essential characteristics On-demand service Self service provisioning, pay-per-use No human interaction US: National Institute of Standards and Technology http://www.nist.gov Essential characteristics On-demand service Broad network access Standard mechanisms over networks Any client US: National Institute of Standards and Technology http://www.nist.gov 5

8. Essential characteristics On-demand service Broad network access Resource pooling Multi-tenant Storage, processing, memory, virtual machines, Location independent US: National Institute of Standards and Technology http://www.nist.gov Essential characteristics On-demand service Broad network access Resource pooling Rapid elasticity Rapid scale in and out Any quantity at any time US: National Institute of Standards and Technology http://www.nist.gov 6

9. Essential characteristics On-demand service Broad network access Resource pooling Rapid elasticity Measured service Controlled resource use Transparency, pay-per-use US: National Institute of Standards and Technology http://www.nist.gov Essential characteristics On-demand service Broad network access Resource pooling Rapid elasticity Measured service Deployment models private cloud community cloud public cloud hybrid cloud Service Models Software as a Service Platform as a Service Infrastructure as a Service US: National Institute of Standards and Technology http://www.nist.gov 7

10. Question What is being moved into the cloud? 15 Pack List 1 2 3 4 5 16 8

11. What is done in the cloud? 500 Consumer Public Private Hybride Community Public *aaS IaaS, PaaS, DaaS, SaaS SaaS Taas *aaS Mail Storage Infrastructure CRM Finance Business processes Data Centre Data Management Business processes 17 Surf and mail Apps Social media Dropbox Google services Spotify Picasa Games Standards Cyber crime Continuity Privacy Legislation Multi platform Impact organisation 143 9

12. Performance Security Availability & Continuity Functionality Risks Manageability 19 Legislation & Regulations Suppliers & Outsourcing Performance Security YOUR Operational Profile Availability & Continuity Functionality Other customers Risks Manageability 20 Legislation & Regulations Suppliers & Outsourcing YOUR Operational Profile PLUS 10

13. Performance Security The idea: its safe Availability & Continuity Functionality Everything over the web Risks Manageability 21 Legislation & Regulations Home gound for hackers Suppliers & Outsourcing Performance Security No free choice of device. Availability & Continuity Functionality Bring Your Own Device Risks Manageability 22 Legislation & Regulations Endless possibilities. Suppliers & Outsourcing 11

14. Performance Taken care of. Security Availability & Continuity Functionality Backup and recovery Risks Manageability 23 Legislation & Regulations Who will support me? Suppliers & Outsourcing Performance Security Planned and controlled Availability & Continuity Functionality Updates, patches, fixes, Risks Manageability 24 Legislation & Regulations Do I have a choice? Suppliers & Outsourcing 12

15. Performance In house. Security Availability & Continuity Functionality Where is my data? And is that OK? Risks Manageability 25 Legislation & Regulations Somewhere Suppliers & Outsourcing Performance Security Availability & Continuity Functionality Risks Manageability 26 Legislation & Regulations Suppliers & Outsourcing 13

16. Question Group Session Lets look at the clustered risks What test measures would you consider applying? 27 Test Measures 1 2 3 4 5 28 14

17. Testing? Interview Check Trial Proof of concept Intake 29 Performance Testing Security Testing Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Interview Proof ofProef concept Testen Intake Functional Testing Migration Testing Testing caused by Legislation & Regulations 30 Testing in Production 15

18. Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 31 Legislation & Regulations Suppliers & Outsourcing Testing in Production Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Testing caused by Legislation & Regulations Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 32 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 16

19. Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 33 Legislation & Regulations Suppliers & Outsourcing Testing in Production Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Architecture Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Testing caused by Legislation & Regulations Availability & Continuity Testing Risks From individual risks Functional Testing to Manageability individual test measures Migration Testing 34 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 17

20. Performance Testing Selection Security Testing Manageability Testing Implementation Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Production Migration Testing Testing caused by Legislation & Regulations Testing in Production 35 Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 36 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 18

21. Performance Testing Security Testing Manageability Testing Selection Criteria Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations Testing in Production 37 Performance Testing Security Testing Manageability Testing Proof of Concept Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations 38 Testing in Production 19

22. Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 39 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production Performance Testing Security Testing Known measures tuned and tweaked Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing New measures developed Migration Testing Testing caused by Legislation & Regulations 40 Testing in Production 20

23. Question Group Session Suppose your CRM appliction is moved into the Cloud. What would you test? 41 Test Measures When CRM Moves into the Cloud 1 2 3 4 5 42 21

24. YOUR Operational Profile Performance Testing Security Testing Manageability Testing Load Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing YOUR Operational Profile PLUS ACTUAL MOMENT 43 Testing caused by Legislation & Regulations Testing in Production Yes, you can! Performance Testing Security Testing Manageability Testing Stress Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Definitly NOT! 44 Testing caused by Legislation & Regulations Testing in Production 22

25. Load and stress. Performance Testing Security Testing Manageability Testing Elasticity Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations Load and elasticity. 45 Testing in Production Question How would you test Elasticity? 46 23

26. Test Elasticity 47 load test down load test up 101 100 boundary values 99 load path test extend? up tc 1: use=99, pay 100 tc 2: use=100, pay 100 tc 3: use=101, pay 200 no yes 200 charged boundary values 100 charged down tc1: use=101, pay 200 tc2: use=100, pay 100 tc3: use=99, pay 100 48 24

27. load test down load test up 101 100 boundary values (Automatic) scaling up or down 99 does not perform as required load At scaling moments functional problems test boundary values path emerge extend? yes no Insight in use not sufficient 200 charged 100 charged up tc 1: use=99, pay based costspay100 is 100 tc 2: use=100, tc 3: use=101, pay 200 down tc1: use=101, pay 200 tc2: use=100, pay 100 tc3: use=99, pay 100 49 Security at: Network Supplier User Performance Testing Security Testing Manageability Testing Encryption Authentication and autorisation Test logs and audit trails Security Audits 50 Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations Testing in Production 25

28. Completeness and correctness of specifications Supplier User Availibilty of test environments Management of: Defects Changes Maintainability of the software 51 Performance Testing Security Testing Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations Testing in Production Manageablity of test environments Everything in the cloud 52 26

29. Manageablity of test environments Link all current environments to the service 53 Manageablity of test environments Link Production to the real service Link other environmnets to a MOCK SERVICE 54 27

30. Question Who will solve the defects? And how? 55 Defect resolution 56 28

31. Defect Management 57 Performance Testing Role of system architecture Security Testing Monitoring and Logging Manageability Testing Guarantees and SLAs Availability & Continuity Testing Test fail-over mechanism Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Functional Testing Test online/offline Migration Testing Testing caused by Legislation & Regulations 58 Testing in Production 29

32. Performance Testing Security Testing Manageability Testing Online Offline Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Use case testing. Global testing. 59 Testing caused by Legislation & Regulations Testing in Production Fail-over testing 60 30

33. Performance Testing Security Testing Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations 61 Testing in Production Question So whats new here? 62 31

34. Functional Testing 63 Multiplatform testing. Performance Testing Security Testing Manageability Testing Any device any platform Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Multiplatform testing. 64 Testing caused by Legislation & Regulations Testing in Production 32

35. Internet Explorer 6 Windows XP Internet Explorer 7 Windows Vista Internet Explorer 8 Windows 7 Firefox 3.5 Windows 2003 server Firefox 3.6 Browsers Windows 8 Firefox 4 Safari 4 Windows CE Safari 5 Linux Operating Systems Chrome11 Unix Opera11 Multiplatform Mac OS Lion PC Mac OS Snowleopard SUN Computer iOS Macintosh Android iPhone .. Windows Mobile Samsung Devices NOKIA Mobile Xxx ASUS.. Blackberry Tablet MOTOROLA Xxx 65 Multiplatform testing. Performance Testing Security Testing Manageability Testing Any device any platform Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Multiplatform testing. 66 Testing caused by Legislation & Regulations Testing in Production 33

36. Performance Testing Security Testing Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Interruption of business processes is minimal All migrated data can be tracked (audit trail) All data is converted correctly All pending transactions are successfully finished after the migration Defects in data before migration do not lead to problems during migration Defects in data are solved and not migrated defective No more data than necessary is migrated to the service Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations Testing in Production 67 Incidental testing. Performance Testing Security Testing Legislation + Regulations = Test basis Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations Compliancy testing. 68 Testing in Production 34

37. Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 69 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production Performance Testing Continuous End-to-End Test Security Testing Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functionals and non-functionals Functional Testing Migration Testing Testing caused by Legislation & Regulations 70 Testing in Production 35

38. Standards Cyber crime Check Continuity Interview Privacy Legislation Proof of concept Trial Multi platform Intake Impact organisation Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability Test starts earlier & Continuity Testing Test scope is widened Test will never Functional Testing stop Manageability Functionality Risks Migration Testing 72 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 36

39. Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Questions? Functional Testing Manageability Risks Migration Testing 73 Legislation & Regulations Suppliers & Outsourcing Testing in Production Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Testing caused by Legislation & Regulations Availability & Continuity Testing Thank you! Functional Testing Manageability Risks Migration Testing 74 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 37

40. About the speaker Ruud Teunissen Polteq Test Services, The Netherlands ruud.teunissen@polteq.com - http://www.polteq.com In the testing world since 1989, Ruud Teunissen has held numerous test functions in different organizations and projects: tester, test specialist, test consultant, test manager, etcetera. Ruud is co-author of several books on software testing and is a frequent speaker at (inter)national conferences and workshops. He was a member of the program committee for Quality Week Europe and EuroSTAR. Ruud is currently Senior Test Consultant at Polteq Test Services BV and responsible for the quality of Polteq services and assignments. 75 38