saas testing on an agile world

About me - Joel Montvelisky

PractiTest – Co Founder & Prod. Architect

QA Instructor & Consultant

Mercury Interactive - QA Manager (retired )

TD, QC, WR, QTP, etc

ITCB (IL) Advisory Board

QABlog.practitest.com

© 2010 -

Today’s Agenda

 SaaS - Software as a Service

 Things we Can Stop Testing

 Non-trivial Stuff to Think About & TEST

 Leveraging Live Feedback into the Testing Lifecycle

 Open Season for Questions…

© 2010 -

© 2010 -

What is SaaS?

Software as a Service

 Hosted Platforms – Total Control – No Maintenance

 No Client Installations

 Customers buys each month – keep them happy

© 2010 -

Users don’t care it is SaaS,

it needs to be better than the alternative

SaaS QA Management

 End-to-End QA Management (Reqs – Tests – Bugs)

 Enterprise Level SaaS Solution - Methodology - Customizations & Flexibility

 Global platform - Communication - Languages - Available 24x7 Worldwide

 Intuitive & Simple

 Agile Dev, RoR, on Amazon EC2 © 2010 -

© 2010 -

What does SaaS mean to the Testing Process?

Things we can STOP Testing

✘  Client or Server installations

✘  Multiplatform backend support

✘  Multiple version upgrades

✘  Backwards compatibility

© 2010 -

“Savings” of approx. 1/3 of the Traditional Enterprise

Application Testing Cycle

The regular testing scenarios

 Functionality - Manual scripts - Exploratory sessions - Checklists - End-2-End Scenarios - Selenium - Cucumber & RSpec

 Load & Stress

 Multi-platform & Multi-browser support

 I18N English, Swedish, German, Chinese Simplified, Hebrew

© 2010 -

Agile Sprints & Updates of 3 – 4 weeks

Each sprint’s cycle is based on its content

Non-trivial stuff to think about & TEST

1. Remote accessibility & usage

  Staging Servers in USA - Amazon EC2 Development & Testing teams in Israel (& WW collaborators)

  Globally distributed monitoring services

(www.pingdom.com)

© 2010 -

Non-trivial stuff to think about & TEST

2. Security

Application-related:

  Cross-site scripting

  SQL injections

  HTTP header injections

etc…

© 2010 -

Infrastructure-based:

  Secured communication

  Backups & storage policies

  Controlled Access to the site & servers

Non-trivial stuff to think about & TEST

3. Live updates & deployments (3 to 4 weeks)

  No concrete separation between Dev & IT

  Minimal service disruption

  Dressed rehearsals (including rollbacks!)

  Automated sanity & manual verification

  Patches & hot-fixes (extraordinary occasions )

  Prove of Concepts & Beta Releases

© 2010 -

QA & Testing as a facilitator for Balance: Stable & Professional, yet Flexible Environment

Non-trivial stuff to think about & TEST

4. Disaster recovery procedures

Two Main scenarios:

(a) System down to be brought up quickly

  Configure machines

  Install & deploy software

  Restore data

(b) Rollback to last known stable data (with or without data restore)

© 2010 -

Define a policy & schedule regular drills to ensure you can achieve it!

© 2010 -

Leveraging Real Data into the Testing Process

Levering live feedback into the Testing Process

(1) Internally developed monitoring tools

- Check usage patterns

- Validate need & usability assumptions

- Analyze patterns and create realistic & prioritized testing scenarios

© 2010 -

No need to continue guessing!!!

Levering live feedback into the Testing Process

(2) Site Monitoring systems   Real-time usage to monitor traffic

  Allow users to contact the company with questions or issues

Pingdom (www.pingdom.com)

  Up to date response times for real projects and data.

  Proactive alerts when things start slowing down

© 2010 -

Levering live feedback into the Testing Process

(3) Proactive mail notifications on issues

(www.hoptoad.com)

  Real-time notifications on issues on the system.

  Information about the issue including system parameters that allow you to do an initial trouble-shoot of the issue

© 2010 -

Levering live feedback into the Testing Process

(4) Ask users for Real Projects

  No better way to test than using real data

  Most users will trust you not to abuse their information in exchange for assurance of working software

  Since the data sits in your system it is easier to copy projects to your testing environment

 Need to be careful with who has access to the data; making sure your testing environment is as secured as your production environment

© 2010 -

© 2010 -

Wrapping up…

Wrapping up

  SaaS is only the delivery model, customers will not compromise on Functionality, Stability or Response Time

  Since the system is hosted you can save all tests related to backend installation and support

  You need to pay more attention to stuff like accessibility, security, deployments & DRPs

  The lines between the R&D and IT organizations in SaaS are blurry

  SaaS application allows better access to real data to be leveraged to improve the effectiveness of our testing operations.

© 2010 -

© 2010 -

Open Season for questions!

© 2010 -

Special thanks to Anne-Marie Charrett

joel@practitest.com http://qablog.practitest.com