201002 automation-1000 tests

8/6/2019 201002 Automation-1000 Tests

1/18

Automation - Getting through thefirst 1,000 tests

Test Types Covered: Functional Testing Fuzz Testing Load Testing Database Testing

By:

Jeremy Carey-dressler


2/18

Overview

What is automation?

What you need

Product choices

How to design the system

Code samples


3/18

What is Automation?

To replace or enhancehuman labor withmachines Wiktionary

Test Automation is meantto do a few things:

Perform tasks not possible oreasy to do

Works when testers are notavailable

Regression, Integration,acceptance testing

Find issues in the system


4/18

Why Aren't Unit Tests Enough?

Only happy path or paths imagined at time ofwriting

Typically written by the developer

Does not verify things come together

Inputs dont change

Not what a user does unless you are selling a

API

Environment is ignored, data is simulated

Can't easily verify the view (particularly html)


5/18

General Question For Automation

Consider the following:

How stable is the system or component (UI, DB,etc.)?

What are managements expectations? How long do you expect the product to last?

Who is writing the automation?

Is it a full time job?

How easy is it to create the automation? Are test hooks created?

How many custom controls exist in the app?

How stable is the test tool?


6/18

Functional Automation

Typically UI based testing

Typically best bang for the buck Each test should be independent Well understood in both automated and manual

testing

Code should be so easy to understand that you

can run it manually 6 months later Rapid feedback

Never blind or faith based

One test boils down to verifying one thing


7/18

Setting up for automation

Software (which should have hooks forautomated)

Available simulations of a customer environment

(VMs, etc) Installed software (unless it is planned to

automate the install)

Test Data Manual Tests

Budget / HR / Willing Management


8/18

Record and Playback

Useful for...

Learning thesystem

Developing ideas Manual tests

Quick and dirtytests

Small test sets

Not useful for...

Long termautomation

Framework-based code

Large test sets

Changing codebase


9/18

API vs All-in-one Tools

API Pros

Not tied to a toolset

Typically better forframeworks

More customizable

API Cons

No reporting Harder to run in

multipleenvironments

All-in-one Pros

Recorder, reportingand running all builtin

Single cost All-in-one Cons

Typically one wayto do things

Hard to developlarge set ofautomation

Recorder-based


10/18

Open Source Options

Slick Test Developer

http://sourceforge.net/projects/slicktest/

WatiN

http://watin.sourceforge.net/ Watir

http://watir.com/

AutoIt

http://www.autoitscript.com/autoit3/

Nunit

http://www.nunit.org/index.php
http://watir.com/http://www.autoitscript.com/autoit3/http://www.autoitscript.com/autoit3/http://watir.com/


11/18

xUnit

Setup

Class

Test (Method)

Clean up

Class

Test (Method)

When test fails by exception or assert, the testscleanup is called


12/18

Framework, framework, framework

Spend time designing the back end

Consider how the UI is likely to change

Look at the cost/benefits in flexibility,readability, runtime speed, anddevelopment speed.

Hide UI implementation (Called a

description, UI Map, ID, etc)

Separate concerns such as Actions,Tests/Asserts, Steps and the actual test


13/18

General Design Architecture

Establish hierarchy and levels of division for tests in aclass by page, module, test type or functionality

Define Page/UI to map a set of elements to a live UI

Define Steps which are a collection of Actions

Define Actions to do one single thing in the UI

Define Data to provide hardcoded, dynamic or random

data that will be used in testing. Define Test/Assert to verify elements


14/18

General Design Architecture Details

Use Magic-words (5 or less) to make work easier

Example: Test.Steps.*, UI.Page.*, DBTest.Data, etc.

Allows for separate people to do tests and framework

If the UI does NOT change often, you can build up the Elements Example: UI.Admin.Settings.Save()

If the UI does change often, you are better off having a moreflat structure

Example: UI.AdminSettings.Save() This method works best with Functional or Database testing


15/18

Load Testing

There are many forms of Load Testing: Performance / Capacity

Endurance

Stress Testing

Spike

Configuration

In most cases, 10-20 test cases capture 70+% of your user base'sactivities

Load Tests are less developed compared to functional testing

Expect them to cost more per test

Rarely will you need more than 100 cases for a single application

Some can get away with just one or two tests


16/18

Fuzz Testing

Automation with random or pseudo-randominputs

Not intended to find functional bugs

You can perform this by API, UI, DB inputs, fileformats or network protocols

Send valid, mostly-valid, invalid or a mixture in

fuzz testing Can be mixed with other testing and can be

helpful for increasing code coverage


17/18

Acknowledgments

McDonalds Image:http://www.flickr.com/photos/xclozano/2913341645/

Some advice inspired by http://www.kaner.com/testarch.html andhttp://www.io.com/~wazmo/papers/seven_steps.html

Unit Test Information: http://www.youtube.com/watch?v=Pq6LHFM4JvE

People from past and present QA groups I have worked with

Tools you may have seen today:

Microsoft Visual Studios

Sharp Develop

Slick Test Developer

WatiN WatiR

Nunit

Resharper
http://www.flickr.com/photos/xclozano/2913341645/http://www.kaner.com/testarch.htmlhttp://www.io.com/~wazmo/papers/seven_steps.htmlhttp://www.youtube.com/watch?v=Pq6LHFM4JvEhttp://www.youtube.com/watch?v=Pq6LHFM4JvEhttp://www.io.com/~wazmo/papers/seven_steps.htmlhttp://www.kaner.com/testarch.htmlhttp://www.flickr.com/photos/xclozano/2913341645/


18/18

Code!

Practical code samples

201002 automation-1000 tests

Documents