how to get automated testing "done"

How to get Automated Testing "Done"

Hans BuwaldaLogiGear

[email protected]@hansbuwaldawww.TestArchitect.comwww.HappyTester.com

©2016 LogiGear Corporationwww.LogiGear.com

Who is your speaker

• Software testing company, around since 1994– Foster City, California– consultancy, training– test development and automation services– "test integrated" development services

• Products:– TestArchitect™, TestArchitect for Visual Studio™– integrating test development with test management and automation– based on modularized keyword-driven testing

• LogiGear Magazine:– themed issues, non-commercial

• Dutch guy, in California with LogiGear since 2001• Background in math, computer science, management• Since 1994 focusing on testing and automation

– keywords, agile testing, big testing

Hans Buwalda

LogiGear [email protected]@hansbuwaldawww.TestArchitect.comwww.HappyTester.com


Relation to code Quality / depth Automation Scalability

Unit TestingClose relationship with the code

Singular test scope, but deep into the code

Fully automated by nature

Scalable, grows with the code, easy to repeat

Functional Testing

Usually does not have a one-on-one relation with code

Quality and scope depends on test design

In particular UI based automation can be a challenge

Often a bottle-neck in scalability

Exploratory Testing

Human driven, not seeking a relation with code

Usually deep and thorough, good at finding problems

May or may not be automated afterwards

Not meant to be repeatable. Rather do a new session

Some test kinds and their scalability (simplified)


• High degree of automation coverage– under Agile and DevOps automation went from nice-to-have to

must-have

• Scrum teams are well suited to, collectively, own testing and automation

• QA members (test development and automation) need to stay in the same sprint with the rest of the team– developers and product owners are a great help for testing, and vice

versa– otherwise the task gets harder, and will delay even more

• However, it is not always practical to have fully automated tests as part of "done" for a sprint . . .

Success factors in Agile (Scrum)


• More test cases needed than is time for• Shooting a moving target, changes are common• Lacking technology• Since QA benefits greatly from close cooperation with PO's and Devs, matters might

get worse once out of sync• Not always practical to let development wait for QA to finish (true meaning of "done")

QA role may have a hard time keeping up with the rest of the team

Sprint Sprint Sprint

PO

Dev

QA

PO

Dev

QA

PO

Dev

QA

ROLES

Sprint

PO

Dev


• Design tests to be automation-friendly– successful automation is not as much a technical challenge as it is a

test design challenge

• Make tests easy to understand for all involved– so, not too technical please– note: "easy to understand" does not necessarily mean "easy"

• Invest in testability of the system under test– one of the main benefits of having a team– works two ways: good tests facilitate "test driven" development

• Organize the activities well– I like to consider tests, and their automation, as "products"– let the testing and automation activities fit the sprint– support the teams well with infrastructure, technology, services, etc

Approaches


Easy to understand ? ? ?

WebElement element = driver.findElement(By.name(name));

element.sendKeys("mystery magic");element.submit();

(new WebDriverWait(driver, 10)).until(new ExpectedCondition<Boolean>() {

public Boolean apply(WebDriver d) {return d.getTitle().toLowerCase().startsWith("mystery");

}}

);

System.out.println(driver.getTitle());


More easy to understand: actions

4 actions, each with an action keyword and arguments

read from top to bottom

fragment from a test with actions

acc nr first last

open account 123123 John Doe

acc nr amount

deposit 123123 10.11deposit 123123 20.22

acc nr expected

check balance 123123 30.33

• The test developer creates tests using actions with keywords and arguments in a easy to understand spreadsheet format

• Checks are, as much as possible, explicit (specified expected values)

• The automation task focuses on automating the keywords, each keyword is automated only once

• In addition to keywords you can also use "given-then-when" scenarios (BDD) for good readability


Supporting BDD with actions

Given a customer previously bought a black sweater from me

And I currently have three black sweaters left in stock

When he returns the sweater for a refund

Then I should have four black sweaters in stockcustomer buys, article, color

a customer previously bought a {color} {article} from me

set stock, article, color, amountI currently have {amount} {color} {article} left in stockmy stock of {color} {sweater} is {amount}

return article, article, colorhe returns the {color} {article} for a refund

check stock, article, colorI should have four {color} {article} in stock

article color

customer buys sweater blackarticle color amount

set stock sweater black 3article color

return article sweater blackarticle color amount

check stock sweater black 4

email me for this tool:[email protected]


• Modularized and keyword-driven method for automated testing

• Particularly suited for large and complex projects

• High degree of agility, even when used in traditional projects

• High level of automation, 95% or more is the norm

• Puts the focus on test design, less on technology

"Action Based Testing"

High Level Test Design - Test Development Plan

Objectives

Test Module 1

Test Cases

Test Module 2 Test Module N

Actions

. . .

AUTOMATION

Objectives Objectives

interaction test business test

define the "chapters"

create the "chapters"

create the "words"

make the words work

Test Cases Test Cases

window control valueenter log in user name jdoeenter log in password car guy

window control property expectedcheck property log in ok button enabled true

user passwordlog in jdoe car guy

first last brand modelenter rental Mary Renter Ford Escape

last totalcheck bill Renter 140.42

Action Based Testing

Test Module Sections

Test Module

Objectives

Initial - setup

Test cases

Final - cleanup


Variables and expressions with keywords

• This test does not need an absolute number for the available cars, just wants to see if a stock is updated

• As a convention we denote an assignment with ">>"• The "#" indicates an expression

TEST CASE TC 02 Rent some more cars

car availableget quantity Chevy Volt >> volt

first name last name carrent car John Green Chevy Voltrent car Jane White Chevy Volt

car expectedcheck quantity Chevy Volt # volt - 2


ACTION DEFINITION check balanceuser

argument customerargument amount

window control valueenter balance inquiry last name # customer

window controlclick balance inquiry view balance

window control expectedcheck balance inquiry balance # amount

Re-use actions to make new actions

• In the below example we make a new action• Existing actions are strung together to create new ones with a

broader scope• Often steps in low level tests are re-used to create these action

definitions

:customer amount

check balance Smith 223.45check balance Jones 0.00check balance James -330.45

:

use many times in tests:

define in one place:


Example action implementation in Python

# get table object, column number and column countwindowName = LIBRARY.NamedArgument("window")tableName = LIBRARY.NamedArgument("table")columnName = LIBRARY.NamedArgument("column")table = ABT.OpenElement(windowName, tableName)column = table.GetColumnIndex(columnName)rowCount = table.GetRowCount()

# check the sort order, row by rowprevious = table.GetCellText(0, column)for i in range(1, rowCount): current = table.GetCellText(i, column) if current < previous : LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0) return previous = current

LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)

The following action script (in Python) will verify whether the rows in a table are sorted:

find the table in the UI

if a value is smaller than before, fail the test

if all rows are ascending, pass the test

get arguments from the test line

def action_checkSortOrder():


Using the new action

• By keeping an action generic it can be applied for a variety of situations

• Some examples of using "check sort order":

window table columncheck sort order view orders orders table ID

window table columncheck sort order annual results regions revenue

window table columncheck sort order inventory cars price

window table columncheck sort order registration students last name


"Lead Deputy" Testing

• For "multi station" testing, when multiple machines have to participate in a test in real-time

• For example if a supervisor needs to approve a withdrawal in a bank teller system

• Can be "sync" and "parallel" (with a rendezvous point)

acc nr amount tx id

withdraw 123123 10,000 >> txname

use deputy supervisortx id amount

approve # tx 10,000

use lead

"bank teller"

"supervisor"

"bank teller"

Multiple System Access

SystemUnder Test

Action Automation

APIaccess

servicesaccess(REST etc)

UIaccess

databaseaccess

Test Modules


Detailed and unfocused tests are hard to automate

Step Description Expected

step 16 Open http://www.bigstore.com The "BIG Store" main page is displayed, with a "sign in" link

step 17 Click on "Sign In", upper right corner A sign in dialog shows, the "Sign in" button is disabled

step 18 Enter "johnd" in the user name field The "Sign In" button is still disabled

step 19 Enter "bigtester" in the password field Now the "Sign In" button is enabled

step 20 Click on the "Sign in" button The page now shows "Hello John" in the upper right corner

step 21 Enter "acme watch" in the search field The "Search" button is enabled

step 22 Click on the "Search" button 5 watches of Acme Corporation are displayed

step 23 Double click on "Acme Super Watch 2" The details page of the Acme Super Watch 2 is displayed

step 24 Verify the picture of the watch The picture should show a black Acme Super Watch 2

step 25 Select "red" in the "Color" dropdown list The picture now shows a black Acme Super Watch 2

step 26 Type 2 in the "Order quantity" textbox The price in the right shows "$79.00 + Free Shipping"

step 27 Click on "Add to cart" The status panel shows "Acme Super Watch 2" added

step 28 Click on "Check out" The Cart Check Out open, with the 2 Acme Super Watches

The above test is hard to automate in a maintainable way. Not even the smartest engineer can "fix" a test design that is not suitable for automation.


First step: breakdown in test modules

• Straightforward Criteria– Functionality (customers, finances, management information, UI, ...)– Architecture of the system under test (client, server, protocol, sub systems,

components, modules, ...)– Kind of test (navigation flow, negative tests, response time, ...)

• Additional Criteria– Stakeholders (like "Accounting", "Compliance", "HR", ...)– Complexity of the test (put complex tests in separate modules)– Technical aspects of execution (picture checks, special hardware, multi-station, ...) – Overall project planning (availability of information, timelines, sprints, ...)– Risks involved (extra test modules for high risk areas)– Ambition level (smoke test, regression, aggressive, …)

• Business or interaction– Business tests: does the application reflect the business well– Interaction tests: can a user (or another system) interact with the application– mixing business and interaction is one of the most common, and one of the most

fatal, "anti patterns" in automated testing


• Business objects– examples: cars, invoices, locations, etc– tests focus on life cycles– a majority of tests tend to fit in this category

• Business flows– end-to-end workflow style test, – example: create, fulfill, pay and close an order

• "Other tests"– functions and features, like premium calculation or PDF output– administration, users, security, authorizations– graphics– technologies, protocols, ...– customization, extensibility– . . .

Example of breakdown considerations


Example Top Level StructureProject create, update, delete/close

copy, movecategorize, enumerate, identifyconvert, serialize, export/import, ...

UI, dialogs, forms, pagesinput (validation, defaulting, dependencies)flows (primary paths, alternate paths)keyboard shortcuts, keyboard controls, ...

. . .

"Cars"

Lifecycles, data operations

Interaction

Functions and Features

Technologies, protocols, controls

Data (handling, quality, ETL, ...)

Security, authorization, admin

Graphics, multi-media, charts, ...

Interoperability

Customizing, extensibility

Business Flows

Concurrency, race conditions, ...

Business Objects

processes, transactions, end-to-end, day in the life,combinations of flows, ...

calculations, analyses, PDF output, ...

"Customers"

. . .


What about existing tests?

• Compare to moving house:– some effort can't be avoided– be selective, edit your stuff,

• look at the future, not the past– first decide where to put what, then put it there– moving is an opportunity, you may not get such chance again soon

• Follow the module approach– define the modules and their scope as if from scratch– use the existing test cases in two ways:

• verify completeness• harvest and re-use them for tests and for actions

– avoid porting over "step by step", in particular avoid over-checking


Scope of a test module

• Always know the scope of the test module

• The scope should be unambiguous

• The scope determines many things:– what the test objectives are– which test cases to expect– what level of actions to use– what the checks are about and which events should generate a

warning or error (if a “lower” functionality is wrong)

• In other words: keep your eye on the ball . . .


Tip: Use right level actions

project taskcheck task in project Drill Assembly Plan of Approach

Better to have a business level action and hide the details in an action definition:

But do show details when it matters. The example below is too high level, requires drill down into action definition to understand what is tested.


Stable and Robust Automation

• Test your automation– develop separate tests for the actions and interface mappings themselves

• Identifying properties of UI elements– a lot of time is spent on identifying and re-identifying UI elements

• "active" timing– wait based on conditions in the system under test, avoid fixed “sleeps”– if needed request developers to provide hooks

• Testing and automation friendly system designs– with well-defined components, services, tiers, etc

• Keep an eye on the test design– unsuccessful automation virtually always can be traced back to poor test design


Mapping an interface

• An interface mapping (common in test tools) will map windows and controls to names• When the interface of an application changes, you only have to update this in one place• The interface mapping is a key step in your automation success, allocate time to design it well,

in particular naming and choosing identifying properties

INTERFACE ENTITY libraryinterface entity setting title {.*Music Library}

name class labelinterface element title text Title:interface element artist text Artist:interface element file size text File size (Kb):

name class positioninterface element playing time text textbox 4interface element file type text textbox 5interface element bitrate text textbox 6

name class positioninterface element music treeview treeview 1


Hidden Identifying Property


Mapping the interface(style we use in our product)

• An interface mapping will map windows and controls to names• When the interface of an application changes, you only have to update this in one place• The interface mapping is a key step in your automation success, allocate time to design it well


• Testability should be understood as a "must have" requirement– first question in a development project: "how do we test this?"

• Define (assign) identifying properties for any interface element (control, HTML element, etc)

• Have hooks for timing– avoid "hard sleeps" at all costs

• Have white-box access to anything interesting:– input data (ability to emulate)– output data (what is underlying data being displayed)– random generators (can I set a seed?)– states (like in a game)– objects displayed (like monsters in a game)– pay attention to security, don't leave hacking doors open

• A well-designed system, with tiers, components, services, is also very helpful for test design and automation

Testability, some key items


Test design and automation in agile

High Level Test Design (organization

of test modules)

Test Module Development

Interface Definition

Action Automation

Test Execution

Sprint ProductsProduct Backlog

Test re-use

Automation re-use

product owner team prod owner

& team

User storiesDocumentation

Domain understanding

Acceptance CriteriaPO Questions

SituationsRelations

Agile life cycle

Test development

Main Level Test Modules

Interaction Test Modules

Cross over Test Modules


Pre-conditions for Sprints

• Agreement on the approach:– is testability a requirement for the software?

• Understanding of subject matter, testing, automation, method, best practices, etc

• Automation technology available and working– UI, non-UI, custom controls, graphics, embedded, etc

• A plan on how you to deal with data and environments

To discuss an approach, consider daily "sit down" meetings with some or all members to coach and evaluate- an end-of-day counterpart to the early-morning "stand up" meetings- short and friendly, not about progress and impediments, but about practices and

experiences with them (like "what actions did you use?")- a few meetings may suffice

TIP


Facilitation outside sprints, example

Sprint

Have a facility (internal or outsourced) to help the agile life cycle• keep up, catch up, continue beyond "done"• one outsourced facility can support multiple teams, smooth out peaks and valleys• allow sprint teams to focus on development priorities• team is responsible, and decides if, what and when to outsource

Sprint

Test Definition & Development

TEAM

Additional Test DevelopmentAdditional Automation

FACILITY

Test DevelopmentAutomation

Automation

Test Definition & Development

Automation


• QA needs to keep up with the rest of the team

• Test design is a key driver for automation success/failure

• Testability, as a team responsibility, is a core requirement

• Organizing the testing well in the sprints gets more of it "done"

• Having an agile facilitation model can support the QA's to stay in sync

Summary