automated softwaretestingmagazine december2011

.......

An AUTOMATED TESTING INSTITUTE Publication - www.automatedtestinginstitute.com

Software teSting MAGAZINE December 2011 $8.95

Cost Effective Test ManagementDevelop a HomeGrown

manaGment Solution

AutomAtedAutomAted

Surviving Obfuscation

How to automate obfuScateD coDe

e s t i n gAutomated AM obile

teSt manaGement GettinG crowDeD: CrowdsourCing inspires new Tool

7230 Lee Deforest Drive, Suite 100 • Columbia, MD 21046 • 301-854-2215 • www.spt-inc.com

Through the SPT PromiseTM, we’ve made a commitment to value our employees, to be understanding and supportive of individual needs and to provide opportunities for career growth. We value and appreciate our employees for the work they do for us as a company and for the clients’ mission.

SPT partners with client organizations to design, develop, implement and deploy high-end, custom, technology-based solutions by integrating existing and emerging concepts. Positions are available for professionals looking for exciting and rewarding challenges with significant impact to ongoing missions. Openings include: • Software Engineers • Systems Engineers • Configuration Managers • Systems Administrators • Systems/Software Testers

Most positions require an active TS/SCI clearance with full scope or CI polygraph. SPT is EOE, M/F/D/V.

© 2011 Software Process Technologies, LLC. (SPT). All rights reserved.

If you’re looking to thrive in a diverse and vibrant professional team, we’re looking for you! Submit your resumé to [email protected] today!

Our promise is to ensure we are committed to continued exellence and success for our clients’ mission. We are devoted to doing what is right for our company, our partners, and our employees to best serve our community and country.

A Sotera Defense Solutions Company

AutomAtedSoftware teStingDecember 2011, Volume 3, Issue 5

Automated Software Testing Magazinewww.automatedtestinginstitute.comDecember 2011 3

Contents

ColumnS & DepartmentSEditorial 4

authors and EvEnts 6 tEstKit is hot! 42

i ‘B’log to u 38the Mobile Evolution

the Kit is Koming

How mobile technology effects test management and automated test execution.

Learn what this year’s conference has to offer.

Enjoy a testing comic strip that is funny and informative.

Learn about AST authors and upcoming events.

The Automated Software Testing (AST) Magazine is an Automated Testing Institute (ATI) publication.

For more information regarding the magazine visithttp://www.astmagazine.automatedtestinginstitute.com

opEn sourcEry 8test Management is getting crowdedHow Crowdsourcing and the Mobile Revolution Have Inspired The Birth of a New QA Tool

go on a rEtwEEt 40Read featured microblog posts from the web.

Cover StoryMoBilE autoMatEd tEsting 19For many, mobile devices are fast becoming the computing environment of choice. With so many different devices and new ones emerging all the time, it’s infeasible to conduct sufficient testing for each, manually. But how will test automation need to adjust from the PC to mobile? Read this article to find out. By Patrick Quilter

featureSoBfuscation: avoiding friEndly firE in thE BattlE for sEcurity 12How do we handle application dynamics introduced by security measures such as obfuscation? Read this article to get strategies for

handling this challenge that results in modern computing. By Dion Johnson

“intEgrating Ms officE for cost-EffEctivE tEst ManagEMEnt 26Need a cost-effective test management solution? Read this article as Tom Wimsatt identifies key test management features, as well as a

step-by-step approach for using common Windows desktop software for creating a tool customized to your needs. By Tom Wimsatt

4 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011

Many computing technologies have been introduced over the years and with them, new approaches for test implementation. Linda Hayes wrote a great article called the “Evolution of Automated Software Testing” that was published in Volume 1, Issue 1 of the Automated Software Testing Magazine; her article chronicled the evolution in computing from Mainframe dominance all the way up to the Internet taking center stage. Each evolutionary phase was accompanied by a test automation evolutionary response that highlighted the changes to how test automation was conducted in order to keep pace with the shift in the computing world at large.

The computing world is at the beginning of a new computing sea change, and I have no doubt that there will need to be a major evolutionary response to this change in the world of test automation. The sea change will move us further into the world of mobile technology, and the move is likely to happen faster than any of the other computing paradigm shifts to date – evidenced by the faster than expected depletion of IPv4 addresses that has forced us quickly into the world of IPv6 (see AST Volume 3, Issue 1).

Mobile technology has introduced new devices, new types of software and a whole host of new challenges for testing and test automation. In the past we were largely concerned about adjusting to a world in which setting up lab environments to test multiple versions and configurations of a handful of browsers was the challenge. That’s nothing compared to the 100s, maybe 1000s of mobile devices and configurations that may be running your organization’s mobile application (app)! We need to quickly come to grips with what this means for testing and the tools that aid our testing.

One thing this technological shift means is that we need to come to grips with the fact that structured, efficient

test management is no longer just a luxury. Many organizations have gone without any real approach or tool for test management, with the excuse that such an approach or tool would be overkill for the way they do testing. With increased test complexity, however, and teams that may span multiple organizations, and/or continents, test management is becoming increasingly important. Another item that we must come to grips with is how automation of our test execution will be changed by the mobile evolution. This issue of the Automated Software Testing Magazine is the first of many to come that focuses on these items and more.

In our cover story entitled, “Mobile Automated Testing”, Patrick Quilter provides a look into how the explosion of mobile device usage has affected the current day implementation of test execution automation. He discusses how traditional tools and strategies can be used for mobile app automation with a little assistance from simulators, but also addresses test execution automation on the actual mobile devices. In addition,

Quilter discusses tools for managing mobile device test cases.

Next, Tom Wimsatt reveals an approach for implementing a homegrown test management solution in his article entitled, “Integrating MS Office for Cost-Effective Test Management”. In addition to identification of key test management features, Wimsatt provides a step-by-step approach for using common Windows desktop software for creating a tool customized to your needs.

In the second feature, Dion Johnson (yours truly) shifts gears slightly to offer strategies for handling another challenge that results in modern computing: the challenge posed by security measures such as obfuscation. Obfuscation often results in dynamic UI properties that render an application non-automatable. In this article entitled, “Obfuscation – Avoiding Friendly Fire in the Battle for Security”, he provides useful mechanisms for creating stable automated scripts on an unstable application.

the Mobile Evolutionby Dion Johnson

editorial

...efficient test management is no longer just a luxury.

Training That’s Process Focused Yet Hands On

Software Test Automation Trainingwww.training.automatedtestinginstitute.com

Public and Virtual Training Available

Come participate in a set

of test automation courses that address both fundamental

and advanced concepts from a theoretical and hands on perspective. These courses focus on topics such as test scripting concepts, automated framework creation, ROI calculations and more. In addition, these courses

may be used to prepare for the TABOK Certification

exam.

Public coursesSoftware Test Automation Foundations

Automated Test Development & ScriptingDesigning an Automated Test Framework

Advanced Automated Test Framework Development

Virtual coursesAutomated Test Development & ScriptingDesigning an Automated Test Framework



Managing Editor Dion Johnson

Contributing EditorDonna Vance Edward Torrie

Director of Marketing and Events Christine Johnson

A PUBLICATION OF THE AUTOMATED

CONTACT US AST Magazine [email protected]

ATI Online Reference [email protected]

tom wimsatt is an ISTQB/ASTQB Fully Certified Advanced Tester with Software Process Technologies, Inc. A Sotera Defense Company in Columbia, Maryland. He has been in the field of testing for more than 20 years beginning in the Parts Analysis labs with NASA at Goddard in Maryland. The last ten years he has been concentrating on testing software specializing in tools and automated test development. He can be reached at: [email protected]

Who’s In This Issue?patrick Quilter Jr. has earned a Bachelor of Science in Computer Science & Mathematics, a graduate certificate in Homeland Security Technology Management, and is in the process of acquiring a Master degree in Technology Management. Patrick has been an automation architect for a number of large companies such as Travelers Insurance, Computer Science Corporation (CSC), and GEICO. While at Travelers Insurance, Patrick received the company’s

Outstanding Achievement Award for an R&D solution that programmatically generated business test cases for automated regression execution. Patrick has also worked as a test consultant providing tools architecture and process guidance for both commercial organizations and federal government agencies. After 10 years of experience and continued development, he is in the process of patenting a cloud based, automation framework that’s been configured with many popular automation engines. He is currently employed as a lead automation architect while also assisting organizations through Quilmont LLC, a test process consulting company and cloud based, test automation service provider.

authors and events

AutomAtedSoftware teSting

dion Johnson has several years of experience in providing IT services to both government and private industry. With a Bachelor of Science degree in Electrical Engineering, he has spent much of his professional career as a consultant in the areas of quality assurance (QA), quality control (QC), software process improvements, requirements analysis and software test automation. As a regular conference speaker

and presenter, Dion has delivered award winning and highly acclaimed presentations at many of the most prestigious industry conferences, including the StarEast and StarWest International Conference on Software Testing, Analysis and Review, and the Quality Assurance Institute Conference. He also teaches college level classes relative to testing and test automation, and has several published articles in various IT publications.

ATI and Partner Events

January 1 2012TestKIT 2012 Conference Call For Presentations Opens

April 1, 20124th Annual ATI Automation Honors Awards Nominations Beginwww.atihonors.automatedtestinginstitute.com

Fall 2012TestKIT 2012 Conference

www.automatedtestinginstitute.comDecember 2011 7Automated Software Testing Magazine

The KIT is ComingThe KIT is Coming

Fall2012

http://www.testkitconference.com

K


test Management is getting crowdedHow Crowdsourcing and the Mobile Revolution Have Inspired The Birth of a New QA Tool

A few years ago, the open source alternatives for test management were pretty sparse. The bug tracking portion of test management didn’t lack for automated tools, but not very many tools filled the needs for features such as test case administration, assigning work, status tracking, reporting, and other key elements of test management. This is far from being the case now, as there are several viable open source solutions for test management. Among the most popular tools are TestLink and XStudio. XStudio isn’t completely open source, but is free and has some open source elements. There are many more open source test management tools besides these two, however, including RTH (Requirements Testing Hub), Salome-TMF, and Testopia. Testopia is an open source test case management extension for Mozilla’s open source bug tracker, Bugzilla. Given this existing list of available tools, we were intrigued to hear of a new test management tool known as CaseConductor entering this already crowded space. A company known as uTest announced in April that they’d be developing a new test case management system (http://www.utest.com/press/utest-announces-partnership-mozilla-build-test-case-management-system), an announcement that came to fruition on October 25, 2011 as they announced the release of the beta version of this system known as CaseConductor (http://blog.utest.com/utest-and-mozilla-release-beta-version-of-caseconductor/2011/10/). These

announcements were not only intriguing due to the fact that several open source test case management tools already exist, but also because uTest indicated in these announcements that they were partnering with Mozilla to produce this new offering. That’s right! They were partnering with the maintainers of Testopia, an already available open source test management tool. AST reached out to uTest to get some answers.

uTest describes themselves as “the world’s largest marketplace for software testing services.” With a community of 45,000+ professional testers from 180 countries around the globe, they are founded on a crowdsource testing business model that involves a number of different testers from different places as opposed to in-house consultants. Testers sign-up with uTest and complete a technical profile indicating their technical expertise, along with the platforms, environments and/or devices they have access to. These testers are then enlisted for testing assignments as necessary when their profiles meet the needs of

uTest customers seeking to have their software tested in diverse, realistic environments. Matt Johnston, CMO of uTest, referred to this type of testing as “in the wild testing”. It was this “in the wild testing” along with the large number of clients involved in mobile application (app) development that helped to lead to the decision to create a new test management tool.

Johnston explained that once it was clear that a test management tool

was necessary to better administer the testing process, they polled nearly 1000 of their customers to get a feel for what solutions were being used. The feedback revealed that the test management market was extremely fragmented. Small companies often didn’t use test management tools at all, while larger companies felt their solutions – whether commercial or open source – had serious deficiencies. Ultimately, while most of their customers were happy with the bug tracking systems they each employed, there was little consensus on the most satisfactory test case management tool. When asked specifically about tools such as TestLink and XStudio, Johnston indicated that his customer base gave little feedback on these at all.

Open source tools such as TestLink and XStudio were, however, evaluated by uTest, along with many commercial tools and, according to Johnston, some key elements necessary for uTest’s approach for doing business were missing. One of the key elements that

open Sourcery


was lacking was the ability to adequately handle remote collaboration. For any given testing project, uTest may have testers from all around the world plus testers from a client’s company (for which the testing is being conducted) involved in test implementation. Tests or even test steps may need to be assigned to various testers in different locations, testers may need to be able to communicate with one another, and status may need to be appropriately tracked. Another missing element that the CMO identified was the ability to appropriately handle a more fluid, agile style of testing. uTest has a lot of customers that develop mobile apps, which Johnston says requires a different testing approach than that employed for larger, more monolithic applications.

For these reasons and others uTest felt that a new tool was necessary, and upon working with uTest customer Mozilla, the makers of the popular Firefox browser, it was found that they had many of the same test management needs that apparently weren’t being filled by Testopia or any other tool. uTest and Mozilla decided to work together to create a new test management solution that culminated into what is now known as CaseConductor.

According to the announcement of

the beta release, CaseConductor was developed with the following three core values:

1. Community-oriented – Allow dynamic distribution of test cases across different user roles. With CaseConductor, your testing team can be a combination of your internal QA members, open source testers and uTesters – all of whom can execute test cases simultaneously. Essentially, you can create an unlimited testing team using a single interface.

2. Environment coverage – In today’s software ecosystems, almost every product needs to support an enormous testing matrix consisting of OS versions, device makers, languages, locations and more. CaseConductor’s design around environment attributes allows testers to easily and visually specify complex compatibility test metrics and select those high priority test cases for quicker and more targeted compatibility testing. The goal for CaseConductor was to enable testing managers to improve their product coverage when doing full

regression tests.

3. Simple interface – The producers of CaseConductor set out to keep the tool as accessible and simple as using Word or Excel since those are the tools being used to manage test cases for a big chunk of the market. Test cases libraries have the potential to be quite large, but the UX has been streamlined to keep assignments and results on one screen.

Currently, there is no integration with functional automated test tools, but according to Johnston, conversations have already begun with organizations that are interested in developing integrations with it. AST will keep an eye out for this and will also be listening to the automation community to find out how well you all receive CaseConductor into the fold of test management solutions.

open Sourcery

For more information on CaseConductor, visit the

CaseConductor repository at https://github.com/mozilla/

caseconductor-platform/

wait! there’s More!

Training That’s Process Focused Yet Hands On

Software Test Automation Trainingwww.training.automatedtestinginstitute.com

Public coursesSoftware Test Automation Foundations

Automated Test Development & ScriptingDesigning an Automated Test Framework


byDion Johnson

ObfuscatiOnavoiding friendly fire in the battle for security

ObfuscatiOnavoiding friendly fire in the battle for security

December 2011www.automatedtestinginstitute.comAutomated Software Testing Magazine12

“ObfuscatiOn is nOt the first technique emplOyed tO prOtect cOde frOm thOse that wish tO reverse engineer Or tamper with it, but is instead anOther weapOn used in a lOng battle between

prOgrammers and thOse that seek tO use Or access the prOgrams in unintended, undesirable ways.”

Obfuscated cOde - Obfuscated code is source or machine code that has been made difficult to understand for humans. Programmers may deliberately obfuscate code to conceal its purpose (security through obscurity) or its logic to prevent tampering, deter reverse engineering, or as a puzzle or recreational challenge for someone reading the source code.

http://en.wikipedia.org/wiki/Obfuscated_code

Cyber security and the ability to protect code from being reverse engineered have become increasingly important as the threats of a system being reverse engineered or compromised have exploded over the years. It is widely accepted that the risk of these threats cannot be eliminated, but can be greatly reduced by making application code more obscure and

costly to decipher in an effort to deter the act of reverse engineering. Increasingly, a method known as obfuscation is being employed to make the code more obscure.

In general, obfuscation is the practice of making something more confusing, unclear, and/or difficult to understand. This same definition applies to obfuscated code as well. Obfuscated code is code that has been modified, typically by a program known as an obfuscator, to be more convoluted and difficult for humans to understand and follow its behavior. The obfuscated code is functionally indistinguishable from the original code, but the details surrounding the implementation of those functions have changed.

Obfuscation is not the first technique employed to protect code from those that wish to reverse engineer or tamper with it, but is instead another weapon used in a long battle between programmers and those that seek to use or access the programs in unintended, undesirable ways. Unfortunately, it is often the software automator that gets caught in the cross fire from this never ending battle, particularly when the obfuscation affects an application’s user interface (UI). When obfuscation affects an application’s UI, test automation is severally impacted because these tests depend on a certain degree of consistency and/or predictability in how UI elements are referenced. Turning to Figure 1 allows us to look at three commonly exercised elements: a Username textbox, Password textbox and a Login button. Automated tests identify these three elements by property values that have been assigned to them in the application code.

For example, each object may have an ID property that defines it. The username textbox may have an ID equal to ‘uname’, the password textbox may have an ID equal to ‘pword’, while the Login button may have an ID equal to ‘Login’. These property names are descriptive and probably fairly constant, thus helping the page remain relatively automatable and maintainable. An obfuscated UI will likely change these IDs in a manner that will negatively affect their ability to be consistently accessed by automated tests.

Table 1 and Figure 2 reveal how the ID properties may be affected by obfuscation. Not only do the properties follow no set pattern, they may change from build to build or as often as each time the application is invoked.

13Automated Software Testing Magazinewww.automatedtestinginstitute.comDecember 2011


This type of UI instability has brought down many test automation efforts, because it makes the test automation un-maintainable, particularly as the automated test bed grows.

There are several proven techniques, however, that may be employed to provide cover for test automation when development injects obfuscation into the coding battles against hackers and the like, including:

Figure 1: Login Screen

labels

ID property = ‘uname’

ID property = ‘pword’

ID property = ‘Login’

Object ID First Accessed ID Second AccessedUser erlwlkjf4 askflaf6

Password pewtlj90 aasksj2Login alsjfafs3 aljfljlaj5

Table 1: Obfuscated Properties

• Pre-obfuscation automation

• Label-based identification

• Obfuscation map utilization

• Image-based automation

Pre-ObFUScATIOn AUTOmATIOn

Probably the most obvious solution to dealing with obfuscated code is to… well…not deal with obfuscated code. Since obfuscated code is typically the product of a baseline set of code being run through an obfuscator, it stands to reason that automation be performed on the code prior to it being obfuscated. Some take umbrage with this solution because it is testing of a code base that is different from that which goes into production. In his blog, Nick Olivo, frequent ATI contributor, takes aim at this criticism:

“I would think that obfuscation would impact any tests that were run after the obfuscation process. So is it safe to say that companies who employ unit tests must be running those tests prior to the obfuscator running? If so, should the manager be concerned that the unit tests were run against non-obfuscated code?

Obfuscation would also prevent any other type of post-build white box testing; so any tools I've written that call methods directly from dlls or invoke web service calls would be stopped dead. So what's the best solution here? Is it to do the bulk of your automated testing against non-obfuscated code and then do manual sanity checks against an obfuscated build? Or is there a better way that I'm not seeing?”

(http://autonomicon.blogspot.com/2009/02/obfuscation-question.html)

LAbeL-bASeD IDenTIFIcATIOn

Obfuscation is typically a process that is transparent to the user of an application. Therefore, while much of the code is changed during the obfuscation process, items that we may reasonably expect to remain stable are text-based UI labels, along with their relationship to fields on a page. For example, Figure 2 has two text boxes, each preceded by a label that provides information to the user on how those textboxes should be used. The labels are not necessarily directly linked to fields that they describe, but there is normally a pattern for assigning labels to fields. Once

Figure 2: Obfuscated Login Screen

labels

ID property = ‘erlwlkjf4’

ID property = ‘pewtlj90’

ID property = ‘alsjfafs3’


the pattern is identified it can be assessed and used for object identification by an automated test at run-time.

Three approaches for using labels to uniquely identify fields are:

1. Location-based identification2. Cell-based identification3. Property-based identification

Location-based identification initially seems closely linked to the volatile coordinate-based automation approach that we’ve been taught to steer clear of. Despite appearances, however, it is not the same thing, but is instead a much more stable approach to identifying application fields. Location-based identification using labels is based on the stability of the label names and the pattern used by programmers for coupling labels with fields, which is a more robust approach than using the actual position of an object on a screen. Location-based identification first requires that your test automation approach provide a mechanism for obtaining all fields on a page into an array or collection, based on the order in which those objects appear on the screen. For web pages, the “all” document object model (DOM) method makes this possible. If you are using xpath for web object identification, “//*” will do the trick. Once the objects are stored in an array, a loop may be established to traverse that array or collection in search of a label element with the desired label text. Then, based on the pattern identified for coupling labels with fields, the desired field can be identified. For example, the application screen represented in Figure 2 has its fields preceded by labels that represent that field, evidenced by the fact that the ‘Username’ label precedes the textbox used for entering a username.

Thus, gathering all of the page’s objects into an array named pageElementArray, would produce an array or collection with the following items:

pageelementArray[0] = Username label

pageelementArray[1] = erlwlkjf4 textbox

pageelementArray[2] = Password label

pageelementArray[3] = pewtlj90 textbox

pageelementArray[4] = alsjfafs3 button

Upon looping through the array or collection of objects and locating the desired label, the automation loop can be set to return the very next item in the array or collection that is a textbox. So, if the automated test needs to access the ‘Password’ textbox without using the dynamic ID property that is currently equal to ‘pewtlj90’, it would simply loop through the array of objects and find the ‘Password’ label. Upon find the ‘Password’ label at array item 2, the loop - which has been set up to return the very next array item that is a textbox - will return array item 3, which is the ‘pewtlj90’ textbox. Provided that the label name text remains constant, the associated field can be found even if the field (and label)

is moved around the screen and/or its properties change.Cell-based identification is useful when the page objects are contained within a table. Provided that each label/field pair is contained within cells separate from other label/field pairs, the cell configuration may be used for uniquely identifying a field. The first step in making this work is to programmatically locate the label that is associated with the desired field on the screen, and then get the table cell in which the label exists. Based on the pattern used in the application for storing label/field pairs, the label cell information will be used for uniquely identifying the desired field. For example, if the label typically exists in the same cell as its associated field, then code can be written to get the child textbox of the same cell element in which the label exists. If the field typically exists in the same row, but an adjacent column of the label, as illustrated in Figure 3, code can be written to get the child textbox element of the adjacent cell.In a roundabout way, the cell-based approach is dependent on location, because you’re expecting the field to be in a location relative to its associated label. The third label-based approach, property-based identification, is the only label-based approach that is not dependent on location at all. This approach is most useful when the label element has an attribute that is directly tied to an attribute of its corresponding field. For example, a label may have a NAME attribute that is identical to the ID attribute of its corresponding field, as illustrated in Figure 4. In this situation, automated test code can be written to programmatically obtain the desired label based on its label

Figure 3: cell-based Object Identif ication

table row 1, column 1

table row 1, column 2

table row 2, column 2table row 2,

column 1

Figure 4: Property-based Object Identif ication

ID property = ‘erlwlkjf4’

ID property = ‘pewtlj90’

ID property = ‘alsjfafs3’

NAME property = ‘erlwlkjf4’

NAME property = ‘pewtlj90’


text, and then get the NAME property associated with that label. The automated code can then be instructed to find the field with an ID property that is equal to the label NAME property that has been captured. Upon identifying the appropriate label-based approach, functions and/or methods can be created to perform the necessary computations in a way that is seamless to the design time test automation process. The automators would simply need to use a field’s associated label text for identification of that field as opposed to properties directly related to the desired field.

ObFUScATIOn mAPS

At times, when code is passed through an obfuscator, in addition to outputting the obfuscated code an obfuscation map may also be produced. This map provides a direct relationship between old property values and the new property values of application fields and other elements. If an obfuscation map is not produced, you may be able to request that the development team modify the obfuscator to produce such a resource, because it could be invaluable. With an obfuscation map, a utility can be written to synchronize the object information used by the automated scripts with the new object information in the application prior to or during script execution.

ImAGe-bASeD AUTOmATIOn

The Test Automation Body of Knowledge

(TABOK) Guidebook describes Image-based automation as follows:

…often based on Virtual network computing (Vnc), [an image-based automation tool} relies on image recognition as opposed to object recognition (context Sensitive) or coordinate recognition (content Sensitive).

VNC-based tools follow the two-computer system, with the automated tool residing on the client machine and functioning as the VNC client. The server machine on which the AUT is installed will run a VNC server that communicates with and transmits images to the VNC client. These tools, therefore, recognize application objects based on analysis of the transmitted images.This mode is more closely related to context sensitive automation than content sensitive, because it is not completely coordinate-based. These images may still be located if they are moved to a different screen position or if their internal properties change. Thus, image-based automation can be useful for handling applications that utilize obfuscation.

cOncLUSIOn

Obfuscation is a battle between software producers and their enemies, and if testing suffers, ultimately the software producer suffers. Follow the approaches defined in this article to avoid being a casualty of obfuscation or any other development approach that injects instability in the application UI field properties.

Community Comments Box

Announcements & Blog Posts

Automation Events

Contribute

Content

Today

As a registered user you can submit content directly to the site, providing you with content control and the ability to network with like minded individuals.

Learn more today at

http//www.about.automatedtestinginstitute.com


With Traditional Tools & Beyonde s t i n gby Patrick Quilter

Has consumer demand already chosen mobile devices to replace the personal computer? Will traditional webpage design with drop downs and text boxes lose ground to mobile touch screen data entry? Will everything we know about creating automated tests for applications change because of mobile devices? Don’t abandon your old automation frameworks just yet. You’ll still need them to expand into the untamed world of automated mobile testing. Mary Meeker, a renowned Internet analyst with Morgan Stanley, highlighted the significance of mobile consumer demand in her latest “State of the Internet” report. Meeker predicted that within the next five years, “more users will connect over mobile devices than desktop PCs.” Since 2010 the trend of smartphone shipments outpacing PC shipments continues to surge and gives credence to Meeker’s forecast (Ingram, 2010). Anyone can see that a multitude of organizations have already embraced this technology swing by developing applications explicitly to establish a mobile presence. Recognized companies continue to offer free or inexpensive apps for both business and entertainment purposes.

While having a mobile presence is significant for organizations, it is Also becoming imperAtive to hAve stAble ApplicAtions And mobile websites. PhoCusWright conducted an 800 participant survey to evaluate user tolerance for buggy mobile software. The survey revealed that 37% of smartphone users indicated that they would be less likely to use that particular application again and 28% indicated that they would seek out competitor applications (Leggatt, 2010). These facts illustrate the importance for sound application development practices and the ability to automate test cases. With so many different devices and new ones emerging all the time, it’s infeasible to conduct sufficient testing for each, manually. Automation’s luster is the ability to execute large volumes of repeatable scenarios that a person could not undertake manually.

“Simulators on Windows or Macs allow teams to employ traditional automation engines like Quick Test Professional or Selenium”

mobile automation

Automated AM obile


To Simulate or Not to SimulateA typical beginning for application development teams journeying the path of mobile testing is to create automated scripts that execute on desktop simulators. A simulator has the same look-and-feel of a mobile application displaying on a device, however, it executes on a desktop operating system. Teams utilize a simulator by constructing business flow, user designs, and verifying basic functionality of their applications. Simulators on Windows or Macs allow teams to employ traditional automation engines like QuickTest Professional or Selenium to create test cases.

Although using a simulator adds value to an application development practice, verifying functionality is limited. Neglecting to conduct tests on a physical device causes vulnerabilities to application performance. This realization has its drawbacks. For one, the list of popular devices is growing rapidly, and testing must occur for all devices on the list. Secondly, how do application development teams execute test cases outside of the PC-based automation tools? Are there any strategies to handle these

challenges?

What’s the strategy?The most challenging aspect of an automation strategy is devising a practice that optimizes test case execution across multiple devices. Ideally we’d like to craft a single test case that can execute on multiple devices, otherwise we would have to maintain a separate copy of each test case, per device. Test sets (a collection of test cases that compose a test plan) would become maintenance intensive over time without this reuse capability. Mobile devices cause a significant amount of variation for development and testing efforts. First, is the question of whether

software under development is going to be application based or web based? If application based, functionality will vary depending on device platform such as iPhones, Androids, Blackberries, and Windows. Although some solutions are materializing to allow developers the ability to create one codebase that deploys over multiple devices (regardless of iPhone, Android, etc.) the typical approach is to maintain a separate codebase per platform. A savvy tester will see the vulnerability in maintaining multiple codebases. This implies multiple builds to verify.

Device manufacturers and operating systems have the biggest impact on how applications function even across the platform families. For example, the HTC EVO and Samsung Galaxy are both part of the Android family, however, the same application installed on them could function differently. The reason for this is the manufacturer can manipulate how processing, GUI rendering, and form factors affect execution on their devices. Web based functionality is more straightforward because developers and testers converge on a single deployment of code to a server. Configuration settings

determine functionality based on the device a user connects with to the mobile website. Normally, functionality across product families will work the same. Although not a guarantee, testing a mobile website with an HTC EVO and Samsung Galaxy that is running the same version of Android should produce the same results. Testing just one of them would most likely be sufficient. The device carrier is also significant when considering a strategy for both mobile applications and web based software. Network speeds and protocol restrictions are common performance hinderers for mobile software. The key for any organization will be how to turn the strategy into reality.

Luckily consumer demand for mobile

device transactions has spurred the growth of application development service providers. A capable service provider will offer an on-demand testing infrastructure, multiple devices, and automated scripting capabilities.

In terms of mobile testing this type of service is invaluable because it accounts for all major strategic characteristics under one solution. But what specific service provider existing today offers all of this?

Meet Keynote DeviceAnywhereA great example of this type of service provider is a company called Keynote DeviceAnywhere headquartered in San Mateo, California. Keynote DeviceAnywhere helps organizations of all sizes bring their products, services, and IT applications to a mobile platform. Their cloud-based solution hosts a phone bank of over 1000 real devices that are used for mobile application testing, test automation, monitoring, and development worldwide. This is possible through their award winning, end-to-end studio platform that cuts operational expenses

and time associated with testing mobile applications (HP/KDA datasheet, 2009).

Accessing the EnvironmentKeynote DeviceAnywhere users first gain access by logging into a web-based portal that provides a management dashboard, training information, and links to support resources. From there, the user downloads the client-side software onto their individual desktop to experiment with the actual devices. This gives them a window into the Keynote DeviceAnywhere devices hosted in the cloud. A visual representation of the physical device displays once the user acquires a device. Users can license any

A capable service provider will offer an on-demand testing infrastructure, multiple devices, and automated scripting

capabilities.


The KIT is ComingK

if you thought ati’s 2011 event was good, wait until you see 2012.h t t p : / / w w w . t e s t k i t c o n f e r e n c e . c o m


of the hosted Keynote DeviceAnywhere devices upon request. Acquiring devices is an important concept because team members can share and schedule times for when they want to test on particular devices.

Once a device has been acquired, the user can control all aspects of the physical device through mouse clicks and keyboard strokes to mimic the actions as if they had the device in their hand. The user can also manipulate the features of the devices such as buttons, LCD screen, speaker, ringer, battery, power-plug, and data-cables. The Keynote DeviceAnywhere control sliders allow the user to change these setting and pre-condition the device before conducting a test case. Another controller known as the Device Console is also accessible for typing text and performing SMS messaging. This Console can monitor HTML traffic between webpages and the device. The importance of controlling these features is in the fact that mobile test cases have some unique verification steps when compared to PC based applications or webpages. For example, a user may want to see what happens to their application when the phone rings or a text message is received (Keynote DeviceAnywhere, 2010). Accessing the devices through the studio and using the Device Console allows a user to set the stage for these types of scenarios.

To accomplish automation, Keynote DeviceAnywhere provides a module specifically for managing requirements, test case steps, execution tracking, and result reporting. Each of the items are organized within a project and shared between team members for collaborative testing. A user can start creating automation steps by accessing Keynote DeviceAnywhere’s device-agnostic, script-less automation technology (HP/KDA datasheet, 2009). The technology is device-agnostic because when a script is

developed for one device it can be cloned to run on multiple devices. It’s script-less because automation steps are created by dragging and dropping icons which are preset to perform specific actions. Stringing together a series of icons produces a whole test case.

For users that are familiar with traditional web-based automation, the biggest adjustment may be getting accustomed to how objects are represented on mobile pages. Tal Barmeir, CEO of Experitest,

explains that, “existing industry-leading test automation tools are restricted by object recognition limitations. Most tools rely on open and published APIs to obtain the discrete properties of a control. Mobile platforms do not provide the traditional APIs, rendering your favorite tools useless in mobile” (Barmeir, 2011). Unlike web page objects, mobile objects are identified differently. Web-based objects are referenced by property tags such as an object name or ID. A scripter can use tools to break down the various types of objects on a page and associate data with those objects. With mobile application rendering, property tags are not referenced in this way. Instead a user interacts with the screen by anchoring on specific text, icon recognition, and pixel locations. Keynote DeviceAnywhere’s

script-less actions correspond to these screen interaction properties.

Walking through an example, a user developing a script for an iPhone will first acquire the device and is then presented with a window view of the iPhone next to a blank execution grid. This tells the scripting component that the particular implementation is going to interact with an iPhone device. The users will drag-and-drop Keynote DeviceAnywhere actions (icons) onto the execution grid in an order that corresponds to test case steps. The actions available to the user can search for some text on the screen, enter text, click on GUI icons, or validate a checkpoint area located on the screen. The actions are separate steps represented as separate icons. During execution, the actions perform the key strokes and touch screen interaction that a human has with the iPhone. The iPhone’s screens are driven by the actions in the execution grid until the last one has been reached.

Once the test case has been created and saved, the test case can then be cloned for other devices. All of the steps that were created for the iPhone could be enabled for an HTC EVO. The benefit of cloning test cases is that for the most part application flow stays consistent from one device to the next. All action command steps are converted for use on the new device. In some cases, actions that contain specific GUI representations will need updating after the cloning process. Typically these types of updates are minimal and identifying where to make the adjustment is intuitive to the users. This is because the user has the advantage of using the Keynote DeviceAnywhere technology to stay within a consistent framework. Users can also mitigate the amount of updates by incorporating an automation development strategy or best practices within the framework that fits their applications. Trial and error will help identify the best strategy.

Script-less automation: “Script-less means no scripting/

programming in the test automation tool’s native language. This

doesn’t mean there are no scripts involved. Explicitly, this means

while automating test cases, there is no more need to program scripts

for each test case”

(Qualitia, 2011).

Most tools rely on open and published APIs to obtain the discrete properties of a control. Mobile platforms do not provide the traditional APIs, rendering your favorite tools

useless in mobile


Incorporating Framework AutomationFor many applications, using the script-less approach is appropriate especially if functionality changes are minimal between code updates or if the particular application doesn’t have an abundance of data entry. An application heavy with links and button pushes would be covered adequately. This is also a useful approach if the test case suite is small and easily maintainable. Users that have applications with heavy data entry in many business scenarios will naturally want to incorporate industry standard practices such as creating a data driven or keyword driven framework. This type of approach enhances script reuse, quicker maintenance, and offers the ability to create test cases without consuming a Keynote DeviceAnywhere license (the last section provides an example of how this can be accomplished). For those well versed in automation design, the script-less approach may sound like a variation on record and playback. Also restricted access to the scripting language may strike automation developers as a hindrance to their effectiveness. In other words there are some things a seasoned programmer can accomplish more effectively with code

than relying on script-less actions. Not to worry because Keynote DeviceAnywhere offers two ways to interact with their automation engine so that a tradition data framework can be created. For organizations that use HP/Mercury Functional Testing (QTP) and Quality Center, there is a tremendous opportunity to continue utilizing these tools and configure them to work with the Keynote DeviceAnywhere Studio. This allows organizations to expand upon the familiar features of QTP and QC as well as enhance their tester’s skillsets. This option allows a QTP developer to extend their current framework to invoke existing actions in the Keynote DeviceAnywhere Studio. Keynote DeviceAnywhere has created

an add-in that a user can download for this capability. For those that are QTP developers, you know that when launching QTP you have the option to select which add-ins are loaded for your development and execution sessions. After installation, the user just needs to check the box in the add-in selection screen for the Keynote DeviceAnywhere add-in. Once this step is complete, a user can reference any of the actions they’ve created to drive the automation capabilities on any of the devices. Interacting with a specific device is managed in the QTP object repository where every device that is going to be tested is a separate object in the repository. The handshake between the repository and the devices are controlled by a unique ID property. From this point a user can design a framework and separate the automation code from their test case data. As a best practice, a user should create a design where QTP does the repetitive calling of a sequence of Keynote DeviceAnywhere actions while passing parameters to those actions. Keystrokes and data entry can be controlled in spreadsheets or other data harnesses. All results and screenshots during execution will roll up into the QTP execution report (Keynote DeviceAnywhere, 2010).

A second option for organizations that do not have Functional Testing (QTP) and don’t plan to purchase it, is to create a data driven or keyword driven framework by utilizing Keynote DeviceAnywhere’s JAVA based APIs to control the actions. In the same way a user can create reusable code and incorporate a datasheet to drive the commands. Keynote DeviceAnywhere provides a JAVA editor internally to create the API programs and conveniently stores all components inside the Keynote DeviceAnywhere Studio. In addition, the Keynote DeviceAnywhere Studio can interact with other JAVA editors such as Eclipse to get the full capabilities of a JAVA environment (Keynote DeviceAnywhere, 2010). This is convenient for managing packages and

editing complex logic before the compile stage. The Keynote DeviceAnywhere APIs provide a great way for organizations that don’t have expensive commercial tools or the budget to purchase them to still get the most out of the mobile automated testing experience.

How We Integrate with Keynote DeviceAnywhereHere at Quilmont we have a pre-developed framework that comprises a user-friendly interface for managing application properties and creating test case steps. Our out-of-the-box function libraries integrate with QTP/QC, Selenium, CodeUI, Rational Functional Tester, and yes, Keynote DeviceAnywhere. For our Keynote DeviceAnywhere implementation, we provide plug-ins for both QTP and JAVA APIs to deliver a framework with keyword and data driven capabilities. In our setup we employ a three layer approach. In the first layer we have several keyword actions packaged at a project level for use in the Keynote DeviceAnywhere Studio. Keynote DeviceAnywhere Studio allows projects to be imported and exported. For us, this means we can import and export our keyword actions to any Keynote DeviceAnywhere Studio installation. Our second layer is a driver script which could be the QTP or JAVA API version depending on the customer request. This is the layer that processes the main execution and calls the keywords depending on the test case information it receives from the third layer. The third layer is our framework

interface. In this layer our customers can manage any application properties (for mobile application this would be text, icon objects, pixel information) and create test case steps. The test case steps call keyword actions in the Keynote DeviceAnywhere studio while also passing parameters to these actions through the driver script. This allows our customers to first create all of their test case information outside of the Keynote DeviceAnywhere Studio which cuts down on license consumption and then log into the Keynote DeviceAnywhere studio when they are ready to execute their test cases. An example of how the three layers work together can be illustrated as follows:

Keystrokes and data entry can be controlled in spreadsheets or other

data harnesses.


Let’s assume the device we are testing is an HTC EVO and our application has a login screen. The login screen has objects for a user name, password, and a login button. The goal of our test case is simply to verify that we can log into the application and get to the next screen. Starting with layer three (framework interface) we would represent each of the objects (user name, password, and login button) in the section of the interface that manages object information. Since these are mobile objects we would most likely reference the text labels for each object. In the test case creation section of the interface we would create the step sequence where each step would reference an object name, value, and keyword action.

framework interfaCe:Step 1: User Name | user1 | EnterData

Step 2: Password | pwd1 | EnterData

Step 3: Login | button | Click

In the second layer, the driver script would read these steps from the framework interface and call the action in the Keynote DeviceAnywhere Studio that corresponds to the action in the step sequence. The driver passes the object name and value to the keyword action.

funCtional teSting (Qtp) or Java api Driver:GET Step 1 from interface, CALL EnterData in Keynote DeviceAnywhere Studio, PASS User Name | user1

GET Step 2 from interface, CALL EnterData in Keynote DeviceAnywhere Studio, PASS Password | pwd1

GET Step 3 from interface, CALL Click in Keynote DeviceAnywhere Studio, PASS Login | button

In the first layer, the pre-developed keywords receive the parameters from the driver script and use the Keynote DeviceAnywhere automation engine to translate the action for use on the HTC EVO.

Keywords stored in Keynote DeviceAnywhere Studio:RUN EnterData, FIND User Name, TYPE user1

RUN EnterData, FIND Password, TYPE pwd1

RUN Click, FIND Login, TOUCH Login

These three calls will log into the application and drive the screen to the next page.

Statistically we can see that mobile devices are becoming the preferred way of conducting business transactions for consumers. Touch screens provide a more desirable method for collecting customer information than keying data into web forms or scrolling through overpopulated dropdown boxes. Mobile devices have arrived and more companies are embracing the change every year. It’s fair to describe the application development techniques for mobile software as still being in its early stages. However, applying discipline through traditional application development practices (test automation being one of them) will assist future standardization. From an automation developer’s standpoint, learning the tools and techniques in this early stage will have great implications on a person’s individual skillset. It’s an amazing opportunity to learn, discover, and help standardize the untamed world of automated mobile testing.

References:

Barneir, Tal (2011), Need for Mobile Test Automation Tool on Various Platforms (Android, iPhone, Blackberry, Symbian & WindowsPhone 7), Mobile Application Testing. Retrieved: November 17th 2011 From: http://www.mobileappstesting.com/category/mobile-application-testing/mobile-application-testing-and-automation-tools/#

HP/KDA Datasheet (2009), Test Automation for Mobile Devices, Hewlett-Packard. Retrieved: July 16th 2011 From: http://support.openview.hp.com/

Keynote DeviceAnywhere (2010), Keynote DeviceAnywhere Test Center User Manual for Enterprise Deployments, Keynote DeviceAnywhere. Retrieved: June 15th 2011

Ingram, Mathew (2010), Mary Meeker: Mobile Internet Will Soon Overtake Fixed Internet, Gigaom. Retrieved: October 25th 2011 From: http://gigaom.com/2010/04/12/mary-meeker-mobile-internet-will-soon-overtake-fixed-internet/

Leggatt, Helen (2010), Travelers display low tolerance for mobile app, website glitches, Gigaom. Retrieved: October 25th 2011 From: http://www.bizreport.com/2010/08/travelers-display-low-tolerance-for-mobile-app-website-glitches.html#

Qualitia (2011), Defining Scriptless, QualitiaSoft. Retrieved: November 17th 2011 From: http://www.qualitiasoft.com/pdfs/Definition%20of%20Scriptless%20Test%20Automation.pdf

Are You Contributing Content Yet?

The Automated Testing Institute relies heavily on the automated testing

community in order to deliver up-to-date and relevant content. That’s why

we’ve made it even easier for you to contribute content directly to the ATI

Online Reference! Register and let your voice be heard today!



>> Community Comments Box - This comments box, available on the home page of the site, provides an opportunity for users to post micro comments in real time.>> AnnounCements & Blog Posts - If you have interesting tool announcements, or you have a concept that you’d like to blog about, submit a post directly to the ATI Online Reference today. At ATI, you have a community of individuals that would love to hear what you have to say. Your site profile will include a list of your submitted articles.>> AutomAtion events - Do you know about a cool automated testing meetup, webinar or conference? Let the rest of us know about it by posting it on the ATI site. Add the date, time and venue so people will know where to go and when to be there.


Automation Events

Learn more today at http//www.about.automatedtestinginstitute.com


We all have the neCeSSary components on

our systems to build a Solution of our oWn.

Tom Wimsatt

by

TesTManageMenT

InTegraTIng Ms OffIce For Cost-EFFECtivE TesT ManageMenT

A Homegrown, Desktop Solution for Test Management

A s in any test effort, we write test procedures and eventually require some way to keep track of all of the documentation.

Many of us become spreadsheet jockeys with workbooks of our tests and other supporting information tucked away in the labyrinth of our disk drives. The management becomes increasingly cumbersome leading to duplication and confusion. We forget if we wrote a procedure to cover a case, a new member joins the team and does not know what has been written, or we are unable to locate the needed procedure. Our typical solution is to write a new test and add it to the workbooks for safe keeping.

Another complication arises when we want to create a test set for a particular purpose, such as a user acceptance test or regression test set. We may also get requests from management about which tests cover which requirements, or about getting some sort of a test report. Pulling the different parts together to create the needed documents, and managing the whole process becomes time consuming and error prone. In other words, a perfect opportunity for automation!

There are a number of nice commercial solutions available that answer the task quite well; but, what if we do not have a large team or a budget that can afford one of the packages? We all have the necessary components on our systems to build a solution of our own. What follows is an explanation of a system I developed with Microsoft (MS) Office, and the same can be adapted for OpenOffice.Org.

“A simple system...to coordinate your test procedure development effort.”

December 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 27


need to share out the location to your team if you choose to store it on your own work station. This organization is optional, so, feel free to adjust to your own desire.

This organization works well, as it helps to keep things organized in a way that seems logical. At the root directory is the actual database – in this case it would be under the Test Procedures directory. Under that are subdirectories for each of the systems of your particular test effort. If this was an effort to test MS Office, they would be Word, Excel, PowerPoint, etc. Under each of the system directories are the subsystem directories and under that are the test files themselves.

The naming convention for the files mirrors the above directory structure. It helps

reQuirementSBefore jumping into the details of the design, a few requirements need to be established. After having used a number of tools over the years, I settled on the following basic functions:

• Viewing of all tests

• Filter and Sort

• Test Set Production

• Track Status and Author

• Allow Attachments

• Associate Requirements

• Track Team Members

• Allow Team Communication

BaSiC arChiteCtureMost of us have the Microsoft Office suite on our machines and it makes sense to develop for something the majority has access to. Most of us, though, forget that Office is more than Word, Excel and PowerPoint. At the heart of this system is the MS Access database. The test procedures are in MS Excel spreadsheets to make the formatting somewhat easier, and allow for some additional functionality without the complication of Word document formatting. In addition, there is a hook into Outlook to provide some automated communication for the team.

Behind the scenes, MS Office uses Visual Basic for Applications (VBA) for the programming language. This makes it easy to add more functionality or modify any of the existing functions.

The database provides a place to store the details about the test procedures, but the file system contains the actual spreadsheet files. Using links in the database, the tests can be displayed and modified. New tests are created from the information entered into the database. The test procedure files are kept as originals, and modified as needed and copies are created and collected to build test sets for the actual runs.

Refer to Figure 1 below. The database is best stored on a networked drive shared out to the team. At a minimum, you will

Figure 1: Test procedure database directory structure

Figure 2: Tables and relationships


locate a file easily as the name contains the basic structure:

<ProcNum><System><Subsystem>.xlsx

The ProcNum is automatically generated by Access, and the initial spreadsheets get the number on the tabs as you will see.

Delving further into the structure of the database, we have tables that not only contain the data for our test procedures but also the various components associated with the test procedures. To make up a test procedure, we need to know certain details such as description, system, subsystem, status, etc. Some of the information can be selected from drop-downs that draw from the data in the tables. Figure 2 shows the tables and their respective relationships within the database. Keep in mind when you modify the structure of the tables (e.g. add or remove columns), you may need to remove a relationship and restore it after you finish the modification.

The tables are:• System – contains the names

of the systems and their related subsystems. Typically you can use work breakdown structures, diagrams or other documents to develop items for this table;

• Members – the team members and related information such as telephone numbers, e-mail addresses, etc;

• Status – stages of the procedure development from In Progress to Obsolete;

• Methods – the test methods for each procedure such as Analysis, Observation, and Test;

• Requirements;• Procedures;• Req Attachments – requirement

attachments;• Attachments – procedure

attachments;• ReqtoProcID – the associated

IDs of the requirements and test procedures.

formS anD reportSMS Access uses forms and reports to serve as the interfaces to the tables and data. The

Figure 3: main Page Form

Figure 4: Test Procedure entry/edit Form


8. Stage – at what point in the SDLC is this tested intended to be performed (e.g. Unit Test, System, Production, etc.);

9. Manual test – Manual Test filename;

10. Automated test – location of the automated version of this test;

11. Status – Ready, Work In Progress (WIP), Obsolete, Ready for Review;

12. Selected – self-explanatory;

13. Requirements – a listing of the requirements currently entered into the system;

14. Attachments Subform – enter the link and a description of any documents that may pertain to this test;

15. Requirements Subform – the list of requirements now associated with this test. Refer to the Select Reqt. button instructions.

Not shown in Figure 4 is the right side of the form containing a listing of the requirements. After a requirement or requirements are selected, clicking the “Select

forms are where most of the automation occurs as we add buttons and code to make the database more useful.

Figure 3 shows the first form which functions as the main control page. This form contains buttons to initialize the most common tasks in the database such as entering test procedures, viewing reports and editing the members listing. The front page allows for a custom graphic to display your project name, company logo or any graphic supported by Office. The buttons have built in functions to display the forms and reports, but they can be programmed for any number of functions enabled by VBA.

The Test Procedure entry form contains buttons to create new test procedures, associate requirements to the tests and create e-mails containing links to the selected test and the database itself. Referring to Figure 4, notice that the fields are related to the columns in the database for the tests themselves, with some minor modifications to make the entries more readable.

Access will use the column name by default if a caption is not supplied when the table is created. The caption gives the developer the option of providing a more readable name in addition to the actual name of the column. This is a good opportunity to further explain the information for each test procedure:

1. ID – auto-generated number. Not editable;

2. System – the highest level at which the project can be classified (e.g. MS Office);

3. Subsystem – the next level of definition for the system. The systems that make up the larger system (e.g. MS Access, MS Excel, MS Word, etc);

4. Feature – a particular aspect of the subsystem that is to be tested (e.g. file save operation);

5. Description – what the test is going to do in general terms (e.g. open a file, modify the contents, save and verify the operation);

6. Version – of the software this test was developed to test;

7. Author – who wrote this test;

Figure 5: requirements Form

Figure 6: build Test Set Form


Reqt.” button associates the requirement to the procedure, and adds to the Requirements subform.

Access provides many useful functions out of the box that can be used right in the forms. The top of my list are the filtering options. In the menu bar above the forms are two filtering functions – Selection and Advanced. Selection will use the currently selected field to filter the data. Advanced allows you to use the form as a query tool. With the filter by form enabled, the entire form becomes the query.

Clicking on Requirements button from the front page brings up associated forms as shown in Figure 4. The requirements form is similar. This is part of the database that may require some modification as the fields may not quite fit your organization’s way of handling this information. Some organizations go overboard with classifying requirements, while others are very simple. Below are some of the more common or desirable fields:

1. ID - auto-generated number; not editable;

2. Type – mandatory, highly desired, desired, nice to have;

3. System - the largest level at which the project can be classified (e.g. MS Office);

4. Subsystem – the next level of definition for the system; the systems that make up the larger

Figure 7: Test Set Template

system (e.g. MS Access, MS Excel, MS Word, etc);

5. Title – self-explanatory;

6. Description – self-explanatory;

7. Spin – agile reference to schedule time frame;

8. Owner – who the requirement is for or who is tracking this item;

9. Disposition – commitment to work or deferred status;

10. Req Attachments Subform - enter the link and a description of any documents that may pertain to this requirement.

The Build Test Set button brings up the form shown in Figure 6. This form is really an abbreviated version of the Test Procedure entry/edit form in Figure 4, and is arranged in a tabular view. The purpose for this form is to allow the selection of tests easily by scrolling through all of the tests and clicking the Selected checkbox for the desired tests. When the form’s Build Test Set button is clicked, the selected tests are all opened and copied into a new workbook made from a template that sits outside of the database.

Refer to Figure 7 for an example of the test set created by the tool. The test set template includes a cover sheet with a table for revisions and room for a title, version number and logo. The second sheet is for results (shown), and is automatically populated by

Figure 8: Team member Form


the VBA code as the tests are copied into the workbook. The rest of the sheets are the actual tests copied from the originals. If there are additional sheets accompanying your tests, they get copied in, as well.

When a new file is created in Excel, three sheets are created by default. If your test does not require the additional worksheets, you will need to delete them before running this or you will have a number of duplicate and empty spreadsheets cluttering up your newly created test set! If this happens, it is no big deal to simply close Excel, edit the individual tests, and build the test set again – easy!

The final form is for your team members as shown in Figure 8. This is a simple form to allow easy editing of the members table. One thing not shown in this form but exists in the table is a selection for notifications. With this, the e-mail addresses can be populated automatically with the people who would be most interested in seeing the test procedures such as development or management. A tool for tracking defects makes this particular feature extremely valuable.

The reports available from the Main Control page are:

• By Auth – report for tests grouped by author

• By System – for tests grouped by system

• Members – self-explanatory

• Reqts by Type – requirements grouped by type (mandatory, desired, etc.)

They are simply the reports as created in Access with buttons to display them.

Fortunately, the MS Office Suite does a good job of sharing data between each of the applications. The reports can be exported to the standard formats, allowing many options for including the information into other reports and report formats. In addition to raw reporting, there are canned macros that can be used to add statistics and additional data to your reports, and to extend what I have developed here.

Private Sub System_AfterUpdate() If Len(System) > 0 Then Subsystem.RowSource = “SELECT DISTINCT System.Subsystem FROM System WHERE System = ![System]” Subsystem.Requery Subsystem.SetFocus Subsystem.Dropdown End IfEnd Sub

the CoDeAll of that is very useful and interesting, but the code is much more fun to talk about and work with. I had previous experience with Visual Basic, and I was, therefore, able to leverage that knowledge for this tool. To start, look at the code in Figure 9, which is for the System and Subsystem drop-down boxes on the Test Procedure Edit form.

When a selection is made from the System drop-down box, the AfterUpdate event fires this piece of code. A SQL query is run against the System table selecting data to populate the Subsystem drop-down. The Requery populates the information to be displayed in the Subsystems, and the two lines that follow put the cursor in the box and open the drop-down. Nice! This bit of code does not work in the Build Test Set form, as each row is a new set of System and Subsystem boxes.

This chunk of code in Figure 10 is from the Build Test Set function.

Figure 9: System Drop-down code

Set rsQuery = dbs.OpenRecordset(“qrySelectedProcs”, dbOpenDynaset) Do While Not rsQuery.EOF If rsQuery![ManualTestName] <> “” Then aryFileName = Split(rsQuery![ManualTestName], “#”) strWkBkName = aryFileName(1) xlApp.Application.Workbooks.Open strWkBkName Set xlNewBook = xlApp.Application.ActiveWorkbook Set xlNewSheet = xlNewBook.Worksheets(1) ‘Move the sheet or sheets from the original file to the Test Set file z = Worksheets.Count For x = 1 To z Set xlNewSheet = xlNewBook.Worksheets(z - x + 1) ‘Move the sheet to the end of the template file xlNewSheet.Move After:=xlRsltSheet Next ‘Update the results sheet (2) xlRsltSheet.Cells(intRsltCellCount, 1) = rsQuery!ID xlRsltSheet.Cells(intRsltCellCount, 2) = rsQuery!System & vbCrLf & rsQuery!Subsystem xlRsltSheet.Cells(intRsltCellCount, 3) = rsQuery!Feature If IsNull(rsQuery!AutomatedTestName) Then xlRsltSheet.Cells(intRsltCellCount, 4) = rsQuery!Author Else xlRsltSheet.Cells(intRsltCellCount, 4) = “Automated” End If rsQuery.MoveNext intRsltCellCount = intRsltCellCount + 1 Set xlNewSheet = Nothing Set xlNewBook = Nothing End If Loop

Figure 10: build Test Set Function code


In this piece of code we have to run a query that selects all of the checked test procedures. For each test procedure returned in the set, the file is opened if it exists and is copied into the template file. Next the ID, System, Subsystem, Feature and Author are copied into the Results page. If there is any text in the Automated Test Name field, it will get the word “Automated” inserted for the author. The rest of the code is left out, as it is line after line of Excel formatting that is not very interesting, IMHO.

Another interesting bit of code is the routine for selecting requirements to associate with the test procedures, illustrated in Figure 11.

The interesting thing to note here is the SQL statement (beginning with “INSERT INTO…”).

In order to make the association, a table has to get the ID’s from each requirement, and put them into the ReqtoProcID table with the corresponding procedure ID. This association is then used to query for the requirements to be displayed in the Requirements subform. This can also be used to create a Requirements Traceability Matrix very easily. That is one report I did not include as there are many ideas on what is to be included in such a document and the format. I have a version I use and can supply on request.

There you have it!

A simple system using MS Office to coordinate your test procedure development effort. I gave the hi-lights as best as I see them of this tool and there is much more that could be said. If you would like a copy of this tool, drop me a line and I will send it to you with some additional documentation to get you going. AT WHAT ADDRESS? HOW DO THEY GET IN TOUCH WITH HIM? If there is enough interest, maybe we can collaborate to make the tool more robust and full-featured! When I first started to develop this tool there was nothing as far as an open source tool to accomplish. I hope this fills that gap or inspires someone else to come up with something better. Finally, I hope

Private Sub SelectReqt_Click()On Error GoTo Err_SelectReqt_Click

Dim dbs As Database Dim varItem As Variant Dim frmEdit As Form Dim ctrlEdit As Control Dim strInsertQ As String Set frmEdit = Forms![Procedures] Set ctrlEdit = frmEdit!Requirements Set dbs = CurrentDb For Each varItem In ctrlEdit.ItemsSelected strInsertQ = “INSERT INTO ReqtoProcID (PID, RID) Values (“ & [ID] & “,” & ctrlEdit.ItemData(varItem) & “)” dbs.Execute strInsertQ Next varItem Forms![Procedures]![Requirements Subform].Requery

Exit_SelectReqt_Click: Exit Sub

Err_SelectReqt_Click: MsgBox Err.Description Resume Exit_SelectReqt_Click End Sub

Figure 11: requirements Traceabil ity code

to recreate this in an open source suite such as OpenOffice.org sometime in the future. Who knows? Maybe one of you will do that before I will, and we will all benefit!



Automation Events

Contribute Content Today


Learn more today athttp//www.about.automatedtestinginstitute.com


i ‘b’log to u

latest from the BlogosphereAutomation blogs are one of the greatest sources of up-to-date test automation information, so the Automated Testing Institute has decided to keep you up-to-date with some of the latest blog posts from around the web. Read below for some interesting posts, and keep an eye out, because you never know when your post will be spotlighted.

Hey there, what’s up with API nowadays?They are hot topics wherever you go. No API = no product for now. Well, Soap UI is on a wave of this trendy tech flop. Why? Let’s see:

1. Easy to get into it, small learning curve

2. Perfectly, thoughtfully built to test SOAP and Restful web services

3. Supports proprietary and specific protocols

4. No coding required

Blog Name: Automated Testing and QAPost Date: October 21, 2011Post Title: SOAP UI is hotAuthor: Peter Kartashov

Read More at:

http://at4qa.blogspot.com/2011_10_01_archive.html

Like most test tools, especially code generating ones, it works most of the time but its not perfect. Po Hu had an early version working, and decided to test this out on a real world product. Po, the developer of RPF, worked with the chrome web store team to see how an early version would work for them.

Blog Name: Google Testing BlogPost Date: November 17, 2011

Post Title: RPF: Google’s Record Playback FrameworkAuthor: Jason Arbon

Read More at:

http://googletesting.blogspot.com/2011_11_01_archive.html


latest from the BlogosphereAutomation blogs are one of the greatest sources of up-to-date test automation information, so the Automated Testing Institute has decided to keep you up-to-date with some of the latest blog posts from around the web. Read below for some interesting posts, and keep an eye out, because you never know when your post will be spotlighted.

I think if one claims that they have test automation, I mean full-fledged test automation, it must imply not only automated tests per say but it should include process automation. Automated tests (the code) are useful in same extent as manual tests which were never executed. It is just having no value unless it is executed at least once. When we execute tests, we kind a returning back these expenses invested in test design, same for automated tests.

Blog Name: Automated Testing and QAPost Date: November 28, 2011

Post Title: Automate Your AutomationAuthor: Peter Kartashov

Read More at:

http://at4qa.blogspot.com/2011_11_01_archive.html

It is done to evaluate the application’s behaviour beyond normal or peak load conditions. It is basically testing the functionality of the application under high loads. Normally these are related to synchronization issues, memory leaks or race conditions etc. Some testing experts also call it as fatigue testing. Sometimes, it becomes difficult to set up a controlled environment before running the test. Example of Stress testing is...

Blog Name: Software Testing StuffPost Date: September 13, 2011

Post Title: Performance vs Load vs Stress Testing

Read More at:

http://www.softwaretestingstuff.com/2011/09/performance-testing-vs-load-testing-vs.html


Go on a retweet

paying a visit to the MicroblogsMicroblogging is a form of communication based on the concept of blogging (also known as web logging), that allows subscribers of the microblogging service to broadcast brief messages to other subscribers of the service. The main difference between microblogging and blogging is in the fact that microblog posts are much shorter, with most services restricting messages to about 140 to 200 characters.Popularized by Twitter, there are numerous other microblogging services, including Plurk, Jaiku, Pownce and Tumblr, and the list goes on-and-on. Microblogging is a powerful tool for relaying an assortment of information, a

power that has definitely not been lost on the test automation community. Let’s retreat into the world of microblogs for a moment and see how automators are using their 140 characters.

The Verify/ATI 2011 Confer-ence was a huge success!!! The LIVE Automation Honors was a big hit...getting ready for 2012...Don’t miss out!!!

Twitter Name: automatedtestPost Date/Time: Oct 19

Topic: Conference

Analysis often doesn’t find causes; finding causes often in-volves additional testing http://bit.ly/o5OIYU+

Twitter Name: sbarberPost Date/Time: Nov 21Topic: Finding Causes


paying a visit to the Microblogs

The KIT is Koming...

Twitter Name: automatedtestPost Date/Time: Nov 22

Topic: TestKIT Conference

India will be a #cloud first IT market from @katiefehren

Twitter Name: fredberingerPost Date/Time: Dec 16Topic: Cloud Computing

Additional testing finds addi-tional manifestations of bugs but doesn’t tell you cause. RCA finds cause of related issues.

Twitter Name: TestingMentor Post Date/Time: Nov 21

Topic: Rebuttal to Finding Causes

jHome is an #opensource plat-form for home automation based on #Java EE 6 and #Glassfish de-veloped b... http://bit.ly/vVLdQg

Twitter Name: FOSSwikiPost Date/Time: Nov 28Topic: Home Automation


the testKit conference is hot!The KIT is Koming

As each of us becomes more mature in the discipline of testing, our respective testing toolkits – that ATI refers to as your testkits - continue to grow. This testkit is filled with knowledge, information and tools that go with us wherever we go, allowing our projects and organizations to quickly reap the benefits of the testkit elements that we’ve amassed. It generally takes time and experience to build up a testkit, however, but events like ATI’s TestKIT Testing and Test Automation conference are specifically designed to help accelerate the process of filling a testing professional’s testkit. Coming off of the highly acclaimed Verify/ATI conference in 2011, there is a lot of enthusiasm for the TestKIT 2012 event scheduled for October 15-17, 2012, which makes this event hot!

TestKIT is more than a miscellaneous collection of speakers and sessions, but instead an event that is created to be engaging, informative, practical and memorable so that the testkits of all participants are destined to vastly grow with elements that are timely and applicable. TestKIT focuses on relevant software testing topics including security, testing techniques and methodologies, the cloud, test automation, test tool implementations and more.

As the K-I-T in TestKIT implies, the conference offers a stellar event that will focus on the following three areas:

• Knowledge• Information• Tools

KNOwLEDGEThe conference gathers together

a group of highly knowledgeable people and provides an environment in which that knowledge may best be relayed. These people include speakers, presenters, vendors and the attendees themselves. We believe that everyone has something to share and through a variety of tutorials, sessions, discussion forums and networking events, TestKIT creates the perfect

environment that allows everyone to learn and share, thus maximizing the growth of everyone’s testkit.

INFOrMATIONKnowledge is information

– or contextualized data – that has been gathered and interpreted in a way that allows it to be usefully and intelligently applied to meet some purpose. TestKIT makes sure you are provided practical information in a manner that helps to ensure the acquired information

may be easily processed so you can make a determination on how it may best be applied. Thus, you leave the event more informed and more knowledgeable.

TOOLSGiven the fact that the event is put

on by ATI, there is a heavy focus on tools. Tools make it possible to more effectively and efficiently implement various processes and procedures that one may be knowledgeable in. Tools may include open source and commercial automated test tools, static analysis tools, code samples, concrete automation techniques, checklists, etc. TestKIT provides a focus on tools like no other event.

Hot topics in automation

The KIT is Koming


http://www.googleautomation.com


The KIT is ComingK

if you thought ati’s 2011 event was good, wait until you see 2012.h t t p : / / w w w . t e s t k i t c o n f e r e n c e . c o m