test automation plan - cabig-1

8/10/2019 Test Automation Plan - CaBIG-1

1/13

EkagracaEHR Test Automation Plan

High Impact High Value Business Results

Document Change HistoryVersion

Number

Date Contributor Description

V0.1 6/24/2010 Jim Jones First DraftV0.2 6/30/2010 Jim Jones Ae too!ing s"reasheet an narrati#e.V0.3 $/21/2010 Jim Jones Changes throughout to ref!ect ecision to use

soa"%& instea of 'er#ice (est. A!so minoreits

V0.4 $/23/2010 Jim Jones %"ate Functiona! (est section to c!arify ro!eof each too! re!ati#e toseria!i)ation/esria!i)ation.

V0.* +/21/2010 Jim Jones Ae Arch Diagram for J%nit Frame,or- ansu""orting tet.

0

CAEHR

Test Automation Plan

%"ate 'e"temer 102010


2/13



Table of Contents

1 INTRODUCTION....................................................................................................... 2

1.1 JC(VV&5.............................................................................................21.2 ''(7VV&5...............................................................................................2

1.3 DC%78(VV&5..........................................................................................21.4 9A(&8'H&((HDC%78('/9A8'.......................................................21.* A''%7(&8'......................................................................................................21.6 C8'(A&8('....................................................................................................... 2

2 AUTOMATION TOOL CAT!ORI"........................................................................#

2.1 ('(7A8A:78(..............................................................................................32.2 F%8C(&8A9('(&8:...........................................................................................32.3 F7A8C('(&8:.......................................................................................32.4 &8(:A(&8F(ACA;&9&(............................................................................42.* C8(&8%%'&8(:A(&8....................................................................................4

# $RO$O"D IM$LMNTATION..............................................................................%

% T"T MANA!MNT TOOL VALUATION.............................................................&

' (UNCTIONAL T"T TOOL VALUATION................................................................&

*.1 5;'V&C'A(&8'('(&8:...................................................................$*.2 '&A9&


3/13



1 Intro/uction(his test automation "!an escries the se!ection an "!anne im"!ementation of too!ing tosu""ort the ?ua!ity Assurance an (esting acti#ities of the Center for ;iomeica! &nformation

an &nformation (echno!ogy C;&&(B Amu!atory nco!ogy !ectronic Hea!th ecor caHB"roect.

1.1Project Overview

(he goa! of the caH "roect is to su""ort the Amu!atory nco!ogy sector in the e!i#ery ofim"ro#e "atient care through the e#e!o"ment of consensus ase hea!th &( stanars anso!utions that meets this sector uniue H nees.

1.2System Overview

(he caH "roect ,i!! e#e!o" ser#ice s"ecifications that ,i!! e ao"te in the Amu!atorynco!ogy sector to meet the uniue H nees of that sector to "ro#ie im"ro#e "atientcare.

1.3Document Overview

(his (est Automation !an e"!ain the e#a!uation "rocess status an "ro"ose im"!ementationfor the too!set "ro"ose to su""ort a!! ?A testing acti#ities.

1.4Relationship to Other Documents/Plans

(his "!an references the fo!!o,ing !ansEerformance (est !an

1.5AssumptionsAt the onset of the "roect there ,as not enough time to o a "rotracte too! se!ection. (he H(est 'uite ,as inc!ue in the "ro"osa! so it ,as assume that this suite ,ou! meet ournees. (he ?A (eam "roceee to e#a!uate the too!set ,ith s"ecia! focus on its o"eration in ac!ou en#ironment an its integration ,ith too!s se!ecte y other teams.

1.6Constraints

J&A 'tuio ,as immeiate!y se!ecte an im"!emente y the 'D (eam. &ntegrationof the ?A too!set ,ith J&A 'tuio ,as therefore a constraint.JA7A Contour ,as se!ecte y the Ana!ysis (eam as their euirements 7anagementtoo!. &ntegration of the ?A too!set ,ith Contour ,as therefore a constraint.

2


4/13



2 Automation Tool Cate0ories

2.1Test ManagementA test management too! is the centra! too! for the ?A (eam to "erform a!! of its asic tas-sEDesign test casesecute test casese"ort/trac- efects

Assure reuirements traceai!ity/co#eragee"ort test metrics

H ?ua!ity ,as e#a!uate an se!ecte as the ?A (eam test management too!.

2.2Functional Testing&n its functiona! test automation a""roach the ?A (eam consiere that co#erage ,as reuirenot on!y for the ,e ser#ice o"erations ut a!so for the seria!i)ation/e=seria!i)ation into anout of @79. (his !ast "art ,as im"erati#e ecause the caH ,e ser#ices are initia!!y einge#e!o"e ,ithout a reference im"!ementation in ,hich to test them.For this reason ?A has ecie u"on a hea!essG c!ient too! for ,e ser#ices testing at the@79 reuest/res"onse !e#e! an Ja#a=ase a""roach using a ,e ser#ices c!ient for testingseria!i)ation/e=seria!i)ation.?A consiere the fo!!o,ing hea!essG c!ients for test of ,e ser#ice o"erations at the @79reuest/res"onse !e#e!E'er#ice (est'oa"%&

'oa"%& ,as se!ecte y the ?A (eam ecause 'er#ice (est >.* has com"atai!ity issues ,ith,s! 2.0=ase ,s!s.

?A is augmenting its functiona! automation ,ith a Ja#a=ase "rogrammatic a""roach aseon the J%nit test frame,or-. (his ,i!! incor"orate a Ja#a ,e ser#ices c!ient ,hich ,i!! testseria!i)ation/e=seria!i)ation.

2.3Performance TestingA "erformance test too! is reuire to eecute ,e ser#ice o"erations in sufficientnumers an in a contro!!e fashion to measure the ,e ser#ices res"onsi#eness.9oaunner an 8eo!oa ,ere e#a!uate for the "ur"ose. ;oth 9oaunner an8eo9oa ha com"atai!ity issues ,ith ,s! 2.0=ase ,s!s. 9oaunner has a moreroust scri"ting ca"ai!ity an "ro#ie the ai!ity to im"ort a ,or-ing reuestgenerate in soa"%&B into the 9oarunner scri"t. 9oaunner a!so integrates ,ith?ua!ity Center. For these reasons 9oaunner ,as se!ecte as the "erfromancetesting too!.

3


5/13



2.4Integration for TraceabilityDue the constraints note in the Constraints section it ,as necessary to e#a!uate an assurethat ?A ha a re!ia!e ,ay to integrate ,ith the 'D an Ana!ysis too!sets. J&A ;rige ,ase#a!uate an se!ecte for integration of efects et,een ?ua!ity Center an J&A 'tuio.Contour/?C &ntegration ,as e#a!uate an se!ecte for integrating reuirements et,een

?ua!ity Center an Contour.

2.5Continuous Integration(he ?A (eam ,ants to integrate its functiona! test suites into the ui!s an e"!oy"rocess. (he e#a!uation "rocess inc!ueE

ecution of a ?C (est 'uite from the comman !ine using ?C A& to integrate

the 'er#ice (est automation ,ith the ui! an e"!oy "rocess.

ecuting soa"%& tests from A8(

ecuting a J%nit suite from A8(.

# $ropose/ Implementation(he fo!!o,ing iagram sho,s the "ro"ose integrate im"!ementation of a!! the ?A too!s.

(he remainer of this ocument ,i!! iscuss the too!s ,ithin this im"!ementation contet.

(he ?A (eam has ientifie ifferent !e#e!s of testing that nee to e "erforme at ifferent"oints in the agi!e e#e!o"ment cyc!e. (he fo!!o,ing s"reasheet sho,s ho, each of the"ro"ose too!s ,i!! e use for those #arious !e#e!s of testing.

4


6/13



TOOL

LOCATION/ACTIVITY

CI !A Tag !A Trunk"er#$rma

nce%AT& %AT' %AT( %AT) %AT* +E,O

EveryCheck in

Deployedfor EachSprint

DeployedDaily (3am)

to QA onSuccessfulBuild (TA)

Deployedperiodicall

y

Deployed!nfre"uentl

y

Deployed!nfre"uentl

y

Deployed!nfre"uentl

y

Deployed!nfre"uentl

y

Deployed!nfre"uently

Deployed!nfre"uen

y

#ava (customcoded)

Toolin$

Smoke

Smoke Smoke Smoke Smoke Smoke Smoke Smoke Smoke Smoke

#ava (customcoded)

Toolin$

%ro$rammati

c Test

eneration

#ava (customcoded)

Toolin$

Seriali&ation'De

seriali&ation

#ava (customcoded)

Toolin$ e* Client

soap+! ,i- .alidation

soap+! /e$ression /e$ression /e$ressio

soap+!Deployme

ntScenarios

Deployment

Scenarios

Deployment

Scenarios

Deployment

Scenarios

DeploymentScenarios

soap+!SystemTestin$

soap+!/e"uiremen

ts Testin$

soap+!

Architecture

' ECC,Testin$

soap+!Desi$n

Testin$

soap+! 01 C,/

soap+!2!%AA

Compliance

!2 SecurityScans

Security

!2 456compliance

test tool

456Compliance

2%7oad/unner

%erforman

ce

soap+!'897Spy

Test Analysis'Test case Construction

2% QualityCenter

Test: Defect: /e"uirements 9$mt

&ssues an is-ssoa"%& ro &nsta!!ation on Demo an at terna! %A( (est 'itessoa"%& nees to e integrate ,ith ?C.(esting in (run- reuires eicate resources for ai!y changes an 'cri"ts maintenance8&H 'ecurity 'canning too! com"atii!ity ,ith ,e ser#ices/test en#ironments8&H *0+ com"!iance test too! com"atii!ity ,ith ,e ser#ices/test en#ironmentseuirements testing ,ith soa"%& = fu!! im"!ementation ,ith Contour ;rige "rouct

*


7/13



8ee CD 224 = successfu! ui! tag im"!ementation.

% Test Mana0ement Tool aluationA test management too! "ro#ies the ?A (eam ,ith a centra! re"ository for esigning testcases eecuting tests re"orting/trac-ing efects an "u!ishing test metrics. (he "rimarycaniate too! for (est 7anagement for caH is H ?ua!ity Center ?CB. ?C is the "remiertest management too! in the mar-et"!ace. Due to its ,e!!=esta!ishe "osition in the mar-et it"ro#ies t,o istinct a#antagesE

?A resources ,ith ?C e"erience are reai!y a#ai!a!e. (hir "arty integrations ,ith other 'D9C too!s are a#ai!a!e.

A -ey oecti#e for the caH "roect ,as traceai!ity. ?C "ro#ies traceai!ity from thereuirements to the test cases to the efects.

?C ,as e#a!uate to assure that it ,i!! meet the nees of the ?A (eam. (he fo!!o,ing actions,ere "erforme as "art of the e#a!uationE

?C ,as insta!!e on an Ama)on c!ou instance %sers an roect aministrators ,ere efine

'am"!e reuirements ,ere efine in the euirements 7ou!e. e!eases ,ere efine in the e!eases 7ou!e an euirements assigne to a

e!ease. 'am"!e test cases ,ere ,ritten 'am"!e tests ,ere automate ,ith H 'er#ice (est Versioning

' (unctional Test Tool aluationFunctiona! (esting of ,e ser#ices is com"!e. &t is uite ifferent from :%&=ase testing.(y"ica!!y the main focus of ,e ser#ices testing is to eercise a!! of the o"erations containe,ithin each ,e ser#ice. %sua!!y this is one from ,ithin a ,e a""!ication ,hich uses the,e ser#ice o"eration. 5hen use ,ithin a ,e a""!ication the seria!i)ation/e=seria!i)ationof oects to m! an m! to oects ha""ens as "art of the testing. (his is sho,n in thefo!!o,ing figure.

6


8/13



'ince caH has no reference im"!ementation the ,e ser#ices are eing testeine"enent of any a""!ication. (herefore our functiona! testing a""roach nees to co#eroth ,e ser#ice o"eration testing as a ,e!! as seria!i)ation/e=seria!i)ation.

5.1Web Services Operations TestingH 'er#ice (est an 'oa"%& from #i,are are commercia!!y a#ai!a!e too!s for functiona!testing of 5e ser#ices. ;oth of these fa!! into a category of too!s referre to as hea!essc!ients.G (hey o"erate at the @79 reuest an res"onse !e#e! an o not test seria!i)ation/e=seria!i)ation of the @79 into/out of coe oects. (he fo!!o,ing figure sho,s ho, these too!s fitinto the ,e ser#ices testing infrastructure.

ach of these too!s offere a#antages an isa#antages ma-ing neither one a c!ear choice.For instance ,hi!e 'er#ice (est offers tight integartion ,ith ?ua!ity Center the 'er#ice (est >.*#ersion oes not su""ort 5'D9 2.0 an ,as una!e to im"ort the first 5'D9 that ,as

e!i#ere y Architecture. H "ro#ie a 'er#ices (est 11.0 eta #ersion ut that #ersion ,asuna!e to access an set the attriutes an #a!ues for the H9$ ataty"es.

'oa"%& on the other han has een use fair!y hea#i!y y the ?A team "ening the "urchaseof the se!ecte too!set. 'oa"%& is a!e to im"ort our 5'D9s that are ase on 5'D9 2.0 utthe too! sometimes is #ery s!o, ecause of the si)e of the resu!ting im"orte reuest. soa"%&oes not integrate ,ith ?ua!ity Center an so!ution nees to e e#e!o"e or foun for this.

After e#a!uation of these t,o too!s the !eaing choice is soa"%& ecause it offere a so!utionto the technica! issues.

&ssue I soa"%& to ?ua!ity Center integration "rouct from Agi!etest,are has recent!y eenientifie an ,i!! e e#a!uate in an u"coming s"rint. & may a!so e "referre to -ee" thesoa"%& artifacts in 'V8 rather than ?ua!ity CenterB for C& "ur"oses an ,rite a customintegration for getting the soa"%& test resu!ts into ?ua!ity Center.

5.2Serialization/De-serialization TestingAs mentione ao#e caH ,e ser#ices are eing teste ine"enent of any a""!ication.(o test seria!i)ation/e=seria!i)ation of the @79 the ?A team is a!so in#estigating aitiona!

$


9/13



testing a""roaches ase on a Ja#a 5e ser#ices C!ient an J%nit test cases. (he fo!!o,ingfigure sho,s ho, this too! fits into the ,e ser#ices testing infrastructure.

5.2.1Java-based Service Test Tool Architecture

(he fo!!o,ing iagram sho,s the architecture of the Ja#a=ase automate testing a""roach.

+


10/13



(he -ey com"onents sho,n at the to" of the iagram are the Data Fi!es an Ja#a C!asses.(he J%nit (est Frame,or- uses a ata=ri#en a""roach of reaing a ata fi!e ,hich efines thec!asses that nee to e ui!t for the ser#ice reuest eing teste. (he Ja#a C!asses aregenerate from the ,s! fi!es "ro#ie y the Architecture team. (hese comine to form aJa#a=ase ,e ser#ice ,hich is eecute #ia a ,e ser#ices c!ient. (his re"resents an en=

to=en test ,hich inc!ues the seria!i)ation/eseria!i)ation of the ,eser#ice from Ja#a oectsto @79 an ac- again ,hen the res"onse is recei#e.

(he architecture iagram further sho,s that the frame,or- has etene ca"ai!ities of"rogrammatica!!y generating test ata integrating ,ith the Continuous &ntegartion "rocessintegrating ,ith ?ua!ity Center an su""orting other test generation methos.

& $erformance Testin0(he asic strategy uring the first roun of Com"onent !e#e! "erformance tests is to ientifythe re!ati#e "erformance of each 'er#ice uner a common set of !oa assum"tions. (,ocategories of tests ,i!! e "erforme enchmar- an egraation tests. &n this ,ay ,e ,i!! e

a!e to ientify ,hich ser#ices "erform re!ati#e!y ,e!! an those ,hich "erform "oor!y for agi#en set of !oa assum"tions.

'useuent rouns of erformance (esting ,i!! inc!ue %se Case testing for !ogica!!y re!ateSets of Service/Operation ca!!s an a!so eference &m"!ementation tests for the initia!De"!oyment sites for caH ser#ices.

6.1Scenario Design and Scripting(here are t,o categories of erformance euirements ,e genera!!y !oo- for 5or-!oa"ectations an then es"onse (ime "ectations for that ,or-!oa. (he 5or-!oa"ectations usua!!y ref!ect the e"ecte transaction #o!ume for the 'er#ices an the

es"onse (ime "ectations ref!ect the reuirements for res"onse times ,hi!e the system is!oae ,ith the e"ecte transaction #o!umes. 'ince ,e ont ha#e those e"ectations yet,e ,i!! use a ;enchmar-ing a""roach an a sca!e Degraation a""roach. (he erformance(est !an ,i!! out!ine a 5or-!oa Characteri)ation 7atri for each ,e ser#ice.

9oaunner test scri"ts ,i!! e e#e!o"e for each e!ease/&teration/'"rint. Any functiona!test scri"ts e#e!o"e in soa"%& ,i!! e !e#erage to ,hate#er etent "ossi!e ,ith aitiona!ata #ariation to "re#ent ser#er caching.

9oaunner oes not ha#e the ca"ai!ity to im"ort a ,s!2.0=ase ,s!. 9oaunnerscri"ting ,i!! in#o!#e im"orting into soa"%& creating the esire test reuest an sa#ing thetest reuest as an m! fi!e. (he sa#e m! fi!e can then e im"rte into a 9oaunner scri"t.

6.2Cloud InfrastructureAn e#a!uation !icense of the 9oaunner Contro!!er ,as insta!!e on a 5ino,s instance in theAma)on c!ou. (he #irtua! users ,ere hoste on the same instance for e#a!uation "ur"oses.For ?A testing se"arate c!ou instance ,i!! e reuire for the Contro!!er an Virtua! %ser Hostmachines.

>


11/13



6.3Test Environment Provisioning(o get meaningfu! an rea!istic "erformance test resu!ts a test en#ironment must e"ro#isione that is re"resentati#e of e"!oyment site infrastructures. (he erformance (est!an ,i!! out!ine the "erformance test en#ironment reuirements in terms of ser#er har,are' soft,are configuration an ataase si)e.

6.4Monitoring9oaunner "ro#ies a great many statistics re!ate to transaction res"onse time an ser#eracti#ity. %sua!!y aitiona! infrastructure monitoring is he!"fu! an necessary to ena!eDe#e!o"ment to effecti#e!y tune an eug "erformance issues. (he erformance (est !an,i!! out!ine the "erformance monitoring that ,i!! e "erforme as "art of the "erformance testeecution.

5hen "erformance testing commences the ?A (eam ,i!! ,or- ,ith the CD (eam toetermine the monitoring o"tions a#ai!a!e ,ithin the Ama)on c!ou an ,ith the De#e!o"ment(eam to assure that informati#e monitoring is eing "erforme.

* Inte0ration for Traceabilit

7.1Traceability to Requirements?ua!ity Center "ro#ies a reuirements mou!e ,here the Ana!ysis (eam can esign ane#e!o" reuirements. (he Ana!ysis (eam e#a!uate this too! a!ong ,ith other euirements7anagement too!s an se!ecte Contour from JA7A as their too! of choice. ?A thereforee#a!uate a JA7A="ro#ie Contour I ?C integration too!. (he fo!!o,ing ste"s ,ere

"erforme to e#a!uate the integration. (he integration soft,are ,as insta!!e on a c!ou instance. &ntegration soft,are ,as connecte to a Contour emo "roect an a ?ua!ity Center

emo "roect oth on c!ou instances. 7a""ings ,ere one from Contour fie!s to ?ua!ity Center fie!s. Verifie that fie! u"ates in Contour ,ere "ro"agate to ?ua!ity Center for ne, an

eisting reuirements Verifie that reuirements cou! ientifie as e!onging to a re!ease

(his integration a!!o,s the Ana!ysis team to ,or- in Contour an synchroni)e the reuirementsto ?ua!ity Center. nce the reuirements are synchroni)e to ?ua!ity Center the ?A (eamuses them to e#e!o" test cases. (he traceai!ity ca"ai!ity ,ithin ?ua!ity Center a!!o,s each

test case to e ma""e to reuirementsB an therey sho, traceai!ity an test caseco#erage.

7a""ings for e!ease #a!ues are har=coe in the ma""ing too!. Venor committe to!oo-ing at more so"histicate ,ays of oing this ase on the e!eases efine in the ?ua!ityCenter e!eases 7ou!e.#a!uation #ersion of the integration i not su""ort a ?ua!ity Center "roect ,ith #ersioningturne on. Venor committe to correcting this as "art of e"!oying the soft,are to us.

10


12/13



7.2Traceability to Defects(he De#e!o"ment 'DB team is using J&A 'tuio as their "rimary ,or- management too!. Assuch 'D ,ou! !i-e to ha#e efects ientifie ,or-e an trac-e ,ithin J&A. (he ?A(eam ,ou! !i-e to use the Defect 7ou!e in ?ua!ity Center to maintain traceai!ity in ?ua!ityCenter et,een the reuirement test case an efect. (he J&A ;rige "rouct from rasi

"ro#ies the ca"ai!ity to synchroni)e efects et,een ?ua!ity Center an J&A. (his a!!o,s a?A "erson to ientify the efect in ?ua!ity Center to maintain this traceai!ity. (he e#e!o"erhas #isii!ity of the same efect in J&A. Changes can e mae to the efect an e seen ineither too!.

(he fo!!o,ing acti#ities ,ere "erforme as "art of the J&A ;rige e#a!uation. (he J&A ;rige soft,are ,as insta!!e y At!assian as an a""ro#e "!ug=in for J&A

'tuio (he J&A ;rige ,as connecte to the J&A De# "roect an a ?ua!ity Center emo

"roect. A efect ,as create in ?ua!ity Center an "ro"agate to J&A. Verifie that a target J&A "roect cou! e ientifie on the ?ua!ity Center efect an

that the efect ,as "ro"agate to the correct "roect in J&A.

Verifie that efect comments ,ere "asse ac- an forth et,een ?C an J&A. Verifie that ini#iua! fie! u"ates i.e. se#erityB ,ere "ro"agate "ro"er!y in oth

irections in accorance ,ith the efine fie! ma""ings Verifie that "ro"agation from J&A to ?ua!ity Center cou! e restricte to ust those

efects that originate in ?ua!ity Center

Defects ha#e een entere irect!y into J&A ,hi!e the ?A too! se!ection ,as

"rogressing. CD is in#estigating ,ays to o an initia! !oaG of efects fromJ&A to ?ua!ity Center. (his ,as not consiere "ass/fai! criteria of thee#a!uation.

) Continuous Inte0rationA goa! of the ?A (eam is to integrate an automate test suite into the ui! an e"!oy"rocess. As escrie in the Functiona! (esting section functiona! testing ,i!! e accom"!ishe,ith oth 'er#ice (est an a J%nit im"!ementation. (his section ,i!! iscuss the integration ofeach of these ,ith the ui! an e"!oy "rocess.

8.1Execution of soapUI Testssoa"%& tests can e eecute from the comman !ine. (herefore the eecution of a soa"%&suite using an A8( scri"t is "ossi!e ,hich can e eecute as "art of the ui! "rocess.&ntegration of soa"%& into the C& "rocess has not een "rototy"e.

8.2JUnit Tests(he choice of ,riting tests in Ja#a an using J%nit as a test frame,or- fits neat!y into the C&en#ironment as it easi!y eecutes an "u!ishes resu!ts. 'tanar Ant ui! scri"ts mayeecute J%nit tests. 'ince Ant is su""orte y the Huson C& ser#er that means a test run maye triggere automatica!!y y a successfu! e"!oyment of 'D coe. u!ishing of J%nit resu!tsis a!so nati#e!y su""orte y the Huson C& ser#er so ,e may e a!e to see the cumu!ati#eresu!ts of tests eecute o#er time. (he test frame,or- may e etene to a!!o, re"ortingac- to ?C a!though this may reuire aitiona! e#e!o"ment.

11


13/13



&m"!ementation of the J%nit frame,or- is sti!! uner e#e!o"ment.

J%nit resu!ts ,i!! nee to e integrate ,ith ?ua!ity Center #ia the ?ua!ity Center

A&.

12