example of taf with batch execution of test cases
TRANSCRIPT
![Page 1: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/1.jpg)
CONSOLIDATED TEST CASESEXECUTION
Dmitry Hes
SEPTEMBER, 2016
![Page 2: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/2.jpg)
2CONFIDENTIAL
CONSOLIDATED TEST CASE EXECUTION
INTRODUCTION
![Page 3: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/3.jpg)
3CONFIDENTIAL
INTRODUTION
![Page 4: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/4.jpg)
4CONFIDENTIAL
INTRODUTION
Takes much time
• long-running process (> 5 min)
• modify database significantly
Integration processes
![Page 5: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/5.jpg)
5CONFIDENTIAL
INTRODUTION
tim ecount
… integration processes are invoked many
times
![Page 6: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/6.jpg)
6CONFIDENTIAL
INTRODUTION
How to speed up?
![Page 7: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/7.jpg)
7CONFIDENTIAL
INTRODUTION
![Page 8: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/8.jpg)
8CONFIDENTIAL
CONSOLIDATEDAPPROACH
![Page 9: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/9.jpg)
9CONFIDENTIAL
TestcaseArrangement
test data(create e-
mail)
Sync processes
(Set data in DB)
Build-mode
lVerify data
REAL EXAMPLETestcase
Arrangement test data(create e-
mail)
Sync processes
(Set data in DB)
Build-mode
lVerify data
…Testcase
Arrangement test data(create e-
mail)
Sync processes
(Set data in DB)
Build-mode
lVerify data
many times with unique test data
![Page 10: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/10.jpg)
10CONFIDENTIAL
REAL EXAMPLE
Could we use parallelization? (setup scope of environments and run TCs on each)
No.• We are unable to run build-model against one server• Only one process must run at the same time• Therefore we need setup whole environment on each VM• It takes about 20 minutes (some installation scripts
execution require• So no sense • Moreover we don’t have so much free VMs
![Page 11: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/11.jpg)
11CONFIDENTIAL
REAL EXAMPLE
Execution time of 1 test case: 10–15 minutes
Number of test cases: 70
Total running time: 875 minutes (14.5 hours)
![Page 12: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/12.jpg)
12CONFIDENTIAL
SOLUTION
Sync processes
(Set data in DB)
0Build-mode
l
Test
case Arrangement
test data(create e-
mail)
Test
case Arrangement
test data(create e-
mail)
…
Test
case Arrangement
test data(create e-
mail)
Test
case Verify
data
Test
case Verify
data
…Te
stca
se Verify data
![Page 13: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/13.jpg)
13CONFIDENTIAL
CONSOLIDATEDFRAMEWORK
![Page 14: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/14.jpg)
14CONFIDENTIAL
CONSOLIDATED FRAMEWORK
Test data csv-file should be created/updated before
running consolidated Test caseby executing
ParameterExportByTestCaseID test method.
Test class implementationStructure
[TestMethod]public void MasterTestSuite()
- testCaseData - <variables to store test data> ...
<Test class>
CNTestCaseBase
+ setup()reads test data from the appropriate csv file into and stores it to local dictionary and variables of method instance testCaseData =
CNTestSuiteExtensions .LoadTestCaseData();+ exec()
create and setup test data needed to TC like create mails, contacts, meeting, etc.
+ validate()method that perform validation steps (after processes execution)
+ cleanup()clean up environment after TC execution
Test caseCSV file:
../../../CNAutomationProject /TFS.Parameters/<test case id>.csv
[TestMethod]<Test class>()
Start
Create TC…
classes instances
For each instanceexecute setup();execute exec()
Execute sync- processes
Execute build-model process,iisreset
For each instanceexecute
validate()
End
For each instanceexecute
cleanup()
[TestClass]TFSParameterExportByTestCaseID[TestMethod]+ ParameterExportByTestCaseID() Gets test parameters data from TFS for specified test cases and stores them in appropriate csv-file
![Page 15: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/15.jpg)
15CONFIDENTIAL
CONSOLIDATED FRAMEWORK
0Build-model0
CRM sync
process
Sync to CRM
example 2
![Page 16: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/16.jpg)
16CONFIDENTIAL
CONSOLIDATED FRAMEWORK
Test class implementation
Structure[TestMethod]public void BatchExecution_TestMethod ()
+ ctor <Test class>
Invokes setup()
<Test class>
TestCaseBase
+ setup()reads test data from the appropriate csv file into and stores it to local dictionary and variables of method instance
[ + Validation1() ]
if required
[ + Validation2() ]
if required
[TestClass]<Test class>()
Start
Create TC…
classes instances
ExecuteProcess 1
End
Implementation 2
For each instanceExec Validation
1
ExecuteProcess 2
For each instanceExec Validation
2…ExecuteProcess N
For each instanceExec Validation
N
…[ + ValidationN() ]
if required
![Page 17: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/17.jpg)
17CONFIDENTIAL
PROS AND CONS
Advantages Disadvantagessignificantly reduced TC run time complex test case structure
able to manage execution order test data should be unique
test data is read from local CSV huge test log
difficult to add new methods
MS test methods like Class/Test initialize or cleanup are not executedhard to adopt existing TCs to consolidated approach
![Page 18: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/18.jpg)
18CONFIDENTIAL
NEXT STEP
![Page 19: Example of TAF with batch execution of test cases](https://reader034.vdocuments.net/reader034/viewer/2022051007/5872910e1a28ab36118b6333/html5/thumbnails/19.jpg)
19CONFIDENTIAL
1. Make test case more flexible
• ability to change testing workflow process by managing pool of executable methods
2. Add execution of ClassInitialize, TestInitialize, ClassCleanup, TestCleanup accordingly
NEXT STEP