1. Asakusa #4 Asakusa Test Driver 2011/08/26 (@apirakun)

2. Asakusa#4 2 lA Brief History of Asaksa Test Driver lTestDriver (ver 0.1) lTestDriver (ver 0.2) lTestDriver lAsakusa CI on AWS lSource Code Reading lTestDriver ver 0.2 API 2 3. Asakusa#4 3 l (@apirakun) lAsakusa Framework nAsakusa Test Driver/ n CI (Jenkins) n (GitHub) n (Maven) n (Sphinx) n (misc.) l n SI:Web n 4. Asakusa#4 4 lSource Code (branch:0.2-develop) lhttps://github.com/asakusafw/asakusafw/ tree/0.2-develop nasakusa-test-driver nasakusa-test-modelator nasakusa-test-data-provider nasakusa-thundergate-test-modelator lDocuments () lhttp://asakusafw.s3.amazonaws.com/ documents/0.2/release/ja/html/testing/ index.html 4 5. Asakusa#4 5 TestDriver (ver 0.1) Hadoop 5 6. Asakusa#4 6 Asakusa Framework lDSL 6 Java Asakusa DSL Map Reduce DSL DMDL Review 7. Asakusa#4 7 Batch Dataflow ProcessDataflow Part DSL l 7 UoW #1 UoW #2 UoW #3 UoW #4 1. 1.1. 1.2. 2. 2.1. 2.2. 2.a. 3. 4. Batch DSL Flow DSL Operator DSL Review 8. Asakusa#4 8 DSL l DSL lOperator lFlowPart lJobFlow lBatch l l l 9. Asakusa#4 9 DSL lOperator lJava lModel/ lFlowPart lDSLMapReduce Job lSequenceFile lModel/ 10. Asakusa#4 10 DSL lJobFlow l(ThunderGate) lBatch l n(experimental.sh) 11. Asakusa#4 11 lModel lModelTemplat e lModel lExcel l l/ lDB lDBUnit 12. Asakusa#4 12 lDB lHadoop n 13. Asakusa#4 13 l lCI l n n nThunderGateHadoopJob nCI Server l n 14. Asakusa#4 14 l lDSL lFlowPartJobFlow nSequencFile/DB lJobFlowBatch n hadoop/experimental.sh 15. Asakusa#4 15 l lAPI n API n n 16. Asakusa#4 16 lJava lEclipse lJUnit 17. Asakusa#4 17 l l nCI n lWindowsHadoop nLinux(AsakusaSDK) l n 18. Asakusa#4 18 Asakusa Test Driver (0.1) lTestDriver(0.1) ltestdriver.XXXTestDriver nFlowPart-/JobFlow-/Batch- l lAshigel Compiler/HadoopJob/ThunderGate lModelGenerator Template(Excel) lDSLAPI l l com.asakusafw 19. Asakusa#4 19 TestDriver TestDriver 19 20. Asakusa#4 20 TestDriver(0.1) l lThunderGate:ThunderGate lExcel:Excel l0.1 l0.1TestDriver () 21. Asakusa#4 21 Asakusa Test Driver (0.2) lNewAPI ltestdriver.XXXTester nFlowPart-/JobFlow-/Batch- l l ThunderGate n l Excel nJSONExcel nExcel0.1(Calc) l SPI n Ashigel Compiler 22. Asakusa#4 22 TestDriver(0.2) l 23. Asakusa#4 23 TestDriver(0.2) l 24. Asakusa#4 24 Test Data TestDriver(0.2) Import Tgt Batch App Import Export Export Tgt M/R Job M/R Job M/R Job Input Resource Expected Resource Verify Resource TestDriver Preparator Inspector Executor Compiler Batch Compile Execute Import Resource Export Resource Asakusa DSL 25. Asakusa#4 25 TestDriver 25 26. Asakusa#4 26 lDistributed Testing l lCloud Testing lAWS 27. Asakusa#4 27 lDefine the relationship betweeninput and output l nMOVE l( ) 28. Asakusa#4 28 l l 29. Asakusa#4 29 Asakusa CI on AWS Asakusa Framework CI 29 30. Asakusa#4 30 Asakusa FrameworkCI lAWS(EC2/S3)CI lhttp://ec2-175-41-241-89.ap- northeast-1.compute.amazonaws.com/ jenkins/ 31. Asakusa#4 31 Amazon S3Amazon EC2 Asakusa CI Jenkins Master GitHub Development Environment git push Post Receive URLs Jenkins Slave distributed build Maven Repository Documents deploy copy tweet Maven Archetype Catalog Twitter 32. Asakusa#4 32 MasterSlave lJenkins Master l lJenkinsWeb lJenkins Slave l lHigh-CPU n lAmazon EC2 Plugin lJenkinsEC2AMI lIdle30 lCI:1,0002,000 33. Asakusa#4 33 Jenkins Amazon EC2 Plugin lJenkins SlaveAMI 34. Asakusa#4 34 GitHubJenkins lGitHubJenkinspush lGitHubPost Receive URLs ngit pushJenkinsget lgit push nPost Receive URLs Request URL nJenkins Job nJobJob Job n nJenkinsDownstream-Ext Plugin n 35. Asakusa#4 35 Build Pipeline l push-hook build- kicker build-main archetype build- report examples deploy GitHub get request Jenkins Master Jenkins Slave 36. Asakusa#4 36 lTwitter l@asakusa_ci 37. Asakusa#4 37 Asakusa Test Driver Source Code Reading 37 38. Asakusa#4 38 Test Data TestDriver(0.2)() Import Tgt Batch App Import Export Export Tgt M/R Job M/R Job M/R Job Input Resource Expected Resource Verify Resource TestDriver Preparator Inspector Executor Compiler Batch Compile Execute Import Resource Export Resource Asakusa DSL 39. Asakusa#4 39 Compiler lDSL lAshigel Compiler nDirectFlowCompiler nDirectBatchCompiler nAsakusa#1 lTestDriver n n 40. Asakusa#4 40 Preparator DataModel Source ModelOutput Importer Preparator Source Provider Model (Reflection) DataModel Source ModelOutput Source Provider DataModel Source Model Output Asakusa DSL Importer Description Import Tgt Import Resource Test Data Input Resource Object TestData SPI 41. Asakusa#4 41 SPI ltestdriver.core.SourceProvider lURIModel (DataModelSource) lver0.2.1 ntestdriver.json.JsonSourceProvider ntestdriver.excel.ExcelSheetSourceProvider 42. Asakusa#4 42 SPI ltestdriver.core.ImporterPreparator lImport (ModelOutput) lver0.2.1 ntestdriver.file.FileImporterPreparator ntestdriver.bulkloader.BulkloaderImporterPreparat or 43. Asakusa#4 43 ThunderGate/Excel DataModel Source ModelOutput BulkloaderImpor terPreparator Source Provider Model (Reflection) DataModel Source ModelOutput ExcelSheetSou rceProvider ExcelSheetDat aModelSource Table Output Asakusa DSL BulkloaderImpor terDescription Import Tgt MySQL Test Data Exeel Sheet Object TestData SPI 44. Asakusa#4 44 Executor l lJobflowInfo lExternalIoCommandProvider nver0.2.1OS(Linux) 45. Asakusa#4 45 Inspector DataModel Source Exporter Retriever Source Provider DataModel Source Source Provider DataModel Source DataModel Source Asakusa DSL Exporter Description Export Tgt Export Resource Test Data Expected Resource Verify Resource VerifyRule Provider DataModel Source DataModel Source Verify Rule Model (Reflection) Model (Reflection) Object TestData SPI 46. Asakusa#4 46 SPI ltestdriver.core.ExporterRetriever lExportModel (DataModelSource) lver0.2.1 ntestdriver.file.FileExporterRetriever ntestdriver.bulkloader.BulkloaderExporterRetriever 47. Asakusa#4 47 SPI ltestdriver.core.VerifyRuleProvider lURI (VerifyRule) lver0.2.1 ntestdriver.excel.ExcelSheetRuleProvider lJava ntestdriver.core.ModelVerifier 48. Asakusa#4 48 l: l nExcel n n 49. Asakusa#4 49 SourceProvider DataModel Source Exporter Retriever Source Provider DataModel Source Source Provider DataModel Source DataModel Source Asakusa DSL Exporter Description Export Tgt Export Resource Test Data DataBase Table Excel Sheet VerifyRule Provider DataModel Source DataModel Source Verify Rule Model (Reflection) Model (Reflection) Object TestData SPI DBTable SourceProvider 50. Asakusa#4 50 lIssue#64:Enable to input expect data from database table. lhttps://github.com/asakusafw/asakusafw/issues/64 l lTableSourceProvider ntestdriver.bulkloader.TableSourceProvider lSPI nsrc/main/resources/META-INF/services/ com.asakusafw.testdriver.core.SourceProvider l lURI n bulkloader:: n :bulkloader:asakusa:ORDER