ai-driven testing - a new era of test automation (camera ... · 5. ai and machine learning skills...

AI-DRIVEN TESTINGA NEW ERA OF TEST AUTOMATION

KI

N G

AR

27

M

2

019

AGENDA

2

ABOUT ME

MOTIVATION: REDEFINING TEST AUTOMATION

BACKGROUND: AI, MACHINE LEARNING, TESTING

AI-DRIVEN TESTING: A NEW ERA OF TEST AUTOMATION

CONCLUSION

3ADJUNCT PROFESSOR, COMPUTER SCIENCE

HEAD, QUALITY CENTER OF EXCELLENCE

3

AUTOMATIONMACHINES WORKING WITHOUT DIRECT HUMAN CONTROL

au·to·ma·tion/ˌôdəˈmāSH(ə)n/ noun

1. Capable of starting, operating, and completing independently.2. Use of various control systems for operating machinery and

equipment with minimal human intervention.

4

Artificial Intelligence for Software Testing (AIST)is an emerging field aimed at the developmentof AI systems to test software, methods to testAI systems, and ultimately designing softwarethat is capable of self-testing and self-healing.

9

AI-DRIVEN TESTING

TESTING AISYSTEMS

TOOLS METHODS

DESIGNS

SELF-TESTINGSYSTEMS10

AI-DRIVEN TESTING

TESTING AISYSTEMS

SELF-TESTINGSYSTEMS

DESIGNS

METHODSTOOLS

11

12

artificial intelligence:

: a branch of computer science dealing with thesimulation of intelligent behavior in computers

: the capability of a machine to imitate intelligenthuman behavior

1

2

13

14

Perceive and act in its environment using sensors and actuators, respectively.

Direct its activity toward goals and use knowledge and learning in achieving those goals.

Analyze itself in terms of behavior, error, and success and adapt online, possibly in real time.

Be organized with other agents, typically in a hierarchy to tackle problems at different abstraction levels.

Typically referred to as bots or AI.

AN AUTONOMOUS ENTITY THAT CAN…

INTELLIGENT AGENT

15

16

machine learning:the science of getting computers to act without being explicitly programmed

17

If a program can improve how it performs a certain task based on previous experience then you can say the program has learnt.

18

If the performance P of a computer program atcompleting a task T improves with experience E,the program is said to learn from experience E.

19

Hidden LayersInput Layer Output Layer

REWARD

PUNISH

20

software testing:dynamic program verification using finite tests from an infinite execution domain

21

Input Output

Program(Function)

22

Program(Function)

Domain(Input)

Range(Output)

d f(d)

Is f (d) acceptable?

¬OK (d)

OK (d)

23

REWARD

PUNISH

Hidden LayersInput Layer Output Layer

24

Program(Function)

Domain(Input)

Range(Output)

OK(t1)OK (t2)OK (t3)OK (t4)OK (t5)

T

t5

t2t1

t3t4

D f(t1)

f(t2)

f(t4)

f(t3)

f(t5)

T is successful iff∀t ∈ $ %&(t)25

Program(Function)

Domain(Input)

Range(Output)

T

t5

t2t1

t3t4

D

T is an ideal test iff∀t ∈ $ %&(t) → ∀d ∈ + %&(d)

OK(d1)OK (d2)OK (d3)OK (d4)OK (d5)OK (d6)OK (d7)OK (d8)OK (d9)

:OK (d∞)

f(d1)

f(d2)

f(d4)

f(d3)

f(d5)

f(d8)

f(d7)

f(d6)

f(d9)

f(d∞)

26

Time

Com

plex

ity

/ Cov

erag

e

Source: AI for Software Testing, Jason Arbon, CEO, test.aihttps:/www.linkedin.com/pulse/ai-software-testing-jason-arbon

SoftwareComplexity

Conventional Test Coverage

Coverage Gap

27

Time

Com

plex

ity

/ Cov

erag

e

Source: AI for Software Testing, Jason Arbon, CEO, test.aihttps:/www.linkedin.com/pulse/ai-software-testing-jason-arbon

SoftwareComplexity

Conventional Test Coverage

AI-DrivenTest Coverage

28

29

testrigor

AI-DRIVEN TESTINGFAD, FICTION, OR FUTURE?

30

AI-DRIVEN TESTINGA NEW ERA OF TEST AUTOMATION

INTERACTGenerate actions to navigate through the application UI, input data, and verify observable outcomes.

LEARNDirect feedback on the quality of an agents actions to help improve future actions.

MODELKeep track of the application

and environment state to help deal with uncertainty

and partial observability.

EXPLOREPerceive the environment

including the application UI structure, behavior and any

differences over builds.

AIDT

APPROACHHOW BOTS LEARN TO TEST SOFTWARE

32

Punish

Rewrite

Reward

Reinforce

cart home

…

ML

PINCH TAP

APPROACHHOW BOTS LEARN TO TEST SOFTWARE

33

Monitor

Analyze Plan

Execute

AI-DRIVEN TEST BOT

Machine LearningKnowledge

ARCHITECTUREDEMYSTIFYING AI-DRIVEN TESTING BOTS

34

SEEDED BASE KNOWLEDGEACQUIRES NEW KNOWLEDGE

MAINTAINS EXISTING KNOWLEDGESHARES ITS KNOWLEDGE

MAPE + KML

EXECUTIONQUEUE

EXECUTIONQUEUE

EXECUTIONQUEUE

AI-GENERATED TESTS

X X

EXPLORE AND GENERATE EXPLORE AND GENERATE EXPLORE AND GENERATERUN GENERATED TESTRUN GENERATED TEST

TEST RESULTS AND HISTORY

Monitor

Analyze Plan

Execute

TEST AGENT<<Coordinator>>

DEDUPE AND DISTRIBUTE

35

/AGENT-X/AGENT

http://github.com/UltimateSoftware

PROTOTYPESAI-GENERATION AND EXPLORATION IN TEST

36

37

38

39

FROM TEXT GENERATION TO TEST GENERATIONA LESSON LEARNED FROM ALICE IN WONDERLAND

~150,000 CharactersTotal in the book

47 Distinct CharactersWhen converted to lowercase

100-Length Subsequence TrainingN-Length Subsequence → Character Output

CHAPT → EHAPTE → R

Example (N=5)

40

be no mistake about it: it wasneither more nor less than a pig,and she felt that it would be quite aelin that she was a little want oetoiet ano a grtpersent to the tas alittle war th tee the tase oa teettee thehad been tinhgtt a little toiee at thecadl in a long tuiee aedun thet sheerwas a little tare gereen to be a gentleof the tabdit soenee the gad ouw iethe tay a tirt of toiet at the was alittle anonersen, and thiu had beenwoite io a lott of tueh a tiie andtaede bot her aeain she cere thth thebene tith the tere bane to tee toaete totee the harter was a little tire the

same oare cade an anl ano the gareeand the was so seat the was a littlegareen and the sabdit, and thewhite rabbit wese tilel an the caoeand the sabbit se teeteer, and thewhite rabbit wese tilel an the cade ina lonk tfne the sabdi ano aroing totea the was sf teet whitg the was alittle tane oo thete the sabeit shewas a little tartig to the tar tf teethe tame of the cagd, and the whiterabbit was a little toiee to be anletite thete ofs and the tabdit was thewiite rabbit, and

gooder head. and said to herself, ‘itwas a little little shated of the mouseof the good of the courte, and it wasa little little shated in a little thatthe was a little little shated of thethmee said to see it was a littlebook of the was a little that she wasso sereated to hare a little the begansitee of the was of the was a littlethat she was so seally and thesabbit was a little lad good to thelittle gooder head of the gad searedto see it was a little lad good to thelittle good

she sabbit said to herself and thesabbit said to herself and the soodway of the was a little that she wasa little lad good to the garden, andthe sood of the mock turtle said toherself, 'it was a little that the mockturtle said to see it said to sea itsaid to sea it say it the marge hardsat hn a little that she was sosereated to herself, and she sabbitsaid to herself, 'it was a little littleshated of the sooe of the coomouse itwas a little lad good to the little

herself lying on the bank, with herhead in the lap of her sister, who wasgently brushing away so siee, and

ABSTRACT TEST LANGUAGE

AI-DRIVEN TEST GENERATIONMACHINES LEARNING FROM HUMAN TESTERS

43

Observe ErrorMessageInput BLANK FirstName Invoke Submit Observe Required TextBox FirstName

First Name* Last Name*

Company

A Model-Based AI-Driven Test Generation System, Dionny Santiago. Masters Thesis Proposal, Florida International University (2018).


44

Observe ErrorMessage

IFPerceived Conditions

Input BLANK FirstName Invoke Submit Observe Required TextBox FirstName


Company



45



WHENInput Actions



Company



46



WHENInput Actions

THENRequired Observations



Company

Please enter first name. Please enter last name.



47


Input Sentence Predicted Next Word

Observe Required

Observe Required Textbox

Observe Required Textbox FirstName

Observe Required Textbox FirstName Input

Observe Required Textbox FirstName Try VALID

Observe Required TextBox FirstName Try VALID FirstName Invoke

Observe Required TextBox FirstName Try VALID FirstName Click Submit

Observe Required TextBox FirstName Try VALID FirstName Click Commit NotObserve

Observe Required Textbox FirstName Try VALID FirstName Click Commit NotObserve ErrorMessage

48

PESTICIDE FREEGenerates New Tests Each Time

Eliminates Pesticide Paradox

GENERAL PURPOSEDifferent Testing Types

Any Application or Domain

LARGE SCALEDeep Learning

Cloud Infrastructure

AI-DRIVEN TEST GENERATIONKEY ADVANTAGES

50

LIMITATIONS?

51

PROGRAM(Function)

INPUT(Domain)

OUTPUT(Range)

53

MACHINE LEARNING(Function)

INPUT(Domain)

OUTPUT(Range)

54

SELF-TESTINGAI-DRIVEN TESTING FROM THE INSIDE-OUT

55

Self-Configure

Self-Optimize

Self-Heal

Self-Protect

Self-Test

Software Component

Monitor

Analyze Plan

ExecuteKnowledge

(Brain)

Autonomic Test Manager

A Self-Testing Approach for Autonomic Software, Tariq M. King. ProQuest Electronic Theses and Dissertations (2009).

AUTONOMIC SELF-TESTINGTESTING DYNAMICALLY ADAPTIVE SYSTEMS

56

AI-DRIVEN TESTING

TESTING AISYSTEMS

SELF-TESTINGSYSTEMS

DESIGNS

METHODSTOOLS

57

AI-DRIVEN TESTING

TESTING AISYSTEMS

TOOLS METHODS

DESIGNS

SELF-TESTINGSYSTEMS58

DIONNY SANTIAGO

DAVID ADAMO

JUSTIN PHILLIPS

PATRICK ALT

KEITH BRIGGS

NICOLETTE CELLI

PETER J. CLARKE

JASON ARBON

THANK YOUACKNOWLEDGEMENTS

59

REFERENCES1. Test Machina: Demystifying AI-Driven Testing Bots, Tariq M. King, Automation Guild, Feb 2019.

2. AI-Driven Testing: A New Era of Test Automation, Tariq M. King, CIO Review Software Testing Special, Dec 2018.

3. A Model-Based AI-Driven Test Generation System, Dionny Santiago, ProQuest Electronic Theses and Dissertations, Dec 2018.

4. Abstract Flow Learning for Web Application Test Generation, Dionny Santiago, Peter J. Clarke, Patrick Alt, and Tariq M. King, European Software Engineering Conference/Foundations of Software Engineering International Workshop on Automating TEST Case Design, Selection, and Evaluation, Nov 2018.

5. AI and Machine Learning Skills for the Testing World, Tariq M. King and Jason Arbon, STAREAST, May 2018.

6. Text Generation with LSTM Recurrent Neural Networks in Python with Keras, Jason Brownlee, Aug 2016.

7. A Self-Testing Approach to Autonomic Software, Tariq M. King, ProQuest Electronic Theses and Dissertations, Jul 2009.

8. Towards a Theory of Test Data Selection, John B. Goodenough and Susan L. Gerhart, International Conference on Reliable Software, Jun 1975.

60

ai-driven testing - a new era of test automation (camera ... · 5. ai and machine learning skills...

Documents