a test scenario automatic generation strategy for intelligent … · 2018. 6. 27. · intelligent...

Research ArticleA Test Scenario Automatic Generation Strategy forIntelligent Driving Systems

Feng Gao 1 Jianli Duan1 Yingdong He2 and ZilongWang3

1School of Automotive Engineering Chongqing University Chongqing 400044 China2Mechanical Engineering University of Michigan MI 48109 USA3China Automotive Technology and Research Center Tianjin 300300 China

Correspondence should be addressed to Feng Gao gaofeng1cqueducn

Received 27 June 2018 Revised 30 September 2018 Accepted 30 December 2018 Published 15 January 2019

Academic Editor Emilio Insfran Pelozo

Copyright copy 2019 FengGao et alThis is an open access article distributed under the Creative Commons Attribution License whichpermits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

In this paper a methodology of automatic generation of test scenarios for intelligent driving systems is proposed which is based onthe combination of the test matrix (TM) and combinatorial testing (CT) methods together With a hierarchical model of influencefactors an evaluation index for scenario complexity is designed Then an improved CT algorithm is proposed to make a balancebetween test efficiency condition coverage and scenario complexity This method can ensure the required combinational coverageand at the same time increase the overall complexity of generated scenarios which is not considered by CT Furthermore the wayto find the best compromise between efficiency and complexity and the bound of scenario number has been analyzed theoreticallyTo validate the effectiveness it has been applied in the hardware-in-the-loop (HIL) test of a lane departure warning system (LDW)The results show that the proposed method can ensure required coverage with a significantly improved scenario complexity andthe generated test scenario can find system defects more efficiently

1 Introduction

In recent years with the development of intelligent driv-ing technologies such advanced driver assistance systems(ADAS) as auto emergency braking (AEB) forward collisionwarning (FCW) lane departure warning (LDW) etc havebeen put into market rapidly [1] Compared with the tradi-tional onboard electronic systems the working condition ofsuch systems is uncontrolled and indefinable exactly whichcauses the traditional test scenario design approach to beinapplicableOn the other hand since these systems influencethe driving safety directly sufficient and comprehensive testsare necessary before they go public [2 3]

At this stage the approaches of designing the test scenariofor ADAS can be divided into the following five types mainlynaturalistic field operational test (N-FOT)Monte Carlo sim-ulation (MCS) accelerated evaluation method (AE) worst-case scenario evaluation (WCSE) and test matrix approach(TM) When using N-FOT vehicles equipped with ADASto be tested are driven by multiple drivers in real trafficfor data collection over a quite long period to achieve

enough coverage [4] It can test the system under the realworking condition but the occurrence probability of criticalcondition is very low and most of the time the system maybe tested under the similar and simple scenarios Moreoverthe real traffic is uncontrollable which makes it difficult toensure a good test consistency [5] To improve N-FOT someresearchers use the data collected from naturalistic traffic tobuild a stochastic driving behavior model to generate testscenarios which is called MCS Based on this approach Yanget al developed a car-following drivermodel to evaluate FCWand AEB [6] Compared with N-FOTMCS can partly extendthe test condition by the driving behavior model ensure agood consistency with real traffic and reduce the similar andsimple scenarios to increase test efficiency Unfortunately thecritical condition is still hardly to be generated because of thelack of original traffic data

To generate more critical scenarios Zhao et al [7] andHuang et al [8] proposed AE which applies importance-sampling theory to speed up the evaluation process by findingthe most critical scenarios However this method only takeslimited influence factors into account for some specific

HindawiMathematical Problems in EngineeringVolume 2019 Article ID 3737486 10 pageshttpsdoiorg10115520193737486

2 Mathematical Problems in Engineering

driving assistance function which restricts its applicationrange In order to directly extract the most critical scenariosWCSE was proposed based on the database built up fromthe traffic accidents Andreas validated the emergency laneassist system based on these most dangerous scenes [9]These extreme scenarios help to find the faults and achievableperformance of ADAS but the coverage of all related factorscannot be measured quantitatively This makes it difficultto determine when to stop the validation and evaluationprocess

As a widely used method to construct the standardizedtest scenarios TM has also been applied in the formulation oftest standards for ADAS eg ISO 15622 for ACC [10] AEBin EuroNCAP [11] etc TM can take all factors relating to thetested system into consideration Therefore compared withother methods it can be used to design higher diversity andcomplex scenarios with an acceptable testing cost better con-trollability and preferable repeatability However the currentTM has the following problems when applied on ADAS test[12] (1) the number of test scenarios and considered factorsis too few to achieve the coverage requirement Otherwisethe mostly used orthogonal experiment method (OE) for TMleads to ldquodimensional disasterrdquo when the number of consid-ered influence factors increases (2) and its test efficiency isvery low when failures are only triggered by the interactionof a small number of factors

To solve these issues of TM we introduce the combina-torial testing method (CT) which has already been widelyused in computer software testing [13] The motivation ofCT is based on the fact that most of the program errorsare caused by the interaction of multiple influence factors[14] The set of test scenarios generated by CT can guaranteefull coverage of the required n-wise combination with asfew scenario number as possible Cohen et al proposedAutomatic Efficient Test Generator (AETG) algorithm whichgenerates a certain number of test scenarios at each timeby using the greedy algorithm and then picks one thatcan cover most of the n-wise combination of factors stilluncovered [15] Czerwonka developed another CT tool calledPairwise Independent Combinatorial Testing tool (PICT)which is unlike AETG algorithm and need not producetest scenarios to be selected beforehand [16] Instead ituses a fixed random seed to ensure the consistency of theoutput To trade off a reduction of the number of testscenarios James developed a new tool called Allpairs tofind a reasonably small set of test scenarios to satisfy thefull coverage of arbitrary pairwise combination [17] Wuet al used the particle swarm optimization algorithm togenerate test suites ensuring the coverage criteriawith smallertest set than other metaheuristic strategies [18] Garvin etal modified the original simulated annealing algorithm tofurther improve the efficiency of test suite generation [19]Gonzalez-Hernandez used the Mixed-Tabu Search to buildthe test suite with uniform strength [20]

All the aforementioned CT methods focus on the reduc-tion of test set while ensuring the combinatorial coverage atthe same timeThe complexity of scenario has not been takeninto account when generating combinatorial test scenarioswhich is beneficial to test effectiveness because a system tends

Factorlayer1

Factorlayer2

Factorlayer

layer

3

Value

Intelligent drivingsystem F11

Environment F21

Self-stateF22

Otherparticipants

F23

WeatherF31

TimeF32

Sunny31

Cloudy32

Rainy33

Foggy34

Figure 1 Tree structure model of influence factors

to malfunction under more complex conditions Therefore anew method called combinatorial testing based on scenariocomplexity (CTBC) is presented in this paper for automaticgenerating the test scenarios for intelligent driving systemwhich can achieve full coverage of n-wise combinationwith a relative compact test set meanwhile increasing thescenario complexity The following paper is organized asfollows Section 2 describes the model of influence factorsand potential problems Section 3 introduces the procedureand principle of the proposed CTBC whose performance isanalyzed theoretically in Section 4 Section 5 validates CTBCby application and comparative analysis and finally Section 6concludes the paper

2 Problem Description

Before discussing the problem of automatic generation oftest scenario for intelligent driving systems a mathematicalmodel is introduced to describe the influence factors ofintelligent driving system which act as the input of theautomatic generation process of test scenarios

21 Tree Structure Model of Influence Factors The compo-nents used to construct the test scenario are called influencefactors in this paper which are related to the functionalityof intelligent driving system These factors can be obtainedthrough a variety ways such as technical specificationsnaturalistic trafficdata etc To ensure the comprehensivenesssuch systematic methods eg classification tree [21] aresuggested to analyze the possible influence factors In generalthe influence factors of an intelligent driving system can bedivided into three types environment self-state and otherroad participants

To realize automatic generation of test scenarios theinfluence factors should be discretized to get distinct valueswhich can be generated by such black box test scenariosdesign methods as equivalence class partition and boundaryvalue processing [12] Finally a tree structure model ofinfluence factors can be derived as shown in Figure 1 asan example Table 3 in the Appendix shows the detailedinfluence factors of LDW

Mathematical Problems in Engineering 3

Factorsin

end

Corres-ponding

value

Weather F1

Time F2

Rapidchanges in light

F3

Sunny 1p1

Day(800-1700)

2p2

Pass underfootbridge

3p3

Combining all factors withtheir values together

Onetest scenario

Figure 2 Test scenario construction process

Then the factors and their values can be described by ageneral mathematical linguistic form as shown by (1) and (2)

F119894119895 = F119894+1119896 119896 isin Ω119894119895119894 isin [1 sdot sdot sdot 119878 minus 1] 119895 isin [1 sdot sdot sdot 119871 119894]

(1)

where F119894119895 is the 119895-th factor at the 119894-th layer Ω119894119895 is composedof all subfactors of F119894119895 119878 is the number of factor layer and 119871 119894is the number of factors in the 119894-th layer

All subfactors belonging to the same parent factor canappear in one scenario while the different values of thesame factor at the bottom layer cannot Therefore we needa different formula to describe all possible values of factorslocating at the bottom factor layer

F119894119895 = V119894119896 119896 isin Ω119894119895 (119894 119895) isin Ω119890119899119889 (2)

where V119894119896 is the 119896-th value of F119894119895 andΩ119890119899119889 is composed of thesubscripts of all factors at the bottom layer

Finally the test scenario can be built by combining allthe factors at the bottom factor layer with one of theircorresponding values together To simplify the descriptionlet |∙| denote the number of elements belonging to a setThenthe end node factors can be denoted as F119902(119902 = 1 sdot sdot sdot 119873)where119873 = |Ω119890119899119889| And each factorrsquos values can be denoted asV119902119901119902(119901119902 = 1 sdot sdot sdot |F119902|) The mathematical expression of testscenarios is

T = [119905119894119895] isin R119872times119873

119905119894119895 isin V1198951 sdot sdot sdot V119895|F119895| 119894 = 1 sdot sdot sdot 119872 119895 = 1 sdot sdot sdot 119873 (3)

where T is a matrix including all test scenarios 119872 is thenumber of test scenario and 119905119894119895 denotes the value of 119895-thfactor in the 119894-th scenario The construction process of a testscenario is shown in Figure 2 as an example

22 ProblemsAnalysis Based on the structuremodel of influ-ence factors of LDW shown in Table 3 in the Appendix theproblem of generation of effective test scenario is discussed inthis section There are 16 influence factors at the bottom level

which have 58 values totally The number of test scenariosdefined in ISO 17361 is only 8 [22] which is far away to coverall possibilities and its error detection capability is pretty badSuch standard test scenarios only can be used to validatethe primary functionality On the other hand the scenariosgenerated by OE can ensure the coverage of all influencefactors but the number of scenarios reaches 349920000which is unacceptable in practice because of the test cost Byusing the PICT the number of scenarios ensuring pairwisecombination is reduced to 53 [23] However the percentage ofscenarios with higher complexity cannot be improved underwhich LDW may malfunction easily Therefore in order toincrease the proportion of scenario with higher complexity tofurther improve the test efficiency an improved CT methodcalled CTBC is to be introduced in the next section

3 Combinatorial Test Generation MethodBased on Complexity

To control the complexity of the generated test scenarios weneed an index to measure the contribution of factor value tothe complexity of scenario

31 Complexity Index of Scenario The contribution of eachfactor or value to the complexity of scenario can be deter-mined by the Analytic Hierarchy Process (AHP) whichis widely used in the field of engineering because of itsability to make quantitative analysis of subjective evaluation[24] The calculation process is based on the tree structuremodel of influence factors as shown in Figure 1 Accordingto their relative contribution a judgment matrix A119894119895 can beconstructed as

A119894119895 = [119886ℎ119891] isin R|Ω119894119895|times|Ω119894119895 |

119894 isin [1 sdot sdot sdot 119878 minus 1] 119895 isin [1 sdot sdot sdot 119871 119894] ℎ 119891 = 1 sdot sdot sdot 10038161003816100381610038161003816Ω11989411989510038161003816100381610038161003816(4)

where 119886ℎ119891 represents a comparison of the relative importanceof any two elements in Ω119894119895 and it can be determined bySaatyrsquos scaling law [24] It also has the following properties

119886ℎ119891 =

1 ℎ = 1198911119886119891ℎ ℎ = 119891 (5)

Then the normalized eigenvector corresponding to themaximum positive eigenvalue of A119894119895 is

W119894119895 = [120596119894119896] isin R1times|Ω119894119895| 119896 isin Ω119894119895 (6)

where 120596119894119896 represents the relative importance of influencefactors or its corresponding values according to [24]

However the relative importance degree is not com-parable for the factors belonging to different categoriesTherefore to measure the importance of different valuesused to construct test scenario uniformly and equally thefollowing importance index of value is used

119894119896 = prod(119886119887)isinΩ119894119896

120596119886119887 119896 isin Ω119894119895 (119894 119895) isin Ω119890119899119889 (7)


Table 1 Variables used in pseudocode of CTBC

Variables Meaningsthreshold The threshold value120573 The complexity improvement coefficient 120573 isin [0 100]uncovered The set of combinations that remained uncoveredcomb The combination in uncoveredcombweight The sum of the valuesrsquo importance indicessetofweight The set of combweighttestscenario The new test scenariobest The first chosen combination with max(combweight) in current uncovered in testscenarionextbest The combination with max(combweight) in current uncovered

where Ω119894119896 is composed of the subscript of the nodes inthe route from V119894119896 to the root node F11 Table 3 shows theimportance index of factor values of LDW Then with theimportance index the complexity index C119894 of the i-th testscenario in T is defined as

C119894 =119873sum119895=1

119894119895 (8)

where 119894119895 is the importance index of 11990511989411989532 Complex Index Based CT Algorithm In this section aCTBC algorithm is introduced to improve the test scenariocomplexity while ensuring required combinational coverageAn idea to improve the scenario complexity is that the factorvalue with higher importance index is preferred to generatethe test scenario Naturally more complex scenarios can befound at the beginning of automatic generation process Butthe importance index of the left factor value not coveredbecomes very small It is impossible to construct complexscenario using these factor values which is bad for the overallcomplexity of all generated scenarios

To overcome this problem a threshold is defined tochoose the aspect to be considered when generating testscenarios When generating a new scenario if the summedimportance index of the selected factor values is larger thanthis coefficient the unselected factor values with the mostcombinational coverage is preferred Otherwise the impor-tance index is considered preferentially when determiningthe value of left factors of a scenario By this way a tradeoffbetween combinational coverage and scenario complexity ismade

For a given tree structure model of influence factorswith their importance index (4) there exists a boundaryof achievable maximum or minimum complexity of sce-nario If the selected threshold is less than the minimumcomplexity the CTBC algorithm generates scenarios onlyconsidering the combinational coverage requirement likeAETG In Section 41 a complexity improvement coefficientis proposed to determine a proper threshold

The CTBC algorithm generates one test scenario at atime and lasts until the combinations of all possible valueshave been covered by at least one scenario Based on the

aforementioned idea and the variables defined in Table 1 thepseudocode of CTBC is shown in Algorithm 1

When programming the executable code the followingshould be noted

(a) In order to accelerate the searching process alluncovered combinations are put in a red-black tree accordingto the sum of importance indices in the descending order[25]

(b) Moreover the requirement of reproducibility andcertainty is important for the test of ADAS Therefore thelexicographical order is used to ensure the consistency of gen-erated scenarios when there exist multiple choices providingthe same sum of importance indices [26]

4 Performance Analysis of CTBC

The generated test scenarios are evaluated mainly by twoaspects (a) test cost measured by the number of scenarios (b)and test effectiveness evaluated by the overall scenario com-plexity in this paper To make an optimal tradeoff betweenthese two aspects a complexity improvement coefficient isproposed by which the best test scenarios can be found witha statistical method

41 Optimization by Complexity Improvement CoefficientThe purpose of introducing this complexity improvementcoefficient 120573 is to improve the overall scenario complexity

120573 = 119905ℎ119903119890119904ℎ119900119897119889 minus C

C minus Ctimes 100 (9)

where C and C are the minimum and maximum achievablescenario complex indexes From the fundamental of CTBCa more complex scenario can be obtained by increasing 120573but the number of needed scenarios ensuring combinationalcoverage may also increase To make a balance between thetest cost and effectiveness to find the best test scenarios theoverall complexity of test scenarios is defined firstly

C (120573) = sum119872(120573)119894=1 C119894 (120573)119872 (120573) (10)

where C(120573)119872(120573) and C119894(120573) are the overall complexity thenumber of test scenarios and the 119894-th scenariorsquos complexity


1 Input F119902(119902 = 1 sdot sdot sdot 119873) V119902119901119902 (119901119902 = 1 sdot sdot sdot |F119902|) 119902119901119902 119899 1205732 Output T3 Obtain uncovered4 for all comb in uncovered do5 119888119900119898119887119908119890119894119892ℎ119905 = sum

119902119901119902isincomb119902119901119902

6 Add combweight to setofweight7 end for

8 119905ℎ119903119890119904ℎ119900119897119889 = 120573 119873sum119902=1

max(119902119901119902)9 while uncovered = 0 do10 best = combination with max(combweight) in uncovered11 for all 119894 in 119899 do12 Assign factors and values from combination that corresponding to best to testscenario13 Remove best from uncovered14 end for15 if best gt threshold then16 while ((length of testscenario) lt 119873) do17 nextbest = combination with max(combweight) in uncovered18 Comparing values of the same factors between nextbest and testscenario19 if there exist conflicting values then20 continue to select the next combination corresponding to nextbest in the descending order of combweight21 end if22 Factors and values that corresponding to nextbest but not exist in testscenario are assigned to testscenario23 Remove nextbest from uncovered24 end while25 else26 Assign factor F119902 that are not contained in testscenario with the value of max(119902119901119902)27 end if28 Add testscenario to T29 end while

Algorithm 1 Pseudocode for CTBC

index which all are functions of the complexity improvementcoefficient 120573

Here the AHP method is used again to determine theweighting of test cost and effectiveness [24] Then the opti-mization problem can be described by

max120573

119885119885 = 1198781Clowast minus 1198782119872lowastClowast = C (120573) minus C

C minus C

119872lowast = 119872(120573) minus119872119872 minus119872

(11)

where 119885 denotes the composite test effect 1198781 and 1198782 arethe normalized weight for scenario complexity and test costdetermined by AHPmethod Clowast and119872lowast are the normalizedvalue of complexity and test cost and 119872 and 119872 are themaximum and minimum number of scenarios In (8) theldquomin-max scalingrdquo process is used to put C(120573) and 119872(120573) inthe same scale to avoid the challenge of selecting a properweight

For a given tree structure model of influence factorsand its importance index values the achievable maximumor minimum complex index of scenario can be derived byselecting the factor value with highest or lowest importanceindex Unfortunately the range of scenario number cannotbe obtained from the previously known information of thetested system which is used to normalize the test cost in(8) In the following section an approximated method ispresented to estimate the range of scenario number

42 Estimation of Test Scenario Number The test scenarionumber in T can be discussed in two situations (a) T is aCovering Array (CA) and (b) T is a Mixed Covering Array(MCA) The difference between CA and MCA is whetherthe number of each factor value is the same ie in CA|F1|=|F2|=sdot sdot sdot =|FN|=119906 and in MCA the number of values isdifferent [27]

Firstly the simpler condition is considered ie T is aCA At this condition if each factor value is only requiredto appear at least once then 119906 test scenarios are needed Ifall combinations of values for any 119899 factors are required tobe covered the lower bound of scenario can be found bythe orthogonal table [28] And Bush proposed a constructionmethod of the orthogonal table forCAwhich has the smallest


Figure 3 Hardware-in-loop test system

number of scenarios [29] At the condition 120573 = 0 CTBCalgorithm gives priority to test cost and the number of itsgenerated scenario is no less than that of the orthogonal table[29]

119872 = 119906119899 (12)

The maximum number of scenarios is reached when 120573 =100This means that when a new test scenario is generatedit can only cover one combination in the set of uncoveredcombinations and the upper bound of scenario number canbe estimated by

119872 = (119873119899)119906119899 = 119873119906119899(119899 (119873 minus 119899)) (13)

where (119873119899 ) denotes the number of options for selecting any119899(119899 le 119873) factors from119873 onesBased on the aforementioned analysis the condition that

T is aMCA is discussed next Firstly all factors are rearrangedaccording to the number of their values from large to smalland we get Flowast1 Flowast2 sdot sdot sdot Flowast119873 satisfying |Flowast1 | ge |Flowast2 | ge ge |Flowast119873|Similar to CA the lower bound of scenario number mayhappen when 120573 = 0 and can be estimated by

119872 = 1003816100381610038161003816Flowast1 1003816100381610038161003816119899 (14)

Analogously when 120573 = 100 the maximum number ofscenarios may be reached which is calculated by

119872 = sumC119894isinC(119873119899)

prod119895isinC119894

10038161003816100381610038161003816F11989510038161003816100381610038161003816 (15)

where C(119873 119899) denotes the set of combinations for selectingany 119899(119899 le 119873) factors from 119873 factors and C119894 is the 119894-thcombination of factors in C(119873 119899)5 Application and Analysis

In this section the proposed test scenario generation methodis applied to evaluate an LDW system to validate its effective-ness by hardware-in-the-loop test as shown in Figure 3

In Figure 3 ldquoArdquo is a workstation where the virtual realitysimulation software ldquoPrescanrdquo runs ldquoBrdquo is the external driverinput device ldquoCrdquo is a host computer that can configure and

10 20 30 40 50 60 70 80 90 1000Complexity improvement index []

minus0006minus0005minus0004minus0003minus0002minus0001

000010002000300040005

Test

effec

tZ

Figure 4 Optimization process of test effect

monitor the real-time simulator ldquoDrdquo is the MicroLabBoxproduced by Dspace and acting as a real-time simulator torun the vehicle dynamical model and ldquoErdquo is the tested LDWsystem

51 Optimization of Complexity Improvement CoefficientWith this application the strength of combination coverageis selected to be 119899 = 2 to find the optimal complexityimprovement indexThe judgment matrix A = [ 1 616 1 ] is usedto determine the normalized weights of scenario complexityand test cost

S = [1198781 1198782]T = [08333 01667]T (16)

The optimization process of test effect 119885(120573) is shown inFigure 4 The best test effect is 00042 when 120573 = 4 At thiscondition M(120573) = 324 and C(120573) = 04769 A tradeoffbetween the test cost and the scenario complexity is success-fully made by the proposed approach to achieve better testeffectiveness

52 Fundamental Validation of CTBC The fundamental ofthe CTBC algorithm is that the ADAS under test is easier tomalfunction under more complex scenarios The scenariosare generated by using 119899 = 2 and 120573 = 4 which isoptimized in Section 51 The number of scenarios generatedby CTBC is 324 among which 10 scenarios whose complexindex distributes uniformly are selected to test the LDWThese scenarios are numbered from 1 to 10 in descendingorder according to their complex index The fault detectionrate under different scenarios is shown in Figure 5

As can be seen from Figure 5 scenario No 1 has a faultdetection rate of 4220 while that of scenario No 10 is only59 The fault detection rate fluctuates between 3565 and4380 as the complexity index falls from 04729 of scenarioNo 1 to 04334 of scenario No 7 And the fault detectionrate reduces dramatically from scenario No 7 to No 10 Theresults show that overall the bigger the complex index ofscenario is the easier it is to find the malfunctions of testedsystemThis fact can be used not only to guide the automatic


1 2 3 4 5 6 7 8 9 10Scenario identifier

Complexity indexFault detection rate

000

1000

2000

3000

4000

5000

Fault detection rate

0

01

02

03

04

05C

ompl

exity

inde

x

Figure 5 Relationship between complexity and fault detection rate

Methods in TMLower boundaryUpper boundary

ISO n=2 ETn=4n=3

Different Test Scenarios Generation Methods in TM

100E+00

100E+01

100E+02

100E+03

100E+04

100E+05

100E+06

100E+07

100E+08

100E+09

Num

ber o

f Tes

t Sce

nario

s

Figure 6 Number of test scenarios designed by different TMmethods

generation algorithm of test scenario but also to evaluate theeffectiveness of test scenarios generated by other algorithms

53 Performance Analysis In this section CTBC algorithmis compared with other TM methods to validate its improve-ment of test effect The number of test scenarios defined inISO 17361 designed by the OE method and generated byCTBC algorithm with the strength of combination coverage119899 = 2 3 4 is shown in Figure 6 Being similar to the optimiza-tion process in Section 51 the best complexity improvementcoefficient is selected to be 120573 = 3 when 119899 = 3 4

It can be seen clearly that the number of the test scenariodefined in ISO 17361 is too few to test the LDW adequatelyand throughly as there are only 8 scenarios Moreover thesescenarios are quite simpleThenumber of scenarios generatedby OE far exceeds others which is almost impossible toundertake because of test costs The scenarios generated by

Table 2 complexity index distribution

PICT Allpairs AETG CTBCMin 01360 00993 01094 01109Lower quartile 02115 02142 01439 04509Median 02546 02475 01935 04769Upper quartile 02997 03101 02760 04884Max 03699 04128 04190 05071

01 02 03 04 05 060Complexity index

0

10

20

30

40

50

60

70

80

Prop

ortio

n [

]

PICTAllpairs

AETGCTBC

Figure 7 Comparison of the complexity index of test scenario

CTBC are more reasonable by synthetically considering thetest effectiveness Furthermore from the previous studies it isknown that a quite small 119899 can reveal most of potential errors[26]

Moreover the bound of scenario number is the basisof the optimization process of test scenario introduced inSection 41 From the results in Figure 6 it is found that thenumber of scenarios generated by CTBC lies in the estimatedrange under different coverage strength which implies thatthe proposed estimation of bound of scenario number inSection 42 is effective

54 Scenario Complexity Index Improvement The bestadvantage of CTBC algorithm is its improvement ofscenario complexity index which is beneficial to findingfault validated in Section 52 The primary focus of thisexperiment is to set comparison with the test set generatedby other CT algorithms which does not consider thecomplexity of scenario and is the basis of CTBC Thereforechoosing such methods can highlight the superiority of theproposed algorithm Moreover PICT Allpairs and AETGhave already been widely applied in engineering applicationsTherefore they are used to validate the effectiveness of CTBCin increasing the overall complexity index of test scenariosThe comparative results are shown in Figure 7 and Table 2


Table 3 Influence factors of LDW system

Influence factor Value Importance index

Traffic environmentparameters Lightning environments

Weather

Sunny 00126Cloudy 00475Rainy 01101Foggy 01419

TimeDay (800-1700) 00101

DuskDawn (1700-1930530-800) 00403Night (1930-530) 00807

Rapid changes in lightPass through tunnel 00098Pass under footbridge 00162

None 00015

Traffic environmentparameters Lane lines parameters

Lane line clarity

Few fade and holes 00023Intermediate fade and holes 00088

Much fade and holes 0026No fade and holes 00009

Lane line integrity Lane line on one side 00038Lane line on both sides 00268

Lane line number Single 00148double 00049

Lane line color White 00007Yellow 00033

Lane line type Dashed 00276Solid 0003

Traffic environmentparameters Road parameters

Curvature

Straight road (0) 00039Bend road (1750) 00083Bend road (1500) 00186Bend road (1250) 00401

Lane number1 000432 00013 00007

Lane marks

One mark 0002Combination of two marks 00039Combination of three marks 0008Combination of four marks 00157

None 0001

SlopeUphill (Slope of +5) 00068

Downhill (Slope of -5) 00044None 00011

Roadside facilities

One facility 00062Combination of two facilities 00103Combination of three facilities 00176Combination of four facilities 00277Combination of five facilities 0052

None 00038

Subject vehicledrivers behavior Longitudinal speed

40 kmh 0001655 kmh 001860 kmh 0014880 kmh 0014890 kmh 00111100 kmh 00094120 kmh 00072140 kmh 00014


Table 3 Continued


Subject vehicledrivers behavior Lateral speed

0 ms 0001301 ms 0007104 ms 0009208 ms 0011410 ms 00102

Other trafficparticipants state Road congestion

0 lane lines missing 0001625 lane lines missing 0004350 lane lines missing 000975 lane lines missing 001590 lane lines missing 00215

53 60 61

324

PICT Allpairs AETG CTBCDifferent Test Scenarios Generation Methods in CT

1

10

100

1000

Num

ber o

f Tes

t Sce

nario

s

Figure 8 Comparison of test scenario number

It can be seen that compared with other CT algorithmsthough the maximum value of complexity index is onlyincreased slightly the number of scenarios with highercomplexity index is increased significantly which implies thatthe scenarios generated by CTBC can find more errors

The number of test scenario generated by these methodsis compared in Figure 8 The number of scenarios generatedby CTBC is increased by 61132 54 and 53115 times respec-tively It is still greatly reduced compared with that of ET (asshown in Figure 6) Besides an appropriate increase of testcost is acceptable in order to give priority to detect the hiddenerrors of ADAS systems which have respect with the drivingsecurity

6 Conclusions

Considering the high security of intelligent driving systemsthis paper proposes a new approach to automatically generatemore effective test scenarios to improve the test effectivenessThe theoretical analysis and application results show thefollowing

(1) The larger the complexity index of scenario is theeasier it is to find out the malfunctions of tested systems

(2) The proposed CTBC algorithm can generatemore complex scenarios compared with the traditional

CT method while ensuring the required combinationalcoverage

(3) Compared to the widely used ETmethod the numberof test scenarios generated by CTBC algorithm is reducedgreatly which leads to a dramatically cut down of test cost

Appendix

See Table 3

Data Availability

The tree structure model of LDW and the importanceindex of each factor are listed in the appendix The ATEGalgorithm is an online program which can be found athttpalarcostestesiuclmesCombTestWebcombinat orialjsp The Allpairs and PICT algorithms are free softwarewhich are available from the corresponding author uponrequest The program of CTBC algorithm programed byPython and its input file are available from the correspondingauthor upon request

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

This research was supported by the National Key RampDProgram of China under grants 2016YFB0101104 and2017YFB0102504 Scientific Technological Plans of Chong-qing under grant cstc2017zdcy-zdzx0042 and IndustrialBase Enhancement Project under grant 2016ZXFB06002

References

[1] R A Auckland W J Manning O M J Carsten and A HJamson ldquoAdvanced driver assistance systems Objective andsubjective performance evaluationrdquo Vehicle System Dynamicsvol 46 no 1 pp 883ndash897 2008

[2] K Bengler K Dietmayer B Farber M Maurer C Stiller andH Winner ldquoThree decades of driver assistance systems review


and future perspectivesrdquo IEEE Intelligent Transportation SystemsMagazine vol 6 no 4 pp 6ndash22 2014

[3] Q Xia J Duan F Gao T Chen and C Yang ldquoAutomaticGeneration Method of Test Scenario for ADAS Based onComplexityrdquo in Proceedings of the Intelligent and ConnectedVehicles Symposium Kunshan China 2017

[4] D LeBlanc J Sayer C Winkler et al Road departure crashwarning system field operational test methodology and resultsVolume 2 appendices University of Michigan Ann ArborTransportation Research Institute 2006

[5] V L Neale T A Dingus S G Klauer J Sudweeks andM Goodman ldquoAn overview of the 100-car naturalistic studyand findingsrdquo in Proceedings of the International TechnicalConference on the Enhanced Safety of Vehicles p 787 2005

[6] H-H Yang H Peng T J Gordon and D Leblanc ldquoDevelop-ment and validation of an errorable car-following drivermodelrdquoin Proceedings of the 2008 American Control Conference ACCpp 3927ndash3932 June 2008

[7] D Zhao X Huang H Peng H Lam and D J Leblanc ldquoAccel-erated Evaluation of Automated Vehicles in Car-FollowingManeuversrdquo IEEE Transactions on Intelligent TransportationSystems vol 19 no 3 pp 733ndash744 2018

[8] Z Huang H Lam D J LeBlanc and D Zhao ldquoAcceleratedEvaluation of Automated Vehicles Using Piecewise MixtureModelsrdquo IEEE Transactions on Intelligent Transportation Sys-tems no 99 pp 1ndash11 2017

[9] A Eidehall ldquoTracking and threat assessment for automotivecollision avoidancerdquo Behavior 2007

[10] Intelligent transport systems - adaptive cruise control systems- performance requirements and test procedures ISO Standard15622 2010

[11] European new car assessment programme - test protocol - AEBsystems NCAP Standard 2015

[12] Q Zhang D Chen Y Li and K Li ldquoResearch on PerformanceTest Method of Lane DepartureWarning Systemwith PreScanrdquoin Proceedings of the SAE-China Congress 2014 Selected PapersLecture Notes in Electrical Engineering pp 445ndash453 BeijingChina 2014

[13] B S Ahmed L M Gambardella W Afzal and K Z ZamlildquoHandling constraints in combinatorial interaction testing inthe presence of multi objective particle swarm andmultithread-ingrdquo Information and So13ware Technology vol 86 pp 20ndash362017

[14] G Seroussi and N H Bshouty ldquoVector sets for exhaustivetesting of logic circuitsrdquo Institute of Electrical and ElectronicsEngineers Transactions on Information eory vol 34 no 3 pp513ndash522 1988

[15] DMCohen S R DalalM L Fredman andG C Patton ldquoTheAETG system an approach to testing based on combinatorialdesignrdquo IEEE Transactions on So13ware Engineering vol 23 no7 pp 437ndash444 1997

[16] J Czerwonka ldquoPairwise testing in real worldrdquo in Proceedings ofthe 24th Pacific Northwest So13ware Quality Conference pp 419ndash430 Portland Ore USA 2006

[17] Satisfice Inc ldquoEpistemology for the rest of usrdquo 2008 httpwwwsatisficecomtoolsshtml

[18] H Wu C Nie F-C Kuo H Leung and C J ColbournldquoA Discrete Particle Swarm Optimization for Covering ArrayGenerationrdquo IEEE Transactions on Evolutionary Computationvol 19 no 4 pp 575ndash591 2015

[19] B J Garvin M B Cohen and M B Dwyer ldquoEvaluatingimprovements to a meta-heuristic search for constrained inter-action testingrdquo Empirical So13ware Engineering vol 16 no 1 pp61ndash102 2011

[20] L Gonzalez-Hernandez ldquoNew bounds for mixed coveringarrays in t-way testing with uniform strengthrdquo Information andSo13ware Technology vol 59 pp 17ndash32 2015

[21] R D Ona and J D Ona ldquoAnalysis of transit quality of ser-vice through segmentation and classification tree techniquesrdquoTransportmetrica A Transport Science vol 11 no 5 pp 365ndash387 2015

[22] Intelligent transport systems ndash lane departure warning systems- performance requirements and test procedures ISO Standard17361 2017

[23] R Kuhn Y Lei and R Kacker ldquoPractical combinatorial testingbeyond pairwiserdquo IT Professional vol 10 no 3 pp 19ndash23 2008

[24] R W Saaty ldquoThe analytic hierarchy process-what it is and howit is usedrdquo Applied Mathematical Modelling Simulation andComputation for Engineering and Environmental Systems vol 9no 3-5 pp 161ndash176 1987

[25] A Mantler and H Cameron ldquoConstructing red-black treeshapesrdquo International Journal of Foundations of Computer Sci-ence vol 13 no 6 pp 837ndash863 2002

[26] D Kuhn and M Reilly ldquoAn investigation of the applicabilityof design of experiments to software testingrdquo in Proceedingsof the 27th Annual NASA GoddardIEEE So13ware EngineeringWorkshop pp 91ndash95 Los Alamitos Calif USA 2003

[27] M B Cohen C J Colbourn and A C Ling ldquoConstructingstrength three covering arrays with augmented annealingrdquoDiscrete Mathematics vol 308 no 13 pp 2709ndash2722 2008

[28] A S Hedayat N J A Sloane and J Stufken ldquoOrthogonalarrays theory and applicationsrdquo Technometrics vol 42 no 4pp 440-440 2000

[29] K A Bush ldquoOrthogonal arrays of index unityrdquo Annals ofMathematical Statistics vol 23 pp 426ndash434 1952

Hindawiwwwhindawicom Volume 2018

MathematicsJournal of


Mathematical Problems in Engineering

Applied MathematicsJournal of


Probability and StatisticsHindawiwwwhindawicom Volume 2018

Journal of


Mathematical PhysicsAdvances in

Complex AnalysisJournal of


OptimizationJournal of



Engineering Mathematics

International Journal of


Operations ResearchAdvances in

Journal of


Function SpacesAbstract and Applied AnalysisHindawiwwwhindawicom Volume 2018

International Journal of Mathematics and Mathematical Sciences


Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom

The Scientific World Journal

Volume 2018

Hindawiwwwhindawicom Volume 2018Volume 2018

Numerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisAdvances inAdvances in Discrete Dynamics in

Nature and SocietyHindawiwwwhindawicom Volume 2018

Hindawiwwwhindawicom

Dierential EquationsInternational Journal of

Volume 2018


Decision SciencesAdvances in


AnalysisInternational Journal of


Stochastic AnalysisInternational Journal of

Submit your manuscripts atwwwhindawicom


driving assistance function which restricts its applicationrange In order to directly extract the most critical scenariosWCSE was proposed based on the database built up fromthe traffic accidents Andreas validated the emergency laneassist system based on these most dangerous scenes [9]These extreme scenarios help to find the faults and achievableperformance of ADAS but the coverage of all related factorscannot be measured quantitatively This makes it difficultto determine when to stop the validation and evaluationprocess

As a widely used method to construct the standardizedtest scenarios TM has also been applied in the formulation oftest standards for ADAS eg ISO 15622 for ACC [10] AEBin EuroNCAP [11] etc TM can take all factors relating to thetested system into consideration Therefore compared withother methods it can be used to design higher diversity andcomplex scenarios with an acceptable testing cost better con-trollability and preferable repeatability However the currentTM has the following problems when applied on ADAS test[12] (1) the number of test scenarios and considered factorsis too few to achieve the coverage requirement Otherwisethe mostly used orthogonal experiment method (OE) for TMleads to ldquodimensional disasterrdquo when the number of consid-ered influence factors increases (2) and its test efficiency isvery low when failures are only triggered by the interactionof a small number of factors

To solve these issues of TM we introduce the combina-torial testing method (CT) which has already been widelyused in computer software testing [13] The motivation ofCT is based on the fact that most of the program errorsare caused by the interaction of multiple influence factors[14] The set of test scenarios generated by CT can guaranteefull coverage of the required n-wise combination with asfew scenario number as possible Cohen et al proposedAutomatic Efficient Test Generator (AETG) algorithm whichgenerates a certain number of test scenarios at each timeby using the greedy algorithm and then picks one thatcan cover most of the n-wise combination of factors stilluncovered [15] Czerwonka developed another CT tool calledPairwise Independent Combinatorial Testing tool (PICT)which is unlike AETG algorithm and need not producetest scenarios to be selected beforehand [16] Instead ituses a fixed random seed to ensure the consistency of theoutput To trade off a reduction of the number of testscenarios James developed a new tool called Allpairs tofind a reasonably small set of test scenarios to satisfy thefull coverage of arbitrary pairwise combination [17] Wuet al used the particle swarm optimization algorithm togenerate test suites ensuring the coverage criteriawith smallertest set than other metaheuristic strategies [18] Garvin etal modified the original simulated annealing algorithm tofurther improve the efficiency of test suite generation [19]Gonzalez-Hernandez used the Mixed-Tabu Search to buildthe test suite with uniform strength [20]

All the aforementioned CT methods focus on the reduc-tion of test set while ensuring the combinatorial coverage atthe same timeThe complexity of scenario has not been takeninto account when generating combinatorial test scenarioswhich is beneficial to test effectiveness because a system tends

Factorlayer1

Factorlayer2

Factorlayer

layer

3

Value

Intelligent drivingsystem F11

Environment F21

Self-stateF22

Otherparticipants

F23

WeatherF31

TimeF32

Sunny31

Cloudy32

Rainy33

Foggy34

Figure 1 Tree structure model of influence factors

to malfunction under more complex conditions Therefore anew method called combinatorial testing based on scenariocomplexity (CTBC) is presented in this paper for automaticgenerating the test scenarios for intelligent driving systemwhich can achieve full coverage of n-wise combinationwith a relative compact test set meanwhile increasing thescenario complexity The following paper is organized asfollows Section 2 describes the model of influence factorsand potential problems Section 3 introduces the procedureand principle of the proposed CTBC whose performance isanalyzed theoretically in Section 4 Section 5 validates CTBCby application and comparative analysis and finally Section 6concludes the paper

2 Problem Description

Before discussing the problem of automatic generation oftest scenario for intelligent driving systems a mathematicalmodel is introduced to describe the influence factors ofintelligent driving system which act as the input of theautomatic generation process of test scenarios

21 Tree Structure Model of Influence Factors The compo-nents used to construct the test scenario are called influencefactors in this paper which are related to the functionalityof intelligent driving system These factors can be obtainedthrough a variety ways such as technical specificationsnaturalistic trafficdata etc To ensure the comprehensivenesssuch systematic methods eg classification tree [21] aresuggested to analyze the possible influence factors In generalthe influence factors of an intelligent driving system can bedivided into three types environment self-state and otherroad participants

To realize automatic generation of test scenarios theinfluence factors should be discretized to get distinct valueswhich can be generated by such black box test scenariosdesign methods as equivalence class partition and boundaryvalue processing [12] Finally a tree structure model ofinfluence factors can be derived as shown in Figure 1 asan example Table 3 in the Appendix shows the detailedinfluence factors of LDW


Factorsin

end

Corres-ponding

value

Weather F1

Time F2


F3

Sunny 1p1

Day(800-1700)

2p2


3p3


Onetest scenario




(1)



F119894119895 = V119894119896 119896 isin Ω119894119895 (119894 119895) isin Ω119890119899119889 (2)



T = [119905119894119895] isin R119872times119873








A119894119895 = [119886ℎ119891] isin R|Ω119894119895|times|Ω119894119895 |



119886ℎ119891 =

1 ℎ = 1198911119886119891ℎ ℎ = 119891 (5)





119894119896 = prod(119886119887)isinΩ119894119896

120596119886119887 119896 isin Ω119894119895 (119894 119895) isin Ω119890119899119889 (7)





C119894 =119873sum119895=1

119894119895 (8)












120573 = 119905ℎ119903119890119904ℎ119900119897119889 minus C



C (120573) = sum119872(120573)119894=1 C119894 (120573)119872 (120573) (10)




119902119901119902isincomb119902119901119902


8 119905ℎ119903119890119904ℎ119900119897119889 = 120573 119873sum119902=1





max120573


C minus C


(11)








119872 = 119906119899 (12)


119872 = (119873119899)119906119899 = 119873119906119899(119899 (119873 minus 119899)) (13)



119872 = 1003816100381610038161003816Flowast1 1003816100381610038161003816119899 (14)


119872 = sumC119894isinC(119873119899)


10038161003816100381610038161003816F11989510038161003816100381610038161003816 (15)






000010002000300040005

Test

effec

tZ




S = [1198781 1198782]T = [08333 01667]T (16)







000

1000

2000

3000

4000

5000


0

01

02

03

04

05C

ompl

exity

inde

x



ISO n=2 ETn=4n=3


100E+00

100E+01

100E+02

100E+03

100E+04

100E+05

100E+06

100E+07

100E+08

100E+09

Num

ber o

f Tes

t Sce

nario

s








0

10

20

30

40

50

60

70

80

Prop

ortio

n [

]

PICTAllpairs

AETGCTBC









Weather


TimeDay (800-1700) 00101

DuskDawn (1700-1930530-800) 00403Night (1930-530) 00807


None 00015


Lane line clarity








Curvature


Lane number1 000432 00013 00007

Lane marks


None 0001



Roadside facilities


None 00038




Table 3 Continued



0 ms 0001301 ms 0007104 ms 0009208 ms 0011410 ms 00102



53 60 61

324


1

10

100

1000

Num

ber o

f Tes

t Sce

nario

s




6 Conclusions






Appendix

See Table 3

Data Availability




Acknowledgments


References







































Journal of












Journal of







Volume 2018






Volume 2018









Factorsin

end

Corres-ponding

value

Weather F1

Time F2


F3

Sunny 1p1

Day(800-1700)

2p2


3p3


Onetest scenario




(1)



F119894119895 = V119894119896 119896 isin Ω119894119895 (119894 119895) isin Ω119890119899119889 (2)



T = [119905119894119895] isin R119872times119873








A119894119895 = [119886ℎ119891] isin R|Ω119894119895|times|Ω119894119895 |



119886ℎ119891 =

1 ℎ = 1198911119886119891ℎ ℎ = 119891 (5)





119894119896 = prod(119886119887)isinΩ119894119896

120596119886119887 119896 isin Ω119894119895 (119894 119895) isin Ω119890119899119889 (7)





C119894 =119873sum119895=1

119894119895 (8)












120573 = 119905ℎ119903119890119904ℎ119900119897119889 minus C



C (120573) = sum119872(120573)119894=1 C119894 (120573)119872 (120573) (10)




119902119901119902isincomb119902119901119902


8 119905ℎ119903119890119904ℎ119900119897119889 = 120573 119873sum119902=1





max120573


C minus C


(11)








119872 = 119906119899 (12)


119872 = (119873119899)119906119899 = 119873119906119899(119899 (119873 minus 119899)) (13)



119872 = 1003816100381610038161003816Flowast1 1003816100381610038161003816119899 (14)


119872 = sumC119894isinC(119873119899)


10038161003816100381610038161003816F11989510038161003816100381610038161003816 (15)






000010002000300040005

Test

effec

tZ




S = [1198781 1198782]T = [08333 01667]T (16)







000

1000

2000

3000

4000

5000


0

01

02

03

04

05C

ompl

exity

inde

x



ISO n=2 ETn=4n=3


100E+00

100E+01

100E+02

100E+03

100E+04

100E+05

100E+06

100E+07

100E+08

100E+09

Num

ber o

f Tes

t Sce

nario

s








0

10

20

30

40

50

60

70

80

Prop

ortio

n [

]

PICTAllpairs

AETGCTBC









Weather


TimeDay (800-1700) 00101

DuskDawn (1700-1930530-800) 00403Night (1930-530) 00807


None 00015


Lane line clarity








Curvature


Lane number1 000432 00013 00007

Lane marks


None 0001



Roadside facilities


None 00038




Table 3 Continued



0 ms 0001301 ms 0007104 ms 0009208 ms 0011410 ms 00102



53 60 61

324


1

10

100

1000

Num

ber o

f Tes

t Sce

nario

s




6 Conclusions






Appendix

See Table 3

Data Availability




Acknowledgments


References







































Journal of












Journal of







Volume 2018






Volume 2018












C119894 =119873sum119895=1

119894119895 (8)












120573 = 119905ℎ119903119890119904ℎ119900119897119889 minus C



C (120573) = sum119872(120573)119894=1 C119894 (120573)119872 (120573) (10)




119902119901119902isincomb119902119901119902


8 119905ℎ119903119890119904ℎ119900119897119889 = 120573 119873sum119902=1





max120573


C minus C


(11)








119872 = 119906119899 (12)


119872 = (119873119899)119906119899 = 119873119906119899(119899 (119873 minus 119899)) (13)



119872 = 1003816100381610038161003816Flowast1 1003816100381610038161003816119899 (14)


119872 = sumC119894isinC(119873119899)


10038161003816100381610038161003816F11989510038161003816100381610038161003816 (15)






000010002000300040005

Test

effec

tZ




S = [1198781 1198782]T = [08333 01667]T (16)







000

1000

2000

3000

4000

5000


0

01

02

03

04

05C

ompl

exity

inde

x



ISO n=2 ETn=4n=3


100E+00

100E+01

100E+02

100E+03

100E+04

100E+05

100E+06

100E+07

100E+08

100E+09

Num

ber o

f Tes

t Sce

nario

s








0

10

20

30

40

50

60

70

80

Prop

ortio

n [

]

PICTAllpairs

AETGCTBC









Weather


TimeDay (800-1700) 00101

DuskDawn (1700-1930530-800) 00403Night (1930-530) 00807


None 00015


Lane line clarity








Curvature


Lane number1 000432 00013 00007

Lane marks


None 0001



Roadside facilities


None 00038




Table 3 Continued



0 ms 0001301 ms 0007104 ms 0009208 ms 0011410 ms 00102



53 60 61

324


1

10

100

1000

Num

ber o

f Tes

t Sce

nario

s




6 Conclusions






Appendix

See Table 3

Data Availability




Acknowledgments


References







































Journal of












Journal of







Volume 2018






Volume 2018










119902119901119902isincomb119902119901119902


8 119905ℎ119903119890119904ℎ119900119897119889 = 120573 119873sum119902=1





max120573


C minus C


(11)








119872 = 119906119899 (12)


119872 = (119873119899)119906119899 = 119873119906119899(119899 (119873 minus 119899)) (13)



119872 = 1003816100381610038161003816Flowast1 1003816100381610038161003816119899 (14)


119872 = sumC119894isinC(119873119899)


10038161003816100381610038161003816F11989510038161003816100381610038161003816 (15)






000010002000300040005

Test

effec

tZ




S = [1198781 1198782]T = [08333 01667]T (16)







000

1000

2000

3000

4000

5000


0

01

02

03

04

05C

ompl

exity

inde

x



ISO n=2 ETn=4n=3


100E+00

100E+01

100E+02

100E+03

100E+04

100E+05

100E+06

100E+07

100E+08

100E+09

Num

ber o

f Tes

t Sce

nario

s








0

10

20

30

40

50

60

70

80

Prop

ortio

n [

]

PICTAllpairs

AETGCTBC









Weather


TimeDay (800-1700) 00101

DuskDawn (1700-1930530-800) 00403Night (1930-530) 00807


None 00015


Lane line clarity








Curvature


Lane number1 000432 00013 00007

Lane marks


None 0001



Roadside facilities


None 00038




Table 3 Continued



0 ms 0001301 ms 0007104 ms 0009208 ms 0011410 ms 00102



53 60 61

324


1

10

100

1000

Num

ber o

f Tes

t Sce

nario

s




6 Conclusions






Appendix

See Table 3

Data Availability




Acknowledgments


References







































Journal of












Journal of







Volume 2018






Volume 2018











119872 = 119906119899 (12)


119872 = (119873119899)119906119899 = 119873119906119899(119899 (119873 minus 119899)) (13)



119872 = 1003816100381610038161003816Flowast1 1003816100381610038161003816119899 (14)


119872 = sumC119894isinC(119873119899)


10038161003816100381610038161003816F11989510038161003816100381610038161003816 (15)






000010002000300040005

Test

effec

tZ




S = [1198781 1198782]T = [08333 01667]T (16)







000

1000

2000

3000

4000

5000


0

01

02

03

04

05C

ompl

exity

inde

x



ISO n=2 ETn=4n=3


100E+00

100E+01

100E+02

100E+03

100E+04

100E+05

100E+06

100E+07

100E+08

100E+09

Num

ber o

f Tes

t Sce

nario

s








0

10

20

30

40

50

60

70

80

Prop

ortio

n [

]

PICTAllpairs

AETGCTBC









Weather


TimeDay (800-1700) 00101

DuskDawn (1700-1930530-800) 00403Night (1930-530) 00807


None 00015


Lane line clarity








Curvature


Lane number1 000432 00013 00007

Lane marks


None 0001



Roadside facilities


None 00038




Table 3 Continued



0 ms 0001301 ms 0007104 ms 0009208 ms 0011410 ms 00102



53 60 61

324


1

10

100

1000

Num

ber o

f Tes

t Sce

nario

s




6 Conclusions






Appendix

See Table 3

Data Availability




Acknowledgments


References







































Journal of












Journal of







Volume 2018






Volume 2018











000

1000

2000

3000

4000

5000


0

01

02

03

04

05C

ompl

exity

inde

x



ISO n=2 ETn=4n=3


100E+00

100E+01

100E+02

100E+03

100E+04

100E+05

100E+06

100E+07

100E+08

100E+09

Num

ber o

f Tes

t Sce

nario

s








0

10

20

30

40

50

60

70

80

Prop

ortio

n [

]

PICTAllpairs

AETGCTBC









Weather


TimeDay (800-1700) 00101

DuskDawn (1700-1930530-800) 00403Night (1930-530) 00807


None 00015


Lane line clarity








Curvature


Lane number1 000432 00013 00007

Lane marks


None 0001



Roadside facilities


None 00038




Table 3 Continued



0 ms 0001301 ms 0007104 ms 0009208 ms 0011410 ms 00102



53 60 61

324


1

10

100

1000

Num

ber o

f Tes

t Sce

nario

s




6 Conclusions






Appendix

See Table 3

Data Availability




Acknowledgments


References







































Journal of












Journal of







Volume 2018






Volume 2018












Weather


TimeDay (800-1700) 00101

DuskDawn (1700-1930530-800) 00403Night (1930-530) 00807


None 00015


Lane line clarity








Curvature


Lane number1 000432 00013 00007

Lane marks


None 0001



Roadside facilities


None 00038




Table 3 Continued



0 ms 0001301 ms 0007104 ms 0009208 ms 0011410 ms 00102



53 60 61

324


1

10

100

1000

Num

ber o

f Tes

t Sce

nario

s




6 Conclusions






Appendix

See Table 3

Data Availability




Acknowledgments


References







































Journal of












Journal of







Volume 2018






Volume 2018









Table 3 Continued



0 ms 0001301 ms 0007104 ms 0009208 ms 0011410 ms 00102



53 60 61

324


1

10

100

1000

Num

ber o

f Tes

t Sce

nario

s




6 Conclusions






Appendix

See Table 3

Data Availability




Acknowledgments


References







































Journal of












Journal of







Volume 2018






Volume 2018












































Journal of












Journal of







Volume 2018






Volume 2018








a test scenario automatic generation strategy for intelligent … · 2018. 6. 27. · intelligent...

Documents