mil testing of highly configurable continuous controllers
DESCRIPTION
TRANSCRIPT
![Page 1: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/1.jpg)
Reza Matinnejad Shiva Nejati Lionel Briand Software Verification and Validation Group SnT Center, University of Luxembourg Thomas Bruckmann Delphi Automotive Systems, Luxembourg
MiL Testing of Highly Configurable Continuous Controllers:
Scalable Search Using Surrogate Models
![Page 2: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/2.jpg)
Closed-loop Continuous Controllers An Example: Conveyor Belt Controller
Controller
1
![Page 3: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/3.jpg)
+ -‐
2
Closed-loop Controller
Closed-loop Continuous Controllers An Example: Conveyor Belt Controller
![Page 4: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/4.jpg)
Closed-loop Continuous Controllers
3
![Page 5: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/5.jpg)
Desired Value
Controller Model
System Output
Model-based Development of Embedded Software
Plant Model
+ -
Model-in-the-Loop Stage (MiL)
Hardware-in-the-Loop Stage (HiL)
4
Software-in-the-Loop Stage (SiL)
![Page 6: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/6.jpg)
Configuration 1
Configuration 2 Configuration 3
Controller Model
Plant Model
5
MiL Testing Continuous Controllers (1) Instantiating the Controller Model
• Step1: Identifying an assignment of values to the configuration parameters of the controller model
![Page 7: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/7.jpg)
Des
ired/
Act
ual V
alue
Des
ired/
Act
ual V
alue
Time Time
MiL Testing Continuous Controllers (2) Running Model Simulation
6
Test Input 1
• Step2: Running simulations of the controller model and examining the output signals
Test Input 2 / Output 1 / Output 2
![Page 8: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/8.jpg)
Controller Requirements (1)
Time Time
Act
ual V
alue
• Stability: The actual value shall stabilize at the desired value
Act
ual V
alue
7
Stability Violation
Stability Violation
![Page 9: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/9.jpg)
Controller Requirements (2)
• Smoothness: The actual value shall not abruptly change when it is close to the desired value
Time
Act
ual V
alue
8
Smoothness Violation
![Page 10: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/10.jpg)
Controller Requirements (2)
9
Time
Act
ual V
alue
Responsiveness Violation
• Responsiveness: The controller shall respond within a certain time-limit
![Page 11: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/11.jpg)
Stability Objective Function: Fst
• Fst (Test Case A) > Fst (Test Case B) • We look for test input that maximizes Fst
Test Case A Test Case B
Act
ual V
alue
Act
ual V
alue
Time Time
10
![Page 12: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/12.jpg)
Smoothness Objective Function: Fsm
• Fsm (Test Case A) > Fsm (Test Case B) • We look for test input that maximizes Fsm
Act
ual V
alue
Time Time
Test Case A Test Case B
11
Act
ual V
alue
![Page 13: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/13.jpg)
Responsiveness Objective Function: Fr
• Fr (Test Case A) > Fr (Test Case B) • We look for test input that maximizes Fr
Act
ual V
alue
Time Time
Test Case A Test Case B
Act
ual V
alue
12
![Page 14: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/14.jpg)
Our Approach: Search-based Testing of Continuous Controllers
13
• Step 1 : Exploration
• Step 2 : Search
Controller-Plant Model
Controller BehaviorOver the
Input Space
1.Exploration
Plant ⌃
Controller
Worst-Case Scenarios
CriticalOperatingRegions of
the Controller
2.Search
![Page 15: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/15.jpg)
FD
ID FD
ID
Our Earlier Work: Search-based Testing of Continuous Controllers
with fixed values for the configuration parameters
Initial Desired (ID)
Fina
l Des
ired
(FD
)
Smoothness HeatMap Smoothness Worst-case Scenario
Controller-plant model
HeatMap Diagram
Worst-Case Scenarios
List of Critical Regions
DomainExpert
1.Exploration 2.Single-statesearch
Controller Input Space
14
Fst Fsm Fr
Published in [IST Journal 2014, SSBSE 2013]
![Page 16: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/16.jpg)
Controller-plant model
HeatMap Diagram
Worst-Case Scenarios
List of Critical Regions
DomainExpert
1.Exploration 2.Single-statesearch
Challenges When Searching the Entire Configuration Space
1. The input space of the controller is larger • 6 configuration parameters in our case study
4. Search becomes slower • It takes 30 sec to run each model Simulation
2. Not all the configuration parameters matter for all the objective functions
3. Harder to visualize the results
15
![Page 17: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/17.jpg)
Our approach for Testing the Controller in the Entire Configuration Space
Dimensionality ReducLon to focus the exploraLon on the variables with significant impact on each objecLve funcLon
VisualizaLon of the 8-‐dimension space using Regression Trees
Surrogate Modeling to avoid running simulaLons for parts of the input space where the surrogate model is the most accurate.
16
Controller-plant model
+Configuration
parametersranges
Regression Trees
Worst-Case Scenarios
List of Critical Regions
DomainExpert
1.Exploration with Dimensionality
Reduction
x<0.2
y<0.3
z<0.1
2. Search with Surrogate Modeling
![Page 18: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/18.jpg)
Exploration with Dimensionality Reduction Elementary Effects Analysis Method
• Goal: IdenLfying variables with a significant impact on each objecLve funcLon
17
Exploringall the
dimensions
Exploring the significant
dimensions
ProportionalGainDerivativeGain
IntegralGainIntgrResetErrLim
...
SmoothnessProportionalGainIntgrResetErrLim
...
ElementaryEffects
AnalysisMethod
...
![Page 19: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/19.jpg)
Regression Tree
• Goal: Dividing the input space into homogenous parLLons with respect to the objecLve funcLon values
Smoothness Regression Tree
18
All Points
ID < 0.7757
Count MeanStd Dev
Count MeanStd Dev
ID >= 0.7757Count MeanStd Dev
FD >= 0.5674 FD < 0.5674Count MeanStd Dev
Count MeanStd Dev
1000 0.007
0.0049
574 0.0059
0.004
426 0.01034250.0049919
182 0.0134
0.005
244 0.0080.003
FD < 0.1254 FD >= 0.1254Count MeanStd Dev
Count MeanStd Dev
182 0.01345550.0052883
244 0.00802060.0031751
Smoothness HeatMap
FD<0.5674
FD>=0.1254
![Page 20: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/20.jpg)
Smoothness Critical Partition
Search With Surrogate Modeling Supervised Machine Learning
• Goal: To predict the values of the objecLve funcLons within a criLcal parLLon and speed up the search
19
• Surrogate model predicts Fsm with a given confidence level
A
B • During the search, surrogate model predicts Fsm for the next point: 1. Not confident about Fsm(B) and
Fsm(A) relation • Run the simulation as before
2. Confident that Fsm(B) > Fsm(A) • Run the simulation as before
3. Confident that Fsm(B) < Fsm(A) • Avoid running the Simulation
![Page 21: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/21.jpg)
Search With Surrogate Modeling Supervised Machine Learning
• Different Machine Learning techniques: 1. Linear Regression 2. ExponenLal Regression 3. Polynomial Regression (n=2) 4. Polynomial Regression (n=3)
• Criteria to compare different techniques: 1. R2 ranges from 0 to 1
• R2 shows goodness of fit 2. Mean RelaLve PredicLon Error (MRPE)
• MRPE shows predicLon accuracy
20
![Page 22: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/22.jpg)
Evaluation: Research Questions
• RQ1 (Which ML Technique): Which Machine Learning technique performed the best?
21
• RQ2 (Effect of DR): What was the effect of Dimensionality Reduction?
• RQ3 (SM vs. No-SM): How did the search with Surrogate Modeling perform comparing to the search without Surrogate Modeling?
• RQ4 (Worst-Case Scenarios): How did our approach perform in finding worst-‐case test scenarios?
![Page 23: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/23.jpg)
RQ1: Which Machine Learning Technique?
22
• The best technique to build surrogate models for all our three objecLve funcLons is Polynomial Regression (n=3) • PR (n=3) has the highest R2 ( goodness of fit ) • PR (n=3) has the smallest MRPE ( predicLon error )
• The surrogate models are accurate and predicLve enough for Smoothness and Responsiveness (R2 was close to 1, MRPE was close to 0) • In future, we want to try other Supervised Leaning
techniques, such as SVM
![Page 24: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/24.jpg)
RQ2: Effect of Dimensionality Reduction
23
• Dimensionality reducLon helps generate beber surrogate models for Smoothness and Responsiveness • Smaller MRPE: More predicLve surrogate model
Smoothness Responsiveness 0.03
0.02
0.01 No DR DR
MR
PE
0.02
0.03
0.04
MR
PE
Mean Relative Prediction Error (MRPE)
No DR DR
![Page 25: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/25.jpg)
RQ3: Search with vs. without Surrogate Modeling
24
• For responsiveness, the search with SM was 8 Lmes faster
0.16
0.165
0.17 H
ighe
st v
alue
of
Fr
After 300 Sec After 2500 Sec
• For smoothness, the search with SM was much more effecLve
Hig
hest
val
ue
of F
sm
After 800 Sec After 2500 Sec
SM No SM
0.22
0.225
0.23
SM No SM 0.22
0.225
0.23
SM
No SM
SM
0.16
0.165
0.17
SM No SM
No SM
![Page 26: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/26.jpg)
RQ4: Worst-Case Scenarios (1)
25
• Our approach is able to idenLfy criLcal violaLons of the controller requirements that had neither been found by our earlier work nor by manual tesLng
MiL Testing Varied Configurations
(Current work)
MiL Testing Fixed Configurations
(Earlier Work)
ManualMiL Testing
(Industry Practice)
Stability
Smoothness
Responsiveness
2.2% deviation - -24% over/undershoot
170 ms response time
20% over/undershoot
80 ms response time
5% over/undershoot
50 ms response time
![Page 27: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/27.jpg)
RQ4: Worst-Case Scenarios (2)
26
• For example, for the industrial controller we idenLfied the following violaLon of the Stability requirement within the given ranges for the calibraLon variables
ID = 0.36
FD = 0.22
Flap
Pos
ition
Time
2.2 % Deviation
![Page 28: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/28.jpg)
27
Conclusion
![Page 29: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/29.jpg)
28
Reza Matinnejad PhD Candidate Software Verification and Validation Group SnT Center, University of Luxembourg Emails: [email protected] [email protected]
MiL Testing of Highly Configurable Continuous Controllers:
Scalable Search Using Surrogate Models
![Page 30: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/30.jpg)
Alternative Heatmap
Stability Responsiveness
ü Our approach was implemented at Technical University of Munich too
![Page 31: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/31.jpg)
ExisLng Tool Support for TesLng Simulink/Stateflow Models
• None of the exis,ng tools specifically test the con,nuous proper,es of Simulink output signals
![Page 32: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/32.jpg)
Related Work
Technique Examples Limitations and Differences
Mixed discrete-continuous modeling techniques
• Timed Automata • Hybrid Automata • Stateflow
• Require model translation • Scalability issues • Verification of logical and
state-based behavior
Search-based testing techniques for Simulink models
• Path coverage or mutation testing
• Worst-case execution time
• Mainly focus on test data
generation • Only applied to discrete-event
embedded systems
Control Engineering • Ziegler-Nichols Tuning Rules
• Design/configuration
optimization • Signal analysis and generation
Commercial tools in automotive industry
• MATLAB/Simulink Design Verifier
• Reactis tool
• Combinatorial and Logical
Properties • Lack of documentation
31
![Page 33: MiL Testing of Highly Configurable Continuous Controllers](https://reader034.vdocuments.net/reader034/viewer/2022051610/5492f20bb479595b4d8b4722/html5/thumbnails/33.jpg)
CoCoTest Results
Stability
Violation
32