error control for probabilistic model checking
DESCRIPTION
Error Control for Probabilistic Model Checking. H åkan L. S. Younes Carnegie Mellon University. Contributions. Framework for expressing correctness guarantees of model-checking algorithms Enables comparison of different algorithms Improves understanding of sampling-based algorithms - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/1.jpg)
Error Control forProbabilistic Model Checking
Håkan L. S. YounesCarnegie Mellon University
![Page 2: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/2.jpg)
Younes Error Control for Probabilistic Model Checing 2
Contributions
Framework for expressing correctness guarantees of model-checking algorithms
Enables comparison of different algorithms Improves understanding of sampling-based
algorithms New sampling-based algorithm for
probabilistic model checking Better error control through undecided results
![Page 3: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/3.jpg)
Younes Error Control for Probabilistic Model Checing 3
Probabilistic Model Checking
Given a model , a state s, and a property , does hold in s for ?
Model: stochastic discrete event system Property: probabilistic temporal logic formula
q
arrival departure
“The probability is at least 0.1 that the queuebecomes full within 5 minutes”
![Page 4: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/4.jpg)
Younes Error Control for Probabilistic Model Checing 4
Temporal Stochastic Logic (CSL)
Standard logic operators: , , …
Probabilistic operator: ≥ [ ] Holds in state s iff probability is at least for
paths satisfying and starting in s Until: ≤T
Holds over path iff becomes true along within time T, and is true until then
![Page 5: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/5.jpg)
Younes Error Control for Probabilistic Model Checing 5
Property Example
“The probability is at least 0.1 that the queue becomes full within 5 minutes”
≥0.1[ ≤5 full ]
![Page 6: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/6.jpg)
Younes Error Control for Probabilistic Model Checing 6
Possible Results ofModel Checking
Given a state s and a formula , a model-checking algorithm can:
Accept as true in s (s ) Reject as false in s (s ) Return an undecided result (s I )
An error occurs if: rejects when is true (false negative) accepts when is false (false positive)
![Page 7: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/7.jpg)
Younes Error Control for Probabilistic Model Checing 7
Ideal Error Control
Bound on false negatives: Pr[s | s ]
Bound on false positives: Pr[s | s ]
Bound on undecided results: Pr[s I ]
![Page 8: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/8.jpg)
Younes Error Control for Probabilistic Model Checing 8
Unrealistic Expectations
Actual probability of holding
s ≥ [ ]s ≥ [ ]
p
Pro
babi
lity
of a
ccep
ting
≥ [
] as
tru
e in
s 1 – –
![Page 9: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/9.jpg)
Younes Error Control for Probabilistic Model Checing 9
Temporal Stochastic Logic with Indifference Regions (CSL)
Indifference region of width 2 centered around probability thresholds
Probabilistic operator: ≥ [ ] Holds in state s if probability is at least +
for paths satisfying and starting in s Does not hold if probability is at most − “Too close to call” if probability is within
distance of
![Page 10: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/10.jpg)
Younes Error Control for Probabilistic Model Checing 10
Error Control forCurrent Solution Methods
Bound on false negatives: Pr[s | s ]
Bound on false positives: Pr[s | s ]
No undecided results: = 0 Pr[s I ] = 0
![Page 11: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/11.jpg)
Younes Error Control for Probabilistic Model Checing 11
Probabilistic Model Checkingwith Indifference Regions
Actual probability of holding + − p
Pro
babi
lity
of a
ccep
ting
≥ [
] as
tru
e in
s 1 –
s ≥ [ ]
s ≥ [ ]s ≥ [ ]
s ≥ [ ]
![Page 12: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/12.jpg)
Younes Error Control for Probabilistic Model Checing 12
Hypothesis TestingYounes & Simmons (CAV’02)
Single sampling plan: n, c Generate n sample execution paths Accept ≥ [ ] iff more than c paths satisfy Probability of accepting ≥ [ ] as true:
Sequential acceptance sampling
c
i
ini ppi
npncF
0
11),;(1
![Page 13: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/13.jpg)
Younes Error Control for Probabilistic Model Checing 13
Statistical EstimationHérault et al. (VMCAI’04)
Estimate p using sample of size n: Choosing n:
Acceptance condition for ≥ [ ]:
n
iix
np
1
1~
1~Pr
2log
21
2 ppn
p~
Same as single sampling plan n, n + 1!
![Page 14: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/14.jpg)
Younes Error Control for Probabilistic Model Checing 14
Statistical Estimation vs.Hypothesis Testing
nest nopt nest ∕ nopt
0.5 10−2 10−2 26,492 13,527 1.96
0.5 10−8 10−2 95,570 39,379 2.43
0.5 10−8 10−8 95,570 78,725 1.21
0.9 10−2 10−2 26,492 4,861 5.45
0.9 10−8 10−2 95,570 13,982 6.84
0.9 10−8 10−8 95,570 28,280 3.38
![Page 15: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/15.jpg)
Younes Error Control for Probabilistic Model Checing 15
Numerical Transient AnalysisBaier et al. (CAV’00)
Estimate p with truncation error :
Acceptance condition for ≥ [ ]: Pr[s | s ] = 0
Pr[s | s ] = 0
ppp ~~
2
~p
2
![Page 16: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/16.jpg)
Younes Error Control for Probabilistic Model Checing 16
Alternative Error Control
Bound on false negatives: Pr[s | s ]
Bound on false positives: Pr[s | s ]
Bound on undecided results: Pr[s I | (s ) (s )]
![Page 17: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/17.jpg)
Younes Error Control for Probabilistic Model Checing 17
Probabilistic Model Checkingwith Undecided Results
Actual probability of holding + − p
1 –
Rejection probability
Acceptance probability
Undecided result withprobability at least 1 – –
![Page 18: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/18.jpg)
Younes Error Control for Probabilistic Model Checing 18
Statistical Solution Method
Simultaneous acceptance sampling plans H0: p against H1: p – H0: p + against H1: p
Combining the results Accept ≥ [ ] if H0 and H0 are accepted
Reject ≥ [ ] if H1 and H1 are accepted Undecided result otherwise
![Page 19: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/19.jpg)
Younes Error Control for Probabilistic Model Checing 19
20
15
10
5
014 14.1 14.2 14.3 14.4 14.5
= 0 = 10–2
Formula time bound (T )
Ve
rific
atio
n ti
me
(se
con
ds)
Empirical Evaluation(Symmetric Polling System)
serv1 ≥0.5[ ≤T poll1]
![Page 20: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/20.jpg)
Younes Error Control for Probabilistic Model Checing 20
Empirical Evaluation(Symmetric Polling System)
result 14.10 14.15 14.20 14.25 14.30 14.35 14.40
accept 0 3 9 50 88 97 100
reject 100 97 91 50 12 3 0
accept 0 0 0 0 32 99 100
reject 100 99 42 1 0 0 0
undecided 0 1 58 99 68 1 0
= = = 10–2
![Page 21: Error Control for Probabilistic Model Checking](https://reader036.vdocuments.net/reader036/viewer/2022081508/56813ded550346895da7c6c7/html5/thumbnails/21.jpg)
Younes Error Control for Probabilistic Model Checing 21
Summary
Statistical estimation is never more efficient than hypothesis testing
Statistical methods are randomized algorithms for CSL model checking
Numerical methods are exact algorithms for CSL model checking
New statistical solution method with finer error control ( parameter)