how to prevent disasters - people | mit csailpeople.csail.mit.edu › dnj › talks › veldhoven10...
TRANSCRIPT
![Page 1: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/1.jpg)
how topreventdisastersDaniel Jackson, MITSiren//NL, Veldhoven · November 2, 2010
![Page 2: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/2.jpg)
a civil engineering disaster
![Page 3: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/3.jpg)
kansas city hyatt regency, 1981
New York Times
![Page 4: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/4.jpg)
the design beam supportsone walkway
illustrations from Matthys Levy and Mario Salvadori, Why Buildings Fall Down
![Page 5: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/5.jpg)
how it failed
as designed as implemented what happened
beam supportstwo walkways
![Page 6: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/6.jpg)
therac 25no argument for success› AECL fault tree (1983) did not include software› P(computer selects wrong energy) = 10-11
hard to extract any lessons› Leveson & Turner (1993): so many flaws, nothing clear
so doomed to fail again› 17 deaths from similar machine in Panama (2001)› 621 target/dose/patient errors (2001-9, NY state)
[2001-2009, New York Times, January 22, 2010]
![Page 7: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/7.jpg)
my conclusions
civil engineers› argue why structure should stand› failure occurs when argument is flawed
software engineers› build and hope for the best› when failure occurs, no story› can’t assign blame or learn for future
![Page 8: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/8.jpg)
![Page 9: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/9.jpg)
a new approach
write downcritical properties R
write downdomain
assumptions A
design a specification S
check thatA∧S⇒R
build machine Mcheck that M⇒S
yes
no
reduce goal
rely on more
fix design
DEPENDABILITY CASE:claimed properties
assumptionsdesign & specs
correctness argumentwrite down
domain assumptions A
write downcritical properties R
design a specification S
![Page 10: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/10.jpg)
the door interlock problem
![Page 11: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/11.jpg)
a textbook problem› see, eg, Engineering a Safer World [Leveson, 2010]
problem: design an interlock
![Page 12: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/12.jpg)
actually, a real problemThe Worlds First Microwave Test Oven
Here's a picture of the world's first commercial microwave during its first field test. I am on the left, my brother on the right. We used to defeat the door interlock and point it at the end of the countertop where we left a plate of eggs. They exploded like little hand grenades. Drove my mom nuts!
http://www.thescubalady.com/Keith%20Lamb%20History.htm
Statistics indicate that five to ten arc-flash accidents that involve a fatality or serious injury to an employee occur every day in the United States.
http://www.iaei.org/magazine/?p=1163
![Page 13: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/13.jpg)
step 1: requirement
SafeSafeOperators
PowerSource
touch
Safe: touch event does not occur in state Live
no touching live power source
![Page 14: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/14.jpg)
SafeOperators
PowerSource
touch
SafeOperators
Door PowerSource
open,close
touch
Exposed
Safe: touch event does not occur in state Live
step 2: domain assumptions
SafeOperators
Door PowerSource
open,close
touch
Sensor Switch
open,close Live
Exposedwhen close occurs,Exposed becomes false
no touch unless Exposed is true
when open occursClosed becomes false when off occurs, Live becomes false
![Page 15: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/15.jpg)
step 3: machine specification
Controller
SafeOperators
Door PowerSource
open,close
touch
on, off
Sensor Switch
Closed
open,close Live
Exposed
SafeOperators
Door PowerSource
open,close
touch
Sensor Switch
open,close Live
Exposed
Safe: touch event does not occur in state Live
when close occurs,Exposed becomes false
no touch unless Exposed is true
when open occursClosed becomes false when off occurs, Live becomes false
every step, send off if Closed became false
send on only when Closed is true
![Page 16: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/16.jpg)
step 4: checking the system argumentmachine specdomain assumptions ∧ ⇒ requirement
one sig Sensor extends Domain { Closed: set Time }
one sig PowerSource extends Domain { Exposed, Live: set Time }
sig Open extends Event { } { not Sensor.Closed.after }
one sig Controller extends Domain { } { all t: Time - (first + last) | not Sensor.Closed.at [t] and Sensor.Closed.at [t.prev] implies Off.happensAt [t] } one sig Safe extends Requirement {} {
all t: Touch | not PowerSource.Live.before [t] }
![Page 17: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/17.jpg)
counterexample!problem:forgot initial conditions
solution:record them
one sig PowerSource extends Domain { Exposed, Live: set Time } { not Live.initially not Exposed.initially }
![Page 18: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/18.jpg)
counterexample again!problem:controller turns power off too late
solution:new domainassumption
sig Touch extends Event { } { PowerSource.Exposed.before no o: Open | this.follows [o] }
![Page 19: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/19.jpg)
no more counterexamples
Alloy’s analysis is› fully automatic› large bounded space› here, analyzed 2366 cases
![Page 20: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/20.jpg)
Controller
SafeOperators
Door PowerSource
open,close
touch
on, off
Sensor Switch
Closed
open,close Live
Exposed
summary
every step, send off if Closed became false
send on only when Closed is true
when open occurs, Closed becomes false
when close occurs, Exposed becomes false
when off occurs, Live becomes false
no touch unless Exposed is true
Live is initially false
Exposed is initially false
touch does not follow within 1 step of open Safe: touch event does not occur in state Live
![Page 21: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/21.jpg)
dependability cases we’ve worked on
Burr Proton Therapy Center› correct dose [Robert Seater]
› emergency stop [with Andrew Rae]
› treatment door interlock [Eunsuk Kang, Joe Near, Aleks Millicevic]
Voting systems› Pret a Voter [Robert Seater]
› Scantegrity [Eunsuk Kang]
Tokeneer› ongoing analysis [Eunsuk Kang]
![Page 22: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/22.jpg)
tokeneer
![Page 23: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/23.jpg)
tokeneer
› commissioned by NSA as exemplar› built by Praxis using Z and SPARK-Ada› not just open source!
![Page 24: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/24.jpg)
problem diagram
Controller
Users
Latch
Card Reader
FingerprintReader Door
Enclave
access enclave => have privilege
privilege access
attach, detach
insert, removeopen, close accessible, blocked
locked, unlocked
access
read tokenread fingerprint
lock, unlock
![Page 25: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/25.jpg)
analyzing the designwhat Praxis did› formal spec in Z (about 120 pages); informal reasoning› code verification with SPARK-Ada
defects found to date› 5 code-level defects› requirements issues (using Alloy for test case generation)
[Aydal & Woodcock 2009]› no defects yet found in design
what we’re doing› translating design to Alloy (about 1000 lines so far)› automatic analysis: design ∧ assumptions ⇒ security
![Page 26: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/26.jpg)
sample argument fragments
![Page 27: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/27.jpg)
sample screenshot
![Page 28: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/28.jpg)
results so far
bug in security property› if door is opened, user must hold token with recently
validated fingerprint or valid authorization certificate
bug in spec for UnlockDoor› timer not checked if token withdrawn after timeout
![Page 29: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/29.jpg)
proton therapy
![Page 30: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/30.jpg)
proton therapy treatment room
![Page 31: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/31.jpg)
correct dose requirement
!"#
!$%&'(%)'#
"&)&*%$
+,
-$%./$01'02)
+&'&3&.%
456
6)'%$7&/%!8%$&10.'
-&'0%)'
9:
,%&(#
;<=01(%)'
"%..&*%.#2)#
>%'?2$@
.%A%/'02)
<=%$B+2.%.C%<=%.'
<=%$B+2.%.C%.=A'
<=%$BD0.'C%.=A'
.%''0)*.
0)'%$1$%'&'02)E2.%
$%&E6+(.*
.%)ED6F!(.*
.%)E6+(.*
$%&ED6F!(.*
)&(%6)72
!"#$%&'()*'$(+$,-..),'/0$+)/),')1
)&(%6)72#G#.%A%/'02)
!23#$(1$(+$(*').%.)')1$&*1$+)*'
(&1H.%A%/'02)#G#.%)E6+(.*
!4#$5)++&6)$&.)$'.&*+5('')1$$&7'8)*'(,&//0
.%)ED6F!(.*#G#$%&ED6F!(.*
.%)E6+(.*#G#$%&E6+(.*
!9&#$:7).()+$.)!),'$13
<=%$BD0.'C%.=A'#G
)&(%.6)72#I#0)&/'0J%
<=%$B+2.%.C%.=A'#G#
<=%$B+2.%.C%<=%.'HE2.%.
!;3#$(1$<.-5$5)++&6)$(+$+)*'$'-$13
<=%$B+2.%.C%<=%.'#G#
$%&E6+(.*
!;,#$:7).()1$1-+)$(+$7+)1$'-$+)'$):7(%5)*'
.%''0)*.H0)'%$1$%'&'02)#G#
<=%$B+2.%.C%.=A'!=#$1-+)$1)/(>).0
K)&(%.6)72H)&(%6)72LHE2.%.#
G#E2.%
!2&#$(*').%.)'&'(-*$.)!),'+$5+6
(&1#G#$%&ED6F!(.*
!;&#$/(+'$(*<-$(+$+)*'
<=%$BD0.'C%.=A'#G#
.%)ED6F!(.*
!?#$@A$-%).&'(-*
.%''0)*.H0)'%$1$%'&'02)#G#E2.%
&AA#)M#>=(3%$#N
##2)%#)H0)'%$1$%'&'02)
!93#$'8).)$(+$-*/0$-*)$(1$<-.$)&,8$*&5)
&AA#)M#F'$0)*#N#2)%#)&(%.6)72H)
)&(%.6)72
E2.%.
)&(%6)72
E2.%
Figure 4-5: Argument diagram for the patient identity subproblem.
117
![Page 32: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/32.jpg)
correct dose case
extraction of models› Alloy models of messaging infrastructure› C code translated to Java, then to Alloy using Forge
resulting insights› very long message delay might cause bad dose› patient identification relies on distinct patient names› SQL injection attack vulnerability
![Page 33: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/33.jpg)
door interlock requirement
BeamManager
Treatment Manager
SafetyControl Unit Beam
Control Unit
DataDaq
RTWorks
TCP/IPRPC
Door
DoorOpen signal
msg
frameOut
callback(rtdaqinDoorOpen)
insertBeamStop
Nozzle
RPC(nsertBeamStop)
BeamStop signal
callback(inhibitBeam)
ACT_INHIBIT_BEAM
frameIn
Door Safety Requirement
opening door causesDoorOpen signal
signal causes (frame : Frame) where frame = signalFrameMap[signal] and TCP_IP.frameIn = frame
frameIn causes (frameOut : Frame) where frameOut = frameIn
frameOut causes (msg : RTWorks.msgs) wheremsg.type = RTMsgTypeMap[frameOut] andmsg.dest = RTMsgDestMap[frameOut]
(rtdaqinDoorOpen : callbacks) causes (msg : RTWorks.msgs)wheremsg.dest = BeamManager andmsg.type = ACT_INHIBIT_BEAM
(msg : msgs) causes (cb : dest.callbacks)wheredest = msg.dest andcb = CallbackMap[msg.type]
(inhibitBeam : callbacks) causes req : RPC.reqs wherereq.dest = BCU andreq.type = InhibitBeamStop
(req : reqs) causes call: dest.callswheredest = req.dest andcall = RPCCallMap[req.type]
BCU.beamInsert causes BeamStop signal
BeamStop signal causes beam stop being inserted
opening door causes beam stop being inserted
System ManagerlogEvent causes req : RPC.reqswherereq.dest = TCU andreq.type = evtReport
logEvent
TreatmentControl Unit
evtReport returns True
evtReport
RPC(evtReport)
![Page 34: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/34.jpg)
door interlock case
high level analysis in Alloy› by modelling each component› simple chain of events
code analysis› to identify side conditions› to extract control paths› but hard due to missing code
approach› lightweight extraction of control flow› abstract interpretation of state› user provides specs for library calls
![Page 35: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/35.jpg)
tracing call paths
tool and analysis by Aleks Millicevic
![Page 36: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/36.jpg)
tracing calls within a component
![Page 37: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/37.jpg)
results so far
entanglement› door safety entangled with logging› if logging fails, safety action is aborted› (but hardware safety system...)
![Page 38: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/38.jpg)
how to cheat
![Page 39: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/39.jpg)
Controller
SafeOperators
Door PowerSource
open,close
touch
on, off
Sensor Switch
Closed
open,close Live
Exposed
identifying the trusted base
every step, send off if Closed became false
send on only when Closed is true
when open occurs, Closed becomes false
when close occurs, Exposed becomes false
when off occurs, Live becomes false
no touch unless Exposed is true
Live is initially false
Exposed is initially false
touch does not follow within 1 step of open Safe: touch event does not occur in state Live
Controller
SafeOperators
Door PowerSource
open,close
touch
on, off
Sensor Switch
Closed
open,close Live
Exposed
![Page 40: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/40.jpg)
reducing the trusted base
Controller
SafeOperators
Door PowerSource
open,close
touch
on, off
Sensor Switch
Closed
open,close Live
Exposed
SafeOperators
Door PowerSource
open,close
touch
on, off
Switch
Live
Exposed
simpler design ⇒ simpler argument
![Page 41: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/41.jpg)
analysis with trusted basesone sig Sensor extends Domain { Closed: set Time }
sig Open extends Event { } { Sensor in OK implies not Sensor.Closed.after }
one sig Controller extends Domain { } { this in OK implies all t: Time - (first + last) | not Sensor.Closed.at [t] and Sensor.Closed.at [t.prev] implies Off.happensAt [t] }
one sig Safe extends Requirement {} { this in OK iff all t: Touch | not PowerSource.Live.before [t] trustedBase = Switch + Controller + Sensor + Door + Operators }
assert BaseSufficient { all r: Requirement | r.trustedBase in OK implies r in OK }
![Page 42: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/42.jpg)
reducing the trusted base: examples
![Page 43: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/43.jpg)
designing emergency stop
pendant with emergency stop button
![Page 44: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/44.jpg)
existing design
FileSystem
UI Agent
HandPendant
BeamBlock
EmergencyStop works
ControllerEventQueue
OperatingSystem
Event Registration
FileSystem
UI Agent
HandPendant
BeamBlock
EmergencyStop works
ControllerEventQueue
OperatingSystem
Event Registration
![Page 45: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/45.jpg)
redesign
FileSystem
UI Agent
HandPendant
BeamBlock
EmergencyStop works
ControllerEventQueue
OperatingSystem
Event Registration
EmergencyStop Unit
FileSystem
UI Agent
HandPendant
BeamBlock
EmergencyStop works
ControllerEventQueue
OperatingSystem
Event Registration
EmergencyStop Unit
![Page 46: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/46.jpg)
alarm clock
Most other alarm clock applications choose
to play the alarms/music via iTunes (via AppleScript). I deliberately decided against this... Consider...
• The alarm is set to play a specific song, but the song was deleted.
• The alarm is set to play a specific playlist, but you renamed the playlist, or deleted it.
• The alarm is set to play a radio station, but the
internet is down.
• iTunes was recently upgraded, and requires you to
reagree to the license next time you launch it. The alarm application launches it for the alarm...
• You had iTunes set to play to your airTunes speakers, but you left your airport card turned off.
• You had the iTunes preference panel open. (Which prevents AppleScript from working)
• You had a "Get Info" panel open. (Which also prevents AppleScript from working)
From Alarm Clock, http://www.robbiehanson.com/alarmclock/faq.html
... It’s only job is to wake you up in the morning, and I believe you'll find that it does it’s job perfectly.
![Page 47: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/47.jpg)
alarm clock
From Alarm Clock, http://www.robbiehanson.com/alarmclock/faq.html
iTunesAlarm
Controller
alarm
goes
o!
request to play
song
generated
song
played
Settings
Internet
Basic
Song
Player
Alarm
Controller
alarm
goes
o!
request to play
song
generated
song
played
![Page 48: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/48.jpg)
example: voting
Check-in
Desk
Optical
Scanner
Election
O!cial
All cast ballots
are counted
reports tally
from scanner
to public
accurately
records choice
on a ballot
computes tally
based on
records
gives one
ballot per
voter
scanner
computes tally
based on
ballots
Voters
standard design,relying on scanner
TabulatorCheck-in
DeskOptical
ScannerVoters
Election
O!cial
All cast ballots
are counted
Auditor
gives one
ballot per
voter
voters checks
their receipts
independent
tallies match
auditor checks
independent
tallies
computes
independent
tally
Scantegrity design,relying on votersand 3rd partytabulators
![Page 49: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/49.jpg)
conclusions
![Page 50: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/50.jpg)
what’s typically (not) done
Controller
SafeOperators
Door PowerSource
open,close
touch
on, off
Sensor Switch
Closed
open,close Live
Exposed
every step, send off if Closed became false
send on only when Closed is true
when open occurs, Closed becomes false
when close occurs, Exposed becomes false
when off occurs, Live becomes false
no touch unless Exposed is true
Live is initially false
Exposed is initially false
touch does not follow within 1 step of open Safe: touch event does not occur in state Live
critical properties not made explicit
phenomenanot designated
domain assumptionsnot recorded
specificationreferencesinaccessiblephenomena
no systematicanalysis
initializationmissed
![Page 51: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/51.jpg)
observations
on dependability cases› if you can’t say why it works, it probably doesn’t
on design› a principle: design for simple argument
on formal methods› two benefits: clarity of requirements, mechanical checks
on cost› key to low cost is upfront investment, non-uniformity
![Page 52: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/52.jpg)
too hard to argue, unsafe to build
The direction and amount of the complicated strains throughout the trussing [would] become incalculable as far as all practical purposes are concerned...Stephenson, explaining why he rejected a suspension design
Brittania Bridge (Robert Stephenson, 1850)
![Page 53: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/53.jpg)
Controller
SafeOperators
Door PowerSource
open,close
touch
on, off
Sensor Switch
Closed
open,close Live
Exposed
every step, send off if Closed became false
send on only when Closed is true
when open occurs, Closed becomes false
when close occurs, Exposed becomes false
when off occurs, Live becomes false
no touch unless Exposed is true
Live is initially false
Exposed is initially false
touch does not follow within 1 step of open Safe: touch event does not occur in state Live
a research question
when close occurs, Closed becomes true
send on when Closed becomes true
‘redundant’ propertiesshould they be included?if so, how?
![Page 54: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/54.jpg)
acknowledgments
joint work with my students› Eunsuk Kang, Joe Near, Aleks Millicevic
phenomenology› Michael Jackson, Problem Frames (2001)
dependability cases study› ‘Sufficient Evidence’ (NAS, 2007)
related work by many› van Lamsweerde, Kelly, etc (goal structuring)› Rushby, Knight, Bloomfield (assurance cases)› ...
support from NSF, Northrop Grumman, Mass General
![Page 55: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/55.jpg)
a paper about this approach
A Direct Path to Dependable Software, CACM, March 2009wordle thanks to Jonathan Feinberg, IBM Research, Cambridge
![Page 56: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/56.jpg)
backup slides
![Page 57: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/57.jpg)
designationseventsopen: operator opens door fully or partiallyclose: operator closes door fullytouch: operator touches poweron: controller issues command to switch to turn onoff: controller issues command to switch to turn off
statesExposed: power source is exposedLive: power in live stateClosed: sensor is in state that reports door closed
Controller
Operators
Door PowerSource
open,close
touch
on, off
Sensor Switch
Closed
open,close Live
Exposed
![Page 58: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/58.jpg)
what if analysis finds no flaws?
informal problems› wrong domain assumption› missing phenomena or interactions› wrong or badly expressed requirement
formal problems› scope not large enough› inconsistent axiomatization› analysis tool is broken› ... or system is actually safe
machine specdomain assumptions ∧ ⇒ requirement
![Page 59: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/59.jpg)
generic modules: domains
module domains
abstract sig Domain {}
abstract sig Property {}
abstract sig Requirement extends Property { trustedBase: set Domain }
sig OK in Domain + Property {}
assert BaseSufficient { all r: Requirement | r.trustedBase in OK implies r in OK }
![Page 60: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/60.jpg)
generic modules: events
module events
open util/ordering[Time] as time
sig Time {}
abstract sig Event { pre, post: Time }
fact Traces { all t: Time - last | some e: Event | e.pre = t and e.post = t.next all t: Time - last | lone e: Event | e.pre = t }
![Page 61: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/61.jpg)
examining side conditions
![Page 62: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/62.jpg)
on software risks
“We have become dangerouslydependent on large software systemswhose behavior is not well understoodand which often fail in unpredicted ways.”President's Information Technology Advisory Committee, 1999
“The most likely way for the world to be destroyed, most experts agree, is by accident.That’s where we come in. We’re computer professionals. We cause accidents.”Nathaniel Borenstein, Programming as if People Mattered, Princeton University Press, 1991
![Page 63: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/63.jpg)
on accidents
“Accidents are signals sent fromdeep within the systemabout the vulnerability andpotential for disaster that lie within”Richard Cook and Michael O’ConnorThinking About Accidents And Systems (2005)
![Page 64: how to prevent disasters - People | MIT CSAILpeople.csail.mit.edu › dnj › talks › veldhoven10 › veldhoven10.pdfhow to prevent disasters Daniel Jackson, MIT Siren//NL, Veldhoven](https://reader033.vdocuments.net/reader033/viewer/2022052612/5f0ef3337e708231d441bcf8/html5/thumbnails/64.jpg)
on design
“There probably isn’t a best way to build the system, or even any major part of it; much more important is to avoid choosing a terrible way, and to have a clear division of responsibilities among the parts.”Butler LampsonHints for computer system design (1983)