the issue of trust in autonomy - d1rkab7tlqy5f1.cloudfront.net safety and... · slow steaming –...
TRANSCRIPT
![Page 1: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/1.jpg)
The Issue of Trust in Autonomy
Scott N. MacKinnon
Division of Maritime Studies
Chalmers University
Gothenburg, Sweden
International Seminar on Safety and Security of Autonomous Vessels
21 March, 2018
Delft, The Netherlands
![Page 2: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/2.jpg)
Issues in Shipping Industry
• SUSTAINABILITY
– Decreased Manning
– Recruitment and Retention
– Time in Port (JIT)
• ENVIRONMENTAL CONCERNS
– Slow Steaming
– Route Optimization
• INCREASED COMPLEXITY
![Page 3: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/3.jpg)
Conceição, V., Carmo, B., Dahlman, J. and Navarro, A. (2017). Visualization in Maritime Navigation: A Critical Review. Presented at
the 8th International Conference on Applied Human Factors and Ergonomics, 17-21 July, Los Angeles, USA.
![Page 4: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/4.jpg)
e-Navigation
A strategy developed by the International Maritime Organization
to improve shipping safety and efficiency through better
organization of data on ships and on shore, and better data
exchange and communication between ships and the ship and
shore.
![Page 5: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/5.jpg)
![Page 6: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/6.jpg)
Automation*
1. the method of making a machine, a process, or a system work
without being directly controlled by a person
2. automatic working of a machine, process, or system by
mechanical or electronic devices that take the place of humans
*Merriam-Webster
![Page 7: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/7.jpg)
Automation*
• Decision-support (human in-the-loop)
• Decision-making (human out-of-the-loop)
- We have to think of the shipping industry as a complex
socio-technical system (which ¨now¨ has both centralized
and distributed systems)
1. the method of making a machine, a process, or a system work
without being directly controlled by a person
2. automatic working of a machine, process, or system by
mechanical or electronic devices that take the place of humans
*Merriam-Webster
![Page 8: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/8.jpg)
The more complex
a system is the
greater the risk that
something,
somewhere,
sometime will fail.
Automation complexity
Automation surprise
Automation irony
Ironies of Automation (Bainbridge, 1983)
![Page 9: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/9.jpg)
… and that failure
might come as a big
surprise
Automation complexity
Automation surprise
Automation irony
Ironies of Automation (Bainbridge, 1983)
![Page 10: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/10.jpg)
Automation complexity
Automation surprise
Automation ironyAutomation is most
reliable in simple tasks.
The more
functionalities needed
the less reliable
automation becomes.
Ironies of Automation (Bainbridge, 1983)
![Page 11: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/11.jpg)
From: SARUMS (SAFETY AND REGULATIONS FOR UNMANNED MARITIME SYSTEMS)
No Trust
Full Trust
![Page 12: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/12.jpg)
Trust (Interpersonal)
Disposition toward the world and the people in it (Rotter,
1967)
Socially learned expectations that vary based on social order
(Barber, 1983)
A willingness to accept vulnerability (Mayer et al, 1995)
The attitude that an ¨agent¨ will help achieve an
individual’s goals in a situation characterised by
uncertainty and vulnerability (Lee and See, 2004)
![Page 13: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/13.jpg)
Trust* (Human Automation)
Performance-based trust: how well an automated system
executes a task
Process-based trust: based on the operator’s
understanding of the methods an automated system uses to
perform tasks
Purpose-based trust: the designer’s intended use for an
automated system
*Hoff and Bashir (2015)
![Page 14: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/14.jpg)
Trust* (Human Automation)
Dispositional trust: represents an individual’s overall
tendency to trust automation, independent of context or a
specific system (culture, gender, personality, AGE)
Situational trust:
- External: depends largely on the type of system,
complexity and difficulty of task
- Internal: interaction with self-confidence
Learned trust: drawn from past experiences of current
interactions
*Hoff and Bashir (2015)
![Page 15: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/15.jpg)
The relationship among calibration, resolution, and automation capability in defining appropriate
trust in automation.
Lee and See (2004). Trust in Automation: Designing for Appropriate Reliance. HUMAN FACTORS, Vol. 46 (1).
![Page 16: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/16.jpg)
Maritime Unmanned Navigation through Intelligence in Networks
– Research the feasibility of autonomous unmanned ship and
human-center automation governance from shore-based
facilities during deep sea voyage
– 3 year EU sponsored Framework 7 project
– Simulated 200 meters long dry bulk carrier
![Page 17: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/17.jpg)
So what did these ¨Operators¨ do?
![Page 18: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/18.jpg)
…and who will be the
¨trusting¨people…
![Page 19: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/19.jpg)
The ¨Generations¨
• Silent (1925-1941)
• Baby Boomer (1946–1964)
• Generation X (1960-1980)
• Millennials (1980-2000)
• Generation Z (2000-+…)
![Page 20: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/20.jpg)
Generation Z
• Always had the internet
• Comfortable with technology
• Non-Technical Skills?
![Page 21: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/21.jpg)
Cognitive
Psychological
School
So where might the errors occur?
![Page 22: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/22.jpg)
Swiss Cheese Model
Preconditions
Organisational influences
Supervision
Unsafe Acts
![Page 23: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/23.jpg)
![Page 24: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/24.jpg)
Memory
Sensory
Processing
Decision /
response
selection
Response
execution
Responses
Feedback
Long-term
memory
Working
memory
PerceptionStimuli
Attention
resources
Figure after Wickens (1992)
SLIPS
MISTAKES
LAPSES &
MODE ERRORS
Knowledge
Rule
![Page 25: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/25.jpg)
Joint Cognitive
Systems School
![Page 26: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/26.jpg)
![Page 27: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/27.jpg)
27
Work-as-imagined
Work-as-done
(What stakeholders believe happens or should happen)
(What actually happens)
What are the implications for the design and execution of work procedures?
What are the implications for recovery from an unexpected event?
![Page 28: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/28.jpg)
Memory
Sensory
Processing
Decision /
response
selection
Response
execution
Responses
Feedback
Long-term
memory
Working
memory
PerceptionStimuli
Attention
resources
Limited Capacity
Growing Complexity
Figure after Wickens (1992)
![Page 29: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/29.jpg)
Creating Trust with the User
• Design for appropriate trust, not greater trust.
• Show the past performance of the automation. (Big Data)
• Show the process and algorithms of the automation by revealing
intermediate results in a way that is comprehensible to the operators. (Data
Transparency)
• Show how automation is related to the users’/system’s goals. (Confirmation
of Mental Model)
• Train operators regarding its expected reliability, the mechanisms
governing its behaviour, and its intended use. (Lessons from Aviation and
Nuclear Industry)
![Page 30: The Issue of Trust in Autonomy - d1rkab7tlqy5f1.cloudfront.net Safety and... · Slow Steaming – Route ... to improve shipping safety and efficiency through better organization of](https://reader033.vdocuments.net/reader033/viewer/2022060316/5f0c358d7e708231d4344728/html5/thumbnails/30.jpg)
Thank you!
Questions?