Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/ 1
Understanding the Understanding the Human in the LoopHuman in the Loop
January 16, 2008
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/ 2
HumansHumans“Humans are incapable of securely storing high-quality cryptographic keys, and they have unacceptable speed and accuracy when performing cryptographic operations. (They are also large, expensive to maintain, difficult to manage, and they pollute the environment. It is astonishing that these devices continue to be manufactured and deployed. But they are sufficiently pervasive that we must design our protocols around their limitations.)”
-- C. Kaufman, R. Perlman, and M. Speciner. Network Security: PRIVATE Communication in a PUBLIC World.
2nd edition. Prentice Hall, page 237, 2002.
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/ 3
Humans are weakest linkHumans are weakest linkMost security breaches attributed to
“human error”
Social engineering attacks proliferate
Frequent security policy compliance failures
Automated systems are generally more predictable and accurate than humans
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/ 4
Why are humans in the loop at Why are humans in the loop at all?all?Don’t know how or too expensive to
automate
Human judgments or policy decisions needed
Need to authenticate humans
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/ 5
The human threatThe human threatMalicious humans who will attack system
Humans who don’t know when or how to perform security-critical tasks
Humans who are unmotivated to perform security-critical tasks properly or comply with policies
Humans who are incapable of making sound security decisions
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/ 6
Need to better understand Need to better understand humanshumansDo they know they are supposed to be
doing something?
Do they understand what they are supposed to do?
Do they know how to do it?
Are they motivated to do it?
Are they capable of doing it?
Will they actually do it?
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/ 7
Proposed frameworkProposed framework Cranor interactions article: What do they
"indicate?": evaluating security and privacy indicators
The Handbook of Warnings, edited by Michael S. Wogalter• Wogalter’s Communication-Human Information
Processing (C-HIP) Model Applied C-HIP to security indicators evaluation
from interactions article Expanded it to model other types of human
interaction with secure systems Developed “Human in the loop security framework”
and “Human threat identification and mitigation process” - paper under review
Need validation and more work on mitigation and how to operationalize process
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/ 8
C-HIP ModelC-HIP Model Communication-
Human Information Processing (C-HIP) Model• Wogalter, M. 2006.
Communication-Human Information Processing (C-HIP) Model. In Wogalter, M., ed., Handbook of Warnings. Lawrence Erlbaum Associates, Mahwah, NJ, 51-61.
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/ 9
Human in the loop security Human in the loop security frameworkframework
Human ReceiverHuman Receiver
Co
mm
un
icat
ion
Pro
cess
ing
Co
mm
un
icat
ion
Pro
cess
ing
Ap
plic
atio
nA
pp
licat
ion
Co
mm
un
icat
ion
Del
iver
yC
om
mu
nic
atio
nD
eliv
ery
IntentionsIntentions
Attention Switch
AttentionMaintenance
Comprehension
KnowledgeRetention
KnowledgeTransfer
Motivation
Attitudes and Beliefs
KnowledgeAcquisition
CommunicationCommunication BehaviorBehavior
Personal VariablesPersonal Variables
Knowledgeand
Experience
Demographicsand Personal Characteristics
Communication Impediments
Communication Impediments
Interference
EnvironmentalStimuli
CapabilitiesCapabilities
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/10
Communication processing Communication processing modelmodelFramework is based on communication
processing model• Many models in the literature• Used to model all sorts of different types of
communications: individual, group, media, etc.
Most end-user security actions are triggered by some form of communication• Pop-up alert, email, manual, etc.
Expert self-discovery of a security process can be modeled as communication to oneself
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/11
CommunicationCommunication
Human ReceiverHuman Receiver
Co
mm
un
icat
ion
Pro
cess
ing
Co
mm
un
icat
ion
Pro
cess
ing
Ap
plic
atio
nA
pp
licat
ion
Co
mm
un
icat
ion
Del
iver
yC
om
mu
nic
atio
nD
eliv
ery
IntentionsIntentions
Attention Switch
AttentionMaintenance
Comprehension
KnowledgeRetention
KnowledgeTransfer
Motivation
Attitudes and Beliefs
KnowledgeAcquisition
CommunicationCommunication BehaviorBehavior
Personal VariablesPersonal Variables
Knowledgeand
Experience
Demographicsand Personal Characteristics
Communication Impediments
Communication Impediments
Interference
EnvironmentalStimuli
CapabilitiesCapabilities
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/12
Types of security Types of security communicationscommunications Warnings• Alert users to take immediate action to avoid hazard
Notices • Inform users about characteristics of entity or object
Status indicators • Inform users about system status information
Training • Teach users about threat and how to respond
Policy • Inform users about policies
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/13
Active versus passive Active versus passive communicationscommunications
FirefoxAnti-Phishing
Warning
FirefoxAnti-Phishing
Warning
Active Passive
Bluetoothindicator in
Mac menu bar
Bluetoothindicator in
Mac menu bar
Indicators with audio
alerts
Indicators with audio
alerts
Indicators with
animation
Indicators with
animation
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/14
Communication impedimentsCommunication impediments
Human ReceiverHuman Receiver
Co
mm
un
icat
ion
Pro
cess
ing
Co
mm
un
icat
ion
Pro
cess
ing
Ap
plic
atio
nA
pp
licat
ion
Co
mm
un
icat
ion
Del
iver
yC
om
mu
nic
atio
nD
eliv
ery
IntentionsIntentions
Attention Switch
AttentionMaintenance
Comprehension
KnowledgeRetention
KnowledgeTransfer
Motivation
Attitudes and Beliefs
KnowledgeAcquisition
CommunicationCommunication BehaviorBehavior
Personal VariablesPersonal Variables
Knowledgeand
Experience
Demographicsand Personal Characteristics
Communication Impediments
Communication Impediments
Interference
EnvironmentalStimuli
CapabilitiesCapabilities
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/15
Environmental stimuliEnvironmental stimuliDivert user’s attention
Greatest impact on passive communication
Examples• Other communications• Ambient light and noise• User’s primary task
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/16
InterferenceInterference Anything that may prevent a communication from
being received as the sender intended
Caused by• Malicious attackers• Technology failures• Environmental stimuli that obscure the communication
Focus of traditional secure systems analysis• How can attacker interfere with communications?
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/17
Human receiverHuman receiver
Human ReceiverHuman Receiver
Co
mm
un
icat
ion
Pro
cess
ing
Co
mm
un
icat
ion
Pro
cess
ing
Ap
plic
atio
nA
pp
licat
ion
Co
mm
un
icat
ion
Del
iver
yC
om
mu
nic
atio
nD
eliv
ery
IntentionsIntentions
Attention Switch
AttentionMaintenance
Comprehension
KnowledgeRetention
KnowledgeTransfer
Motivation
Attitudes and Beliefs
KnowledgeAcquisition
CommunicationCommunication BehaviorBehavior
Personal VariablesPersonal Variables
Knowledgeand
Experience
Demographicsand Personal Characteristics
Communication Impediments
Communication Impediments
Interference
EnvironmentalStimuli
CapabilitiesCapabilities
“The human in the loop”
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/18
Communication deliveryCommunication delivery Attention switch• Noticing communication
Attention maintenance• Paying attention long enough to process
communication
Breakdowns• Environmental stimuli, interference• Characteristics of communication• Habituation
Tendency for the impact of stimuli to decrease over time
Just because the communication appeared on the user’s screen, doesn’t mean the user actually saw it
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/19
Communication processingCommunication processingComprehension• Ability to understand communication
Knowledge acquisition• User’s ability to learn what to do in response
Breakdowns• Unfamiliar symbols, vocabulary, complex
sentences, conceptual complexity
Even if a user understands the communication, they still may not know what they are supposed to do
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/20
ApplicationApplicationKnowledge retention• Ability to remember communication
Knowledge transfer• Ability to recognize situations where the
communication is applicable and figure out how to apply it
Some security communications are always applied immediately (for example, pop-up warnings) so retention and transfer may not be necessary
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/21
Personal variablesPersonal variablesDemographics and personal characteristics• Age, gender, culture, education, occupation,
disabilities
Knowledge and experience• Education, occupation, prior experience
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/22
IntentionsIntentions Attitudes and beliefs• Beliefs about communication accuracy• Beliefs about whether they should pay attention• Self-efficacy - whether they believe they can complete
actions effectively• Response-efficacy - whether they believe the actions
they take will be effective• How long it will take• General attitudes - trust, annoyance, etc.
Motivation• Incentives, disincentives
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/23
CapabilitiesCapabilitiesUser’s level of ability• Cognitive or physical skills• Availability of necessary software or devices
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/24
BehaviorBehavior
Human ReceiverHuman Receiver
Co
mm
un
icat
ion
Pro
cess
ing
Co
mm
un
icat
ion
Pro
cess
ing
Ap
plic
atio
nA
pp
licat
ion
Co
mm
un
icat
ion
Del
iver
yC
om
mu
nic
atio
nD
eliv
ery
IntentionsIntentions
Attention Switch
AttentionMaintenance
Comprehension
KnowledgeRetention
KnowledgeTransfer
Motivation
Attitudes and Beliefs
KnowledgeAcquisition
CommunicationCommunication BehaviorBehavior
Personal VariablesPersonal Variables
Knowledgeand
Experience
Demographicsand Personal Characteristics
Communication Impediments
Communication Impediments
Interference
EnvironmentalStimuli
CapabilitiesCapabilities
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/25
BehaviorBehaviorUsers may complete recommended action,
but do so in a way that follows a predictable pattern that can be exploited by attackers• Example: password choice
Users may intend to comply, but may fail to complete necessary action
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/26
GulfsGulfsDon Norman. The Design of Every Day
Things.1988.
Gulf of Execution• Gap between a person’s intentions to carry out an
action and the mechanisms provided by a system to facilitate that action “I can’t figure out how to make it do what I want it to do”
Gulf of Evaluation• When a user completes an action but is unable to
interpret the results to determine whether it was successful “I can’t figure out whether it worked”
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/27
Generic Error-Modeling SystemGeneric Error-Modeling SystemJames Reason. Human Error. 1990.
Mistakes• When people formulate action plans that will
not achieve the desired goal
Lapses• When people formulate suitable action plans,
but forget to perform a planned action (for example, skipping a step)
Slips• When people perform actions incorrectly (for
example, press the wrong button)
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/28
Human threat identification Human threat identification and mitigation processand mitigation process
TaskIdentification
TaskIdentification
TaskAutomation
TaskAutomation
FailureIdentification
FailureIdentification
FailureMitigation
FailureMitigation
Human-in-the-loopFramework
User Studies
User Studies
Task identification• Identify all points where the system relies on humans to perform security-
critical functions
Task automation• Find ways to partially or fully automate some of these tasks
Failure identification• Identify potential failure modes for remaining tasks
Failure mitigation• Find ways to prevent these failures
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/30
Typical password policyTypical password policyPick a hard to guess password
Don’t use it anywhere else
Change it often
Don’t write it down
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/31
Typical password practiceTypical password practice
Bank = b3aYZ Amazon = aa66x!Phonebill = p$2$ta1
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/32
Why don’t users follow password Why don’t users follow password policies?policies?
TaskIdentification
TaskIdentification
TaskAutomation
TaskAutomation
FailureIdentification
FailureIdentification
FailureMitigation
FailureMitigation
Human-in-the-loopFramework
User Studies
User Studies
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/33
Why don’t users follow password Why don’t users follow password policies?policies?
Human ReceiverHuman Receiver
Co
mm
un
icat
ion
Pro
cess
ing
Co
mm
un
icat
ion
Pro
cess
ing
Ap
plic
atio
nA
pp
licat
ion
Co
mm
un
icat
ion
Del
iver
yC
om
mu
nic
atio
nD
eliv
ery
IntentionsIntentions
Attention Switch
AttentionMaintenance
Comprehension
KnowledgeRetention
KnowledgeTransfer
Motivation
Attitudes and Beliefs
KnowledgeAcquisition
CommunicationCommunication BehaviorBehavior
Personal VariablesPersonal Variables
Knowledgeand
Experience
Demographicsand Personal Characteristics
Communication Impediments
Communication Impediments
Interference
EnvironmentalStimuli
CapabilitiesCapabilities
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/35
Do users notice them?Do users notice them?“What lock icon?”• Few users notice lock icon in browser chrome,
https, etc.
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/36
Do users know what they Do users know what they mean?mean?Web browser lock icon:• “I think that it means secured, it symbolizes
some kind of security, somehow.”
Web browser security pop-up:• “Yeah, like the certificate has expired. I don’t
actually know what that means.”
J. Downs, M. Holbrook, and L. Cranor. Decision Strategies and Susceptibility to Phishing. In Proceedings of the 2006 Symposium On Usable Privacy and Security, 12-14 July 2006, Pittsburgh, PA.
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/37
Do they do what they advise?Do they do what they advise?“I would probably experience some brief, vague sense of unease and close the box and go about my business.”
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/38
Why don’t users heed browser security Why don’t users heed browser security warnings?warnings?
TaskIdentification
TaskIdentification
TaskAutomation
TaskAutomation
FailureIdentification
FailureIdentification
FailureMitigation
FailureMitigation
Human-in-the-loopFramework
User Studies
User Studies
Usable Privacy and Security • Carnegie Mellon University • Spring 2008 • Lorrie Cranor • http://cups.cs.cmu.edu/courses/ups.html/39
Why don’t users heed browser security Why don’t users heed browser security warnings?warnings?
Human ReceiverHuman Receiver
Co
mm
un
icat
ion
Pro
cess
ing
Co
mm
un
icat
ion
Pro
cess
ing
Ap
plic
atio
nA
pp
licat
ion
Co
mm
un
icat
ion
Del
iver
yC
om
mu
nic
atio
nD
eliv
ery
IntentionsIntentions
Attention Switch
AttentionMaintenance
Comprehension
KnowledgeRetention
KnowledgeTransfer
Motivation
Attitudes and Beliefs
KnowledgeAcquisition
CommunicationCommunication BehaviorBehavior
Personal VariablesPersonal Variables
Knowledgeand
Experience
Demographicsand Personal Characteristics
Communication Impediments
Communication Impediments
Interference
EnvironmentalStimuli
CapabilitiesCapabilities