mobile attention management identifying and harnessing factors that determine reaction to mobile...
TRANSCRIPT
Mobile Attention Management Identifying and Harnessing Factors that Determine
Reaction to Mobile Notifications
Dr Veljko Pejović Faculty of Computer and Information Science
University of Ljubljana, Slovenia
FreeUniversityofBozen-Bolzano,September2016
Mobile Notifications • Mobile phone is the most
direct point of contact • Increasingly interactive lives • For recipients, notifications
are a means of real-time information awareness
• For senders, a way to initiate remote communication
Mobile Notifications • If not managed properly:
– Interrupt ongoing tasks resulting in poor completion rates, errors
– Interfere with users’ lifestyle – Arrive at socially inappropriate
moments – Increase frustration – May result in negative brand perception – Get ignored
Design and develop a system for intelligent notification scheduling on a mobile device
Goal
Premise: notification timing
is the key!
Identify opportune moments to deliver information to a mobile user
But first…
Sensing Interruptibility • Hypothesis: sensed context reveals interruptibility
Sensing Interruptibility • Hypothesis: sensed context reveals interruptibility • Measuring interruptibility
– Reaction presence ? t
Sensing Interruptibility • Hypothesis: sensed context reveals interruptibility • Measuring interruptibility
– Reaction presence – Time to reaction
ttreaction
Sensing Interruptibility • Hypothesis: sensed context reveals interruptibility • Measuring interruptibility
– Reaction presence – Time to reaction – Sentiment I hate you
t
Sensing Interruptibility • Test with data:
– SampleMe: Android experience sampling app • (Reactions to) notifications • Sensor data • 20 users, two weeks, approx
eight notifications per day
V. Pejovic and M. Musolesi, InterruptMe: Designing Intelligent Prompting Mechanisms for Pervasive Applications, ACM UbiComp’14, Seattle, WA, USA
Modeling Interruptibility
Context at the time of
notifying (tN)
Classifiers: - Batch, online - Naïve
Bayesian, Tree-based, Boosting
Response presence
Response time
Sentiment
• Findings: – Reaction presence
• Precision is important
Classifier Precision Recall
AdaBoost 0.64 0.41
BayesNet 0.58 0.37
NaïveBayes 0.57 0.52
Baseline 0.39 0.38
Modeling Interruptibility
• Findings: – Reaction presence
• Precision is important – Time to reaction
• Coarse-grain inference
00.10.20.30.40.50.60.7
5 1015202530354045505560
Prec
isio
n
Answer threshold [min]
Baseline Naïve Bayes Bayes Net Boosting
Modeling Interruptibility
Modeling Interruptibility • Findings:
– Reaction presence • Precision is important
– Time to reaction • Coarse-grain inference
– Sentiment • Difficult to identify very
much suitable moments to interrupt
0
0.2
0.4
0.6
0.8
1
1 2 3
Prec
isio
n
Min acceptable sentiment [1-"a bit", 2-"some", 3-"very much"]
Baseline Naïve Bayes Bayes Net Boosting
Design and develop a system for intelligent notification scheduling on a mobile device
Goal
Towards real-time interruptibility recognition on
the mobile!
InterruptMe • Interruptibility lib:
– Connect sensed context with user interruptibility
– Build personalised or shared models
– Ensure persistence
ApplicaHon
AndroidOS
InterruptMe
NoHficaHon
Sensorstream
InterrupHonmanager
Machinelearninglib
Predictedoutcome
Sensinglib
Sensorfeatures
Decision
Outcomefeedback
Trainclassifier
InterruptMe publicclassNo9fica9onServiceextendsServiceimplementsIntelligentTriggerReceiver{IntelligentTriggerManagerinterrupHonMngr;publicvoidonCreate(){interrupHonMngr=IntelligentTriggerManager.getTriggerManager(this); }publicvoidonTriggerReceived(Stringa_triggerID,ArrayList<LearnerResultBundle>a_bundles){//…sendanoHficaHonusingNoHficaHonManager}
//…incaseuserrespondstoanoHficaHoninterrupHonMngr.trainLearnerFromFeedback(Constants.MOD_INTERRUPTIBILITY,
currentTriggerNo,"yes");
InterruptMe Evaluation • SampleMe + InterruptMe:
– Context-aware experience sampling – N-of-1 trial
(random vs intelligent notification triggering)
– Ten users for one month
InterruptMe Evaluation • Faster response to
InterruptMe-based notifications
0
0.2
0.4
0.6
0.8
1
0 50 100 150 Response time [min]
Random InterruptMe
InterruptMe Evaluation • Faster response to
InterruptMe-based notifications
• No significant difference in the sentiment towards being interrupted
Very much Some A bit Not at all
0.00
0.20
0.40
0.60
0.80
1.00
CDF of min reported sentiment
Random InterruptMe
InterruptMe Evaluation • Faster response to
InterruptMe-based notifications
• No significant difference in the sentiment towards being interrupted
• Amount of interruptibility 0
0.2 0.4 0.6 0.8
1 1.2 1.4
0 1 2 3 4 5 6 7 8 9 10
Sent
imen
t
Number of notifications in previous two hours
InterruptMe Evaluation • Isolated InterruptMe
notifications received better than random notifications
Very much Some A bit Not at all 0.00
0.20
0.40
0.60
0.80
1.00
CDF of min reported sentiment
Random InterruptMe
InterruptMe – the first context-aware real-time mobile notification management system that leads to faster responses and better received
notifications.
Conclusion
However…
… no significant effects of notification
scheduling on the usage of a
behavioural change intervention app
Conclusion
What Did We Miss? • We had a limited dataset – only a single
type of a notification
• We did not take content into account
• We overlooked the role of a sender-receiver relationship
NotifyMe Dataset • Sensed context • Reaction to a notification • Notification data
– Category – Sender ID
• User preferences: – Where and when would you like
to receive notifications with similar content A. Mehrotra, M. Musolesi, R. Hendley and V. Pejovic,
Designing Content-Driven Intelligent Notification Mechanisms for Mobile Applications, ACM UbiComp’15, Osaka, Japan.
NotifyMe Dataset • Participants: 35 • Study duration: 3 weeks • Notification samples: 70,000 (approx) • Questionnaire responses: 4,069
NotifyMe Dataset • Application name and
notification title extracted – Manual app name
classification
NotifyMe Dataset • Recipient-sender
relationship – Users classified their
notifications to: Family, Friend, Work, Other
Chat and email divided into eight
sub-categories
The Impact of Content/Sender • Notification click
count differs between application types (i.e. content type) and sender-receiver relations
Predicting Opportune Moments • Moments in which a user reacts quickly and accepts a
notification • Predictor flavours (depending on features):
1. User-defined rules and context data 2. Context data 3. Information type and context data 4. Social circle, information type and context data
• Machine learning algorithms: – Naïve Bayes, Ada Boost, Random Forest
Prediction Results
By using information type and social circle we were able to predict the acceptance of a notification within 10 minutes from its arrival time with an average sensitivity of 70% and a specificity of 80%.
We identified a number of factors that impact the acceptance of a notification. However,
reactions to notifications are diverse. How exactly will a user react to a notification?
Detailed Analysis of User Reactions
swipe
click
seen time decision time
Seen Time: time from the notification arrival until the notification was seen by the user
A. Mehrotra, V. Pejovic, J. Vermeulen, R. Hendley and M. Musolesi, My Phone and Me: Understanding People’s Receptivity to Mobile Notifications
ACM CHI’16, San Jose, CA, USA.
Detailed Analysis of User Reactions
swipe
click
seen time decision time
Decision Time: time from the moment a user saw a notification until the time they acted upon it (click or dismiss) A. Mehrotra, V. Pejovic, J. Vermeulen, R. Hendley and M. Musolesi,
My Phone and Me: Understanding People’s Receptivity to Mobile Notifications ACM CHI’16, San Jose, CA, USA.
My Phone And Me App • Automated logging:
– Notification time of arrival, seen, removal – Notification response – Notification details (title, app) – Alert type – Context (physical activity, location, etc.)
• Experience sampling: – Questions about how the notification was
handled, sender-receiver relationship, cognitive context, personality
A B
C D
Key Results – Reaction Times • Alert modality, current task type and complexity impact
the seen time – We are faster to react when engaged in a complex task
• Relationship with the sender impacts the decision time – We are faster to decide on a message that comes from a close
person (partners, immediate family members), than on a message that come from a less familiar person (extended family, service providers)
Key Results – Disruption • Sender/receiver relationship:
messages from subordinates and system messages are considered as most disruptive. Extended family members are considered as the least disruptive.
• Task engagement: more disruptive if it arrives when the user is in the middle of or finishing a task. Perceived disruption increases with the complexity of an ongoing task.
Confirmed in another study: V. Pejovic, A. Mehrotra and M. Musolesi,
Investigating The Role of Task Engagement in Mobile Interruptibility,
Smarttention workshop with ACM Ubicomp’15.
Key Results – Personality • Data is limited as only 11 user
have fully completed the personality test
• Hints towards extroverts being more inclined to be disrupted by a notification
Linear regression with the average disruption as a dependent variable and
personality traits as independent variables
How does a thought get disrupted?
Theory of Multitasking Resources • Perceptual and motor • Cognitive
– Procedural memory – Declarative memory
Mechanisms • Resource use is exclusive
– one task at a time per resource
• Multiple problem threads run in parallel, but processing is still serial [Salvucci2008]
Theory of Multitasking • Interference when two or more threads ask for the same
resource at a time
Example from [Borst2010]
Theory of Multitasking • Complex tasks require problem state saving/retrieving
Example from [Borst2015]
Practical Implications on Mobile Attention Management
• Interruptions are more disruptive if they require problem state switching
• Make them less disruptive by interrupting: – At moments when a task is not fully active (e.g. just starting, or
just finished) – At moments when a task does not require a problem state – At moments when a user is working on a task that is well
practiced, a routine
Can we automatically infer task engagement?
G. Urh and V. Pejovic, TaskyApp: Inferring Task Engagement via Smartphone Sensing
Ubittention workshop with ACM UbiComp’16, Heidelberg, Germany.
TaskyApp • Can smartphones sense that their users
are busy (in an office setting)? • TaskyApp data collection app
– Background sensing of device movement, ambient sound, collocation with other devices
– Data labelling via experience sampling and retroactive assisted labelling
– Recruited eight office workers for five weeks • 232 labelled instances (3035 unlabelled) • Most data between 8am and 6pm
TaskyApp – Data Analysis • Linear regression fit with sensed
features as independent variables and task difficulty (1-5 on a Likert scale) as a dependent variable – Movement data gives the most
informative features – The regression explains only a small
part of the data
TaskyApp – Data Analysis • Classify a task engagement moment as either “easy” or
“difficult” depending on the sensed features – We experimented with different classifiers but Naïve Bayes
seems to work best (probably due to the low amount of data) • 62.5% accuracy compared to 52.8% baseline • Also, leads to “favourable” errors –
few difficult tasks predicted as easy
Task Engagement Inference • Even in a restricted office setting smartphone-based task
inference is difficult • Movement features seem to be the most informative • Next step – wearables
– Sense heart rate and skin temperature
Conclusions • Notification success depends on the content, sender-
receiver relationship, channel, context in which a user is • Our reaction and the disturbance levels depend on the
alert type, sender, existing task engagement • User behaviour is highly personalised (our personalised
data have performed better than the general ones) • Personality may play a role in the way we perceive
notifications • Task engagement inference requires additional sensors
References • [Pejovic2014] V. Pejovic and M. Musolesi, InterruptMe: Designing Intelligent Prompting Mechanisms for
Pervasive Applications, ACM UbiComp’14, Seattle, WA, USA • [Mehrotra2015] A. Mehrotra, M. Musolesi, R. Hendley and V. Pejovic, Designing Content-Driven Intelligent
Notification Mechanisms for Mobile Applications, ACM UbiComp’15, Osaka, Japan. • [Pejovic2015] V. Pejovic, A. Mehrotra and M. Musolesi, Investigating The Role of Task Engagement in Mobile
Interruptibility, Smarttention workshop with ACM Ubicomp’15. • [Mehrotra2016] A. Mehrotra, V. Pejovic, J. Vermeulen, R. Hendley and M. Musolesi, My Phone and Me:
Understanding People’s Receptivity to Mobile Notifications, ACM CHI’16, San Jose, CA, USA. • [Urh2016] G. Urh and V. Pejovic, TaskyApp: Inferring Task Engagement via Smartphone Sensing, Ubittention
workshop with ACM UbiComp’16, Heidelberg, Germany. • [Salvucci2008] Salvucci, Dario D., and Niels A. Taatgen. "Threaded cognition: an integrated theory of
concurrent multitasking." Psychological review 115.1 (2008): 101. • [Borst2010] Borst, Jelmer P., Niels A. Taatgen, and Hedderik van Rijn. "The problem state: a cognitive
bottleneck in multitasking." Journal of Experimental Psychology: Learning, memory, and cognition 36.2 (2010): 363.
• [Borst2015] Borst, Jelmer P., Niels A. Taatgen, and Hedderik van Rijn. "What Makes Interruptions Disruptive?: A Process-Model Account of the Effects of the Problem State Bottleneck on Task Interruption and Resumption." CHI’15, 2015.
Collaborators Abhinav Mehrotra University of Birmingham UK
Dr Mirco Musolesi University College London UK
Gašper Urh University of
Ljubljana Slovenia
Prof Robert Hendley University of Birmingham
UK
Dr Jo Vermeulen University of Calgary
Canada
Thank you! Veljko Pejovic
University of Ljubljana [email protected]
http://lrss.fri.uni-lj.si/Veljko @veljkoveljko
bitbucket.org/veljkop/intelligenttrigger github.com/vpejovic/MachineLearningToolkit