EC-Web - September 2014, Munich, Germany
Usability Assessment of a Context-Aware and Personality-Based Mobile Recommender
System
Matthias Braunhofer, Mehdi Elahi and Francesco Ricci!
Free University of Bozen - BolzanoPiazza Domenicani 3, 39100 Bolzano, Italy{mbraunhofer,mehdi.elahi,fricci}@unibz.it
EC-Web - September 2014, Munich, Germany
Outline
2
• Context-Aware Recommender Systems and their Challenges
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions, Lessons Learned and Future Work
EC-Web - September 2014, Munich, Germany
Outline
2
• Context-Aware Recommender Systems and their Challenges
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions, Lessons Learned and
EC-Web - September 2014, Munich, Germany
Context is Essential
• Main idea: users can experience items differently depending on the current contextual situation (e.g., season, weather, temperature, mood)
• Example:
3
EC-Web - September 2014, Munich, Germany
Context-Aware Recommender Systems (CARSs)
• CARS extend Recommender Systems (RSs) beyond users and items to the contexts in which items are experienced by users
• Rating prediction function is: R: Users × Items × Context → Ratings
4
3 ? 4
2 5 4
? 3 4
1 ? 1
2 5
? 3
3 ? 5
2 5
? 3
5 ? 5
4 5 4
? 3 5
EC-Web - September 2014, Munich, Germany
Challenges for CARSs
• Identification of contextual factors (e.g., weather) that are worth considering when generating recommendations
• Acquisition of a representative set of contextually-tagged ratings
• Development of a predictive model for predicting the user’s ratings for items under various contextual situations
• Design and implementation of a human-computer interaction (HCI) layer on top of the predictive model
5
EC-Web - September 2014, Munich, Germany
Challenges for CARSs
• Identification of contextual factors (e.g., weather) that are worth considering when generating recommendations
• Acquisition of a representative set of contextually-tagged ratings
• Development of a predictive model for predicting the user’s ratings for items under various contextual situations
• Design and implementation of a human-computer interaction (HCI) layer on top of the predictive model
5
Focus of this research
EC-Web - September 2014, Munich, Germany
Outline
6
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions and Future Work
• Context-Aware Recommender Systems and their Challenges
EC-Web - September 2014, Munich, Germany
• Effectiveness of a RS depends not only on the underlying prediction algorithm but also on the proper design of the human-computer interaction (Swearingen and Sinha, 2001)
• User’s interaction with RSs:
HCI Perspective on RSs
7
Recommendation Algorithms
Input from user (ratings)
Output to user (recommendations)
• No. of ratings • Time to register • Details about item
to be rated • Type of rating scale • …
• No. of good recs. • No. of new, unknown recs. • Information about each rec. • Confidence in prediction • Is system logic transparent? • …
EC-Web - September 2014, Munich, Germany
Usability Assessment of RSs (1/2)
• Evaluation of the usability of a context-aware and group-based restaurant RS using the System Usability Scale (SUS) (Park et al., 2008)
• The SUS is a 10-item instrument to measure the user’s perceived usability of a system (Brooke, 1996)
• Major finding: the SUS score with 13 test users was 70.58, a rating between “ok” and “good”, and corresponding to a “C” grade, which is an acceptable level of usability
8
EC-Web - September 2014, Munich, Germany
Usability Assessment of RSs (2/2)
• Usage of eye tracking, clickstream analysis and SUS to determine the usability of a constraint-based travel advisory system called VIBE (Jannach et al., 2009)
• Major findings:
• Average SUS score was 81.5, a rating between “good” and “excellent” and corresponding to a “B” grade, which is a very high level of usability
• Identification of several usability issues:
• Inadequate positioning of VIBE on the online portal
• Too many recommendation results
• Too little information displayed in the recommendation results
9
EC-Web - September 2014, Munich, Germany
Outline
10
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions, Lessons Learned and F
• Context-Aware Recommender Systems and their Challenges
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Welcome screen
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Registration screen
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Personality questionnaire
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Questionnaire results
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Active learning
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Suggestions screen
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Context settings
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Details screen
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Rating dialog
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Routing screen
EC-Web - September 2014, Munich, Germany
Interaction with the STS System
11
Bookmarked items screen
EC-Web - September 2014, Munich, Germany
Software Architecture and Implementation
12
Android Client
Spring Dispatcher Servlet Spring Controllers
Apache Tomcat Server
Service / Application Layer
JPA Entities Hibernate
Objects managed by Spring IoC Container
Database
JSON HTTP
Web Services
EC-Web - September 2014, Munich, Germany
Software Architecture and Implementation
12
Android Client
Spring Dispatcher Servlet Spring Controllers
Apache Tomcat Server
Service / Application Layer
JPA Entities Hibernate
Objects managed by Spring IoC Container
Database
JSON HTTP
Web Services
EC-Web - September 2014, Munich, Germany
Software Architecture and Implementation
12
Android Client
Spring Dispatcher Servlet Spring Controllers
Apache Tomcat Server
Service / Application Layer
JPA Entities Hibernate
Objects managed by Spring IoC Container
Database
JSON HTTP
Web Services
EC-Web - September 2014, Munich, Germany
Software Architecture and Implementation
12
Android Client
Spring Dispatcher Servlet Spring Controllers
Apache Tomcat Server
Service / Application Layer
JPA Entities Hibernate
Objects managed by Spring IoC Container
Database
JSON HTTP
Web Services
EC-Web - September 2014, Munich, Germany
Software Architecture and Implementation
12
Android Client
Spring Dispatcher Servlet Spring Controllers
Apache Tomcat Server
Service / Application Layer
JPA Entities Hibernate
Objects managed by Spring IoC Container
Database
JSON HTTP
Web Services
EC-Web - September 2014, Munich, Germany
Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF) (Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is available
13
r̂uic1,...,ck = i + bu + bicjj=1
k
∑ + qiT ⋅(pu + ya
a∈A(u )∑ )
ī average rating for item ibu baseline for user ubicj baseline for item i and contextual condition cjqi latent factor vector of item ipu latent factor vector of user uA(u) set of user attributesya latent factor vector of user attribute a
EC-Web - September 2014, Munich, Germany
Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF) (Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is available
13
r̂uic1,...,ck = i + bu + bicjj=1
k
∑ + qiT ⋅(pu + ya
a∈A(u )∑ )
ī average rating for item ibu baseline for user ubicj baseline for item i and contextual condition cjqi latent factor vector of item ipu latent factor vector of user uA(u) set of user attributesya latent factor vector of user attribute a
EC-Web - September 2014, Munich, Germany
Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF) (Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is available
13
r̂uic1,...,ck = i + bu + bicjj=1
k
∑ + qiT ⋅(pu + ya
a∈A(u )∑ )
ī average rating for item ibu baseline for user ubicj baseline for item i and contextual condition cjqi latent factor vector of item ipu latent factor vector of user uA(u) set of user attributesya latent factor vector of user attribute a
EC-Web - September 2014, Munich, Germany
Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF) (Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is available
13
r̂uic1,...,ck = i + bu + bicjj=1
k
∑ + qiT ⋅(pu + ya
a∈A(u )∑ )
ī average rating for item ibu baseline for user ubicj baseline for item i and contextual condition cjqi latent factor vector of item ipu latent factor vector of user uA(u) set of user attributesya latent factor vector of user attribute a
EC-Web - September 2014, Munich, Germany
Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF) (Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is available
13
r̂uic1,...,ck = i + bu + bicjj=1
k
∑ + qiT ⋅(pu + ya
a∈A(u )∑ )
ī average rating for item ibu baseline for user ubicj baseline for item i and contextual condition cjqi latent factor vector of item ipu latent factor vector of user uA(u) set of user attributesya latent factor vector of user attribute a
EC-Web - September 2014, Munich, Germany
Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF) (Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is available
13
r̂uic1,...,ck = i + bu + bicjj=1
k
∑ + qiT ⋅(pu + ya
a∈A(u )∑ )
ī average rating for item ibu baseline for user ubicj baseline for item i and contextual condition cjqi latent factor vector of item ipu latent factor vector of user uA(u) set of user attributesya latent factor vector of user attribute a
EC-Web - September 2014, Munich, Germany
Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF) (Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is available
13
r̂uic1,...,ck = i + bu + bicjj=1
k
∑ + qiT ⋅(pu + ya
a∈A(u )∑ )
ī average rating for item ibu baseline for user ubicj baseline for item i and contextual condition cjqi latent factor vector of item ipu latent factor vector of user uA(u) set of user attributesya latent factor vector of user attribute a
new
EC-Web - September 2014, Munich, Germany
Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF) (Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is available
13
r̂uic1,...,ck = i + bu + bicjj=1
k
∑ + qiT ⋅(pu + ya
a∈A(u )∑ )
ī average rating for item ibu baseline for user ubicj baseline for item i and contextual condition cjqi latent factor vector of item ipu latent factor vector of user uA(u) set of user attributesya latent factor vector of user attribute a
EC-Web - September 2014, Munich, Germany
Outline
14
• Context-Aware Recommender Systems and their Challenges
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions, Lessons Learned and Future Work
EC-Web - September 2014, Munich, Germany
Experimental Methodology
• Live user study where we compared our system (STS) with a variant (STS-S) that has the same graphical UI but does not use the weather context when generating recommendations
• We have designed a specific user task and used a questionnaire for assessing the perceived recommendation quality (Knijnenburg et al., 2012) and system usability with the System Usability Scale (SUS) (Brooke, 1996)
• 30 subjects that were randomly divided in two equal groups assigned to STS and STS-S (15 each)
15
EC-Web - September 2014, Munich, Germany
User Task
• Users were supposed to:
• have an afternoon off and to look for attractions / events in South Tyrol
• consider the contextual conditions relevant for them and to specify them in the system settings
• browse the attractions / events sections and check whether they could find something interesting for them
• browse the system suggestions (recommendations), and select and bookmark the one that they believed fits their preferences
• fill out a survey on recommendation quality and system usability
16
EC-Web - September 2014, Munich, Germany
Results (1/3)
Box-and-whisker plot of the SUS points for each statement given by all users
17
S1 I think that I would like to use this system frequently.
S2 I found the system unnecessarily complex.S3 I thought the system was easy to use.
S4 I think that I would need the support of a technical person to be able to use this system.
S5 I found the various functions in this system were well integrated
S6 I thought there was too much inconsistency in this system.
S7 I would imagine that most people would learn to use this system very quickly.
S8 I found the system very cumbersome to use.
S9 I felt very confident using the system.S10 I needed to learn a lot of things before I
could get going with this system.
EC-Web - September 2014, Munich, Germany
SUS scores for all users
SUS
scor
e
50
55
60
65
70
75
80
85
90
Users1 2 3 4 5 6 7 8 9 10 11 12 12 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Benchmark Average
Results (2/3)
18
EC-Web - September 2014, Munich, Germany
Results (3/3)
Comparison of the SUS scores for STS and STS-S users
19
Statement STS STS-S p-value
S1 I think that I would like to use this system frequently. 3.0 3.2 0.27
S2 I found the system unnecessarily complex. 3.2 3.5 0.16S3 I thought the system was easy to use. 3.1 2.8 0.18S4 I think that I would need the support of a technical person to
be able to use this system.3.3 3.4 0.40
S5 I found the various functions in this system were well integrated 3.1 2.8 0.14
S6 I thought there was too much inconsistency in this system.
3.2 2.8 0.08
S7 I would imagine that most people would learn to use this system very quickly.
2.8 3.0 0.25
S8 I found the system very cumbersome to use. 3.4 3.1 0.19
S9 I felt very confident using the system. 2.7 2.8 0.40S10 I needed to learn a lot of things before I could get going
with this system.3.4 3.1 0.11
Overall SUS 78.8 77.0 0.19
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (1/3)
• Five-Item Personality Inventory (FIPI)
• We replaced the Ten-Item Personality Inventory (TIPI) with the Five-Item Personality Inventory (FIPI), which is less time-consuming and still provides sufficient personality data.
• Built-in help
• Users can click the “?” icon next to each questionnaire question to access on-screen help with term definitions.
20
…Before
…After
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (1/3)
• Five-Item Personality Inventory (FIPI)
• We replaced the Ten-Item Personality Inventory (TIPI) with the Five-Item Personality Inventory (FIPI), which is less time-consuming and still provides sufficient personality data.
• Built-in help
• Users can click the “?” icon next to each questionnaire question to access on-screen help with term definitions.
20
…Before
…After
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (1/3)
• Five-Item Personality Inventory (FIPI)
• We replaced the Ten-Item Personality Inventory (TIPI) with the Five-Item Personality Inventory (FIPI), which is less time-consuming and still provides sufficient personality data.
• Built-in help
• Users can click the “?” icon next to each questionnaire question to access on-screen help with term definitions.
20
…Before
…After
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (1/3)
• Five-Item Personality Inventory (FIPI)
• We replaced the Ten-Item Personality Inventory (TIPI) with the Five-Item Personality Inventory (FIPI), which is less time-consuming and still provides sufficient personality data.
• Built-in help
• Users can click the “?” icon next to each questionnaire question to access on-screen help with term definitions.
20
…Before
…After
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (1/3)
• Five-Item Personality Inventory (FIPI)
• We replaced the Ten-Item Personality Inventory (TIPI) with the Five-Item Personality Inventory (FIPI), which is less time-consuming and still provides sufficient personality data.
• Built-in help
• Users can click the “?” icon next to each questionnaire question to access on-screen help with term definitions.
20
…Before
…After
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the lengthy AL process during registration, we give them freedom to decide when to initiate it through in-app notifications within the POI suggestions screen.
• User profile page
• We implemented a new user profile page, making it easier to access and change context settings, basic user information, personality information, etc.
21
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the lengthy AL process during registration, we give them freedom to decide when to initiate it through in-app notifications within the POI suggestions screen.
• User profile page
• We implemented a new user profile page, making it easier to access and change context settings, basic user information, personality information, etc.
21
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the lengthy AL process during registration, we give them freedom to decide when to initiate it through in-app notifications within the POI suggestions screen.
• User profile page
• We implemented a new user profile page, making it easier to access and change context settings, basic user information, personality information, etc.
21
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the lengthy AL process during registration, we give them freedom to decide when to initiate it through in-app notifications within the POI suggestions screen.
• User profile page
• We implemented a new user profile page, making it easier to access and change context settings, basic user information, personality information, etc.
21
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the lengthy AL process during registration, we give them freedom to decide when to initiate it through in-app notifications within the POI suggestions screen.
• User profile page
• We implemented a new user profile page, making it easier to access and change context settings, basic user information, personality information, etc.
21
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the lengthy AL process during registration, we give them freedom to decide when to initiate it through in-app notifications within the POI suggestions screen.
• User profile page
• We implemented a new user profile page, making it easier to access and change context settings, basic user information, personality information, etc.
21
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the lengthy AL process during registration, we give them freedom to decide when to initiate it through in-app notifications within the POI suggestions screen.
• User profile page
• We implemented a new user profile page, making it easier to access and change context settings, basic user information, personality information, etc.
21
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (3/3)
• Many other minor UI improvements
• Revised the contextual factors and contextual conditions
• Improved the UI for displaying personality questionnaire results
• Cleaned up the POI details screen
22AfterBefore AfterBefore
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (3/3)
• Many other minor UI improvements
• Revised the contextual factors and contextual conditions
• Improved the UI for displaying personality questionnaire results
• Cleaned up the POI details screen
22AfterBefore AfterBefore
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (3/3)
• Many other minor UI improvements
• Revised the contextual factors and contextual conditions
• Improved the UI for displaying personality questionnaire results
• Cleaned up the POI details screen
22AfterBefore AfterBefore
EC-Web - September 2014, Munich, Germany
Corrective Actions Based on the Results (3/3)
• Many other minor UI improvements
• Revised the contextual factors and contextual conditions
• Improved the UI for displaying personality questionnaire results
• Cleaned up the POI details screen
22AfterBefore AfterBefore
EC-Web - September 2014, Munich, Germany
Outline
23
• Context-Aware Recommender Systems and their Challenges
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions, Lessons Learned and Future Work
EC-Web - September 2014, Munich, Germany
Conclusions
• Novel and highly usable mobile CARS called STS (South Tyrol Suggests) that offers various innovative features
• Learns users’ preferences not only using their past ratings, but also exploiting their personality
• Uses personality to actively acquire ratings for POIs the user has likely experienced, and to produce more accurate POI recommendations
• Live user study to test the usability of STS
• Results confirm high usability of the proposed system
• Allowed to uncover and resolve some usability issues, such as moderate confidence in the system and poor integration of some features
24
EC-Web - September 2014, Munich, Germany
Lessons Learned
• Only ask users for the minimum required information
• The more information you ask of users, the less likely they will provide it
• Make the system as simple as possible to use
• Keep the system as simple as possible and provide useful on-screen help or tutorials to instruct users on how to get things done
• Give users control over the system
• Instead of telling users how to use the user interface, give them the ability to control where they go and what they do. Moreover, always ensure that the user knows what things are and what they will do
25
EC-Web - September 2014, Munich, Germany
Future Work
• Evaluate the usability of the revised user interface
• Provide users with proactive recommendations and rating requests
• Consider additional important contextual factors in the recommendation process (e.g., parking availability, traffic conditions)
• Improve explanations to make the recommendation process more transparent to users
26
EC-Web - September 2014, Munich, Germany
Questions?
Thank you.
