design of a hybrid recommender system: a study of the cold

Design of a Hybrid Recommender System: A Study of the Cold-Start User Problem

Sigurd Støen LundØystein Tandberg

Master of Science in Informatics

Supervisor: Helge Langseth, IDICo-supervisor: Professor Agnar Aamodt, IDI

Department of Computer and Information Science

Submission date: May 2015

Norwegian University of Science and Technology

i

Abstract

Recommender systems are used to help users discover the items they mightbe interested in, especially when the number of alternatives is big. In mod-ern streaming websites for music, movies, and TV-shows, E-commerce, socialnetworks, and more, recommender systems are widely used. These recom-mender systems are often looking at the ratings on items for the current andother users, and predicting a rating on the items the user have not seen.Others match the content of an item itself against a user profile. A mix ofthe two is often used to make the predictions more accurate, and this can alsohelp to the problem when a new user sign up where we have no knowledgeabout him. This issue, is a well-known problem for recommender systemsoften described as the cold-start problem, and much research has been doneto find the best way to overcome this.

In this thesis, we look at previous approaches to recommender systems andthe cold-start problem in particular. We have developed our application,Eatelligent, which is recommending dinner recipes based on our study ofprevious research. Eatelligent has been designed to examine how we canapproach the cold-start problem e�ciently in a real world application, andwhat kind of feedback we can collect from the users.

iii

Sammendrag

Anbefalingssystemer brukes til a hjelpe brukere a oppdage ting de kan væreinteressert i, og spesielt nar det finnes mange alternativer a velge fra. Imoderne strømningstjenester for musikk, tv-serier og film, eller nettbutikkerog sosiale medier, er anbefalingssystemer mye brukt. Disse anbefalingssys-temene ser ofte pa historikk i form av vurderinger for en bruker, og prediktereren vurdering pa objektene brukeren ikke har vurdert selv. Andre systemersammenlikner innholdet av et objekt mot brukerens profil. Ofte brukes enkombinasjon av disse to metodene for a gjøre prediksjonene enda mer nøyak-tige. Dette kan ogsa hjelpe til a løse utfordringen med nye brukere som manikke har noen kunnskap om. Dette problemet omtales ofte som kaldstart-problemet, og mye arbeid er lagt ned tidligere for a overkomme dette.

I denne masteroppgaven ser vi pa tidligere tilnærminger for anbefalingssyste-mer og kaldstart-problemet spesielt. Vi har laget var egen applikasjon, Eatel-ligent, som er en middagsoppskrift-anbefaler basert pa vart studie av tidligerearbeid. Eatelligent er designet for a utforske hvordan vi kan tilnærme osskaldstart-problemet e↵ektivt i en reel applikasjon, og hvilken type tilbake-meldinger vi kan samle fra brukerne for a overkomme dette.

v

Preface

This Master’s thesis is written by Sigurd Støen Lund and Øystein Tandbergfrom August 2014 to June 2015 at the Norwegian University of Science andTechnology (NTNU). The thesis is the final delivery of the master programin Informatics, with specialization in intelligent systems.

We came up with the idea for the thesis ourselves, wanting to build somethingrelated to recommender systems in a real world application. Our supervisorsliked the idea and helped us making this into a thesis.

We would like to thank our supervisors, Helge Langseth and Agnar Aamodt,for constructive feedback and discussions, and for always being in pleasantmood throughout the year. We would also like to thank the fellow studentsin room 363, Korp, for a good working atmosphere, along with the otherstudents taking lunch breaks on the second floor.

We would also like to thank Matprat AS for letting us use their recipes inour application.

Trondheim, Friday 29th May, 2015

Sigurd Støen Lund Øystein Tandberg

Contents

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Sammendrag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

1 Introduction and Overview 3

1.1 Background and Motivation . . . . . . . . . . . . . . . . . . . 3

1.2 Goals and Research Objectives . . . . . . . . . . . . . . . . . . 4

1.3 Thesis Structure . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.4 Project Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Background Theory and Motivation 7

2.1 Recommender Systems . . . . . . . . . . . . . . . . . . . . . . 7

2.1.1 Content-Based Filtering . . . . . . . . . . . . . . . . . 8

2.1.2 Collaborative Filtering . . . . . . . . . . . . . . . . . . 9

2.1.3 Hybrid Recommender Systems . . . . . . . . . . . . . . 11

2.1.4 Context-Aware Recommender Systems . . . . . . . . . 12

2.2 Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.2.1 Explicit Feedback . . . . . . . . . . . . . . . . . . . . . 16

2.2.2 Implicit Feedback . . . . . . . . . . . . . . . . . . . . . 17

2.3 The Cold-Start Problem . . . . . . . . . . . . . . . . . . . . . 19

2.3.1 Cold-Start User Problem . . . . . . . . . . . . . . . . . 19

vii

viii CONTENTS

2.3.2 Cold-Start Item Problem . . . . . . . . . . . . . . . . . 23

2.4 Case-Based Reasoning . . . . . . . . . . . . . . . . . . . . . . 23

2.4.1 The CBR Principles . . . . . . . . . . . . . . . . . . . 24

2.4.2 CBR as a Recommender System . . . . . . . . . . . . . 26

2.5 Examples of Earlier Projects . . . . . . . . . . . . . . . . . . . 27

2.5.1 Amazon.com . . . . . . . . . . . . . . . . . . . . . . . 27

2.5.2 DieToRecs . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.5.3 CHEF . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.5.4 JULIA . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.5.5 Intelligent Food Planning: Personalized Recipe Rec-ommendation . . . . . . . . . . . . . . . . . . . . . . . 30

2.5.6 Food Recommendation Using Ontology and Heuristics 30

2.6 Relating the Theory Towards Eatelligent . . . . . . . . . . . . 31

2.6.1 Recommender System . . . . . . . . . . . . . . . . . . 31

2.6.2 Cold-Start . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.6.3 Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.6.4 Taking Advantage of the Contextual Information . . . 34

3 Architecture/Model 37

3.1 Requirements for Design . . . . . . . . . . . . . . . . . . . . . 37

3.2 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.3.1 Client . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.3.2 Backend . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.4 Recommender System . . . . . . . . . . . . . . . . . . . . . . 51

3.4.1 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . 52

3.4.2 Feedback in the Client . . . . . . . . . . . . . . . . . . 58

3.4.3 Cold-Start Item Problem . . . . . . . . . . . . . . . . . 59

CONTENTS ix

3.4.4 Composition of Recommendations . . . . . . . . . . . . 59

3.5 How it Works . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

3.5.1 A New User Receiving a List of Recommendations . . . 60

3.6 Making Eatelligent Public . . . . . . . . . . . . . . . . . . . . 65

4 Experiments and Results 67

4.1 Experimental Plan . . . . . . . . . . . . . . . . . . . . . . . . 67

4.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 68

4.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 69

4.3.1 Yes/No ratings . . . . . . . . . . . . . . . . . . . . . . 69

4.3.2 Star Ratings . . . . . . . . . . . . . . . . . . . . . . . . 70

4.3.3 Comparing Yes/No and Star Ratings . . . . . . . . . . 73

4.3.4 Cold-Start Questions . . . . . . . . . . . . . . . . . . . 73

5 Evaluation and Conclusion 77

5.1 Evaluation and Discussion . . . . . . . . . . . . . . . . . . . . 77

5.1.1 RO1 - Gain an understanding of what it means to rec-ommend food recipes . . . . . . . . . . . . . . . . . . . 77

5.1.2 RO2 - Study and evaluate di↵erent strategies to rec-ommend items for a new user using both collaborativeand content-based approaches . . . . . . . . . . . . . . 78

5.1.3 RO3 - Study existing solutions to the cold-start problem 79

5.1.4 RO4 - Study what kind of data we can collect from theapplication used by a user over a period of time thatis relevant to build more knowledge about the user . . 79

5.1.5 RO5 - Study the challenges that arise when buildingan application with a recommender system from scratch 80

5.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

5.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

x CONTENTS

A SQL-scheme 85

B Endpoints 91

B.1 GET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

B.2 PUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

B.3 POST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

B.4 DELETE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

C Recipe Features 95

C.1 Ingredient Tags . . . . . . . . . . . . . . . . . . . . . . . . . . 95

C.2 Recipe Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

C.3 Full Recipe Description . . . . . . . . . . . . . . . . . . . . . . 97

C.3.1 Paprika Chicken with Asparagus . . . . . . . . . . . . 97

C.3.2 Babi Asam Manis . . . . . . . . . . . . . . . . . . . . . 98

C.3.3 Asian Omelet . . . . . . . . . . . . . . . . . . . . . . . 99

Bibliography 100

List of Figures

2.1 We want to learn the function between the user and the itemshe like the most . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2 Multidimensional model for the User ⇥ Item ⇥ Time recom-mendation space [5] . . . . . . . . . . . . . . . . . . . . . . . . 14

2.3 The CBR cycle . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.4 The mapping from problem to solution. The dotted arrowshows a new case and its solution adapted from the previouslysolved cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.1 System overview . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.2 Class hierarchy for the mobile client . . . . . . . . . . . . . . . 41

3.3 Entity relation diagram . . . . . . . . . . . . . . . . . . . . . . 44

3.4 The sign up process . . . . . . . . . . . . . . . . . . . . . . . . 47

3.5 The user answer 1/5 questions . . . . . . . . . . . . . . . . . 48

3.6 The users answers “yes” or “no” to a recommendation . . . . . 49

3.7 Detailed view of the recipe . . . . . . . . . . . . . . . . . . . . 50

3.8 Rating the recipe . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.9 Illustration of the recipe tag tree . . . . . . . . . . . . . . . . 57

3.10 Illustration of the ingredient tag tree . . . . . . . . . . . . . . 57

4.1 “Yes/no” ratings per algorithm . . . . . . . . . . . . . . . . . . 69

4.2 Normalized “Yes/no” ratings per algorithm . . . . . . . . . . . 70

xi

xii LIST OF FIGURES

4.3 Star ratings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

4.4 Percentage of star ratings for each algorithm . . . . . . . . . . 72

4.5 Comparison between yes/no and star ratings. The data foreach recipe corresponds vertically between the two charts. Inthe bottom chart, the impulses represent the number of rat-ings, and the bars show the average star rating. . . . . . . . . 74

List of Tables

2.1 The goal is to predict the missing votes for the active user,e.g., User B . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2 Common types of explicit and implicit feedback . . . . . . . . 17

2.3 Evaluation of di↵erent strategies from A. M. Rashid et al. . . 22

3.1 The di↵erent response codes used by the server . . . . . . . . 46

3.2 The weighting from the cold-start questions . . . . . . . . . . 53

3.3 The impact of star ratings . . . . . . . . . . . . . . . . . . . . 55

3.4 An example on a composed recommendation list . . . . . . . . 60

3.5 The current state of the weights after the cold-start questions. 61

3.6 The current state of the weights after the “yes/no” ratings . . 62

3.7 The updated weights after a rating of “Paprika Chicken withasparagus” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4.1 The number of ratings for each algorithm . . . . . . . . . . . . 71

4.2 Answers to the cold-start questions . . . . . . . . . . . . . . . 75

4.3 Recommendations made by each algorithm based on cold-startquestions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

xiii

xiv LIST OF TABLES

List of Algorithms

1 Nearest User Rated Recipes . . . . . . . . . . . . . . . . . . . . 54

2 Content-based filtering . . . . . . . . . . . . . . . . . . . . . . . 56

xv

xvi LIST OF ALGORITHMS

Listings

2.1 The KNN algorithm . . . . . . . . . . . . . . . . . . . . . . . 253.1 Sign up request . . . . . . . . . . . . . . . . . . . . . . . . . . 473.2 Sign up response . . . . . . . . . . . . . . . . . . . . . . . . . 473.3 Cold-Start Request . . . . . . . . . . . . . . . . . . . . . . . . 483.4 Cold-Start Response . . . . . . . . . . . . . . . . . . . . . . . 483.5 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . 493.6 Recipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.7 Rating request . . . . . . . . . . . . . . . . . . . . . . . . . . . 513.8 Rating response . . . . . . . . . . . . . . . . . . . . . . . . . . 513.9 First list of recommendations to a user . . . . . . . . . . . . . 633.10 New recommendation after interactions . . . . . . . . . . . . . 64

1

2 LISTINGS

Chapter 1

Introduction and Overview

Recommender systems help people to make better decisions in a world filledwith choices. Since the recommender systems in most cases are targetedtowards individuals, it is a problem to recommend items to a new user youbarely know anything about, or a new item you do not know which typeof user will like. This is known as the cold-start problem. In this thesis,we will study di↵erent approaches to the cold-start problem in a problemdomain where we have no knowledge about new users. We will also studyhow feedback can be collected, to evaluate the recommender engine, and usethe feedback for better recommendations in the future.

By exploring this in a real world application we develop on our own, we getthe opportunity to be more flexible about how we structure our data, whattype of knowledge we try to extract from the user, and what feedback wecollect while the user uses the system. A well-suited domain for this researchis the recommendation of food recipes, which will be implemented as anapplication from now called Eatelligent.

1.1 Background and Motivation

Composing recommendations of items to a user is usually done by one of twostrategies, or both of them combined. Either by looking at the content of theitems itself and matching it against a user profile, or predicting a rating basedon other users ratings on the items. Both of the strategies have challenges

3

4 CHAPTER 1. INTRODUCTION AND OVERVIEW

relating to the cold-start problem, and a combination of the two is often usedto overcome some of these. In this thesis, we want to try combinations ofdi↵erent methods, and see how these perform on new users.

To build knowledge about a new user, we want to study di↵erent methods tocollect data about how the user is using the application and ask the user toadd additional information. We also want to design, structure, and describethe data ourselves, to be able to build knowledge about the items and usersin an e�cient way. The only way we are free to do all this without anyrestrictions is if we develop our own application. Many datasets exist to beused for testing recommender systems, but those datasets contain historicaldata, typical user ratings on items like books or movies. By building a realworld application, we can collect both implicit and explicit ratings, look atthe content of the items ourselves and therefore be more flexible in the designof the recommender system.

A good test domain would be something where you start with no knowledgeabout the user and have to extract some knowledge and turn it into predic-tions. Thinking of a domain where people make choices on a regular basisand there are a lot of options out there that takes some e↵ort for a person toexplore, we thought of what people are deciding to make for dinner. Thereexist a lot of websites that contains food recipes, but they are not necessarilyhelping the user to make an easy choice of what to go for without having tobrowse through a lot of recipes. By implementing an application with dinnerrecipes along with a recommender system, we could help people save time ona daily basis. It is also hard to know what kind of food an unknown personlikes, so it gives us a good domain for studying the cold-start problem. In aworld where almost everybody have smartphones and more and more of theInternet activity is happening on those devices, we decided to go for a mobileapplication, to test the software as a real world application.

1.2 Goals and Research Objectives

The goal for this thesis is to study methods for a recommender system ina real world problem situation, explore the challenges with recommendingitems to new users, and implement and evaluate di↵erent strategies to solvethis problem. Di↵erent recommender strategies will be tested to see whichone is most accurate. The di↵erent methods will be assessed by star ratings

1.3. THESIS STRUCTURE 5

from the users, and also by looking at which recipes the user looks at indetail. Some of the recommended recipes will be drawn from a uniformrandom distribution to be used as a baseline to compare the other methods.

Below are the research objectives for this thesis:

• RO1 Gain an understanding of what it means to recommend foodrecipes.

• RO2 Study and evaluate di↵erent strategies to recommend items for anew user using both collaborative and content-based approaches.

• RO3 Study existing solutions to the cold-start problem.

• RO4 Study what kind of data we can collect from the application usedby a user over a period of time that is relevant to build more knowledgeabout the user.

• RO5 Study the challenges that arise when building an application witha recommender system from scratch.

1.3 Thesis Structure

Chapter 2 describes the background material to cover the most relevant the-ory related to recommender systems and the cold-start problem, and materialthat is either considered or used in our application. In Chapter 3 we describethe design and implementation of the whole application, which is everythingfrom how the server and client are structured to how we approach the rec-ommendation and cold-start problem. Chapter 4 presents the results of ourwork, and in Chapter 5 we evaluate how this solution worked along withsuggestions for future work.

1.4 Project Scope

The scope for this thesis is to study the nature of the cold-start problemwithin the described problem domain and investigate the results of our ap-proach to solve this problem, by building an application for recommendation

6 CHAPTER 1. INTRODUCTION AND OVERVIEW

of dinner recipes. This also includes studying previous research, to make surewe build this application while standing on the shoulders of the giants.

Since this application is built from scratch, evaluate the recommender sys-tem by the data we collect throughout our test period. The application isreleased on the open market so that the user can be any user with access toa smartphone. Eatelligent is available for free in App Store for iPhone usersand Google Play for Android users.

Chapter 2

Background Theory andMotivation

In this chapter, we will present the subjects that will form the backgroundfor this thesis. The chapter covers research regarding recommender systemsand its features from early stages up until recent studies. We start o↵ by de-scribing recommender systems in general, and the di↵erent approaches thatare out there. Then we write about feedback followed by the cold-start prob-lem, which both are relevant challenges in recommender systems, and thatwe will study deeper in this thesis. Case-based reasoning (CBR) is a way ofproblem-solving and can also be used as a strategy in a recommender sys-tem. Section 2.4 is therefore dedicated to this methodology. In Section 2.5we describe some relevant projects to recommender systems and food rec-ommendation in particular. The chapter ends with Section 2.6 relating thetheory described earlier towards Eatelligent.

2.1 Recommender Systems

Recommender systems suggest items that should interest the user. The rec-ommendations are usually based on knowledge about the user, either it isa user profile with preferences or history. Recommendations of items likemovies, books, and music widely use this strategy. It is usually targeted toindividuals and used when the number of alternatives is big.

7

8 CHAPTER 2. BACKGROUND THEORY AND MOTIVATION

Figure 2.1: We want to learn the function between the user and the items helike the most

In this section, we will discuss the di↵erent approaches to recommender sys-tems that are common today.

The key task for every di↵erent recommender system can vary. One can beto sort a list of items in a personalized fashion. Another can be to pick apersonalized top 10 list from a large item set. Recommender systems alsohelp users widen their horizon by recommending new items, based on similarusers or similar items the user liked before. Figure 2.1 illustrates the functionbetween the user and items that a recommender engine want to learn. For anaive recommender system, it is an easy task to recommend the most popularitems to all users. In this naive approach, the result is not personalized.

2.1.1 Content-Based Filtering

Content-based filtering methods analyzes the items itself and match themagainst a user profile [39]. The representation of an item is important incontent-based recommender systems since the recommendation techniquemight be based on the representation. In a structured way, attributes liketags or categories can easily be stored, so it is easy to compute from them.Unrestricted text is more complicated, and by counting words and representtheir importance, an unrestricted text can be transformed into structureddata. This process is often done by representing a weight for each term TF-IDF (term frequency times inverse document frequency). Before this is doneit is common to use techniques like stemming, which means to represent dif-ferent versions of a word as one term. For instance, “comput” can representthe terms “computation” and “computers”.

Since the items are going to be matched to a user, the recommender systemneeds to have a way of computing a score for a user on the items. Pazzaniet al. [39] describe two types of user profiles: (1) The user’s preferences arestored, e.g. in a food recommender a user might have “vegan” attached to hisprofile, to describe that he wants vegan dishes. Preferences like this could

2.1. RECOMMENDER SYSTEMS 9

be collected by having the user fill out a profile with checkboxes. (2) Thehistory of the user interacting with the system. This information might behistory like which items he has looked at before, ratings, or queries. Thisfeedback can either be collected implicitly or explicitly.

Many classification learning algorithms can be used to learn the functionthat computes an estimate of the probability that a user likes an item, likeNaive Bayes along with many others. We will not go into detail with thesealgorithms here.

2.1.2 Collaborative Filtering

Collaborative filtering is a recommender approach based on a collection of theuser’s data like ratings, behaviors, and preferences, as well as analyzing thesedata and recommend items to a similar user. The task can be described as“To predict the utility of items to a particular user (the active user) based ona database of user votes from a sample or population of other users (the userdatabase)” [11]. Collaborative filtering can be in di↵erent forms; memory-based or model-based methods.

Inmemory-based algorithms, we predict the votes of the active user basedon some partial information regarding the active user and a set of weightscalculated from the user database. The user database contains a set of votesvu,i, meaning the vote for user u on item i. If Iu is the set of items on whichuser u has voted, then the mean vote for user u is as follows:

vu =1

|Iu|X

i2Iu

vu,i (2.1)

In memory-based algorithms, we predict the votes of the active user basedon partial information regarding the active user and a set of weights fromthe user database. The assumed predicted vote for the active user u for itemi, pu,i, is a weighted sum of the votes of the users:

Pu,i = vu + kX

u02U

w(u, u0)(vu,u0 � vu0) (2.2)

Where u is the current user, U is the set of users in the database with nonzeroweights. The weights, w(u, u0), can describe the correlation or similarity


Item A Item B Item C Item D

User A 1 5

User B 1 ? 3 ?

User C 2 4 5

Table 2.1: The goal is to predict the missing votes for the active user, e.g.,User B

between each user in the database and the active user. k is for normalization.In this thesis, we will cover the correlation weighting model. The correlationbetween two users u and u0 is defined as follows [43]:

w(u, u0) =

Pi(vu,i � vu)(vu0,i � vu0)pP

i(vu,i � vu)2P

i(vu0,i � vu0)2(2.3)

Where i is each item for which both user u and u0 have recorded votes.

Model-based algorithms is using a probabilistic approach where the col-laborative filtering task can be viewed as calculating the expected value ofa vote, given what we know about the user. For the active user, we wantto predict the votes for unobserved items. The probability expression for anitem i and user u with the rated items for user u is Iu is as follows (Thisassumes that the votes are integers in range from 0 to m):

pu,i = E(vu,i) =mX

j=0

Pr(vu,i = j|vuk, k 2 Iu)i (2.4)

The probability, Pr, in Equation 2.4 is the probability that the given userwill rate a value of the input item, given the previously rated items. Bayesiannetworks or clustering models is two of the probabilistic models that can beused for model-based collaborative filtering.

Collaborative filtering helps with predicting ratings for an active user basedon previous votes in a database of votes. The input data is very often asparse matrix of votes for di↵erent items. The example in Table 2.1 is asmall example showing ratings for some users, and where the data is sparse.


2.1.3 Hybrid Recommender Systems

A hybrid recommender system is used to help avoid certain limitations ofcontent-based- or collaborative filtering methods. Collaborative methodshave a relatively big weakness around the cold-start problem since the userneeds to have rated some items to receive recommendations. A real-life prob-lem domain is much more complicated than a straight movie recommendersystem [4]. A real business case will have significant amounts of parame-ters, where a hybrid will help learning a more complex model of the user,hence giving more accurate recommendations. Hybrid solutions are a mixof the methods described earlier. Since di↵erent methods can be combined,multiple hybrid solutions can be created. Burke [13] covers seven di↵erentapproaches to combining recommenders/components:

Weighted hybrid recommender systems go through a training phase wherethe weights in the recommender system are learned from a training set duringa training period. The output of this training phase is the weights of eachcomponent. The candidates from each component are weighted in respectof these numeric values each time a recommendation is done. The result isthen either a union or intersection of these lists. This approach does notdi↵erentiate between users; there is an assumption that each component willhave a consistent performance for all users.

Mixed hybrid recommenders present a composed list of items side-by-sidefrom each of the components. There is no actual combining of the di↵erentrecommender algorithms. The challenge in mixed hybrid is the ordering andconfidence in each of the components composed together. A technique oftenincludes merging the lists on a normalization the predicted rating.

A switching hybrid recommender have a confidence in each component andselects the recommendation that gives the best results at the given time. Thismethod, similar to the mixed approach, does not combine the algorithms intoa new one. A substantial part of a switching hybrid is the selection criteria.Both statistical confidence and external criteria can be used to select whichrecommender to select in each user case.

Cascading hybrids tries to create a strict hierarchy of recommender com-ponents. The di↵erent components are ranked, and the secondary ones fill infor the primary recommender when there is a broken tie.

Meta-level hybrid recommender systems are first going through a training


phase. This training phase is to train one recommender to have a concept ofa learned model. The actual recommender then uses this model to producerecommendations for the end user. Not all pairs of di↵erent recommendationcomponents are suitable for this hybrid.

Feature combination is a hybrid where one of the components uses itsoutput as input to the actual recommender. For instance, the output froma collaborative filtering algorithm can be used as input to a content-basedalgorithm. This method achieves that the first component is a contributorgiving a bias to the algorithm giving recommended list, every time.

Feature augmentation is a strategy where one component is used to pro-duce a new augmented feature set to the next component. This approachalso needs a training phase that learns the augmented profile. Feature aug-mentation can derive new content features at this stage that can be valuablewhen generating candidate items.

2.1.4 Context-Aware Recommender Systems

Most of the recommender systems do not consider the context of the userwhen finding the items to recommend [5]. However, it can often be useful toconsider the context that the user is in when giving recommendations. Forsome domains, various information like the day of a week and where the useris can have a significant impact on what the user would want.

What is a Context?

Before explaining context-aware recommending systems (CARS) any further,it is important to define what a context is. As G. Adomavicius and A.Tuzhilin [5] write, the term context is used in so many di↵erent disciplines,and the definition often vary across the di↵erent ones (computer science,linguistics, philosophy, psychology, and organizational sciences). They focuson the definitions in the fields that are most relevant to recommender systems,and we will just summarize them up here.

• In data mining, the context is sometimes defined as those events thatcharacterize the life stages of a customer and can determine a change inhis preferences, status, and value for a company [8]. This information


can be events like a new job, the birth of a child, marriage, divorce, orretirement.

• E-commerce Personalization systems sometimes look at the intent ofa purchase made by a customer as contextual information [36]. Forexample if someone buy something as a gift, he does not necessary wantrecommendations based on that when he visit again buying somethingfor himself.

• In ubiquitous and mobile context-aware systems, the context was ini-tially defined as the location of the user, the identity of people near theuser, the objects around, and the changes in these elements [48]. Date,season, temperature, physical and conceptual statuses of interest for auser, emotional statuses have been added later [12, 46, 19].

• In marketing and management research they have studied the di↵erentbehavior and decision making a customer can adopt and prefer di↵er-ent products based on the context [9, 33]. C. K. Prahalad [41] definescontext as “the precise physical location of a customer at any giventime, the exact minute he needs the service, and the kind of techno-logical mobile device over which that experience will be received”. Hedistinguishes among three dimensions: temporal (when to deliver cus-tomer experiences), spatial (where to deliver), and technological (howto deliver).

Modeling

A typical recommender system consists of users and items and is trying tolearn a function to estimate the rating for a user on a given item.

Rating = User⇥ Item (2.5)

When we want to take the context of the user into consideration as well, weend up with the following function instead:

Rating = User⇥ Item⇥ Context (2.6)


Figure 2.2: Multidimensional model for the User ⇥ Item ⇥ Time recommen-dation space [5]

If we use the example by G. Adomavicius and A. Tuzhilin [5] where theyconsider time as a context, we can model it in three dimensions as shown inFigure 2.2.

G. Adomavicius and A. Tuzhilin [5] describes two di↵erent approaches touse contextual information in a recommendation process; (1) recommenda-tion via context-driven querying and search, and (2) recommendation viacontextual preference elicitation and estimation. Mobile and tourist systems[2, 15, 49] use the first approach, by using the contextual information whenquerying for some information, like querying for restaurants in the restau-rants that are nearby.

The latter approach, recommendation via contextual preference elicitationand estimation, represents a more recent trend for context-aware recom-mender systems [3, 35, 37, 51]. This approach tries to model and learnthe user’s preferences by obtaining information while the user is using theapplication, and by watching how he interact with previously given recom-mendations. G. Adomavicius and A. Tuzhilin [5] describes three ways that


this recommendation process can be implemented:

• Contextual pre-filtering are done by selecting data filtered on the con-text, and then it can do a traditional 2D recommendation process onthe retrieved dataset.

• Contextual post-filtering is the opposite where the recommendationsare first predicted without considering the context, and filter on thecontext afterward.

• Contextual modeling is a technique where the contextual informationis used directly within the recommendation function. Instead of usinga traditional 2D approach either before or after as the first two ap-proaches, this will use a multidimensional model. Di↵erent algorithmshave been developed, but we will not go into further detail about thathere.

As we have talked about in previous sections, combining di↵erent recommen-dation techniques, it is possible to do the same within CARS, by combiningmultiple of the strategies described above.

Obtaining the Context

G. Adomavicius and A. Tuzhilin [5] describes three ways to obtain contextualinformation:

• By explicitly asking the user questions that help to define his currentcontext.

• Implicitly gather it from the data or environment, e.g. using the GPSon a mobile phone to determine the location of the user.

• Inferring the context using statistical or data mining methods. Forexample, we might be able to determine who is watching the TV in afamily of dad, mother, and son, based on what is being watched rightnow.


Other than knowing how to gather the information, we also need to decidewhich information to collect. G. Adomavicius et al. [3] propose that a widerange of attributes can be used, which are selected by domain experts thatthey believe can be useful information. After the data is collected, variousstatistical tests can find the most important attributes.

Issues

G. Adomavicius and A. Tuzhilin [5] also describes some issues with the cur-rent state of the research in context-aware recommender systems. More re-search need to be done on how to combine the di↵erent approaches and thetrade-o↵s for each of them. Adding the context to recommender systems alsoincreases the complexity, and so far most of the work has been conceptual.To be able to use this in real world applications with tons of data, moreresearch about problems like e�cient data structures and storage methodsneed to be done.

It is also a problem in the cases where the context is obtained by askingthe user. These questions will require more e↵ort from the user as well, andmost recommender systems want to retrieve as much relevant information aspossible without putting more stress on the user [14].

2.2 Feedback

Recommender systems are depending on feedback from the users, to build upknowledge and be able to have more accurate recommendations in the future.Feedback can be given in two forms, explicit and implicit feedback. In thissection, we will cover the di↵erences between those methods of gatheringfeedback from the users.

2.2.1 Explicit Feedback

Explicit feedback is an active action by the user on the system. This kindof feedback heavily depends on the willingness of the user to give feedback.Explicit ratings are often preferred over implicit feedback, because of theaccuracy this provides. D. Cosley et al. [16] found that explicit ratings can

2.2. FEEDBACK 17

Explicit feedback Implicit feedback

Rating (E.g. stars) Time tracking

Reviews GPS positions when a choice is made

Like or dislike buttons Keyboard/mouse inputs

Surveys Clicks

Eye tracking (lab)

Microphone input (lab)

Facial expression (lab)

Table 2.2: Common types of explicit and implicit feedback

be of di↵erent qualities. Users may rate movies less carefully than they ratebigger choices, e.g. a review after a holiday trip.

In a study by X. Amatriain et al. [7], they found that extreme explicitratings often are more consistent than mild opinions. The consequence isthat we cannot get a better resolution of the user preference model just byincreasing the number of stars it is possible to rate. The vast majority areusing the extreme values. Another finding in their research was that similaritems grouped together gave more consistent ratings and that fast ratings donot yield more inconsistencies. This strategy opens up for gathering a goodamount of data about the user in a short period to acquire a basic knowledgequickly.

Explicit ratings required an action from the user. This interruption of theuser flow can be critical in a negative fashion for the user to come back ona later occasion. It is hard to ensure that users leave their explicit feedback[40]. The laziness of the average user results in a relatively low ratio of ratingsper user.

2.2.2 Implicit Feedback

Implicit feedback takes advantage of the users usage pattern. This techniqueis in many ways the opposite of explicit feedback where the user does nottake an explicit action to give the system more information about themselves.Implicit feedback has the benefit of that it is easier to collect than explicitfeedback because of the user willingness. In Table 2.2, examples of di↵erentimplicit feedback methods are listed.


Y. Hu et al. [26] found that implicit feedback have four prime characteristics.The first one is that implicit feedback gives no negative feedback. It is hardto say something about items the user does not have seen. The user may notknow about the item or dislike it, and it reveals a fundamental asymmetrybetween explicit and implicit feedback.

The second characteristic is that it contains a vast sum of noise. Why isthe user browsing right now and why is he browsing the item at all can onlybe guessed with this type of feedback. For example viewing a shirt in astore does not explain that the user likes this particular shirt. However, if hecontinues to look at it multiple times, it is very likely that we can assume aninterest in the product.

The third discusses the di↵erence between numerical values of explicit andimplicit feedback. Numerical values of implicit feedback can, for example,describe how often a customer watches a TV series. An explicit feedbacktells more about how well the user relates to the item, e.g. like or dislikebuttons.

The last characteristic is the evaluation of implicit feedback. The challengescome with how to evaluate items purchased, viewed or rated more than once.Y. Hu et al. [26] comes with an example where it can be hard to comparetwo TV shows broadcasted at the same time, where the user cannot watchboth shows.

Some implicit measures do not bring any knowledge about the user at all,and some are only useful in combination with others [27]. Implicit feedbackcan as mentioned above be noisy, but with enough feedback, an appropriatemodel can be made to help the other parts of di↵erent systems be better.The more data about the user, the better, because the user patterns oftenreveal something (either in one or the other direction) about the end user.Implicit feedback can also be used to construct an explicit type of feedback.For example, if one user is only browsing novels in a book selling service,then after enough data, the system can assume the user does not have anyother preferences at all. This action is the same as picking a default searchcategory (explicit feedback).

2.3. THE COLD-START PROBLEM 19

2.3 The Cold-Start Problem

Well known for all types of recommender systems, is the cold-start problem.Some authors define the problem as when a new item is added to the dataset,and it is no ratings on that item. Another problem when a new user signs upthat the system do not have any knowledge about is often called either cold-start user problem or new-user problem. In this thesis we will describe theseissues as cold-start item problem and cold-start user problem [47]. When bothof these problems occur at the same time, e.g. a totally fresh system, S. Parket al. [38] defines it as the cold-start system problem. In this section, wedescribe how previous research have approached these problems.

2.3.1 Cold-Start User Problem

Since the recommendations are supposed to be targeted for individuals, it isimpossible to know how to compose good recommendations for a new user thesystem know nothing about. By giving bad recommendations to begin with,the user might be scared away. He could get disappointed by the mistakesof the recommender system, so he will quit using the software before enoughknowledge about the user is built up. A crucial feature of most recommendersystems is therefore to extract enough knowledge about the new user to beaccurate enough. Many di↵erent strategies to solve this problem has beentried out, and we will describe some of the most common ways to do this.

Demographics

The user can either enter some information about the user, or retrieve itfrom social medias, which can help grouping the user into similar groups.Depending on what kind of system, some groups of people might share thesame interests, so grouping people by attributes like age, sex, occupation,nationality, city, education, income, marital status, might help. Asking theuser to enter all this information is however a little time consuming for theuser, and might scare him away. By looking at the domain of the recom-mender system you want only to collect the data that will be most valuableto give recommendations. More and more applications also allow people tosign up using their social network profile, and this could help retrieving thisdata without having the user entering it manually.


Y. Wang et al. [50] implemented a demographic recommender system ontourist attractions on TripAdvisor. Using a crawler to collect ratings anddemographic data from TripAdvisor, they had information including “Age”,“Gender”, “Travel style”, “Travel for”, “Great vacation” and “Travel with”.To classify the ratings, they used traditional machine learning approaches,Naive Bayes, Support Vector Machine (SVM) and Bayesian Network. SVMperformed slightly better than Naive Bays, and both outperformed BayesianNetwork. All three methods performed better than the baseline, and canbe useful for recommendations. However, they were inaccurate on the rat-ing values 1-4, and only accurate in identifying the users that have given5 (attractions were rated 1 - 5). This outcome was probably because thedataset was very unbalanced, where most people only rated 4 or 5. The au-thors suggest for future work to implement a hybrid recommender that canincorporate more information about the attractions and reviews.

While demographic data can help a lot in some recommender systems, it isalso less general than other techniques since it depends on domain knowledgeand only apply to certain domains as A. M. Rashid et al. [42] mentions intheir paper using another approach to the cold-start problem.

Asking the Users to Rate

As A. M. Rashid et al. [42] says, the most direct way of acquiring informationabout new users, is to ask them explicitly to rate items. However, since itis bad to put more stress on the user, it is important to not ask the newusers more questions than necessary. Therefore, it becomes very crucial thatthese questions will distinguish the users, and give as much information aspossible. As mentioned in their paper, it does not help to ask questions whichall users agree on, e.g. a food recommender should not ask their users “Doyou like vanilla ice cream?”, since most people will say yes to this question.It is also important to ask questions that the user is likely to have an opinionabout. The goal must be to extract general knowledge, and too specificquestions will only make it hard for some users to know what they shouldanswer, and it gives little knowledge in return. A. M. Rashid et al. exploreapproaches for finding the best items to present to new users for rating byusing the MovieLens movie recommender. They identify four dimensionsthat a system can be judged on; (1) User e↵ort: How hard was it to sign up?(2) User Satisfaction: how well did the user like the sign up process? (3)

2.3. THE COLD-START PROBLEM 21

Recommendation accuracy: how well can the system make recommendationsto the user? (4) System utility: how well will the system be able to serveall users, given what it learns from this one? A. M. Rashid et al. choose tofocus on e↵ort and accuracy, since those two are easy to measure.

They are discussing and testing five di↵erent strategies for presenting itemsto rate to the user, and we will summarize them up here.

• Random strategies select random items to present to a user. Anotherapproach is to select one random item from a popular list, and the restat random of all items. The advantage is that all the items can collectinformation, but it is also likely that the user will not have an opinionon that item.

• Popularity is a strategy where the items are ranked in popularity, andthey are presented to the user in descending order. This makes surethat the user is likely to have an opinion about it, but it is also a bigrisk of presenting items that almost all users like, and hence will notbe able to learn anything useful.

• Pure entropy presents the items that the users usually either dislikeor like very much. An item with most ratings being either 1 or 5 willhave high entropy. This method can also su↵er from the problem thatthe user might not have an opinion about the items.

• A balanced strategy combine the popularity and entropy methods.This helps on the problems with using one of the methods, since theuser is more likely to have an opinion about the item, and the itemswill also be the ones that you can learn something about the user from.

• An item-item personalized approach are at first presenting the itemsusing one of the above methods. Other items from the users that havepreviously rated an item that the user rates, are retrieved, and pre-sented to the user.

In Table 2.3 is the results from their experiment with the di↵erent strategies.They did both an o✏ine experiment with the dataset from MovieLens1, andan online experiment with new users. Item-item su↵ers from that the items

1MovieLens dataset: http://grouplens.org/datasets/movielens/


Strategy User E↵ort Accuracy

Random ? ??

Popularity ? ? ? ? ? ? ? ??

(log) Pop*Ent ? ? ? ? ? ? ? ?

Item-Item ? ? ? ? ? ??

Table 2.3: Evaluation of di↵erent strategies from A. M. Rashid et al.

presented to rate for the users will often be clustered together, and it doesnot gain knowledge about all the items. The user e↵ort is evaluated by howmany users dropped out during sign up, and accuracy is the mean absoluteerror (MAE). MAE is here the sum of the absolute di↵erences between eachprediction and corresponding rating divided by the number of ratings.

Which strategy to use in a recommender system depends a lot on the domain.A system that works within e-commerce, should probably suggest the mostpopular items to begin with when there is no data about the user, and thenit could move on to use the item-item strategy. In other systems, it mightwork better to build up better knowledge about the user using the balancedstrategy.

Tags

Collaborative filtering typically uses a matrix User⇥ Item, which will be hardto fill out for a new user. If the items are grouped into tags, a matrix User⇥ Tag, will be denser, since for every tag you know the users opinion about,you will be able to translate that information towards multiple items. Thisapproach was developed by H. Kim et al. [28], and they tested it on a datasetof bookmarks collected from del.icio.us, a site containing bookmarks taggedby the users. They and achieved better results than regular collaborativefiltering on the items when measured with recall.

Their approach to the problem was first to build a model of the candidatetags for a user, using collaborative filtering. Using that model of the tags,they generate the top-N recommendations using Naive Bayes.

H. Kim et al. [28] mention an issue with their method that is noisy tagsmakes the performance worse. This issue could be a problem for some real

2.4. CASE-BASED REASONING 23

world applications where the users create the tags.

User Select Some Users He Trusts

Another approach that has been used is making the user select one or mul-tiple users that he trusts, and then give recommendations based on whatthese users like [34]. This technique gives the recommender engine an initialinput about the new user that can be valuable in recommending items in thebeginning.

2.3.2 Cold-Start Item Problem

When no ratings exist on an item, it is not possible to get a predicted ratingfrom a traditional collaborative filtering approach [47]. A. I. Schein et al.[47] presents a solution to this problem by using a probabilistic approachthat combines content and collaborative information by using expectationmaximization (EM) [18] learning to fit the model to the data. They testtheir approach on a movie dataset, and they look at the cast of actors in amovie, and how similar it is to what the user have rated before.

In a content-based filtering system, the cold-start item problem can be eas-ier to work around compared to the collaborative filtering approach. Withcontent-based filtering, the content of the items can be matched against auser profile, even if the item have not been seen by any other user before.

The ask to rate strategy described earlier can also help the user discover newitems if the items are selected randomly from all the items.

2.4 Case-Based Reasoning

CBR is a way to solve new problems based on previously known solutions tosimilar problems. It can be done by adapting old solutions to new problems,and be used for problems like explaining new situations, and avoid mistakesthat have been done before [45]. It has a lot in common with the way humanssolve problems already and is therefore also connected to cognitive science.In this section, we first describe the CBR approach in general, and then showhow it has been used in a recommender system.


RE

TA

IN

Problem

General

Knowledge

Past

Cases

Suggested

Solution

REVISE

Tested/ Repaired Case

Confirmed

Solution

Solved Case

New Case

New Case

Retrieved Case

Learned Case

Aam

od

t&P

laza 1

99

4

Figure 2.3: The CBR cycle

2.4.1 The CBR Principles

For understanding CBR, it is essential to know what a case is. A case repre-sents specific knowledge. J. Kolodner [29] defines it as: “A case is a contex-tualized piece of knowledge representing an experience that teaches a lessonfundamental to achieving the goals of the reasoner.”

The process in a CBR application can be described as 4 steps [1, 31] asshown in Figure 2.3 The retrieve step is looking up similar cases in the casedatabase. Then one or multiple old cases are reused to make a solution to thenew target. Revise will test the solution and change it if necessary. Whenthe new solution is successful, it will be retained in the case database.

d(p, q) = d(q, p) =

vuutnX

i=1

(qi � pi)2 (2.7)

Case-based reasoning uses a similarity measure to retrieve similar cases. Fig-ure 2.4 shows a new case and its mapping to the previously learned cases.

2.4. CASE-BASED REASONING 25

A computed mathematical distance measure the distance between two cases.The di↵erent parameters of each case are normalized/converted to a nu-meric value, then used in this similarity measure. Di↵erent similarity mea-sures are used, but the Euclidean Distance, shown in Equation 2.7, has beenvery popular and given good results [6]. The distances between cases in theknowledge-base is by itself not very useful, but can be used to retrieve thenearest neighbors. The K-nearest neighbors (KNN) [17], is an algorithm thattakes the k as a parameter saying how many neighbor nodes to return. KNNcan use this similarity measures to compare the node we now are comparingto all the others. It is shown that nearest neighbor algorithms can strugglewith high dimensions where the query case is not part of a cluster [10]. TheKNN is a lazy learning algorithm.

Listing 2.1: The KNN algorithm

For each training example <x,f(x)>:Add the example to the list of training_examples.

Given a query instance x to be classified,Let x1,x2...xk denote the k instances from training_examplesthat are nearest to x.Return the class that represents the maximum of the k instances.

In a standard KNN, the nodes takes the class of the neighbors (dependingon what class there is most of). In a case-based approach, the KNN isused, as mentioned above, to retrieve cases similar to itself. This retrievalis done because of the next step, reuse, where solutions from previous casesare mapped to the current case. Often this is not a one-to-one match, somodifications to previous solutions may be necessary. For example, one canuse di↵erent parts of each neighbor to come up with a new tailored solution.A suggested solution for the new case we are currently working on is theoutput of the reuse step.

After a solution proposed from the previous step, based on old cases andtailored to the target case, a testing/simulation of the new solution takesplace. Testing is an essential step in knowing the impact on the knowledge-base. Simulation of how well the solution performs in the environment revealsweaknesses in the solution for this particular case, and can be repaired inthis step. The goal of this step is to come up with a repaired and confirmedsolution what performs well.


Figure 2.4: The mapping from problem to solution. The dotted arrow showsa new case and its solution adapted from the previously solved cases.

The last step of the CBR-cycle is the retain step. This final step is to preserveold successful cases with good solutions. This knowledge goes back to theknowledge-base and will be one of the cases considered when a new case isassessed with the KNN algorithm.

2.4.2 CBR as a Recommender System

A generic CBR system can be seen as very similar to a recommendationprocess, hence also used as a recommender system. F. Lorenzi and F. Ricci[32] covers four recommender systems taking advantage of CBR principlesand six recommendation techniques (covered below):

• Interest Confidence Value: The assumption of this technique is thatthe user’s interest in new items is similar to the user’s interest in pastitems.

• Single Item Recommendation: Simply a recommendation of one singleitem.

2.5. EXAMPLES OF EARLIER PROJECTS 27

• Seeking for Inspiration: Prompts the user with explicit feedback toupdate the recommendations, hence the inspiration for the system.

• Travel completion: This technique o↵ers items that “completes” thecurrent selected/purchased items, e.g. accessories.

• Order-based retrieval: Like in the name, this technique emphasizes theordering of the items retrieved from the case base.

• Comparison-based Retrieval: Users preferences into query parametersto receive recommended items, e.g. cold-start preferences.

2.5 Examples of Earlier Projects

In this section, we will cover some examples of projects both in research andthe industry, which we consider to be relevant for our recommender systemthat will operate in the food recipe domain. These examples shows thediversity of recommendation methods operating in di↵erent domains, andsome projects related to food. We picked these examples since they haveinspired us in the design and development of Eatelligent.

2.5.1 Amazon.com

One of the lead industry companies in recommender systems is Amazon.com.Amazon.com is one of the largest online retailer in the world 2, started outselling books. Amazon o↵ers recommendations to their users to help themselect their purchases based on many of the techniques mentioned above. Ina real-world problem domain, we often enter a new issues not emphasized inresearch with fictive users and user responses, with respect to the originalproblem domain. Amazon is mainly using collaborative filtering and clustermodels [30]. In Amazons algorithm recommending new items it is importantto find items that are similar to the items that are already purchased by theuser, hence item-to-item collaborative filtering and clustering items in cate-gorizing clusters. Real-time recommendations with collaborative filtering is a

2The 5 Largest Online Retailers in the World:http://www.insidermonkey.com/blog/the-5-largest-online-retailers-in-the-world-331292/


challenge due to its complexity and the vast amount of items and users. Oneof the solutions to this problem has been for Amazon to introduce clustermodels. Cluster models are in many ways a similar way to tag items in thesame category to have some knowledge about the relation between items. Amajor upside of this approach due to the expectations of the user to have theresult within half a second is that it can be run o✏ine in advance. Ratherthan combining each of the users to other similar users, an item-to-item col-laborative filtering matches the users previous scored items to similar items,then merges those related items into a recommendation list. The intriguingaspect of this system is the retrieval of similar items (based on the tags foreach item), in respect of previous choices made by the user. This methodscopes down the search space both for items and users, and is very interestingdue to scalability and real-time recommendations.

2.5.2 DieToRecs

DieToRecs is a case-based travel planning recommender system [44]. Thissystem is used to make a travel plan for holiday trips to a selected destina-tion. That includes problems like which hotels to stay at and which beachesto visit. DieToRecs implemented three di↵erent recommendation techniques:Single item recommendation, travel completion and seeking for inspiration.The first door the user may use is the single item recommendation. It firstasks the user for a wide set of general questions and gradually tightens thescope until a single recommendation remains or else a tightening function isused. This technique is typically used to make the major choices for a trip (atleast for a machine that can prune possibilities), say finding a destination.The second technique is travel completion, where the system o↵ers itemsor services that complete the trip. The system retrieves similar cases (travelplans previously built by other travelers). Collaborative features for each caseis used to generate logical constraints to the travel completion step. Seekingfor inspiration is the third step in DieToRecs. This step involves promptingthe user with complete recommendations, hence giving inspiration to the sys-tem. Each iteration in a loop the user are given six choices of complete travelrecommendations. These six cases is retrieved from a randomly selected caseand retrieved by using the K-nearest neighbors to the initial case. The userselects one of the six to move gradually towards the travel recommendationthe user wants.

2.5. EXAMPLES OF EARLIER PROJECTS 29

This type of recommender systems could be used to solve problems like plan-ning of a weeks menu of dinner recipes. DieToRecs prompts the user forfeedback during the recommendation process, and this can be utilized in thefood recommendation domain to enhance recommendations by do some lastminute changes to the final recommendations.

2.5.3 CHEF

CHEF [23] is a case-based planner, which domain is recipe creation. It takesin an input consisting of goals that recipes can achieve and outputs a recipethat can reach those goals. Recipe creation is outside the scope of this thesis,but it is interesting to see the knowledge CHEF have about the role of theingredients. It knows that chicken is meat, and if some goals match an oldrecipe in everything except it is chicken instead of beef in the new goal, CHEFcan understand that chicken can replace the beef.

When CHEF is finished with making a new plan, it will use feedback to knowif it worked out or not. If not, it will try to learn from the situation, andmake an explanation of why, which will be indexed in the general knowledge,and then can use for repairing the faulty plan.

While CHEF uses the knowledge to make recipes that satisfy the goals andto modify recipes, we will not necessarily need to modify the recipes to makea good recommendations. With enough recipes in the database, it shouldbe possible to find good enough matches, but it can be done to learn moreabout the ingredients for individual users. If a user does not like salmon, thesystem should be able to learn that it is salmon the user do not like, and notjust the recipes he has tried with salmon in it.

2.5.4 JULIA

JULIA [24] is a case-based designer that works with meal planning. Theproblems are described in terms of constraints that needs to be achieved,and the solution describe a model that fills as many constraints as possible.

As JULIA is a designer, it is more suitable for complex situations where arecipe needs to be reconstructed to fit constraints. JULIA also has much


knowledge about the ingredients, and it tries to not repeat the main ingredi-ents when creating a plan of several dishes. JULIA’s focus is here on creatinga starter, main dish, and dessert, but we are focusing on recommending amain dish. However, the same approach as JULIA can be used to avoid rec-ommending multiple recipes with the same main ingredient in a row. Manypeople will prefer to have some variety in their diet, e.g. not eat salmon everyday.

2.5.5 Intelligent Food Planning: Personalized RecipeRecommendation

Personalized Recipe Recommendation [21] is a recipe recommender systemwhere the recipe is split up into its ingredients and the actual recipe. Theweighting for the recommendation is based upon not just the recipe, but alsothe ingredients used to make the dish.

This system is using collaborative filtering to make the calculations for eachuser and discusses issues around the cold-start problem that collaborativefiltering su↵ers from. The system shows some statistics of results from theproject with 183 users and 136 recipes, and 337 classified food items (ingre-dients).

This system is connected to this thesis in the way it defines the items (recipesand ingredients), where it sees the recipe as a set of ingredients. Across therelated recipes, these ingredients are recycled. On a normalized level wherethe ingredients are comparable across recipes, the learning can then be done.Section 3.4 describes how this idea is implemented in this project along withthe tag abstraction.

2.5.6 Food Recommendation Using Ontology and Heuris-tics

This paper describes a system using term frequency–inverse document fre-quency (TF-IDF) to do the food recommendation for users. Collaborativefiltering is as mentioned in [21] su↵ering from the cold-start problem, andthat is solved by looking on previous meals by each user.

2.6. RELATING THE THEORY TOWARDS EATELLIGENT 31

Nutrition is an important factor in this system, where the system gather allnutrition information for each ingredient from United States Department ofAgriculture (USDA). Nutrition is used to make a more healthy recommen-dation.

This system uses a query language to do the requests (the further plan is todefine the grammar to standardize the query language.) This language is acontent- and semantics-based approach where the attributes like ingredientsand description has a significant role. The results presented in the paperdescribes a rather good accuracy for TF-IDF.

2.6 Relating the Theory Towards Eatelligent

In this section, we will discuss the theory and background described earlier,and look into what will be implemented in our recommender system. Someof the strategies are general and can be used in any recommender system,and some are more specified to the domain of recommending food recipes.

2.6.1 Recommender System

The implemented recommender system in this thesis is a hybrid recommenderwhere some of the topics covered in this chapter are used. The mixed ap-proach to the hybrid recommender is used, because it is easy to compare thedi↵erent methods. The implementation is focused on the study of the cold-start problem in connection with a real-world example. The system startsout with zero users and zero ratings. Each of the items is never seen by anyuser (cold-start system). This matter excludes the model-based collabora-tive filtering method because it is nothing to build a learning model from. Amemory-based approach is chosen because of the online learning part that isdeveloped during the user data gathering.

For the CBR part of this hybrid recommender it mainly looks at the nearestneighbors in connection to tags for both items and users, covered in Sec-tion 3.4.1.


2.6.2 Cold-Start

The cold-start problem can be helped in many di↵erent ways, and have fo-cused on trying out a few strategies that we believe can assist in recommend-ing food recipes to a new user.

Demographics

Some demographic facts about the user can be relevant for grouping peopletogether that might have a similar taste. Attributes like age, location, sex,income and occupation could all be helpful information to know what typeof food the user likes. However, we also have to take into consideration whatkind of information the user will be willing to enter into a mobile application.First of all everything the user has to enter at sign up will add more stress andmake it more likely that the user will quit using the application even beforehe has signed up successfully. Secondly some information might sound toosensitive to most users, e.g. people will most likely not want to enter theirincome in an application for recipes.

Location, or at least city or nationality, can be valuable information fora global application since people around the world can have di↵erent foodtraditions, and di↵erent groceries available. We will let the user enter the cityon their profile, but since this application will be implemented for Norwegianusers to begin with, we will not use this information for now. Age can also beentered into their profile, but this is also something that we do not think is themost e�cient way to recommend recipes to a new user. Our implementationfocuses therefore on other methods.

Ask to Rate and Tagging of Items

To extract some knowledge explicitly from the user in a few questions, wehave designed a strategy with the knowledge that it is food recipes that weare going to recommend. Instead of just asking the user to rate recipes, weask the user some yes/no questions that we extract into knowledge aboutboth recipes and ingredient. This knowledge is extracted by grouping theingredients and recipes into tags, and each answer will make a score for theuser on a tag. This strategy will be described more in detail in Chapter 3.


The user answers to these questions can help in two ways: (1) Knowing ascore for the user on di↵erent tags, makes it possible to aggregate a score onthe recipes, and the best score are the recipe that fits the user’s model thebest. (2) The answers can also be used for finding a similar user, by lookingat the distance between two answers, and finding the recipes that the mostsimilar user likes the most.

The tagging model also work after the initial questions have been answered.Every time a recommendation is given to a user, he can select yes or no, andthe score on the tags will be updated. A user provides therefore informationto the system by just viewing di↵erent recipes.

The tags also help the system to know more about the ingredients. LikeCHEF, it knows that chicken is meat or that pepper is a spice. This knowl-edge can help identifying which tags are more important in a dish. It is likelythat a user who dislikes some of the greens in a dish are willing just to skipor replace them and still make the dish. However, a user who does not likesalmon will probably not want a recipe containing that.

Since the tags are supposed to be entered in the recipes themselves, andthe ingredients are also tagged, this approach helps on the cold-start systemproblem. This tagging help since it will be able to score each recipe in thesystem regardless of if ratings from other users exist.

Trusted Users

Asking the user to select one or multiple users he trusts is something thatcould be helpful, since a user often know some people that have the similarfood taste like him. This feature could also be implemented by using a socialmedia so that the user can easily select friends from a list. Some might befollowing somebody that blogs about food, and also know that they trusttheir taste. However this feature needs many users to work, and since oursystem will su↵er from having very few users to begin with, we will not focuson implementing this strategy.

2.6.3 Feedback

The implementation in this thesis covers a mobile application. As mentionedin Section 2.2, the gathering of explicit ratings from a user can be challenging,


so we used the benefits of smartphones to report implicit ratings to the server.The final mobile application uses tracking on every screen and how long theuser is on the page. This is implicit time tracking which can be used to saywhether or not the user has made the recipe at a later point in time. Eachtime the user asks for a recommendation, it is logged to the server. Thisinformation is then used to evaluate the recommendations for this particularuser.

As mentioned in Table 2.2, some of the implicit feedback methods is notsuitable for a real-world problem domain, because of privacy. Eye tracking,microphone input, and facial expression are therefore excluded from thisexperiment. For a pure recommender system, it is also limited how muchthis type of feedback will bring to the table. Locations for a user, when arecommendation is made, is related to context, and in some conditions grouprecommendations. GPS positions are chosen to not be implemented becausethis is falling outside of the research questions.

Primary explicit feedback methods for the implemented system is as follows:Star ratings for the explicit ratings of the di↵erent items. Like and dislikepoints for the cold-start methodology, e.g. yes or no to chicken. For theimplicit feedback, we choose to emphasize time tracking and clicks to learnmore about the usage pattern for each user.

2.6.4 Taking Advantage of the Contextual Information

We have considered some contextual information that could have an impacton what the user would want to make for dinner.

• How much time the user has available is can be very relevant. It isnot preferable to recommend a recipe that takes one hour to make ifthe user has a busy day and only 20 minutes available. However, itcan put too much stress on the user if we ask him explicitly everytime he makes a request for new recommendations about how muchtime he has available. Therefore, we have concluded that this will be abad approach. A better strategy would be to extract this informationimplicitly, e.g. by learning the days the user want something fast, andwhen he wants something fancier.


• What groceries the user already has available at home, can also beconsidered as contextual information. Some days the user would notwant to have get new groceries, and by giving the application someitems, a recipe containing those would be a good recommendation.This problem will, however, be more similar to a recipe compositionproblem, than a recommendation problem, and will not be the focus ofthis thesis.

• The fact if the user is going to make food for only himself or others,could also a↵ect what he wants to make. If all the people that weregoing to be served, also had this application and a user profile, thesystem could turn this into a group recommendation if the user selectedall the people that he would serve. Again this makes the problem morecomplicated and also require many users to get it to work in the realworld, so this will not be the focus here.

• Season, Holidays, weekends, could also be interesting to consider in thecomposition of recommendations. The season of the year could have animpact of which groceries are available, and people might have di↵erenttraditions depending on warm and cold weather. On the weekends,people often have more time and put more e↵ort into making goodfood. Some holidays include food traditions, and you might want therecommendations to represent this, and at the same time, we do notwant to recommend Christmas food outside Christmas.

Using contextual information in a recommender system for recommendingfood items at restaurants is shown to improve the accuracy compared to atraditional collaborative filtering approach [25]. We have reason to believethat this could also improve the recommendation of food recipes. Howeverit also makes the problem a lot more complicated, and in this thesis we willonly discuss it for future work in Section 5.3. We also believe that it is morelikely that the users who signs up with our application will be mainly ask forrecommendations during the weekdays, since most people tend to have theirown traditions for holidays and weekends, so the dataset will consist mostlyof recipes that can be made on a regular day.

Chapter 3

Architecture/Model

This chapter will cover the design and implementation of Eatelligent, thedinner recipe recommender. As mentioned earlier, we found it necessary todevelop our application to promote flexibility to what data we can use forthe recommendations. This chapter, therefore, describes the full applicationin detail, and not only the recommender system, since both of them play asignificant part in this thesis.

Eatelligent is released on the open market as a mobile phone application. Thesystem consists of two main components. A client, the mobile application,and the server, which serves the client with data and stores user information,recipes, and feedback. The recommender engine is implemented on the server.

We will start by describing the functionality of the application, implementa-tion details, and in the end of the chapter the details about the recommenderengine.

3.1 Requirements for Design

This section lists the requirements for the system as user stories to give anoverview of what the system should fulfill, and why we need each requirement.

• As a user, I can sign up and login so that the information about mecan be used next time I use the application to give more accuraterecommendations in the future.

37

38 CHAPTER 3. ARCHITECTURE/MODEL

• As a new user, I will be prompted with some “yes/no” questions toextract some basic knowledge about my food preferences so the systemwill have some knowledge before recommending recipes.

• As a user, I retrieve recommended recipes so I can easily explore newrecipes and find out what to eat for dinner.

• As a user, I can indicate if I want a recipe by answering “yes” or “no”.If “yes”, have a detailed look at the recipe, if “no”, view the next rec-ommendation.

• As a user, I can rate the recipes 1 to 5 stars, to give an indication ofhow much I liked a recipe, to build up the knowledge of which recipesI like.

• As a user, I can add recipes to favorites so that I can easily retrieve thebest recipes.

• As a user, I can add a recipe to a shopping cart so that the ingredientsfor the recipe I choose to make, will be easy to go and get at the grocerystore.

• As a user, I can search for and retrieve recipes knowing part of the title.

• As a user, I can look at the last viewed recipes, to easy look at therecipes I just made if I want to give them rating or make them again.

• As a user, I can reset my knowledge base, in case I wish to start over andanswer the cold-start questions again, and reset the model the systemhave developed on me.

• The application stores all the recommendations given to a user, so thatthe user will not get the same recommendations in a time interval, andthen the system will also know where the user got the recipe from if herates the recipe.

• The application stores how long a user looks at a recipe, to give anindication if the user did make this recipe or not, and the longer youlook at a recipe, the more interested you might be.

• The application gives out some recommended recipes as random recipes,to compare the rest of the recommendations with a baseline.

3.2. SYSTEM OVERVIEW 39

• As an administrator, I can group ingredients and recipes into tags sothat it can make a relation of what kind of tags the user likes or dislike.

• As an administrator, I want to add, modify, delete and publish newrecipes into the application.

3.2 System Overview

Before we go into detail about the di↵erent parts of the system, we want togive an overview of the whole user process. We describe the process from auser sign up with the mobile application, and how the client interacts withthe server and recommender system.

We store the information on the server when the user signs up, so that allthe knowledge to the user can be used every time the user logs in. Sinceno knowledge about the user is available at this point, the user is supposedto answer some “yes/no” questions. These responses are stored to give aninitial value of the preferences for the user. When a user receives recom-mendations, the server stores all of them with a field source which describeswhich algorithm promoted the recipe. This information makes it possible toalways know where the user discovered the recipe when he or she rates it.Stored along with the information is the new rating, and for recommenda-tions containing a predicted rating, we can evaluate the di↵erence betweenthe predicted and the actual rating. Figure 3.1 shows the flow between theclients through the backend to retrieve recommendations.

3.3 Implementation

In the following section, we describe the di↵erent parts of the system, discussthe various choices that we have done, and describing the implementationthat we ended up using in the final product. The three main parts is theclient, backend (server), and the recommender system.


Figure 3.1: System overview

3.3.1 Client

Since more and more of the web activity happens through smartphones, wedecided to address the end user with a smartphone application. The client forthis project is an HTML51 application embedded in a web view and packedinto an application running on all devices, available in the respective appli-cation stores for Android and iPhone. With limited resources and time todevelop a client for both Android and iPhone, we decided to develop one codebase that was able to deploy the application on the di↵erent mobile phones.This approach is possible by using HTML5 with the Cordova project2, andit also makes it very easy to communicate with the servers Application Pro-gramming Interface (API).

The client is fully backed from the API (Section 3.3.2), with Hypertext Trans-fer Protocol3 (HTTP) requests to acquire all the content visible for the enduser.

As described in the Figure 3.2, the classes are strictly distinguished between

1HTML5: http://www.w3.org/TR/html5/2Cordova: https://cordova.apache.org/3HTTP: https://tools.ietf.org/html/rfc2616

3.3. IMPLEMENTATION 41

Figure 3.2: Class hierarchy for the mobile client


authorized and not authorized classes with abstract parent classes. Di↵erentfunctionalities are split up into modules. For example, Favorites and Settingsare both standalone modules. This object-oriented approach was helping tokeep track of more complex tasks.

We put some e↵ort into user experience because the average smartphoneuser is used to use great application of di↵erent sorts. This client followsconventions for the modern smartphone application, to make the user feellike it is a good experience to use the application.

Recommendation Cycle

When the end user asks for recommendations, the server gives a list of recipespresented to the client. This list is ordered based on the recommendationfor the requesting user. Starting from the top of the list, the user have toanswer either “No thanks” or “Yes please”. If he selects “No thanks”, it isrecorded to the API and taken into consideration for later recommendations.The next recipe in the list will be presented until the user pick one recipe(answers “’Yes please”). When he select “yes”, it is also synchronized withthe server and taken into consideration for later recommendations. Then wepresent a detailed view of the selected recipe for the user, so he gets all theinformation needed to make the dish.

After the user has come up with his opinion on the recipe, the user willhopefully rate the recipe from 1 to 5 stars. All this represents the cycle ofhow the user interacts with the recommender system.

Search

To open up the entry to view the other recipes, the client also provides asearch for other recipes. When the user finds something and clicks it, it willbe collected as a view of the recipe.

Favorites

The users of the client have the possibility to add recipes to a list of favorites.The favorites module is located in the sidebar along with the other parts ofthe application.


Shopping Cart

The shopping cart is an add-on feature just to make it easy for the user tobuy the ingredients for one or multiple recipes.

Last Viewed Recipes

To view the history of recipes for one user, the client contains this module.This history can be useful for several reasons, for instance, some users maysay “Yes please” to a recipe from the recommender, and afterward figure outthat it was not a good time for this recipe after all (for several reasons). Theuser will not have to remember the name of the recipes because they arelisted here.

Data Collection

To be able to study the results of this project, we had to collect every actiondone on the client. To collect these data, we used Google Analytics to trackevery user’s user pattern. There are many di↵erent tracking services that canprovide this type of tracking, but since this is an HTML5 application, GoogleAnalytics is a relatively easy integration to use instead of custom tailoringfor each platform. Google Analytics is free and helped us collect our resultspresented in Chapter 4.

3.3.2 Backend

The backend (or server) of this system is where all the clients get their recipes,recommendations and post their feedback. In other words, this is wherethe clients get the content they display for the user, and the place theysave their information. The server is implemented using the modern webarchitecture style Representational State Transfer (REST) [20]. It is donewith the programming language Scala4 with Play Framework5, which gaveus a modern language and framework, and at the same time it was possibleto use any Java library if necessary. We hosted the server on the Heroku

4The Scala Programming Language: http://www.scala-lang.org/5Play Framework: https://www.playframework.com/


cloud service6, which enabled easy deployment without too much configuringon our own.

The server stores all the data, and the recommendation engine is also imple-mented there, which is essential since we need the data for all the users to beable to make recommendations for one user. All the di↵erent models and thestructure of the data we store in the database have been designed for beingable to recommend recipes, and evaluate the system in the best way.

Figure 3.3: Entity relation diagram

Model

Here is some information about every di↵erent object, which helps to under-stand the rest of this chapter. The relations between the entities are shownin Figure 3.3, and Appendix A shows the database scheme in detail.

6https://www.heroku.com/


• Recipe: Containing all the information needed to show a recipe to theuser.

• Ingredient: The recipes have links to the ingredient so that two recipeswith “tomatoes” in it will link to the same ingredient. This is essentialto be able to learn which ingredients an user likes or do not like.

• Recipe Tag: Tags on recipes to group them with similar recipes. Onerecipe can have many tags. The tags are made to group similar recipestogether and help to know what categories a user typically likes thebest. It is not shown to the user, and are only here to give informationto the recommender system about the recipe.

• Ingredient Tag: Same as Recipe Tag, but just for ingredients. Theseare not supposed to be shown to the user either, and are only hereto make knowledge about ingredients. These tags are manually set byourselves, and makes it easy to know if an ingredient is from an animal.It also makes it possible to know if the ingredient is a main ingredientor something not that important in a dish.

• Unit: The units that are used to describe the amount of an ingredientin a recipe.

• Star Rating: A user’s rating of a recipe. The rating is an integerbetween 1 and 5. Along with the rating the time stamp when it is ratedand the source (where the recipe was discovered, e.g. collaborativefiltering, search, etc.) with additional data are stored for evaluation.

• “Yes/No” Rating: When a user gets recommendations, he can eithersay “yes” and look up the recipe or “no” to get the next. We store thisinformation, in order to not show a recipe that the user already havesaid no to multiple times. This information can be aggregated, e.g. weknow if an user have said no 3 times.

• User Viewed Recipe: To have an indication if the user made the recipeor not, other than the rating, we store how long time the user lookedat a recipe.

• Cold-start Questions: These are “yes/no” questions that the user cananswer to give us more knowledge. The answer and the time are stored.


• Given Recommendation: All the recommended recipes given to a userare stored to be used for evaluation.

Status Code DescriptionOK 200 Everything went OKBad request 400 The request was not under-

stood by the serverUnauthorized 401 The user requesting is not au-

thenticatedForbidden 403 The user is not allowed to re-

trieve the resource requestedNot found 404 The server could not find the

resource that was requestedConflict 409 Used for duplicates that are

not allowedInternal server error 500 Unexpected server error

Table 3.1: The di↵erent response codes used by the server

Requests

The client uses the server by making requests on di↵erent URLs which iscalled endpoints. Appendix B lists the documentation for each endpoint.Di↵erent HTTP methods7 are used for di↵erent types of requests. GETrequests are used when the client only asks for data, and nothing will bechanged on the server. Requests for storing data, e.g. a rating on a recipe,will be done using POST requests. If something is supposed to be updated itis done using PUT. The response from the server contains a response code8

that tells the client if the request succeeded, or if something went wrong.In Table 3.1 the di↵erent codes used by our server are listed and explainedbriefly.

All the data passed to or from the server is in a format called JSON9. In

7HTTP Method Definitions: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html8HTTP Status Code Definitions: http://www.w3.org/Protocols/rfc2616/rfc2616-

sec10.html9The JavaScript Object Notation standard: http://www.json.org/


Figure 3.4 - 3.8, we show the screenshots of the app, requests from client,and response from the server.

Listing 3.1: Sign up request{"firstName": "Ola","lastName": "Nordmann","email": "[email protected]","password": "ola"

}

Listing 3.2: Sign up response{"ok": true,"message": {"id": 14,"loginInfo": {"providerID": "credentials","providerKey": "[email protected]"

},"firstName": "Ola","lastName": "Nordmann","email": "[email protected]","image": null,"role": "user","created": "2015�04�13T16:47:04.402","recipeLanguage": null,"appLanguage": null,"city": null,"country": null,"sex": null,"yearBorn": null,"enrolled": false,"metricSystem": true

}}

Figure 3.4: The sign up process


Listing 3.3: Cold-Start Request{

"coldStartId":3,"answer": true

}

Listing 3.4: Cold-Start Response{

"ok": true,"answer": {

"userId": 14,"coldStartId": 3,"answer": true,"answerTime": "2015�04�15T12:12:36.591"

}}

Figure 3.5: The user answer 1/5 questions


Listing 3.5: Recommendations{ok: true,recommendations: [{__meta__: {userId: 1,recipeId: 14,source: "CF",predictedRating: "4.5779"

},recipe: {id: 14,name: "Burger med asiatisk smak",image: "http://res.cloudinary.com/

hnjelkrui/image/upload/v1423563013/jmdigci5h3k7si69e7wz.jpg",

description: "...",spicy: 2,time: 30,difficulty: "Enkel"

}},...

]}

Figure 3.6: The users answers “yes” or “no” to a recommendation


Listing 3.6: Recipe{ok: true,recipe: {id: 14,name: "Burger med asiatisk smak",image: "http://res.cloudinary.com/hnjelkrui

/image/upload/v1423563013/jmdigci5h3k7si69e7wz.jpg",

description: "...",language: 1,calories: "0",procedure: "...",spicy: 2,time: 30,difficulty: "Enkel",source: "http://www.matprat.no/oppskrifter/

familien/burger�med�asiatisk�smak/",created: "2015�02�10T10:09:51.636",modified: "2015�03�12T14:31:51.965",published: "2015�03�11T14:11:55.016",ingredients: [{id: 70,name: "Kjøttdeig",unit: "gram",amount: "125"

},...

],tags: ["Familien","Asiatisk"

],currentUserRating: null,averageRating: "4"

}}

Figure 3.7: Detailed view of the recipe

3.4. RECOMMENDER SYSTEM 51

Listing 3.7: Rating request{

"recipeId": 14,"rating": 4.0

}

Listing 3.8: Rating response{"ok": true,"rating": {"userId": 14,"recipeId": 14,"rating": "4","created": "2015�04�15T14:46:54.497"

}}

Figure 3.8: Rating the recipe

3.4 Recommender System

The client reaches the recommender system through one of the API end-points, and the recommender engine is a part of the server. This was done to


have easy access to the database. This section describes the implementationdetails of the recommender systems.

3.4.1 Algorithms

We have implemented three di↵erent algorithms that work separately to rec-ommend recipes for a user. Here we will describe each of them in detail.

Nearest User Rated Recipe (NURR)

The first thing the user does in the application is to answer a few questionsregarding taste and expertise on the kitchen. The five questions asked is thefollowing:

• Do you like meat?

• Do you like spicy food?

• Do you view yourself as a good chef?

• Do you like fish?

• Do you like chicken?

Each of the cold-start questions a↵ects the weights for the current user. Thesenumbers correspond to how much each ingredient tag is weighted by theEuclidean distance for the KNN algorithm (see Section 2.4.1). Table 3.2displays the matrix of weights for each question. This information will helpthe KNN calculate distances between users the first few times before it hasenough ratings from explicit star ratings. A negative answer to one of thequestions will result in updating the corresponding ingredient tags with anegative sign. The weights have a maximum cap at 5 and minimum of -5.This maximum cap is to avoid that weights do not sum up to infinite valuesand to make it possible to change the taste.

All these questions will have a boolean answer (“yes” or “no”) which we areusing for similarity measure between two users. We define ~ua as the vectorof the responses for user a. Equation 3.1 defines the similarity measure used


Meat Spicy Skills Fish ChickenSpice – 0.5 – – –Spicy – 1.0 – – –Salty – 0.2 – – –Composed – – -1.0 – –Fish – – – 1.0 –Seafood – – – 0.8 –Animal product 0.3 – – 0.1 0.2Chicken – – – – 1.0Meat 1.0 – – – –Pork 0.8 – – – –Beef 0.8 – – – –

Table 3.2: The weighting from the cold-start questions

to compare two users cold-start answers. The di↵erence between two vectorsis defined in Equation 3.2 where ua,i is the ith answer from user a and n isthe number of questions, in this case, 5.

sim( ~ua, ~ub) =1p

di↵( ~ua, ~ub) + 1(3.1)

di↵( ~ua, ~ub) =nX

i=1

|ua,i � ub,i| (3.2)

For example, if user A and user B have the following responses:

User A responses: yes, yes, no, no, yesUser B responses: yes, no, yes, no, yes

The similarity between these users is calculated in Equation 3.3.

sim(A,B) =1p

di↵(A,B) + 1

sim(A,B) =1p

|1� 1|+ |1� 0|+ |0� 1|+ |0� 0|+ |1� 1|+ 1

sim(A,B) =1p3

(3.3)


Algorithm 1: Nearest User Rated Recipes

Data:currentUser - current user,users - all other users,n - number of recipes to return,foreach user in users do

user.similarity sim(currentUser, user)endusers users filtered on user.similarity > minThresholdrecipes empty collectionforeach user in users do

r retrieveTopRatingsForUser(user, maxRecipesFromOneUser)foreach ratedRecipe in r do

ratedRecipe.score user.similarity ⇥ ratedRecipe.ratingrecipes recipes + ratedRecipe

endendreturn recipes.sortBy(recipe.score)[0 : n]

Based on this similarity measure, we can find the most similar users. Theseusers ratings will be measured with the same similarity, and be the productof their actual rating and the similarity to that similar user. For example,say the similarity is 0.8, and the other user have rated a recipe 4 stars, thenthe score for this recipe will be 0.8 ⇤ 4 = 3.2. Note that this score is not arating, but a scoring used to rank the di↵erent recipes to return the top nrecipes. Algorithm 1 describes this procedure in pseudocode. This algorithmis designed by ourselves and is inspired by the first two steps from CBR (seeSection 2.4).

Content-Based Filtering (CBF)

One or multiple recipe tags marks every recipe, to group the recipes. Thesetags are set by ourselves. We do the same thing with the ingredients, withanother set of tags that we call ingredient tags. The system is developed witha relation between user and both the recipe and ingredient tags so that a scorefor the user is stored for the tags. This forms a user profile as mentioned inSection 2.1.1, and Figure 3.9 and 3.10 illustrates this relation. The scores on


the tags are made up from three di↵erent sources:

• Cold-start questions: Some of the questions maps to some ingredienttags. For example, if a user says that he does not like fish, the systemwill store a negative score for the tags seafood and fish. A value between-1 and 1 updates the score, and are set by ourselves with the best guessthat we can do between the relations.

• “Yes” or “no” to a recipe: When the user answers “yes” or “no” to a rec-ommended recipe in the application, we take that as a weak indicationthat the user likes or dislikes the recipe. A value of -0.3 or 0.3 updatesthe corresponding recipe tags and the tags for the ingredients in therecipe.

• Rating: A rating on a recipe is considered to be a stronger indicationif the user likes or dislikes it. The di↵erent possible values decide howmuch the score should be updated. These impacts is listed in Table 3.3.

? ?? ? ? ? ? ? ?? ? ? ? ? ?-1 -0.5 0 0.5 1

Table 3.3: The impact of star ratings

The score each user has for the ingredient and recipe tags, makes it possibleto compute a score for each recipe for a user, by aggregating the tag scores ofthe tags in a recipe. Algorithm 2 shows how this is calculated. The recipesare first filtered on some preferences for the user, based on two questions heanswered when he started using the app. If he answered no to “Do you likespicy food”, we filter out the recipes that are spicy. The recipes belong toone out of three levels of spiciness, and the people who do not like spicy food,will only get the lowest category. If the user likes spicy food, we do not filteranything because we believe that people who like spicy food not necessarilyonly want the spicy recipes. The same strategy are done with how long timeit takes to make the food, and how hard the recipe is to make. If the useranswered no to “Do you view yourself as a good chef?”, we filter out therecipes that are not in the easy category (out of easy, medium, and hard),and the ones that take more than 45 minutes to make. After this filtering isdone, a score is calculated for each recipe based on the scores the user have


on the recipe and ingredient tags in the recipe, and the top n recipes arereturned. We designed this algorithm ourself, but inspiration is taken fromearlier projects. E.g., when CHEF (see Section 2.5.3) knows about the linkbetween chicken and meat.

Algorithm 2: Content-based filtering

Data:recipes - all recipes,user - user preferencesurt - user score on all recipe tagsuit - user score on all ingredient tagsn - number of recipes to return,Result: Top n recipesif not user.likeSpicyFood then

recipes recipes filtered on not spicyendif not user.isAGoodChef then

recipes recipes filtered on time < 45 min. and easy di�cultyendforeach r in recipes do

score 0foreach recipeTag in r.recipeTags do

score score+ urt[recipeTag]endforeach ingredient in r.ingredients do

foreach ingredientTag in ingredient.ingredientTags doscore score+ uit[ingredientTag]

endendr.score score

|recipeTags|+|ingredientTags|endrecipes recipes filtered on score > 0return recipes.sortBy(r.score)[0 : n]


Figure 3.9: Illustration of the recipe tag tree

Figure 3.10: Illustration of the ingredient tag tree


Collaborative Approach

As a helper for the collaborative filtering mechanism we used the open sourceproject from GroupLens called LensKit10. LensKit is an implementation ofcollaborative filtering algorithms along with a set of tools support of suchsystems. LensKit is split up in di↵erent modules to help developers only touse the parts needed for the particular implementation. This projects usestwo of this modules, lenskit-core and lenskit-data-structures.

When building a recommender with LensKit, the core module is required andis a dependency for the other modules. For this implementation, the core isused to initialize the building of the collaborative filtering recommendation.An item recommender is built from an access object to the database where itwill gain access to the other users, ratings and items (recipes). This methoduses KNN to retrieve similar items where the similarity is calculated by acosine vector similarity. This approach is a memory-based method where allthe calculations are done when the user asks for a list of recommendations.

Some problem specific configuration is needed, to help LensKit compute therecommendations. The lenskit-data-structures module provides data struc-tures to help with this configuration

3.4.2 Feedback in the Client

As described in Section 2.2, feedback is necessary to understand what pref-erences each of the users have. In the client, we gather both explicit andimplicit feedback. Two di↵erent explicit feedback mechanisms are used: (1)After a recommendation list is presented, the user answers either“Yes please”or “No thanks”. (2) When a user views a recipe, he can report a star ratingfrom 1 to 5 stars.

We gather the implicit feedback in two ways: (1) When a user looks at arecipe, a timer is started. This timing gives us an indication whether or notthe user made the dish. (2) For every request, timestamps are logged. Thesetimestamps helps us to not recommend the same recipes within the sameperiod for a user.

10The LensKit project: http://lenskit.org/


3.4.3 Cold-Start Item Problem

Pushing items that do not have ratings would help to promote new items,and let the users discover the whole set of recipes. By randomly pickingsome recipes, both new and recipes with no rating all have a small chanceof being promoted. This strategy makes the evaluation of the system easiersince the baseline are the random recipes, and the cold-start item problem isnot the primary focus of Eatelligent at this moment. Therefore, no specialpromotion strategy is implemented for new items.

3.4.4 Composition of Recommendations

The composition of the recommendation list of any size is based mainly onthe collaborative filtering engine and the CBF algorithm. These two algo-rithms compose 80% of the list (40% from each). When there is not enoughknowledge for the collaborative filtering or CBF algorithms to find enoughrecipes, the NURR algorithm fills in the rest. For CBF, this happens becauseof the threshold that is defined. With a lower score than this threshold, therecipes are excluded. For CF, this can be when the user have not rated anyrecipes and the engine has nothing to compare against. The last 20% of thelist is a set of random recipes. This strategy is done to be able to comparethe other algorithms to random in the evaluation, and to help integratingnew recipes without ratings. To give all methods a fair chance we randomizewhich of the algorithms to get the top of the list. Therefore, the recommen-dation list is not deterministic. The random contribution helps the unratedrecipes to gather some ratings as well.

All the di↵erent set of recipes from the recommendation methods except theones we choose at random are filtered on additional attributes. The recipesthat are not rated the last 14 days, not been seen in total the last sevendays, nor been said no to in the last three days, are returned. This filteringis done to make sure the user do not get the same recipes recommended everytime. Table 3.4 shows an example of a list of recommendations where thecollaborative filtering results on top. The next 5 to 7 items is from the CBFalgorithm. Since CBF only filled three out of four possible slots, the NURRalgorithm had to fill in one recipe for this example.


# FromPredictedRating

Similarityto User

Score RecipeId

1 CF 4.4 – – 52 CF 4.1 – – 63 CF 3.7 – – 44 CF 3.7 – – 75 CBF – – 3.88 426 CBF – – 2.75 27 CBF – – 1.27 98 NURR – 0.7 – 19 Random – – – 1010 Random – – – 23

Table 3.4: An example on a composed recommendation list

3.5 How it Works

To describe the functionality with the weights and new recommendationssystem in the system, we will show an example of a user going through thefollowing steps: A recommendation, answering “yes” or “no” on a few recipes,picking a recipe and rating the recipe and receiving new recommendations inthe end.

3.5.1 A New User Receiving a List of Recommenda-tions

After the user have signed up, they will be prompted with five questions asmentioned earlier. The ingredient tag weights will be updated from 0 to amore personalized weighting in respect to the cold-start questions answered.For this example, say the user have answered the following:

Meat: Yes, Spicy: No, Skills No, Fish: Yes, Chicken: Yes

After processing these questions, the user will have the following weightsstored in the user ingredient tag and user recipe tag relations shown in Ta-ble 3.5. Note that composed, spicy, spice, and salty got the negative valuedue to the negative answer on the previous questions. An explanation ofeach ingredient tag and recipe tags is listed in Appendix C. Answering this

3.5. HOW IT WORKS 61

Ingredient Tag ValueMeat 1.0Animal product 0.3 + 0.2 + 0.1 = 0.6Pork 0.8Beef 0.8Chicken 1.0Seafood 0.8Fish 1.0Composed 1.0Spicy -1.0Spice -0.5Salty -0.2

Recipe tag ValueHealthy 0Asian 0Quick 0

Table 3.5: The current state of the weights after the cold-start questions.

questions does not a↵ect the recipe tags. These tags are updated after rat-ings of the recipes (described in Section 3.4.1), not by answering cold-startquestions.

The user is presented a list of recommendations from di↵erent recommenda-tion engines, and then the list is logged in the database to gather results.For our example, the output from the recommendation endpoint is shown inListing 3.9. Note that the CF algorithm returns no items before the user per-forms a star rating on a recipe. This is because the CF algorithm exclusivelydoes calculations based on this type of rating, and cannot do a comparisonwith an empty user-rating vector.

After the ordered list of 10 items is displayed, the user will say yes or noto the recipes recommended. For our example, say the user says no to thefirst two recipes and wants to check out the third one. This feedback is thenused to update the users weights. The update depends on the recipe andwhat ingredients that the recipe contains because of the ingredient tags thatis updated. Of course, this is a rather “harmless” choice, unlike a star rating.The results of this examples behavior is listed in Table 3.6


Ingredient Tag Di↵erence New valueMeat 1 negative = �0.3 0.9Animal product 1 negative, 1 positive = 0 0.6Pork 1 negative = �0.3 0.5Beef 1 negative = �0.3 0.5Chicken 1 positive = 0.3 1.3Seafood – 0.8Fish – 1.0Composed 1 negative, 1 positive = 0 1.0Spicy 1 negative, 1 positive = 0 -1.0Spice 1 negative, 1 positive = 0 -0.5Salty 1 positive = 0.3 0.1Extras 1 negative, 1 positive = 0 0.0Green 2 negative, 1 positive = �0.3 -0.3Fruit 1 negative = �0.3 -0.3Milk product 1 positive = 0.3 0.3

Recipe tag Di↵erence New valueHealthy 1 positive = 0.3 0.3Asian 2 negative = �0.6 -0.6Quick 1 negative, 1 positive = 0 0

Table 3.6: The current state of the weights after the “yes/no” ratings

3.5. HOW IT WORKS 63

Listing 3.9: First list of recommendations to a user{

ok: true ,

recommendations: [

{

recipe: { name: "Babi Asam Manis" }

},

{

recipe: { name: "Asian omelet" }

},

{

recipe: { name: "Paprika Chicken with asparagus" }

},

...

]

}

The selected recipe, “Paprika Chicken with asparagus” is chosen by the useras todays dinner (for a full description of the recipe, see Section C.3.1). Atimer is present and reported back to Eatelligents backend while the recipeis viewed by the user. The user then rates the recipe, say 4 star rating. InTable 3.3, a 4 star rating corresponds to a weight of 0.5 times every ingredienttag. Table 3.7 shows the updated weights after the feedback is given.

Now, when the user have given both implicit and explicit feedback to therecommendation engine, we have some more content to base the next recom-mendations on. This information helps the other users (collaborative part),and the user itself the next time some user asks for a recipe list recommenda-tion. In the last part of the example, the next recommendation list is aimedmore at the users preferences. Table 3.7 shows that the tags chicken, animalproduct and composed ingredients is something this user likes.

Listing 3.10 shows the next recommendation after the actions described aboveis carried out. The CF recommender proposes a chicken-based recipe, due tosimilar users and their rating history. From the NURR method, a fish dishis suggested. The most similar users ratings explain this suggestion. TheCBF algorithm presents another chicken dish due to the positive weight tothe chicken and animal product tags. The items ranking in the list are ran-domized, where di↵erent parts of the list are the top. This randomization isbecause we want to expose recommendations from all recommender methods


Ingredient tag New valueSpicy �1.0 + 0.5 = �0.5Salty 0.1 + 0.5 = 0.6Spice �0.5 + 0.5 = 0Green �0.3 + 0.5 = 0.2Extras 0 + 0.5 = 0.5Milk product 0.3 + 0.5 = 0.8Chicken 1.3 + 0.5 = 1.8Animal product 0.6 + 0.5 = 1.1Composed 1.0 + 0.5 = 1.5

Recipe tag New valueHealthy 0.3 + 0.5 = 0.8Asian -0.6Quick 0

Table 3.7: The updated weights after a rating of “Paprika Chicken withasparagus”

Listing 3.10: New recommendation after interactions{

ok: true ,

recommendations: [

{

__meta__: { source: "CF", predictedRating: 4.45 },

recipe: { name: "Curry chicken with banana and fennel" }

},

{

__meta__: { source: "NURR", similarityToUser: 1.0 },

recipe: { name: "Fishburger" }

},

{

__meta__: { source: "CBF", score: 4.1 },

recipe: { name: "Chickenwok with sweet and sour" }

},

...

]

}

3.6. MAKING EATELLIGENT PUBLIC 65

as the first recipe. Furthermore, this ensures that the user is unbiased as towhich recommender is employed in retrieving the various recipes.

3.6 Making Eatelligent Public

As described in Section 3.3.1, architectural choices is made to make the publi-cation to most users a relatively simple process. The 25th of March 2015, themobile client was published in Google Play and Apple App Store. To haveboth Android and iPhone users, help gather diversity in the user mass to re-veal possibly di↵erences [22]. The project’s homepage at http://eatelligent.nopromoted the application along with social media and the word of mouth.Chapter 4 describes the results of the gathered data.

http://eatelligent.no

Chapter 4

Experiments and Results

In this chapter, we describe our experiments along with the corresponding re-sults. We start by listing the experimental plan, followed by the experimentalsetup, and in the end we show the results.

4.1 Experimental Plan

Multiple ways exist to evaluate the recommender system. The most generalone is to look up the di↵erence between the user ratings on recipes pickedby a random generator, and compare it with the ratings on recipes given bya recommender algorithm. With enough data, this approach makes it easyto say if the algorithms are intelligent at all, and it is easy to compare thedi↵erent recommendation approaches. Since the logs of the recommendedrecipes always contain the name of the algorithm promoting it, this can beaggregated easily. For both the star ratings and the “yes/no” ratings, thisstrategy will be done. The stars contain the strongest indication of whetherthe user liked the recipe or not, and the “yes/no” ratings tell us if the userfound that recipe interesting.

For the collaborative filtering algorithm, the recommended recipes containa predicted rating, which also enables us to evaluate the di↵erence betweenthe predicted and the actual user rating, once a user rates a recipe. Thisevaluation is measured by the mean absolute error (MAE). This error isdefined as the absolute value of predicted rating, p, subtracted by the actual

67

68 CHAPTER 4. EXPERIMENTS AND RESULTS

user rating, a. The mean of these errors is the MAE and explains how farthe algorithm is from the optimal predictions.

MAE =1

n

X|p� a| (4.1)

For all the di↵erent recommender algorithms, it is interesting to plot theperformance over time since the accuracy should increase. Both the timelinefor a new user until he has several ratings, but also how an algorithm performswith few vs. many users in the system. It could also be interesting to evaluatethe cold-start item problem after a while, by releasing some new ones, andsee if they are picked up by the recomender system or not. However duringour test period, we do not have enough data to answer all these questions,and a deeper evaluation must wait until later work.

Since a recommender system needs much data to get better, the application isreleased on the open market in both App Store and Google Play. Therefore,we did not know beforehand how many users we were going to include in ourtests, but we report the numbers in the experimental setup.

When the user gets a recommendation, the server gives ten for each request,but the user only sees one. He can swipe “yes” or “no”, and if “no”, he getsthe next one. This approach enables us to evaluate how many recipes theuser on average looks at before he chooses a recipe. A lower number wouldbe better.

4.2 Experimental Setup

The data for this chapter describes the application after ten weeks on theopen market, available for anyone to test. Explicit feedback from the users,either by rating a recipe 1-5 or by answering “yes” or “no”, forms the basis ofour results.

The number of items (recipes) the recommender can choose from is 100. Thenumber of users is 109. These users have together used the application for1126 minutes, giving each user 10.5 minutes in average. The users have given5071“yes/no” ratings and 87 star ratings. All users have answered the 5 cold-start questions described in Section 3.4.1. They have together viewed 413recipes and a list of ten recommended recipes is generated 924 times.

4.3. EXPERIMENTAL RESULTS 69

4.3 Experimental Results

This section will present the results collected during our test period.

4.3.1 Yes/No ratings

When presenting the recommendations, the users answer either “yes” or “no”to the recipe. Figure 4.1 shows the distributions of answers between the algo-rithms. The di↵erent algorithms do not have an equal number of responses.Out of the ten recommendations given each time, some of the algorithmswill not always output recipes, and on average collaborative filtering outputsmore. This outcome happens because both CBF and NURR requires therecommendations to above a certain threshold so that the algorithms ratheroutputs nothing than bad recommendations.

200

400

600

800

1000

1200

1400

CF NURR CBF Random

PositiveNegative

Figure 4.1: “Yes/no” ratings per algorithm

To compare the di↵erent algorithms on a normalized level, Figure 4.2 showsthe same data as a stacked histogram.


0

20

40

60

80

100

CF NURR CBF Random

%of

total

PositiveNegative

Figure 4.2: Normalized “Yes/no” ratings per algorithm

We can see that CF and CBF do a relatively good job in recommendingrecipes that the user wants to check out further. Compared to the randomitems given, all three procedures performs better (ratio between positive andnegative answers).

4.3.2 Star Ratings

Figure 4.3 shows the distribution in rating of stars and this reveals the databasis of star ratings. As we can see, it is a lot more of 4 and 5 ratingscompared to the others. We assume this is because people are more likely tomake the recipes they think they will like, and, therefore, most people willfilter out the recipes before they consider rating them.

Figure 4.4 shows the normalized distribution of stars ratings per algorithm.We can see that random have a relatively large amount of low ratings com-pared to the other algorithms. It is also small di↵erences between the di↵erentalgorithms, but we should be careful about analyzing it too much since the


0

5

10

15

20

25

30

35

40

1 star 2 stars 3 stars 4 stars 5 stars

Figure 4.3: Star ratings

Algorithm No. ratings

NURR 8

CF 38

CBF 27

Random 12

Table 4.1: The number of ratings for each algorithm

number of ratings is low as shown in Table 4.1.

2.38 is the average number of recommended recipes the user checks outbefore he select one. This average number reflects that the users both wantto explore the recommendations deeply, but at the same time receives recipesthat are interesting in a reasonable time. A number that is high (closer to 10,because of the limitation of recommendations in one request), would a↵ectthe users judgment of the recommendations. A small number would describea user mass with a sense of “yes to everything” attitude.

The recommended recipes from CF that ended up with a rating had an MAEof 0.71. This number is only possible to calculate for this algorithm becausethe other algorithms does not predict a rating, but calculates a score or asimilarity. This number does not have a rock solid ground due to the usermass, but it describes a good prediction at least in an early stage of the


0

20

40

60

80

100

CF NURR CBF Random

%of

total

1 star2 stars3 stars4 stars5 stars

Figure 4.4: Percentage of star ratings for each algorithm


system.

The results presented is from a tiny data set and, therefore, these results donot necessarily describe the correct nature of the algorithms. Still Figure 4.3and 4.4 shows a di↵erence between the intelligent methods vs. randomlygiven recipes.

4.3.3 Comparing Yes/No and Star Ratings

The recommendations are based upon three types of user feedback; answerto cold-start questions, “yes/no” ratings, and star ratings on the recipes.However, the di↵erent algorithms do not consider all the types of feedbackthemselves. NURR and CF both focus on finding similar users and theirhighly-rated recipes, but they lack learning by the “yes/no” feedback. The“yes/no” ratings can be aggregated for each recipe, and if we consider “yes”as 1 and “no” as -1, we can calculate a sum for each recipe. Based on resultsfrom Figure 4.1, we see there is an overweight of “no” ratings. Therefore,a negative sum is common. However, as shown in the results in the upperchart in Figure 4.5 (which illustrates this aggregation), some of the recipesare doing significant worse than the average. These recipes all have relativelygood average rating (bottom chart). We have reason to believe that this iscaused by people not giving star ratings on the recipes they are not interestedin. Therefore, the number of stars will remain high while people say “nothanks” to the recipe. Since both NURR and CF do not learn from the“yes/no” feedback, they continue to promote the recipes with good rating,and this explains why a “bad” recipe is recommended many times.

4.3.4 Cold-Start Questions

The GINI index can be used to say something about the cold-start questionsability to split the user mass into di↵erent buckets. This technique calculatesa number between 0 and 0.5 where 0.5 is a perfect split on a boolean question.Equation 4.2 shows how to calculate the GINI index for each question.

GINI = 1�⇣ No. yes

No. users

⌘2

�⇣ No. no

No. users

⌘2

(4.2)

Table 4.2 lists the number of positive and negative responses to the cold-startquestions. The three of the questions with GINI score below 0.25 is relatively


�50�40�30�20�10

0

10

20

30

012345

0

5

10

15

Aggregatedyes/noratings

Avg

.stars

No.

ratings

Figure 4.5: Comparison between yes/no and star ratings. The data for eachrecipe corresponds vertically between the two charts. In the bottom chart,the impulses represent the number of ratings, and the bars show the averagestar rating.


bad questions compared to the others. Still, these questions help the few ofthe users to get in the right direction according to the weights. The questionregarding spicy and skills in the kitchen does, on the other hand, separatethe user mass to a greater extent. However, this numbers does reflect theimpact of each question to the final recommendations.

Question Yes No GINI

Do you like meat? 96 13 0.21

Do you like spicy food? 82 27 0.37

Do you view yourself as a good chef? 75 34 0.43

Do you like fish? 94 15 0.24

Do you like chicken? 94 15 0.24

Table 4.2: Answers to the cold-start questions

Table 4.3 lists the number of recommendations made based on the answersfrom the cold-start questions. Positives are recommendations given that sup-ports the response from the cold-start questions. Negatives are recommen-dations where the user have said no to a category in the cold-start questions,but the recommenders have disregarded this input. Note that the number ofrecommendations that is associated with spicy is relatively high. We havereason to believe that this is because some recipes got a good rating at anearly point in time, and, therefore, got snapped up by the collaborative al-gorithms.

The numbers in Table 4.3 are not surprising due to the implementation of thealgorithms but are relevant to demonstrate how they behave regarding thecold-start questions. CF is only considering star ratings and does not takethe cold-start questions into consideration at all. Therefore, this algorithmrecommends more recipes disregarding the answers. NURR is similar toCF since it looks at the ratings, but it looks at the cold-start questions forfinding the similar users. However some users might have given a high scoreto recipes even if they do not correspond to their answers for the questions,and this might make the data noisy. CBF is not surprisingly doing best (lessnegative recommendations) here since that approach looks at the content,and the weights are initially set by the cold-start questions.

It would also be interesting to evaluate the ratings the users give the recipes


CF NURR CBF

Pos. Neg. Pos. Neg. Pos. Neg.

Meat 575 171 429 163 882 131

Spicy 4537 1867 1809 897 2623 0

Skills 262 108 50 22 70 0

Fish 23 13 0 0 71 9

Chicken 535 83 47 18 496 21

Table 4.3: Recommendations made by each algorithm based on cold-startquestions

that are recommended negative vs. positive regarding the answers to theinitial questions. However, we were not able to do this due to a tiny dataset.

Chapter 5

Evaluation and Conclusion

This chapter will evaluate the work covered in this thesis. First we presenta discussion regarding the link to the research objectives presented in theintroduction, followed by summarizing our contributions. In the end, wepresent some thoughts about what future work can do both with this projectand recommendations of food recipes.

5.1 Evaluation and Discussion

Chapter 2 covers previous research regarding recommender systems, alongwith techniques for gathering feedback and methods to approach the cold-start problem. Previous ideas have inspired us in the making of the algo-rithms used in Eatelligent. We covered more theory and approaches to rec-ommender systems than we had the time and data to test in our application,so in Section 5.3 we discuss some of the ideas for future research.

5.1.1 RO1 - Gain an understanding of what it meansto recommend food recipes

We have done a deep dive into what features that are truly important inthe food domain, with respect to recommendations. Our studies have un-veiled that the basic primitives (ingredients) can be generalized into higherabstractions to gain more knowledge about the true model. This thesis does

77

78 CHAPTER 5. EVALUATION AND CONCLUSION

not cover the nutrition element that is clearly a dimension of relevance forsome groups of users, which can discover new interesting abstractions. Wewould also like to mention context as a factor in understanding what it meansto recommend food recipes. The lack of understanding the context of a useris a weakness because we believe context is a significant factor of what a userchoose to make for dinner.

5.1.2 RO2 - Study and evaluate di↵erent strategies torecommend items for a new user using both col-laborative and content-based approaches

Since we wanted to evaluate di↵erent recommendation strategies we choseto implement this as a hybrid recommender system, by using the hybridmixed approach which uses di↵erent algorithms separately. This approachenables us to compare the di↵erent algorithms easily with each other. Sincecollaborative filtering is the most common machine learning algorithm in therecommender systems today, we found it obvious to include this techniquein our system. We also wanted to have one content-based approach, and thiswas achieved by grouping ingredient and recipes in di↵erent tags, and updatea user profile continuously for these tags. These two approaches turned outto have very similar results for various reasons. Both CF and CBF havedi↵erent shortcomings and strengths. CF recommends popular recipes andis doing worse on exploring the whole dataset. CBF gives recommendationsbased on learned personalized weights and pays attention to the cold-startquestions and“yes/no” ratings, but it does not care about the star rating andhence can recommend less popular recipes.

The last approach, NURR, was created to have a technique that looks onlyat the answers to the five cold-start questions presented to a new user. Thismethod performs worse than the other two but still better than random. Thebiggest problem with this approach is that NURR sometimes find one userthat matches on the cold-start questions but if his best recipes are consideredby most people to be bad, the algorithm does not consider any new feedback.As shown i Section 4.3.4, many people gives the same responses to the cold-start questions, and if one user with a popular response vector and goodrating on a “bad” recipe, NURR will recommend this recipe to many people.To improve NURR, the algorithm needs to take advantage of“yes/no”ratings.

5.1. EVALUATION AND DISCUSSION 79

All three algorithms perform better than just giving out random recipes,so the results are promising, and all the approaches show to be useful inrecommending recipes. However, all of them are still recommending morerecipes which the user is not interested in compared to the recipes the userlook at closer. One explanation is that people seems to have a browsinghabit when looking at food recipes, and they want to look at multiple recipesbefore they decide to take one.

5.1.3 RO3 - Study existing solutions to the cold-startproblem

In Chapter 2 we presented existing solutions to the cold-start problem. Onthis subject, much research is done, and we picked out the most commonstrategies and the ones that we believed could be useful in Eatelligent.

Recommender systems are continuously evolving to be more accurate andare widely used and researched by the industry. In the search for literature,it is always possible that we have been biased and that some methods arenot discovered. However, we think we have discovered the most common andessential approaches forming a good foundation for this thesis.

5.1.4 RO4 - Study what kind of data we can collectfrom the application used by a user over a periodof time that is relevant to build more knowledgeabout the user

We found that the collection of data from a user in the application contextwas a fairly easy task within limitations. To acquire more knowledge aboutwhy a user answers“yes”or“no”to a dish, a better model of the context needsto be understood. The context is not only important for a domain where foodrecipes are in focus, but also in other domains of recommendation.

Upon three types of user feedback, the recommendations are based; the an-swers to cold-start questions,“yes/no”ratings, and star ratings on the recipes.Our results show that all the types are relevant to learn from, but the dif-ferent algorithms lack by not taking advantage of all of them. Improvementscan be done by combining the “yes/no” and star ratings. Implicit ratings


often have roots in the usage patterns of the user mass, and due to the smallamount of usage data, these type of recommendation enhancements was notimplemented.

We have gained a deep knowledge of what types of feedback that are impor-tant, and how this feedback can be used to build up knowledge about theuser.

5.1.5 RO5 - Study the challenges that arise when build-ing an application with a recommender systemfrom scratch

The thesis have faced challenges regarding the development of a full featuredapplication with a mobile application and a server with a recommender sys-tem as a main focus. Evaluation of the system has been a challenge due tothe number of users and usage within our time limits.

Not being able to evaluate the accuracy of the recommender systems duringthe development process made it hard to adjust parameters and the designof the recommender algorithms throughout the process. In the design of rec-ommender systems with a dataset, it is easy to evaluate every step by testingthe accuracy on the test set, but we had to wait until we had released theapplication. By releasing the application earlier we could have collected morefeedback, and then improvements could be done to improve the recommen-dations. Within our time limits, this was not possible, and in future workthe algorithms need to be tuned when more evaluation is done.

Since we developed this system as a new application and collected all the datain a short period with new users, the results are very sparse. The experimentsshould continue for a longer period, to derive more definitive conclusions.

5.2 Contributions

This section summarizes our contributions from this thesis.

Chapter 2 provides an overview of theory and previous research related torecommender systems, including problems like the cold-start problem and

5.2. CONTRIBUTIONS 81

user feedback. CBR is also described there as a problem-solving strategy tomake recommender systems.

Eatelligent is available for any user (only recipes in Norwegian so far) in AppStore and Google Play and is free to download. Improvements can be done tomake the application more user-friendly, but we are happy to see that peoplealready can have use of our product.

While star ratings or other similar ratings on items are the typical feedbackof many recommender systems, we provide two types of explicit feedback;stars and a binary “yes/no”’ rating. The latter is less explicit and extractsknowledge about more items, not only the ones that the user likes. Ouralgorithms do not take enough advantage of the combination of these twotypes of ratings, but we provide the results and analysis that will help futureimplementations to do so.

It is not revolutionary to use tags to structure the data and use this informa-tion for recommendations, and we found this to be an e�cient data structurefor machine readable recipes. The tagging on ingredients is especially goodsince the number of ingredients does not increase significantly after addingenough recipes, and the tags can be entered manually without too muchwork. If we were to let users add recipes in the future, they would not needto add tags to the ingredients since those already exist.

With more user data, better evaluation can be done, but we provide aninitial evaluation of the results in this thesis. We are careful to make toostrong conclusions, but the results are already helping us to see what kind ofimprovements that need to be done.

All code written for Eatelligent is open source1 and released under the MITlicense. Hopefully, some people can take advantage of this in future research,and the di↵erent parts of the system can easily be modified to solve othertypes of problems than the recommendation of food recipes. The mobile ap-plication is self-contained, and can easily be modified to retrieve other typesof data. Greater changes need to be done for the server and recommendationcode to recommend in other domains, but for food recommendation it is agood framework to start with.

1Source code: https://github.com/eatelligent

https://github.com/eatelligent


5.3 Future Work

While developing this recommender system, and also when studying the pre-vious research, we discovered many ideas and useful features. It is not pos-sible to make everything at once, and we had to prioritize what is the keyfunctionality to answer the research objectives. In this section, we present theideas that we think should be tried out in future research and new versionsof Eatelligent.

A similar evaluation of the system needs to be done after collecting moredata. A new evaluation will help to draw stronger conclusions and to tunethe algorithms more precisely.

As we mentioned in Section 2.6.4, contextual information can be very in-teresting to consider in the composition of recommendations in Eatelligent.This approach was not implemented and tested, and should be tried out infuture work. Some demographic info could be extracted by letting the usersign up with a social network profile. Geographic location can be used totrack where the user is, and see if it can learn any patterns. If the user is onthe way to the grocery store, he might care less about going for a recipe withmany ingredients that he needs to buy. However, if he is home and want tomake something with the ingredients he already has, the system should beable to take advantage of this. Either by taking some ingredients as input andoutput a recipe consisting of those or it could also assume some ingredientsthat he is likely to have.

This system focuses on the recommendation to one user. However, peopleare often making dinner for a party, and extracting this contextual informa-tion, and implement group recommendations, is an interesting problem. Thisfeature is also likely to be something that many people would find useful ina food recommender system.

More work is needed with respect to which parameters are important in afood recipe. For example, in this thesis, nutrition is not emphasized and couldhelp boost the personalization both for one user and group recommendations.This improvement could make it possible to use the number of calories as afeature, and also be able to recommend recipes that fit in a user’s diet. Forexample, a user on a low-carbohydrate diet does not want recommendationscontaining many carbohydrates.

In this study, only the mixed hybrid method is used, and other approaches

5.3. FUTURE WORK 83

should also be tried out. We think it is best to start to test the algorithmsseparately like we did, to evaluate them. Once we discover useful algorithms,combinations of these should be tried out and might result in better accuracy.

Mixtures of other hybrid approaches along with other algorithms such asmodel-based collaborative filtering could be an interesting comparison to theresults covered in this thesis.

The cold-start weights listed in Table 3.2 is taken by our best guess based onour knowledge about food. From a data set to optimize the score for eachuser on the recipe and ingredient tags, these weights could be learned.

As mentioned in Section 3.4.3, methods for promoting new items and itemswithout ratings are not implemented. This aspect of recommender systems isimportant for real world problems when new items are added frequently, andto find good strategies towards this issue more research needs to be done.

Appendix A

SQL-scheme

This appendix lists the PostgreSQL scheme used in this project.

CREATE TABLE language (id s e r i a l 8 PRIMARY KEY,l o c a l e t ex t NOT NULL UNIQUE,name text NOT NULL UNIQUE

) ;

CREATE TABLE r e c i p e t a g (id s e r i a l 8 PRIMARY KEY,name text NOT NULL UNIQUE

) ;

CREATE TABLE un i t (id s e r i a l 8 PRIMARY KEY,name text NOT NULL UNIQUE

) ;

CREATE TABLE i n g r ed i e n t (id s e r i a l 8 PRIMARY KEY,name text NOT NULL UNIQUE,

) ;

CREATE TABLE use r s (id s e r i a l 8 PRIMARY KEY,f i r s t name text ,

85

86 APPENDIX A. SQL-SCHEME

last name text ,emai l t ex t NOT NULL UNIQUE,image text ,role text ,c r ea t ed timestamp ,r e c i p e l anguage in t8 REFERENCES language ( id ) ,app language in t8 REFERENCES language ( id ) ,c i t y text ,country text ,sex text ,year born int

CONSTRAINT age checkCHECK ( year born > 1900 AND year born < 2016) ,

e n r o l l e d boolean ,metr ic system boolean

) ;

CREATE TABLE r e c i p e (id s e r i a l 8 PRIMARY KEY,name text NOT NULL,image text ,d e s c r i p t i o n text ,language in t8 NOT NULL REFERENCES language ( id )

ON DELETE CASCADE,procedure text ,sp i cy int

CONSTRAINT sp i cy checkCHECK ( sp i cy > 0 AND sp i cy < 4) ,

time intCONSTRAINT t ime checkCHECK ( time > 0) ,

d i f f i c u l t y text ,source text ,c r ea t ed timestamp NOT NULL,modi f i ed timestamp NOT NULL,pub l i shed timestamp ,d e l e t ed timestamp ,c reated by in t8 NOT NULL REFERENCES use r s ( id )

ON DELETE CASCADE) ;

87

CREATE TABLE f a v o r i t e s (u s e r i d in t8 REFERENCES use r s ( id )

ON DELETE CASCADE,r e c i p e i d in t8 REFERENCES r e c i p e ( id )


CREATE UNIQUE INDEX f a v o r i t e s i d xON f a v o r i t e s ( u se r id , r e c i p e i d ) ;

CREATE TABLE r e c i p e i n t a g (r e c i p e i d in t8 REFERENCES r e c i p e ( id )

ON DELETE CASCADE,t a g i d in t8 REFERENCES r e c i p e t a g ( id )


CREATE UNIQUE INDEX r e c i p e i n t a g i d xON r e c i p e i n t a g ( r e c i p e i d , t ag i d ) ;

CREATE TABLE i n g r e d i e n t t a g (id s e r i a l 8 PRIMARY KEY,name text NOT NULL UNIQUE

) ;

CREATE TABLE i n g r e d i e n t i n t a g (i n g r e d i e n t i d in t8 REFERENCES ing r ed i e n t ( id )

ON DELETE CASCADE,t a g i d in t8 REFERENCES ing r ed i e n t t a g ( id )


CREATE UNIQUE INDEX i n g r e d i e n t i n t a g i d xON i n g r e d i e n t i n t a g ( i n g r ed i en t i d , t ag i d ) ;

CREATE TABLE i n g r e d i e n t i n r e c i p e (r e c i p e i d in t8 REFERENCES r e c i p e ( id )

ON DELETE CASCADE,i n g r e d i e n t i d in t8 REFERENCES ing r ed i e n t ( id )


ON DELETE CASCADE,u n i t i d in t8 REFERENCES uni t ( id )

ON DELETE CASCADE,amount r e a l

) ;

CREATE TABLE u s e r s t a r r a t e r e c i p e (u s e r i d in t8 REFERENCES use r s ( id )


ON DELETE CASCADE,r a t i ng r e a l

CONSTRAINT r a t i ng checkCHECK ( r a t i ng >= 0.0 AND r a t i ng <= 5 . 0 ) ,

c r ea t ed timestamp ,c r e a t ed l ong int8 ,source text ,data j son

) ;

CREATE UNIQUE INDEX u s e r s t a r r a t e r e c i p e i d xON u s e r s t a r r a t e r e c i p e ( use r id , r e c i p e i d ) ;

CREATE TABLE u s e r y e s n o r a t e r e c i p e (u s e r i d in t8 REFERENCES use r s ( id )


ON DELETE CASCADE,r a t i ng int ,l a s t s e e n timestamp

) ;

CREATE UNIQUE INDEX u s e r y e s n o r a t e r e c i p e i d xON u s e r y e s n o r a t e r e c i p e ( use r id , r e c i p e i d ) ;

CREATE TABLE us e r v i ewed r e c i p e (u s e r i d in t8 NOT NULL REFERENCES use r s ( id )

ON DELETE CASCADE,r e c i p e i d in t8 NOT NULL REFERENCES r e c i p e ( id )

ON DELETE CASCADE,

89

durat ion in t8 NOT NULL,l a s t s e e n timestamp NOT NULL

) ;

CREATE UNIQUE INDEX u s e r v i ewed r e c i p e i dxON us e r v i ewed r e c i p e ( use r id , r e c i p e i d ) ;

CREATE TABLE l o g i n i n f o (id s e r i a l 8 PRIMARY KEY,p r ov i d e r i d text ,p rov ide r key text

) ;

CREATE TABLE u s e r l o g i n i n f o (u s e r i d in t8 NOT NULL,l o g i n i n f o i d in t8 NOT NULL

) ;

CREATE TABLE password in fo (hasher text ,password text ,s a l t text ,l o g i n i n f o i d in t8

) ;

CREATE TABLE c o l d s t a r t (id s e r i a l 8 PRIMARY KEY,image text NOT NULL,i d e n t i f i e r t ex t UNIQUENOT NULL,d e s c r i p t i o n text NOT NULL,i n g r e d i e n t t a g s hstore ,r e c i p e t a g s hs to r e

) ;

CREATE TABLE u s e r c o l d s t a r t (u s e r i d in t8 NOT NULL REFERENCES use r s ( id )

ON DELETE CASCADE,c o l d s t a r t i d in t8 NOT NULL REFERENCES c o l d s t a r t ( id )

ON DELETE CASCADE,answer boolean ,


answer time timestamp ,) ;

CREATE TABLE u s e r i n g r e d i e n t t a g (u s e r i d in t8 REFERENCES use r s ( id )

ON DELETE CASCADE,i n g r e d i e n t t a g i d in t8 REFERENCES ing r ed i e n t t a g ( id )

ON DELETE CASCADE,value r e a l

CONSTRAINT va lue checkCHECK (value >= �5.0 AND value <= 5 . 0 )

) ;

CREATE UNIQUE INDEX u s e r i n g r e d i e n t t a g i d xON u s e r i n g r e d i e n t t a g ( use r id , i n g r e d i e n t t a g i d ) ;

CREATE TABLE u s e r r e c i p e t a g (u s e r i d in t8 REFERENCES use r s ( id )

ON DELETE CASCADE,r e c i p e t a g i d in t8 REFERENCES r e c i p e t a g ( id )

ON DELETE CASCADE,value r e a l

CONSTRAINT va lue checkCHECK (value >= �5.0 AND value <= 5 . 0 )

) ;

CREATE UNIQUE INDEX u s e r r e c i p e t a g i d xON u s e r r e c i p e t a g ( use r id , r e c i p e t a g i d ) ;

CREATE TABLE given recommendation (id s e r i a l 8 PRIMARY KEY,u s e r i d in t8 REFERENCES use r s ( id )


ON DELETE CASCADE,type text ,ranking int ,c r ea t ed timestamp ,data j son

) ;

Appendix B

Endpoints

The backend described in Section 3.3.2 consists of endpoints. These end-points are used by the client to interact with the backend through HTTP.Every endpoint is prefixed with the server host.

B.1 GET

Endpoints that is reached with the GET method uses the auth token (currentuser) and returns either one entity or a list of entities. Below is a list of thisprojects endpoints that goes under the GET method.

• /api/users - Returns a list of all the users

• /api/recommendation/recipes - Returns a personalized a list withrecipes

• /api/users/:user id - Takes a user_id as a URL parameter and re-turns the user with its fields

• /api/user - Returns the current user entity

• /api/ingredients - Returns a list of all ingredients

• /api/ingredients/:ingredient id - Takes a ingredient_id and returnsa verbose version of the ingredient entity

91

92 APPENDIX B. ENDPOINTS

• /api/ingredients/tags - Returns a list of the ingredient tags

• /api/recipes - Returns a list of all recipes

• /api/recipes/:recipe id - Takes a recipe_id as a URL parameter andreturns a verbose version of the recipe entity

• /api/recipes/tags - Returns a list of the recipe tags

• /api/favorites/recipes - Returns a list of the favorite recipes for thecurrent user

• /api/recipes/viewed - Returns a list of the ten last viewed recipesfor the current user

• /api/coldstart - Returns the cold-start questions

B.2 PUT

PUT endpoints are used to update an entity. These endpoints input a payloadof what to update on the entity.

• /api/user - Inputs new user fields and returns the current users up-dated entity.

• /api/recipes/:recipe id - Takes a recipe_id in the URL and newrecipe fields and returns the updated recipe entity.

B.3 POST

Endpoints that use the POST method sends a payload to the server. Thispayload often used to create a new entity.

• /api/users - Creates a new user entity

• /api/recipes - Creates a new recipe along with new ingredients if theyare not already stored in the database

B.4. DELETE 93

• /api/authenticate - Inputs the user credentials and authenticates theuser. This sets the current user for this session with a cookie

• /api/ratings/recipes - Inputs the recipe_id and a star_rating to givea star rating on a recipe from the current user

• /api/favorites/recipes - Inputs the recipe_id to save the recipe tothe favorites list for the current user

• /api/ratings/recipes/binary - Inputs recipe_id and a boolean togive a “yes/no rating” to a recipe

• /api/recipes/viewed - Inputs the duration of how long a recipe wasviewed for the current user

• /api/coldstart - Inputs the cold_start_question_id and a boolean togive feedback to the cold start questions

• /api/knowledge/ reset - Resets the weights for the current user

B.4 DELETE

DELETE endpoints delete an entity. The deletion does not permanentlydelete the entity but sets a flag in the database, marked is as deleted.

• /api/recipes/:recipe id - Takes a recipe_id in the URL and deletesthe recipe with this id

• /api/favorites/recipes/:recipe id - Takes a recipe_id in the URLand removes the recipe from the favorites list for the current user.

94 APPENDIX B. ENDPOINTS

Appendix C

Recipe Features

This appendix lists important features regarding recipes and how a they arestructured.

C.1 Ingredient Tags

All the ingredients are mapped to either zero or some of the listed tags:

• Animal product

• Beef

• Berries

• Cheese

• Chicken

• Composed

• Extras

• Fish

• Flour

95

96 APPENDIX C. RECIPE FEATURES

• Fruit

• Green

• Meat

• Milk product

• Pasta

• Pork

• Potatoes

• Salty

• Seafood

• Spice

• Spicy

Most of the ingredient tags is self-explanatory, and what ingredients goesunder this category.Composed and extras can be more di�cult to understandand is described below:

The composed tag is ingredients that are made up from di↵erent otheringredients. Pizza sauce is typically a composed ingredient due to the com-position crushed tomatoes and di↵erent types of spices. Skillful chefs mayuse few of these ingredients.

Extras can be referred to as ingredients that are used as side dishes. Pastaand corncob are associated with this ingredient tag.

C.2 Recipe Tags

All recipes are associated with at least one recipe tags. Below is the list ofall recipe tags used in the set of recipes:

• Family

C.3. FULL RECIPE DESCRIPTION 97

• Fast

• Healthy

• Comfort food

• Guests

• Asian food

• Italian food

• Mexican food

• Indian food

• Norwegian food

C.3 Full Recipe Description

Listed below is three recipes used in the example in Section 3.5.

C.3.1 Paprika Chicken with Asparagus

This recipe contains the following ingredients for one portion:

• Salt (0.5 tablespoons)

• Pepper (0.5 tablespoons)

• Green fresh asparagus (4 pieces)

• Sour cream (1 tablespoon)

• Chicken filet (1 pieces)

• Corn flakes (2 tablespoons)

• Paprika powder (1 tablespoon)


These ingredients resolves to the following ingredient tags:

• Salt ! Spicy, Salty, Spice

• Pepper ! Spice

• Green fresh asparagus ! Green, Extras

• Sour cream ! Milk product

• Chicken filet ! Chicken, Animal product

• Corn flakes ! Composed

• Paprika powder ! Spice

C.3.2 Babi Asam Manis


• Onion (0.25 pieces)

• Garlic (0.75 cloves)

• Pork into strips (150 grams)

• Chili powder (0.25 teaspoon)

• Canned pineapple chunks (0.25 can)

• Paprika (0.5 pieces)

• Ready to cook wok sauce (0.25 bag)

• Pineapple juice (0.5 deciliter)


• Onion ! Green

• Garlic ! Green, Spice

C.3. FULL RECIPE DESCRIPTION 99

• Pork into strips ! Pork, Meat

• Chili powder ! Spice, Spicy

• Canned pineapple chunks ! Composed, green

• Paprika ! Green

• Ready to cook wok sauce ! Composed

• Pineapple juice ! Fruit

C.3.3 Asian Omelet


• Egg (1.5 pieces)

• Oyster sauce (0.5 tablespoons)

• Beef meat in strips (100 grams)

• Bean sprouts (0.25 can)

• Oyster mushrooms (50 grams)


• Egg ! Animal product

• Oyster sauce ! –

• Beef meat in strips ! Beef

• Bean sprouts ! Extras

• Oyster mushrooms ! Green

Bibliography

[1] A. Aamodt and E. Plaza. Case-based reasoning: Foundational issues,methodological variations, and system approaches. AI communications,7(1):39–59, 1994.

[2] G. D. Abowd, C. G. Atkeson, J. Hong, S. Long, R. Kooper, andM. Pinkerton. Cyberguide: A mobile context-aware tour guide. Wirelessnetworks, 3(5):421–433, 1997.

[3] G. Adomavicius, R. Sankaranarayanan, S. Sen, and A. Tuzhilin. Incor-porating contextual information in recommender systems using a mul-tidimensional approach. ACM Transactions on Information Systems(TOIS), 23(1):103–145, 2005.

[4] G. Adomavicius and A. Tuzhilin. Toward the next generation of rec-ommender systems: A survey of the state-of-the-art and possible ex-tensions. Knowledge and Data Engineering, IEEE Transactions on,17(6):734–749, 2005.

[5] G. Adomavicius and A. Tuzhilin. Context-aware recommender systems.In Recommender systems handbook, pages 217–253. Springer, 2011.

[6] D. W. Aha. Case-based learning algorithms. In Proceedings of the 1991DARPA Case-Based Reasoning Workshop, volume 1, pages 147–158,1991.

[7] X. Amatriain, J. M. Pujol, and N. Oliver. I like it... i like it not: Eval-uating user ratings noise in recommender systems. In User modeling,adaptation, and personalization, pages 247–258. Springer, 2009.

[8] M. J. Berry and G. Lino↵. Data mining techniques: for marketing, sales,and customer support. John Wiley & Sons, Inc., 1997.

101

102 BIBLIOGRAPHY

[9] J. Bettman, M. Luce, and J. Payne. Consumer decision making: A con-structive perspective. Consumer Behavior and Decision Making, pages1–42, 1991.

[10] K. Beyer, J. Goldstein, R. Ramakrishnan, and U. Shaft. When is “near-est neighbor” meaningful? In Database Theory—ICDT’99, pages 217–235. Springer, 1999.

[11] J. S. Breese, D. Heckerman, and C. Kadie. Empirical analysis of pre-dictive algorithms for collaborative filtering. In Proceedings of the Four-teenth conference on Uncertainty in artificial intelligence, pages 43–52.Morgan Kaufmann Publishers Inc., 1998.

[12] P. J. Brown, J. D. Bovey, and X. Chen. Context-aware applications:from the laboratory to the marketplace. Personal Communications,IEEE, 4(5):58–64, 1997.

[13] R. Burke. Hybrid web recommender systems. In The adaptive web,pages 377–408. Springer, 2007.

[14] P. G. Campos, I. Fernandez-Tobıas, I. Cantador, and F. Dıez. Context-aware movie recommendations: An empirical comparison of pre-filtering,post-filtering and contextual modeling approaches. In E-Commerce andWeb Technologies, pages 137–149. Springer, 2013.

[15] F. Cena, L. Console, C. Gena, A. Goy, G. Levi, S. Modeo, and I. Torre.Integrating heterogeneous adaptation techniques to build a flexible andusable mobile tourist guide. AI Communications, 19(4):369–384, 2006.

[16] D. Cosley, S. K. Lam, I. Albert, J. A. Konstan, and J. Riedl. Is seeingbelieving?: how recommender system interfaces a↵ect users’ opinions. InProceedings of the SIGCHI conference on Human factors in computingsystems, pages 585–592. ACM, 2003.

[17] T. Cover and P. Hart. Nearest neighbor pattern classification. Informa-tion Theory, IEEE Transactions on, 13(1):21–27, 1967.

[18] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihoodfrom incomplete data via the em algorithm. Journal of the royal statis-tical society. Series B (methodological), pages 1–38, 1977.

BIBLIOGRAPHY 103

[19] A. K. Dey, G. D. Abowd, and D. Salber. A conceptual framework anda toolkit for supporting the rapid prototyping of context-aware applica-tions. Human-computer interaction, 16(2):97–166, 2001.

[20] R. T. Fielding and R. N. Taylor. Principled design of the modernweb architecture. ACM Transactions on Internet Technology (TOIT),2(2):115–150, 2002.

[21] J. Freyne and S. Berkovsky. Intelligent food planning: personalizedrecipe recommendation. In Proceedings of the 15th international confer-ence on Intelligent user interfaces, pages 321–324. ACM, 2010.

[22] T. J. Gerpott, S. Thomas, and M. Weichert. Characteristics and mo-bile internet use intensity of consumers with di↵erent types of ad-vanced handsets: An exploratory empirical study of iphone, androidand other web-enabled mobile users in germany. TelecommunicationsPolicy, 37(4):357–371, 2013.

[23] K. J. Hammond. Chef: A model of case-based planning. In AAAI, pages267–271, 1986.

[24] T. R. Hinrichs and J. L. Kolodner. The roles of adaptation in case-baseddesign. In AAAI, volume 91, pages 28–33, 1991.

[25] M. Hosseini-Pozveh, M. Nematbakhsh, and N. Movahhedinia. A multi-dimensional approach for context-aware recommendation in mobile com-merce. arXiv preprint arXiv:0908.0982, 2009.

[26] Y. Hu, Y. Koren, and C. Volinsky. Collaborative filtering for implicitfeedback datasets. In Data Mining, 2008. ICDM’08. Eighth IEEE In-ternational Conference on, pages 263–272. IEEE, 2008.

[27] D. Kelly and J. Teevan. Implicit feedback for inferring user preference:a bibliography. In ACM SIGIR Forum, volume 37, pages 18–28. ACM,2003.

[28] H.-N. Kim, A.-T. Ji, I. Ha, and G.-S. Jo. Collaborative filtering basedon collaborative tagging for enhancing the quality of recommendation.Electronic Commerce Research and Applications, 9(1):73–83, 2010.

[29] J. Kolodner. Case-based reasoning. Morgan Kaufmann, 1988.

104 BIBLIOGRAPHY

[30] G. Linden, B. Smith, and J. York. Amazon. com recommendations:Item-to-item collaborative filtering. Internet Computing, IEEE, 7(1):76–80, 2003.

[31] R. Lopez De Mantaras, D. McSherry, D. Bridge, D. Leake, B. Smyth,S. Craw, B. Faltings, M. L. Maher, M. T. Cox, K. Forbus, et al. Retrieval,reuse, revision and retention in case-based reasoning. The KnowledgeEngineering Review, 20(03):215–240, 2005.

[32] F. Lorenzi and F. Ricci. Case-based recommender systems: a unifyingview. In Intelligent Techniques for Web Personalization, pages 89–113.Springer, 2005.

[33] D. A. Lussier and R. W. Olshavsky. Task complexity and contingentprocessing in brand choice. Journal of Consumer Research, pages 154–165, 1979.

[34] P. Massa and B. Bhattacharjee. Using trust in recommender systems: anexperimental analysis. In Trust Management, pages 221–235. Springer,2004.

[35] K. Oku, S. Nakajima, J. Miyazaki, and S. Uemura. Context-aware svmfor context-dependent information recommendation. In Mobile DataManagement, 2006. MDM 2006. 7th International Conference on, pages109–109. IEEE, 2006.

[36] C. Palmisano, A. Tuzhilin, and M. Gorgoglione. Using context to im-prove predictive modeling of customers in personalization applications.Knowledge and Data Engineering, IEEE Transactions on, 20(11):1535–1549, 2008.

[37] U. Panniello, A. Tuzhilin, M. Gorgoglione, C. Palmisano, and A. Pedone.Experimental comparison of pre-vs. post-filtering approaches in context-aware recommender systems. In Proceedings of the third ACM conferenceon Recommender systems, pages 265–268. ACM, 2009.

[38] S.-T. Park, D. Pennock, O. Madani, N. Good, and D. DeCoste. Naıvefilterbots for robust cold-start recommendations. In Proceedings of the12th ACM SIGKDD international conference on Knowledge discoveryand data mining, pages 699–705. ACM, 2006.

BIBLIOGRAPHY 105

[39] M. J. Pazzani and D. Billsus. Content-based recommendation systems.In The adaptive web, pages 325–341. Springer, 2007.

[40] D. Poo, B. Chng, and J.-M. Goh. A hybrid approach for user profil-ing. In System Sciences, 2003. Proceedings of the 36th Annual HawaiiInternational Conference on, pages 9–pp. IEEE, 2003.

[41] C. Prahalad. Beyond crm: Ck prahalad predicts customer context is thenext big thing. American Management Association MwWorld, 2004.

[42] A. M. Rashid, I. Albert, D. Cosley, S. K. Lam, S. M. McNee, J. A.Konstan, and J. Riedl. Getting to know you: learning new user prefer-ences in recommender systems. In Proceedings of the 7th internationalconference on Intelligent user interfaces, pages 127–134. ACM, 2002.

[43] P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl. Grou-plens: an open architecture for collaborative filtering of netnews. InProceedings of the 1994 ACM conference on Computer supported coop-erative work, pages 175–186. ACM, 1994.

[44] F. Ricci, D. R. Fesenmaier, N. Mirzadeh, H. Rumetshofer, E. Schaum-lechner, A. Venturini, K. W. Wober, and A. H. Zins. Dietorecs: acase-based travel advisory system. Destination recommendation sys-tems: behavioural foundations and applications, pages 227–239, 2006.

[45] C. K. Riesbeck and R. C. Schank. Inside case-based reasoning. Psychol-ogy Press, 2013.

[46] N. S. Ryan, J. Pascoe, and D. R. Morse. Enhanced reality fieldwork:the context-aware archaeological assistant. In Computer applications inarchaeology. Tempus Reparatum, 1998.

[47] A. I. Schein, A. Popescul, L. H. Ungar, and D. M. Pennock. Methodsand metrics for cold-start recommendations. In Proceedings of the 25thannual international ACM SIGIR conference on Research and develop-ment in information retrieval, pages 253–260. ACM, 2002.

[48] B. N. Schilit and M. M. Theimer. Disseminating active map informationto mobile hosts. Network, IEEE, 8(5):22–32, 1994.

106 BIBLIOGRAPHY

[49] M. Van Setten, S. Pokraev, and J. Koolwaaij. Context-aware recommen-dations in the mobile tourist application compass. In Adaptive hyper-media and adaptive web-based systems, pages 235–244. Springer, 2004.

[50] Y. Wang, S. C.-f. Chan, and G. Ngai. Applicability of demographic rec-ommender system to tourist attractions: A case study on trip advisor. InProceedings of the The 2012 IEEE/WIC/ACM International Joint Con-ferences on Web Intelligence and Intelligent Agent Technology-Volume03, pages 97–101. IEEE Computer Society, 2012.

[51] Z. Yu, X. Zhou, D. Zhang, C.-Y. Chin, X. Wang, et al. Support-ing context-aware media recommendations for smart phones. PervasiveComputing, IEEE, 5(3):68–75, 2006.

design of a hybrid recommender system: a study of the cold

Documents