An Agent-based Movie Recommender SystemCombining the Results Computed based on

Heterogeneous Semantic Datasets

Andreas Lommatzsch, Benjamin Kille, and Sahin Albayrak

DAI-Lab, TU BerlinErnst-Reuter-Platz 7, D-10587 Berlin, Germany

{andreas.lommatzsch,benjamin.kille,sahin.albayrak}@dai-labor.de

Abstract. In a growing number of domains, recommender systems sup-port users’ decision making processes in finding entities (such as book,movies, or newspaper articles) matching their individual preferences. In-dividual preferences vary extensively thus hampering the creation of goodrecommender systems. Additionally, the relevance of items to users de-pends on a plethora of criteria. Such relevance criteria may considerusers’ current contexts, their ratings (“likes”/“dislikes”) and content de-scribing the entities (e. g., a movie’s leading actors). In this paper, weintroduce an agent-based recommender approach that integrates het-erogeneous recommender agents and combines their results based on apersonalized weighting scheme. The framework is open for new agents,allowing us to integrate new algorithms and data sources. The use ofa personalized weighting scheme for combining the results from hetero-geneous agents as well as the integration of personalized recommenderagents allows the system to adapt the results to individual users’ pref-erences. For improving the trust in unknown items, the system providesdetailed explanations based on the semantic relations. We present a rec-ommender system for the movie domain (based on the framework) anddiscuss the evaluation results.

1 Introduction

Recommender systems support users in finding relevant entities by reducing thenumber of alternatives to those that they will likely prefer. Traditional recom-mender systems focus on Collaborative Filtering (CF). User-based CF is builton the assumption that entities (items or products) are potentially relevant forusers if like-minded users liked them (or rated them positively). As discussed bySchein et al. [1], CF cannot provide adequate recommendations unless a compre-hensive set of ratings is available for all users and all items. This phenomenonis called the “cold start problem” [2]. New users as well as items entering thesystem are particularly affected thereof. The problem incurs as similarities be-tween existing entities and the newly added ones are undefined. In addition,CF-based algorithms cannot provide good human understandable explanations.

The lack of explanations describing in what extent a recommended entity isrelevant, often leads to missing trust in the system [3]. CF-based recommenda-tions are often biased towards popular items, probably already known to theusers. Recommending already known entities to the user is neither useful norserendipitous.

In contrast to Collaborative recommender algorithms, content-based recom-mender algorithms do not handle entities as black boxes. Instead, an entity’srelevance is determined by its properties. Thus, content-based recommenders areable to compute similarities for new (still unrated) entities. Moreover, content-based recommender systems may provide explanations describing which proper-ties of the recommended entities match the user’s preferences. The recommenda-tion process requires detailed descriptions to ensure high-quality recommenda-tions. This has been a big problem for years, due to the fact that adding these de-scriptions is costly and time consuming. With the rapidly growing availability ofbig semantic data collections, content-based recommender systems receive grow-ing attention. The fine-grained machine readable, “semantically represented”reusable, often “open” descriptions for huge entity sets provide a good basis forcontent-based recommendations. Comprehensive semantic data collections forthe entertainment domain are available, for instance from Freebase1, DBpe-dia2, and IMDb3. Semantic data sets represent knowledge as graphs consistingof nodes (representing entities) and edges (representing relationships betweennodes). Nodes as well as edges are equipped with labels providing detailed in-formation about them. Entities in semantic dataset are identified by uniqueURIs allowing interlinking entities with other datasets and additional features.This universal representation of information is well-suited for content-based anduser-specific data. Since every dataset and relationship set might have specificsemantics, optimized edge algebras and models must be learned for every dataset.

Since every user has specific preferences and criteria on what properties de-termine an entity’s relevance, recommender systems should provide personalizedrecommendations. Typically, preferences can be seen as a complex, continuouslychanging mixture of many different criteria (e.g., influenced by friends or newexperiences). Moreover, new data sources, providing relevant information, maybecome available. Thus, the recommender system’s design should support theintegration of new recommender strategies and be able to constantly adapt tochanging user preferences.

In this paper, we present an agent-based approach that encapsulates varioussemantic recommender algorithms and combines their suggestions based on in-dividual user preferences. We evaluate the recommender models and show thatthe personalized aggregation of results from heterogeneous recommender agentsprovides highly precise recommendation. The remaining paper is organized asfollows: Section 2 reviews related work and discusses current approaches. Sec-tion 3 describes the analyzed scenarios and formulates the research questions.

1 http://www.freebase.com2 http://dbpedia.org3 http://www.imdb.com

Section 4 introduces the implemented system and discussed the developed algo-rithms in detail. A discussion of our evaluation results can be found in Section 5.Finally, we conclude and give an outlook to future work in Section 6.

2 Related Work

In this section we discuss the properties of different recommender approaches.Collaborative and Content-based Recommender Collaborative recommender

systems suggest items based on preference ratings [4]. If sufficient ratings foreach user and item exist, CF-based systems provide high-quality results. In casethere is only a small number of ratings available, CF-based recommenders can-not provide good recommendations. This phenomenon occurs particularly fornew users and items entering the system. In addition, CF does not provide goodexplanations.Content-based filtering algorithms compute the relevance of entities based ontheir features [5]. Textual as well as semantic representations of such featureshave been investigated. Textual investigations focus on the most importantterms. Having extracted those from textual descriptions, the recommendationalgorithm computes similarities between entities [6, 7]. The system recommendsthe most similar entities for a given entity (e.g., based on a weighted term over-lap). The need to handle multi-lingual content, homonyms, synonyms as wellas anaphors constitutes an additional challenge for approaches utilizing textualdata on items. The increasing availability of so-called linked open data enablesitem comparison based on well-structured, “semantic” features. Content-basedapproaches are generally less personalized compared to collaborative methods,due to focusing on item features rather than user features.

Semantic Recommenders Semantic recommenders compute recommenda-tions using semantically represented data. Blanco-Fernandez et al. [8] introducedthe AVATAR TV program recommender system. Their approach uses an ontol-ogy of TV-related entities. The authors determine an entity’s relevance basedon semantic intensity, connectedness and user preferences. A relevance thresholdallows disregarding unappealing TV programs reducing computational complex-ity. Cantador et al. [9] present the News @ hand system for recommending newsarticles based on semantic relations. Their approach encapsulates recommenda-tion algorithms using semantic similarities for either individual users, groups ofusers or even user clusters. Challenges building semantic recommenders consistin defining adequate metrics and efficient prediction models considering the se-mantics of the edges and reducing the noise in large datasets [10, 11]. Semanticrecommenders can be seen as hybrid recommenders due to the fact that collabo-rative and content-based knowledge (semantically represented) are combined forimproving the recommendation quality [12].

2.1 Problems with existing Approaches

Current recommender approaches are usually tailored for one specific scenarioeven though approaches considering several criteria at a time are reported to

provide superior results [13]. Choosing an adequate parameter setting requiresexpert knowledge with respect to the specific scenario [4]. Furthermore, mostrecommender systems cannot handle large semantic networks with differentlylabeled edges but rely on bipartite datasets (such as preference relations). Exist-ing hybrid recommenders often statically combine a content-based and a collab-orative approach based on expert defined rules. Such adaptions require manualadjustments once a new recommender system is started. Most existing frame-works for processing semantic data do not fulfill the requirements of opennessand flexibility. Many systems are designed for a single scenario but lack exten-sibility and the support for several semantic relationship types. In the followingsection, we introduce our recommender framework that provides a comprehen-sive collection of components for processing large semantic data collections andfor building adaptive recommender applications.

3 Problem description

In our research, we investigate the problem of combining different recommenderapproaches in one system and analyze approaches for the personalized aggre-gation of suggestions from different agents taking into account individual userpreferences.

3.1 Computing Recommendations based on Semantic Data Sets

In recent years, the amount of information published as Linked Open Data growsrapidly. For the entertainment domain several large data sets are available as “se-mantically represented data” providing comprehensive knowledge about movies,actors, singers, authors, songs, etc. These datasets (e.g., DBpedia) contain datafor a huge number of entities and the relations between them. Unfortunately, theunified knowledge representation based on a huge graph does not imply a unifiedsimilarity measure for entity types. Due to the fact that relations have differentsemantic meanings, additional effort is required, to define similarity measuresoptimized for computing recommendations based on the relations. In general,encyclopedic knowledge (e.g., the relationship movie-director) needs to behandled differently compared to user generated knowledge (e.g., the relationshipmovie-tags). Thus, a recommender system tailored to heterogeneous semanticdata sets should provide several different algorithms optimized to the specificproperties of the respective relationship sets.

3.2 Aggregating Recommendations from Different Agents

An advanced recommender system should be able to consider many differentcriteria (semantic relationships) when computing personalized recommendations.Since new semantic datasets might become available in the future, recommendersystems should support the integration of new recommender strategies as wellas the integration of additional data sources, thus keeping the recommender

system open and adaptive. Due to the fact, that every user has her individualpreferences, recommender systems should learn optimal, personalized weights foreach criterion. The weights can be learned based on user feedback (e.g., ratings)and should be continuously adapted to match the changes within her preferences.User preferences may relate to the properties of recommended entities (similarityof a suggested entity with the user profile) or to properties of the result set (e.g.,diversity, coverage).

In the next section, we present an approach that allows us to build an adap-tive recommender system for heterogeneous semantic dataset taking into accountthe discussed challenges.

4 Approach and System Architecture

We define an agent-based architecture for a recommender system optimized forsemantic datasets. Each semantic relationship is wrapped by at least one agent.The wrapping allows us to learn optimized recommender models for each rela-tionship set. A manager agent aggregates the agents’ recommendations into oneresult list considering individual user preferences. This approach ensures highflexibility, since we can easily add new data or algorithms (wrapped in an agent)to the system. We achieve the personalization by combining recommendationsfrom different agents based on user specific weights as well as by combiningresults computed based on (“general”, content-based) encyclopedic data andresults based on user ratings (“collaborative data”). This pattern allows us toprovide personalized recommendations using encyclopedic data. Thus, users canshare recommender agents based on encyclopedic data, significantly reducing theamount of required resources.

Figure 1 visualizes the system’s architecture. The recommender system’s ar-chitecture follows the idea that users demand items relevant according to sev-

costumedesigner

composerwriter

Σ ΣΣ

directoractor RDFRDF RDF RDF RDF

Fig. 1. The movie recommender system’s architecture: Each semantic relationship isrepresented by one recommender agent optimized with respect to its specific character-istics. The resulting recommendations are aggregated taking individual user preferencesinto account. The application server provides users with personalized recommendationsas a web application. Since each user has individual preferences, not every semanticrelationship set must be considered by every personal result aggregation agent.

eral criteria such as preferred genres, favorite actors, and directors. Additionally,users may have different expectations to the list of recommended items: Someusers may prefer homogeneous result list (strongly focused on “globally” opti-mal recommendations); some may prefer diverse result lists (created based on“locally” optimal results from each agents), animating them to browse throughthe results and to discover relevant, but less popular items. In the movie recom-mendation domain the system must decide whether the user in interested in themost relevant movie according to actor, director, etc. or whether the user prefersmovies having the highest score over all criteria. For this reason, our movie rec-ommender provides different algorithms for combining the recommender agents’results.

4.1 Computing Recommendations

Semantic datasets represent knowledge as a graph consisting of nodes and edges.Nodes represent entities while edges describe the relationship between entities.Both nodes and edges are annotated with features describing them in detail.We must define appropriate similarity measures to compute recommendationsbased on semantic data. In general, entities connected by several parallel pathsare more similar than entities connected by a single path. Additionally, we con-sider the path length: Entities connected by short paths (consisting of two edges)are more similar than entities connected by longer paths (e. g., consisting of fouredges). We need to assign appropriate edge weights to algebraically computerecommendations on the semantic graph. The weights must reflect the enti-ties’ similarity induced by the semantic meaning (“label”) as well as the edge’srelevance. Usually, edge weights are scaled to values in the interval [0; 1]. Wecalculate the similarity between two entities by analyzing all parallel paths be-tween them with a defined length. The weights for each path are computed bymultiplying the edge weights along the path. Weights computed for the parallelpaths are summed up. This ensures that two entities are the more similar themore parallel paths exist and the higher the weights are for the connecting path.

4.2 Combining Results from different Recommender Agents

A movie recommender should consider several criteria when computing recom-mendations. Our movie recommender wraps each criterion into one agent. Thesystem delegates user requests to a set of appropriate agents. In a second step,the manager agent combines the recommender agents’ results. The manageragent has different aggregation approaches available:(1) The final list contains each recommender agent’s best ranked entities. Thiscreates a heterogeneous recommendation list and ensures that the suggested en-tities are highly relevant according to at least one criterion. E.g., in a movierecommender scenario, the aggregated recommendation list contains the moviemost similar to the given user profile based on the relationship movies-actors

along with the entity most similar according to the relationship movie-genre.Since this strategy disregards the sum over the relevance predictions, it might

fail to recommend movies relevant to several criteria but not listed at the top ofany recommender agent’s list.(2) We aggregate the recommender agents’ suggestions by computing a weightedsum of their relevance predictions. This approach follows the idea, that moviesare the more relevant the more recommender agents assigned them a positiverelevance rating and the higher these relevance scores. This is based on theobservation that good recommendations should be relevant according to severalcriteria. The weights for each recommender agent control the importance of eachcriterion to the user.Both approaches have specific advantages. The aggregation of results by sum-ming up the relevance predictions (CombSUM [14]) shows a higher recommenda-tion precision (in the evaluation using a rating datasets); but the results containa higher fraction of popular, probably already known movies. Aggregating thebest ranked movies from each agent increases the result list’s diversity and mo-tivates the user to discover new, less popular movies.

5 A Hybrid Movie Recommender

Based on the presented architecture, we have implemented a movie recommendersystem (in Java). We use Freebase and IMDb as semantic data sources. Thedata is represented as graph consisting of approximately 6 million entities and 29million edges (Table 1). The user interface is implemented as a web application(using Grails).

Table 1. The Table lists the number of nodes and edges extracted from IMDb. Thisdata provide the basis for the implemented movie recommender prototype. The entitytype misc annotates various persons supporting the production such as location scoutsor stuntmen.

entity type # entities relationship set # edges density

actors 2,269,883 movies – actors 16,381,711 1.76E-06composers 113,759 movies – composers 635,377 2.73E-07directors 244,852 movies – directors 1,381,188 5.28E-07genres 30 movies – genres 1,267,667 6.07E-07keywords 115,359 movies – keywords 3,845,036 1.65E-06misc 601,563 movies – misc 3,230,682 9.24E-07writers 334,269 movies – writers 2,253,637 7.98E-07

movies 2,043,004

Queries and User Profiles Queries for the semantic recommender system con-sist of a list of entities describing user preferences, e. g. movies a user liked.Keeping the entrance barrier as low as possible, users may use the recommendersystem anonymously. In this case, the system stores user data in the session andusers have to enter their favorite entities explicitly (Fig. 2). The auto-completion

Fig. 2. The screenshot shows how users define an entity based query in the web ap-plication. The system compares the character input provided by the user with movietitles (“entity labels”) and lists the most similar movies. This allows the system to useunique entity URIs avoiding linguistic problems, e. g. ambiguity of words.

ensures that a valid entity input is found. Due to the fact that the IMDb andFreebase datasets are user-generated, the number of edges (data connectedwith the movies) varies between entities. In general, the more popular a moviethe more details are available for its node representation. Additional to thesemantic dataset, IMDb provides text based movie descriptions and movie re-views. These data are handled using standard text processing techniques suchas language detection, stemming, and stop word removal.

Recommender Agents As discussed in Section 4.1, we model each semantic re-lationship with one agent. We restrict each agent to a bipartite dataset. Pathsconnecting three or more entity set (e. g., movie-actor-location-movie) aredisregarded, since the semantic meaning of “complex” paths is controversialand may confuse users. Experiments on benchmark data sets have shown thatcomplex paths barely affect predicted movies’ relevance. Moreover, we limit thepath length to 4, ensuring that users understand why an entity has been recom-mended. Considering longer paths increases the computational complexity, but(in general) does not improve the recommendation quality.The movie recommender system integrates agents computing recommendationsbased on either semantic relationship sets (e. g., movie-actor) or on text-basedsimilarities (e. g., movie-plot). Recommender agents based on semantic rela-tionship sets determine the entities best-connected according to the respectiveentities’ relationship set (e. g., movies) to the entities the user liked in the past(see Fig. 3). The considered paths are visualized to provide explanations to theuser. The text-based recommender agents compute the similarity between twomovies based on the assigned texts’ similarity. We compute the term overlap assimilarity measure for texts [15]. We calculate each term’s weight based on thetf-idf-value (term frequency – inverse document frequency) of terms appearingin the considered texts (e.g. plot); stop words are excluded.

Combination Strategies The recommender strategies presented in the previousparagraph compute recommendations based on one semantic relationship set.The results provided by such “simple” recommendation agents are aggregatedinto a unified result set. We implement the aggregation by summing up the rel-evance ratings. The weights assigned to each recommender agent are predefined(computed based on a training data set). Users may adapt the weights accordingto their individual preferences. This allows the system to provide personalizedrecommendations based on the “simple” recommender agents. In addition tothe weights, users may influence the result set’s diversity by assigning higherweights to movies relevant according to many different criteria. Assigning higherweights to movies recommended by many recommender agents reduces the riskof suggesting bad movies.

Fig. 3. The Figure shows how recommendations are visualized. The left area displays alist of aggregated recommended movies (ordered by relevance). For each recommendedmovie the system provides an explanation as well as additional information such asIMDb’s description and Freebase’s data. Additionally, users may use the links toYouTube to watch the movie’s trailer. Movie descriptions and explanations (e. g., thegraph deployed for computing the results) are presented in the page’s right section.Users may explore each single agent’s recommendations by browsing the tabs Actor,Director, . . . (located below the query input field). The button Refine allows usersto personalize the weights for each recommender agent. Moreover, users may activateadditional filters for movie types (TV serial, movie, video game), for age classificationas well as for controlling the aggregated result list’s diversity.

5.1 User-centric Evaluation

We conducted a user study to evaluate our recommender system. In discussionswith participants, we realized that movies unknown to the user cause problems inthe evaluation: On the one hand, users expect the recommender to provide them

with unknown entities (following the idea that a recommender should providenew suggestions); on the other hand, users cannot assess the system’s recom-mendation quality, if all recommended items are unknown to them. The meta-recommender (aggregating the results from the “simple” recommender agents)should consider the popularity of entities and ensure that the result sets pre-sented to users contain popular as well as less popular entities to cope with thisproblem: Relevant popular entities (probably known to users) improve the trustin the system. Less popular entities (and thus probably unknown to users) coverthe requirement for providing serendipitous, new movies. Especially for “un-known” movies, the system should provide human understandable explanationsdescribing why the suggested movie matches the individual user preferences andmotivating users to spend time or money on watching it. Explanations generatedbased on the semantic data are very helpful, since they describe the aspects inwhich a recommendation is potentially relevant to users. Additional information,such as trailers, images, reviews or plot summaries help users to get a compre-hensive impression of the movie without much effort.When discussing with participants about the movie recommender, we foundthat the users’ preferences differ from one another. E.g., some participants areonly interested in the final result list and the movie trailers, while others pre-fer browsing the recommendations from the different agents in detail. Moreover,each user has specific expectation what criteria a good movie should fulfill. Thecombination of different recommender agents has been seen as a good approachto consider different preference schemes. The explanations help users to improvethe requests and to adapt the recommender agent’s weighting scheme.

5.2 Data-driven Evaluation

We evaluate our recommender based on a dataset. For ensuring the results’ repro-ducibility, we use a movie dataset originally created for the second internationalWorkshop on Information Heterogeneity and Fusion in Recommender Systems4.The dataset has been created by combining data from the MovieLens datasetand IMDb. In contrast to the data used in the movie recommender web appli-cation, the HetRec dataset provides only a fraction of movie features; ratings(extracted from MovieLens) describe at what time a movie has been rated bya user. We try to predict what movies a user will rate in the future (based ondata about the rating behavior in the past). We split the dataset based on therating timestamps into a training set and a test set. Users having less than tenratings in at least one of the sets are filtered out to ensure that an adequatenumber of ratings exists while learning and evaluating the recommenders (in theevaluation 760 user profiles have been considered). We compute the recommen-dation quality for each recommender agent, such as the actor, director, andgenre agents. In the next step, we create a recommender ensemble aggregatingthe rating predictions using weighted CombSUM. We compute the optimal weightfor each agent over all users (“global optimization”). In addition, we learn for

4 HetRec ’2011, http://ir.ii.uam.es/hetrec2011/

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

0.20

MAP

Memory-based

Recommender

(Path length = 2)

Memory-based

Recommender

(Path length = 4)

Personalized Ensemble

Recommender

(personalized weights)

Ensemble

Recommender

(globally optimized weights)

Fig. 4. The Figure depicts an excerpt of our evaluation. The agents’ weights in the en-semble are displayed in brackets. The recommender ensembles achieve precision supe-rior to any single semantic relation based recommender. The personalized recommenderensembles achieve an improvement compared to the ensembles based on globally opti-mized weights.

each user an optimized recommender ensemble resulting in personalized weight-ing schemes for the recommender agents. The evaluation (Fig. 4) shows that thecombination of different content-based recommenders highly improves the preci-sion. The recommender ensembles’ (based on globally optimized weights) meanaverage precision (MAP) outperforms the best one-feature based recommenders.The personalized recommender ensemble achieves even superior MAP. Thus, al-lowing users to adjust the weights for the respective movie features as well aslearning the optimal weights based on user feedback is a promising approach forpersonalized recommendation.

6 Conclusion and Outlook to Future Work

In this paper, we present a framework for efficiently processing large semanticdatasets (by means of a variety of supported recommender models). We showhow to create an adaptive movie recommender system based on this framework.Wrapping the recommender algorithms and semantic relationships in agents al-lows us to integrate new recommender algorithms and relevance criteria at run-time. For combining the recommender agents’ results different algorithms havebeen implemented controlling the results’ diversity and reliability. We achievepersonalization of the results by learning weighting schemes, assigning personal-ized weights to every agent. This weighting is subject to be continuously adaptedby algorithms based on user feedback. The evaluation shows that high-qualitypersonalized recommender systems can be implemented based on comprehensive,semantic knowledge. Providing explanations for recommendations improves thetrust in the system (especially when new, unknown movies are suggested). En-

abling the user to explicitly control the weights for criteria used to determine therelevance of recommended entities allows users to adapt the system to the indi-vidual preferences quickly. In future work, we will focus on improving the supportof unstructured data. Therefore, we will integrate an additional component forNamed Entity Recognition and Name Entity Disambiguation in our framework.Additionally, we plan to test the context-aware adaption of personalized weightsfor the agents.

References

1. Schein, A., Popescul, A., Ungar, L., Pennock, D.: Methods and metrics for cold-start recommendations. In: Proc. of the 25th ACM SIGIR Conf. on Research andDevelopment in IR. SIGIR ’02, ACM 253–260

2. Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommendersystems. IEEE Computer (2009)

3. Adomavicius, G., Zhang, J.: On the stability of recommendation algorithms. In:Proceedings of the fourth ACM conference on Recommender systems. (2010) 47–54

4. Adomavicius, G., Tuzhilin, A.: Towards the next generation of recommender sys-tems: A survey of the state-of-the-art and possible extensions. Transactions onKnowledge and Data Engineering 17 (2005)

5. Lops, P., de Gemmis, M., Semeraro, G.: 3. In: Content-based RecommenderSystems: State of the Art and Trends. Springer (2011)

6. Lang, K.: Newsweeder: Learning to filter netnews. In: Proc. of the 12th ICML,Morgan Kaufmann (1995) 331–339

7. Melville, P., Mooney, R., Nagarajan, R.: Content-boosted collaborative filteringfor improved recommendations. In: 18th national conf. on AI, Menlo Park, CA,USA, AAAI (2002) 187–192

8. Blanco-Fernandez, Y., Pazos-Arias, J., Gil-Solla, A., Ramos-Cabrer, M., Lopez-Nores, M.: Providing entertainment by content-based filtering and semantic rea-soning in intelligent recommender systems. IEEE Transactions on Consumer Elec-tronics 54(2) (2008)

9. Cantador, I., Bellogin, A., Castells, P.: News@hand: A semantic web approachto recommending news. Adaptive Hypermedia and Adaptive Web-based Systems5149 (2008) 279–283

10. Kunegis, J., Schmidt, S., Lommatzsch, A., Lerner, J., Luca, E.W.D., Albayrak, S.:Spectral analysis of signed graphs for clustering, prediction and visualization. In:Proc. SIAM Int. Conf. on Data Mining. (2010) 559–570

11. Fouss, F., Yen, L., Pirotte, A., Saerens, M.: An experimental investigation of fivegraph kernels on a collaborative recommendation task. In: Proc. of the 6th IEEEIntl. Conf. on Data Mining (ICDM 2006). (2006)

12. Katz, G., Ofek, N., Shapira, B., Rockach, L., Shani, G.: Using wikipedia to boostcollaborative filtering techniques. In: Proceedings of the fifth ACM conference onRecommender systems. RecSys ’11, New York, NY, USA, ACM (2011) 285–288

13. Bell, R., Koren, Y.: Lessons from the netflix prize challenge. ACM SIGKDDExplorations 9(2) (2007)

14. Hull, D., Pedersen, J., Schutze, H.: Method combination for document filtering.In: Proc. of the 19th ACM SIGIR conf. SIGIR ’96, NY, NY, USA (1996) 279–287

15. Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Addison Wesley(1999)