a multi-criteria collaborative filtering recommender ... · pdf filea multi-criteria...

Electronic Commerce Research and Applications xxx (2015) xxx–xxx

Contents lists available at ScienceDirect

Electronic Commerce Research and Applications

journal homepage: www.elsevier .com/ locate/ecra

A multi-criteria collaborative filtering recommender system for thetourism domain using Expectation Maximization (EM) and PCA–ANFIS

http://dx.doi.org/10.1016/j.elerap.2015.08.0041567-4223/� 2015 Elsevier B.V. All rights reserved.

⇑ Corresponding authors.E-mail addresses: [email protected] (M. Nilashi), othmanibrahim@

utm.my (O. bin Ibrahim).

Please cite this article in press as: Nilashi, M., et al. A multi-criteria collaborative filtering recommender system for the tourism domain using ExpeMaximization (EM) and PCA–ANFIS. Electron. Comm. Res. Appl. (2015), http://dx.doi.org/10.1016/j.elerap.2015.08.004

Mehrbakhsh Nilashi a,⇑, Othman bin Ibrahim a,⇑, Norafida Ithnin a, Nor Haniza Sarmin b

a Faculty of Computing, Universiti Teknologi Malaysia, 81310 UTM Johor Bahru, MalaysiabDepartment of Mathematical Sciences, Faculty of Science, Universiti Teknologi Malaysia, 81310 UTM Johor Bahru, Malaysia

a r t i c l e i n f o a b s t r a c t

Article history:Received 12 November 2014Received in revised form 10 August 2015Accepted 10 August 2015Available online xxxx

Keywords:Expectation MaximizationClusteringPCAANFISMulti-criteria CFTourist recommendation

In order to improve the tourist experience, recommender systems are used to offer personalized informa-tion for online users. The hotel industry is a leading stakeholder in the tourism sector, which needs toprovide online facilities to their customers. Collaborative Filtering (CF) techniques, which attempt to pre-dict what information will meet a user’s needs based on data coming from similar users, are becomingincreasingly popular as ways to combat information overload. They use a single rating as input. However,the multi-criteria based CF presents a possibility to provide accurate recommendations by consideringthe user preferences in multiple aspects and they can be an appropriate choice for the tourist. In thispaper, we propose a new hybrid method for hotel recommendation using dimensionality reductionand prediction techniques. Accordingly, we have developed the multi-criteria CF recommender systemsfor hotel recommendation to enhance the predictive accuracy by using Gaussian mixture model withExpectation Maximization (EM) algorithm and Adaptive Neuro-Fuzzy Inference System (ANFIS). We havealso used the Principal Component Analysis (PCA) for dimensionality reduction and to address multi-collinearity induced from the interdependencies among criteria in multi-criteria CF dataset. Our experi-ments confirmed that the proposed hybrid method achieved high accuracy for hotel recommendation forthe tourism sector.

� 2015 Elsevier B.V. All rights reserved.

1. Introduction

Tourism as a strategic sector has provided a significant contri-bution to the economies of many nations around the world (WanLee and Brahmasrene 2013). It has provided a remarkable impacton the global economic development in which the contributionof this sector for the employing people and economic activity haveestimated around 7.6% of global employment and US$ 5474 billionof 9.4% of global GDP (World Travel and Tourism Council 2009).According to the World Tourism Organization (2006), it is pre-dicted that by 2020, tourist arrivals around the world will increaseby over 200%. Impressive changes in the Information and Commu-nications Technologies (ICTs) and the Internet has resulted inextensive transformation of the industry. According to the TravelIndustry Association of America (www.tia.org) cited by (Lucaset al. 2013), in 2003, the major United States adult population(around 30%) has used Internet as a tool to check prices and sched-ules and seek information regarding destinations. 66% of tourists

booked travel needs using the Internet. In addition, the ICTs haveconsiderably improved the innovations in the tourism sector inmanagement and marketing of tourism packages and broughtabout new paradigm shifts in this sector as discussed in manyresearches (Polo Peña et al. 2013, Chiu et al. 2009, Popescu andGrefenstette 2011, Morrison et al. 2001, Singh and Kasavana2005, Connolly and Lee 2006, Pan et al. 2007, 2011, Buhalis andLaw 2008, Xiang and Pan 2010, Buhalis and O’Connor 2005).

Tourism is an activity closely linked with personal interests andpreferences (Chou et al. 2008, Wang et al. 2002, Benítez et al. 2007).Recommender systems designed in the tourism domain and appli-cations known as Travel Recommender Systems (TRSs) or destina-tion recommendation system, are a valuable tool for customersand travel agencies (Loh et al. 2004, Werthner and Ricci 2004). Thatis why many tourism web applications incorporate recommendersystems. With this, they try to simulate the interaction with ahuman travel agent. Through the introduction of tourism recom-mending systems, tourists can easily access information about thehotels they need, thus, resulting in shorter lead-time for bookings,making last-minute decisions and generally, tailoring their prefer-ences. Tourism recommender systems are a class of intelligent sys-tems that render tourism related information services in the form of

ctation

http://www.tia.org

http://dx.doi.org/10.1016/j.elerap.2015.08.004

mailto:[email protected]




http://www.sciencedirect.com/science/journal/15674223

http://www.elsevier.com/locate/ecra


2 M. Nilashi et al. / Electronic Commerce Research and Applications xxx (2015) xxx–xxx

guides and suggestions to users. This class of systems can bebroadly classified as web-based tourism recommender systemsand mobile recommender systems. Web-based tourism recom-mender systems are intelligent systems that are usually embeddedin e-Tourism portals in order to deliver travel information guide,travel advice and travel planning recommendations.

Collaborative Filtering (CF) can be an appropriate choice fortourism object recommendation. Recommender systems basedon CF are those in which recommendations only consider the sim-ilarity of terms between users. That is, collaborative systems rec-ommend items that other users with similar interests like.However, traditional CF use a single rating as input, usually anoverall numerical ranking by a user to an item. Hence, in someapplications, this kind of recommendation does not meet users’personalized needs and multi-criteria ratings are considered.

Multi-criteria based CF presents a possibility of providing accu-rate recommendations by considering the user preferences in multiaspects of items. According to Adomavicius and Kwon (2007), pureCF-based recommender systems rely solely on product ratings pro-vided by a large user community to generate personalized recom-mendation lists for each individual online user. In traditional CFsystems the assumption is that customers provide an overall ratingfor the items which they have purchased, for example, using a5-star rating system. However, given the value of customer feedbackto the business, customers in some domains are nowadays giventhe opportunity to provide more fine-grained feedback and to rateproducts and services along various dimensions (Jannach et al.2012a, Adomavicius et al. 2011, Nilashi et al. 2014c). According toAdomavicius and Kwon (2007), multi-criteria system providesmore information about user preferences than a single-rating sys-tem. And by adopting a decision theory, multi-criteria systemscan provide rich tools for system designers to build more interest-ing systems as well (Lakiotaki et al. 2011). In addition, nowadays,allowing online visitors to provide fine-grainedmulti-criteria ratingfeedback is common in the travel and tourism sector. TripAdvisortourism is one of the popular platforms which has provided usersto rate hotels according to different criteria such as cleanliness,service or value for money (Nilashi et al. 2015a).

Adomavicius and Kwon (2007) developed a number of basicstrategies to exploit multi-criteria ratings for improving the predic-tive accuracy of a recommender in terms of typical informationretrieval measures. Later on, a number of additional techniquesto leverage the detailed ratings in the recommendation processwere proposed (Liu et al. 2011, Sahoo et al. 2012, Shambour andLu 2011a,b, Jannach et al. 2012a,b, 2014, Nilashi et al. 2014a,b,2015a). The work presented in this paper continues on these linesof research.

Overall, our work is similar to the works of which has used,clustering, combined with methods such as methods developedby Adomavicius and Kwon (2007) and Jannach et al. (2012a,b),where we use Neuro-Fuzzy techniques to predict the overall rat-ings from the given multi-criteria ratings. Furthermore, our workis, in some sense similar to that of Liu et al. (2011) and Nilashiet al. (2015a) where we apply clustering.

From the literature on multi-criteria CF, at the moment there isno implementation of PCA, Neuro-Fuzzy and clustering recom-menders in multi-criteria CF, and this research tries to develop arecommender system in the tourism sector based on theseapproaches. Thus, in order to improve predictive accuracy ofmulti-criteria CF, we propose a newmodel using fuzzy logic, neuralnetworks and clustering techniques. To the best for our knowledge,an artificial intelligence method (ANFIS), clustering method (EM)and dimensionality reduction (PCA) is applied for the first timein this research in the context of multi-criteria CF recommenda-tions in particular for hotel recommendation based on multi-criteria CF.

Please cite this article in press as: Nilashi, M., et al. A multi-criteria collaborativMaximization (EM) and PCA–ANFIS. Electron. Comm. Res. Appl. (2015), http://

1.1. Recommendation problem

In multi-criteria CF problem, there are m users, n items and kcriteria in addition to the overall rating. Users have provided anumber of explicit ratings for items; a general rating R0 must bepredicted in addition to k additional criteria ratings (R1,. . ., Rk). Itcan be configured to push new items to users in two ways, eitherby producing a Top-N list of recommendations for a given target,or by predicting the target user’s likely utility (or rating) for a par-ticular unseen item. We will refer to these as the recommendationtask and the rating prediction task in multi-criteria CF, respec-tively. Fig. 1 demonstrates the multi-criteria CF problem in caseof prediction and recommendation tasks for an active user Ua

and active item Ij.Recommendation is a list of N products, TP = {Tp1, Tp2,. . ., TpN},

that the active user will like the most. The recommended list usu-ally consists of the products not already purchased by the activecustomer. This output interface of multi-criteria CF algorithms isalso known as Top-N recommendation. Multi-criteria CF algo-rithms represent the entire m � n � k user-item-criteria data as atensor of ratings, A. Each entry ai,j in tensor A as shown in Fig. 1represents the preference score (ratings) of the ith user on thejth item (hotel) as overall preference in addition to criteria ratingsin the 3rd dimension. Each overall and criteria rating is within anumerical scale and it can as well be 0, indicating that the userhas not yet rated that item.

Thus, the algorithm for a multi-criteria recommender systemcan be extended from a single-rating recommender system. Fol-lowing this approach, Adomavicius and Kwon presented twoapproaches to leverage multi-criteria ratings through extendingsingle-rating CF (Adomavicius and Kwon 2007). One is computingthe overall user similarity through aggregating the similarities cal-culated from each individual criterion (Adomavicius and Kwon2007).The other approach is aiming for a more holistic calculationof user similarity through multidimensional distance metrics. Eachrating is presented in a multivariable format, such as

ru;i ¼ f ðr0; r1; . . . ; rkÞ ð1Þwhere r0 is the overall rating that user u has rated item i and r1,. . ., rkpresents the rating of criterion 1,. . ., k.

In this paper, we use Pearson correlation coefficient approachfor users and items similarity calculation. The Pearson correlationcoefficient (the most commonly used weighting approach) mea-sures the degree to which a linear relationship exists betweentwo variables (McLaughlin and Herlocker 2004). In this research,it is used to evaluate how a certain user is related to an active userwith respect to their preferences on given items. The Pearson cor-relation coefficient is derived from a linear regression model (seeEq. (2)). The range of the result of the equation is from �1 to 1,inclusively. More specifically, a result of ‘‘1” means the two usersare positively related (absolute agreement), ‘‘�1” denotes theyare negatively related (absolute disagreement), and ‘‘0” indicatesno relation at all.

simu;v ¼

ifXi2Iuv

ðru;i � ruÞ2 ¼ 0

0 or ifXi2Iuv

ðrv;i � rvÞ2 ¼ 0

or if jIj ¼ 0Pi2Iuv

ðru;i�ruÞðrv ;i�rv ÞffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPi2Iuv

ðru;i�ruÞ2q ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiP

i2Iuvðrv ;i�rv Þ2

q Otherwise

8>>>>>>>>>>><>>>>>>>>>>>:

ð2Þwhere Iu,v in Eq. (2) is a set of items that both user u and v rate, Iu is aset of items that user u rates and Iv is a set of items that user v rates.

e filtering recommender system for the tourism domain using Expectationdx.doi.org/10.1016/j.elerap.2015.08.004


MC-CF algorithms

II II … IIjj II

pp II

nn

U

U

Ua

Um

C1

...

Criteria

Items (Hotels)

Users

Ck ,a jP Prediction on item jI for the

active user aU

{ 1 2, ,...,p p pNT T T }top -N list of

aU

Ou

tpu

t In

terf

ace

PredictionPrediction

RecommendationRecommendation

Active User Item for which prediction is sought

Fig. 1. The multi-criteria CF processes.

M. Nilashi et al. / Electronic Commerce Research and Applications xxx (2015) xxx–xxx 3

2. Related work

The fuzzy logic field has grown considerably in a number ofapplications across a wide variety of domains like in the semanticmusic recommendation system (Lesaffre and Leman 2007), movierecommendation (Nilashi et al. 2014a,b) and product recommen-dations (Cao and Li 2007, Stormer et al. 2006). Castellano et al.(2007) developed a Neuro-Fuzzy strategy combined with soft com-puting approaches for recommending URLs to the active users.They used fuzzy clustering for creating a user profile consideringthe similar browsing behavior. de Campos et al. (2008) proposeda model by combining Bayesian network for governing the rela-tionships between the users and fuzzy set theory for presentingthe vagueness in the description of users’ ratings. A conceptualframework based on fuzzy logic-based was proposed by Yager(2003) to represent and then justify the recommendation rules.In the proposed framework, an internal description of the itemswas used that relied solely on the preferences of the active user.Carbo and Molina (2004) developed an algorithm based on CF thatratings and recommendations were considered as linguistic labelsby using fuzzy sets. A model proposed by Pinto et al. (2012) thatcombines fuzzy numbers, product positioning (from marketingtheory) and item-based CF.

In the context of web recommendation systems, traditionalsingle-rating CF recommender systems have been highly success-ful however, the research area regarding of the CF with multi-criteria ratings for items has been rarely touched and fairly thisissue is unexplored. Especially, few number of researches havebeen conducted in the tourism recommendations context usingmulti-criteria ratings.

According to Stock et al. (2005), tourism is a worthy area for theapplication of Artificial Intelligence (AI), and, especially, in DecisionSupport Systems (DSSs) and recommender systems (Felfernig et al.2007). Due to the importance of tourism as a strategic sector, sev-eral attempts have been made in developing recommender sys-tems for the tourism domain (Wallace et al. 2003, Loh et al.2004, Castillo et al. 2008, Ricci and Nguyen 2007). These researchesoften have used AI techniques for their purpose. For example,Schiaffino and Amandi (2009) uses intelligent agents, Lenar andSobecki (2007) and Ngai and Wat (2003) apply fuzzy approaches,Huang and Bian (2009) employs Bayesian networks and many


researchers have incorporated semantic approaches for the tour-ism domain (Jakkilinki et al. 2007, Kanellopoulos 2008). Berkaand Plnig (2004) developed a recommendation system for travelrecommender systems (TRSs) using fuzzy association rules. Opti-mization techniques have been used in many researches for tour-ism recommendation systems (Lee et al. 2009, Vansteenwegenand Souffriau 2011, Garcia et al. 2010, 2013, Castillo et al. 2008,Meehan et al. 2013). Several researchers also have used automaticclustering in the tourism recommender systems (Fenza et al. 2011,Castillo et al. 2008, Gavalas and Kenteris 2011, Noguera et al. 2012,Moreno et al. 2013, Batet et al. 2012, Kurata and Hara 2013,Ruotsalo et al. 2013, Lucas et al. 2013).

Collaborative and content-based approaches have been widelyused in recommendation systems for tourism domain. Vaca-tionCoach’s expert advice platform and Triplehop’s TripMatcher(Ricci 2002) are good examples of content-based systems. Basi-cally, these systems match the users characteristics, preferencesand needs with the set of destinations variables and features. Incase of collaborative type recommendation for tourism, there arefew works which have employed only the memory-based CFapproach which TripAdvisor is probably the most popular tourismrecommender system of this type. The recommendations inTripAdvisor are mainly based on ratings and comments collectedfrom users. In this way, the recommendation system follows theCF procedure and suggest items to the target/active by comparingitems of other users with similar interests like. In this type recom-mendation systems users are grouped based on items’ ratings.Hence, similar ratings will be in a one group. However, despitepopularity of CF recommendation systems, they also suffer fromtypical recommender systems drawbacks such as sparsity, scalabil-ity and cold-start.

The use of model-based CF approaches such as case-based rea-soning in recommender systems for tourism has been chosen toovercome some shortcomings associated with memory-basedapproaches. Accordingly, Ricci and Werthner (2002) have provideda recommender system for tourism domain based on case-basedreasoning, which data is gathered from existent external tourismportals. Lucas et al. (2013) implemented a recommendationmethodology in a recommender system in the tourism domain thatthe classification was based on association. In addition theirmethod was composed by several data mining algorithms;




however, the key aspect of the method was the join use of classifi-cation based on association and fuzzy sets.

Several researchers have attempted to combine collaborativeand content-based filtering in travel recommender systems(Pazzani 1999, Delgado and Davidson 2002). By hybrid of thesetwo approaches, they could achieve significant accuracy improve-ments on the travel recommender systems.

In case of multi-criteria CF, few researches has been conductedto develop the similarity calculation of the traditional memory-based CF approach to investigate multi-criteria rating (Tang andMcCalla 2009, Manouselis and Costopoulou 2007, Adomaviciusand Kwon 2007) that the similarities between users are estimatedthrough aggregating traditional similarities from individual criteriaor applying multidimensional distance metrics. In order to devel-oping the idea of Adomavicius and Kwon (2007), Sahoo et al.(2006, 2011) extended the flexible mixture model (FMM) devel-oped by Si and Jin (2003) to multi-criteria recommender systems.Li et al. (2008) presented a multicriteria rating approach toimprove personalized services in mobile commerce using multi-linear singular value decomposition (MSVD). The aim of theirpaper was to exploit context information about the user as wellas multi-criteria ratings in the recommendation process. Liu et al.(2011) presented a multi-criteria recommendation approachwhich is based on the clustering of users. Their idea is that for eachuser one of the criteria is ‘‘dominant” and users are groupedaccording to their criteria preferences. They applied linear leastsquares regression, assigning each user to one cluster, and evalu-ated different schemes for the generation of predictions. Theyapplied the methods on hotel domain dataset with five criteria,Value, Location, Rooms, Service and Cleanliness. Jannach et al.(2012a) further developed the accuracy of multi-criteria CF byproposing a method using SVR for automatically detecting theexisting relationships between detailed item ratings and the over-all ratings. In addition, the learning process of SVR models was peritem and user and lastly combined the individual predictions in aweighted approach.

Shambour and Lu (2011b) implemented a hybrid Multi-CriteriaSemantic enhanced CF (MC-SeCF) approach to alleviate limitationssuch as sparsity and cold-start of the item-based CF techniques.The experimental results on MovieLens dataset demonstrated theeffectiveness of their proposed approach in alleviating the sparsityand cold-start items problems. They achieved high accuracy andmore coverage in very sparse and new items datasets than thebenchmark item-based CF recommendation algorithms.

In this study, we consider the proposed method for Tourismdomain recommender systems. However, the method can also beadopted for e-business and e-government applications recom-mender systems Shambour and Lu (2011a,b) for e-business ande-government applications. In our proposed method for buildinga model using PCA, ANFIS and clustering, the explicit ratings areneeded. However, based on Nielsen’s 90-9-1 principle (2006) morepeople will lurk in a virtual community than will participate.Hence, with considering the Nielsen’s 90-9-1 principle, appropriatestrategies are required to be incorporated in multi-criteria CF suchas developed method by Shambour and Lu (2011b) which usessemantic information of items. Generally, we view the semanticbased approaches to be complementary to our method. The oppor-tunity for future work is therefore to combine the predictions Withrespect to the achieved improvements by Shambour and Lu(2011b), major problems such as sparsity and cold-start can beremarkably alleviated.

In some researches the multi-criteria has been incorporated tothe CF recommendation systems to improve the accuracy ofthese systems. As an example, Jannach et al. (2012b) showedthrough an empirical evaluation based on a real-world dataset from the tourism domain that the predictive accuracy of an


recommender systems can be significantly improved when themulti-dimensional rating information is taken into account. Theyevaluated the method on a real-world datasets provided by a majorEuropean tourism platform and Yahoo!Movies and compare it withstate-of-the-art baseline algorithms based on matrix factorization.

In the context of personalization applications, traditionalsingle-rating CF have been highly successful. However, theresearch area regarding of the CF with multi-criteria ratings foritems has been rarely touched and fairly this issue is unexplored.According to Adomavicius and Kwon (2007), the problem ofmulti-criteria recommendations with a single and overall ratingis still considered an optimization problem. In addition, comparedto approaches proposed in the literature over the last years thatrely on single-rating feedback, research on building recommendersystems using multi-criteria rating information is still limited.

Hence, in this paper we incorporate the multi-criteria ratings tothe CF for hotel recommendation in the tourism domain andpropose a new approach of recommendation using ExpectationMaximization (EM) clustering technique, the Principal ComponentAnalysis (PCA) and Adaptive Neuro-Fuzzy Inference Systems(ANFISs).

Thus, in comparison with research efforts found in the litera-ture, our work has the following differences. In this research:

� A hybrid recommendation model using EM, PCA and ANFIS isproposed for increasing the predictive accuracy of the multi-criteria CF in tourism domain.

� ANFIS is used for knowledge discovery from the multi-criteriaratings ratings provided by users without the human expertintervention.

� PCA is used for dimensionality reduction and dealing with themulti-collinearity problem exists in the multi-criteria ratings.

The remainder of this paper is organized as follows: Section 3provides the research methodology along with all approaches usedin the proposed model. Section 4 presents the multi-criteria mod-eling using proposed approaches the along with the evaluations.Finally, conclusions and future work is presented in the Section 5.

3. Methodology of research

This section introduces the proposed recommendation modelbased on multi-criteria ratings to be validated in a tourist recom-mender system. The general framework of proposed model isshown in the Fig. 2. As can be seen in this figure, the recommenda-tion model is composed by several data mining techniques; how-ever, in relation to prior researches, the key aspect of thismethod is the join use of supervised and unsupervised dimension-ality reduction techniques. Accordingly, in this study, we use ANFISas a supervised approach for constructing the models of prediction.However, prior to the ANFIS, we perform clustering on data usingEM algorithm as a unsupervised machine learning technique. Thedata clustering assists ANFIS to construct effective prediction mod-els of users and items. We also use PCA for dimensionality reduc-tion because the greatest source of difficulties in using ANFISmethod is the existence of ‘‘multi-collinearity” in many sets of datathat in this research PCA will overcome this problem. We selectANFIS approach to learn prediction models for users and items ineach cluster and then combine those models in a weightedapproach in the online phase. The form of discovered knowledgeusing ANFIS is fuzzy rules that are used for predicting overall rat-ings. The extracted rules are employed for prediction unknown rat-ings and also revealing real level of user preferences on items’features (criteria). In addition, non-stochastic uncertainty emerg-ing from vagueness and imprecision is handled using Membership



Tensor of Users Ratings

Users

Hotels

Criteria

Data pre-processing

Clustering Using Expectation Maximization Algorithm

5-Fold Cross Validation

ANFISi ANFIS Prediction Models (fi)

PCAi…

Clusteri

Cluster1Clustern

Retrieve Corresponding ANFIS Prediction Models (fi)

Return Rating Prediction

Finding Nearest Cluster

Select Active User and Unseen Items Identification

Recommendation and Prediction Tasks 2

1

Offline Phase

Calculating Criteria Ratings

Online Phase

Training Testing

Fig. 2. Proposed model for the multi-criteria CF.


Functions (MFs) produced by ANFIS. They are used for representa-tion and reasoning users’ behavior in providing rating based ontheir perception of items’ features. The MFs formed by ANFIS arecontinues and are more accurate in representing the features ofitems and user feedbacks. Furthermore, to prevent the problemof overfitting discussed in the previous researches (Jannach et al.2012a, Sen et al. 2009), checking data is used to minimize andsolve overfitting problem in the training data. Moreover, for pre-dicting the unknown overall ratings, we try to solve the sparsityproblem in criteria ratings by the neighborhood formation in eachcluster using Pearson correlation coefficient. It should be notedthat tensor decomposition techniques such as Higher Order Singu-lar Value Decomposition (HOSVD) can also be used.

Therefore, for alleviating sparsity problem in individual criteriaratings, after forming the neighborhood in each cluster, we haveused the average of the other criteria ratings as an estimate ofthe user’s rating. Two methods have been widely used in recom-mender systems for neighbors selection: the Top-N method andthe correlation weight threshold. In Top-N, a predefined numberof users with greatest correlation are selected and in the correla-tion weight threshold all users with similarity correlation exceed-ing a certain threshold are selected. We use the Top-N method asrecommended by Herlocker et al. (2002). To recommend an itemto a target user, first assign the target user u to one of the clustersdetermined in the offline phase. We thus calculate a ‘‘mean ratingvector” for the target user by calculating the user’s average ratingvalue for each dimension given the user’s past ratings. The result-ing rating vector is then compared to the center of each of the clus-ters and the user is finally assigned to the target cluster whosecenter has the highest similarity with the user’s mean rating vectorin terms of the Pearson correlation coefficient. It should be alsonoted that in the proposed recommendation model and the meth-ods used, K-Fold Cross-Validation (CV) (k = 5) is used for estimatingthe models using part of the training data and then validatingthose models with the remaining part of the training data.


3.1. ANFIS

The Fuzzy Logic (FL) was introduced by Zadeh (1965) to providea solution for decisions making and handle uncertanity in vague,ambiguous and imprecise situations (Prihasto et al. 2014). It repre-sents models or knowledge using IF-THEN rules (Nilashi et al.2015b,c). Neural Networks (NNs) is a powerful technique learn sys-tem behavior by using system input–output data. NNs have goodlearning and generalization capabilities. The capabilities of learn-ing and generalization in NNs enable the system to effectivelyaddress real-world problems. They can be a good choice when suf-ficient data be available for the problem. Then, NNs can effectivelysolve problems which cannot be solved or inefficiently solved byexisting techniques, including fuzzy logic.

Both fuzzy logic and NNs have been very successful in solvingmany real-world problems. However, both technologies have somelimitations as well which have prevented them from providing effi-cient solutions for multi-criteria CF problems. In fuzzy logic, it isusually difficult to determine the correct set of rules and member-ship functions from the users’ preferences in multi-criteria CF.Moreover, fine-tuning a fuzzy solution is even more difficult andtakes longer. In neural networks, it is difficult to understand the‘‘Black Box”, i.e., it is incomplete compared to a fuzzy rule basedsystem description.

An appropriate combination of these two technologies (Neuro-Fuzzy) can effectively solve the problems of fuzzy logic and neuralnetworks and, thus, can more effectively address the multi-criteriaCF problems. A Neuro-Fuzzy approach was used to take advantageof the neural network’s ability to learn, and membership degreesand functions of fuzzy logic. The weights of the neural networksare mapped to fuzzy logic rules and member functions. Expressingthe weights of the neural network by fuzzy rules also provides abetter understanding of the ‘‘Black Box” and thus helps the betterdesign of the neural network itself. Thus, while the learning of neu-ral network is parameterized by the variation in input data, the




learning of ANFIS is fixed by the rules and membership functionvalues that we define. A Neuro-Fuzzy system is functionally equiv-alent to a FIS. A FIS mimics a human reasoning process by imple-menting fuzzy sets and approximate reasoning mechanism thatuses numerical values instead of logical values. A FIS requires adomain expert to define the Membership Functions (MFs) and todetermine the associated parameters in both the MFs, and the rea-soning section. However, there is no standard for the knowledgeacquisition process and thus the results may be different if a differ-ent knowledge engineer is at work in acquiring the knowledgefrom experts.

A Neuro-Fuzzy system can replace the knowledge acquisitionprocess by humans using a training process with a set of input–output training dataset. Thus instead of dependent on humanexperts the Neuro-Fuzzy system will determine the parametersassociated with the Neuro-Fuzzy system through a training pro-cess, by minimizing an error criterion. A popular Neuro-Fuzzy sys-tem is called an ANFIS. ANFIS is a fuzzy system that uses artificialneural network theory to determine its properties (fuzzy sets andfuzzy rules).

The main objective of ANFIS modeling is to map the inputs to

outputs to find a function f_

for a given input vectorX ¼ ðx1; x2; x3; . . . ; xnÞ in order to predict output y as close aspossible to its actual output y. Assume m observations ofmulti-input–single-output data pairs be available such asX ¼ ðxi1; xi2; xi3; . . . ; xinÞ, andyi ¼ f ðxi1; xi2; xi3; . . . ; xinÞ ði ¼ 1;2; . . . ;mÞ ð3Þit is now possible to build a model using ANFIS in prediction task for

any new input vector X ¼ ðxi1; xi2; xi3; . . . ; xinÞ. This prediction, y_

i, is

an approximation of y that can be presented as

y_

i¼ f

_

ðxi1; xi2; xi3; . . . ; xinÞ ði ¼ 1;2; . . . ;mÞ ð4Þ

In that direction, the goal is to minimize the difference betweenthe actual output and the predicted one by determining an ANFISmodel.Xmi¼1

f_

ðxi1; xi2; xi3; . . . ; xinÞ � yi

� �2! min : ð5Þ

Indeed, in ANFIS the linguistic Takagi and Sugeno (TSK) type,fuzzy IF-THEN rules are used for prediction task. These rules are

generated by training the model to approximate f by f_

using mobservations of n-input–single-output data pairs (Xi, yi). ANFIShas a structure that consists of nodes and directional links throughwhich the nodes are connected. Back-propagation strategy is usedto train the MFs, while the least mean squares algorithm determi-nes the coefficients of the linear combinations in the consequentpart of the model. TSK type fuzzy IF-THEN rules are used in ANFISmodel, for example:

IFðx is A1Þ AND ðy is B1Þ; THEN f 1 ¼ p1xþ q1yþ r1IF ðx is A2Þ AND ðy is B2Þ; THEN f 2 ¼ p2xþ q2yþ r2

ð6Þ

where x and y are the inputs, fi is the output, pi, qi and ri are thedesign parameters that are determined by the users during thetraining process. Ai and Bi are the fuzzy sets according to pre-defined MFs. An ANFIS model with two inputs and two fuzzy rulesare implemented in Fig. 3.

The first hidden layer is for fuzzification of the input variables.The outputs of layer 1 are fuzzy membership grade of the inputs,which are given by:

O1i ¼ aAi

ðxÞ; i ¼ 1;2

O1i ¼ aBi�2

ðyÞ; i ¼ 3;4ð7Þ


where aAiðxÞ and aBi�2 ðyÞ are MF.There are fixed number of nodes in the second layer, labeled

with M. The outputs of the second layer can be defined as:

O2i ¼ wi ¼ aAi

ðxÞ � aBi ðyÞ; i ¼ 1;2 ð8Þwhere wis are so-called firing strength of the rules.

In the third layer, the number of nodes is also fixed, labeledwith N. It normalizes the rule strengths from the second layer.The output of this layer can be defined as:

O3i ¼ �wi ¼ wi

w1 þw2; i ¼ 1;2 ð9Þ

which are the so-called normalized firing strengths. The consequentparameters of the rule are determined in the fourth layer. The out-put of each node in this layer is the product of the normalized firingstrength and the polynomial defined in fuzzy rule, shown as:

O4i ¼ �wif i ¼ �wiðpixþ qiyþ riÞ; i ¼ 1;2 ð10ÞThe fifth layer computes the overall output as the summation of

all incoming signals. There is only one node in this layer, labeledwith S. Hence, the output of this layer can be presented as:

O5i ¼

X2i¼1

�wif i ¼P2

i¼1wif i� �w1 þw2

ð11Þ

There are two adaptive layers in the ANFIS architecture, namelythe first layer and the fourth layer. There are three modifiableparameters in the first layer ai, bi so-called premise parameters,which are related to the shape of the MF. In the fourth layer, thereare also three modifiable parameters pi, qi, ri, so-called consequentparameters, which are related to the output of the first order poly-nomial. When the premise parameters are fixed, the output of theANFIS can be written as:

f ¼ w1

w1 þw2f 1 þ

w2

w1 þw2f 2 ¼ �w1f 1 þ �w2f 2 ð12Þ

Substituting the Eq. (10) into the Eq. (12):

f ¼ �w1ðp1xþ q1yþ r1Þ þ �w2ðp2xþ q2yþ r2Þ ð13Þwhich is a linear combination of the modifiable consequent param-eters p1, q1, r1, p2, q2 and r2. The least squares method can be used toidentify the optimal values of these parameters easily. In eachepoch, the Least-Squares Estimator (LSE) method is used to opti-mize the consequent parameters, while the premise parametersare fixed. The output of the ANFIS is calculated by employing theconsequent parameters found in the forward pass. Once the optimalconsequent parameters are found, the Back-Propagation (BP)method will immediately start to adjust the premise parameterscorresponding to the fuzzy sets in the input domain immediately,according to the output error. It has been proven that this hybridalgorithm is highly efficient comparing with a standard gradientmethod in training the ANFIS.

3.2. Clustering using EM algorithm

It is well known that the k-means algorithm is an instance ofExpectation Maximization (EM) algorithm which is a generalalgorithm of density estimation. This algorithm is based on dis-tance. Gaussian mixture model with EM algorithm is a power fullapproach for clustering. EM algorithm is model based iterativealgorithm for solving the clustering problem where the data isincomplete or considered incomplete. EM algorithm is an opti-mization algorithm for constructing statistical models of the data(Mitra et al. 2003). In this algorithm each and every data instancebelongs to each and every cluster with a certain probability. EMalgorithm starts with initial estimates and iterates to find the



Fig. 3. The ANFIS architecture.


maximum likelihood estimates for the parameters. The quality ofEM algorithm becomes very good when using huge dataset. Ithas been also demonstrated that EM is a good clustering methodin terms of computation time and accuracy (Jung et al. 2014,Nathiya et al. 2010). In addition, in this study EM is chosen to clus-ter data for the following reasons among others (Ordonez andOmiecinski 2002). (1) It has a strong statistical basis, (2) it is linearin database size, (3) it is robust to noisy data, (4) it can accept thedesired number of clusters as input, (5) it can handle high dimen-sionality, and (6) it converges fast given a good initialization.

The mathematical background of EM algorithm is shown here inthis section (Mitra et al. 2003).

Given a dataset fxigNi¼1 the task of assigning a cluster for eachinstance in the dataset, is the goal that we aspire for. Let therebe N data points in the dataset and let us assume that the numberof clusters is k. Let the index of the cluster be modeled as a randomvariable z = j and let its probability be given by a multinomial dis-tribution satisfying

Ppj ¼ 1, Such that

pj ¼ pðz ¼ jÞ; 8j; j ¼ 1; . . . k ð14ÞIt is assumed that pðxjz ¼ jÞ � Nðlj;rjIjÞ is a Gaussian distribu-

tion. Ij denotes the identity matrix of order j. The unknown param-eters of the model namely the mean lj variance

Pj = diag (r1,

r2,. . .rj) and the distribution function pj are estimated.

h ¼ lj;Xj

; pj

( )k

j¼1

pðxjhÞ ¼Xk

z¼1

ðpðxjz; hÞpðzjhÞpj; ð15Þ

where z is an unknown hidden variable. The total log likelihood ofall data is given by

lðh;DÞ ¼ logYNi¼1

Xk

j¼1

pj exp � jjxi � ljjj22r2

i

" #ð16Þ

The parameter values that maximize the likelihood functionlðh;DÞ are the ones that are chosen. Here D denotes the data. Thisoptimization is complicated and to solve this some of theunknowns are assumed to be known, while estimating the othersand vice versa. For each class, the conditional expectation of z = jgiven the data and the parameters.

wj ¼ pðz ¼ jjx; hÞ ¼ pðxjz ¼ j; hÞpðz ¼ jjpjÞpðxjhÞ ¼ pjNðxijlj;

PjÞPk

i¼1pjNðxijlj;P

jÞð17Þ

Since each point x contributes to wj in some proportion, forparticular xi we have


wij ¼pjNðxijlj;

PjÞPk

i¼1pjNðxijlj;P

jÞ: ð18Þ

The optimization algorithm is called EM and has the followingsteps: Assume we have some random initial estimates of the

means and variances of the model lð0Þj ;

Pð0Þj ;pð0Þ

j . Algorithm 1,describes the EM algorithm.

Algorithm 1 EM Algorithm.

Initialize: means and variances of the model lð0Þj ;

Pð0Þj ;pð0Þ

j .

Step 1. Expectation: Using the estimates of

hðtÞ ¼ lðtÞj ;

PðtÞj ;pðtÞ

j

n o, parameters compute the estimate of

wij

wðiÞij ¼ pðz ¼ jjxi; hðtÞÞ ¼

pijpðxi jzi¼j;hÞPk

m¼1ptmpðxm jzm¼m;hðtÞÞ

Step 2. Maximization: Using estimates of wðiÞij , update the

estimates of the model parameters

lðtþ1Þj ¼

PN

i¼1wðtÞ

ij xiPN

i¼1wðtÞ

ij

rðtþ1Þj ¼

PN

i¼1wðtÞ

ijjjxi�li jj2PN

i¼1wðtÞ

ij

pðtþ1Þi ¼ 1

N

PNi¼1w

ðtÞij

Step 3. Repeat steps expectation and maximization until theparameter change gets small enough.

3.3. Solving multi-collinearity issue using PCA

Principal Component Analysis (PCA) is a tool for data compres-sion and information extraction (Nilashi et al. 2015a). In some sit-uations, there are many correlated or redundant data which mustbe compressed in a manner to retain the essential information.Among the widely used multivariate statistical methods, PCA is apowerful tool for analyzing such data because of its ability to han-dle large numbers of highly correlated, noisy and redundant vari-ables. Using PCA, a number of related variables are transformedto a set of uncorrelated variables. It is concerned with explainingthe variance–covariance structure of a set of variables throughfew linear combinations of these variables. Its general objectivesare data reduction and interpretation.

By using initial analysis of the data in multi-criteria experimen-tal dataset, an overlap of information can be found that affectoverall rating predictions. In addition, there exists significant




correlation among the input variables which are used as inputs ofmulti-criteria CF to build an inferential model. However, the datacan be compressed to retain the essential information and makethe input variables uncorrelated.

In the field of recommender systems, PCA has for example beenapplied by Goldberg et al. (2001) in the Jester joke recommender.In their approach, PCA was performed in an offline phase and theyapplied clustering on the resulting projection of the data in a two-dimensional space. Our approach is different from their work. Weapplied PCA after the initial clustering process individually on eachcluster and determined a suitable number of principal componentsto retain for each cluster. Then, as inputs in ANFIS, we used the PCsfor overall ratings prediction. Following this approach, it allowedus to achieve a highly accurate and up-to-date recommendationswith lower computation time in predicting the overall ratingsprediction.

From the experimental dataset, if we consider seven variables inthe matrix X, the procedure of dimensionality reduction for over-coming the multi-collinearity can be defined in two steps asfollows:

� Perform PCA on matrix X that consists of user ratings on items’criteria.

� PCs selection from PCA.

The selected number of PCs along with the desired output f(overall rating) are employed in developing the inferential models.Fig. 4 illustrates the PCA–ANFIS network structure with two PCs.

Reducing the dimensionality of a dataset which includes of alarge number of interrelated variables is the main objectives ofPCA. It keeps as much variation as possible in the original datasetand performs this process by transforming the original variablesto a new set of variables which are called Principal Components(PCs). The generated PCs are basically uncorrelated and orderedwhere the first of them includes most of the variation providedby the original variables. For constructing a PCA initializationmodel of multi-criteria CF, the multi-criteria dataset can be suffi-ciently described using some chosen parameters in relation tothe original variables with no significant loss of information. Theissue of multi-collinearity in the multi-criteria CF data is also elim-inated. The number of PCs that sufficiently represents the originaldata set is then selected.

As in multi-criteria CF there exist interdependencies betweenvariables of multi-criteria dataset, in such situation, the main issueis the multi-collinearity of the data that needs to be solved. Ifthe input variables of dataset are highly collinear, using theoriginal data for supervised learning methods such as regression,

Fig. 4. The structure o


classification and also Neuro-Fuzzy methods will result an ill-conditioned problem. Thus, in this situation, by reducing thedimension of dataset, PCA can be used to address multi-collinearity problems. The compressed model generated by PCAwhich consists of PCs provides linear combinations of the originalvariables of dataset.

Hence, instead of using the original variables as inputs in theANFIS, for the multi-criteria CF, selected PCs are used from thePCA algorithm. The developed PCA–ANFIS architecture is illus-trated in Fig. 5. From this figure, it can be seen that using thePCA approach, dimensionality of multi-criteria dataset can be ade-quately reduced. Also, later, we will demonstrate that developingPCA–ANFIS helps the proposed multi-criteria recommendationsystem to overcome the issue of multi-collinearity in the dataand accordingly accuracy improvement in relation to solely usingANFIS.

The inputs to the ANFIS were the selected PCs of the dataset byapplying PCA. N MFs was used for each input in fuzzification pro-cess which the total number of MN rules were generated. SincePCs are used as inputs in ANFIS model, the first-order Sugeno fuzzymodel provides the following rule-based structure for four PCs:

If PC1 is A and PC2 is B and PC3 is C and PC4 is D; then f

¼ p � PC1þ q � PC2þ r � PC3þ s � PC4

where A, B, C, and D indicate fuzzy sets for the input of system, {p, q,r, s} is the consequent parameter set, and f denotes the output afteraggregating the fuzzy rules. For the ANFIS employed in this study,the nodes in Layer 1 (premise parameters) were all generalizedGaussian MFs which have a flexible parameterization. For example,for the fuzzy set A, the generalized Gaussian MF takes the form:

lAiðPC1Þ ¼ e

�ðPC1�bi Þ22a2

i ð19Þ

where fai; bi; cig is the parameter set of the MFs in the premise partof fuzzy IF-THEN rules that change the shapes of the MFs.

The next layer (Layer 2) multiplies the inputs from the nodes inLayer 1 and generates the firing strength of the rules. The output ofthis layer is given by:

wi ¼ lAiðPC1Þ � lBi

ðPC1Þ � lCiðPC1Þ � lDi

ðPC1Þ ð20Þ

where wi is the firing strength of rule i.Accordingly, the overall ratings (O) can be calculated by Eq.

(13).

f PCA and ANFIS.



……

X1

PCn PC1

PCA

Xn

Uncorrelated Variables

1 2 3 q 1 2 3 q

M1Mq

1 23 q 1 2 3 q

M1Mq

Layer 1 Layer 2 Layer 3 Layer 4

Criteria 1

Layer 5

Overall

Rating

Criteria p

Fig. 5. PCA–ANFIS for multi-criteria CF.


4. Experimental results

4.1. Analyzing and pre-processing the dataset

In order to analyze the effectiveness of the proposed method,several experiments were conducted on TripAdvisor datasets pro-vided by TripAdvisor website (www.tripadvisor.com). TripAdvisorrepresents the world largest and most successful social networkingand community site in tourism (O’Connor 2008). The platformfacilitates the reviewing of hotels around the world and bringstogether individuals in discussion forums and provides users withindependent travel reviews and comments. In TripAdvisor websiteusers can rate a hotel according to 7 different aspects: Valueaspect, Rooms, Location, Cleanliness, Check in/front desk, Serviceand Business Service. In addition, users provide overall ratings onhotels. Ratings ranges from 0 to 5 stars, and �1 indicates thisaspect rating is missing in the original html file.

Generally, the information of ratings is presented in four-fold < userID; itemID; Overall rating; Criteria rating >. This data


can easily be converted into a tree dimensional rating tensor,whereas the first dimension (m) spans the number of users andthe second dimension (n) spans the number of items and the thirddimension (k) spans the number of criteria (Am�n�k).

The experimental dataset which has been used in this studyincludes 1264 hotels, 85,424 users and 7 criteria. Tables 1 and 2present the sample and statistics of raw TripAdvisor dataset forseven criteria and overall rating, respectively. In Table 2, thereare 28,500 tuples of ratings that the range of ratings is between�1 and 5 on criteria and overall ratings. The average evaluationgrade is 2.69, 1.45, 2.70, 2.20, 2.84, 1.36, 2.72 and 0.46 for ‘‘Overallrating”, ‘‘Location aspect”, ‘‘Rooms aspect”, ‘‘Value aspect”, ‘‘Clean-liness aspect”, ‘‘Check in/front desk aspect”, ‘‘Service aspect andBusiness Service aspect”, respectively.

In Table 2 show that there are 28,500 tuples of rating in theoriginal dataset, however as we can see the users have rated hotelsare very few. The sparsity rate is:

Sparsity ¼ numRatings� 100%numUsers� numItems

¼ 1� 8542428500� 1264

’ 0:9997


http://www.tripadvisor.com



That means, the sparsity level of the experimental dataset is99.97%

In order to check the collinearities between the criteria’ vari-ables, we also compute the linear correlations between the inputattributes as shown in Table 3. As can be seen high collinearities(r2 value) can be found among the 7 criteria.

4.2. Clustering with EM

As indicated in Section 3, we applied the EM clustering on users’ratings in 3-order tensor. In every clustering method, choosing theright number of clusters is important. In EM clustering, with theGaussian mixture model, the likelihood must be optimized. Hence,for this optimization, the best cluster number is selected by evalu-ating various values for the number of clusters. It should be notedthat according to Pelleg and Moore (2000), we used informationtheoretic criterion like the Akaike Information Criterion (AIC)(Akaike 1974) to choose the value optimal number of cluster.Accordingly, in the experimental dataset, we have used a resubsti-tution AIC estimate and evaluated a number of clusters from 1 to20. In addition, in the clustering procedure, we applied 5-fold crossvalidation to obtain unbiased result. In Fig. 6, we present the vari-ous numbers of clusters to select the best cluster based on chosencriterion. This figure shows that the best criterion value(660091.311541) is obtained when 6 clusters are generated byEM. In Fig. 7, the clusters generated by EM are visualized. For visu-alizing the dataset clusters into the original space, a PCA is used inorder to obtain a 2D representation. It was used to visualize clus-ters in the scatter plot using the first and second PCs. Accordingly,

Table 1A sample of the multi-criteria rating of TripAdvisor dataset.

User ID Overall rating Value Rooms Location Cleanliness

18 5 5 4 �1 514 5 5 5 5 520 3 3 4 2 5

..

. ... ..

. ... ..

.

22 4 3 3 3 445 5 5 5 5 511 5 �1 5 5 5

Table 2Statistics of hotel rating on seven criteria and overall rating.

Overall rating Value Rooms Location

N 28,500 28,500 28,500 28,500Mean 2.69 2.20 2.70 1.45Min. 0 �1 �1 �1Max. 5 5 5 5

Table 3Correlation among criteria.

Variable Y Variable X r2

Value Rooms 0.8372Value Location 0.8422Value Cleanliness 0.7521Value Check in/front desk 0.8605Value Service 0.8514Value Business Service 0.7878Rooms Location 0.8111Rooms Cleanliness 0.8256Rooms Check in/front desk 0.7987Rooms Service 0.9123Rooms Business Service 0.8125


the cluster centers are presented in Table 4. These cluster centersare used to assign newly arriving data points to a cluster basedon their Euclidian distance. From the Table 4, it can be seen thatEM has obtained the 6 clusters from our experimental dataset. Itshould be noted that 6 clusters have been automatically generatedby EM. These clusters are used in PCA and then ANFIS for predic-tion models.

4.3. Dimensionality reduction with PCA

We applied the PCA on the clusters obtained from the experi-mental dataset using EM algorithm that the results in the followingare presented. It should be noted the results were obtained fromdataset with seven criteria without considering the overall ratings.Tables 5–10 describe the eigenvalues associated to the factors forsix clusters. We have also the percentage of the total variance(individual and cumulative). In Fig. 8, Scree plot from PCA resultsfor six clusters are demonstrated.

Table 5 describes the eigenvalues associated to the factorsobtained by PCA for first cluster. The obtained results also havethe percentage of the total variance (individual and cumulative).In addition, from Table 5, it can be noticed that first two PCs pro-vide 76.95% and 84.93% of information.

Table 6 describes the eigenvalues associated to the factorsobtained by PCA for second cluster. The obtained results also havethe percentage of the total variance (individual and cumulative). Inaddition, from Table 6, it can be noticed that first two PCs provide65.79% and 80.04% of information.

Check in/front desk Service Business Service Hotel ID

�1 5 -1 hotel_5655505 5 5 hotel_5660771 1 2 hotel_566077

..

. ... ..

. ...

3 4 3 hotel_5708883 3 5 hotel_5708885 5 5 hotel_572859

Cleanliness Check in/front desk Service Business Service

28,500 28,500 28,500 28,5002.84 1.36 2.72 0.46�1 �1 �1 �15 5 5 5

Variable Y Variable X r2

Location Cleanliness 0.7974Location Check in/front desk 0.8421Location Service 0.7824Location Business Service 0.8374Cleanliness Check in/front desk 0.8623Cleanliness Service 0.7826Cleanliness Business Service 0.8378Check in/front desk Service 0.8682Check in/front desk Business Service 0.7837Service Business Service 0.8937



Number of Clusters Criterion Diagram

1 704812.560326 2 863749.183603 3 836959.462740 4 744986.404010 5 716333.967586 6 660091.311541 7 821898.619479 8 1013541.275427 9 1205808.277350 10 1195185.920633 11 1136947.545377 12 1191156.158037 13 1128803.235662 14 1119202.838241 15 1029133.154537 16 899327.018714 17 997434.313315 18 1046513.971724 19 895661.602131 20 1160162.357771

0 2 4 6 8 10 12 14 16 18 206

7

8

9

10

11

12

13x 10

5

Number of Clusters

Crit

erio

n V

alue

Criterion

Fig. 6. Best cluster based on chosen criterion.

Fig. 7. Visualization of clusters on PCA axes.

Table 4Cluster Centers.

Attribute Cluster #1 Cluster #2 Cluster #3 Cluster #4 Cluster #5 Cluster #6

OverallRatings 2.053703 2.998649 2.200389 3.092038 2.932179 4.143281ValueAspect 0.628450 4.215042 2.762975 4.613662 4.409396 0.006359RoomAspect 0.797953 4.143887 2.782404 4.581060 4.354645 4.029250LocationAspect 0.624540 0.000000 3.553150 4.681969 4.589191 0.000000CleanlinessAspect 0.835557 4.448322 3.167083 4.794411 4.619569 3.782111CheckinFrontDeskAspect 0.610281 0.000000 2.936997 4.751386 4.540092 0.000000ServiceAspect 0.715271 4.248818 2.712739 4.743402 4.539032 4.112760BusinessServiceAspect 0.352921 0.000000 2.141826 4.371036 0.003179 0.000000


Table 7 describes the eigenvalues associated to the factorsobtained by PCA for the third cluster. The obtained results alsohave the percentage of the total variance (individual and cumula-tive). In addition, from Table 7, it can be noticed that first twoPCs provide 51.50% and 65.72% of information.


Table 8 describes the eigenvalues associated to the factorsobtained by PCA for the fourth cluster. The obtained results alsohave the percentage of the total variance (individual and cumula-tive). In addition, from Table 8, it can be noticed that first twoPCs provide 40.21% and 53.63% of information.



Table 5PCA result for Cluster 1.

Axis

1234567

Eigen value

5.386500 0.558341 0.403429 0.290596 0.183814 0.105533 0.071787

e Differenc

4.828150.1549120.112830.1067820.078280.03374

-

ce Proporti

9 76.952 7.983 5.762 4.151 2.637 1.51

1.03

ion (%)

5 % 8 % 6 % 5 % 3 %

% 3 %

Histogram Cumul

768490949798100

lative (%)

.95 %

.93 %

.69 %

.84 %

.47 %

.97 % 0.00 %


Axis Eigen value Difference Proportion (%) Histogram Cumulative (%)

1 2.814895 1.875728 40.21 % 40.21 %

2 0.939167 0.156714 13.42 % 53.63 %

3 0.782453 0.029452 11.18 % 64.81 %

4 0.753001 0.053812 10.76 % 75.56 %

5 0.699190 0.15032 9.99 % 85.55 %

6 0.548848 0.086402 7.84 % 93.39 %

7 0.462446 - 6.61 % 100.00 %



1 3.604651 2.609078 51.50 % 51.50 %

2 0.995573 0.255513 14.22 % 65.72 %

3 0.740059 0.058082 10.57 % 76.29 %

4 0.681978 0.273723 9.74 % 86.03 %

5 0.408255 0.073267 5.83 % 91.86 %

6 0.334988 0.100493 4.79 % 96.65 %

7 0.234495 - 3.35 % 100.00 %



1 2.631675 2.061667 65.79 % 65.79 %

2 0.570008 0.148335 14.25 % 80.04 %

3 0.421673 0.045028 10.54 % 90.58 %

4 0.376645 0.376645 9.42 % 100.00 % 5 0.000000 0.000000 0.00 % 100.00 %6 0.000000 0.000000 0.00 % 100.00 % 7 0.000000 - 0.00 % 100.00 %


Table 9 describes the eigenvalues associated to the factorsobtained by PCA for the fifth cluster. The obtained results also havethe percentage of the total variance (individual and cumulative). Inaddition, from Table 9, it can be noticed that first two PCs provide40.01% and 54.31% of information.

Table 10 describes the eigenvalues associated to the factorsobtained by PCA for the sixth cluster. The obtained results alsohave the percentage of the total variance (individual and


cumulative). In addition, from Table 10, it can be noticed that firsttwo PCs provide 69.83% and 44.75% of information.

The eigenvalues that are associated with the factors areindicators for their importance. In our work, we decided to usethe rule proposed by Cattell (1966) and create ‘‘scree” plots asshown in Fig. 8 where we plot the eigenvalues of the factors todetect ‘‘elbows” that indicate possible changes in the structure ofthe data.





1 2.800359 1.799299 40.01 % 40.01 %

2 1.001060 0.117358 14.30 % 54.31 %

3 0.883702 0.096034 12.62 % 66.93 %

4 0.787668 0.112935 11.25 % 78.18 %

5 0.674733 0.224823 9.64 % 87.82 %

6 0.449910 0.047341 6.43 % 94.25 %

7 0.402569 - 5.75 % 100.00 %

(a) (b) (c)

(d) (e) (f)

0 2 4 6 80

1

2

3

4

5

6

Component

Eig

enva

lue

0 2 4 6 80

0.5

1

1.5

2

2.5

3

Component

Eig

enva

lue

0 2 4 6 80

1

2

3

4

Component

Eig

enva

lue

0 2 4 6 80

0.5

1

1.5

2

2.5

3

Component

Eig

enva

lue

0 2 4 6 80

0.5

1

1.5

2

2.5

3

Component

Eig

enva

lue

0 2 4 6 80

0.5

1

1.5

2

Component

Eig

enva

lue

Fig. 8. Scree plot from PCA results for six clusters.

Table 10PCA analysis for Cluster 6.

Axis Eigen value Difference Proportion (%) Histogram Cumulative (%) 1 1.790024 0.786923 44.75 % 44.75 % 2 1.003101 0.206362 25.08 % 69.83 % 3 0.796739 0.386602 19.92 % 89.75 % 4 0.410137 0.410137 10.25 % 100.00 % 5 0.000000 0.000000 0.00 % 100.00 % 6 0.000000 0.000000 0.00 % 100.00 %7 0.000000 - 0.00 % 100.00 %


For the first plot as shown in Fig. 8a, we can include the elbowinto the selection i.e. we select k = 3 factors. Indeed, the eigenvalue(k3 ¼ 0:403429) associated with the 3rd factor is high. It corre-sponds to 90.69% of the variance. For the second plot as shownin Fig. 8b, we can include the elbow into the selection i.e. we selectk = 3 factors. Indeed, the eigenvalue (k3 ¼ 0:421673) associatedwith the 3rd factor is high. It corresponds to 90.58% of the variance.For the third plot as shown in Fig. 8c, we can include the elbow intothe selection i.e. we select k = 6 factors. Indeed, the eigenvalue(k6 ¼ 0:408255) associated with the 6th factor is high. It corre-sponds to 91.86% of the variance. For the fourth plot as shown inFig. 8d, we can include the elbow into the selection i.e. we select


k = 6 factors. Indeed, the eigenvalue (k6 ¼ 0:548848) associatedwith the 6th factor is high. It corresponds to 93.39% of the variance.For the fifth plot as shown in Fig. 8e, we can include the elbow intothe selection i.e. we select k = 6 factors. Indeed, the eigenvalue(k6 ¼ 0:449910) associated with the 6th factor is high. It corre-sponds to 94.25% of the variance. For the sixth plot as shown inFig. 8f, we can include the elbow into the selection i.e. we selectk = 4 factors. Indeed, the eigenvalue (k6 ¼ 0:410137) associatedwith the 4th factor is high. It corresponds to 100% of the variance.

In Table 11, we have summarized the selected PCAs for allclusters. From the Table 11, it can be found that for Cluster 1 andCluster 2 three PCs are selected as they provides significant



Table 11Result of PCA on 8 clusters.

PC1 PC2 PC3 PC4 PC5 PC6 PC7

Cluster 1p p p

– – – –Cluster 2

p p p– – – –

Cluster 3p p p p p

– –Cluster 4

p p p p p p–

Cluster 5p p p p p p

–Cluster 6

p p p p– – –


percentage of information. For Cluster 3, five PCs are selected asthey provide 91.86% of information. For Cluster 4 and Cluster 5, 6PCs are selected as these cluster provide 98.97% and 100.00% oforiginal data, respectively. And, for Cluster 6, four PCs are selectedas they provide 100.00% of information.

4.4. Evaluating the PCA–ANFIS models

In this study, the fuzzy rule based system was developedthrough several consequent steps. In the fuzzification step, ANFIStakes the inputs and determine the degree to which they belongto each of the appropriate fuzzy sets via membership functions(Gaussian). After developing membership functions, ANFISextracted fuzzy rules from the users’ ratings on items to be usedin the fuzzy rule based system. Then, in the defuzzification step,the fuzzy outputs are converted into a scalar output quantity, asthe output of each rule is fuzzy. It should be noted that as weimplemented the fuzzy rule based system in Matlab software,the centroid of area (COA) method (Hellendoorn and Thomas1993) was used for defuzzification purpose. COA is the mostpopular defuzzification method, which returns the center of areaunder the curve. The results of defuzzification step are then usedfor overall rating prediction and revealing the importance level ofitems’ criteria.

After applying PCA on clusters, ANFIS models were developed tofind the relative importance of criteria and predict overall ratingsbased on input variables. 6 ANFIS models were totally developedbased on inputs and output of data for the clusters. Since PCs wereselected as inputs of ANFIS models, in the fuzzification steps, for allPCs the degree to which they belong to each of the appropriatefuzzy sets via MFs were determined. Because of its smoothnessand concise notation, Gaussian MF is popular method for specify-ing fuzzy sets. The curves in this type of MF have the advantageof being smooth and nonzero at all points. In addition, this typeof MF provided ANFIS models with minimum prediction errors

Table 12The information of MFs for first cluster.

Variables Type Linguist

MF1

Inputs PC1 Gaussian [1.2 �3.PC2 Gaussian [0.8548PC3 Gaussian [0.5473

Table 13The information of MFs for second cluster.

Variables Type Linguistic

MF1

Inputs PC1 Gaussian [0.3693 �PC2 Gaussian [0.3136 �PC3 Gaussian [0.3693 �


compared to the other types of MFs. Hence, in this paper, weselected Gaussian MF and developed the ANFIS models base on thistype of MF.

Tables 12–17 present the MFs for PCs generated by PCA–ANFIS.From these tables, it can be seen that Gaussian MFs are consideredfor PCs by three linguistic variables MF1, MF2 and MF3. In Table 12,for each PC, the Gaussian MFs are generated by PCA–ANFIS in threemain groups. The range of PC1 and PC2 and PC3 for linguistic vari-able MF1, MF2 and MF3 are defined [1.2 �3.206], [1.201 �0.3775]and [1.2 2.452], respectively. And, in Table 13, the range of PC1 forlinguistic variables MF1, MF2 and MF3 are defined [0.3693�0.08725], [0.3899 0.8032] and [0.4535 1.645], respectively.

Fig. 9 illustrates the interdependency of four inputs parameters(PCs) and the overall rating obtained from the fuzzy rules gener-ated by PCA–ANFIS through control surface. The level of overallrating can be depicted as a continuous function of its input param-eters as PC1, PC2 and also other PCs. The surface plots in this figuredepict the variation of overall rating based on identified fuzzyrules.

From the fuzzy rule viewer of established PCA–ANFIS modelshown in Figs. 10 and 11, the process of overall rating predictionby selecting the MFs can be better visualized. They indicate thebehavior of users over the change in values of all 7 inputs reducedin the PCs for overall rating. From the fuzzy rule viewer in Fig. 10,when the input PCA1 is at 0.86, PCA2 at �1.01, and PCA3 at 0.225,an output of overall rating at 2.56 out of 5 is obtained. In addition,from the fuzzy rule viewer in Fig. 11, when the input PCA1 is at1.32, PCA2 at 1.74, PCA3 at 0.0517, PCA4 at 0.787, and PCA5 at0.0297, an output of overall rating at 2.03 out of 5 is obtained. Itshould be noted that COA was used for defuzzification purpose.

From Figs. 10 and 11, it can be seen that the overall ratings canbe predicted using generated PCs instead of using original vari-ables. Hence, choosing the right number of PCs is important foroverall rating prediction. As we noted earlier, the eigenvalues thatare associated with the factors in each cluster are indicators oftheir importance and we used those factors as inputs for overallrating prediction in ANFIS.

For evaluating the PCA–ANFIS model, two measures of accuracyare used to determine the model capability for predicting the over-all rating. For this reason, the models are evaluated by two estima-tors Mean Squared Error (MSE) and coefficient of determination(R2). Usually, in the training process, MSE measure is used to testthe prediction model; however, in this study, other performancemeasures are used to search for a more effective performance eval-uation which is coefficient of determination R2. The coefficient ofdetermination R2 provides a value between [0, 1] about the

ic values and ranges of MFs

MF2 MF3

206] [1.201 �0.3775] [1.2 2.452]�1.989] [0.858 0.02164] [0.8631 2.027]�1.65] [0.5651 �0.3525] [0.5512 0.9216]

values and ranges of MFs

MF2 MF3

0.08725] [0.3899 0.8032] [0.4535 1.645]2.437] [0.3122 �1.6] [0.3714 �0.8225]0.08725] [0.3899 0.8032] [0.4535 1.645]



Table 14The information of MFs for third cluster.

Variables Type Linguistic values and ranges of MFs

MF1 MF2 MF3

Inputs PC1 Gaussian [0.8824 �2.049] [0.8827 0.03151] [0.8818 2.112]PC2 Gaussian [0.4804 �0.2267] [0.48 0.915] [0.4802 2.042]PC3 Gaussian [0.2964 �1.014] [0.3001 �0.3058] [0.2949 0.4022]PC4 Gaussian [0.5356 �1.058] [0.5368 0.2075] [0.5354 1.473]PC5 Gaussian [0.4164 �1.248] [0.4207 �0.2589] [0.4151 0.73]

Table 15The information of MFs for fourth cluster.


MF1 MF2 MF3

Inputs PC1 Gaussian [0.2775 2.064] [0.2769 2.724] [0.2752 3.385]PC2 Gaussian [0.2141 �0.106] [0.2168 0.4081] [0.2125 0.9228]PC3 Gaussian [0.1384 �1.379] [0.1368 �1.038] [0.1377 �0.6987]PC4 Gaussian [0.2632 �0.3008] [0.2639 0.3254] [0.2608 0.9525]PC5 Gaussian [0.1727 �0.4013] [0.1781 0.02287] [0.1733 0.4475]PC6 Gaussian [0.1733 0.4475] [0.1545 �0.06758] [0.1538 0.3103]

Table 16The information of MFs for fifth cluster.


MF1 MF2 MF3

Inputs PC1 Gaussian [0.2436 1.415] [0.2429 1.992] [0.2449 2.571]PC2 Gaussian [0.2171 �0.9726] [0.2198 �0.4532] [0.2169 0.0653]PC3 Gaussian [0.1032 �0.6502] [0.1025 �0.3936] [0.1012 �0.1379]PC4 Gaussian [0.08673 �1.665] [0.09183 �1.445] [0.08028 �1.226]PC5 Gaussian [0.1406 �0.5477] [0.1423 �0.2067] [0.141 0.134]PC6 Gaussian [0.1732 �0.3511] [0.1723 0.05982] [0.1704 0.4712]

Table 17The information of MFs for Sixth cluster.


MF1 MF2 MF3

Inputs PC1 Gaussian [0.4024 �1.938] [0.3993 �0.9892] [0.4044 �0.03521]PC2 Gaussian [0.4216 �2.314] [0.4284 �1.31] [0.4239 �0.3021]PC3 Gaussian [0.1833 �0.4454] [0.1748 �0.00878] [0.1762 0.4226]PC4 Gaussian [0.05175 0.27] [0.05812 0.4379] [0.06356 0.605]


training of the proposed network. A value closer to 1 stands for thesuccess of learning. These estimators are determined by Eqs. (19)and (20).

MSE ¼Pn

O¼1ðactual ðOÞ � predictionðOÞÞ2n

ð21Þ

R2 ¼ 1�Pn

O¼1ðactualðOÞ � predictionðOÞÞ2PnO¼1ðactual ðOÞ � actual ðOÞÞ2

ð22Þ

where actual (O) indicates the real overall rating provided by user,prediction (O) implies the predicted overall rating value and ncorresponds to the number of used user ratings.

For error estimation in the clusters of EM, after 200 epochs, theaverages MSE and R2 for validating, testing and training data werecalculated as presented in Table 18. The MSE and R2 were calcu-lated based on overall ratings prediction. It should be noted thatwe used 5-fold cross validation and average test accuracy for eachcluster.


For error estimation in the clusters provided by EM, it can beconcluded that prediction errors for PCA–ANFIS models of EMclusters are significantly low with high values of coefficient ofdetermination.

Comparison of performance for overall rating prediction of bothMultiple Linear Regression (MLR) and ANFIS on experimental dataset show that the proposed PCA–ANFIS method is more accuratethan the MLR in overall rating prediction. For ANFIS models, weselected the best configurations in terms of MFs type, type of train-ings and number of training. The Gaussian MF type shows the bestperformance in relation to the Triangular one. Note that as theoverall ratings ranges are [0 5], the prediction models ofPCA–ANFIS also obtained this range for the overall ratings. How-ever, the prediction range for MLR was not always between [0 5],in many cases it was above than this range.

In this paper, we selected hybrid learning (training) algorithmin ANFIS. This learning algorithm combines the least squares esti-mator and the gradient descent method. Using the hybrid method,the ANFIS models generated rules by enumerating all possible



Fig. 9. Interdependency of any two PCs and overall rating for the (a) first and (b) third clusters.

Fig. 10. Prediction of overall rating based on 3 PCs in the first cluster.


combinations of MFs of all original inputs and PCs generated byPCA. Compared with the ANFIS for overall rating prediction, themodels that used ANFIS with incorporating PCA had lower compu-tation time in all models. In addition, the computation time forANFIS is moderately large when the number of inputs (curse ofdimensionality) (Brown et al. 1995). This can be a main disadvan-tage of using solely ANFIS for the problem of overall rating predic-tion in multi-criteria CF. Hence, this problem connected to theANFIS has been eliminated with incorporating the PCA beforeapplying ANFIS. This incorporation of PCA caused the reductionin number of inputs and accordingly hidden layers, number ofMFs and rules. Evidently, the training time of prediction modelswas significantly reduced also as the computational overhead asso-ciated with the PCA algorithm is negligible. Note that all the pre-diction models by ANFIS have been developed in the offlinephase and when new ratings were added to the dataset, the modelsneeded to be retrained. An opportunity for future work is thereforeto add newly arriving data points to the clusters and train the pre-diction models incrementally by ANFIS. The advantage of having


incremental prediction models is that when incremental updatesare supported, the scalability of multi-criteria CF will be improved.

4.5. Evaluating the hybrid multi-criteria CF

To evaluate the accuracy of the proposed method, we conducteda set of experiments. We determine the precision and recall of theTop-N list for recommender system. We relied on the typical eval-uation protocol and accuracy measures described, e.g., in (Shaniand Gunawardana 2011), to determine the quality of the recom-mendations and to compare our method with previous works.Specifically, we considered the data into training and test setsand tried to predict the rating or ranking of the hidden items inthe test dataset. To factor out random effects, we apply a fivefoldcross-validation procedure. For these measures we split the datainto 80% training and 20% test data, used random subsamplingand repeated the experiments appropriately to factor out effectsof randomness.



Fig. 11. Prediction of overall rating based on 5 PCs in the third cluster.

Table 18MSE and R2 for 6 clusters in PCA–ANFIS modeling.

Accuracy Measure Training Testing Validating Training Testing Validating Training Testing Validating

Cluster 1 Cluster 2 Cluster 3MSE 0.046525 0.04853 0.052925 0.052153 0.032734 0.045285 0.021711 0.037897 0.042253R2 0.97460 0.96850 0.95260 0.94345 0.98564 0.971171 0.988835 0.981124 0.970224

Cluster 4 Cluster 5 Cluster 6MSE 0.0562 0.0583 0.0571 0.0551 0.0434 0.054 0.0321 0.0442 0.0503R2 0.9581 0.9544 0.959 0.942 0.9736 0.964 0.9711 0.972 0.9691

Table 19RMSE for proposed method.

Method RMSE

Standard CF (Adomavicius and Kwon 2007) 0.672Total-Reg (Adomavicius and Kwon 2007) 0.653PCA 0.618Clustering and Multiple Linear Regression 0.608Clustering and ANFIS 0.513ANFIS and HOSVD (Nilashi et al. 2014b) 0.489Clustering and PCA–ANFIS 0.454


The recommenders’ prediction accuracy is measured by rootmean squared error (RMSE), which is a widely used metric for eval-uating the statistical accuracy of recommendation algorithms,given by

RMSE ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPni¼1ðx

_

i�xiÞ

n

vuutð23Þ

where x_

iis the rating values predicted by the recommender from

the true rating xi. A lower value of RMSE indicates a higher accuracyof the recommendation system.

The accuracy result measured by RMSE is shown in Table 19.The results are based on fivefold cross-validation and 80% data inthe training set with 5 test trials. The results show that the combi-nation of the clustering and the PCA technique helps us to measur-ably decrease the prediction error. The coverage of methods thatuse Clustering with ANFIS and Clustering with PCA and ANFISapproaches was 100% compared to ANFIS and HOSVD (Nilashiet al. 2014b) with 100% coverage, Standard CF (Adomavicius andKwon 2007) with 41% coverage, Total-Reg (Adomavicius andKwon 2007) with 53% coverage, PCA with 74% coverage, andClustering and Multiple Linear Regression with 85% coverage. Thismeans that predictions for all user-item pairs in the test set couldbe made by the proposed method.


From Table 19, it can be seen that proposed model has outper-formed other recommendation methods and obtained a betterRMSE error. It should be noted that a two-tailed paired t-test hasbeen performed and based on the results the differences betweenproposed model (PCA–ANFIS) and the compared methods werestatistically significant (p < 0.01) for the experimental dataset.

We also employe the recall and precision metrics, which arewidely used in recommender systems to evaluate the quality ofrecommendations. Recall indicates the ability of a system to pre-sent all relevant items (Sarwar et al. 2000). In reality, it may notbe possible to retrieve all the relevant items from a collection,especially when the collection is large. A system may be able toretrieve a proportion of the total relevant items. Thus, the perfor-mance of a system is often measured by the recall ratio, which



Table 20Contingency table for computing precision and recall.

Selected Unselected Total (Selected Unselected)

Relevant NSR NUR NTR

Irrelevant NSI NUI NTI

Total NST NUT NTT


denotes the percentage of the relevant items retrieved in a givensituation. Precision implies the ability of a system to present onlythe relevant items. This relates to its ability to not retrieve non-relevant item (Sarwar et al. 2000). This factor, that is how far thesystem is able to withhold unwanted items in a given situation,is measured in terms of precision ratio. Table 20 shows Contin-gency table for computing precision and recall. These precisionand recall measures are presented by Eq. (24) and Eq. (25),respectively.

PrecisionðReclistÞ ¼jfrelevant itemsg \ ftop� Nitemsgj

jftop� N itemsgj ¼ NSR

NST

ð24Þ

Re callðReclistÞ ¼jfrelevant itemsg \ ftop� Nitemsgj

jfrelevant itemsgj ¼ NSR

NTR

ð25Þ

F-measure is a metric defined as the harmonic mean of the pre-cision and recall, is also widely used to evaluate the quality of rec-ommendations. We used the F1-metric in our evaluation, as shownin Eq. (25).

F1 ¼ ð1þ b2Þ:precision:recallb2:precisionþ recall

ð26Þ

where parameter b 2 ½0;1� determines the relative influence of bothmetrics (the value b ¼ 1 is commonly used).

In order to compare the proposed method with previous work,we evaluated our approach on YM-10-10 using MAE. The MAE isdetermined as the average absolute deviation between predictedratings and true ratings that showed in Eq. (27).

MAEðpred; actÞ ¼XNi¼1

predu;i � actu;iN

�� ð27Þ

Suppose N is the number of items that user u has expressed anopinion.

To experimentally show the effectiveness of clustering and PCA-ANFIS, we perform the experiments on the TripAdvisor dataset. Theaim is to calculate the recommendation precision and MAE of pre-diction of proposed method (see Table 21). We compared the pro-posed method with the Standard CF and Total-Reg developed byAdomavicius and Kwon (2007), matrix factorization method usingPCA (Yin and Peng 2012), and tensor factorization method (Nilashiet al. 2014b).

Table 21Precision at top 5 and top 7 and MAE at original 1–5 scale dataset.

Method Precis

Standard CF (Adomavicius and Kwon 2007) 62.20Total-Reg (Adomavicius and Kwon 2007) 65.10PCA 73.14Clustering and Multiple Linear Regression 78.36Clustering and ANFIS 82.45ANFIS and HOSVD (Nilashi et al. 2014b) 84.86Clustering and PCA–ANFIS 86.21


From the results presented in Table 21, it can be found that thePrecision@5 (recommended items for a user of size 5) and Preci-sion@7 (recommended items for a user of size 7) of the proposedmethod is higher than recommendation method which only usesANFIS. According to the experiments results, the proposed hybridmethod also provides better prediction accuracy with lower MAEin relation to the Standard CF and Total-Reg developed byAdomavicius and Kwon (2007). These show the effectiveness ofincorporating the clustering and PCA approaches regarding theprediction accuracy of multi-criteria CF in the Tourism Domain.In addition, the results in Table 21 indicates that, compared toStandard CF and Total-Reg, our clustering and noise removal tech-niques help to improve the prediction accuracy by more than 20%in all tested scenarios in term of the Precision@5 and [email protected], the results show that compared to the matrix factor-ization method using PCA, our clustering and noise removal tech-niques help to improve the prediction accuracy by more than15% in all tested scenarios in term of the Precision@5 and Preci-sion@7. It can also be observed that the matrix factorization usingPCA works relatively well and is better than methods such as Stan-dard CF and Total-Reg. This supports the findings of (Jannach et al.2012a) with respect to the accuracy as their method based onregression approach outperformed the methods that used matrixfactorization and methods developed by Adomavicius and Kwon(2007). Moreover, compared with the methods that use Clusteringand Multiple Linear Regression, the proposed method improves theprecision by more than 8%. Again, we can see that the clusteringand PCA techniques help to increase precision and at the same timereduce the MAE compared to method that uses Clustering andANFIS.

Over method that uses EM clustering and ANFIS, the predictionaccuracy improvement is more than 6% in term of the MAE. Inaddition, compared with the method which uses ANFIS and HigherOrder Singular Value Decomposition (HOSVD), the accuracyimprovement was more than 2% in all tested scenarios in term ofthe Precision@5 and Precision@7 and at the same time reducedthe MAE about 3%. This supports the findings of (Nilashi et al.2014b) with respect to the accuracy as their method based onANFIS technique outperformed the recommendation methods thatonly used tensor and matrix factorization approaches and recom-mendation methods developed by Adomavicius and Kwon(2007). In addition, it can be supported by Jannach et al. (2012a)work as they evaluated the improvements of their proposedmethod on Yahoo!Movies and tourism domain datasets. Theirexperiments showed that the suggested improvements not onlylead to better results than those achieved with the techniques pre-sented in Adomavicius and Kwon (2007), but also that the predic-tions are more accurate than more recent single-rating approachesbased on matrix factorization.

In term of F1, the recommendation accuracy performance of themulti-criteria CF using PCA, ANFIS and clustering approach is pre-sented in Fig. 12. For this purpose, in the experiments, we variedthe number of neighbors and computed the corresponding F1(b ¼ 1) value for proposed recommendation method. We ran the

ion@5 Precision@7 MAE

61.45 1.3763.31 1.2871.13 1.1676.16 1.1181.21 0.9283.21 0.8985.71 0.86



1 20 40 60 80 100 120

0.81

0.82

0.83

0.84

0.85

0.86

0.87

0.88

Top-N

F-m

easu

re

5 neighbors 10 neighbors 20 neighbors 30 neighbors 40 neighbors 50 neighbors 60 neighbors

Fig. 12. Top-N and F-measure for different neighborhood size.


experiments on TripAdvisor dataset for N equal 1, 20, 40, 60, 80,100 and 120, where N is the number of items to be recommendedby the Top-N recommender systems. From all F1 curves in Fig. 12,we can notice that the proposed method gives high level of accu-racy when the size of neighbors is increased versus the Top-N rec-ommendation. In this figure, it can be seen that while returningTop-1 to Top-120 recommendation, the proposed method canachieve an improvement above 0.81 for at all neighborhood sizes.In addition, it can be concluded that the optimal neighbor size canbe obtained by considering the maximum value of F1. Thus, theneighborhood size 60 can be chosen as the optimal value in pro-ducing the best performance of the proposed method. This out-come demonstrates the significance of combining PCA–ANFISmethod with EM clustering for enhancing the accuracy of multi-criteria CF in the tourism domain.

5. Conclusion and future work

In this paper, a new approach, called PCA–ANFIS, was proposedto increase the predictive accuracy and efficiency of the multi-criteria CF. The proposed method was developed for tourismdomain using EM clustering, PCA and ANFIS. We selected ANFISapproach to learn the prediction models for users and items sepa-rately in each cluster and then combined two predictions in theonline phase. In addition, PCA was used for dimensionality reduc-tion and to address multi-collinearity and interdependency prob-lems existing between criteria in the multi-criteria dataset.

We analyzed the predictive accuracy of proposed method in thedomain of hotel recommendation on a real-world dataset providedby TripAdvisor. The proposed method was evaluated using RMSE,MAE, F1-measure, Precision@5 and Precision@7 using precisionmetric. The experimental results on TripAdvisor dataset clearlydemonstrated the capability of ANFIS modeling using MFs andfuzzy rules without the human expert intervention in multi-criteria CF in the tourism domain. Our experiments confirmed thatthe combination of PCA–ANFIS with clustering as a hybrid methodsignificantly leads to the improvement in predictive accuracy oftourism multi-criteria CF measured by standard accuracy metrics.

In the proposed method, ANFIS models of items and users areupdated offline; however, in the multi-criteria CF recommenders,data is dramatically updated and therefore incremental learningapproaches are needed to consider new ratings. Thus, future stud-ies will focus on further improvement of the multi-criteria CF rec-ommendation accuracy for tourism domain by incorporating fuzzysemantic technique. In addition, we will focus on developing anincremental Neuro-Fuzzy learning approach as newly arriving rat-ings can be crucial for the success of a practical recommendation


system and have an immediate impact on the accuracy of the pre-dictions. Recommendations for future study in this area are asfollows:

� Although multi-criteria ratings can be a good choice for pure CFrecommendation, the accuracy of multi-criteria CF can beimproved more with incorporating other resources such as tagsand content of users and items to the tensor of ratings. Withthis incorporation, the fuzzy semantic techniques can beapplied to better alleviate sparsity problem and enhance themulti-criteria CF recommendation accuracy.

� The proposed multi-criteria recommendation model can beextended to the incremental based recommendation. Therefore,future studies will focus on further improvement of the multi-criteria CF recommendation accuracy and efficiency by incorpo-rating incremental techniques using incremental ANFIS.

Acknowledgements

The authors would like to thank Research Management Centre(RMC) at Universiti Teknologi Malaysia (UTM) for providing ade-quate facilities to conduct this research and for their financial sup-ports of this research through the Post Doctorate ResearchUniversiti Grant with Vote No. 02E50. Appreciation also goes tothe editor and anonymous reviewers for their valuable commentsand suggestions, which were helpful in improving the paper.

References

Adomavicius, G., Kwon, Y., 2007. New recommendation techniques for multicriteriarating systems. Intelligent Systems, IEEE 22 (3), 48–55.

Adomavicius, G., Manouselis, N., Kwon, Y., 2011. Multi-criteria recommendersystems. In: Recommender Systems Handbook. Springer, pp. 769–803.

Akaike, H., 1974. A new look at the statistical model identification. AutomaticControl, IEEE Transactions on 19 (6), 716–723.

Batet, M., Moreno, A., Sánchez, D., Isern, D., Valls, A., 2012. Turist@: agent-basedpersonalised recommendation of touristic activities. Expert Systems withApplications 39, 7319–7329.

Benítez, J.M., Martín, J.C., Román, C., 2007. Using fuzzy number for measuringquality of service in the hotel industry. Tourism Management 28 (2), 544–555.

Berka, T., and Plnig, M. Designing recommender systems for tourism. In The 11thInternational Conference on Information Technology in Travel and Tourism,Cairo, Egypt, 2004.

Brown, M., Bossley, K.M., Mills, D.J., Harris, C.J., 1995. High dimensional neurofuzzysystems: overcoming the curse of dimensionality. In: Fuzzy Systems (Ed.),International Joint Conference of the Fourth IEEE International Conference onFuzzy Systems and The Second International Fuzzy Engineering Symposium,Proceedings of 1995 IEEE Int, Vol. 4. IEEE, pp. 2139–2146.

Buhalis, D., Law, R., 2008. Progress in information technology and tourismmanagement: 20 years on and 10 years after the internet—the state ofeTourism research. Tourism Management 29 (4), 609–623.

Buhalis, D., O’Connor, P., 2005. Information communication technologyrevolutionizing tourism. Tourism Recreation Research 30 (3), 7–16.

Cao, Y., Li, Y., 2007. An intelligent fuzzy-based recommendation system forconsumer electronic products. Expert Systems with Applications 33 (1), 230–240.

Carbo, J., Molina, J.M., 2004. Agent-based collaborative filtering based on fuzzyrecommendations. International Journal of Web Engineering and Technology 1(4), 414–426.

Castellano, G., Fanelli, A., Torsello, M., 2007. A neuro-fuzzy collaborative filteringapproach for web recommendation. International Journal of ComputationalScience 1 (1), 27–29.

Castillo, L., Armengol, E., Onaindía, E., Sebastiá, L., González-Boticario, J., Rodríguez,A., et al., 2008. Samap: an user-oriented adaptive system for planning touristvisits. Expert Systems with Applications 34, 1318–1332.

Cattell, R.B., 1966. The scree test for the number of factors. Multivariate BehavioralResearch 1 (2), 245–276.

Chiu, D.K.W., Yueh, Y.T.F., Leung, H.F., Hung, P.C.K., 2009. Towards ubiquitoustourist service coordination and process integration: a collaborative travelagent system architecture with semantic web services. Information SystemsFrontiers 11 (3), 241–256.

Chou, T.Y., Hsu, C.L., Chen, M.C., 2008. A fuzzy multi-decision model forinternational tourist hotels location selection. International Journal ofHospitality Management 27 (2), 293–301.


http://refhub.elsevier.com/S1567-4223(15)00059-9/h0005














































Connolly, D.J., Lee, S., 2006. Developing information technology proficiencies andfluency in hospitality students. Journal of Hospitality & Tourism Education 18(3), 15–29.

de Campos, L.M., Fernández-Luna, J.M., Huete, J.F., 2008. A collaborativerecommender system based on probabilistic inference from fuzzyobservations. Fuzzy Sets and Systems 159 (12), 1554–1576.

Delgado, J., Davidson, R., 2002. Knowledge bases and user profiling in travel andhospitality recommender systems. In: Proceedings of the IFITT’s Global Forumfor Travel and Tourism Technology and eBusiness, Focusing on Multi-Channel-Strategies, ENTER 2002. Springer.

Felfernig, A., Gordea, S., Jannach, D., Teppan, E., Zanker, M., 2007. A short survey ofrecommendation technologies in travel and tourism. OEGAI Journal 25 (7), 17–22.

Fenza, G., Fischetti, E., Furno, D., and Loia, V. A hybrid context aware system fortourist guidance based on collaborative filtering. In Proceedings of IEEE Int.Conference on Fuzzy Systems, 2011, 131–138.

Garcia, A., Arbelaitz, O., Linaza, M. T., Vansteenwegen, P., and Souffriau, W.Personalized tourist route generation. In F. Daniel and F. M. Facca (eds.), 10thInternational Conference on Web Engineering 2010, ICWE2010: Vol. 6385.LNCS, 2010, 486–497.

Garcia, A., Torre, I., Linaza, M.T., 2013. Mobile social travel recommender system. In:Information and Communication Technologies in Tourism 2014. SpringerInternational Publishing, pp. 3–16.

Gavalas, D., Kenteris, M., 2011. A web-based pervasive recommendation system formobile tourist guides. Personal and Ubiquitous Computing 15, 759–770.

Goldberg, K., Roeder, T., Gupta, D., Perkins, C., 2001. Eigentaste: a constant timecollaborative filtering algorithm. Information Retrieval 4 (2), 133–151.

Hellendoorn, H., Thomas, C., 1993. Defuzzification in fuzzy controllers. Journal ofIntelligent and Fuzzy Systems 1 (2), 109–123.

Herlocker, J., Konstan, J.A., Riedl, J., 2002. An empirical analysis of design choices inneighborhood-based collaborative filtering algorithms. Information Retrieval 5(4), 287–310.

Huang, Y., Bian, L., 2009. A Bayesian network and analytic hierarchy process basedpersonalized recommendations for tourist attractions over the Internet. ExpertSystems with Applications 36 (1), 933–943.

Jakkilinki, R., Georgievski, M., Sharda, N., 2007. Connecting destinations with anontology-based e-tourism planner. Information and CommunicationTechnologies in Tourism 2007, 21–32.

Jannach, D., Karakaya, Z., Gedikli, F., 2012a. Accuracy improvements for multi-criteria recommender systems. In: Proceedings of the 13th ACM Conference onElectronic Commerce. ACM, pp. 674–689.

Jannach, D., Gedikli, F., Karakaya, Z., and Juwig, O. Recommending hotels based onmulti-dimensional customer ratings. In eTourism Present and Future Servicesand Applications: Proceedings of ENTER 2012, Helsingborg, Sweden, 2012b,320–331.

Jannach, D., Zanker, M., Fuchs, M., 2014. Leveraging multi-criteria customerfeedback for satisfaction analysis and improved recommendations.Information Technology & Tourism 14 (2), 119–149.

Jung, Y.G., Kang, M.S., Heo, J., 2014. Clustering performance comparison using K-means and expectation maximization algorithms. Biotechnology &Biotechnological Equipment 28 (sup1), S44–S48.

Kanellopoulos, D.N., 2008. An ontology-based system for intelligent matching oftravellers’ needs for Group Package Tours. International Journal of DigitalCulture and Electronic Tourism 1 (1), 76–99.

Kurata, Y., Hara, T., 2013. CT-planner4: toward a more user-friendly interactive day-tour planner. In: Information and Communication Technologies in Tourism2014. Springer International Publishing, pp. 73–86.

Lakiotaki, K., Matsatsinis, N.F., Tsoukiàs, A., 2011. Multicriteria user modeling inrecommender systems. Intelligent Systems, IEEE 26 (2), 64–76.

Lee, C.S., Chang, Y.C., Wang, M.H., 2009. Ontological recommendation multiagent forTainan city travel. Expert Systems with Applications 36, 6740–6753.

Lenar, M., Sobecki, J., 2007. Using recommendation to improve negotiations inagent-based systems. Journal of Universal Computer Science 13 (2), 267–286.

Lesaffre, M., Leman, M., 2007. Using fuzzy logic to handle the users’ semanticdescriptions in a music retrieval system. In: Theoretical Advances andApplications of Fuzzy Logic and Soft Computing. Springer, pp. 89–98.

Liu, L., Mehandjiev, N., Xu, D.L., 2011. Multi-criteria service recommendation basedon user criteria preferences. In: Proceedings of the Fifth ACM Conference onRecommender Systems. ACM, pp. 77–84.

Loh, S., Lorenzi, F., Saldaa, R., Licthnow, D., 2004. A tourism recommender systembased on collaboration and text analysis. Information Technology and Tourism6 (3), 157–165.

Lucas, J.P., Luz, N., Moreno, M.N., Anacleto, R., Almeida Figueiredo, A., Martins, C.,2013. A hybrid recommendation approach for a tourism system. Expert Systemswith Applications 40 (9), 3532–3550.

Manouselis, N., Costopoulou, C., 2007. Experimental analysis of design choices inmultiattribute utility collaborative filtering. International Journal of PatternRecognition and Artificial Intelligence 21 (02), 311–331.

McLaughlin, M.R., Herlocker, J.L., 2004. A collaborative filtering algorithm andevaluation metric that accurately model the user experience. In: Proceedings ofthe 27th Annual International ACM SIGIR Conference on Research andDevelopment in Information Retrieval. ACM, pp. 329–336.

Meehan, K., Lunney, T., Curran, K., and McCaughey, A. Context-aware intelligentrecommendation system for tourism. In Proceedings of the 11th IEEEInternational Conference on Pervasive Computing and Communications, 2013,328–331.


Mitra, P., Pal, S.K., Siddiqi, M.A., 2003. Non-convex clustering using expectationmaximization algorithm with rough set initialization. Pattern RecognitionLetters 24 (6), 863–873.

Moreno, A., Valls, A., Isern, D., Marin, L., Borràs, J., 2013. SigTur/e-destination:ontology-based personalized recommendation of tourism and leisure activities.Engineering Applications of Artificial Intelligence 26 (1), 633–651.

Morrison, Alastair, Jing, Su, O’Leary, Joseph, Cai, Liping, 2001. Predicting usage of theInternet for travel bookings: an exploratory study. Information Technology andTourism 4 (1), 15–30.

Nathiya, G., Punitha, S.C., Punithavalli, M., 2010. An analytical study on behavior ofclusters using k means, em and k⁄ means algorithm. International Journal ofComputer Science and Information Security 7 (3), 155–190.

Ngai, E.W.T., Wat, F.K.T., 2003. Design and development of a fuzzy expert system forhotel selection. Omega 31 (4), 275–286.

Nielsen J. Participation inequality: encouraging more users to contribute. JakobNielsen’s alertbox, 2006.

Nilashi, M., bin Ibrahim, O., Ithnin, N., 2014a. Hybrid recommendation approachesfor multi-criteria collaborative filtering. Expert Systems with Applications 41(8), 3879–3900.

Nilashi, M., bin Ibrahim, O., Ithnin, N., 2014b. Multi-criteria collaborative filteringwith high accuracy using higher order singular value decomposition and Neuro-Fuzzy system. Knowledge-Based Systems 60, 82–101.

Nilashi, M., Jannach, D., bin Ibrahim, O., Ithnin, N., 2014c. Clustering-and regression-based multi-criteria collaborative filtering with incremental updates.Information Sciences.

Nilashi, M., Jannach, D., bin Ibrahim, O., Ithnin, N., 2015a. Clustering-and regression-based multi-criteria collaborative filtering with incremental updates.Information Sciences 293, 235–250.

Nilashi, M., Ibrahim, O., Mirabi, V.R., Ebrahimi, L., Zare, M., 2015b. The role ofSecurity, Design and Content factors on customer trust in mobile commerce.Journal of Retailing and Consumer Services 26, 57–69.

Nilashi, M., Zakaria, R., Ibrahim, O., Majid, M.Z.A., Zin, R.M., Chugtai, M.W., et al.,2015c. A knowledge-based expert system for assessing the performance level ofgreen buildings. Knowledge-Based Systems 86, 194–209.

Noguera, J.M., Barranco, M.J., Segura, R.J., Martínez, L., 2012. A mobile 3D-GIS hybridrecommender system for tourism. Information Sciences 215, 37–52.

O’Connor, P., 2008. User-generated content and travel: A case study onTripadvisor.com. In: Information and Communication Technologies in Tourism2008. Springer, pp. 47–58.

Ordonez, C., Omiecinski, E., 2002. FREM: fast and robust EM clustering for large datasets. In: Proceedings of the Eleventh International Conference on Informationand Knowledge Management. ACM, pp. 590–599.

Pan, Bing, Litvin, Stephen W., O’Donnell, Thomas E., 2007. Understandingaccommodation search query formulation: the first step in putting ‘heads inbeds’. Journal of Vacation Marketing 13 (4), 371–381.

Pan, Bing, Xiang, Zheng, Fesenmaier, Daniel R., Law, Rob, 2011. The dynamics ofsearch engine marketing for tourist destinations. Journal of Travel Research 50(4), 365–377.

Pazzani, M., 1999. A framework for collaborative, content-based and demographicfiltering. Artificial Intelligence Review.

Pelleg, D. and Moore, A. W. X-means: extending K-means with efficient estimationof the number of clusters. In ICML, 2000, 727–734.

Pinto, M.A., Tanscheit, R., Vellasco, M., 2012. Hybrid recommendation system basedon collaborative filtering and fuzzy numbers. In: Fuzzy Systems (FUZZ-IEEE),2012 IEEE International Conference on. IEEE, pp. 1–6.

Polo Peña, A.I., Frías Jamilena, D.M., Rodríguez Molina, M.A., 2013. Impact ofcustomer orientation and ICT use on the perceived performance of rural tourismenterprises. Journal of Travel & Tourism Marketing 30 (3), 272–289.

Popescu, A., Grefenstette, G., 2011. Mining social media to create personalizedrecommendations for tourist visits. In: Proceedings of the 2nd InternationalConference on Computing for Geospatial Research & Applications. ACM,p. 37.

Prihasto, B., Irawan, M.I., Masduqi, A., 2014. Fuzzy MADM method for decisionsupport system based on artificial neural network towater quality assessment inSurabaya river. Journal of Soft Computing and Decision Support Systems 1 (1),24–29.

Ricci, F. Travel recommender systems. Tech. rep., eCommerce and Tourism ResearchLaboratory, ITC-Irst, Trento, Italy, 2002.

Ricci, F., Nguyen, Q.N., 2007. Acquiring and revising preferences in a critique-basedmobile recommender system. Intelligent Systems, IEEE 22 (3), 22–29.

Ricci, F., Werthner, H., 2002. Case base querying for travel planningrecommendation. Information Technology & Tourism 4 (3/4), 215–226.

Ruotsalo, T., Haav, K., Stoyanov, A., Rochee, S., Fanid, E., Deliaic, R., et al., 2013.SMARTMUSEUM: a mobile recommender system for the web of data. WebSemantics: Science, Services and Agents on the World Wide Web 20,50–67.

Sahoo, N., Krishnan, R., Duncan, G., and Callan, J. P. Collaborative filtering withmulti-component rating for recommender systems. In Proceedings of theSixteenth Workshop on Information Technologies and Systems, 2006.

Sahoo, N., Krishnan, R., Duncan, G., Callan, J., 2012. Research note-the halo effect inmulticomponent ratings and its implications for recommender systems: thecase of Yahoo! movies. Information Systems Research 23 (1), 231–246.

Sarwar, B., Karypis, G., Konstan, J., and Riedl. J. Application of dimensionalityreduction in recommender system-a case study. DTIC Document, 2000.

Schiaffino, S., Amandi, A., 2009. Intelligent user profiling. In: Artificial IntelligenceAn International Perspective. Springer, Berlin Heidelberg, pp. 193–216.























































































































































Sen, S., Vig, J., Riedl, J., 2009. Tagommenders: connecting users to items throughtags. In: Proceedings of the 18th International Conference on World Wide Web.ACM, pp. 671–680.

Shambour, Q., Lu, J., 2011a. A hybrid trust-enhanced collaborative filteringrecommendation approach for personalized government-to-business e-services. International Journal of Intelligent Systems 26, 814–843.

Shambour, Q., Lu, J., 2011b. A hybrid multi-criteria semantic-enhanced collaborativefiltering approach for personalized recommendations. In: Proceedings of the2011 IEEE/WIC/ACM International Conferences on Web Intelligence andIntelligent Agent Technology-Volume 01. IEEE Computer Society, pp. 71–78.

Shani, G., Gunawardana, A., 2011. Evaluating recommendation systems. In:Recommender Systems Handbook. Springer, US, pp. 257–297.

Si, L., and Jin, R. Flexible mixture model for collaborative filtering. In ICML, Vol. 3,2003, 704–711.

Singh, Arjun J., Kasavana, Michael L, 2005. The impact of information technology onfuture management of lodging operations: a Delphi study to predict keytechnological events in 2007 and 2027. Tourism & Hospitality Research 6 (1),24–37.

Stock, O., Not, E., Zancanaro, M., 2005. Intelligent interactive informationpresentation for cultural tourism. In: Multimodal Intelligent InformationPresentation. Springer, Netherlands, pp. 95–111.

Henrik Stormer, Nicolas Werro and Daniel Risch. Recommending Products with aFuzzy Classification. CollECTeR Europe 2006, 2006.

Tang, T.Y., McCalla, G., 2009. The pedagogical value of papers: a collaborative-filtering based paper recommender. Journal of Digital Information 10 (2).


Vansteenwegen, P., Souffriau, W., 2011. Trip planning functionalities: state-of the-art and future. Information Technology and Tourism 12 (4), 305–315.

Wallace, M., Maglogiannis, I., Karpouzis, K., Kormentzas, G., Kollias, S., 2003.Intelligent one-stop-shop travel recommendations using an adaptive neuralnetwork and clustering of history. Information Technology & Tourism 6 (3),181–193.

Wan Lee, J., Brahmasrene, T., 2013. Investigating the influence of tourism oneconomic growth and carbon emissions: evidence from panel analysis of theEuropean Union. Tourism Management 38, 69–76.

Wang, Y., Yu, Q., Fesenmaier, R.D., 2002. Defining the virtual tourist community:Implications for tourism marketing. Tourism Management 23 (4), 407–417.

Werthner, H., Ricci, F., 2004. E-commerce and tourism. Communications of the ACM47 (12), 101–105.

World Tourism Organization, 2006. Tourism Market Trends 2005: World Overview& Tourism Topics. World Tourism Organization, Madrid, Spain.

World Travel & Tourism Council. Travel & Tourism Economic Impact, 2009.Xiang, Zheng, Pan, Bing, 2010. Travel queries on cities in the United States:

implications for search engine marketing for tourist destinations. TourismManagement 32 (1), 88–97.

Yager, R.R., 2003. Fuzzy logic methods in recommender systems. Fuzzy Sets andSystems 136 (2), 133–149.

Yin, C.X., Peng, Q.K., 2012. A careful assessment of recommendation algorithmsrelated to dimension reduction techniques. Knowledge-Based Systems 27, 407–423.

Zadeh, L.A., 1965. Fuzzy sets. Information and Control 8 (3), 338–353.
















































a multi-criteria collaborative filtering recommender ... · pdf filea multi-criteria...

Documents