A Rule-Based Recommender System for OnlineDiscussion Forums

Fabian Abel2, Ig Ibert Bittencourt1, Nicola Henze2, Daniel Krause2, JulitaVassileva3

1 Federal University of Alagoas, Computer Science InstituteMaceio, AL, Brazil

ibert@dsc.ufcg.edu.br2 IVS - Semantic Web Group, Leibniz University of Hannover

Appelstr. 4, D-30167 Hannover, Germany{abel, henze, krause}@kbs.uni-hannover.de

3 Department of Computer Science, University of Saskatechwan110 Science Place, Saskatoon SK S7N 5C9, Canada jiv@cs.usask.ca

Abstract. In this paper we present a rule-based personalization frame-work for encapsulating and combining personalization algorithms knownfrom adaptive hypermedia and recommender systems. We show how thispersonalization framework can be integrated into existing systems byexample of the educational online board Comtella-D, which exploits theframework for recommending relevant discussions to the users. In ourevaluations we compare different recommender strategies, investigate us-age behavior over time, and show that a small amount of user data issufficient to generate precise recommendations.

1 Introduction

Online discussion forums allow people to discuss different topics using the WorldWide Web. While a discussion forum has often one large overall topic, it isnormally divided into subtopics, called subforums or topics. The subforums arefurther divided into single threads. In these threads, one specific question, definedby the thread opener, is discussed by several users. Every user who wants tocontribute can create text snippets, called posts which are ordered by the timethey have been created. Posts can be displayed as a list and enable other usersto follow the discussion easily.

The tree-like structure of the discussion forums enables users to navigatequickly to the topics which they are interested in. A drawback of the structureis that it is hard to find interesting threads if the thread is either not classifiedcorrectly by the thread opener or the user does not know how a specific topicof his interest is classified into the static discussion forum’s hierarchy. Anotherdrawback is that every thread can be assigned to one category, making threadsmatching to multiple categories hard to find. A commonly used approach to han-dle the problems caused by the described classification is to display a flattenedlist of all topics which encountered recent changes as a starting point. However,when forums become popular and hence large or the community discusses vari-ous topics, these lists contain a high percentage of threads which are not relevant

for the user. This results in the situation that interested users encounter seriousproblems to find relevant threads when the forum grows.

Collaborative recommender systems can be used to cope with the issue ofbringing users and relevant tasks together. In an E-Learning Online DiscussionForum like Comtella-D there are different kinds of input data which can be usedto create recommendations. In this paper we evaluate a) which kind of inputdata fits best and b) how much input data is required to generate appropriaterecommendations. Based on the results of this evaluation, we propose a rulebased framework which chooses the optimal input data source: Recommendersystems based on different input data sources are implemented as Web Servicesof the Personal Reader Framework [1]. A rule layer enables on the one handto pass parameters to single personalization Web Services, PServices for short,and on the other hand to combine the results of different PServices. Defaultrules allow Comtella-D users to use this personalization framework immediatelywithout any interaction. Moreover, user adjustable parameters in the rule allowfor fine tuning of the rules if a user is not satisfied with the recommendations.

The paper is structured as follows: In Section 2 we describe our rule-basedpersonalization system. In Section 3 we describe the Comtella-D system andoutline the need for personalization in this system. Afterwards, by evaluatingdifferent recommender strategies, we define a flexible personalization rule inSection 4 that performs best in creating recommendation in different scenarios.Section 5 contains related work and Section 6 gives a conclusion and some furtherideas to be exploited.

2 Rule-based Personalization System

Personalization techniques have been investigated extensively in different areasof computer science. Especially in the domains of recommender systems [2] andadaptive hypermedia [3], personalization algorithms have been developed anddeployed in various systems.

These personalization techniques are generic as the algorithms can be de-ployed in different domains, changing the domain specific input data withoutthe need of modifying the algorithm itself. Hence, personalization algorithmsare perfect candidates for being encapsulated to become reusable. However, incurrent systems these algorithms are often strongly coupled with the system asthe data is often domain specific pre- or postprocessed or combined with otheralgorithms. In our system we decouple personalization algorithms, data sourcesand pre- and postprocessing from each other and allow the creation of ruleswhich describe the interaction of the single components.

Figure 1 shows the architecture of the rule-based recommender system. Itemphasizes two aspects. First, it assures the integration of different recommen-dation algorithms based on the use of Web Services. Second, show how to inte-grate external personalization functionality like recommendations. A descriptionof the components of the architecture is listed below. However, it is not the focusof this paper to describe each component in detail. The technologies used in the

development of the system were Java, Protege4, SWRL5, OWL-S Editor6, andMindSwap7.

Fig. 1. Architecture of the System

DB represents all databases that can be used for personalization, for e.g. userprofiles or data provided in the Web

DS each data source represents an encapsulated personalization algorithm likea collaborative recommender system

SWS (Web Services) – each data source can be accessed as Web ServiceComtella Application it represents the Comtella application (more details

are described in the next section)Rule-based Recommendation Interface this interface is used to specify per-

sonalization rules. Section 4.5 gives an example of such a rule.SWRL (Semantic Web Rule Language) it is used to specify the conversion

of information between the Comtella-D application and the data sources

3 The Comtella-D System

Comtella Discussions (Comtella-D) [4] is an online community for discussingthe social, ethical, legal and managerial issues associated with information tech-nology and biotechnology. It was used to support the coursework related to a4th year undergraduate class on Ethics and IT taught in the spring of 2006at the University of Saskatchewan. Access to content is restricted to registeredmembers, but anyone may create an account at http://fire.usask.ca after con-senting to release their access data for research purposes. A nickname/alias,e-mail address, and password are required to create an account. Members arerelatively anonymous because they are identified just by their alias. The purposeof using Comtella-D in the class was sharing and discussing information (Inter-net publications, popular magazine, articles, etc.) related to the course topics.The students had to share at least one link to an online article related to theweekly topic and summarize the article in a way that it stimulates discussion.4 http://protege.stanford.edu/5 http://www.w3.org/Submission/SWRL/6 http://owlseditor.semwebcentral.org/7 http://www.mindswap.org/

As a part of their coursework, the students also had to reply/discuss two oftheir colleagues’ postings each week. In parallel with the students of the Ethicsand IT class, (4th year Computer Science students), the Comtella-D system wasused in a class on Ethics and Technology offered by the Philosophy department.These students used the system only as an additional resource, recommended bythe instructor. The system was not related to their coursework and it was usedentirely voluntary.

In the context of Comtella-D, a ’forum is an initial theme related to a coursetopic (usually weekly), defined and created by the instructor. A ’thread is startedwhen a student contributes a link (URL) of a paper related to the topic ofthe forum. The first ’post in a new thread contains the URL and a summaryof the paper (usually half a page). Further ’posts in the thread are added asother students respond to/discuss the first post of the thread. Each post can becommented. A ’comment is usually a very specific local comment to the postrather than to the entire thread. In Comtella-D comments were used mostly bythe marker to give feedback on the quality of arguments raised in the studentsposts.

Comtella-D allows students to rate posts by adding or removing ’energy toor from it. A user can rate every post once, but only if there is free energy inthe system available. The system provides a limited number of energy units,depending on the level of activity in the system. The number of energy unitsin the system increases every time when a new post is created (2 new units areadded), and it decays with time. In this way, the scarcity of energy in the systemprevents users from overrating their colleagues posts, and encourages them tocarefully read a post before assigning energy to it. This mechanism is describedin [4].

As every week several new threads are started and popular threads attractmany posts, keeping an overview of the discussion is a time consuming task. Astudent who does not spend the time to read all new posts could easily missimportant topics of his/her interest. Hence, a recommender system is neededwhich points the student to relevant posts. Using our rule-based personaliza-tion framework, we can utilize collaborative recommender services to solve thistask. Based on the features of Comtella-D, there are different possibilities onwhich input data such a collaborative recommender can perform: a) recommen-dations based on explicit feedback gained from the user’s energy rating and b)recommendations based on implicit feedback gained from co-posting in the samethread.

In the following section we will evaluate which kind of user feedback fitsbetter to recommend threads a user might be interested in. Therefore, we alsoevaluate how much input data is required and over which time frame this inputdata has to be provided to generate high quality recommendations.

4 EvaluationFor the evaluation we took a snapshot of the Comtella-D system of the Ethicsand Computer Science course 2006. Overall, there were 110 registered users.From these users only 36 contributed actively by posting a least one message in

the discussion forum. Users rated other users 183 time and posted 756 messagesin 173 threads over a time period of approximately 3 months. In these threemonths, the lectures deal every week with a new topic.

To define a personalization rule which recommends threads a user could beinterested in, we use the existing user interaction with the system. Before creatingthis rule, we have to examine different questions: a) How much training data isrequired to generate precise recommendations (Section 4.1)? b) What kind ofinput data (explicit or implicit) gives the best quality to recommend threads(Section 4.2)? c) Does the behavior of users in the discussion forum change overtime (Section 4.3)? d) Are active users, i.e. users who have posted frequentlyand hence are more experienced, more reliable as source for recommendations(Section 4.4)?.

For all of the following measurements, we used a recommender library8 whichimplemented the collaborative recommender algorithm described in [5].

4.1 Required Amount of Training DataAccording to the first question we divided our data set into weeks correspondingto the different topics of the lectures. Afterwards, we iterated over the weeks,selecting every week x as training set and tried to calculate the posts a specificuser will create in week x + 1. Therefore, we classified the users into differentclasses, these classes contain sets of users who have posted at least y posts indifferent threads and at least 1 post in the test set. Furthermore, as a non-personalized baseline algorithm, we recommend the top-k threads having themost posts. Our hypothesis is that the more data from a user is available in thetraining set, the more precise the recommendation for the test set are.

The precision-recall distribution is build by iterating over all users in the classand calculating the top-k recommendations for these users. k is chosen from 1to the number of all posts. For every k, the precision and recall is calculatedas the average mean of all precision and recall values of all users in the class.Therefore, the recommendation system is invoked as follows: First, the postsgenerated in the training set are passed to the recommender system to determinethe similarity between the users. Afterwards, the recommendations are calculatedby passing all posts to the recommender system which were created in the testset.

Figure 2 displays the precision-recall distribution for the non-personalizedbaseline algorithm and the personalized recommendations based on users whohave contributed at least 2, 3, 4, or 5 posts in the training set. While for k <= 3the classes 3 to 5 perform better than class 2, class 2 performs better for k > 4.However, none of the different classes results in significantly better results thanthe other classes. Furthermore, all approaches are able to retrieve not more than80% of the threads the users have contributed to. This can be explained by thecharacteristics of the recommendation process: When a thread is recommended,a user who is similar to the current user must have contributed to this thread.Hence, threads which are discussed by only a few users are recommended rarely.8 http://www.l3s.de/˜diederich/SW/renkground-2006-09-07-1030.zip

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Pre

cisi

on

Recall

min posts = 5min posts = 4min posts = 3min posts = 2

baseline

Fig. 2. The precision-recall diagram based on implicit user feedback for users who haveposted at least 2, 3, 4, or 5 times in the training set week.

This issue is known as new item problem in collaborative recommender systems[6].

Overall, the results imply that a) the non-personalized baseline algorithm isoutperformed by the personalized algorithm and that b) two posts in a week aresufficient to generate precise personalized recommendations while more posts donot improve this the results significantly.

4.2 Implicit vs. Explicit User Feedback

Based on the classes defined in the previous section which used implicit userfeedback by engaging the posts a user created, we define equivalent classes ofexplicit user feedback: These classes contain users who have at least added orremoved x energy points to posts from other users in the training set week andhave at least posted once in the test set week.

To recommend posts by using user ratings we modified the similarity functionof the recommender system. Instead of comparing the similarity of user vectorscontaining threads a user has posted in, we use vectors containing the energydistribution. Two users are considered as similar when they gave energy to thesame post, hence expressing interest in the same post. We did not take intoaccount if users added or removed energy as we interpreted every form of energyassignment as interest in a post. The recommender algorithm itself was notmodified.

Figure 3 gives an overview of the precision-recall ratio of recommendationsbased on explicit feedback for the classes of users having rated at least 2, or 3other users in the training set period. The class with 5 energy assignments wasomitted as it contained not enough users to deliver reliable results. The graphoutlines that – like in the previous section – a comparable small amount ofinput data, namely two energy assignments, are sufficient to create appropriate

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Pre

cisi

on

Recall

min energy = 2min energy = 3min energy = 4

baseline

Fig. 3. The precision-recall diagram based on explicit user feedback for users who haverated at least 2, 3, or 4 posts of other users in the training set week.

recommendations and that increasing the amount of input data does not increasethe precision or recall of the recommendations significantly. Compared to theprecision-recall distribution generated by implicit user feedback, the quality ofthe results generated by explicit feedback, in respect of both, precision and recall,are lower.

We also tried to combine explicit feedback and implicit feedback as we ex-pected that input from different sources could improve the overall performance.We used the average mean to combine the weighted result sets of the recom-mendations based on explicit feedback and implicit feedback. We examined thatthe more we increased the weight of the explicit user feedback, the worse ourrecommender system performed. Our conclusion for the given setting is that ex-plicit feedback performs always worse than implicit feedback and cannot be usedto improve recommendation based on implicit feedback. However, if no implicitfeedback is given for a specific user, explicit feedback performs better than thenon-personalized baseline algorithm. Hence, explicit feedback based recommen-dations can be used as a fallback if no implicit feedback is available.

Based on these results we used implicit user feedback as source for the re-commendations applied in the following evaluations.

4.3 User BehaviorThe Comtella-D system was strongly coupled with the timeline of the lectures.This means that the users discussed every week a new topic. We assume that thebehavior of users changes over time (and over different topics) which means thatthe more weeks ahead recommendations are created, the more imprecise theyare. Furthermore, as topics discussed in a given week should be still somewhatfresher in the memory of the students, we assume that the forecast for the nextweek would be more precise than forecasts for two or more weeks ahead.

To verify our assumptions, we iterated over all weeks and used them astraining data. We calculate the recommendations for n weeks ahead, where n =1, 2, .., 7 and compared them with the test data. Afterwards, we created theprecision-recall diagram displayed in Figure 4.3.

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Pre

cisi

on

Recall

1 week ahead2 weeks ahead3 weeks ahead4 weeks ahead5 weeks ahead6 weeks ahead7 weeks ahead

Fig. 4. User behavior over time

The figure displays a result which does not comply with our assumptions:The one week ahead precision-recall values for small top-k result sets are worsethan all other forecasts. Furthermore, the forecasts for more weeks ahead do notcomply to any rule or trend. This means that the behavior of the users indeedchanges over time and topic, but that the change of behavior is not monotonicand cannot be forecasted. However, we have to remark that our dataset coversonly three months of data. Thus, we can only infer about the short time behaviorof users but cannot conclude that there is not a long time trend in user behavior.

4.4 Size of the Time frame

In the previous section we have shown that the user behavior changes over theweeks making a constantly high forecast for several weeks ahead impossible. Tolower this effect, we increase the input data timeframe by aggregating severalweeks as training set and creating recommendations for one week ahead. Weexpect that aggregating several weeks of input data normalizes the behavior ofa user on one hand and increases the amount of input data one the other. Botheffects should result in an increased quality of the recommendations. Figure5 displays the measurement aggregating one to five weeks of input data andcalculating the precision and recall of the recommendations for the followingweek.

All input periods result in similar results. Our expectation that more inputweeks could improve the result could not be proven. This also underlines our

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Pre

cisi

on

Recall

1 input week2 input weeks3 input weeks4 input weeks5 input weeks

Fig. 5. Variation of the amount of weeks used as training data

previous observation that the changes of quality regarding precision and recallseem to follow no rule or trend.

4.5 ResultsThe results show that a small amount of input data (two posts or energy assign-ments) is enough to generate precise information. Furthermore, we have shownthat the implicit user feedback, given by the posting behavior of users givesmuch better recommendations than explicit user feedback given by the energyassignment of the users. Also we have shown that more input data does not au-tomatically result in better recommendations. According to these observations,an optimal personalization strategy to recommend threads in the Comtella-Dsystem is the following:

if exist 2 or more posts of the user:-> recommendation based on implicit feedbackelse if exist 2 or more energy assignments of the user:-> recommendation based on explicit feedbackelse use the non-personalized baseline algorithm

5 Related WorkOur framework aims on decoupling personalization functionality from a specificapplication. For single domains, for e.g. the e-learning domain, there alreadyexist approaches that realize such a decoupling [7]. However, to the best of ourknowledge there exists no generic approach describing such an encapsulation ofpersonalization functionality.

Also different personalization techniques are already combined to overcomethe disadvantages of single personalization techniques. In the domain of re-commender systems, these combination techniques are known as hybrid re-commender systems systems [6], utilizing for example both, collaborative and

content based recommender systems to overcome the new item or new userproblem. The rule-based framework, however, does not only allow for a staticcombination of different personalization techniques. Instead, every rule can com-bine arbitrary personalization techniques.

6 Conclusion and Future Work

In this paper we presented a rule-based framework to combine arbitrary per-sonalization techniques. Therefore, personalization techniques are encapsulatedand separated from their input data to be reusable in different applications. Weused the Comtella-D system to outline how the framework could be used torecommend forum threads. We specified a rule which selects the optimal recom-mendation technique according to the existing user information. To determinethe best strategies, we evaluated which kinds of input data and which quantityis required to provide accurate recommendations.

In the future, we plan to make the rule user-adjustable. This can be doneby introducing user adjustable weights in the rule which enable the combinationof different techniques according to a user’s preferences. Additionally, we planto engage content-based recommender systems which take the text of the postsinto account to improve the quality of recommendations further.

References

1. Abel, F., Baumgartner, R., Brooks, A., Enzi, C., Gottlob, G., Henze, N., Herzog,M., Kriesell, M., Nejdl, W., Tomaschewski, K.: The personal publication reader,semantic web challenge 2005. In: 4th International Semantic Web Conference. (nov2005)

2. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems:A survey of the state-of-the-art and possible extensions. IEEE Transactions onKnowledge and Data Engineering 17(6) (2005) 734–749

3. Brusilovsky, P.: Adaptive Hypermedia. User Modeling and User-Adapted Interac-tion 11 (2001) 87–110

4. Webster, A., Vassileva, J.: Visualizing personal relations in online communities. InWade, V.P., Ashman, H., Smyth, B., eds.: AH. Volume 4018 of Lecture Notes inComputer Science., Springer (2006) 223–233

5. Shardanand, U., Maes, P.: Social information filtering: Algorithms for automating“word of mouth”. In: Proceedings of ACM CHI’95 Conference on Human Factorsin Computing Systems. Volume 1. (1995) 210–217

6. Burke, R.: Hybrid recommender systems: Survey and experiments. User Modelingand User-Adapted Interaction 12(4) (2002) 331–370

7. Brusilovsky, P., Henze, N.: Open corpus adaptive educational hypermedia. InBrusilovsky, P., Kobsa, A., Nejdl, W., eds.: The Adaptive Web. Volume 4321 ofLecture Notes in Computer Science., Springer (2007) 671–696