Expert Systems with Applications 39 (2012) 9797–9806

Contents lists available at SciVerse ScienceDirect

Expert Systems with Applications

journal homepage: www.elsevier .com/locate /eswa

Adaptive neuro-fuzzy pedagogical recommender

Zoran Sevarac ⇑, Vladan Devedzic, Jelena JovanovicDepartment of Software Engineering, Faculty of Organizational Sciences, University of Belgrade, Jove Ilica 154, 11000 Belgrade, Serbia

a r t i c l e i n f o a b s t r a c t

Keywords:E-learningRecommender systemsNeural networksFuzzy logic

0957-4174/$ - see front matter � 2012 Elsevier Ltd. Adoi:10.1016/j.eswa.2012.02.174

⇑ Corresponding author. Tel.: +381 63 8427 533, +32461 221.

E-mail address: sevarac@gmail.com (Z. Sevarac).

Neuro-Fuzzy Pedagogical Recommender (NFPR) is adaptive recommender based on neuro-fuzzy infer-ence, that can be used to create pedagogical rules in Technology Enhanced Learning (TEL) systems. NFPRis domain independent, provides easy to use API for integration with other systems, and comes with spe-cialized tool (wizard) for creation of NFPR software components. The most important feature of NFPR isits flexibility, that allows teachers to create their own set of pedagogical rules. The proposed model hasbeen implemented and tested with simulated data. Our effort in bringing neuro-fuzzy recommender tothe field of TEL (Technology Enhanced Learning) seems to be the first attempt of its kind.

� 2012 Elsevier Ltd. All rights reserved.

1. Introduction

In general, recommender systems are used for recommendingsome items that might be of interest to the users. Recommenda-tions are typically given based on information such as user profiles,item properties (content based recommenders), and users prefer-ences (collaborative filtering) expressed explicitly (e.g., by user rat-ings and ‘likes’) and/or implicitly (e.g., by the frequency of visits/downloads) (Jameson, Konstan, & Riedl, 2002). By combining thisinformation with a set of recommendation rules, a recommendersystem tries to predict which items will be of interest to the user,so that he/she can achieve some predetermined goals.

Important questions to be addressed when designing recom-mender systems include (but are not limited to):

1. What are the most effective techniques for recommendation inspecific domain?

2. What information about the users is needed; how to collect andrepresent it?

3. What information about the items is needed; how to collect andrepresent it?

4. How to evolve and adapt recommendations in order to makethem continuously effective (i.e., to sustain their effectivenessdespite the changes e.g., in users preferences or any otherrequirements)?

In addition to these challenges, information gathered about theusers is often incomplete or unreliable, and that makes the gener-ation of useful recommendations even more difficult.

ll rights reserved.

81 11 3950 853; fax: +381 11

In the context of Technology Enhanced Learning (TEL), recom-mender systems are used for suggesting learning activities, materi-als and/or topics to students in order to assist them in achievingtheir desired learning goals – in general, to increase their level ofknowledge on some subject (Tang & McCalla, 2003). In this case,the recommendation problem can be defined as the student’srequest to the system: ‘‘Given a representation of my current knowledgeand preferences, recommend me the next topic/content/activity in order tohelp me learn the given subject’’ (Basu, Hirsh, Cohen, & Nevill-Manning,2001).

To address this request, the system generates recommendationsbased on the student model (i.e., its internal representation of thestudents’ knowledge and preferences), and the teaching model (i.e.,the chosen pedagogical strategy usually defined by the teacher).Student model typically contains information about the student’sknowledge, preferences, learning style and the accomplishedlearning activities. This information is often extracted from historyof interaction between the student and the learning environment.Teaching model defines a pedagogical strategy typically as a set ofrules that determines the optimal way for learning some topic forcertain type of students.

The challenging issues in educational recommender systems arethe same as those recognized in other recommender systems:

1. How to collect and represent relevant information about stu-dents, and how to structure the student’s model?

2. How to use the student-related data (stored in the studentmodel) to generate useful recommendations, i.e., how to defineand evolve pedagogical recommendation rules?

3. From the practical point of view, how to implement the recom-mendation rules in the most efficient way?

The first two questions can be addressed by leveraging the re-search work and the results achieved by other researchers in the

9798 Z. Sevarac et al. / Expert Systems with Applications 39 (2012) 9797–9806

field (see Section 2). The last question is especially challenging con-sidering that at the moment there is a lack of open implementa-tions of general pedagogical recommenders that could be reusedin different domains and TEL systems. Aiming to address this tech-nical challenge, we have developed an open and adaptive softwarecomponent, named Neuro Fuzzy Pedagogical Recommender(NFPR), for creating pedagogical recommenders in learning envi-ronments. NFPR, which is the central topic of this paper, providesa wizard-style user interface and an easy-to-use API, which makesit suitable for easy integration with various learning environments.

The paper is organized as follows: related work which includesrecommender and neuro-fuzzy systems for TEL is given in Section2; Section 3 outlines current challenges in the field of pedagogicalrecommender systems and clearly states the problem this workaims to address; the overall architecture of the proposed softwaresystem and the algorithms it is based upon are given in Section 4;these are followed by a basic working example (Section 5) and asample application (Section 6); usability and pedagogical evalua-tion is given in Section 7, whereas Section 8 outlines conclusionsof this research.

2. Related work

Related work for this research includes recommender systemsin general, recommender systems in TEL and neuro-fuzzy systemsin TEL. Accordingly, this section gives a brief overview of thesethree research areas.

Currently, a widely used state of the art approach in recom-mender systems is Matrix Factorization (Bell, Koren, & Volinsky,2009) which belongs to collaborative filtering family of recom-mender systems (Bobadilla, Hernando, Ortega, & Bernal, 2011).The most general data representation technique applied in thistype of recommender systems is a matrix of n users and m items,where each matrix cell corresponds to the rating given to item iby the user u (Melville & Sindhwani, 2010, chap. 00338). The Ma-trix Factorization algorithm is used to predict which item will havethe highest rating for some user, based on the ratings of otheritems by that user and ratings of other users. The main issue withthis approach is the so called ‘cold start’ problem, which means thatin the beginning there is not enough data (ratings) to make goodrecommendations, and it is not possible to give recommendationsfor new users (before they provide some ratings) and new items(before they get some ratings). In practice these issues are resolvedusing simple average ratings, by creating hybrid recommenders incombination with content filtering techniques (Hummel et al.,2007), or using some more sophisticated methods (Gantner, Dru-mond, Freudenthaler, Rendle, & Schmidt-Thieme, 2010; Preisach,Marinho, & Schmidt-Thieme, 2010). A wide range of other tech-niques, including statistics and machine learning based techniques,are also used in order to analyze data and give recommendations(Melville & Sindhwani, 2010, chap. 00338).

In the area of recommender systems for Technology EnhancedLearning (Manouselis, Drachsler, Vuorikari, Hummel, & Koper,2010), research is focused on the construction of recommendersystems for recommendation of learning resources (materials orpeers to provide help) or learning activities to the learners (Ghauth& Abdullah, 2010; Manouselis et al., 2010). Recommender systemsfor educational purposes are challenging research direction(Drachsler, Hummel, & Koper, 2009) since preferred learning activ-ities of students might pedagogically not be the most adequate(Tang & McCalla, 2004) and recommendations in e-learning shouldbe guided by educational objectives, and not only by the user’spreferences (Santos & Boticario, 2010). Also, there are a numberof specific features that have to be taken into account, such as(Drachsler et al., 2009):

– the importance of context (which is not taken into account incommon recommender systems);

– the inherent novelty of most learning activities;– the need for a learning strategy;– the need to take changes and learning processes into account.

There are many different approaches for recommenders in TEL,from collaborative and content filtering to hybrid approaches andeach of them has some advantages and disadvantages dependingon the context in which they have been used and how they areevaluated (Manouselis et al., 2010). For example, the above men-tioned Matrix Factorization technique which has already provedto be very successful in e-commerce and movie recommendationdomains (Melville & Sindhwani, 2010, chap. 00338), is promisingfor educational domain, as well (Thai-Nghe et al., 2011), but it lacksone important feature – the ability to adapt to teacher’s pedagog-ical strategy.

Neuro-fuzzy techniques have been used in TEL so far, mainly forstudent modeling. For instance, Kinshuk, Nikov, and Patel (2001)have used fuzzy back-propagation to develop a neuro-fuzzy sys-tem aimed at maximizing adaptability in business education tutor-ing. The system learned from the data that came from theinteraction with students. Al-Hammadi and Milne (2004) designeda student classification method using neuro-fuzzy technique calledNEFCLASS, in order to obtain learners’ performance prediction. Sta-thakopoulou, Magoulas, Grigoriadou, and Samarakou (2005) at-tempted to create and update a student model, using a neuralnetwork-based fuzzy model, where the fuzzy component wasaimed at simulating human decision-making process and the neu-ral networks were incorporated to provide the system with learn-ing and generalization abilities. Mir Sadique and Ghatol (2004)used the architecture of the adaptive neuro-fuzzy inference system(ANFIS) in the field of intelligent tutoring systems. Learners weretested in concept understanding, memorizing skills and possiblemisconceptions, and results of these tests are given as input tothe inference system. Finally, learners’ performance is categorizedas poor, fair, good or excellent.

Neuro-fuzzy techniques have also being used in recommendersystems as means for discovering and learning meaningful recom-mendation rules. For example, Castellano, Fanelli, and Torsello(2008) proposed a neuro-fuzzy approach for the extraction of a rec-ommendation model from the usage data encoding user naviga-tional behavior. The model is expressed as a set of fuzzy ruleswhich may be exploited to provide personalized link suggestionsto the users visiting a web site. In particular, a neuro-fuzzy net-work is trained using information about user categories to discovera set of fuzzy rules capturing the associations between user behav-ior models and relevance degrees of pages to be recommended.

To the best of our knowledge, at the moment there are noimplementations of neuro-fuzzy recommenders in TEL, and this re-search tries to fill that gap. Although various data mining and sta-tistical based techniques used by the aforementioned systems doprovide efficient ways for extracting recommendations from data,they do not provide a way for teachers to implement their ownpedagogical strategies. Furthermore, recommendation given bythese systems often cannot be explained. By utilizing neuro-fuzzyapproach, this research aims at creating a more user-centeredadaptive recommender system for TEL.

3. Problem statement

The work presented in this paper addresses three challenges re-lated to pedagogical recommender systems:

� A pedagogical recommender should support any number of cri-teria for recommendation.

Fig. 1. The conceptual model of neuro-fuzzy pedagogical recommender.

Z. Sevarac et al. / Expert Systems with Applications 39 (2012) 9797–9806 9799

� A pedagogical recommender system should be adaptable, inorder to support different pedagogical approaches and differentdomains. This may be the key for a wider adoption of such rec-ommenders. However, to the best of our knowledge, most (ifnot all) implementations of pedagogical recommenders at themoment are very domain- and problem-specific, and they can-not be reused in different environments. This also means thatthe development of every pedagogical recommender startsalmost from scratch.� Last but not the least, a pedagogical recommender system

should be intuitive and easy-to-use for end users (pedagogicalexperts). Once a pedagogical strategy is implemented with therecommender, it should be easy for an end user to extend, mod-ify it to suit the changes in his/her teaching practice.

To this end, the paper proposes Neuro Fuzzy Pedagogical Rec-ommender (NFPR), a general pedagogical recommender systembased on neuro-fuzzy techniques. It can be used to create pedagog-ical recommendation rules for different domains. In addition, NFPRallows for adapting the recommendation rules to include specificlearning and teaching preferences, such as prerequisite knowledge,learning style, and other elements that can be extracted from thestudent and domain model. Accordingly, for students, NFPR canprovide personalized learning experience by providing learningcontent recommendations based on their knowledge, learningstyle and other properties from student model. For teachers, itaims to provide great flexibility in defining pedagogical modelsthat support their own teaching style and experience.

4. Proposed solution

An important feature of NFPR is its flexibility: it can be used oncustom input and output data sets (which correspond to, e.g., thestudent model and the recommended learning content, respec-tively), and allows for the creation of personalized recommenda-tion rules. Fuzzy set theory is used to transform high-levelpedagogical rules into a computational model, whereas a neuralnetwork is used to provide adaptivity to the teacher’s preferences.Thanks to the wizard-style user interface, using the system doesnot require in-depth knowledge of fuzzy sets and neural networks.NFPR is available as an open source implementation1 that can beeasily integrated with almost any TEL system.

Fig. 1 illustrates how NFPR can be used with the student model(comprising the student’s knowledge and learning style) as the in-put, and the recommended learning content as the output. In whatfollows we present the main building blocks of NFPR and their rolein this recommender system.

4.1. Domain model

NFPR’s domain model contains structured knowledge of the do-main in the form of a topic map (Amruth, 2006). Topics are relatedto one another with the prerequisite relation which means that onetopic is a prerequisite for learning another topic. This type of do-main model is chosen since it is intuitive to the end users (i.e.,teachers). Other, more complex techniques for domain modeling(such as ontologies), do offer advantages, but have one significantdisadvantage: they are difficult to accept by the end users and thuspose problems to wider adoption in educational practices (Hatala,Gasevic, Siadaty, Jovanovic, & Torniai, 2009). This model is the basefor creating pedagogical recommendation rules.

1 NFPR is available for download at http://neuroph.sourceforge.net/NFPR.zip.

4.2. Student model

Student model stores information about student’s current stateof knowledge and personal characteristics (Stathacopoulou, Mago-ulas, & Grigoriadou, 1999). The student model used in NFPR is theoverlay student model which represents student’s knowledge as asubset of expert/system’s knowledge of domain (Kass, 1989). Themost important information it contains is a list of topics to learnand corresponding test results for those topics which representthe current state of student’s knowledge.

4.3. Inputs

The inputs for NFPR are extracted from the student model. Theycan be, for example, the student’s knowledge of some topics andthe preferred learning style. The student’s knowledge can be eval-uated with tests, while the learning style can be elicited through anappropriate questionnaire (Kinshuk et al., 2001).

4.4. Outputs

The output of NFPR is the recommended learning content thatcorresponds to some domain concept (i.e., a concept from the do-main model). Possible outputs are identified by relating the avail-able learning content to the appropriate concepts from the domainmodel.

4.5. Recommendation rules

Recommendation rules define the mapping of the inputs to theoutputs of a recommender system and are based on the followingset of high-level pedagogical assumptions:

IF (Student has good knowledge of Topic1) THEN Studentshould learn Topic2

Topic1 and Topic2 are topics (concepts) defined in the Domainmodel and they are related through the prerequisite relationship.Student’s knowledge of these topics is stored in the Student model.If Topic2 is related to some learning content, e.g., LearningCon-tent2, then the above rule gets the following form:

IF (Student has good knowledge of Topic1) THEN Studentshould study LearningContent2

The conditional part of the rule may be more complex and in-clude several conditions, like:

IF (Student has good knowledge of Topic1) AND (Student hasexcellent knowledge of Topic2) THEN Student should learnTopic3

Fig. 2. Fuzzy membership functions for student’s knowledge.

9800 Z. Sevarac et al. / Expert Systems with Applications 39 (2012) 9797–9806

Or, if Topic3 is related to some learning content (e.g.,LearningContent3):

IF (Student has good knowledge of Topic1) AND (Student hasexcellent knowledge of Topic2) THEN Student should studyLearningContent3

If the student’s learning style is also considered, then the ruletakes the following form:

IF (Student has good knowledge of Topic1) AND (Student hasexcellent knowledge of Topic2)AND (Student learning style is SomeLearningStyle1)THEN Student should learn LearningContent4

These are high-level rules, typically used and understood byteachers. In NFPR such rules are converted to a computationalmodel using the fuzzy set theory. Students’ knowledge and learn-ing style are considered to be linguistic variables, which can takevalues of the corresponding fuzzy sets. It is also assumed that eachlearning topic has the corresponding learning content.

In the current implementation, three fuzzy sets are used to ex-press the student’s knowledge of some domain topic:

POOR – insufficient knowledge of the topicGOOD – basic understanding of the topicEXCELLENT – advanced understanding of the topic

The implementation allows for an easy extension to a morefine-grained representation (e.g., five or more fuzzy sets).

In the following example two fuzzy sets are used to express thelearning style:

EXTENSIVE_READER – student who needs to read all availablematerial in detail and understand prerequisites before proceed-ing to the next topic;PRACTITIONER – student who learns by doing practicalexercises.

These learning styles are intuitively defined based on our ownteaching experience. However, a teacher can define students learn-ing styles based on his/her own experience or can opt for an exist-ing, well-founded learning style theory. For example, in ourprevious work (Jovanovic, Gaševic, & Devedzic, 2009), we have suc-cessfully applied Felder–Silverman model of learning styles (Felder& Silverman, 1988) which categorizes a student’s learning style onfour dimensions: sensing-intuitive, visual-verbal, active-reflectiveand sequential-global. Thanks to the great flexibility of the proposedsolution, it can support both formal and intuitively defined learn-ing styles, which is very important for wide adoption of the pro-posed solution in different domains.

Considering all these fuzzy sets, the above recommendationrules can be expressed in the following way:

IF (Knowledge of Topic1 is GOOD) AND (Knowledge of Topic2 isEXCELLENT)THEN Topic3IF (Knowledge of Topic1 is GOOD) AND (Knowledge of Topic2 isEXCELLENT)THEN LearningContent3IF (Knowledge of Topic1 is GOOD)AND (Knowledge of Topic2 is EXCELLENT)AND (LearningStyle is EXTENSIVE_READER)THEN LearningContent5

Even though the first two rules look almost the same, they havedifferent purpose. The first rule recommends a topic to study,while the second rule recommends learning content related tothe recommended topic.

The expressions in the IF part of the given recommendationrules (like Knowledge of Topic1 is GOOD) are evaluated to the valueof the fuzzy membership function, which is then used for fuzzyreasoning and rule calculation. Specifically, the level of the stu-dent’s knowledge of a certain topic (originating, e.g., from test re-sults) is turned into the corresponding fuzzy value using thefuzzy membership functions shown in Fig. 2.

Fuzzy reasoning used for calculating the value of the expres-sions in the IF part of the rule is a simple MIN function, which iscommonly used as the fuzzy logical AND operator (Boukezzoula,Galichet, & Foulloy, 2007).

4.6. Neural network

A neural network is used to customize the mappings betweenthe premise (IF part) of the rule and the corresponding recommen-dation. Each input (neuron in the input layer) corresponds to somedomain topic, and will be fed with a test result for that topic. Eachoutput (neuron in the output layer) also corresponds to some do-main topic, and after the network input is set and the output is cal-culated, the recommended neuron/topic in the output layer will beactivated according to the recommendation rule.

Fig. 3 shows the architecture of the neural network. It has fourlayers of neurons, and each layer plays a specific role. The first twolayers represent the fuzzy interface, which transforms real inputsinto the corresponding fuzzy values, while the third and the fourthlayers form a perceptron network (Rosenblatt, 1962) capable oflearning a set of IF–THEN rules (Sevarac, 2006). More precisely,the roles of the network layers are as follows:

Input layer – just distributes inputs to the next, fuzzyficationlayer. Each input neuron is connected only to the correspondingneurons in the fuzzyfication layer.Fuzzyfication layer – each neuron in this layer corresponds toone fuzzy set, and calculates the value of fuzzy membershipfunction for the given input.Premise layer – each neuron in the premise layer corresponds toone combination of fuzzy sets from the fuzzyfication layer. Itrepresents the Cartesian product (all possible combinations)of fuzzy sets that belong to different inputs.Output layer – each neuron in the output layer corresponds toone recommendation, either a topic or a learning content.

The neural network has two modes of operation: adaptationmode and recommendation mode. In the adaptation mode, thenetwork creates relations between the premise and the output lay-ers based on the data presented to the network (the so-calledtraining set). These data take the form of input–output pairs that

Fig. 3. The architecture of the NFPR neural network.

Z. Sevarac et al. / Expert Systems with Applications 39 (2012) 9797–9806 9801

consist of the input and desired network output for that input. Forexample, if there are four learning topics (Variables_and_Data_Types, For_Loops, Arrays, Classes_and_Objects), there will be four in-puts and four outputs where each one of them corresponds to onelearning topic. These are considered to be the input and desiredoutput vectors. Each element of the input vector takes as its valuethe test result for the corresponding learning topic, while the ele-ments of the output vector have the value 1 if the correspondingtopic is recommended according to some rule, or 0 otherwise. Eachrule is presented with one input/output pair of vectors. For exam-ple, consider the following simple pedagogical rule for a basic pro-gramming domain:

IF (Knowledge of Variables_and_Data_Types is EXCELLENT) AND(Knowledge of For_Loops is GOOD) THEN Arrays

This rule says that the learning topic Arrays will be recom-mended to the student if he/she has a high level of understandingof the topic Variables and Data Types and a basic understanding ofthe topic For Loops.

For this simple rule the excerpt from the corresponding trainingset is:

Input: [80, 50, 0, 0]Output [0, 0, 1, 0]

where the elements and their positions in the vectors corre-spond to the following learning topics, respectively: [Vari-ables_and_Data_Types, For_Loops, Arrays, Classes_and_Objects].Input values come from characteristic points of fuzzy membershipfunctions.

Adaptation is accomplished by using simple single-layer super-vised learning rule, known as Delta Rule (Widrow & Lehr, 1995).The Delta Rule, also known as Widrow–Hoff Rule or Least MeanSquares, is an iterative learning rule for updating the weights ofartificial neurons in a single-layer perceptron. During the learning

process inputs and desired outputs are presented to the network,and adaptation algorithm automatically adjusts the weights sothe output responses to the inputs will be as close as possible totheir desired responses. After adaptation, when in recommenda-tion mode, network will provide desired input–output mappingas defined with training set.

In the recommendation mode, some input is presented to thenetwork, and after the calculation is done, the active neuron inthe output layer is considered to be the recommendation. Contin-uing the previous example, if the network input is [85,57,0,0] thetrained network will output [0,0,1,0]. The third neuron is in acti-vated state (outputs 1) so the recommended topic is Arrays.

4.7. Advantages of the proposed solution

The proposed model of pedagogical recommender system isvery flexible as it allows for various customizations in order to sup-port individual pedagogical strategy. It supports intuitive, non-for-mal pedagogical models that can be created by teachers based ontheir teaching experience, and can also be adapted to the teacher’spreferences. Verbal pedagogical model can be easily translated tothe corresponding fuzzy model by using fuzzy sets and rules. Fur-thermore, a neural network can be automatically generated andtrained thanks to its straightforward architecture and learningrule. This means that different pedagogical recommenders can beautomatically generated without a need to change low level imple-mentation details. Accordingly, it is possible to create very sophis-ticated tools with intuitive and easy-to-use user interface, that canproduce ready to use neuro-fuzzy pedagogical recommenders. Pos-sible customizations include:

1. any number of inputs (theoretically), which means that it cansupport any number of criteria for recommendation (e.g., cus-tomized number of learning topics, learning styles, and evenother criteria constituting learning context);

2. each recommendation criterion can have a customized set ofcorresponding fuzzy sets, so the translation from the verbalpedagogical model to fuzzy computational model can also becustomized (for example, instead of POOR, GOOD and EXCEL-LENT, some may want to have five levels of grading with differ-ent naming);

3. any number of outputs, which means customized number oflearning topics for recommendation;

4. adaptation of recommendation rules to some specific prefer-ences; this is possible due to the fact that high level pedagogicalrules transformed to fuzzy domain are automatically learned bythe neural network.

An additional benefit lies in the fact that regardless of all theabove mentioned customizations, the internal operations of theproposed pedagogical recommender remain the same. This furthermeans that the same implementation can be applied to a widerange of learning domains.

Possible constraints could be faced when working with largenumber of inputs and fuzzy sets, which can cause rule layer togrow fast, so it may require more memory than usual (than stan-dard configurations provide). However, having in mind theamounts of memory that modern systems provide, this can beeasily resolved through appropriate system configuration (e. g.,by assigning more heap memory to Java Virtual Machine).

5. Basic working example

Basic principles of operation of NFPR are described on a simplebut realistic example of a pedagogical recommender for an intro-

9802 Z. Sevarac et al. / Expert Systems with Applications 39 (2012) 9797–9806

ductory programming course. Fig. 4 shows a tiny portion of the do-main model which consists of domain concepts/topics, as well asthe relations between them. Relations describe the minimum levelof knowledge of some topic that is required to learn some other to-pic. These relations are defined by the teacher, based on his/herown teaching experience. This example considers domain topicsonly; for simplicity reasons learning styles are not considered,but could be easily added (as explained in the previous section).Note that different learning styles may require different levels ofknowledge of some topics.

A set of fuzzy pedagogical rules extracted from this model is:

IF (Variables_and_Data_Types is GOOD) THEN For_LoopsIF (Variables_and_Data_Types is GOOD) THEN Classes_and_ObjectsIF (Variables_and_Data_Types is EXCELLENT) AND (For_Loops isGOOD) THEN ArraysIF (Arrays is GOOD) AND (Classes_and_Objects is EXCEL-LENT)AND (Inheritance is GOOD) THEN Collections

This set of rules can be represented with the recommendationmatrix as shown in Table 1. This matrix provides a tabular repre-sentation of the prerequisite relationship among topics in the do-main model and how it determines the recommended topics. Itrelies on the fact that a pedagogical strategy can be modeled as arelation on the Cartesian product of topic sets (Xu, Wang, & Su,2002). Each column in the matrix is related to one possible recom-mendation (i.e., one recommended topic) and contains a fuzzy rep-resentation of the knowledge level for all the prerequisite topics.For example, the second column indicates that in order to be rec-

Fig. 4. A part of the domain model for an

Table 1Recommendation rule matrix.

Recommendation

Variables and data types Classes and objects

Prerequisite knowledgeVariables and data types x GOODClasses and objects x xFor loops x xArrays x xInheritance x xInterfaces x xCollections x x

ommended to learn the topic Classes and Objects, the student musthave good knowledge of Variables and Data Types.

This kind of tabular representation is suitable for visual over-view of all possible combinations of domain topics. Irrelevant com-binations are marked with x, while the empty fields are optionaland are left open, so they could be customized later by the user.If the number of relevant concepts in the domain model grows be-yond a certain limit, the matrix should be divided to display relatedconcepts in several groups.

The NFPR’s neural network for the set of recommendation rulesdefined in Table 1 has:

– 7 input neurons which correspond to seven learning topics(given in Table 1).

– 21 neurons in the fuzzification layer, where each neuron corre-sponds to a knowledge-level fuzzy set (see Fig. 2) related toeach learning topic (since there is three fuzzy sets, 7 � 3 = 21).

– 441 neurons in the premise layer (7 � 3 � 7 � 3 = 441) – forseven topics and three fuzzy sets for each topic.

– 7 output neurons, where each neuron corresponds to a learningtopic that could be recommended to the student.

Training data is automatically generated from the recommen-dation rule matrix (Table 1) and corresponding fuzzy sets (Fig. 2).It is assumed that the level of knowledge lower than the one givenin the matrix does not fulfill the prerequisites, while a higher levelsatisfies the prerequisites. For example, to learn For Loops, a stu-dent must have at least good knowledge of Variables and DataTypes, so if he/she has excellent level of knowledge that will alsosatisfy the prerequisite.

introductory programming course.

For loops Arrays Inheritance Interfaces Collections

GOOD EXCELLENTEXCELLENT EXCELLENT

x GOODx x GOODx x x GOOD GOODx x x xx x x x x

Fig. 5. NFPR wizard Step 1: Creation of recommendation rules.

Z. Sevarac et al. / Expert Systems with Applications 39 (2012) 9797–9806 9803

The adaptation of the rules is accomplished by training the neu-ral network with the generated training data set. Training adjuststhe connections between the premise and output/recommendationlayers, so the network gets the desired behavior. Since there is onlyone layer to adapt (the output layer), the simple DeltaRule algo-rithm is used.

If the learning style is also taken into account in a pedagogicalstrategy, then one recommendation matrix is required for eachlearning style. From the perspective of the used neural network,it will be just another input for the network, and will need corre-sponding neurons in the fuzzification and premise layers.

6. Application

The figures bellow show NFPR’s wizard-style user interface forcreating recommendation rules (Fig. 5) and neural network thatimplements those rules (Fig. 6). Neural network is created withNeuroph, an open source Java framework for neural network devel-opment.2 Neuroph provides simple Java API for using neural net-works within Java applications, and a tool, called easyNeurons,which offers rich and intuitive GUI (Graphical User Interface) for cre-ating and training neural networks. NFPR is created as an applicationsample within easyNeurons tool. Figures bellow show how to createand test NFPR using two step wizard.

Step 1. Define recommendation rules. In this step user (teacher)loads all domain topics, prerequisites and possible recommenda-tions, from QTI files, and system generates recommendation matrix(as the one given in Table 1). QTI (Question and Test Interoperabil-ity specification) defines a standard format for the representationof assessment content and results.3 Generated recommendationmatrix contains all possible combinations of prerequisite relation-ships between domain topics, and the teacher selects recommenda-tion for each combination (Fig. 5), thus creating a recommendationrule. Each row in the recommendation matrix represents one recom-mendation rule. Each field contains the name of a domain topicappearing in the corresponding rule, whereas its color indicatesthe knowledge level of the topic (expressed as a fuzzy set): green

2 Neuroph is available for download at http://neuroph.sourceforge.net/download.html.

3 QTI specification is available online at http://www.imsglobal.org/question/.

for EXCELLENT, yellow for GOOD and red for POOR. Once rules aredefined, user clicks the Next button, and neural network and trainingset are automatically created.

Step 2. Train and test NFPR (Fig. 6). In this step, the user (tea-cher) just has to click the Train button to train the neural networkwith the training set (created at the end of the previous step), andthat is how the neural network learns the rules. When network istrained, the user can load some student’s test results from a QTI filewith test results and see the recommendations.

The trained neural network can be serialized as a Java objectand used as a Java component in any TEL application. It pro-vides a simple API with only two methods for setting the inputand getting the output (i.e., recommendation) from the network.The following sample code (Listing 1) illustrates how easy it isto use the created neural network with an external, e.g., TELapplication:

Listing 1. The use of NFPR’s API for calculating recommendations based on thegiven set of test results.

// get test results read from a QTI file

double[] someTestResults = . . .

// load neural network from file

NeuralNetwork nfpr =

NeuralNetwork.load(‘‘MyNfpr.nnet’’);

// set neural net input

nfpr.setInput (someTestResults);

// calculate network

nfpr.calculate();

// get recommendation

double[] recommendation = nfpr.getOutput();

// do something with the recommendation . . .

If end users (teachers) wish to change the pedagogical strategy,the network needs to be retrained. Existing rules can be modified,new domain topics and learning styles can be added, and even newpedagogical criteria can be introduced. To accomplish this teacherjust needs to re-run the NFPR wizard and make the desiredchanges in Step 1 (Fig. 5).

Fig. 6. NFPR wizard step 2: Training and testing the neural network.

9804 Z. Sevarac et al. / Expert Systems with Applications 39 (2012) 9797–9806

7. Evaluation

Despite the increasing number of systems proposed for recom-mending learning resources, a closer look at the current status oftheir development and evaluation reveals a lack of systematic eval-uation studies in the context of real-life applications. As indicated in(Manouselis et al., 2010), more than a half of the analyzed systems,namely 12 out of 20 the authors considered, were still in the designor prototyping stage of development, while only 10 systems were re-ported as being evaluated through trials that involved human users.Another observation is that, very often, experimental investigationof the recommendation algorithms does not take place, although itis a common evaluation practice in recommender systems exam-ined for other domains (e.g., Breese, Heckerman, & Kadie, 1998).

Deshpande and Karypis (2004), Papagelis, Plexousakis, andKutsuras (2005), and Herlocker, Konstan, Terveen, and Riedl(2004), indicate that careful testing and parameterisation has tobe carried out before a recommender system is finally deployedin a real-world setting. One of the main reasons is that the perfor-mance of recommendation algorithms seems to depend on the par-ticularities of the application context. Hence it is advised toexperimentally analyze recommender system before its actualdeployment.

Following this advice, NFPR was evaluated at the University ofBelgrade with a group of 24 teachers. The group has been intro-duced to the idea of pedagogical rules based on test results as wellas the steps needed to create pedagogical rules and save them asexpert knowledge. Then the group was introduced to the NFPR tooland its features, and asked to create a set of pedagogical rules thatreflects their pedagogical strategy. Participants were then asked toassess the recommendations given by NFPR, by using the previ-ously prepared set of test results. These test results which corre-spond to typical student’s knowledge levels, were created by theteachers based on their teaching experience.

Having finished the given tasks, the study participants wereasked to fill in a survey which included the following questions re-lated to their interaction with NFPR:

Q1: I learned to use the software quicklyQ2: I was able to use it without detailed instructionsQ3: Working with this software is satisfyingQ4: Sometimes I didn’t know what to do next with thissoftware.Q5: I need to learn a lot of things before I can get going with thissystemQ6: Student’s knowledge estimation was satisfactoryQ7: Recommendations given by the system are satisfactoryQ8: I was able to create a pedagogical strategy very close tothose I use in my teaching practiceQ9: Students would benefit from the given recommendationsQ10: I would recommend this software to my colleaguesQ11: Your backgroundQ12: Familiarity with information systems and programmingQ13: Additional comments about NFPR wizard

The questionnaire included three sections. The first group ofquestions (1–5) was related to the users’ experience with the NFPRwizard. These questions were focused on usability of the NFPR wiz-ard, and they were based on software usability surveys given in(Arnold, Measuring usability with the USE questionnaire, & Octo-ber, 2001). The second group of questions (6–10) was related tohow well NFPR was estimating the students’ knowledge and thequality of its recommendations. The third group of questions wereopen-ended questions to gather additional info about participantsand their personal experience in using NFPR. The answers to thefirst two groups of questions in the survey were collected using afive-point Likert scale (Strongly disagree, Disagree, Neutral, Agree,Strongly agree). To make the survey easily accessible to the studyparticipants, we used surveymonkey.com tool to make web-basedsurvey.

Survey results are summarized in Table 2.When analyzing the collected data, we also took into account

the teachers’ background and familiarity with information systemsand programming, which were collected from open-endedquestions.

Table 2Survey results.

Questions Answers

Strongly disagree Disagree Neutral Agree Strongly agree Mean Standard deviation

1 3 2 0 12 7 3.75 1.332 4 7 0 8 5 3.13 1.483 2 3 1 12 6 3.71 1.234 13 6 0 3 2 1.96 1.375 7 10 2 1 4 2.38 1.416 2 1 2 11 8 3.92 1.187 1 3 1 10 9 3.96 1.168 1 2 2 13 6 3.88 1.039 1 2 2 11 8 3.96 1.08

10 1 2 2 12 7 3.92 1.06

Z. Sevarac et al. / Expert Systems with Applications 39 (2012) 9797–9806 9805

Some of the teachers obviously found the user interface to bevery intuitive, since they were able to understand how to createpedagogical rules without detailed instructions (Q2; 3.13 out of5, SD 1.48). Most of them find working with software satisfying(Q3; 3.71 out of 5, SD 1.23), and just few of them got confusedwhile working with it (Q4; 1.96 out of 5, SD 1.41). Important re-sults are that participants found students’ knowledge estimationsatisfactory (Q6; 3.92 out of 5, SD 1.18) and same is with the givenrecommendations (Q7; 3.96 out of 5, SD 1.16; and Q9, 3.96 out of 5,SD 1.08). Also, the facts that participants where able to successfullycreate their own pedagogical strategy (Q8; 3.88 out of 5, SD 1.03)and that they would recommend this tool to their colleagues(Q10; 3.92 out of 5, SD 1.06) show that this kind of tool wouldbe accepted by the intended audience (teachers). However, someof the participants pointed in their comments that they had prob-lems with understanding colors representing fuzzy rules. Overallthe possibility of creating an automatic list or a map of educationalcontent for every student, as if he/she had a personal, ‘‘one on one’’interaction with the teacher, was most welcomed from users ofNFPR in their comments.

The review of the data gathered from the survey has alsoshowed that:

� the participants with little experience with information systemsand programming exhibited a better understanding of the con-cept of pedagogical rules after a short presentation on how touse NFPR;� the wizard used for generating pedagogical rules was easy to

use for 55% (Q2) before and 80% (Q1) after help presentationgiven during introduction;� a general opinion about NFPR’s capability to imitate expert

decisions has been shown through the survey feedback – 79%of the teachers had a positive opinion about NFPR’s recommen-dation of educational content based on test results (Q9); inthese cases, NFPR was giving recommendations as teacherswould, based on personally created pedagogical rules; the ideaof saving these pedagogical rules on a computer for later use insimilar or identical tests was suggested in one way or anotherby many participants in their comments (Q13);� overall these results show that proposed solution could success-

fully address adaptivity and usability issues outlined in Section 3.

8. Conclusion

The Neuro Fuzzy Pedagogical Recommender (NFPR) is an open,adaptive software component for creating recommenders in learn-ing environments. It makes use of the fuzzy set theory which al-lows for easy translation of high level verbal pedagogical rulesinto fuzzy computational model. In addition, it offers an automatedprocedure for the creation and configuration of neural networkbased recommender.

For students, NFPR can provide personalized learning experienceby generating learning content recommendations based on theirknowledge, learning style and other properties that might be rele-vant for some specific subject domain. For teachers, it providesgreat flexibility in defining pedagogical models that support theirown teaching style and experience. Preliminary evaluation showedthat the system is easy to use for teachers, and the teachers aresatisfied with the recommendations it can provide. Once created,the pedagogical model can be adjusted or extended without needto change implementation details. Likewise, low level neuro fuzzyprinciples are domain independent, which means that same com-ponent could be used in different domains. All that is required isto run the customization wizard which will configure the neuro-fuzzy model according to the chosen domain and teaching prefer-ences. In a nutshell, the main advantages of NFPR are flexibility,domain independence and easy implementation/application indifferent environments. Future research will be focused on develop-ment of ready to use pedagogical recommenders for different learningdomains and integration in existing learning environments.

References

Al-Hammadi, A. S., Milne, R. H., 2004. A neuro-fuzzy classification approach to theassessment of student performance. In Proceedings of the IEEE internationalconference on fuzzy systems (pp. 837–841).

Amruth, N. K. (2006). Using enhanced concept map for student modeling in amodel-based programming tutor. In Proceedings of 19th international FLAIRSconference on artificial intelligence (FLAIRS 2006) special track on intelligenttutoring systems (pp. 527–532). Melbourne Beach, Florida, USA.

Arnold, M. Lund, Measuring usability with the USE questionnaire, STC usability SIGnewsletter, October 2001 issue (Vol. 8, No. 2). <http://www.stcsig.org/usability/newsletter/0110_measuring_with_use.html>.

Basu, C., Hirsh, H., Cohen, W., & Nevill-Manning, C. (2001). Technical paperrecommendations: A study in combining multiple information sources. Journalof Artificial Intelligence Research, 1, 231–252.

Bell, R., Koren, Y., & Volinsky, C. (2009). Matrix factorization techniques forrecommender systems. IEEE Computer, 42(8), 30–37.

Bobadilla, J., Hernando, A., Ortega, F., & Bernal, J. (2011). A framework forcollaborative filtering recommender systems. Expert Systems with Applications,38(12), 14609–14623.

Boukezzoula, R., Galichet, S., & Foulloy, L. (2007). MIN and MAX operators for fuzzyintervals and their potential use in aggregation operators. IEEE Transactions onFuzzy Systems, 15(6), 1135–1144. ISSN: 1063-6706.

Breese, J., S., Heckerman, D., & Kadie, C., 1998. Empirical analysis of predictivealgorithms for collaborative filtering. In: Proceedings of the 14th Conference onUncertainty in Artificial Intelligence (Vol. 461, No. 8, pp. 43-52). San Francisco,CA. ISBN: 155860555X.

Castellano, G., Fanelli, A. M., Torsello, M. A., 2008. NEWER: A neuro fuzzy webrecommendation system. In Applications of first international conference on thedigital information and web technologies, 2008. ICADIWT 2008 (pp. 162–167).doi:10.1109/ICADIWT.2008.4664338. ISBN 978-1-4244-2623-2.

Deshpande, M., & Karypis, G. (2004). Selective Markov models for predicting Webpage accesses. Transactions on Internet Technology, 4(2), 163–184 [Associationfor Computing Machinery].

Drachsler, H., Hummel, H. G. K., & Koper, R. (2009). Identifying the goal, user modeland conditions of recommender systems for formal and informal learning.Journal of Digital Information, 10(2) [Texas Digital Library. ISSN: 1368-7506].

Felder, R., & Silverman, L. (1988). Learning and teaching styles in engineeringeducation. Journal of Engineering Education, 78(7), 674–681.

Gantner, Z., Drumond, L., Freudenthaler, C., Rendle, S., Schmidt-Thieme, L. (2010).Learning attribute-to-feature mappings for cold-start recommendations. In

9806 Z. Sevarac et al. / Expert Systems with Applications 39 (2012) 9797–9806

Proceedings of the 10th IEEE international conference on data mining (ICDM 2010).IEEE Computer Society.

Ghauth, K., & Abdullah, N. (2010). Learning materials recommendation using goodlearners’ ratings and content-based filtering. Educational Technology Researchand Development, 1042–1629.

Hatala, M., Gasevic, D., Siadaty, M., Jovanovic, J., Torniai, C. (2009). Can educatorsdevelop ontologies using ontology extraction tools: An end user study. InProceedings of the 4th European conference on technology enhanced learning(ECTEL09), LNCS 5794 (pp. 140–153). Springer-Verlag.

Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004). Evaluatingcollaborative filtering recommender systems. ACM Transactions on InformationSystems, 22(1), 5–53 [Association for Computing Machinery].

Hummel, H. G. K., van Den Berg, B., Berlanga, A. J., Drachsler, H., Janssen, J., Nadolski,R., et al. (2007). Combining social-based and information-based approaches forpersonalized recommendations on sequencing learning activities. InternationalJournal of Learning Technology [Interscience].

Jameson, A., Konstan, J., Riedl, J. (2002). AI techniques for personalizedrecommendation. Tutorial notes, AAAI-02. Edmonton, Canada.

Jovanovic, J., Gaševic, D., & Devedzic, V. (2009). TANGRAM for personalized learningusing semantic web technologies. Journal of Emerging Technologies in WebIntelligence, 1(1), 6–21. ISSN: 1798-0461.

Kass, R. (1989). Student modelling in intelligent tutoring systems-implications foruser modelling. In A. Kosba & W. Wahlster (Eds.), User models in dialog systems(pp. 386–410). Berlin: Springer-Verlag.

Kinshuk, A., Nikov, A., Patel (2001). Adaptive tutoring in business education usingfuzzy backpropagation approach. In Proceedings of the ninth internationalconference on human–computer interaction 2001 (Vol. 1, pp. 465–468). <http://www2.sta.uwi.edu/~anikov/publ/adaptive-tutoring-HCII2001.pdf>.

Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., & Koper, R. (2010).Recommender systems in technology enhanced learning. In P. B. Kantor, F. Ricci,L. Rokach, & B. Shapira (Eds.), 1st Recommender systems handbook (pp. 1–29).Berlin: Springer.

Melville, P., Sindhwani, V. (2010). Encyclopedia of machine learning (1st ed.).Springer. ISBN 978-0-387-30164-8.

Mir Sadique, A., Ghatol, A. A. (2004). A neuro fuzzy inference system for studentmodeling in web-based intelligent tutoring systems. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.9687&rep=rep1&type=pdf>.

Papagelis, M., Plexousakis, D., Kutsuras, T. (2005). Alleviating the sparsity problemof collaborative filtering using trust inferences. In Proceedings of the 3rd

international conference on trust management. LNCS 3477 (pp. 224–239). Berlin,Heidelberg, New York: Springer.

Preisach, C., Marinho, L. B., Schmidt-Thieme, L. (2010). Semi-supervised tagrecommendation-using untagged resources to mitigate cold-start problems.In 14th Pacific-Asia conference on knowledge discovery and data mining (PAKDD2010), Lecture Notes in Computer Science (Vol. 6118, pp. 348–357). Springer.

Rosenblatt, F. (1962). Principles of neurodynamics: Perceptrons and the theory of brainmechanisms. Washington DC: Spartan Books [ASIN: B0006AXUII].

Santos, O. C., & Boticario, J. G. (2010). Modeling recommendations for theeducational domain. Procedia Computer Science, 1(2), 2793–2800 [Proceedingsof the 1st Workshop on Recommender Systems for Technology EnhancedLearninRecSysTEL 2010].

Sevarac, Z. (2006). Neuro fuzzy reasoner for student modelling. In Proceedings of thesixth IEEE international conference on advanced learning technologies 2006 (pp.740–744). <http://www.computer.org/portal/web/csdl/doi/10.1109/ICALT.2006.245>.

Stathacopoulou, R., Magoulas, G. D., Grigoriadou, M. (1999). In Neural network-basedfuzzy modeling of the student in intelligent tutoring systems, international jointconference on neural networks (pp. 3517–3521). Washington DC, USA.doi:10.1109/IJCNN.1999.836233. ISBN 0-7803-5529-6.

Stathakopoulou, R., Magoulas, G., Grigoriadou, M., & Samarakou, M. (2005). Neuro-fuzzy knowledge processing in intelligent learning environments for improvedstudent diagnosis. Information Sciences, 170, 273–307.

Tang, T. Y., Mccalla, G. (2003). In Smart recommendation for an evolving e-learningsystem workshop on technologies for electronic documents for supporting learning,international conference on artificial intelligence in education (AIED).

Tang, T., McCalla, G. (2004). Beyond learners interest: Personalized paperrecommendation based on their pedagogical features for an e-learningsystem. In Pacific rim international conference on artificial intelligence – PRICAI2004 (pp. 301–310).

Thai-Nghe, N., Drumond, L., Horváth, T., Krohn-Grimberghe, A., Nanopoulos, A.,Schmidt-Thieme, L. (2011). In O. C. Santos, J. G. Boticario, (Eds.), Factorizationtechniques for predicting student performance. Educational recommender systemsand technologies: Practices and challenges (ERSAT 2011). IGI Global. doi:10.4018/978-1-61350-489-5. ISBN13: 9781613504895.

Widrow, B., & Lehr, M. A. (1995). Perceptrons, adalines, and backpropagation. In M.A. Arbib (Ed.), Handbook of brain theory and neural networks (pp. 719–724). MITPress. ISBN 978-0-262-01148-8.

Xu, D., Wang, H., Su, K. (2002). Intelligent student profiling with fuzzy models. InProceedings of the 35th Hawaii international conference on system sciences.