TAŞCI, T., PARLAK, Z., KİBAR, A., TAŞBAŞI, N., & CEBECİ, H. İ. (2014). A Novel Agent-Supported Academic Online Examination System. Educational Technology & Society, 17 (1), 154–168.

A Novel Agent-Supported Academic Online Examination System

Tuğrul TAŞCI1*, Zekeriya PARLAK2, Alpaslan KİBAR3, Nevzat TAŞBAŞI1 and H. İbrahim CEBECİ1

1Faculty of Informatics of Computer Engineering Department, Sakarya University, Turkey // 2Faculty of Engineering of Mechanical Engineering Department, Sakarya University, Turkey // 3Distance Education Center, Sakarya

University, Turkey // ttasci@sakarya.edu.tr // zparlak@sakarya.edu.tr // kibar@sakarya.edu.tr // ntasbasi@sakarya.edu.tr // hcebeci@sakarya.edu.tr

*Corresponding author

(Submitted August 9, 2012; Revised December 8, 2012; Accepted March 13, 2013) ABSTRACT

Proper execution of exams aimed at assessment & evaluation is of critical importance in Learning Management Systems (LMS). Problems arising from human-centered errors or technical difficulties may lead to questioning of exams, and thus of reliability and efficiency of the distance education systems. Online examination system architecture proposed in this paper provides for integrated management of main functions such as question pool creation and update, exam authoring, execution and evaluation, and management of the feedbacks from students, along with ensuring use of analysis reports related to the questions and exams created by an intelligent agent in the decision-making processes. After conducting analyses on the distance education (DE) system of Sakarya University, it was found that the proposed intelligent agent supported online exam system detects the problems that arise to a large extent and enables the instructors to decide more easily in a shorter time. This system, which is possible to expand with its flexible structure to include additional intelligent features with the aim of resolving different problems, can also be adapted to different institutions that use online examination systems.

Keywords

Architectures for educational technology system, Distance education and telelearning, Evaluation methodologies, Intelligent tutoring systems

Introduction Following the rapid developments in the information technologies, online education gains importance as an alternative to the traditional teaching models. In online education, teaching independent of time and space constraints brings a fast increase in the numbers of students. For evaluation of such high numbers of students, online test methods are more used than classical methods (Zhang et al., 2006). Structural problems of classical evaluation such as requiring too much time (Gawali & Meshram, 2009) and not allowing sufficient means for student and instructor feedbacks (Guo & Mao, 2010; Zhang et al., 2006) can be eliminated owing to the functions of online test system such as efficient management of evaluation process, feedback system and diversity and pace of student evaluation (Tallent-Runnels et al., 2006). Studies on online examination systems constitute a major part of the studies on online education in the recent years (Tallent-Runnels et al., 2006). In many of these studies, theoretical models relating to the exam systems are provided along with application approaches. In their study, Mustakerov & Borissova (2011) predicted that the two-layered examination system they offered ensured significant benefits owing to the flexible adjustment approach during the testing process. Authors also asserted that the system they prepared could be used for formal and informal evaluation processes. And in other theoretical studies in the literature, examination systems are designed with 3 or more layers with the use of broader architectures. Although featured with differing names in the literature, the examination system is composed of preliminary preparation layer that covers administrator appointments and student enrollments (Brusilovsky & Miller, 1999; Gawali & Meshram, 2009; Keleş et al., 2009; Aye & Thwin, 2008; Guo et al., 2008; Zhang et al., 2006), exam preparation layer consisting of questions, question pools and exams (Brusilovsky & Miller, 1999; Gawali & Meshram, 2009; Keleş et al., 2009; Jun, 2009; Aye & Thwin, 2008; Guo et al., 2008; Li & Wu, 2007; Zhang et al., 2006) and evaluation layer where the results of the exams are calculated (Brusilovsky & Miller, 1999; Gawali & Meshram, 2009; Keleş et al., 2009; Jun, 2009; Li & Wu, 2007; Zhang et al., 2006). Aimin & Jipeng (2009) designed the three layers as administrator, student and instructor, whereas Jin & Ma (2008) created the exam architecture by the help of student and instructor layers alone.

154 ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.

Appropriate management of feedbacks received from students and reporting and analysis processes performed on the exam results are important for proper execution of the examination systems. In his study on collaborative learning process, Wilson (2004) investigated the extent the feedback process affects the student performance by comparing with the students who receive classical face-to-face education. The author emphasized that student feedbacks may render evaluation process more efficient. Crisp & Ward (2008), on the other hand, put that getting feedbacks in the systems with too many students can only be achieved by help of online systems. In their study, Shen et al. (2001) emphasized the importance of adjustment and update processes based on student feedbacks for proper execution of online exams. Also, the authors designed an intelligent monitoring system with automatic reporting and filtering capability. In another study, where importance of the monitoring systems is emphasized (Tinoco et al., 1997), a module, in which the exam results are evaluated, reported and comments derived from these reports by help of analyses, is added to the examination system architecture. Analyses on exam result are also included. Hang (2011) expressed that an online examination system architecture must include re-examination, automatic scoring, question pool update, flexible question design and system security topics. Architectural flexibility and strong information technology infrastructure of online examinations systems make it likely for them to possess processes suitable for intelligent approaches. In this context, there is high number of studies in the literature based on intelligent approaches or supported by them. Keleş et al. (2009) added two intelligent agents named Advisor and Planner to their learning platform called ZOSMAT, which they designed to support students online and face-to-face. Advisor agent makes a proposal by mainly analysing the test results and reports and by taking the education science facts into consideration, while the Planner agent matches the information received from the Advisor agent with the system resources. In the decision making process, Advisor agent employs a rule-based expert system approach. In the agent-based study by Jin & Ma (2008), formation of the exam and evaluation of the results are carried out by two different agents. Alexakos et al. (2006) likewise executed evaluation process by the help of intelligent agents and used genetic algorithms along with Bayesian Networks for these processes. Unlike other studies, Gawali & Meshram (2009) designed the examination system in its entirety, but no in part, on the basis of three agents, which are Stationary, Mobile and Main. Aimin & Jipeng (2009) on the other hand incorporated intelligent approaches to the exam system architecture through linear algorithms that make random choices in the question selection and exam formation processes, without making use of agent technology. Proper evaluations of students affect their learning performances (Wang et al., 2008). It can be said that elimination of human and/or system centered errors in the examination systems and good execution indirectly affect student performance and satisfaction. In this regard, it is important to design an auxiliary system or an intermediary layer that will identify the possible difficulties in the main examination processes via reports and student feedbacks and generate solution proposals. As a result of effective use of information technologies during this process, it can be said that intelligent agents can be used for reporting and filtering works in the intermediate layer (Shen et al., 2001). Also, it appears as a necessity that the system gives the student another exam entry right in case of potential problems, update of erroneous or deficient questions in the question pool, ability to update the scores in case of a correction and such components as flexible question design (Hang, 2011). When all these literature findings are considered, there is an obvious need for design and implementation of an integrated and flexible examination system architecture possessing an intelligent intermediate layer, which is capable of producing solutions for differing problems. In this study, an integrated online exam system architecture that includes Administration, Implementation and Finalization main layers and a Support Layer (SL) is proposed. Owing to this architecture, integrated management of concurrent changes triggered in the processes on all related layers, depending on the decisions made in the SL (like update of question text and difficulty level; giving the student another exam entry right; and update of scores and reports) are ensured. Also, a Monitoring Agent (MA) is designed to help students through creating reports by analyzing the online exam results in a way that enables students to review all possible problems and make decisions more easily, in order to reduce the problems that arise in the implementation process of the online examination system to a negligible level. Primary contribution of this study is the proposed online examination system having an integrated and flexible structure. The system manages a number of functions such as collection of feedbacks from students, generating

155

reports and submitting them to the managers and providing proposals in the decision-making processes by help of analysis within a single module as distinct from the other studies found in the literature. Agent-supported academic online examination system architecture Agent-Supported Academic Online Examination System (ASOES), the subject matter of this study, is designed as a basic component of the Learning Management System called AkademikLMS (SAUPORT) that aims to execute online exams of the distance education programs of Sakarya University. While developing ASOES, the goal was to alleviate the potential problems that could be encountered in the online exams and make the exam management process easier. ASOES, which is developed to ensure effective management of assessment and evaluation processes of academic institutions that have many distance education programs and to offer ease of use to the students and the instructors, has an architecture comprising three main layers and one support layer as shown in Figure 1. Operating process of ASOES starts with an integration application that automatically replicates course, user and user-course enrollment information from databases into an external system to the ASOES database at certain intervals. The Administration Layer (AL) covers such operations as creation of distance education program, courses and users of this program, as well as definition of course-user associations. Question and exam question pools are created and the exams are published for registered students in the Implementation Layer (IL). Finalization Layer (FL) covers calculation and update of exam results and creation of reports relating to these. Support Layer (SL) possesses a Help Desk (HD) operated on the basis of problem feedbacks and an exam monitoring module that analyzes the system reports and forwards the results to the Help Desk.

Figure 1. ASOES architecture

Administration layer

Figure 2. Administration layer structure

156

Within the AL, using the information automatically replicated from the external system to the ASOES database at certain periods, distance education programs are created in the AL, and course and users are registered in these programs. Then, data relating to student-course and instructor-course associations are used for matching. Associations of students with the courses they take are defined by Enrollment, and associations of instructors with the courses they give are defined by Management records (See Figure 2). Each student has to be enrolled for the course related to the exam so that they can take the exam and apply to the HD for problems; and this registration information is sent to Publishing process in the IL and Help Desk process in the SL through the Enrollment record. Likewise, in order for each instructor to create an exam and manage the HD, information of the course relating to the exam is sent to the Authoring process in the IL and Help Desk process in the SL through the Management record. Implementation layer

Figure 3. Implementation layer structure

As seen in Figure 3, IL is composed of Authoring process where the question and exam question pools are created and their features are defined and the Publishing process where the exams are published. Users with the Instructor authority coming through the Management record from the AL, add the questions to the question pools of their courses by entering them in the Question Pool Creation sub-process together with their features. Features such as question type, difficulty level, keyword and shuffle for each question are entered by the instructor. Exam Formation sub-process is another part of the Authoring process. In this process, basic exam features come optionally from the AL for each distance education program. If such information does not come, they are created by the instructor with Custom Features in this process. Later, exam question pool is created and recorded in the Exam Features and Exam Question Pool Database together with the exam features.

157

In the authoring process, if there is a decision taken to change question text, answer choice or question category as a result of the problems notified by the MA or came to the HD based on student problem feedbacks after performing the exam, these will again be updated by the instructors so that problems in the coming exams are prevented. In the Publishing process, exam question pool and the defined exam features are combined by the Enrollment record from the AL and the exam is published for each student who start a session. Support layer In an online education system, there is possibility that tens of thousands of online exams are performed. Also, it is always possible to face human-centered or technical problems in online exams. When these are taken into consideration, auxiliary tools are needed to overcome the difficulty of administration of the exams and to identify and resolve the problems that arise. SL, which is designed for this purpose, is used for smooth execution of the exams (See Figure 4). SL is composed of MA process creating reports and HD process where these reports and student feedbacks are evaluated.

Figure 4. Support layer structure

Monitoring agent It is seen from literature that e-learning platforms need having a kind of item analysis module in order to execute large numbers of online exam sessions smoothly. Post & Hargis (2012) indicated that providing item analyses and statistics are essential for any examination system. In another study it is emphasized that the information provided by the item analysis assists not only in evaluating performance but in improving item quality as well (Fotaris et al, 2010). Online examination modules of the popular LMS platforms (Blackboard, Canvas and Moodle) are evaluated within this context. Blackboard LMS has built-in item analysis module which produces question and exam reports with discrimination, difficulty, average score, standard deviation and standard error statistics. Moodle has an add-in type item analysis module which is capable of presenting reports similar to Blackboard with additional statistics including skewness and kurtosis of the grade distribution, coefficient of internal consistency and discriminative efficiency. Canvas, has no item analysis module in spite of incoming requests in this direction.

158

The proposed system with monitoring agent, on the other hand, does not only provide reports and statistics but also generate suggestions for the instructors. By virtue of these suggestions, an instructor gains opportunity to give additional exam-entry right to a particular student, and to change the difficulty level or to update body and options of a specific question. Compared to the other platforms, one of the most significant functions of ASOES is that it creates only the list of problematic questions and exams for the related instructor rather than screening standard reports for every question and exam. Instructors may perform updates in the direction of system suggestions or may insist on their previous decisions by evaluating the related questions and reviewing reports of related students. All of these functions are presented to the instructors within a dashboard-type interface so that they may operate with ease. MA processes the reports it receives from the Reporting process within FL, subject to the rules created by expert opinion, and conducts Category, Exam Response and False Exam Analyses. ˙ Category Analysis In category analysis, information regarding difficulty level of the questions coming from the IL and the item difficulty index calculated by the system is compared. MA calculates the item difficulty index ( ) as follows (Bachman, 2004):

(Eq. 1) Here and gives the number of students that answer the question correctly in upper and lower groups, while value shows the total number of students in the groups (Bachman, 2004). Kelley (1939) put forth that upper and lower groups should be selected equally as first 27 % and last 27 % according to success ranking. After the value is calculated, the difficulty level interval of this value is determined. In the literature it is emphasized that determination of the difficulty level intervals should be undertaken by the instructor who prepares the exam by taking the purpose of the exam into consideration (Atılgan et al., 2006; Kubiszyn & Borich, 1990). Difficulty level intervals used by MA are provided in Table 1.

Table 1. Difficulty Level Intervals Difficulty Level Difficulty Level Interval Easy 0.65 – 1.00 Moderate 0.35 – 0.65 Hard 0.00 – 0.35 Report of proposal to change the difficulty level prepared by help of Rule 1 is sent to the HD process. Rule 1: “If the difficulty level interval that corresponds to the p value calculated by the system is different from the difficulty level determined by the instructor, create a report of proposal to change the difficulty level.” ˙ Exam Response Analysis In the exam response analysis, distribution of the incorrect answer choices are investigated; and for all false choices, ratio of number of checking (CF) for the false choice to the number of checking (CT) for the true choice is calculated as in Equation 2:

(Eq. 2)

If the largest of these ratios are larger than a threshold value (T) then Rule 2 is applied. A report is created according to this rule to review the correct answer, which is then sent to the HD. Rule 2:“If , create a false answer report.”

159

˙ False Exam Analysis In false exam analysis, for all situations, except when students end their session voluntarily or the system terminates it at the end of the duration of the exam, the error codes in Table 2 are created with the parameters they are linked to.

Table 2. Error Codes Error Code Description Parameter E1 Server based error P1 E2 Client-server communication error P2, P3, P4

Table 3. Error Parameters

Parameter Description P1 Presence of active exam sessions during time when the server does not reply P2 Abandoning the session deliberately by student P3 Ratio of number of students experiencing problems to all student entered for the same exam P4 The completion status of the student regarding his/her other exam sessions Rule, that includes the parameters in Table 3 and the error codes, is inferred as follows: Rule 3:“IF P1=1 THEN S{NEER}=Yes ELSE IF P2=1 THEN S{NEER}=No ELSE IF P3>T1 THEN S{NEER}=Yes ELSE IF P4> T2 THEN S{NEER}=Yes ELSE S{NEER}=No” *S{NEER}: new exam entry right proposal Here T1 and T2 are threshold values determined by P3 and P4 parameters. These values are calculated before the exam based on the frequency tables belonging to past exams. Reports prepared as per the inference in Rule 3 are sent to the HD process. HD mainly includes decision-making processes relating to the problems encountered during exams by receiving information from problem feedbacks notified by the students and the analysis reports prepared by the MA. This decision-making process is undertaken by the instructors in the role of exam administrators. The instructor may receive special reports from the Reporting process if he/she deems it necessary, in order to receive help to decide. Students who could not complete the exam session or who claim there are errors in the questions choose one of the categories in Table 4. A problem ticket specific to the student is created together with this category information, which is then sent to the relevant decision-making process within HD.

Table 4. Problem notification category and the relevant decision-making process Problem notification category Category ID Relevant decision-making process False Question FQ False Question Validation False Exam FE New Exam Entry Right Problems labeled with the FQ category code in the problem database are sent to the False Question Validation decision-making process. Instructor reviews the related question and decides whether it is false or not. If the question is not false, the ticket is closed by sending an information message to the student. If the question is false, then notification is sent to the Evaluation process in the FL for re-evaluation and to the Authoring process in the IL so that the problem in the question is fixed. Help desk Problems recorded with the FE code in the problem database is sent to the New Exam Entry Right decision-making process. Also, proposal reports specific to the exam session of the student created at the end of False Exam Analysis in the MA are matched with the problem notifications during the New Exam Entry Right decision-making process.

160

Problems that cannot be detected by the MA and therefore cannot be matched by the problems in the problem database are sent to the New Exam Entry Right decision-making process as a separate report. In the decision-making process, the instructor may give a new exam entry right to the student or close the relevant without giving such right after evaluating the problem. In both of these cases, a notification mail is sent to the student. Reports automatically generated by the MA as a result of Exam Response and Category analyses are sent to the Analysis Based Question and Category Validation decision-making process. The instructor considers the data from the Exam Response analysis and decides whether the pre-specified answer choices of the questions are false or not; and decides whether the difficulty levels of the questions are defined correctly or not using the data from the Category analysis. He/she sends the update request regarding the correct choice and/or difficulty level of the question to the Authoring process. Finalization layer

Figure 5. Finalization layer structure

As shown in Figure 5, FL consists of Evaluation process where exam result information is processed and turned into scores and Reporting process where all information relating to the exam is processed and turned into reports. In the Evaluation process, answers of the student who has completed the exam are matched with the answer key and a score is calculated for that student. In the Reporting process, all reports needed for the analyses to be made by MA are automatically generated after the exam. Also, all kinds of reports regarding the exam can be created within this process upon request of the instructor. After the reports are evaluated within SL, if re-evaluation decision is made, scores of the student is updated in the Evaluation process. ASOES implementation Distance education at Sakarya University was launched in 2001 with IBM Lotus LearningSpace. Online examinations of 400 students were hardly conducted because of the huge technical problems notified via e-mail, phone or discussion forums. It was obtained from exam reports that only 65% of exam sessions were started and 84% of those were completed without any error. Beginning from the year 2003, the number of students was raised to 2500. Therefore, it became a necessity to find an alternative solution due to the unmanageable structure of LearningSpace and expanding institutional requirements. Thus, a new LMS software called SAULMS was developed in 2005 with useful functions such as calendar-based

161

online exam execution and exam-entry-right management. It was measured that 73% of exam sessions was started during the period (2005-2010) and 94% of those have been finalized by success. Actually, it was clearly identified by the year 2008 that a high-end LMS software was needed in conjunction with the increase of conducted e-learning programs as well as the number of students. Throughout the searching period, worldwide LMS platforms including IBM LMS, Blackboard, Moodle and Sakai was evaluated. Because those e-learning systems under evaluation have either no customization support or a quite complicated configuration and maintenance process, anew institutional-requirement-specific software (SAUPORT) development attempt was started. SAUPORT has been employed since 2010 for serving more than 20000 students so far in 23 e-learning programs. Number of online examination sessions today is over 75000 for each semester and execution of those has been performed more efficient than ever before by means of the latest added functions including improved examination calendar, multimedia supported question and options, question bank, elaborative user or course based reporter and advanced exam-entry right manager. Based on the latest reports, it was extracted that the average shares of entrance and errorless completion of exam sessions between 2010 and 2012 were 82% and 99% respectively. As seen in Table 5, 76156 exam sessions were defined during 2011-2012 fall semester, students started 62280 of these sessions and 61285 of these sessions were completed successfully. In this case, 995 exams were labeled as false exam session.

Table 5. Numbers of exam sessions

Degree Total number of exam sessions

Number of started exam sessions

Number of completed exam sessions

Associate 45318 33095 32433 Undergraduate 1538 1129 1111 Master 4829 4076 4026 Undergraduate completion 24471 23980 23715

Figure 6 shows number of failed exam sessions and the number of problems written by students to the HD. As can be understood from the graph, there are differences in the number of incomplete sessions and the number of messages received. Most important reason behind this is that the student thinks that he/she has completed the exam successfully or he/she does not write anything to the HD despite facing a problem. As it can be seen in the figure, about 2/3 of incomplete exam sessions are not notified as problems by students. By use of the MA application in the proposed system, it was possible to develop a solution to the problems of the students who do not make any notification to the system along with those who make notification.

Figure 6. Incomplete exam sessions and problems notified

162

Exam implementation Integrated with the external system (Student Affairs database), ASOES is included in the course-instructor/students database. In order to prepare an exam, firstly there should be enough number of questions in the question pool relating to the course. The instructor makes choice of questions from among the questions he/she has earlier entered and publishes the exam. When the student enters into published exam, a session starts. Exam score of the student, who has completed the exam session without a problem, is calculated by evaluating the session data. Students experiencing a problem can notify to HD of their problems. Problem notified by the students and the results of the analysis made by the MA are evaluated by the instructor and he/she makes his/her decision regarding new exam entry rights, re-calculation of the scores, and update of question text, answer key and difficulty levels of the questions as deemed necessary after which the exam process is finalized and exam results are publicized. Monitoring agent implementation MA has three basic functions as Category, Exam Response and False Exam analysis. MA analysis results may be managed on the system by the instructor through a single screen (See Figure 7).

Figure 7. Monitoring agent

In the Category analysis, item difficulty index in calculated based on the answers given to the exam questions, and new difficulty level proposal, difficulty level averages of the same question in the previous exams and the difficulty level pre-specified by the instructor are submitted to the instructor. Under light of this information, the instructor can update the difficulty level through the MA interface. As an example to the Category analysis provided in this study, data of the Computer for Beginners course in the 2011/2012 fall semester of the distance education programs of Sakarya University was used. According to the category analysis performed, MA proposed to change the difficulty level for 9 out of 40 questions belonging to this course (See Table 6). Proposals to change difficulty level are made based on whether the item difficulty index values are within limits of (Table 1). It is shown in the instructor decision column of Table 6 in bold that the instructor has reckoned 4 out of 9 proposed cases and decided to change the difficulty level. While the instructor decides to change the difficulty level, he/she can consider how much the item difficulty index value converges to difficulty level limits. In the MA interface, calculated difficulty index values are submitted to the instructor together with the new level information.

163

As can be seen in Figure 8, 29th question assigned to the Easy category and 2nd, 3rd and 18th questions assigned to the Moderate category are quite far away from the limits. In such case, it can be seen that instructor’s decision to change the difficulty level is a reasonable approach. For values near limits, the instructor may choose not to change the difficulty level. It can be understood from Table 6 that the instructor decided not to change the difficulty level for questions 5, 20, 33, 38 and 38, as item difficulty indices are very close to limits.

Table 6. Category analysis result table summary

Question No. Difficulty Level Specified by Instructor

Item Difficulty Index*

Difficulty Level Proposed by MA

Decision of Instructor

2 MODERATE 0.211 HARD HARD 3 MODERATE 0.121 HARD HARD 5 EASY 0.628 MODERATE EASY

18 MODERATE 0.112 HARD HARD 20 MODERATE 0.322 HARD MODERATE 29 EASY 0.513 MODERATE MODERATE 33 MODERATE 0.347 HARD MODERATE 37 MODERATE 0.343 HARD MODERATE 38 MODERATE 0.342 HARD MODERATE

* Calculated by Equation 1

Figure 8. (a) Distribution of item difficulty index: Easy

Figure 8. (b) Distribution of item difficulty index: Moderate

164

Figure 8. (c) Distribution of item difficulty index: Hard

As the instructors create questions and add them to the question pool together with their answers, they can make errors in assigning the answer choices. It is almost impossible for these students to identify such problems. Exam Response analysis is performed on questions that are used for the first time in the system in order to determine if there are answer choice errors. In the Exam Response analysis, choices selected by the students are compared with the answer key from the IL. At the end of the comparison, number of correct answers as well as distribution of the incorrect answers is determined for each question. Ratio of each incorrect choice to the choice defined as correct by the instructor is calculated. It is inquired if this ratio is larger than threshold value identified as 0.70 based on experience in the previous years. Values for the questions for which proposal of change was made at the end of the analysis concerning questions used for the first time in all online midterm exams in the 2011/2012 fall semester are provided in Table 7.

Table 7. Exam Response analysis results table Distribution

Correct Choice Convergence Ratio Decision of Correct Choice

Update A B C D E A B C D E 7 8 11 27 20 D 0.26 0.30 0.41 0.74 Yes 5 47 10 19 48 E 0.10 0.98 0.21 0.40

No

53 30 38 2 11 A

0.57 0.72 0.04 0.21 No 12 10 17 43 39 D 0.28 0.23 0.40 0.91 Yes 9 68 6 51 3 B 0.13

0.09 0.75 0.04 No

5 53 4 6 49 E 0.10 1.08 0.08 0.12

No 47 33 15 13 23 A

0.70 0.32 0.28 0.49 No

10 16 9 36 26 D 0.28 0.44 0.25

0.72 Yes 0 51 9 11 57 E 0.00 0.89 0.16 0.19

No

5 9 14 37 41 D 0.14 0.24 0.38

1.11 Yes 31 85 7 79 39 B 0.36

0.08 0.93 0.46 Yes

6 3 45 4 17 E 0.35 0.18 2.65 0.24

Yes 3 15 15 26 21 D 0.12 0.58 0.58

0.81 No

3 3 15 3 21 C 0.20 0.20

0.20 1.40 Yes 62 19 26 20 88 E 0.70 0.22 0.30 0.23

No

57 33 26 32 50 E 1.14 0.66 0.52 0.64

Yes 68 41 61 21 7 C 1.11 0.67

0.34 0.11 No

3 4 27 7 4 D 0.43 0.57 3.86

0.57 Yes 59 2 1 0 54 A

0.03 0.02 0.00 0.92 Yes

Exam Response analysis was performed for 148 new questions and MA created reports for 19 questions for possible erroneous answer choice (See Table 7). At the end of the review by the instructor, answer choice error was detected in 10 questions, and information was sent to the Authoring process in the IL for the update of the answer key, and to the Evaluation process in the FL for re-calculation of the scores.

165

In the False Exam analysis, proposal decisions for the exam sessions, which the students could not complete, are created by help of the inference algorithm in the MA (See Rule 3). Here, a four-step algorithm is executed where P1, P2, P3 and P4 parameter values determined by the system are queried for the decision. In the first step of the algorithm, P1 parameter that corresponds to the presence of active exam sessions during time when the server does not reply is inquired to create giving new right for such exam sessions. Otherwise, it is passed to the second step of the algorithm to check whether the student abandoned the session deliberately (P2). If the session is abandoned deliberately, then the proposal will be not to give new right. Otherwise, we pass to the third step. In order to make a proposal at this stage, T1 threshold value should be calculated by exam-based evaluation of the exam sessions of previous semesters, and P3 parameter for the active exam. In this study, T1 value is calculated by analyzing 370 exams in the 2010/2011 spring semester. When the frequency values in Table 8 are investigated, maximum false exam session rate was found as 4.88 % in 351 exams, which is 95 % of all exams, and this value was taken as T1 threshold value.

Table 8. False exam sessions summary frequency table Cumulative Number of Exams Cumulative / Total Number of Exams (%) Maximum Error Ratio (%)

141 38.1 0.00 185 50 0.70 222 60 0.96 259 70 1.45 296 80 2.14 332 90 3.23 351 95 4.88 370 100 14.81

If P3 values are larger than T1 threshold value, MA proposes to give a new right, in the other case it is passed to the fourth step of the algorithm. In the fourth step, P4 value for the student whose problem is being investigated should be calculated, along with T2 threshold values derived from frequencies of all P4 parameter values calculated in all exams in the active semester. In this study, T2 value was calculated by analyzing the exam sessions of 5585 students in the 2011/2012 fall semester.

Table 9. Student exam completion status summary frequency table Cumulative Number of Students Cumulative / Total Number of

Students (%) Minimum Session Completion Rate

(%) 4347 77.8 100 4469 80 97.67 4741 85 95.92 5023 90 92.98 5307 95 86.84 5585 100 0.00

When the frequency values in Table 9 are investigated, it was found that 95 % of the students successfully completed at least 86.84 % of the exam sessions they started, and this value was taken as the threshold value (T2). Conclusion Online exams are used as an assessment-evaluation tool in the distance education systems that have quite a number of students today. For such systems, good execution of exams aimed at assessment & evaluation is of critical. Problems arising from human-centered errors or technical difficulties may lead to questioning of the exams, and thus (reliability and efficiency of) the distance education systems. In this study, an intelligent agent-supported integrated online examination system architecture is proposed. This architecture provides for integrated management of main functions such as question pool creation and update, exam authoring, execution and evaluation, and management of the feedbacks from students, along with ensuring use of

166

analysis reports relating to questions and exams created by an intelligent agent in the decision-making processes. This way, all problems that can arise in the operation process of the exam system will be able to reduce to an acceptable level, owing to the ability to decide more easily and rapidly provided by the system to the instructors. Student feedbacks are important for efficiency of online examinations systems. However, these feedbacks are not always enough to identify all problems. In addition, it is possible that such cases as failure to determine the question difficulty levels appropriately, erroneous answer keys, and incomplete sessions, for which no problem is notified, remain without a solution. And in some other cases, problem feedback does not provide sufficient information to make decisions such as giving new exam entry right, re-calculation of the exam score and changing question content and the answer key. In this regard, a mechanism that generates reports by help of certain rules was deemed necessary. For this purpose, a mechanism named MA was designed. This intelligent agent supported online examination system, ASOES, has been effectively used since 2011 for operation of 23 distance education programs of Sakarya University, aiming at 7000 students studying for associate’s degree, undergraduate degree, undergraduate completion degree and master’s degree. In practice, it was experienced that instructors became aware of the problems only when students informed them and that they spent too much time to solve these problems. In such case, it is understood from the results of the analyses conducted that intelligent agent-supported online examination system largely identified the problems and allowed the instructors to decide more easily. In this study, an intelligent agent structure with three functions was designed to meet the needs of Sakarya University, with the aim of solving the most frequent problems. Yet, it is possible to encounter different error conditions in the examination systems depending on the purpose of use of different institutions. The proposed system can be expanded to include additional intelligent functions aimed at solving these different errors with its flexible structure. Also, the intelligent agent structures used in this study can be expanded using semantics and fuzzy inference methods. References Aimin, W., & Jipeng, W. (2009, May). Design and implementation of web-based intelligent examination system. WRI World Congress on Software Engineering, 2009 (Vol. 3, pp. 195-199). doi: 10.1109/WCSE.2009.77

Alexakos, C. E., Giotopoulos, K. C., Thermogianni, E. J., Beligiannis, G. N., & Likothanassis, S. D. (2006, May). Integrating e-learning environments with computational intelligence assessment agents. Proceedings of World Academy of Science, Engineering and Technology (Vol. 13, pp. 233-238). Retrieved from http://waset.org/publications/10745/integrating-e-learning-environments-with-computational-intelligence-assessment-agents

Atılgan, H., Kan, A., & Doğan, N. (2006). Eğitimde ölçme ve değerlendirme [Assessment and Evaluation in Education]. Ankara, Turkey: Anı Press.

Aye, M. M., & Thwin, M. M. T. (2008). Mobile agent based online examination system. Proceedings of the 5th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (Vol. 1, pp. 193-196). doi: 10.1109/ECTICON.2008.4600405

Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge, UK: Cambridge University Press.

Blackboard. (2012). Running item analysis on a test. Retrieved November 30, 2012, from http://help.blackboard.com

Brusilovsky, P., & Miller, P. (1999). Web-based testing for distance education. WebNet World Conference on the WWW and Internet (Vol. 1999, No. 1, pp. 149-154). Chesapeake, VA: AACE.

Canvas (2012). Canvas instructor guide - Quizzes. Retrieved November 29, 2012, from http://guides.instructure.com/s/2204/m/4152/c/23861

Crisp, V., & Ward, C. (2008). The development of a formative scenario-based computer assisted assessment tool in psychology for teachers: The PePCAA project. Computers & Education, 50(4), 1509-1526.

Fotaris, P., Mastoras, T., Mavridis, I., & Manitsaris, A. (2010). Extending LMS to support IRT-based assessment test calibration. Technology Enhanced Learning. Quality of Teaching and Educational Reform, 73, 534-543.

Gawali, R. D., & Meshram, B. B. (2009). Agent-based autonomous examination systems. Proceedings of the International Conference on Intelligent Agent & Multi-Agent Systems (pp. 1-7). doi: 10.1109/IAMA.2009.5228095

167

Guo, P., Yu, H., & Yao, Q. (2008). The research and application of online examination and monitoring system. Proceedings of IEEE International Symposium on IT in Medicine and Education (pp. 497-502). doi: 10.1109/ITME.2008.4743914

Guo, S., & Mao, Y. (2010). OPES: An on-line practice and examination system based on web. Proceedings of the International Conference on E-Business and E-Government (ICEE) (pp. 5470-5473). doi: 10.1109/ICEE.2010.1370

Hang, B. (2011). The design and implementation of on-line examination system. Proceedings of the International Symposium on Computer Science and Society (ISCCS) (pp. 227-230). doi: 10.1109/ISCCS.2011.68

Jin, X., & Ma, Y. (2008). Design and implementation to intelligent examination system for e-business application operation. Knowledge-Based Intelligent Information and Engineering Systems (pp. 318-323). Berlin, Germany: Springer doi: 10.1007/978-3-540-85565-1_40

Jun, L. (2009). Design of online examination system based on web service and COM. Proceedings of the International Conference on Information Science and Engineering (ICISE) (pp. 3276-3279). doi: 10.1109/ICISE.2009.484

Keleş, A., Ocak, R., Keleş, A., & Gülcü, A. (2009). ZOSMAT: Web-based intelligent tutoring system for teaching–learning process. Expert Systems with Applications, 36(2), 1229-1239.

Kelley, T. L. (1939). The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30(1), 17.

Kubiszyn, T., & Borich, G. D. (1990). Educational testing and measurement: Classroom application and practise. New York, NY: Harper Collins Publishers

Li, X., & Wu, Y. (2007). Design and development of the online examination and evaluation system based on B/S structure. Proceedings of the International Conference on Wireless Communications, Networking and Mobile Computing (pp. 6223-6225). doi: 10.1109/WICOM.2007.1525

Moodle. (2012). Quiz statistics calculations. Retrieved November 29, 2012, from http://docs.moodle.org/dev/Quiz_statistics_calculations

Mustakerov, I., & Borissova, D. (2011). A conceptual approach for development of educational Web-based e-testing system. Expert Systems with Applications, 38(11), 14060-14064.

Post, G. V., & Hargis, J. (2012). Design features for online examination software. Decision Sciences Journal of Innovative Education, 10(1), 79-107.

Shen, R., Tang, Y., & Zhang, T. (2001). The intelligent assessment system in web-based distance learning education. Proceedings of the Frontiers in Education Conference (Vol. 1, pp. TIF-7). doi: 10.1109/FIE.2001.963855

Tallent-Runnels, M. K., Thomas, J. A., Lan, W. Y., Cooper, S., Ahern, T. C., Shaw, S. M., & Liu, X. (2006). Teaching courses online: A review of the research. Review of Educational Research, 76(1), 93-135.

Tinoco, L. C., Barnette, N. D., & Fox, E. A. (1997, March). Online evaluation in WWW-based courseware. ACM SIGCSE Bulletin, 29(1), 194-198. doi: 10.1145/268085.268156

Wang, T. H., Wang, K. H., & Huang, S. C. (2008). Designing a Web-based assessment environment for improving pre-service teacher assessment literacy. Computers & Education, 51(1), 448-462.

Wilson, E. V. (2004). ExamNet asynchronous learning network: Augmenting face-to-face courses with student-developed exam questions. Computers & Education, 42(1), 87-107.

Zhang, L., Zhuang, Y. T., Yuan, Z. M., & Zhan, G. H. (2006). A web-based examination and evaluation system for computer education. Proceedings of the Sixth International Conference on Advanced Learning Technologies (pp. 120-124). doi: 10.1109/ICALT.2006.1652383

168