soa peer&collaborative assesment final

Learning Infrastructure, WP7.1, D7.1.2, UNIGE 27

Peer/collaborative e-‐assessment case studies We have collected different case studies descriptions and settings available online or from the literature and submitted to the online survey available at: https://docs.google.com/forms/d/11rEtD-oE2D1ClcJ_4aaCWDKVeFjA6VhRxVyrojmhEg0/viewform Each case study is briefly described, analysed and discussed.

Case 1: Using Google drive for peer-‐assessment Technology: Google drive (Form and Spreadsheet) Type of assignment: Writing essay Type of assessment: Formative From http://gettingsmart.com/2012/12/the-sidekick-the-superhero-using-google-drive-for-peer-assessment/ This case study is conducted in AP language classes (but can be extended and applied to any discipline) for essay writing assignments. The ICT tool involved is Google Drive and more precisely Goole Form and Google Spreadsheet (both tools are working together as Form results are stored in Spreadsheet documents. Students have to write consecutively 3 free response essays in 3 weeks. Each essay assignment is based on a four days scenario. Each essay is peer assessed at least two times. The assessment is conducted in double blind, using a numbering system for the essays and the students. The allocation of essays to reviewer students is random. Reviewer students have to fill in a review form available through the Google Form service. The review form includes grades (such as “rate the thesis statement”) and arguments (such as “review the student thesis statement” or “suggest how it may be improved”). The resulting Google Spreadsheet is publicly shared among students and they are requested to consult the feedbacks for their essay as homework. The assignment is concluded with a self-assessment to review peers’ constructive remarks and critics. The self-assignment is submitted as another Google Form. This last assessment is not shared among students of the class. The advantages identified are:

• Students are clear with what is expected for the assignment. • Students get a diagnosis about their strengths and weaknesses in writing. • Students can be actively involved in their own learning.

No particular disadvantages or requirements are mentioned.

Case 2: Developing an essay through peer-‐review on a discussion board Technology: Discussion board Type of assignment: Essay writing Type of assessment: Formative From http://serc.carleton.edu/introgeo/peerreview/examples/sharks.html Before writing their essay, students are helped to develop the topic they have chosen for their essay through a peer-assessment exercise that is ran on a discussion board. The student has first to answer a few questions about the essay topic in a discussion board. Two peers have


then to evaluate the student’s answers. Finally, the student has to answer the peers’ comments. No particular advantage or drawback is indicated for this case apart from the development of students’ electronic communication skills and constructive feedback capacity.

Case 3: An assignment using anonymous electronic peer review with a Dropbox Technology: Dropbox Type of assignment: Essay writing Type of assessment: Formative From http://serc.carleton.edu/introgeo/peerreview/examples/warming.html Students write their essay anonymously, without indicating their name or any other information that could help to recognize them. They upload their essay documents onto the Dropbox and send the address to the teacher. The teacher can then assign the essay to be reviewed by one or two peers using a criteria grid. The final mark can be assigned based on the average of the peers’ marks. The only advantage indicated for this case is that peer reviewing and assessment avoid teachers to assess students’ work themselves. The author suggests 1) having a test peer review assessment in class to train students; 2) providing students with clear explanations about the assignment and the peer review process (including the review form itself).

Case 4: Calibrated peer assignment Technology: CPR online web software Type of assignment: Writing essay Type of assessment: Formative/summative From http://serc.carleton.edu/introgeo/peerreview/cpr.html, http://serc.carleton.edu/introgeo/peerreview/examples/dinosaurs.html, http://serc.carleton.edu/introgeo/peerreview/examples/why_study_geo.html, http://serc.carleton.edu/introgeo/peerreview/examples/petroleum.html Calibrated peer assessment is a process organized in four successive steps:

1) Assignment: The student writes an essay and submits it. 2) Peer assessment training or calibration: the student has to peer review 3 example

essays: calibration essays that have already been evaluated by teachers using a rubric form. If the calibration test meets the requirements, the student can move to the next step. Otherwise, the student has to pass a second peer assessment trial.

3) Peer assessment: The student has to assess and grade three peers’ essays. If the student failed the first calibration evaluation, his/her impact of the peers’ mark is lowered.

4) Self-assessment: The student self-assess his/her own essay. The introduction of calibration in peer assessment is particularly important. It trains students to review and assess their peers.


The advantages expressed in the different related cases include: promotion of critical thinking, improvement of writing skills from one assignment to the next one.

Case 5: Getting to know Coursera: peer assessments Technology: Coursera Type of assignments: NA Type of assessment: Summative From http://cft.vanderbilt.edu/2013/01/getting-to-know-coursera-peer-assessments/ http://cft.vanderbilt.edu/2012/11/getting-to-know-coursersa-assessments/ This case study is more a critical analysis of the way peer assessment is implemented inside Coursera, a xMOOC platform than a real case study. xMOOCs are a particular type of Massively Open Online Courses. They propose a model of teaching based on the traditional teacher-centric approach with the purpose to provide teaching to huge online classes (Figure 4). In the next case we will review an current experiment conducted inside Coursera that aims at addressing some of the issues raised in this one.

Figure 4 – x MOOCs and cMOOCs (figure from Wikipedia)

However, it gives some good feedback about the problems to consider when applying peer assessment for very large online classes. With respect to traditional classes, one must notice, that MOOCs are completely online with distant students from all over the world. This situation introduces many more constraints for teaching in general but also for peer assessment. Peer assessment is particularly critical with respect to the context of MOOC. It is indeed impossible to believe that the teachers and tutors staff can provide assessment and feedback to thousands of online students. As raised in the state of the art section of this document, peer assessment can be viewed as a possible solution for large classes. The author translates this into "Who, after all, has got the time to read 10,000 essays? The answer, for Coursera at least, is other students." In other words, peer assessment is the only way platforms such as Coursera can cope with assessing thousands of essays, which results in crowdsourcing assessment. According to the author, “the model of peer assessment supported by Coursera folds together two assumptions: that peers can approximate or replace the kinds of substantive, constructive expert feedback critical to deeper understanding and that a grade is necessary to learn, full stop". The main issues raised by the authors are:

• Students have to learn to grade


• Grading peers requires lots of efforts from students • What is the outcome of peer grading for students

The author brings a particular importance to anonymity and privacy. As peer feedback on Coursera is anonymous, follow-up on a comment and discussions are mainly impossible. Therefore, how to create learning communities if peers are not accountable for their feedbacks? Similar feedbacks are available in http://hackeducation.com/2012/08/27/peer-assessment-coursera/

• the variability of feedback, • the lack of feedback on feedback, • the anonymity of feedback, • the lack of community.

Case 6: First massive-‐scale class with self and peer assessment in Coursera Technology: Coursera Type of assignment: Project/problem-based Type of assessment: Summative From http://hci.stanford.edu/research/assess/ This case study describes an experiment to introduce peer assessment in an xMOOC. This strategy has been implemented inside Coursera3. As raised by the authors, "providing feedback and assessment of design and other creative work is extremely time consuming -- this bottleneck is the major capacity constraint for scaling peer assessment". In this example, students are not only grading peers, but they are trained to prior to effectively grade them. The proposed method is based on an existing method called “calibrated peer assessment” where students learn grading through training examples before grading their peers. The peer assessment is combined with self-assessment. The objectives of the peer assessment strategy are: 1) training students to assess others accurately; 2) define a grading system robust to errors; 3) provide qualitative and personalized feedback to students. The authors use rubrics to grade. Rubrics are exemplified in Figure 5. Each row corresponds to a rubric and each cell corresponds to a level of performance. Assessing is mainly numeric, with very few text for feedback (as language is an issue for a MOOC).

3 http://www.coursera.org/


Figure 5 – Rubrics-based grading form

Students have first to train assessing. They are submitted examples to assess. They get the right to assess their peers once they grade the example close to the grading result assigned by the teaching staff for the example. Each time they perform an example, they get a feedback to explain them is they are higher, lower or close to the staff grade and why the staff has assigned the grade. The peer assessment is a 3 steps process. The student first assesses 5 peers’ assignments. Among the five assignments, the teaching staff has marked one. It serves as “ground truth” for comparison between staff grading and students grading. In the next step, the student self-assess his/her own assignment. For each assignment, a grade is computed as the median grade of the five peers’ assessments. This peers’ grade is compared to the student grade from the self-assessment. If the student’s grade is close to the peers’ grade, then the student gets his/her own grade. Otherwise he/she gets the peers grade. Other strategies can be used, such as assigning the maximum of the two grades.

Figure 6 – 3 steps peer assessment

The authors use data analysis to guide improvements. They update the rubrics according to the results in order to clarify them. In term of assessment feedbacks, they propose feedback


templates. Students are proposed basic feedback templates that they can customize by completing the template. From the data analysis, they have noticed that the staff grades correlated with the peers’ grades. They are currently exploring other ways to weight the peers grades than using the median. They have noticed what they call a “patriotic” grading, where peers tend to grade their compatriots higher. Grading errors are evaluated by comparing with ground truth grades. The results show that errors are quite balanced. One interesting outcome raised by the authors is that the process stimulates collaborative learning, with students sharing resources, creating assignment aids, answering forum questions, and providing extra peer assessment…

Case 7: Web-‐based peer assessment: a case study with civil engineering students Technology: Google drive Type of assignment: Writing essay Type of assessment: Formative

From http://online-journals.org/i-jep/article/viewArticle/2411 and http://www.slideshare.net/gmatos/icl-2012-final

The assignment process takes place in five successive steps:

1) The student selects an article from an online source. 2) The student uploads the article as a Google Drive document. 3) The student summarizes the article. 4) The student analyses the article. 5) The student gives his/her opinion about the article.

Steps 3 to 5 are written in a Google Drive document that is then shared by the author student with the teacher and with one assigned peer reviewer. The peer reviewer assesses the document using the comment features of Google Drive and grades the work. The teacher then review and grades the author student and the peer reviewer. The author student can then review the feedback and update his/her work. Finally, the teacher reviews the updated online document to give the final grade. During the assignment, additional online documents are provided through Google Drive: the orientation document describes the objectives and tasks to be performed; a table to connect authors to peer reviewers and a table for the management and coordination of the tasks between author, peer and teacher. The authors observed that only a small number of students used their peers’ feedback to improve their essay. The main observations are:

• The use of a digital environment to support the assignment and assessment process did not present any difficulty to the students.

• The support material used to present the assessment process in face to face was very important for the good achievement of the process.

• There is an obvious need to improve students’ feedbacks and communication skills. • The teacher’s grade seems to influence the use of peer’s feedback to improve the

work. • The over evaluation of the teacher’s grade and the small difference between student’s

grade and teacher’s grade question the necessity to have a double intermediate assessment (peer and teacher).


Case study 8: Online peer-‐assessment in a large first-‐year class Technology: Workshop module (Moodle) Type of assignment: Writing essay Type of assessment: Formative

From http://www.academia.edu/665555/Where_Angels_Fear_to_Tread_Online_Peer-Assessment_in_a_Large_First-Year_Class

This case study aims at providing formative feedback to widen participation and develop writing skills for large and diverse classes (800 students).

The Moodle Workshop module was limited to 3 basic features: 1) the submission and the random distribution of assignments; 2) the grading and feedback based on grid forms; and 3) sharing of work with peers’ feedbacks. The peer assessment if only formative and peer feedbacks are anonymous. As students are used to peer assessment, no example essays are provided and no self-assessment is requested. At the end of the assignment period, the best peer scored five essays are publicly published. Students are provided with a rationale about the advantages of peer assessment and detailed information about the assessment process and forums were used for scaffolding.

The assessment process itself is organized into 3 main steps: 1) each student submits the first version of the essay; 2) at least two peers review the essay; and 3) each author student has to answer the peer feedbacks and submit the final essay. The teaching staff grades the final essay. Peers’ assessments are not evaluated nor marked. Engagement was expected by rewarding students: peer assessment replaced one exercise.

Students’ evaluation of the assessment process is mixed, but around ¾ of the students find it more useful to provide than to receive feedback.

Case study 9: Enquiry-‐based peer assessment Technology: BlackBoard, ASK (Assignment Survival Toolkit) Type of assignment: Writing essay Type of assessment: Formative/summative

From : http://www.academia.edu/1495031/Online_peer_assessment_helping_to_facilitate_learning_through_participation

The objective of this case study is to embed enquiry-based learning, information literacy and e-learning in an peer e-assessment assignment.

According to the authors, in enquiry-based learning, students are working in groups to solve problems with the help of a wide range of information resources. The teacher intervenes as a facilitator that enables students to self-regulate their learning.

Thee peer assessment assignment includes four stages and lasts three weeks:

1. Students write a 500 words essay answering a question submitted by the teacher. This first version of the essay is formatively reviewed in face-to-face with tutors.

2. A second longer version of the essay is then written, which is formatively assessed by peers on BlackBoard.


3. Student are then organized in groups and based on ASK (Assignment Survival Toolkit) that provides resources and individualized step-by-step planning for writing an essay, they have to write a third full essay. This full essay includes introduction, body and conclusion. During three weeks, the peers in a group have to reciprocally submit weekly feedbacks about the essays. Students are taught to submit productive feedbacks (based on setting the criteria, selecting the evidence, making a judgement).

4. Students have then to review and update their essay to submit a final version of the essay that is marked.

The results of the students survey raise the importance of feedback both for tutors and peers. The outcomes come both from the feedbacks received and from the peers’ works reviewed and assessed. But it also points out the credibility of feedback: some students favour tutors’ feedbacks against peers’ ones or wonder how to trust the feedbacks of people who are at the same stage of knowledge than themselves.

Another result that arose is that peer assessment facilitates and enhances learning. These results matches the literature ones that indicate that peer assessment encourages students to collaborate, share and reflect.

Case study 10: Coursera Peer Assessment -‐ Writing in the Sciences Technology: Coursera Type of assignment: Writing essay Type of assessment: Summative

From http://scienceoftheinvisible.blogspot.co.uk/2012/10/coursera-peer-assessment-writing-in.html

This case study is another MOOCs related example of peer assessment. This one is described from the point of view of a student (at least a teacher who followed the course as a student).

The peer assessment assignment is organized as follows:

• Each student has 7 days to write an short essay of few hundred words. • Each student has 7 days to assess 5 essays of peers and grade them on 0-3 scale with short free

text feedback on different rubrics. The assessment is done twice, the first time with updates suggestions and the second one on the revised version for a final mark.

The author indicates that the process worked well for him. However, when wondering if he would apply this model to his students the answer is: “I'd like to think so but I'm not sure. For one thing it's not clear that our students are as confident or motivated as the participants in this course. For another, there is the issue of marking cartels as students indulge in the prisoner's dilemma (as they perceive it) with summative assessment. Sadly, I can't see a system like this being a goer for us.”

Case study 11: Peer feedback sessions Technology: Skillshare (@ skillshare.com) Type of assignment: Video presentation of a project results Type of assessment: Formative

From: http://moocnewsandreviews.com/massive-mooc-grading-problem-stanford-hci-group-tackles-peer-assessment/


This case study describes how an online course platform organizes a lightweight peer assessment system based on “peer feedback sessions”4. Students can opt in joining peer feedback sessions. When doing so, their projects are then submitted to two peers who will be able to submit constructive feedbacks. In return, the peers’ projects are submitted to the student for the sale session. It initiates a discussion between the reviewers and the reviewed.

Figure 7 – Student’s request to submit a project to peers

The author describes a session he has been participating to for a course on video production. He mentions that the peer feedback process has involved him in diving deeper into the course.

Case study 12: Reliability and validity of web-‐based portfolio peer assessment Technology: E-portfolio Type of assignment: Project Type of assessment: Formative/summative

From C.-C. Chang, K.-H. Tseng, P.-N. Chou, et Y.-H. Chen, « Reliability and validity of Web-based portfolio peer assessment: A case study for a senior high school’s students taking computer course », Computers & Education, vol. 57, no 1, p. 1306-1316, août 2011.

This case study concerns a class of around 70 students who have to implement and present a project. The assessment process is organized as follows:

1. Students are first provided with portfolio samples, assessment criteria and guidelines. The rubrics of the assessment criteria are tailored according to students’ feedbacks.

2. Students have then to develop their portfolios, monitor peers’ portfolios and participate to forums.

3. Students have finally to perform peer assessment. Peer assessment is anonymous and group-to-group. At the same stage, the teaching staff scores the portfolios.

The global result of this case study is the lack of reliability of portfolio peer assessment. The authors identify the need to avoid or attenuate grading bias. They argue about the burdensome features of portfolio assessment, particularly from the point of view of the teaching staff. They also suggest that “advanced trainings and support so that students would be more likely to get involved in the assessment process with proper abilities".

Case study 13: Teamwork skills assessment for cooperative learning Technology: Ad-hoc platform Type of assignment: Group work Type of assessment: Formative/summative From P. S. Strom et R. D. Strom, « Teamwork skills assessment for cooperative learning », Educational Research and Evaluation, vol. 17, no 4, p. 233-251, 2011 and D. Brown, « Implementation of the Teamwork Skills Inventory among adolescents », 2010. [En ligne]. Disponible sur: http://hdl.handle.net/2286/9c2jizip6q5. [Consulté le: 08-mai-2013].

4 http://help.skillshare.com/customer/portal/articles/1104466-what-is-a-peer-feedback-session-


The TSI is a method based on peer and self-assessment to evaluate teamwork skills (currently, 25 skills are defined). It is an anonymous online assessment tool where students answer questions regarding the individual contribution of each peer and then evaluate their own contribution. Teachers are expected to provide instructions and have discussion about teamwork, the skills… The authors of the method have defined a 5 lessons curriculum to train and teach students about teamwork peer assessment. Teachers have also to express their trust in the fairness that students can achieve during the assessment. Once the workgroup is achieved, each student assesses and marks each of the 25 skills for himself/herself and peers. Once the assessment is achieved, each student get a profile organized into two columns to compare self-assessment with a aggregated view of peers’ assessments. The method integrates features to attenuate over evaluation of peers: a warning pop-up message when the maximum grade is provided for a skill and the inflation rating index that indicates that a student needs additional guidance information for improving assessment. For students, the method helps them to compare their self-evaluation to peers’ one. It also helps them to improve their self- evaluation. For teachers, the process allows evaluating weaknesses for individuals and groups in order to adapt learning and evaluating teachers’ own skills for training students for work group. The difficulties are related to the level of trust that teachers can provide to student to embed them in the assessment process. Peer assessment is also a challenge for the teachers who need to share with students the way he/she assess them. Time and efforts are required to setup proper assignments. Peer assessment supports collaborative learning.

Case study 14: Facilitating peer and self-‐assessment Technology: WebPA Assignment type: Work group Assessment type: Formative/summative From http://www.jisc.ac.uk/media/documents/programmes/elearning/digiassess_assessingselfpeers.pdf This case study describes how WebPA has been used in the Universities of Hull and Lancaster. The platform has been applied for peer assessment of work group in many disciplines, ranging from English to Civil Engineering. The functioning of the WebPA platform has been described in the state of the art section of the document. Therefore we will focus on the advantages, drawbacks and limitations identified in this case study. The authors rise that tutors have not recorded any complaints for malpractice (which does not mean that there were no problems, but that students did not report them). It seems critical to take the time to explain and demonstrate in face to face: it indicates the importance that teachers give to the process; it allows addressing basic questions; it avoids problems during the assessment process. It is even suggested to improve students’ involvement by defining the assessment criteria in collaboration with them. Based on their experience, the authors indicate that students:

- acquire a greater sense of ownership and control over their learning


- work harder to get a successful assessment from their peers. Peer assessment also favours dialogue and social interaction between students and can therefore smooth newbie students’ integration. For the teachers, there are obvious practical advantages over “paper”-based peer assessment: it can be accessed from any place and at any time, the results are instantly and securely collected. From a pedagogical viewpoint, it enables assessing skills that are normally difficult, or even impossible, to assess.

Case study 15: Formative collaborative quiz with clickers Technology: Clicker; votamatic (votamatic.unige.ch) Assignment type: Quiz Assessment type: Collaborative/formative Submitted by: [email protected] This case study has been investigated at the University of Geneva in the context of a first year bachelor course dedicated to multimedia technology with a class of approximately 120 students. Students are using their own devices: laptops, tablets or smartphone. This assessment is performed in face to face at the beginning of a class. A few simple quiz questions have been setup with the votamatic tool. Students have not been advised about the assessment. They are explained about the goal of the assessment; they are told that their answers are anonymous and that there will be no mark for this exercise. Students are then requested to self-organize themselves in groups of 2 or 3 (but students who want to remain alone are authorized to do so). They are given a simple URL to reach the quiz (without any login or authentication process). The quiz can be accessed either on a laptop, tablet or smartphone. By grouping students in teams of 2 or 3, there are enough available devices in the class. Students are given 15 to 20 minutes to answer the quiz. They can ask questions to the teacher, discuss between them during this period. Once the period is over, the teacher stops the quiz and the results are displayed on the screen and all students can view them at the same time on their screen of their device. During this last period, the teacher goes through each question, discusses the results and explains the answers. For the teacher, it is a good way to evaluate the global level of the class and identify weaknesses. The exercise initiates a discussion between the teacher and the class and among students. As the answers are anonymous, students are comfortable to participate and do not express any reluctance to participate. The assessment tool is lightweight and easy to use for the teacher and the student.. The whole exercise takes around 45 to 60 minutes, but it is possible to reduce the time by submitting the quiz between two classes and only discuss the results during the class.


Figure 8 – a snapshot of a quiz

Figure 9 – A snapshot of the display of the quiz results


Case study 16: Gamified work group assessment Technology: User points; elgg (hec-onnect.unige.ch) Assignment type: Project Assessment type: Formative/summative Submitted by: [email protected], [email protected] This case study has been investigated at the University of Geneva in the context of a first year bachelor course dedicated to an introduction to web services with a class of approximately 300 to 400 students. Gamification is one among various approaches that are applied to engage and organize participation. It consists in introducing game mechanics in non-game contexts. The main objective is to increase the user engagement. This approach has raised a lot of interest and development in education with the expectation to improve students’ engagement in learning activities. One of the techniques involved in gamification is based on user’s reward. The reward is usually based on a score that the user is earning throughout his/her interactions with the system. Whenever the user is acting positively his/her score is increasing. Once the score reaches a pre-defined threshold, the user is getting a reward (a badge that is displayed on his/her profile for example). The basic idea consists in adapting the user points approach in order to estimate students’ individual contribution to the global effort. At the end of the group work, students’ scores are used to assign a mark that is then integrated to compute the final mark. Group work is supported with an online shared workspaces platform. The platform is used for collaborative learning so that students can tutor their peers and provide them feedback during the group work project. The tutoring can apply to the activities of the work group assignment but also to the technical and organisational skills required to use the collaborative platform. We consider each activity that a student can have with the platform and evaluate it according to its contribution to the increasing of the global knowledge of the whole class. A student who publishes a public bookmark is considered as being willing to share a resource with the peers. A student who comments a content produced by another student is considered as being willing to provide a feedback to the peers. These two activities will be positively rewarded. We do not evaluate the quality of the production. Only the intention to contribute is rewarded. We are aware that we may reward “useless” contributions. Our policy is to favour contributions by considering that learning students are not systematically able to perform efficiently from the beginning of the group work. The process takes also into consideration the actions that students can perform to increase their own knowledge. For example, when a student reads a content produced by a peer, we consider that the student is willing to learn from others. Therefore, his/her score will increase. A pre-defined ranking of all the possible actions is established. The ranking is defined according to the weight of the contribution that a given action may have to the global knowledge. Sharing a bookmark will be for example considered as a less significant contribution than commenting a content. The amount of points a student can earn for a given action is depending on the rank of the action. The teacher can monitor the assignment of the user points at any time and get the final amount for each student. He/She can then define by himself/herself how to integrate this scoring of the student’s individual contribution in the final mark of the group work. Students are made aware about the fact that their individual contribution and support to the global platform knowledge is evaluated. The collaborative platform is developed with the open source Elgg social network engine. The core engine is augmented with various plugins. Shared workspaces are defined as groups. Each group has its own workspace and toolbox (the toolbox integrates wiki, blog, forums,


question/answer, brainstorming tool…). Professors, teaching assistants and staff as well as students are given the same rights on the platform. They can for example create a group for formal or informal learning activities. A gamification plugin has also been partly integrated. The user points system is activated whereas the badges system is disabled. Students cannot access their score. The platform has been used since 2010 for a 1st year bachelor course in Information Systems for students in commercial and management studies. Every year, the class varies between 300 to 400 students. They have to work in groups for the project semester. The project is organized into multiple phases. For each phase they have to produce outputs that are increasing in complexity. During the project they are continuously provided with resources and guidelines (online and face to face) so that they can gradually learn to use the platform and tools, and get used to collaborate. The final mark is computed from the individual contribution score and the evaluation of the final group production. The individual contribution is estimated by defining ranges of user points. The ranges correspond to different levels of contributions from inactive to very active. For each student the individual mark is assigned according to the user points range in which his/her score stands. Therefore, the students in the same group may receive different final project marks. We have already raised the issue of useless contributions with the risk of rewarding them unfairly. From our experience, we have noticed that if we provide students with differentiated types of content, it is possible to discriminate and orient low-level contributions. For example introducing a shoutbox allows gathering most of the “logistics” messages (such as “where do we meet?”). Moreover, by assigning individual contribution marks according to pre-defined ranges of user points, we avoid fostering students who are over-contributing. The resulting collaborative learning platform encourages students to contribute and collaborate. It addresses the “free rider” problem by providing an indicator of the student’s individual contribution. This indicator allows defining a mark that can be taken into consideration for the final mark. Further developments include the refinement of the rules to assign user points and the introduction and evaluation of intra and inter-group peer assessment. The refinement of the user points rules is expected to bring a better estimation of students’ individual participation. The rules to define the ranges of user points to assign marks can also probably be enhanced. The intra-group peer assessment is expected to adjust the individual contribution score with the evaluation from the peers. The inter-group peer assessment is expected to adjust the global group contribution.

Case study 17: Portfolio-‐based collaborative assessment Technology: Blog/Portfolio Assignment type: Project Assessment type: collaborative/formative Submitted by: [email protected], [email protected] At the University of Geneva, physical education is currently taught in dual system: students share their studies between theory at the University and practice in schools. During the theory periods, student trainees are taught by university trainers and are organized in classes. During the practice periods, student trainees stay in primary and secondary schools, they are supervised by field trainers and are organized in binome teams. During their stay in schools, student trainees have to prepare lessons and apply them with classes under the supervision of field trainers (field trainers are themselves physical education teachers).


This combined peer-tutoring (where student trainees teach to their peers) and peer-assessment (where student trainees assess their peers) approach allows increasing the academic gain for both tutors and tutees. The main issues are: the number of categories of participants involved; the lack of continuity and contacts between the participants (Figure 10.a). This lack of contact does not only affect students and trainers. It also concerns university trainers and field trainers. The objective of this project is to introduce distance learning technology in order to keep the participants connected and stimulate exchanges and feedbacks. The selected approach consists in organizing participants’ interactions around student trainees’ activities with e-portfolios (Figure 10.b). According to the specific context induced by the dual education system, the training platform must be at the same time:

- A common place where the different categories of participants: student trainees, university trainers, field trainers can continuously exchange, harmonize and converge.

- A place where student trainees can depict their activities, get feedbacks, monitor their progress and be evaluated.

The evaluation aspect is particularly important. Trainees feel that the creation of an e-portfolio content is more of a process that is required rather than a product that can demonstrate the development of professional growth. They usually feel this last stage at the end, once they can browse their portfolio content. Therefore, including the portfolio content in the evaluation creates an initial constraint to engage the trainees in the production of content for the portfolio.

(a) (b)

Figure 10 - E-portfolio as a virtual common shared space to overcome dual system barriers

Another important issue is that the structure of the platform needs to reflect the structure of the pedagogical organization: classes, binomial teams and students. It must also reflect the roles of the different participants in the pedagogical organization, particularly in terms of interactions such as feedbacks from the trainers to the trainees. We consider three levels corresponding to the three levels of integration of student trainees: 1) individual level (for individual progress and evaluation) 2) binomial team level (for co-elaboration, feedback and evaluation) and 3) class level (for global management and information and theoretical material dissemination). The levels are organized hierarchically so that when a student trainee submit some contribution at the binomial level, it also appears at the individual level (so that he/she can monitor his/her own progress), but does not appear at the class level as it does not correspond to that level (however, the contribution can be reviewed by other student trainees as all published contribution are made public to all users).

University trainer

Student trainee

Field trainer

University trainer

Student trainee

Field trainer

E-portofolio

university schools


Implementing such a platform requires taking users skills into consideration regarding the design. The first constraint is that none of the users of the platform is particularly skilled in information technology. Therefore, the platform should not be overloaded with functionalities. It is of course not possible to avoid the additional workload induced by the need to master the platform. However there are few design rules that can be applied to make it simpler. We have devised the following approach: -Rule 1: provide users with only the features that are required, so that they do not get lost with too many options that they have to test and acquire. Each role is clearly identified and gives access to the required features. For example, field trainers do not need to any content other than feedbacks. Therefore, they have no personal blog and are only limited to produce comments. -Rule 2: provide tools that are similar to the ones that users may use in their personal practice of information technologies. For example, the students’ e-portfolio is a blog, which may be already familiar to some of the student trainees. Our main objective at the implementation level is to reflect the same structure as the one developed at the pedagogy level. Figure 11 describes the global architecture organized around the student trainees’ e-portfolios. The class organization is reflected through the use of groups: a student trainee is an individual user with his/her own blog. He/she is member of a binomial team, which is also equipped with a blog. And finally each binomial team is a subgroup of the class group. This structure ensures the dissemination of contents among the appropriate levels. Depending on at which level a post is submitted, it appears in different blogs levels (posts always appear in the individual blog of their author).

Figure 11 - Global overview of the platform structure and organization

We also assign a role to each participant (University trainer, Field trainer, Student trainee). Each role is assigned with some rights to produce contents (corresponding to their possible interactions with the platform and the other roles). For example, student trainees need to submit lessons preparation and practice reports, so they are assigned with a blog and ability to


submit blog posts. They are also expected to provide feedbacks to their peers, so they are assigned with the ability to submit comment. They are expected to communicate with trainers, so they are assigned with the ability to submit forum topics and posts.

Case study 17: Acadima Technology: Dedicated platform Assignment type: Flashcards Assessment type: Formative Submitted by: [email protected] Acadima - http://www.acadima-information.ch Acadima provides university students throughout Switzerland with the opportunity to write, edit and share both learning and test flashcards targeted on their modules and final exams. Students can enhance the quality of the cards by providing mutual feedback and quality assessments. In this way, a dynamic question pool is developed that can be readily retrieved by students via their AAI access. The flashcards can be called up on smartphones or via a web interface. The skills developed include: - self-organized learning (cooperative, reflexive, motivational, emotional, cognitive --- self competence) - knowledge about how to write good MC-questions --- competence in didactics - peer review --- competence in didactics --- discovering miss concepts - competence in media (critic, skilful, usage, design) The technologies components involve: - collaborative question pool for exam preparation (card status: published - visible for peers) - peer review - "reviewed" flag (through feedback) - prof review - "profproofed" flag - peer voting (card quality voting, card level of difficulty voting) - feedback - gamification (boost your peers), push mechanism - mobile learning. The workflow has to be simple and clear. Students need to see immediately their data. Simple login such as AAI. No advertisement and cost free. Sharing of small amount of data for exam preparation (e.g. learning and test flashcards). Providing feedback features (feedback, voting); building of communities; gamification; freedom equality; trust; community; collaboration and usability. The main goal is not the provision of learning and test flashcards as such, but the contents, which are created and made available by students. Students can enhance the quality of the cards by providing mutual feedback and quality assessment. In this way, a dynamic question pool is developed that can be readily retrieved by students via their AAI access. This will motivate students to work together with each other, such as through the option of marking cards as favorites, rating them as excellent or criticising them. This serves to increase the quality of the cards, because together we are wiser. Acadima promotes two fundamental cooperative processes. On the one hand, cards can be compiled on an altruistic basis for others to use as well and, on the other hand, students can work together on a joint basis. A shared benefit results. People are naturally disposed to cooperate, to exchange information and tasks, and to share their aims. Acadima can be used by university students and teachers for creating and working with flashcards as well as for sharing them with their fellow students. Acadima has been designed in cooperation with universities and students. It is an optimized form of learning and retaining new information. By sharing with others, student


can be saved from having to create their own sets of flashcards. Acadima enriches learning, teaching and campus life not only for students but also for individual faculties and the wider community. It enables teacher to engage more students in exciting new way, reaching them on their terms and via their devices, and keeping students both informed and involved. An attractive aspect of Acadima is the link it provides between different universities and the resultant openness. Individual cards can, and ought to be swapped for each other, and it is possible to collect useful cards in personalised card collections. The heterogeneous nature of the subject matter becomes clear, and new facts can be linked together. Explorative learners can move forward into previously unknown areas. The focus is on the contents. It would thus be possible to refer to Acadima as "crowd-sourcing for the crowd". Acadima is a straightforward knowledge-imparting tool for teachers and students that is fun to use. We want to tie Acadima into a wider solution, since, together; we can create a meaningful application. Winning applications have to be sufficiently unique. We can attain this goal through the uniqueness of Switzerland's university network. Since 2011, we build student expert groups for the modules of the basic studies curriculum in biology, Division of Biology at University of Zurich. Students of higher semesters such as the advanced studies curriculum join these groups as reviewer. In the end we build up to 20 learning communities covering the main topics of the basic studies curriculum. Their task is to create meaningful learning and test flashcards for exam preparation. In a didactic course they learnt to design good MC questions. Other students can profit of the work of their fellow students and at the same time contribute with annotations, feedbacks and ratings. Lecturers can set 'profproofed' icons.

soa peer&collaborative assesment final

Documents