Tutors: Who Needs them?Tutors: Who Needs them?

Self assessment using Grade Related Self assessment using Grade Related CriteriaCriteria

http://www.rgu.ac.uk/celt/learning/http://www.rgu.ac.uk/celt/learning/page.cfm?pge=7347#Tutorpage.cfm?pge=7347#Tutor

Aims of this SessionAims of this Session

To Review the literature on Self To Review the literature on Self assessmentassessment

To outline the research design of an To outline the research design of an “experiment” with final honours year “experiment” with final honours year studentsstudents

To review the resultsTo review the results To discuss the implications of the research To discuss the implications of the research

for assessmentfor assessment

Aims of AssessmentAims of Assessment

Assessment has traditionally four main roles: Assessment has traditionally four main roles: FormativeFormative, to provide support for future learning; , to provide support for future learning; SummativeSummative,, to provide information about to provide information about

performance at the end of a course; performance at the end of a course; CertificationCertification, selecting by means of qualification , selecting by means of qualification

andand EvaluativeEvaluative,, a means by which stakeholders can a means by which stakeholders can

judge the effectiveness of the system as a judge the effectiveness of the system as a whole.whole.

Role of AssessmentRole of Assessment

Assessment as an aid to learning?Assessment as an aid to learning? Assessment as a means of identifying Assessment as a means of identifying

ways of improving?ways of improving? Assessment as a skill to be acquired by Assessment as a skill to be acquired by

students?students?

Aims of Self assessmentAims of Self assessment

An aid to student learningAn aid to student learning To help students understand more clearly To help students understand more clearly

the basis on which they are assessedthe basis on which they are assessed To develop self assessment skillsTo develop self assessment skills To encourage students to be more self To encourage students to be more self

critical about their workcritical about their work To give students more effective feedbackTo give students more effective feedback To improve the efficiency of the To improve the efficiency of the

assessment processassessment process

The ContextThe Context Module Economics of Taxation and Corporate Module Economics of Taxation and Corporate

Taxes (BS4214)Taxes (BS4214) Part of Accounting Module on taxationPart of Accounting Module on taxation Accounting students used to assessments which Accounting students used to assessments which

involve a known or correct answerinvolve a known or correct answer How do you assess an economics answer?How do you assess an economics answer? Session 2002/3Session 2002/3 48 final year accounting honours students48 final year accounting honours students Using the Common Grading scheme with grade Using the Common Grading scheme with grade

related criteriarelated criteria If GRC Scheme is as transparent as it claims If GRC Scheme is as transparent as it claims

then students should be able to use it to assess then students should be able to use it to assess themselves accurately themselves accurately

Literature ReviewLiterature Review

How Reliable is Self Assessment?How Reliable is Self Assessment? What factors influence Self Assessment?What factors influence Self Assessment?

Literature ReviewLiterature Review

Mabe and West (1982)Mabe and West (1982) Review of 55 studies from 1942-1977 Review of 55 studies from 1942-1977

involving 267 correlations involving 267 correlations Findings:Findings: Poor relationship between students and Poor relationship between students and

tutor ratingstutor ratings R= 0.29 SD 0.25 (high degree of R= 0.29 SD 0.25 (high degree of

variability)variability)

Literature ReviewLiterature Review Boud and Falchikov (1995)Boud and Falchikov (1995) Total of 68 studies from 1932 -1994Total of 68 studies from 1932 -1994 Do students over-rate themselves? ( 17 studies)Do students over-rate themselves? ( 17 studies) Good students better at rating themselves than Good students better at rating themselves than

bad students? (11 studies)bad students? (11 studies) Advanced PG students better than “freshmen” Advanced PG students better than “freshmen”

(7 studies found PG better at self assessment)(7 studies found PG better at self assessment) More practice made self assessment better? More practice made self assessment better?

(7 studies found that it did not!)(7 studies found that it did not!) Gender differences? (6 studies – 3 showed Gender differences? (6 studies – 3 showed

women more accurate than men; the rest no women more accurate than men; the rest no evidence)evidence)

Literature ReviewLiterature Review Larres, Ballantine and Whittington (2003) Self Larres, Ballantine and Whittington (2003) Self

Assessment with Accounting StudentsAssessment with Accounting Students Computer Literacy in two UK universities with Computer Literacy in two UK universities with

sample of accounting studentssample of accounting students ““Vast majority”Vast majority” over-estimated their computer over-estimated their computer

knowledgeknowledge Conclusion: Conclusion: ““Self Assessment is not an appropriate means of Self Assessment is not an appropriate means of

determining computer literacy”determining computer literacy” But it did provide: But it did provide: “a useful adjunct into students’ “a useful adjunct into students’

attitudes to computing and stimulated reflection attitudes to computing and stimulated reflection on their abilities.”on their abilities.”

Literature ReviewLiterature Review

Self Assessment as an aid to learningSelf Assessment as an aid to learning

Fitzgerald (1997) Found significant Fitzgerald (1997) Found significant improvements in learning amongst improvements in learning amongst medical students when self assessment medical students when self assessment introduced introduced

Literature ReviewLiterature Review Self Assessment as an aid to learningSelf Assessment as an aid to learning Rust (2003)Rust (2003) Developed the use of grade related criteria with Developed the use of grade related criteria with

a group of 290 second year undergraduate a group of 290 second year undergraduate students students

140 attended a workshop in which they used the 140 attended a workshop in which they used the criteria to assess work by students from previous criteria to assess work by students from previous years; 150 did not attend: Experimental v years; 150 did not attend: Experimental v control;control;

Performance monitored and samples controlled Performance monitored and samples controlled for ability for ability

Results: significant gains both short run and Results: significant gains both short run and long-runlong-run

MethodologyMethodology Set three topics for students to chooseSet three topics for students to choose Set up iNET discussion forum using Salmons Five stage Set up iNET discussion forum using Salmons Five stage

model Salmon (2003)model Salmon (2003) Gave out the criteria and ran a one hour workshop on Gave out the criteria and ran a one hour workshop on

what was meant be each of the criteriawhat was meant be each of the criteria Conducted an online Q & A sessionConducted an online Q & A session Students handed in coursework and completed a Self Students handed in coursework and completed a Self

assessment proforma using the same criteria as the assessment proforma using the same criteria as the tutor.tutor.

Students completed an evaluation of the work. Ways in Students completed an evaluation of the work. Ways in which it could be improved. Analysis of self evaluation which it could be improved. Analysis of self evaluation feedback + comments on iNET(the qualitative data)feedback + comments on iNET(the qualitative data)

Proformas submitted but not read by tutor-assessor Proformas submitted but not read by tutor-assessor Coursework assessed internally, double markedCoursework assessed internally, double marked Comparisons made student v tutors assessmentComparisons made student v tutors assessment

CriteriaCriteria

PresentationPresentation 10%10% ResearchResearch 10%10% Knowledge and UnderstandingKnowledge and Understanding 20%20% AnalysisAnalysis 30%30% EvaluationEvaluation 30%30%

Overall Grade obtained by “averaging” + Overall Grade obtained by “averaging” + profile of gradeprofile of grade

Accuracy of Self AssessmentAccuracy of Self Assessment

Actual GradesActual Grades Degree of MatchDegree of Match Combinations of bothCombinations of both

Tests usedTests used CorrelationsCorrelations Kruskall Wallis, Mann WhitneyKruskall Wallis, Mann Whitney Pearson Chi SquarePearson Chi Square

ResultsResultsVariable Min Max Mean Median St Dev

TUTOR Grades

Presentation 4 6 5.84 6 0.426

Research 2 6 5.31 6 1.045

Knowledge and Understanding 3 6 5.37 6 0.883

Analysis 3 6 4.96 5 0.841

Evaluation 2 6 4.10 4 0.984

Overall grade 3 6 4.94 5 0.827

STUDENT Grades

Presentation 4 6 5.45 6 0.709

Research 3 6 5.04 5 0.865

Knowledge and Understanding 3 6 4.71 5 0.645

Analysis 3 6 4.41 4 0.674

Evaluation 3 6 4.37 4 0.636

Overall grade 3 6 4.61 5 0.571

ResultsResults

Dimension Correlation between tutor grade and student grade

Presentation 0.179 ns

Research 0.470 ***

Knowledge 0.444 ***

Analysis 0.471 ***

Evaluation 0.438 ***

OVERALL GRADE(Whole integer data)

0.611 ***

ResultsResults

Element Same Grade

One Grade Variance

Two Grade Variance

Students Grade Exceeds the Tutor Grade

Presentation 59.2 30.6 10.2 6.1

Research 42.9 46.9 10.2 16.3

Knowledge and Understanding

20.4 69.4 10.2 12.2

Analysis 38.8 51.0 10.2 8.2

Evaluation 36.7 55.1 8.2 42.8

Overall 53.1 44.9 2.0 8.2

HypothesesHypotheses H1: There will be no statistically significant H1: There will be no statistically significant

differences in the degree of match in grade tutor v differences in the degree of match in grade tutor v studentstudent

H2: There will be no statistically significant gender H2: There will be no statistically significant gender differences in the degree of match between tutor and differences in the degree of match between tutor and students assessments.students assessments.

H3 There will be no statistically significant H3 There will be no statistically significant differences in the degree of match made by good differences in the degree of match made by good and poor students and the tutorand poor students and the tutor

Three definitions of a good studentThree definitions of a good student 1. Grade 5 and above (Broad)1. Grade 5 and above (Broad) 2. Grade 6 only (Narrow)2. Grade 6 only (Narrow) 3. First (Honours Narrow)3. First (Honours Narrow) 4. First + 2:1 (Honours Broad)4. First + 2:1 (Honours Broad)

Hypothesis 1Hypothesis 1

ElementElement Kruskall-WallisKruskall-Wallis Mann-WhitneyMann-Whitney

PresentationPresentation 10.087***10.087*** 3.176***3.176***

ResearchResearch 3.859***3.859*** 1.964***1.964***

Knowledge & Knowledge & UnderstandingUnderstanding

20.528***20.528*** 4.531***4.531***

AnalysisAnalysis 12.343***12.343*** 3.513***3.513***

EvaluationEvaluation 1.6111.611 1.2691.269

OverallOverall 6.013***6.013*** 2.452**2.452**

Hypothesis 2Hypothesis 2

Gender (Pearson Chi Square )

Presentation 1.584

Research 6.756 ***

Knowledge and Understanding

1.197

Analysis 0.063

Evaluation 0.023

Overall 0.000

Hypothesis 3Hypothesis 3

(1) Student Ability

(Pearson Chi Square)

(2) Student Ability

(Pearson Chi Square)

(3) Student Ability

(Pearson Chi Square)

(4) Student Ability

(Pearson Chi Square)

Presentation 2.019 0.005 0.152 1.642Research 2.069 3.679 * 1.838 9.317***Knowledge and

Understanding0.206 1.426 0.057 0.611

Analysis 2.561 6.204 *** 0.348 0.987Evaluation 0.942 0.166 0.330 0.234Overall 1.181 12.765

***0.016 0.023

Conclusions on Quantitative DataConclusions on Quantitative Data

There were statistically significant differences There were statistically significant differences between the grades by the tutor and grades by between the grades by the tutor and grades by students. Students rated themselves significantly students. Students rated themselves significantly below the tutor on all dimensions except below the tutor on all dimensions except “Evaluation”“Evaluation”

There were no significant gender differences on There were no significant gender differences on any of the dimensions with the exception of any of the dimensions with the exception of “Research” where female students underscored “Research” where female students underscored themselves on this in comparison to male themselves on this in comparison to male studentsstudents

No strong evidence that the “best” students No strong evidence that the “best” students rated themselves better than weaker studentsrated themselves better than weaker students

Qualitative EvidenceQualitative Evidence

Used quotes from the self evaluation form Used quotes from the self evaluation form + iNET discussion forums to ascertain + iNET discussion forums to ascertain whether students were more aware of the whether students were more aware of the criteria against which they were assessedcriteria against which they were assessed

Did a content analysis of responsesDid a content analysis of responses

Qualitative EvidenceQualitative Evidence “ “ I feel I have enhanced my ability to perform I feel I have enhanced my ability to perform

researchresearch and critical and critical analysis analysis through this through this assignment”assignment” Female student Grade 4Female student Grade 4

“ “ I feel the strengths of this report was (sic) the I feel the strengths of this report was (sic) the researchresearch conducted as well as the conducted as well as the knowledgeknowledge and and understandingunderstanding I gained from this” I gained from this” Female student Female student overall grade 5overall grade 5

““As a result of this work I have learned that tax As a result of this work I have learned that tax can be interesting!. The strengths of this work is can be interesting!. The strengths of this work is (sic) in its (sic) in its presentation, application of knowledge presentation, application of knowledge and analysisand analysis of the issues identified” of the issues identified” Male student Male student overall grade 4overall grade 4

Feedback on the exerciseFeedback on the exercise

“ “ Thank you! I cant believe I got that mark ( I’m Thank you! I cant believe I got that mark ( I’m still shaking!) I honestly did think this was one of still shaking!) I honestly did think this was one of my poorer pieces of coursework, but I’m very my poorer pieces of coursework, but I’m very glad you didn’t agree! Thanks also for such a glad you didn’t agree! Thanks also for such a detailed feedback, it’s not often we get this and I detailed feedback, it’s not often we get this and I found it very useful” found it very useful” Female student Grade 6Female student Grade 6

Quote from PaperQuote from Paper Biggest mismatches on Research and Biggest mismatches on Research and

EvaluationEvaluation

ConclusionsConclusions Using a university–wide grade related criteria Using a university–wide grade related criteria

scheme improves the accuracy of self scheme improves the accuracy of self assessmentassessment

Self Assessment:Self Assessment: helps students “unpack” the criteria by which they are helps students “unpack” the criteria by which they are

assessedassessed improves feedback that tutors can give to studentsimproves feedback that tutors can give to students identifies criteria that need to be made clearer in the identifies criteria that need to be made clearer in the

future with more detailed briefings (or giving students future with more detailed briefings (or giving students actual coursework from previous years to practice actual coursework from previous years to practice assessing)assessing)

has the potential to change the role of tutor from has the potential to change the role of tutor from “front-line” assessor to “ moderator” of the “front-line” assessor to “ moderator” of the assessment process.assessment process.

has the potential to improve both effectiveness and has the potential to improve both effectiveness and efficiency in assessmentefficiency in assessment