iccs field trial marking of open-ended response items hamburg, july 2007

ICCS Field TrialMarking of open-ended

response items

Hamburg, July 2007

ICCS Marker Training, HamburgJuly 2007

Purpose of Marker Training

1. To develop a common, shared understanding of the purpose of each item and the corresponding item codes for scoring student responses.

2. To confirm the procedures for setting up and running a marking operation.

– The aim for us is that after 1+n days there are shared common understandings of these two aspects

• Day 1 – confirming the scoring of the open-ended response items

• Day 2 – confirming the marking operations procedures


Why are there open-response questions?

(CIVED only MCQ)

• Open-response questions provide:– Information about broader conceptual

understandings (e.g. citizenship motivations, benefits to community)

– Information about students’ capacity to generate rather than recognise concepts (useful in CCE where correct responses can often stand-out)

• Still partly exploratory


The Open-response item scoring guides

Crucial aspects:

- the descriptions of acceptable responses

- the scored sample responses


Considerations when using the guides

• Spelling, quality of expression, grammar and punctuation are not assessed.

• HOWEVER – Civics and Citizenship Education provides an unusual set of challenges when considering the language of student responses.



• In many countries students have not had extensive formal classroom experience of CCE.

• Many, many students do not have the explicit CCE vocabulary to express conceptual understandings efficiently.

• One significant challenge in assessing written student responses is to ‘unpack’ student attempts to express conceptual understandings using their ordinary (i.e. non-technical) language.



• This challenge has two dangers that can affect marker reliability.

1. Some markers can mistakenly believe that because students do not have the language skill to express answers succinctly that all answers are correct.

2. Some markers believe that unless students use appropriate CCE language they cannot be credited with providing correct responses.



• BOTH misconceptions must be dealt with in the marker training.

• Student responses can only be scored on the basis of what they have written – markers should not try to infer student responses on the basis of what has not been written.

• To this end, the students’ choice of the words they use MUST be given careful attention.



• For example– What are two different benefits of

people discussing issue even if they may never agree with each other’s opinions? (CI2RR01)

– ‘They may learn to agree with each other’ (0)

– ‘They may learn to get along with each other’ (1)

– ‘They might end up happy’. (0)– The first and third responses are too

vague to receive credit (and the first response contradicts the stem).



• Many students provide ‘feel-good’ responses (expressions of value rather than understanding) that can be very well expressed but still without substance.

• Many students provide complex re-wordings of the question stem without answering the question – these can be deceptive as often they are eloquent.


Questions or comments?


Comments

• It doesn’t matter where the students write their responses (1, 2)

• We will introduce a ‘spare’ code for response types that are not accounted for at all in the scoring guide but are worthy of credit. We will detail the way to use this in the SOP(III). You will be asked to document how the codes work for consideration in the main study.

iccs field trial marking of open-ended response items hamburg, july 2007

Documents