quality assurance in lagre scale e-assessments

1Prof. Dr. H.-W. Wollersheim

Quality assurancein Large Scale E-Assessments

Tempus Workshop Laubusch 2016

QUALITY ASSURANCEIN LARGE SCALE E-ASSESSMENTS

Prof. Dr. Heinz-Werner Wollersheim




Content

1) Quality assurance: more than a technical problem

2) Aspects of quality assurance in MCA

3) Shortcut: important terms

4) Quality assurance process in MCA

5) Prospects




1) More than a technical problem

qualitymanagement

qualityassurance

analysiscurrent state

implementingimprovementsdocumentation

measuringeffects

1. detecting & making awareprocesses

2. detecting & fixing frictions





optimizationcircle

ofquality

development

qualityassurance

analysiscurrent state

implementingimprovementsdocumentation

measuringeffects





qualitymanagement

Goals to achieve:

Doing the right things. (effectiveness)

Doing the things in the right manner. (efficiency)

Doing the things at the right time. (efficiency)

„Our mission is to do the right things right at the right time.“





Beyond technical “controlmania“

§ technical control is working just in addition to social control (not asa substitute)

§ reaching high acceptance: university students, colleagues

§ needs lots of communication: The right things done right at the right time?

§ But: Assessing verifiable objectives is indicating a shift from holistic„education“ to specific „training“





Doing the right things: validity

kinds of validity relevance determining/ indicators

content validity best way to operationalize contents ratings by experts

construct validity Does the test measure the intendedlearning outcomes and competences?

a) convergent validity(data of tests designed to measuresimilar competences are highly correlating)

b) divergent validity(data of tests designed to measuredifferent competences are lowly correlating)

criterion validity correlation between measurementinstrument and empirical criteria

a) diagnostic validity

a) prognostic validity





Academic teaching: typical issues

§ common practice at the high schools and universities:modules and courses are designed at most

§ challenge:developing quality assured assessments in this context

§ validity reduced to content validity

§ Modified Constructive Alignment (MCA)




2) Aspects of quality assurance in MCA on

§ objectivity: implementing objectivity

objectivity of analysis

§ validity: content validity à ratings by expertsà assessement plan (blueprint)

§ reliability: internal consistency à consistency analysis




2) Aspects of quality assurance in MCA on

analysis of items:

§ empirical difficulty: index of difficulty 20 ≤ Pi ≤ 80

§ discrimination: coefficient of discrimination r ≥ .20





§ objectivity: objectivity of implementation

objectivity of analysis

§ validity: content validity à ratings by experts, à assessement plan (blueprint)

§ reliability: internal consistency à consistency analysis

§ empirical difficulty: index of difficulty à 20 ≤ Pi ≤ 80

§ discrimination: coefficient of discrimination à r ≥ .20





§ selected response

– selected response: multiple possible answers, choosing the right one(s)

– legal term (Germany): Antwort-Wahl-Verfahren

– terms that describes the wayto give an answer: multiple-choice question type (MCQ),

matching, sequence, hotspot

– assessment question: item





§ item

“The smallest separately identified question or task within an assessment plus, its associated information (for example mark scheme, curriculum reference, media content, performance information etc), usually a single objective question. Distinguished from a ‘question’, which may be a longer and less-objective task but often used synonymously.”

(Qualifications and Curriculum Authority 2007, 107)





§ selected response

“The selected-response item format is the best choicefor test developers interested in efficient, effective measurement of cognitive achievement or ability.”

(Downing 2006, 287)





§ item structureDas Bild kann derzeit nicht angezeigt werden.high quality structure

item stem

vignette and question

options

correct answerand

several incorrect answer(distractors)

more detailed stem

short stem

(A) short options (A) long options

avoid

(B)

(C)

(D)

(E)

(B)

(C)

(D)

(E)





§ constructive alignment: the german view

Ø „Beim 'Constructive-Alignment'-Konzept geht es im Kern darum, dassdie intendierten Outcomes des Lernprozesses klar definiert und denStudierenden explizit verdeutlicht werden und die Prüfungs- undLernaktivitäten stringent auf die Learning Outcomes abgestimmtwerden.“

(Schaper 2012, 62)

à In nuce C.A. means coherence betweenlearning outcomes,assessment andlearning process





§ constructive alignment

“In constructive alignment, we start with the outcomes we intend students to learn, and alignteaching and assessment to those outcomes. The outcome statements contain a learning activity, a verb, that students need to perform to best achieve the outcome, such as “apply expectancy-value theory of motivation”, or “explain the concept of … “. That verb says what the relevant learning activities are that the students need to undertake in order to attain the intended learning outcome. Learning is constructed by what activities the students carry out; learning is about what they do, not about what we teachers do. Likewise, assessment is about how well they achieve the intended outcomes, not about how well they report back to us what we have told them. […] Constructive alignment can be used for individual courses, for degree programmes, and at the institutional level, for aligning all teaching to graduate attributes.”

Source: http://www.johnbiggs.com.au/academic/constructive-alignment/





§ learning outcomes

Ø “Learning outcomes describe what a learner is expected to know,understand and be able to do after successful completion of a processof learning.”

(ECTS Users‘ Guide, Europäische Gemeinschaft 2009, 11)





§ focus:





2 dimensions of quality assurance

pla

nnin

g m

odel

for

did

act

ics

in h

ighe

red

uca

tion

constructive alignment

analysis evaluation

process stages

development implementation





Planning stages according to constructive alignment

learning outcomes§ defining intended learning outcomes

before starting the course§ align teaching and assessment to those

outcomes

designing tests§ developing assessment according to

intended learning outcomes § adapting those outcomes if necessary

designing learning process§ designing learning process according to

intended learning outcomes after developing assessment

1

2

3

learningoutcomes

learning process and

learning activities

assessment

1

2

3





Workflow: Creating e-assessements according to constructive alignment

Das Bild kann derzeit nicht angezeigt werden.

plan

ning

leve

ls

modulcourse (seperate course unit)

modul

workload/ ECTS

contentstrucutredesign

of the course

topi

c of

a c

ours

e

indexing of topics and

contents

previous knowledge

competence descriptions

DQR/ HQR

issues to be considered

lear

ning

out

com

es th

atar

ere

lava

ntfo

ras

sess

men

ts(if

nece

ssar

y)

form

atan

d m

etho

dof

asse

ssm

ent

asse

ssm

ent p

lan

(blu

eprin

t) of

targ

etst

ate

(con

tent

s, p

erfo

rman

cele

vels

, ty

pes

ofte

stite

ms)

blue

prin

tof

cur

rent

stat

e

peer

-revi

ew p

roce

ss

crea

ting

item

s

crea

ting

test

item

poo

ls

designing tests compiling tests

crea

ting

test

s

plan

ning

stag

es

topi

car

eas

spec

ific

cont

ents

lear

ning

out

com

es





2 dimensions of quality assurance

pla

nnin

g m

odel

for

did

act

ics

in h

ighe

red

uca

tion

constructive alignment

analysis evaluation

process stages

development implementation

specific implementation?





simplified process model for e-assessment

developingan e-exam

implementingan e-exam

adaptingan e-exam

analysing and evaluatingan e-exam




developingan e-exam


adaptingan e-exam


§ constructive alignment (Biggs and Tang 2007)§ taxonomy for learning, teaching, and assessing

(Anderson and Krathwohl 2001)§ learning outcomes § assessment plan (blueprint)§ scoring model§ content and form of test items§ peer-review

analysis of tests and items§ empirical difficulty§ discrimination§ reliability

renormalization

§ practice e-exam session§ instructions for working

with test items§ risk management

2 levels:§ content criteria for organizing the

course and developing the tests§ ideas for innovating the electronical

assessments system




§ knowledge of constructive alignment (Biggs and Tang 2007) and the taxonomy for learning, teaching, and assessing (Anderson and Krathwohl 2001)

§ knowledge of the form and content of correct test items

§ different work assistance tools to organize the developing process§ defining learning outcomes for courses§ designing the assessment plan (blueprint):

number of items, performance levels, forms of knowledge, topics; ensuring one-dimensionality

§ calculating guessing probability

§ organizing the item developing (work assistance tools)§ organizing the peer review process § creating tests:

selecting items according to the assessment plan (blueprint)equal opportunities in case of more than one test in course

Entwicklung der E-Klausur

developingan e-exam





§ training version of the e-exam session: university students get to know about testprocedure (e.g. navigation) and kinds of item types and designs to become familiar withundertaking e-exams

§ instructions for working with test items

§ transparency of scoring model, e.g. getting marks according to all-or-nothing-principle orfor each part of the right answer

§ risk management: controling examination server, back-up server, log files, bug documentation




analysing andevaluating an e-Exam

analysis of items and test:

item level:§ empirical difficulty: index of difficulty 20 ≤ ≤ 80 § selectivity (corrected item-total discrimination):

coefficient of item discrimination r ≥ .20

test level:§ reliability: internal consistency à consistency analysis .80 ≤ KR 20 ≤ .90

renormalization





simplified process model for e-assessment

developingan e-exam


adaptingan e-exam





5) Prospects




5) Prospects

objectivity

validity

reliability

empirical difficulty

selectivity / item-total discrimination

security against miscalculation

scaling

dealing with guessing of university students

dealing with different valuate modes in item analysis

quality criterion/ quality feature actuallysolved

implementedin

ILIAS 4.3.7




5) Prospects

§ descriptive statistics + inferential statistics are necessary toverification the one-dimensionality of the items

„Die einzig angemessene Umgehensweise mit dem Problem des Ratens ist der Einsatz ganz spezieller Methoden der IRT, die bei der Schätzung des jeweils gesuchten Fähigkeitsparameters einer Person aufgabenspezifisch das faktische Erfolgsausmaß beim Versuch des Lösungerratens mit ein-kalkulieren (d.s. insbesondere das 3-PL Modell und das Difficulty plus Guessing PL Modell; vgl. wieder Kubinger, 2009).“

(Kubinger 2014, 170)

§ Beispiel: Verrechnungsregel im Rahmen der Skalierung(z.B. Anzahl gelöster Aufgaben)à sollte mit Hilfe der Item-Response-Theorie auf empirische Angemessenheit geprüft werden




5) Prospects

dealing with guessing – proposal for solution I:

§ guessing probability included in the calculation of scoring model

à issue: different behaviour of university studentsin relation to guessing




5) Prospects

dealing with guessing – proposal for solution II:

§ increasing the number of distracors§ “1 of 6“, “1 of 7“, “1 of 8“§ “x of 5“, scoring model with all-or-nothing-principle

à issues:1. valid assessment required appropriate distracors

(e.g. plausible, homogeneous)à analysing distractor quality

2. scoring model with all-or-nothing-principle (applied to “x of 5“) assumes that partly knowledge is not enough

(cf. Kubinger 2014)




5) Prospects

minimum goals to reach validity

creating workflow tomanaging the working process, observing the standards,improving the usability




literature reference

Amtsblatt der Europäische Union (AblEU) Nr. 2008/C 111/01 v. 6.5.2008.Source: http://eur-lex.europa.eu/LexUriServ/LexUri-Serv.do?uri=OJ:C:2008:111:0001:0007:DE:PDF

Downing, Steven M. (2006): Selected-Response Item Formats in Test Development. In: Downing, Steven M. / Haladyna, Thomas M.: Handbook of Test Development. Mahwah, N.J., S. 287-301.

Europäische Gemeinschaft (2009): ECTS-Leitfaden. Source: http://ec.europa.eu/education/tools/docs/ects-guide_de.pdf

Fisseni, H.-J. (1990): Lehrbuch der psychologischen Diagnostik. Göttingen: Hogrefe.

Kubinger, K. D. (2009): Psychologische Diagnostik - Theorie und Praxis psychologischen Diagnostizierens. Göttingen: Hogrefe.

Kubinger, K. D. (2014): Kubinger, Klaus D. (2014): Gutachten zur Erstellung „gerichtsfester” Multiple-Choice-Prüfungsaufgaben. In: Psychologische Rundschau 65 (3), S. 169–178.

Lienert, G. A. & Raatz, U. (1998): Testaufbau und Testanalyse. Weinheim: Beltz PVU.

Schaper, N. (2012): Fachgutachten zur Kompetenzorientierung in Studium und Lehre. Source: http://www.hrk-nexus.de/fi leadmin/redaktion/hrk-nexus/07-Downloads/07-02-Publikationen/fachgutachten_kompetenzorientierung.pdf

Qualifications and Curriculum Authority (2007): e-Assessment. Guide to effective practice.Soruce: http://www.e-assessment.com/wp-content/uploads/2014/08/e-assessment_-_guide_to_effective_practice_full_version.pdf




THANK YOU FOR LISTENING.

Prof. Dr. Heinz-Werner WollersheimLeipzig University

Institute of Educational SciencesChair for General Pedagogy

[email protected]