victoria crisp & christine ward the pepcaa project: formative scenario-based caa in psychology...

Victoria Crisp & Christine Ward

The PePCAA Project:

Formative Scenario-based CAA in

Psychology for Teachers

Crisp & Ward: The PePCAA project

Topics

• The PePCAA project and its aims

• Scenarios: development, types and CAA features

• Examples of scenarios

• Results of trialling

• Problems in sharing scenarios transnationally

• Achievements.


PePCAA – Pedagogical Psychology Computer Assisted Assessment

• FIM-NeuesLernen, University of Erlangen

• LabSET, University of Liege

• Universitat Oberta de Cataluña (OU of Catalonia)

• PGU, Göteborg University

• UCLES - University of Cambridge Local Examinations Syndicate


PePCAA

Scenario-based computer-delivered formative assessments in pedagogical psychology for teachers and trainee teachers.

Main aims:

• to improve the quality of teaching, by linking psychological theory to practical instances of its use in the classroom

• to explore more innovative techniques of computer assisted assessment


Expected benefits of the approach (1):

• Intrinsic interest of scenarios

• Showing relevance of psychological theory

• Increasing trainee awareness of psychological theory and research

• Developing awareness of why some strategies are more effective

• Stimulating discussion.


Expected benefits of the approach (2):

• Allowing trainees to assess own knowledge

• Motivational benefits of CAA

• Opportunity to experiment with CAA.


Scenario development

• Early discussion and drafting of examples

• Drafting by one partner

• Comment by at least one other partner

• Revision and trialling

• Development of PLAC (PePCAA Learning Assessment Circle) .


Scenario types:

• Problem Solving: diagnose causes of a classroom difficulty, suggest possible solutions and their theoretical basis, predict effects of proposed intervention.

• Best Practice: plan the best strategy to achieve a goal, justify the strategy by reference to theory, predict the outcomes .


Metacognition and confidence ratings

• Promote awareness of one’s own knowledge

• Confidence rating for each automatically scored answer (in some assessments)

• Calculation of Confidence, Prudence and Nuance

• Required a work-around in Perception.


Some examples of scenario assessments

http://www.pepcaa.odl.org/







Scenario / software features (1):

• a scenario (or ‘case study’) description

• option to read a short explanation of relevant psychological ideas first

• ‘essay’ question, followed by feedback stating the main points expected

• ‘essay’ question followed by a multiple-response question asking which points included, with feedback on choices made


Scenario / software features (2):

• range of objective question types

• division of some assessments into blocks

• ‘branching’ dependent upon the candidate’s response to a question

• confidence measurement

• explanatory feedback, with weblinks


Trialling in the UK

• Small groups of trainee teachers

• Scenarios from both UK and non-UK partners

• Feedback collected through both open and closed questions

• Some follow up in interviews.


Trialling – format of feedback

Statement (eg ‘The assessment is easy to understand’) to be rated as follows:

+2 strongly agree

+1 agree

0 neither agree or disagree

-1 disagree

-2 strongly disagree


Scenarios trialled in the UK - primary

2: PE lesson with 10 year olds (6 students)

3: Number facts (6)

4: Assessing Science (6)

6: Problem Solving (7)

7: Attachment Theory and Parenting (6)

8: Friendships (6)

9: Play (6)

PGU 7 (Sweden): Teaching conceptual change . (6)


Trialling of primary scenarios in UK –Average ratings (scale – 2.0 to + 2.0)

Knowledge of pedagogical psychology is very important to me 0.64

My current knowledge of pedagogical psychology is very good - 0.39

Assessment is easy to understand 0.88

Assessment is about a real world situation 1.22


Trialling of primary scenarios in UK –Average ratings (scale – 2.0 to + 2.0)

Assessment is relevant for teachers 1.36 Assessment is well structured 0.88

Assessment is too long - 0.33

Assessment is stimulating 0.87

Working on assessment was helpful 0.95

Feedback given was helpful 0.97


Trialling of primary scenarios in UK –Key findings

• Generally favourable responses to all scenarios

• Ratings for Swedish scenario similar to those for UK

• One scenario (UCLES 7) shown to be too long

• Amendments made in the light of responses


Scenarios trialled in the UK - secondary

1: History Lesson (7 students)

5: Dance lesson with 11 year olds (7)

Labset 2 (Belgium): Why success isn’t explained like failure (7)

FIM 8 (Germany): Self-managed learning (7)


Trialling of secondary scenarios in UK –Average ratings (scale – 2.0 to + 2.0)

Knowledge of pedagogical psychology is very important to me 0.46

My current knowledge of pedagogical psychology is very good - 0.46

Assessment is easy to understand 0.04

Assessment is about a real world situation 0.54


Trialling of secondary scenarios in UK –Average ratings (scale – 2.0 to + 2.0)

Assessment is relevant for teachers 0.54 Assessment is well structured 0.14

Assessment is too long - 0.43

Assessment is stimulating - 0.04

Working on assessment was helpful 0.04

Feedback given was helpful - 0.21


Trialling of secondary scenarios in UK: Key findings

• Average ratings much lower than for primary scenarios

• Ratings for Belgian and German scenarios much lower than those for UK

• Amendments made to UK scenarios in the light of responses.


Trialling of scenarios in UK: Some general findings

• Mixed response to open-response questions not automatically marked

• Mixed response to multiple-response questions

• Improvements required to candidate information and end of test feedback

• Some students found the psychological language difficult.


Problems in transnational sharing of scenarios

• Mismatch in the coverage of psychology

• Depth of treatment and detail differed, even where topics were common

• More practical approach in the UK & Sweden

• Assumptions about the background to the scenario might not fit another country

• Careful editing of language needed.


Achievements of the PePCAA project

• A framework for scenario development, plus actual examples

• PePCAA Learning Assessment Circle (PLAC)

• Showed benefits of scenarios in motivating students and prompting reflection

• Demonstrated taking CAA beyond simple multiple-choice.

Potential applications outside pedagogical psychology


Try the assessments for yourself on







victoria crisp & christine ward the pepcaa project: formative scenario-based caa in psychology...

Documents

pepcaa project crisp

pepcaa projectcrisp

assessment crisp ward

weblinks crisp ward

pepcaa projectmetacognition

pepcaa projecttrialling

teacherscrisp ward

pepcaa projectscenario