challenges of feedback in chilean universities...challenges of feedback in chilean universities dr....
TRANSCRIPT
CHALLENGES OF FEEDBACK IN CHILEAN UNIVERSITIES
Dr. Daniela Bruna J.
Centro de investigación y Mejoramiento de la Educación
Faculty of Psychology Universidad del Desarrollo
25th of November, 2020 International Virtual Meeting in Teaching, Learning and Assessment
Why focus on learning feedback?
• Backwash effect of assessment and feedback for learning
• Permanent feedback allows students to correct and improve their performance
• Promotes the development of critical judgment
• Understanding of the quality of performance transfer of competences to other contexts, especially work.
• It allows a more autonomous, conscious and metacognitive work, together with greater motivation and perception of self-efficacy
Feedback that is the engine behind learning ...
Enables to expertly judge what is a good performance on the
- It is done before and after
- Exemplars are used
- It involves self and co-assessment processes.
- Expected performances are modeled.
It allows to determine how current performance is related to the expected
- Use of formative, process assessment
- There are opportunities to show that they have incorporated the feedback received.
- Inclusion of guidelines and rubrics, with co-constructed or negotiated criteria
Drives action to reduce the gap between real and ideal performance
It focuses on highlighting strengths to help students set goals,
select strategies, feel self-effective, and motivated.
It also describes weaknesses, allowing students to specifically
identify improvement areas
Provides directive advice on what to do to improve, illuminating the
way forward.
Feedback that is the engine behind learning ...
Feedback for Learning: Closing the Assessment Loop
(Henderson, Boud, Molloyd, Dawson, Phillips, Ryan Walker, Mahoy 2018)
Participants
• Monash University, Deakin University, The University of Melbourne
Aim/Goal
• Our goal was to deliver an empirically based study of feedback designs and conditions to guide educators.
First Phase
• Sought to identify feedback practices and experiences
• Large-scale survey (4514 students and 406 teachers)
http://newmediaresearch.educ.monash.edu.au/feedback/about/about-the-project/
The Chilean Study
• Translation and adaptation of the survey.
• Application to students and teachers of 6 universities, private and traditional in the O'Higgins, Metropolitan and Bio-Bío Region.
• Evaluation of the feedback received in the second semester of 2018.
• Data collection, from October to March 2019.
• Survey applied in online format.
• Voluntary participation of 537 students and 212 teachers.
What did we investigate in this students? Dimensions items
Assessment System Type of assessment Type of the items Perception of difficulty of the items
Feedback System: Relationship with the teacher
Confidence in the teaching experience Teacher knowledge of each student Comfort to dialogue with the teacher
Feedback System: Characteristics
Understanding feedback Comments detail Relationship of the messages with the assessment instrument Inclusion of positive comments Motivating quality Comment customization
Feedback System: Utility
Specific feedback on performance Help for the improvement on future performances Guide for selecting strategies
Results Sociodemographic Variables
31,5
68,5
0 10 20 30 40 50 60 70 80
Masculino
Femenino
Gender
41,6
34,4
24
0 5 10 15 20 25 30 35 40 45
Bachillerato
Licenciatura
Título
Level
Female Male
Degree Bachelor Undergraduate
26,3
26,9
27,2
14,4
5,2
0 5 10 15 20 25 30
Salud
Educación
Ciencias Sociales
Ingenierías
Otras
Discipline
Results Sociodemographic Variables
Others Engineering Social Sciences Education Health
Results Assessment System
18,7
36 37,1
14,2
12,1
9,5
19,2
Ensayo escrito Presentaciónoral
Pruebasescrita
Proyecto Test escrito laboratorios Trabajo grupal
TYPE OF INSTRUMENTS
49,2
14,8
36
TYPE OF INSTRUMENT
Pruebas escritas TBD orales TBD escritas
Written Essay Oral presentation Written exam Project Written Test Labs Group Work
Written Exams Orals Written tests
8,7
19,1 21,7
38,1
43,1
73,1 73,1
84
0
10
20
30
40
50
60
70
80
90
Términos Pareados Completación Verdadero y Falso Opción Múltiple Análisis de caso Desarrollo Breve Resolución deproblemas
Desarrollo Extenso
TYPE OF ITEMS
Results Assessment System
Paired Completion True or False Multiple option Case Analysis Brief Development Problem Solving Extensive development Terms
Assessment System Perception of items’ difficulty
33,5
66,5
23,6
76,4
22,1
77,9
12,7
87,3
53,5
66,5
73,8
26,2
72,8
27,2
63,3
36,7
0
10
20
30
40
50
60
70
80
90
100
DIFFICULTY PERCEPTION ACCORDING TO ITEM
Difficult Easy Difficult Easy Difficult Easy Difficult Easy Difficult Easy Difficult Easy Difficult Easy Difficult Easy
Multiple True or False Completion Paired Terns Brief Extensive Case Study Problem Solving Option Development Development
Results for feedback system Relationship with the teacher
10,9 4,9
84,1
0
20
40
60
80
100
En desacuerdo Neutro De acuerdo
TRUST ON THE TEACHER’S EXPERIENCE
27,4
19,8
52,8
0
10
20
30
40
50
60
En desacuerdo Neutro De acuerdo
KNOWLEDGE OF TEACHER OF EACH STUDENT
17,5 12,1
70,4
0
20
40
60
80
En desacuerdo Neutro De acuerdo
COMFORT TO DIALOGUE WITH THE TEACHER
In disagreement Neutral Agree
In disagreement Neutral Agree
In disagreement Neutral Agree
Results for feedback system Characteristics
11,3
47,3
54,1 55,5
67,9 68,5 69,1
81,6
0
10
20
30
40
50
60
70
80
90
Sentimiento negativos Comentarios positivos Relación con laevaluación
Tiempo de dedicación Calidad motiovadora Detalle Personalización Comprensión
Feedback Characteristics
Negative feelings Positive Comments Relation with assessment Time dedicated Motivating quality Detail Personalization Understanding
57,5
72,1 76
80,4 82,9
87,2
0
10
20
30
40
50
60
70
80
90
100
Evaluación formativa Guía para la selección estrategias Comentarios específicos sobre eldesempeño
Utilización en el futuro Influencia en trabajosposteriores
Ayuda para la mejora dedesempeños futuros
Feedback Utility
Results for feedback system Characteristics
Formative Assessment Guide for selecting strategies Specific comments of performance Use of future Influence in future tasks Help for enhancing future performance
How is it explained a useful perception of feedback?
Predictors Coef. EE
Gender (Ref: Masculino) -0.009 0.308
Age (Ref: 17-19)
20-24 -0.591 0.469
>= 25 -0.784 0.522
Level (Ref: Undergraduate)
Bachelor 0.181 0.333
Degree 0.458 0.383
Discipline (Ref: Health) Education -0.297 0.380
Social Sciences 0.219 0.364
Degree -0.036 0.424
Given information (Ref: Only Comments)
Only grades 1.051 0.540
Grades and comments 1.358* 0.381
Level of understanding (Ref: No) 2.584* 0.290
Int. -0.633 0.571 *: p < 0.001
Model obtained by Ordinal Regression VD: Feedback Utility
Feedback is perceived as useful when, in addition to delivering a grade, it is accompanied by oral and written comments, and these are deeply understood, independent of the other variables
Predictors Coef. EE
Gender(Ref: Femenino) -0.017 0.232
Age (Ref: 17-19) 20-24 -0.293 0.356
>= 25 0.088 0.402
Nivel (Ref: Undergraduate) Bachelor 0.165 0.262
Degree -0.282 0.275
Discipline (Ref: Health) Education -0.165 0.292
Social Sciences -0.718* 0.273
Engeneering -0.341 0.335
Comfort of dialogue (Ref: No) 1.140** 0.300
Quality of the relationship(Ref: No)
0.069 0.306
Intercept 0.495 0.419 *: p < 0.01; **: p < 0.001
Feedback motivates less students of careers belonging to Social Sciences. Feedback motivates when students feel comfortable talking with their teachers.
How is it explained a motivating perception of feedback?
Model obtained by Ordinal Regression VD: Feedback Utility
Discussion
• Improve the quality of written tests. Challenge 1
• Reduce closed answer questions, eliminating some from the higher education system. Challenge 2
• Provide institutional conditions for the design and correction of this type of instrument. Challenge 3
• Generate mechanisms so that students perceive that we know them and that feedback is individualized.
Challenge 4
• Maintain the good feedback climate. Especially in the current context of remote education and pandemic Challenge 5
• Help students who do not feel good weather. Although they are fewer, they are the ones that most require support. Challenge 6
Discussion
• Improve feedback characteristics: time, positive comments and relationship with the assessment Challenge 7
• Increase and improve formative assessment strategies. Challenge 8
• Worry about delivering meaningful comments, helping students to "go beyond the grade." Challenge 9
Discussion
• Dialogic feedback for a deeper understanding Challenge 1o
• What happens with the Social Sciences students? Challenge 11
• How has this process with the pandemic digital transformation? Challenge 12
• How is our reality like with in relation with the Australian Universities? Challenge 13
Discussion
References
Boud, D. & Falchikov, N. (2006). Alligning assessment with long-term learning. Assessment and Evaluation in Higher Education, 31(4), 399-413.
Boud, D. & Molloy, E. (2013). Rethinking models of feedback for learning: the challenge of design. Assessment & Evaluation in Higher Education, 38(6), 698-712.
Loureiro, M., Pombo, L. & Moreira, A. (2012). The quality of peer assesment in a wiki-based online context: a qualitative study. Educational Media International, 49(2), 139-149.
Nicol, D. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning, a model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218.
Panadero, E., Brown, G. & Strijbos, J. W. (2016). The Future of Student Self-Assessment: a Review of Known Unknowns and Potential Directions. Educational, Psychology Review, 28(4), 803- 830.
Sadler, R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144.
Sadler, R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment & Evaluation in Higher Education, 30(2), 175-194.