assessment and teaching of 21st century skills · assessment and teaching of 21st century skills...
TRANSCRIPT
Assessment and Teaching of 21st Century Skills
Claire Scoular
Assessment Research Centre
University of Melbourne
2nd April 2014
San Jose, Costa Rica
2
Outline
1. Background of ATC21S
2. ATC21S: The Latin American Chapter
3. Costa Rican results and outcomes
4. The future of the project in Costa Rica
4
CHANGE BEGINS WITH GLOBAL PARTNERSHIP
ATC21S is a joint effort to mobilize international, educational, governmental and business communities to bring 21st-century skills into the classroom.
5
ATC21S GOAL
To develop new assessment approaches matched to new 21C skills and to advise systems, schools, and teachers on the uses of assessment data to help students develop higher order performances
6
ATC21S PROCESS
Phase 1
Conceptualize C21 skills and
education output needs
Phase 2
Skill identification
and hypotheses
Phase 3
Task development
Phase 4
Pilot studies and trials
Phase 5
Dissemination scale and policy
formation
8
PREPARING STUDENTS TO SUCCEED IN THE 21ST CENTURY
• Changes in employment trends
• Increase in workforce by 2050
• For Latin America expected increase of 26% (Wang)
9
PREPARING STUDENTS TO SUCCEED IN THE 21ST CENTURY
e are surrounded by technology, and kids are digital natives. They go into the classroom, and they don’t see the same use of technology in their education as they might in their outside lives. If education doesn’t start to address that issue, there will be a disconnect in what kids see in their lives growing up and when they get out into the workplace, which is one of the motivations for us to join this project.”
• — Michael Timms, National Project Manager ATC21S for USA
“We are surrounded by
technology, and kids are digital
natives. They go into the
classroom, and they don’t see
the same use of technology in
their education as they might in
their outside lives. If education
doesn’t start to address that
issue, there will be a disconnect
in what kids see in their lives
growing up and when they get
out into the workplace, which is
one of the motivations for us to
join this project.”
— Michael Timms, National Project
Manager ATC21S for USA
11
PHASE 1: CONCEPTUALIZING THE SKILLS
Assemble experts
Define methods
Explore practical and technical needs in the
classroom
Create a new framework
14
SELECTED SKILLS
WAYS OF WORKING
AND THINKING
Collaborative problem
solving
TOOLS FOR WORKING AND
LIVING IN THE WORLD
ICT digital literacy
Personal and social responsibility
through social networking
15
PROCESS OF ASSESSMENT DEVELOPMENT
Formulation of tasks
Teacher view Cognitive laboratory Pilot Trial
Formulation of developmental continua
Identify the dimension/s Characterise the evidence Draft three profile levels
Definition of constructs
Collaborative Problem-solving ICT Literacy
16
COLLABORATIVE PROBLEM SOLVING: Definition
16
Collaborative problem solving as a 21st Century Skill may be seen as a combination of its component parts. These are collaboration, problem solving and the new technologies currently believed to characterise the 21st century
• Collaboration – working together with a common goal
– Communication – Cooperation
• Problem solving – Knowledge – Strategies
17
Purpose, functions and outcomes
• Why? – When different expertise, knowledge or skills are
both needed and available
• What? – Formulate and express ideas; argue and defend
– Value, review, evaluate
• Outcomes – Problem solution
– Enhanced skills and knowledge
18
COLLABORATIVE PROBLEM SOLVING FRAMEWORK
18
Collaborative problem solving
Social
Participation Perspective
taking Social
regulation
Cognitive
Task regulation
Knowledge buiilding
19
Social skills strands
Participation
Action
Interaction
Task completion
Perspective taking
Adaptive responsiveness
Audience awareness
Social regulation
Negotiation
Self evaluation
Transactive memory
Responsibility initiative
20
Cognitive skills strands
Task regulation
Resource management
Information collection
Systematicity
Ambiguity tolerance
Organisation
Goal setting
Knowledge building
Knowledge acquisition
Represents relationships
Identifies consequences
Hypothesises
26
Indicative Behaviours
Skill Behaviour in task
Interaction - Chatting with partner
Presence of chat before allowing partner to make a move
Audience awareness - Adapts contributions to increase understanding for partner
Number of ball moves attempted before stopping and waiting for partner to respond
Reponsibility initiative - Takes responsibility for progress for the group task
Communicating with partner before first half of balls have been used
Resource management Realises that balls are meant to be shared and only uses allocated half
Relationships - Identifies patterns between information
Both students coming to agreement on how their machine works
28
Indicative Behaviours
Skill Behaviour in task
Action - Active in scaffolded tasks
Student A passes B a mass
Task completion – undertaking part of task individually
Follows instructions, moves 100g to position 4
Responsiveness – responding to contributions of others
Realising that some masses cannot balance. If Student A resends 50 or 500, B returns it immediately
Systematicity - Implements possible solutions to a problem
Trying out all combinations of masses on different beam positions
Solution - Correct answer
Number of successful balances achieved (3 optimum)
29
Skill progressions
SOCIAL
LEVELS PARTICIPATION
Level 4 The student initiates and promotes interaction with their partner; sharing resources,
information and ideas.
The student perseveres in solving the task as shown by repeated attempts and/or
multiples strategies
Level 3 The student is actively participating in the task in familiar and unfamiliar contexts.
The student interacts with their partner more (i.e. responds to communication cues
and requests) and shares information and/or resources with partner.
Level 2 The student is actively participating in the task when it is familiar and/or scaffolded.
There is limited interaction with partner and this is mainly prompted by task
instructions.
Level 1 The student participates in the task but is working quite independently
Task activity is usually directed by system instructions
30
Reporting to Students: Learning Readiness
Real time reports based on empirically developed learning progressions. They will identify a point of readiness to learn for each student. This type of report can be linked to teaching interventions associated with readiness to learn as indicated by the black bar in the spine of the chart.
37
Objectives and Benefits to Costa Rica
• Validate the assessment through pilot and trials
• Results could continue the agenda and lead Latin America in 21st Century skills
• Contributing to large international project
• Gain ideas about future implementation of 21C skills
• Evaluate curriculum
• Regional dialogues with peers as well as feedback from stakeholders in the global initiative
38
ASSESSMENT OF COMPETENCIES OF THE 21ST CENTURY
Three exemplary contributions:
– Language adaptions
– Customization of ICT Literacy Tasks
– Localized website
41
Componente cognitivo
Descripciones de nivel
Nivel 5 Resolución refinada de problemas
En este nivel, los estudiantes pueden reconocer errores previos y transferir y usar ese conocimiento en las secciones más complejas de las pruebas. Reconocen qué recursos han causado errores previos y los evitan. Están desarrollando confianza en el manejo de las pruebas y esto les permite corregir o superar las respuestas/acciones de sus parejas. Los estudiantes comparten con sus parejas recursos apropiados y usan estrategias consistentes que han identificado como efectivas en pasos anteriores de la solución del problema. Todo el planeamiento y la mayoría de las estrategias implementadas se realizan dentro del contexto del equipo.
Nivel 4 Trabajo eficiente
En este nivel, los estudiantes comienzan a resolver problemas de manera colaborativa y a usar estrategias de cooperación más efectivas. En este momento pueden simplificar el problema y revisar varias veces la ejecución de secciones previas de las pruebas para determinar si la regla es transferible a las páginas o secciones siguientes. En este nivel, el intercambio de información entre estudiantes incrementa y se hace más eficiente. Su secuencia y el juicio y la error acciones requieren pocos intentos y por lo tanto se concluyó en una cantidad óptima de tiempo / intento. Las completan en un tiempo óptimo. Las acciones parecen estar bien pensadas y planeadas, y cada acción parece tener un propósito. Pueden identificar causas y efectos y usar estrategias adecuadas para lograr una vía de solución correcta. El estudiante dirige y guía a su pareja hacia acciones y recursos apropiados.
Nivel 3 Planeamiento y ejecución estratégicos
En este nivel, los estudiantes pueden completar tareas de menor dificultad de manera correcta e independiente. Son capaces de completar el último paso o sección y ayudar a su pareja en este proceso. Incrementan su trabajo juntos al planear estrategias de trabajo, establecer objetivos y compartir sus recursos. El estudiante tiende a enfocarse en los recursos relevantes y desecha otros que no fueron beneficiosos en intentos anteriores. Son capaces de llegar a respuestas de común acuerdo cuando esto es necesario. En pruebas con varias secciones el estudiante tiende a identificar estrategias y recursos exitosos y aplicar las mismas reglas repetidamente. El estudiante continúa intensificando la investigación secuencial y el comportamiento sistemático en las siguientes secciones de mayor dificultad. Esto lleva a intentos por probar hipótesis junto con la pareja, así como a un comportamiento de prueba y error cada vez más minucioso.
Nivel 2 Prueba y error sistemáticos
En este nivel, el estudiante identifica posibles causas y efectos de las acciones, muestra una comprensión inicial del objeto de la prueba y comienza a probar hipótesis y reglas. El estudiante usa pruebas secuenciales cada vez mejores o exploración sistemática para incrementar su conocimiento. Pueden tratar de cambiar las técnicas que usan para investigar o explorar la prueba. Comienzan a compartir información con sus parejas, y a identificar diferencias en los recursos y en la información que posee cada miembro de la pareja. Buscan información sobre los recursos que poseen sus parejas y proporcionan información sobre los suyos. Reorganizan la información y comienzan a distinguir entre los recursos relevantes y los irrelevantes dentro del entorno del problema.
Nivel 1 Exploración
Los estudiantes a este nivel están principalmente explorando el entorno de la prueba, haciendo click en varios recursos, a menudo al azar. Realizan intentos individuales/unilaterales de prueba y error tratando de conocer el espacio del problema. Participan individualmente en la actividad e intentan resolver el problema aparentemente mediante una estrategia no sistemática de “adivinación”. Su estrategia se limita a seguir las instrucciones proporcionadas, pero no son capaces de avanzar sin ellas. Tienden a repetir errores o acciones no provechosas sin clara señal de avance en la prueba tras varios intentos. Pueden interactuar con su pareja para discutir sobre el espacio del problema de una manera descriptiva o para obtener más información.
LANGUAGE ADAPTIONS
45
Trialing the assessment tools
Trials 2011 (data collection CPS)
16 elementary and high schools
90 11-year-old students, 222 13-year-olds, and 188 15-year-olds
Trials 2013 (ICT and system testing)
9 high schools
776 students
47
ALL COUNTRIES COMBINED
Valid CPS construct
Item Separation reliability 0.999
Person separation reliability 0.814
Regression variable -0.182
48
ALL COUNTRIES COMBINED
Two separate dimensions (social & cognitive)
Social Cognitive
Separation reliability 0.998
Person separation reliability
0.754 0.770
Mean Latent Ability 0.560 -0.824
Correlation between dimensions
0.788
49
Summary of data in Costa Rica
• 575 students were scored
• Sample only not representative
• Focus is to validate the test not to compare countries
50
COUNTRY COMPARISON
CostaRica AllotherCountries
MeanofLatentAbility(StDError) -0.121(0.021) 0.145(0.01)
EAP-PVreliability(SimilartoCronbach’salpha) 0.711 0.783
ReliabilityofItemSeparation 0.993 0.999
The psychometric properties of the tasks in Costa Rica were
similar as in other countries, indicating that these tasks can
be completed in Costa Rica with little to no language,
cultural, or country specific bias.
51
Differences in the difficulty of the items were minor or non-existent
Out of 150 items only two were more difficult for CR than other countries
Difficulty = Frequency
Could be explained by time constraints
Hexagons - Answering the final answer correctly
Small Pyramids – Responding to questions from partner
52
Sunflower - presence of chat when growth appears
Survey (self assessment) - Replying to chats
Two items were easier for CR than other countries
However, there are many items that assess each skill and these differences are based on single items so they are not representative of overall ability in a specific skill
There is no particular task that is more difficult for Costa Rica than the other countries.
55
Now we need…
Developmental assessments
PD for teaching
Resources for teaching
Strategies for teaching
56
Expected to teach subject content and develop 21st century skills
• assess and intervene in both key learning area and general capabilities
Required to think beyond discipline expertise
• current teacher training mainly discipline-based pedagogy
• most effective methods of teaching skills in one discipline may not be same in another
Domain specific or general curriculum
57
Literacy Numeracy
ICT Capability Critical and Creative
Thinking
Personal and Social Capability
Ethical Understanding
Intercultural Understanding
Science
History
Math
English
LEARNING
AREAS GENERAL CAPABILITIES
Australian Curriculum and Reporting Authority, ACARA (2013).
EMBEDDED INTO CURRICULUM : Australian Curriculum
58 http://www.moe.gov.sg/media/press/2010/03/moe-to-enhance-learning-of-21s.php
EMBEDDED INTO CURRICULUM : Singapore Curriculum
59 CONVIVENCIA
VIDA COTIDIANA Y ARTES INDUSTRIALES
SEXUALIDAD
ÉTICA, ESTÉTICA Y CIUDADANÍA PRIMARIA (MÚSICA, ARTES PLÁSTICAS, EDUCACIÓN FÍSICA,
CÍVICA Y ESTUDIOS SOCIALES)
MATEMÁTICAS
ESPAÑOL
ÉTICA, ESTÉTICA Y CIUDADANÍA SECUNDARIA
(MÚSICA, ARTES PLÁSTICAS, CÍVICA, EDUCACIÓN FÍSICA)
EMBEDDED INTO CURRICULUM : Costa Rica
66
Why engage in the assessments?
Establish a baseline for
academic standards
and
Influence curricula at local, regional,
national and global levels
Influence curricula at local, regional,
national and global levels
Generate new approaches to
evaluation and teaching
© Copyright The University of Melbourne 2014
Claire Scoular [email protected]
Esther Care
www.unimelb.edu.au/arc
www.atc21s.org
www.actrc.org