Design experiments for assessment

of “21st century” learning

ROSEMARY HIPKINS, New Zealand

Presentation at VUW Assessment Conference, Wellington, September 1-3, 2011

An “evidence-centered assessment architecture”:

1. Select and develop tasks based on construct

2. Present tasks to learners

3. Learners generate evidence with respect to constructs

4. Evidence used to make inferences about construct of

interest

(Almond et. al. 2002, cited in Scalise and Wilson, 2011)

OECD Key competencies

• Holistic - integrate knowledge, skills, attitudes and values, including a focus on dispositions

• Demonstrated in authentic contexts when students adapt the competency to do something in a new setting.

• Participatory in intent: consistent with situated and socio-cultural views of learning.

• A focus on doing and being, not just knowing

How should the two parts of NZC relate to each other?

The “front end”

• Vision

• Values

• Principles

• Key competencies

• Effective pedagogy

The “21st century” vision

The “back end”

8 levels

8 learning areas

8 sets of achievement

objectives per level

The more traditional “content”

New ‘nature of science’ integrating strand

(based on generic Key Competencies)

Understanding about science Investigating in science

Communicating in science Participating and contributing

Living

WorldMaterial

World

Physical

World

Planet

Earth and

Beyond

Content included through integration

Working “like a scientist”: two inter-related roles

Constructing claims via age appropriate

investigations and development of simple

explanations

Critiquing claims of others

(ask questions, cite evidence, defend

arguments etc)

Research to explore teaching for key competencies

(and hence potential transfer of learning)

• Reframing traditional Assessment Resource Bank

tasks to support formative assessment

• ARB items trialed with at least 200 students and in a

range of schools

• Each item is a small “design experiment” to identify

and support “next learning steps”

• The question here: what can students do with

classification (is there simple inquiry potential?)

Item analysis

• Students from 16 schools in trial

• 169 @ year 6, 171 @ year 8

• First part of item: basic frequency counts

• Second part coded: code combinations (A, AA, ABB, BC, etc) recorded in data base

• Some cross-tabulation to address emergent questions

• “Next steps” advice worked out and published with results

Features of reptiles

have a backbone

are cold-blooded

breathe with lungs

have dry skin covered with

scales

(some also have a hard shell)

have ear holes but no ears

have 4 legs or no legs

lay eggs which have a strong,

soft shell. A few reptiles have

live babies.

Yr 6 Yr 8

yes 88% 91%

no 9% 6%

I can’t

tell

2% 2%

Features of reptiles

have a backbone

are cold-blooded

breathe with lungs

have dry skin covered with

scales

(some also have a hard shell)

have ear holes but no ears

have 4 legs or no legs

lay eggs which have a strong,

soft shell. A few reptiles have

live babies.

Yr 6 Yr 8

yes 53% 60%

no 37% 28%

I can’t

tell

10% 11%

Features of reptiles

have a backbone

are cold-blooded

breathe with lungs

have dry skin covered with

scales

(some also have a hard shell)

have ear holes but no ears

have 4 legs or no legs

lay eggs which have a strong,

soft shell. A few reptiles have

live babies.

Yr 6 Yr 8

yes 9% 8%

no 75% 79%

I can’t

tell

14% 12%

Scoping the potential for key competency/

Nature of Science development

What would the teacher learn from only assessing

for “correct” answers?

What if the “next steps” were about the use of

evidence?

What role could “I can’t tell” play in developing an

inquiry focus for classification?

What is discipline-specific about classification?

• For many students the task was not seen as one of skills-based observation and decision-making

• From what I have learnt at school I’ve learnt frogs are amphibians.

• I just know it’s an amphibian. • I know a weta is an insect. It has 6 legs, no

scales, no hard shell, doesn’t have a backbone (emphasis added).

• Many students took “I can’t tell” to mean “I don’t know”, but one or two did see its intended purpose

• I can’t tell if it has a dry skin.

Understanding the affordances that students

could bring to the task: some or all of….

• Perceived purpose of activity

• Own knowledge of each animal

• Observation skills

• Ability to read and use list provided

• Disposition to suspend answering until all

possibilities considered

Many students used a combination of

observable and not observable features (own

knowledge as one affordance)

Fewer cited only observable features (coded

as A, AA, AAA etc combinations)

They were most likely to resort to use of their

background knowledge for the non-reptilian

vertebrates (where disconfirming evidence

becomes a bit less obvious)

Nature of

evidence

cited

Fish

%

Frog

%

Bird

%

Weta

%

Snake

%

Lizard

%

Turtle

%

Used own

knowledge

only

43 36 34 29 27 24 10

Used

observable

plus own

knowledge

25 30 25 23 38 34 36

Used

observable

only

16 15 23 22 26 27 34

Missing or

other 15 20 18 26 9 15 20

Year 6 students’ responses

• Cross-tabulation of the two parts of

the task confirmed that “I can’t tell”

was likely to mean “I don’t know” for

many but not all students

Evidence cited

(2nd part of

question)

Year 6 classification

(1st part of q)

Year 8 classification

(1st part of q)

Yes No Can’t

tell

Yes No Can’t

tell

Observation

only

2 35 2 45 1

Observable plus

own

knowledge

1 37 1 4 39 2

Own knowledge

only

10 34 3 4 40 3

No evidence

cited

2 20 19 3 11 15

X-tab data for responses to weta photo

A “thinking curriculum” for the 21st century (Laura

Resnick)

• Intellectually rigorous

• Discipline specific so requires deep teacher expertise

• For all students not just elite

The investigative potential in this science task:

• evidence-based decision making;

• recognition of disconfirming evidence;

• re-examination of own “just knowing”

• Active practice of both constructor and critiquer

roles

©NZCER

Acknowledgements

My science team colleagues: Chris Joyce, Lorraine

Spiller, Ally Bull, and statistician Hilary Ferral

New Zealand’s Ministry of Education for research funding

Teachers who help and students who complete ARB trials