computerized adaptive testing and multistage testing: in

智慧评测与智慧学习

张华华 ([email protected])

美国伊利诺伊大学教育心理兼心理和统计学教授

华东师范大学长江学者讲座教授

全通教育特邀嘉宾

2015-10-25 中国杭州

内容：“互联网+评测” 1. 智慧评测？ 2. 大数据带来哪些挑战？ 3. 自适应技术

1. 统计模型 2. 智慧题库 3. 自适应测验 4. 认知诊断

4. 智慧评测帮助智慧学习 – Making Teaching/Testing Adaptive – Paper/pencil Test Adaptive

5. 总结

2

从智慧评测到智慧学习---如何使用评测技术帮助新一代的教育

张苏苏、张华华（2015）

3

Knewton：世界最大的自适应学习平台

“A is for Adaptive-- Personalized learning is

poised to transform education. Can it enrich students

and investors at the same time?” (Time, June 17, 2013)

7

Bradley: “Knewton is based on what you can do, not what the class can do” Clenna: “It adapts to you, so it starts easy and then gets harder”

可汗学院(KHAN Academy)

8

SMART EDUCATION – INNOVATIONS AND ATTEMPTS Personalized learning routes • Knewton system Blended learning • Khan Academy • Intelligent Tutoring Systems • e-Schoolbags/student portfolios Open Educational Resources and Massive Online Open Courseware • Coursera • EdX • International expansions and personal device use

9

MOOCS

MIT OCW and the OER initiative Massive Online Open Courseware (interactive learning): • Instruction • Resource provision • Peer grading • Forum discussion • Certification edX, Udacity, Coursera (>10 mil users, 839

courses, 114 institutes as of 2014)

Personal device and mobile interfaces • Self-sufficient

10

智慧教育当前面临的最大挑战 • MOOCs退学率居高不下

• 可能原因: – 成年人时间宝贵

– 无因人而异的教学起点

– 与基于实际表现的人员选拔冲突

• 有那么一点“放任自流”

挑战: 如何可靠地确定个人的需求来筛选教学资源、并“量身定制”学习内容?

如何在MOOC’s中加入“自适应”

– 基于序列设计的自动化控制

11

12

个体化学习---因材施教祝智庭提出的模型：

“智慧教育”需要“智慧评测” 以促成个体化的 “自适应”学习

13

智慧评测包括以下几个方面

• 计算机自适应考试 (CAT)

• 认知诊断模型 (CD)

• CAT + CD

• 在线估计(Online calibration)

• 客观题阅卷的自动化 (Performance based assessment)

• 无偏化评测、考试公平性 (DIF and test equity)

• 纸笔考试自适应化 (Paper pencil based adaptive testing)

14

是“题库”还是“题堆”？题库自动化组卷实例

16

自动化试卷生成

原始卷

卷 1 卷 2 … 卷 N

题库

Parallel Parallel

Parallel

Select items from the item pool

Parallel Assembly

• Bottom-up strategy

• Parallel Assembly for subtest

Priority index: based on classical item characteristic(item difficulty and

discrimination parameters)

Item Pool Analysis

Sub pool Listening

Sub Pool Reading

Sub Pool Writing

Form 1

…

Item Pool Analysis

Sub pool Listening

Sub Pool Reading

Sub Pool Writing

Form 1

Form 2

Item Pool Analysis

Sub pool Listening

Sub Pool Reading

Sub Pool Writing

Form 1

Form 2

Form N

用CAT 帮助学习

• The idea is not new. But before technology is ready it is impossible to provide 1-to-1 teaching on a large scale.

• CAT can help! – Selecting items sequentially helps students better understand the

concepts being taught

– CAT provides more flexibility in a CAT + Learning setting

• Examples in China: – As the government increasingly invest in schools’ computers and

Internet infrastructure, some schools are using CAT during on-line teaching to offer individualized assessment and get immediate feedback to students

22

CAT can be utilized to get Cognitive Diagnostic

Information: CD-CAT

• A-matrix

• Q-matrix

ik

jkq

Item

1

2

3

4

0 1 1 0 0

0 0 1 1 1

1 0 0 0 1

0 0 0 0 1

Attributes 1 2 3 4 5

0 1 1 0 0

0 0 1 1 1

1 0 0 0 1

0 0 0 0 1

Attributes 1 2 3 4 5

Exam

inee

1

2

3

4

23

智慧学习: (1) 电脑教学教室；(2) 老师教学教室

In Zhang and Chang (2015)

25

COMBINING CD-CAT AND SMART LEARNING --COMPUTER-BASED CLASSROOM

26

COMBINING CD-CAT AND SMART LEARNING --TEACHER-BASED CLASSROOM

27

如何用 “INTERNET +”来帮助一线教师？

我们的目的不是取代课堂教学和一线教师

28

How to make P&P Test Adaptive ?

z

zzv

来自中国大连的实例-- 纸笔考试“自适应化”

美国国家科学基金会研究项目

CD-CAT 帮助 STEM 教育

• 解决科学、技术、工程及数学学科中退课率居高不下的状况

• 考试成绩不好导致很多学生放弃STEM课程

• 这些低分学生 (包括尚未准备充分) 不善于预测自己考试前后的学习能力，为了避免因此失败并退学，通常最终放弃STEM专业并转向其他专业。

• 我们使用 CD-CAT 来帮助低表现学生在物理课程中的表现。

• 物理211 是伊利诺伊大学第一门用到微积分的物理课

30

伊利诺伊大学香槟分校（UIUC）

• 成立于1867;公立大学中的常青藤

• 图书馆藏书量全世界大学第三

• 学校SCI论文总数全美名列前5

• 有24位教授/校友荣获诺贝尔奖

• 著名的华人校友包括:

– 姚期智,华罗庚,黄万里，邢其毅，竺可桢，李安等。

• 该校还是许多心理测量专家的母校

31

CD-CAT in STEM Fields

• Research Design in the UIUC Physics Course

Hourly Exam 1 Mid-term P&P



Hourly Exam 4 Final P&P

Identify students scoring

below 70%

Three CD-CATs

Three CD-CATs

Intrv 1: CD-CAT Intrv 2: CD-CAT + Worked Examples Intrv 3: CD-CAT + Interactive problems Intrv 4: CD-CAT + Human tutoring Control group

Pass

Fail

Ex1. CD-CAT in STEM Fields

• Project Stages

Collect & Analyze data

Administer high-stakes

in-class tests

Provide remedial

interventions based on CD-

CAT results

Recruit students and administer

CD-CAT

Develop a web-based

CD-CAT platform

Build item pools by analyzing

legacy items


• Example Data Coding


• Snapshot of CD-CAT Web-Delivery

Login page via university course website



Student’s mode of taking CD-CAT



Administrator’s mode of maintaining CD-CAT

CD-CAT 在中国大规模试验实例

39

40

In December 2011, 30,000 Grade 5 Students in Dalian China were taking a cognitive diagnostic CAT for their English proficiency assessment.

A Large Scale CAT with 2000 PC’s in Dalian, China

“圆的面积” 课例展示（北京市海淀区西颐小学六年级二班）

Utilizing CAT in Classroom Teaching, Students

are learning “Area of a Circle”

图片说明：1．集体学习系统中“圆的面积”的视频内容； 41

CAT Is Revolutionarily Changing the Way We Address

Challenges in Learning

█ Students really enjoy the new mode of testing, which makes learning more enjoyable comparing with regular teaching and P&P testing

42

Web-delivered CD-CAT Brings Equality

█Thanks to the Internet, whether schools are equipped with fancy LEDs or bulky CRTs, students receive the same quality of assessment.

43

Most Students said the CD-CAT is helpful!

greatly,

113, 58%

yes, 65,

33%

no, 17, 9%

greatly yes no

Is CD-CAT helpful to your learning?

Teachers (郑州金水实验区) ：Assigning different items to each student, CAT encourages critical thinking, and makes students more independent in problem solving, and offers remedy according to their individual needs, which makes learning more interesting.

44

never, 47,

26%

once, 40,

22% 2 times,

39, 21%

3 and

above, 58,

31%

never

once

2 times

3 and above

How many times/week you use CD-CAT without teacher’s assignment

< 20

minutes,

49, 24%

20-40

minutes, 112,

55%

40-60

minutes, 27,

13%

> 60minutes,

15, 8%

< 20 minutes

20-40 minutes

40-60 minutes

> 60minutes

How many minutes each time

45

Help Teachers Know Their Students Better. According to

the diagnostic report, remedial planning is on the way

█ The in-class CAT provides more information to teachers, which facilitates research and career development

【郑州市金水区纬一路小学】的老师在实验中，借助易学通系统对习题进行钻研，通过对学生学习情况的不断分析总结，促进教师在反思中提高自身的教学技能，在提高教学质量的同时，也使自身的专业素养得到提升。

图片说明：三位实验教师在讨论学习内容

46

总结

• 智慧学习是离不开智慧评测的

• 智慧评测的关键技术是自适应算法，应该也是“联网+ 教育”的关键技术

• 感谢“全通教育”的邀请，使我有机会与大家共享我团队的工作

“自适应评测正在实质性地影响社会的日常运转，通过影响人们如何选拔、分配

归类、和诊断决策; 自适应方法的研究将引导致更有效的评估，并因此造福社

会。”

摘自张华华的2013世界心理测量学会主席报告联系方式： [email protected]

47

computerized adaptive testing and multistage testing: in

Documents