presented by daphne minner alina martinez abt associates june 15, 2012
DESCRIPTION
Community for Advancing Discovery Research in Education (CADRE) A Review of Research Instruments for STEM Teacher Practices, PCK, and Content Knowledge. Presented by Daphne Minner Alina Martinez Abt Associates June 15, 2012 DR-K12 Principal Investigator Annual Meeting Arlington, VA. - PowerPoint PPT PresentationTRANSCRIPT
CADRECOMMUNITY FOR ADVANCING
DISCOVERY RESEARCH IN EDUCATION (CADRE)
A Review of Research Instruments for STEM Teacher Practices, PCK, and
Content KnowledgePRESENTED BYDAPHNE MINNERALINA MARTINEZAbt Associates
JUNE 15, 2012DR-K12 PRINCIPAL INVESTIGATOR ANNUAL MEETING
ARLINGTON, VA
CADREDR K-12 Portfolio Overview
Purpose• Describe important characteristics• Provide evidence of progress toward goals• Identify areas for targeted studies• Inform CADRE’s support activities
Questions• What are the characteristics of projects? • What types of projects are being funded?• What research is being conducted?• Are projects evaluating their work?• How are projects going to disseminate their work?
June 15, 2012
CADRE
Data from 248 Projects
Cohort 4(71 Projects)
Cohort 1(75 Projects)
Cohort 3(51 Projects)
Cohort 2(51 Projects)
Includes projects that submitted at least a proposal
Sources: Provided by PIs
voluntarily in response to requests
Found in the public domain
June 15, 2012
CADRE
Project Materials were Reviewed and Coded
Project materials were reviewed and
coded
Coded information was recorded in a
database
Descriptions of DR K-12 portfolio characteristics
were compiled for the annual report
June 15, 2012
CADRE
RFP Guides Innovation in STEM Education
Grand Challenges (2006)
• K-12 math and science assessments• Elementary grade science• Cutting-edge STEM content in K-12
Challenge Strands (2008)
• Contextual challenges• Frontier challenges
Challenge Areas (2009, 2010, 2011)
• Assessments• Opportunity to learn for all students• Ability of teachers• Promising innovations• Cyber-enables learning (2011 only)
Strands (2012)
• Assessment• Learning• Teaching• Scale-up and sustainability
June 15, 2012
CADREDisciplines Addressed in Projects
34%
41%
19%
3%
2%
1%
MathematicsOther
Engineering
Multi-disciplinary
Sciences
Computer
June 15, 2012
CADRE
Target Segment of the Education Population
0%
20%
40%
60%
80%
100%
80%75%
13%5% 4%
Students
OtherHigher Education Faculty
Teachers
School Adminis-trators
June 15, 2012
CADRE
Outcomes Studied
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
66% 65%
14%10%
6% 5%
Implementation Fidelity
Administrators
TeachersStudents
Research Quality, Progress
Other & Unknown
June 15, 2012
CADRE
Types of Student and Teacher Outcomes Studied
Students
0%10%20%30%40%50%60%70%80%90%
100%
62%
32%
18%
Attitudes
Behavior
Teachers
0%10%20%30%40%50%60%70%80%90%
100%
49%
35% 34%
23%
Attitudes
Classroom Practices
Pedagogical Content
Knowledge
Content Knowledge
June 15, 2012
Achievement
CADRETeacher Instrument Study
Overview
• Rationale
• Methods
June 15, 2012
CADRE
75 projects assessed teacher outcomes with existing instruments
Number of Teacher Instruments Used
Number of Projects
1 35
2 21
3 9
4 65 4
June 15, 2012
Outcome Domain Number of Projects
Practice 32
PCK 14
PCK & Practice 8
Content 7Practice & Content 6PCK & Content 4All three 4
Table A Table B
CADRE
82 unique instruments identified
42 used to assess Practices Reformed Teaching Observation Protocol (modified)—15 studiesInside the Classroom Observation and Analytic Protocol—8Surveys of Enacted Curriculum (modified)—5
24 used to assess ContentViews of Nature of Science (Form C)—3 studiesNational Assessment of Educational Progress (modified)—3Praxis content tests/ Earth & physical science (modified)—2TIMSS content tests (modified)—2
June 15, 2012
CADRE82 instruments identified
22 used to assess Pedagogical Content Knowledge
Mathematical Knowledge for Teaching—14 studies Knowledge of Algebra for Teaching (modified)—2Science Teaching Efficacy Belief Instrument—2Reformed Teaching Observation Protocol (modified)—2
June 15, 2012
CADRE
More Refined Look at Constructs
• Instruction practices• Content knowledge• Instructional beliefs; teaching self-efficacy• Class management• Assessment practices/ skills• Classroom social context• Classroom physical context• Administrative or policy context• Student / teacher demographics• Instructional planning
June 15, 2012
CADRE
Instructional PracticesAFM Assessment of the Facilitation of Mathematizing obs PK-6 M
EMCO Early Mathematics Classroom Observation obs PK METAP EdTech Assessment Profile survey - Tech
ISCOP Instructional Strategies Classroom Observation Protocol obs mid SC
LFCPO Lesson Flow Classroom Observation Protocol obs mid SC
LoFTI Looking for Technology Integration obs - Tech
O-TOP OCEPT-Classroom Observation Protocol obs post SC, M
STIR Science Teacher Inquiry Rubric obs elem SC
TIDES Transforming Instruction by Design in Earth Science--Teacher assignment quality rubrics rubric - SC
Scoop Notebook Artifact rubric mid SC
The Quality of Instruction Measure rubric - M
June 15, 2012
CADREInstructional Practices plus
CETP-COPThe Collaboratives for Excellence in Teacher Preparation core evaluation classroom observation protocol
Physical context, demo obs, inter K-16 SC,M
EQUIP Electronic Quality of Inquiry Protocol demo obs K-12 SC,M
KAT Knowledge of Algebra for Teaching content survey, assessment Mid, high M
ICOT ISTE Classroom Observation Tool tech use obs K-12 Tech
MQI Mathematical Quality of Instruction content obs Elem, mid M
PRAXIS Praxis Teaching Foundations: Science Class mgmt, assessment survey Mid, high SC
PRISM Preschool Rating Instrument for Science and Mathematics Physical context obs PK SC,M
SESAME Self-Evaluation of Science and Math Education planning - PK SC,M
SIOP Sheltered Instruction Observation Protocol planning obs, rubric elem Gen
TIMSS Third International Mathematics and Science Video Study (TIMSS) social, demo obs, survey mid SC,M
Ohio Middle Level Mathematics and Science Education Bridging Study - Teacher Questionnaire
content, demo survey K-12 SC
June 15, 2012
CADRE
Teachers’ Content KnowledgeDTAMS Diagnostic Teacher Assessments in
Mathtematics and Science PCK test mid SC,M
DTAMS Diagnostic Mathematics Assessments for Elementary Teachers--algebra PCK test elem M
M-SCAN The Mathematics Scan instruction, assessment obs elem M
ACT American College Testing St. test post GENAP Anvanced Placement St. test high GEN
CST Content Specialty Test Earth Science for New York Teacher Certification test K-12 SC
FACETS Diagnoser Tools St. test Mid, high SC,M
FCI Force Concept Inventory assessment St. test High, post SC
GRE Graduate Record Exam test post GEN
ITBS Iowa Test of Basic Skills St. test Elem, mid GEN
MAP Missouri Assessment Program St. test Elem, mid SC,M
MKT Mathematical Knowledge for Teaching test Elem, mid M
June 15, 2012
CADRE
Teachers’ Content KnowledgeMOSART Misconceptions-Oriented Standards-Based Assessment Resources
for Teachers test K-12 SC
NAEP National Assessment of Educational Progress St. test K-12 GEN
PISA Program for International Student Assessment St. test high GEN
PRAXIS content tests/ Earth & physical science (modified) test post SC
Regents New York State Regents exam St. test K-12 GEN
SAT Stanford Achievement Test St. test high GEN
TAGLIT Taking a Good Look at Instructional Technology test post TECH
TIMSS content tests St. test Elem, mid SC,M
West-E Washington Educator Skills Test-Endorsements test K-12 GEN
WESTEST Science WESTEST St. test Gr. 4-9 SC
American Chemical Society Division of Chemical Education Examinations Institute test post SC
Classroom Test of Scientific Reasoning St. test - SC IL Certification Testing System Study Guide-Science: Biology test post SC
June 15, 2012
CADRE
Instructional BeliefsIMBS Indiana Mathematics Beliefs Scale survey - M
MTEBI Mathematics Teaching Efficacy Belief Instrument survey preservice M
PSI-T Principles of Scientific Inquiry-Teacher survey high SC
SETAKIST Self-Efficacy Teaching and Knowledge Instrument for Science Teachers survey elem SC
STEBI Science Teaching Efficacy Belief Instrument survey elem SC
TBI Teacher Belief Interview interview high SC
TSES Teachers' Sense of Efficacy Scale survey PK-6 GEN
TSI Teaching Science as Inquiry survey elem SC
VNOS-C Views of Nature of Science Form C survey - SC
June 15, 2012
CADRE
MultidimensionalSYSTEM-WIDE REFORM FOCUSED
Practices
Content
Beliefs
Management
Assessment
Social
Physical
Admin context
Demographics
planning
CIP Inside the Classroom Teacher Interview Protocol COP Inside the Classroom Observation and Analytic
Protocol CTRI Coaching /Teacher Reflection Impact Surveys FFT Danielson's Framework for Teaching Domains LSC LSC Core Evaluation Classroom Observation
Protocol SEC Surveys of Enacted Curriculum SII Study of Instructional Improvement
TIMSS-R TIMSS-R Science Teacher Questionnaire TIMSS-R TIMSS-R Mathematics Teacher Questionnaire
Inside the Classroom Teacher Questionnaire: Math or Science version
June 15, 2012
CADRE
System-wide Reform FocusedCIP Inside the Classroom Teacher Interview Protocol interview K-12 SC,M
COP Inside the Classroom Observation and Analytic Protocol obs K-12 SC,M
CTRI CoachingTeacher Reflection Impact Surveys survey K-8 M
FFT Danielson's Framework for Teaching Domains rubric K-12 GEN
LSC LSC Core Evaluation Classroom Observation Protocol obs K-12 SC,M
SEC Surveys of Enacted Curriculum survey K-12 SC,M,LIT
SII Study of Instructional Improvement survey elem M, LIT
TIMSS-R TIMSS-R Science Teacher Questionnaire survey elem, mid SC,M
TIMSS-R TIMSS-R Mathematics Teacher Questionnaire survey elem, mid SC,M
Inside the Classroom Teacher Questionnaire: Math or Science version survey K-12 SC,M
June 15, 2012
CADRE
June 15, 2012
DIOLOGUE FOCUSEDPractices
Content
Beliefs
Management
Assessment
Social
Physical
Admin context
Demographics
planning
CLASS The Classroom Assessment Scoring System CLO Classroom Lesson Observation Instrument COEMET Classroom Observation of Early Mathematics
Environment and Teaching DAISI The Dialogic Activity in Science Instruction EAS The Emergent Academic Snapshot ELLCO The Early Language and Literacty Classroom
Observation IQA Instructional Quality Assessment ISIOP Inquiring into Science Instruction Observation
Protocol RTOP Reformed Teaching Observation Protocol SPC Standards performance continuum Classroom Snapshot Mathematics Classroom Observation Protocol Science Classroom Observation Guide (NCOSP)
CADRE
Dialogue FocusedCLASS The Classroom Assessment Scoring System obs K-12 GEN
CLO Classroom Lesson Observation Instrument obs post GEN
COEMET Classroom Observation of Early Mathematics Environment and Teaching obs PK-6 M
DAISI The Dialogic Activity in Science Instruction obs elem M,LIT
EAS The Emergent Academic Snapshot obs PK-6 GEN
ELLCO The Early Language and Literacty Classroom Observation obs, interview PK-6 LIT
IQA Instructional Quality Assessment obs, artifact elem M,LIT
ISIOP Inquiring into Science Instruction Observation Protocol obs, survey mid, high SC
RTOP Reformed Teaching Observation Protocol obs K-16 SC,M
SPC Standards Performance Continuum obs elem, mid GEN
Classroom Snapshot obs K-12 GEN
Mathematics Classroom Observation Protocol obs, rubric elem, mid M,LIT
Science Classroom Observation Guide (NCOSP) obs K-12 SC
June 15, 2012
CADRE
Reliability Information
June 15, 2012
• Reliability type– missing information (19, 38%)– internal stability alpha (11, 22%) (6 acceptable, 5 good)– interrater, % agreement (7, 14%) (2 low, 2 accept., 3 good)– interrater, Kappa (2, 4%) (acceptable)– Interrater, Spearman rank-order correlation (1, good)– Other (2, good and N/A)– More than one type (8, 16%) (all acceptable to good)
• Level– missing information– <.59 or equivalent: low– .6-.79 or equivalent: acceptable– > .8 or equivalent: good
CADRE
Validity Evidence
June 15, 2012
• Validity Type– Missing information (28, 54%)– Content/ face (11, 21%)– Criterion/ predictive (9, 17%)– Construct (8, 15%)– Concurrent/ convergent (5, 10%)– Discriminant /divergent (2, 4%)
CADRE
Trends• Instructional practices (n=22)
– 11 instruments predominantly observation– 11 instruments with 1-2 other constructs,
observation supplemented with survey methods• Teachers’ content knowledge (n=25)
– primarily assessed domain-specific conceptual or procedural knowledge by multiple choice test
– majority of these instruments being used for teachers were originally designed for students
– 8 science, 3 math, 1 technology-specific tests
June 15, 2012
CADRE
Trends cont.• Instructional beliefs (n=9)
– Predominated by survey methods– 6/9 were science specific with only 2 for math
• Multidimensional (n=23)– System-wide reform (predominantly surveys) (10)– Dialogue focused observation instruments to
capture classroom climate and social aspects (13)• Making reliability and validity evidence more
readily available with information about instruments is essential
June 15, 2012
CADRE
Discussion
• Surprises?• Gaps?• Strategies?• Resources?
June 15, 2012
CADRE
Contact InformationAbt Associates
Daphne Minner
(617) 520-2454
Alina Martinez
(617) 349-2312
June 15, 2012