richard vineyard nevada department of education edys quellmalz and matt silberglitt wested

COPYRIGHT WESTED, 2011

Integrating Simulation-Based Science Assessments into Balanced State Science

Assessment Systems:Findings and Implications

Richard VineyardNevada Department of Education

Edys Quellmalz and Matt SilberglittWestEd

Presented to the Council of State Science Supervisors, San Francisco, CA, March 9, 2011

Enhanced Assessment Grant


Enhanced Assessment Grant State Partners

• Nevada (Lead State) • Utah • North Carolina • Massachusetts • Connecticut • Vermont


Advisors

• Rebecca Kopriva, Wisconsin Center for Education Research–Assessments for English Learners

• Martha Thurlow, National Center on Education Outcomes–Assessments for Students with

Disabilities


Subcontractors

• WestEd STEM (SimScientists)– Project Management– Content Development

• Redhill Studios– Interactive (Flash) Programming

• UCLA/CRESST– External Evaluation


EAG Goals

• Establish the technical quality of simulation-based assessments – for two topics, ecosystems and force and motion

• Study the feasibility and utility of simulation-based assessments in classrooms from 3 states


EAG Goals

• Study the measurement of science knowledge and inquiry by simulation-based curriculum-embedded assessments and unit benchmark assessments in comparison to traditional items for:– All students– ELLs– Students with disabilities

• Propose alternative models for integrating simulation-based assessments into state assessment systems


Relevance to Current Assessment Programs

• Need innovative, technology-enhanced assessments that align with new frameworks that focus on – fewer, deeper, more integrated core knowledge

targets • E.g., the new Framework for Science Education and

next generation national science standards • Models as structures for understanding and studying

science systems (model-based learning)• Science practices for using knowledge and inquiry in

significant, recurring authentic tasks


Relevance to Current Assessment Programs

Need innovative, technology-enhanced assessments that• Target 21st century skills within STEM• Use technology to engage students in use of 21st

century “tools of the trade” • Provide evidence supporting one type of technology-

enhanced performance assessment– For summative and formative purposes – Called for in the RTT Consortia


EAG Design

• Align to national science standards and to science standards of the pilot states– Ecosystems and force & motion– At middle school level– Nevada, North Carolina, Utah

• Accommodations– Extended time– Audio– Zoom


Ecosystem Target Model

System Model Levels

Model LevelDescriptions

Content Targetsby Model Level

Inquiry Targets

Components What are the components of the system and their rules of behavior?

Every ecosystem has a similar pattern of organization with respect to the roles (producers, consumers, and decomposers) that organisms play in the movement of energy and matter through the system.

Use principles to identify role of organisms.

Interactions How do the the individual components interact?

Matter and energy flow through the ecosystem as individual organisms participate in feeding relationships within an ecosystem.

Observe interactions among organisms.

Emergent Behaviors What is the overall behavior or property of the system that results from many interactions following specific rules?

Interactions among organisms and among organisms and the ecosystem’s nonliving features cause the populations of the different organisms to change over time.

predict observe explain investigate


Multiple Modes of RepresentationActive Inquiry

Organism box shows icons that appear and disappear

Graph shows size of population over time.

Data inspector shows population for one point on graph.

Table shows population at start, end, and point selected by data inspector.


EMBEDDED IN CLASSROOM INSTRUCTION

Assessment Module Components for Each Topic/System

Online assessment without feedback

Teacher scores constructed responses

Bayes Net

Proficiency report

Benchmark Summative Unit Assessments

Online assessment with feedback and coaching

Follow up Classroom Reflection ActivityProgress report

Embedded Assessments for Formative Purposes (2 to 4)


Ecosystem Embedded Assessment: Feedback,

Coaching, and Self-AssessmentDemo


Progress Reports to Students


Progress Reports to Teachers

NH = needs help P = making progress OT = on track


Grouping Recommendations for Classroom Reflection Activity


Classroom Reflection Activity

• Formative use of assessment results– Students assigned to teams based on embedded results

• Transfer to different, more complex system• Jigsaw structure

– Allows differentiated instruction via tasks of varying difficulty– Promotes small and large group discourse and collaboration

• Guidance for teacher– Teacher review of key points in simulation– What to look for during group work and questions to pose in response– Presentations– Evaluation of presentations


Benchmark Assessment:Assess Transfer to New Context

Demo


Benchmark Report: Class Summary


Benchmark Report: Student Details


Samples

• 28 think aloud sessions• Pilot testing in 4 states

• 60 participating teachers• ~5,800 students• 40 schools• 28 districts– Response data on the assessments, post tests and

classroom observations


Research Evidence• Quality of the assessments (external review)

– Alignment with standards and curriculum– Coherence– Science content and inquiry– Item design

• Implementation– Feasibility in schools and classrooms– Utility for students and teachers


Research Evidence

• Technical Quality (Validity & Reliability)– Alignment with science standards– Cognitive labs– Analyses of responses

• Comparison of student responses on simulation-based and traditional items– Cognitive labs �– IRT model fits�


Reliability of Benchmark and Post Tests

• Ecosystems– Benchmark (n=3844)

• Mean score was 28.14 (max 39)• Standard Deviation 6.64• Coefficient Alpha: 0.85

– Post Test (n=3422)• Mean score: 16.52 (max 30) • Standard deviation: 6.68• Coefficient Alpha (KR 20): 0.88


Reliability of Benchmark and Post Tests

Force and Motion– Benchmark (n=1504)

• Mean score: 28.15 (max 40) • Standard Deviation: 5.51• Coefficient Alpha: 0.79

– Post Test (n=2166)• Mean score: 14.87 (max 28) • Standard Deviation: 5.73• Coefficient Alpha (KR 20): 0.82


Correlation of Benchmark to Posttest

Ecosystems(n=2924)

Force and Motion(n=1496)

ContentAbility

estimate

InquiryAbility

estimate

ContentAbility

estimate

InquiryAbility

estimateCorrelation with ability estimates on posttest

.51** .24** .40** .45**

** Correlation is significant at the 0.01 level (2-tailed)


CRESST Evaluation Questions

• How effective was SimScientists implementation?

• Was SimScientists feasible and useful in middle school classrooms?

• How effective were SimScientists’ accommodations?


CRESST Evaluation Methods

• Convenience sample of teachers from all three pilot states, both topics– Eight teachers with 3-4 classes– Five schools

• Observation and interview protocols– Assessments– Reflection Activities

• Teacher survey


Teacher’s Role (Assessment)


Class Organization (RA)


CRESST Interviews

“I think that [students] are way more engaged … using this kind of interactive assessment … it was more engaging them simply, “Here is a pencil, here is a paper, answer questions” .”


CRESST Interviews• “…I like that kind of feedback when it doesn’t just go

to the next page and they don’t know whether they did it right or not.”

• “…Yes, the science content is really being tested. Students are asked to conduct experiments, investigate, and draw conclusions and to use scientific skills. Students are not able to guess on the multiple choice questions because it probes them until they choose the right answer. Students are also taught about food webs in one biome and they are tested on another biome.”


CRESST Evaluation Conclusions

• Observations showed that students were actively engaged most of the time during assessments.

• Both teachers and students generally believed that the SimScientists program was beneficial to learning.

• Teachers found the automatically scored, immediate feedback—especially the reports generated by the questions—helpful to students. The instant reports allowed teachers to easily see which questions students had the most difficulty with so that they could tailor their lessons accordingly.


CRESST Evaluation Conclusions

• Teachers collectively agreed that the simulation assessments had greater benefits than traditional paper-and-pencil tests because of the simulation’s instant feedback, interaction, and visuals.

• Teachers agreed that the assessments would be useful in measuring their individual state standards.


Multilevel Balanced Assessment System Models

• Reporting benchmark results alongside district and state data

• Matrix sampling of short “signature” tasks from different topics


Matrix Sampling Short “Signature Tasks” Across Topics


Policy Brief

• Summary of alternative models for integrating simulation-based assessments into balanced state assessment reporting systems

• Summary of study findings on feasibility, utility, and technical quality

• Implications for further study


Related Activities• Increased use of technology in classroom instruction and

assessment• Continued (and growing) frustration around measuring

student’s knowledge and skills above conceptual level• Growth in costs associated with hand scoring of CR items on

science assessments at grades 5 and 8• Development of electronic and on-line resources for

instruction, assessment, and professional development for math and science – MINES, Science Builder, Pathways to Nevada’s Future


Other Projects• Race to the Top• Assessment consortium related to CCSS in

math and reading• Multilevel Assessments of Science Standards• Science Assessments with UDL • EAG proposal for 2011?

– Study series of assessment suites within a grade


Contact information

Email: [email protected]@[email protected]

Website: www.simscientists.org

mailto:[email protected]



http://www.simscientists.org/

richard vineyard nevada department of education edys quellmalz and matt silberglitt wested

Documents