the role of information in systems · professional development ... and professional development)...
TRANSCRIPT
Problem States, districts and schools are being asked to
conceptualize and implement broad, coherent systems Assessment Instruction Curricula Professional development
Goal of these systems is to increase student achievement
But the intricacies of these multi-component systems threaten to overwhelm the ability to understand and successfully plan these systems
Problem How are states and districts responding to the demand to conceptualize, award, develop and implement these coordinated and
complex systems? This session is intended to--
Help conceptualize the problem, Identify important aspects of assessment, curriculum,
instruction and professional development, Describe Systems Thinking as a conceptual tool, and Describe simulation as a statistical tool
Systems Thinking What is Systems Thinking? Systems Thinking means being able to see the underlying
web of ongoing, reciprocal relationships which are cycling to produce the patterns of behavior that a system is exhibiting
Systems Thinking means being able to see the forest (of relationships amongst Assessment, Instruction, Curricula and Professional development) for the trees (of individual tests)
Hard to do because we are crossing boundaries of expertise and authority
Session Jeff Barker, Executive Director, Student
Accountability and Assessment, Gwinnett County Public Schools
Jeff will discuss concerns in creating a coherent system of assessment, curriculum, instruction and professional development from the district perspective
Session Melissa Fincher, Associate Superintendent,
Assessment and Accountability, Georgia Department of Education
Melissa was going to discuss concerns in creating a coherent system of assessment, curriculum, instruction and professional development from the state perspective
Melissa was unable to attend
Session Paul Nichols, Principal Research Scientist, Pearson Paul will discuss using simulations to help design
coherent systems of assessment, curriculum, instruction and professional development
Session Kristen Huff, Senior Fellow, Regents Research Fund,
New York, NY Kristen will pull the two parts of this session together--
the state and district perspectives and a formal simulations-- and move toward incorporating systems thinking into assessment design and development.
Please Go in Order! Please rotate through the presentations in
order! (1) Jeff (2) Paul (3) Kristen
They will make more sense that way!
Paul Nichols
Center for Next Generation Learning & Assessment Pearson
Problem Recognition that improved student learning requires
coordinated assessment, instruction, curricula and professional development
States and districts are being asked to conceptualize and implement assessments as a component of broad and coherent systems
For example, comprehensive assessment systems or systems for learning.
New burdens on district and state staff
Problem Think beyond a single test or set of tests to
coordinated system of assessment, curriculum, pedagogy and professional development
Manage and held accountable for the award, development and implementation of multi-component systems
Cross department boundaries, engage multiple-vendors and involve many times the cost of a testing program
Status Quo State and district staff already have or can find the
training and tools to help with test development and implementation for individual tests
Currently no theoretical framework and no indices to help state, district and other educational leaders in planning, developing and evaluating systems for learning
Systems Approach Move beyond hand waving to offer a conceptual
framework and quantitative tools used by district and state staff to conceptualize and implement systems for learning (SFL)
Note the components that probably should be included in an SFL
Present simulation allowing district and state staff to predict SFL coherence across different configurations
Defining Systems What is a system? From the field of systems thinking:
A system is “a collection of parts which interact with each other to function as a whole”
An SFL is a set of components which interact with each other to function as a whole with the intention of improving student learning
Conceptual Framework The Learning Framework offers a conceptual
framework
Conceptual Framework Curriculum consists of the knowledge and skills in subject
areas that teachers teach and students learn. Instruction refers to methods of teaching and the learning
activities used to help students master the content and objectives specified by a curriculum.
Assessment is the means used to measure the outcomes of education and the achievement of students with regard to important competencies.
Professional development may be in schools of education or provided by schools and districts
Theory of learning is the model of student cognition and learning in the domain
Conceptual Framework Coherence is provided by the theory of learning The components must “talk to each other” in a
common language The language of communication is the learning theory
Quantitative Tool Simulation of the conceptual framework built with the
visually oriented graphical programming language STELLA
STELLA is a commercially available graphical simulation program
STELLA was chosen for this project because it is a powerful yet easy-to-use dynamic system modeling computer language
Quantitative Tool High level view of the simulation of the Learning
Framework under default assumptions
Leave Teaching Force Grd3
Year Begin InstrucPD
Leave Teaching Force Grd3
Receiving Instruc Innovation
Receiving Curr Innovation
Receiving Assess Innovation
Year Begin CurrPD
Year Begin AssessPD
InGrd3 ChCA
Grd3 ConCh Instruct PD
Grd3 ConCh Curr PD
Grd3 ConCh Assess PD
In Grd3 ChIPD ?
In Grd3 ChCPD ?
In Grd3 ChAPD ?
Grd3 ConCh Intruc Attmpt
Out Grd3 ChIPD
Out Grd3 ChCPD
Out Grd3 ChAPD
Grd3 ConCh Curr Attmpt
Grd3 ConCh Assess Attmpt
Grd3 ConCh Intruc NO Attmpt
Grd3 ConCh Curr NO Attmpt
Grd3 ConCh Assess NO Attmpt
InGrd3 ChIA
InGrd3 NoChIA
InGrd3 NoChCA
InGrd3 ChAA
InGrd3 NoChAA
OutGrd3 ChIA
OutGrd3 NoChIA
OutGrd3 ChCA
OutGrd3 NoChCA
OutGrd3 ChAA
OutGrd3 NoChAA
Grd3 ConCh Intruc Implmnt
InGrd3 ChII OutGrd3 ChIIGrd3 Assoc Intruc NO Implmnt 2
InGrd3 NoChIIOutGrd3 NoChII
Grd3 ConCh Curr Implmnt
InGrd3 ChCI OutGrd3 ChCIGrd3 ConCh Curr NO Implmnt
InGrd3 NoChCI OutGrd3 NoChCI
Grd3 ConCh Assess Implmnt
InGrd3 ChAI OutGrd3 ChAI
Grd3 ConCh Assess NO Implmnt
InGrd3 NoChAI OutGrd3 NoChAI
Leave Teaching Force Grd3Leave Teaching Force Grd3
Leave Teaching Force Grd3
Quantitative Tool View of the simulation of the Student Achievement
Subsystem
Grd3@A Grd4@A Grd5@A Grd6@A Grd7@A
InGrd3@A InGrd4@A InGrd5@A InGrd6@A InGrd7@A
OutGrd3@A OutGrd4@A OutGrd5@A OutGrd6@A OutGrd7@A
Cohort Size A
SuccessRate@A
SuccessRate@A
SuccessRate@A
CtchUp@A
SuccessRate@A
SuccessRate@A
SuccessRate@A
Grd8@ABlowGrd3@A
InBlowGrd3@A InGrd8@A
OutBlowGrd3@A
SuccessRate@A
OutGrd8@A
Remdl@ARemdl@A
Abv eGrd8@A
InAbv eGrd8@A
OutAbv eGrd8@A
Remdl@A
Remdl@ARemdl@ARemdl@A
InitializeTime A
InitializeTime A InitializeTime A InitializeTime A InitializeTime A InitializeTime A InitializeTime A InitializeTime A
CtchUp@A
CtchUp@A
CtchUp@A
CtchUp@A
Quantitative Tool A default static simulation Assumptions: Instruction, curriculum and assessment
implicitly or explicitly reflected an associationist learning approach
No professional development was taking place No changes in classroom practice across years
Quantitative Tool Distribution of student achievement across grades 3 to
8 under a default simulation
Grade
Proficient Below Grade
3 Proficient at
Grade 3 Proficient at
Grade 4 Proficient at
Grade 5 Proficient at
Grade 6 Proficient at
Grade 7 Proficient at
Grade 8
Proficient Above
Grade 8
3 100 1800 100 0 0 0 0 0
4 10 317 1587 86 0 0 0 0
5 1 38 415 1471 75 0 0 0
6 0 4 67 456 1407 66 0 0
7 0 1 10 90 468 1374 59 0
8 0 0 1 15 108 464 1359 53
Quantitative Tool Distribution of student achievement across grades 3 to
8 under a default simulation
0
200
400
600
800
1000
1200
1400
1600
1800
2000
1 2 3 4 5 6 7 8
Grade 3
Grade 8
Grade 5
Below 3 3 4 5 6 7 8 Above 8 Proficient at Grade
Quantitative Tool An evolving simulation in which teachers integrate an
innovation into their classrooms Innovative program where curriculum, instruction and
assessment based on a conceptual change theory of learning, i.e., a learning progression
Students are evaluated using a conventional standardized test, based on an associationist model, at the end of the year
Teacher retention 5% of grade 5 teachers leaves the district each year 5% are hired as replacements each year
Quantitative Tool Groups of 20 grade 5 teachers complete professional
development every year All teachers who attend professional development do
experiment with the innovations in their classrooms All teachers who experiment with the innovations find
that the innovation fit their classroom conditions Grade 5 students are slightly more likely to be increase
an achievement level on the standardized test by the end of the year
None of the newly hired teachers have received training in learning progressions
Quantitative Tool The distribution of grade 5 teachers across the
categories of implementing learning progressions
Year Receiving
Professional Development
Exploring Innovations
Implementing Innovations
Implementing Associationist
Approach
Ratio Implementing
Innovation/ Total
1 20 0 0 80 0.00
2 20 19 0 61 0.00
3 20 19 18 43 0.18
4 20 19 35 26 0.35
5 20 19 51 10 0.51
6 10 19 67 4 0.67
Quantitative Tool Distribution of Grade 5 students across levels of
proficient over six years of implementing learning progressions
Year
Proficient Below
Grade 3 Proficient at Grade 3
Proficient at Grade 4
Proficient at Grade 5
Proficient at Grade 6
Proficient at Grade 7
Proficient at Grade 8
Proficient Above
Grade 8
1 4 92 515 1323 66 0 0 0
2 4 92 515 1323 66 0 0 0
3 4 89 501 1339 67 0 0 0
4 2 87 488 1355 68 0 0 0
5 1 79 476 1373 71 0 0 0
6 0 69 462 1394 75 0 0 0
Quantitative Tool Distribution of Grade 5 students across levels of
proficient over six years of implementing learning progressions
0
200
400
600
800
1000
1200
1400
1600
1 2 3 4 5 6 7 8
Year 1 Year 6
Below 3 3 4 5 6 7 8 Above 8 Proficient at Grade
Conclusion Simulations can provide unexpected results Under the approach to professional development in
the learning progressions simulation, Entire grade 5 teaching force is not trained after 6 years Impact on EOY test performance is minimal
Suggestions for improving outcomes-- Hire teachers already trained in learning progressions Construct an EOY test that is more coherent with the
learning progression innovation
Summary State and district staff are asked to conceptualize and
implement SFLs State and district staff need a conceptual framework
and quantitative tools to help plan and evaluate SFLs Measurement theory focuses on individual tests or
groups of tests Instructional theory focuses on instructional
techniques and activities Simulations offer some help…