i am sam: the self- assessment of multi-tiered systems of support€¦ · •kevin stockslager...
TRANSCRIPT
This product was developed by Florida’s Positive Behavior Support Project through the University of South Florida, Louis de la
Parte Florida Mental Health Institute funded by the State of Florida, Department of
Education, Bureau of Exceptional Education and Student Services, through federal assistance under the Individuals with
Disabilities Education Act (IDEA), Part B.
I Am SAM: The Self-Assessment of Multi-Tiered
Systems of Support
Karen Elfner Childs, M.A.Natalie Romer, Ph.D.
Jose Castillo, Ph.D.Kevin Stockslager, Ph.D.Amber Brundage, Ph.D.
I Am SAM: The Self-Assessment of
Multi-Tiered Systems of Support
Welcome
and
Introductions
No, not SAM I AM
Advance Organizer
• Our Vision of MTSS
• Self-Assessment of MTSS (SAM)– SAM
Development Procedures
– Components of the SAM
– SAM Psychometric Properties
• SAM and PBIS
• Questions and Comments
OUR VISION OF MTSS
What is MTSS?
• MTSS=Multi-Tiered Systems of Support– Evidence-based instruction delivered to students
in varying intensities (multiple tiers) based on student need
– Utilization of data-based problem-solving to integrate academic and social-emotional instruction and intervention
– Use of data to make educational decisions
Broader than PBIS
• Both social-emotional &academic throughout
• The collaborative vision of the PS/RtI and the PBIS:MTSS Projects is to:
– Enhance the capacity of all Florida school districts to successfully implement and sustain a multi-tiered system of student supports with fidelity in every school
Why This Instrument?
• Our Inter-Project vision: to enhance capacity of districts to support MTSS with fidelity in schools
• Current tools not adequate for assessing all components of MTSS
• Desire for an instrument to guide action planning towards improved implementation
SAM DEVELOPMENT
SAM Development Steps
1. Literature review2. Construct/theory development3. Item generation4. Expert review panel5. Cognitive interviewing6. Initial pilot (Fall 2013)7. National pilot (Spring 2015)
• “Gold standard” survey development procedures recommended by DeVellis (2012)
Initial Item Development
• Review of SAPSI and additional school-level measures of RtI/MTSS/PBIS
• Review of literature on RtI/MTSS/PBIS implementation, systems change, educational reform initiatives
• Feedback from Inter-Project Leadership Team and Project staff members
• Development of 3-point scoring rubric
Expert Review Panel
• 11 district-, state-, and national-level experts on RtI/MTSS and PBS implementation
• Feedback on item relevancy and clarity/conciseness
• 80% criterion used to identify quality items
• Items not meeting criteria revised by the evaluation team
e
Cognitive Interviews
• 6 cognitive interviews were conducted
• Provided feedback on difficult terms or jargon
• Items identified as problematic were revised by the evaluation team
• Common feedback from interviewees included:– Some items were too wordy
– Defining terms like “staff,” “stakeholders,” “implementation fidelity,” and “parent involvement”
– One person cannot have all the information needed to rate every item
Content Domains
• Leadership
• Building the Capacity/Infrastructure for Implementation
• Communication and Collaboration
• Data-Based Problem Solving
• Three-Tiered Instructional/Intervention Model
• Data-Evaluation
Leadership
• Administrators and school leadership teams are critical to implementing MTSS in schools
• Critical roles for leadership teams:– Continuous professional development on implementing MTSS– Strategically plan for MTSS implementation– Model a problem-solving process for school improvement
• Critical roles for administrators:– Communicating a mission and vision to school staff– Providing resources for planning as well as implementing
instruction and intervention– Ensuring staff have data necessary to engage in problem-
solving.
Building Capacity/Infrastructure for MTSS Implementation
• Critical components:– Ongoing professional development and coaching– Emphasis on data-based problem solving– Implementing a multi-tiered instruction and
intervention model– Scheduling that allows staff to engage in planning and
implementing instruction and intervention– Processes and procedures for engaging in data- based
problem-solving
Communication and Collaboration
• The following increase the likelihood that practices will be implemented and sustained – Consensus among key stakeholders
– Feedback to implementers to support continuous improvement
– Involving stakeholders in planning
– Infrastructure to communicate and collaborate with families and community partners
Data-Based Problem Solving
• Data-based problem solving at a school-, classroom-, and student-level is a critical to MTSS implementation
• Includes problem solving for student outcomes across content areas, grade levels and tiers, and to address barriers to school wide implementation of MTSS
• Four-step problem solving evaluated in this instrument includes, – 1) defining the goals and objectives to be attained– 2) identifying possible reasons why the desired goals are not being
attained– 3) developing and implementing evidence-based strategies to
attain goals– 4) evaluating the effectiveness of the plan
Three-Tiered Instructional/Intervention Model
• The three-tiered instructional/intervention model is another critical component of MTSS implementation
• In a typical model, the three tiers include:– Tier 1: instruction that all students receive– Tier 2: additional, supplemental instruction or intervention
provided to students not meeting benchmarks– Tier 3: intensive, small-group or individual interventions
provided to students exhibiting significant barriers to learning the skills required for school success
• Within an MTSS model, it is critical to consider both academic and social-emotional/behavioral instruction and interventions when examining this domain
Data/Evaluation
• Importance of data-based problem solving in MTSS establishes a need for data and evaluation system
• School staff need to understand and have access to data sources that address four purposes of assessment (i.e., screening, diagnostic assessment, formative evaluation, and summative evaluation)
• Efficient & useful data system, procedures and decision making protocols for assessing and using data
• Data on the fidelity of MTSS implementation for leadership to examine status and make systemic changes for improving implementation
Sample Items (Scoring Rubric 0-1-2-3)
Preliminary Pilot - Fall 2013
• 2 states, 7 districts, 155 schools• Reviewed descriptive statistics and qualitative
feedback• Revised survey (e.g., item clarity, errors, etc.)
and added 5 items based on feedback– Scheduling (2)– Disaggregating data across groups– Allocation of resources– Monitoring of data sources
2015 National Pilot: Sample
• 8 states, 15 districts, 436 schools
• School type– 269 elementary
– 75 middle
– 69 high
– 23 other (Alternative, Combined, etc.)
• Recruitment
SAM Domain Averages
Confirmatory Factor Analysis
• Strong Factor Structure (6-factor model)– 6-factor conceptual CFA based on 6 SAM domains– Chi-square test of fit = 1734.06 (p <.0001)– Root Mean Square Error of Approximation (RMSEA) = .059
(criteria < .08)– CFI = .96 (criteria > .95)
• High Internal Consistency Reliability .79-.91
SAM and Behavior Outcomes
• SAM (overall score) significantly correlated with BOQ: r(188)= .31, p<.001
• SAM (overall score) significantly correlated with Out of School Suspension (OSS) days r(243)= -.14, p<.05
SAM Feasibility Items
• Average time to complete: 1-1.5 hours
• Will SAM help your school implement?– Avg. score 3.55 (1: Not at all-5: Very)
• Will your school use the SAM again?– Avg. score 3.57 (1: Not at all-5: Very)
1 - Not at all Very -5
23
4
SAM – SWPBIS Fidelity Tools Crosswalk• All BoQ & BAT items were related with SAM items. • 7 SAM items were not on BoQ or BAT items.
– Infrastructure for family involvement (as opposed to practices), academic practices, procedures for databased decision making, and coaching.
• PBIS measures focused on components of a data system, but not problem solving itself
• Coaching and the use of data to problem solve and make decisions were not emphasized in the PBIS tools despite being considered critical feature of implementation.
More on Use of the SAM with SWPBIS
• Blended District or School Leadership Teams
• PBIS Coaches’ expertise in systems implementation
• Build connections through common features
• Other?
District Example: Broward County
Diversity, Prevention & Intervention
CLIMATE Grant
Data Collected Focus of ToolSAM MTSS with FidelityBoQ MTSS with FidelityCHAMPs 7 UP classroom management skillsHERO Usage Reinforcement of positive
behaviorsBASIS Documentation of strategies &
ODRsODRs Misbehavior frequencyInternal Suspensions out of seat timeExternal Suspensions out of seat timeReferrals to PROMISE possible arrest misbehaviorsAnnual Climate Survey Climate, safety
▪ Self-Assessment of Multi-Tiered System of Supports (SAM)
▪ SAM School-level Inventory
▪ Critical Component Checklist (CCC) Implementation Fidelity
▪ Observation Checklist Implementation of Fidelity
▪ Beliefs and Perceptions Skills Survey
▪ Needs Assessment
▪ Pre/Post Test Surveys
Quality Assurance & Evaluation Measure
31
32
33
Potential Data Sources
• Self-Assessment of MTSS (SAM)
• Tiers I/II and Tier III CCCs
• Tier I/II Observation Checklist, Problem-Solving Team Meeting Checklist
• PBIS Implementation data– Benchmarks of Quality (BOQ)
– PBS Implementation Checklist (PIC)
– Benchmarks for Advanced Tiers (BAT)
• Walkthrough data, lesson plans
• Tier 2 and 3 intervention documentation worksheets
• Others?
Data Dissemination
• Identify key stakeholders (e.g., District Leadership Teams, School Leadership Teams, instructional staff)
• Share data quickly and frequently• Use Questions to guide discussions• Graph data to more easily identify trends• Compare different types of MTSS fidelity data if
possible (self-report, permanent product reviews, observations)
• Inform professional development planning
Interested in Using the SAM?
• User’s Manual in Development
• Article on Validation in Development
• Email us!
Connect with Us!
Florida’s Problem-Solving/Response to Intervention Project
– http://www.floridarti.usf.edu/
– Email: [email protected]
– Facebook: flpsrti
– Twitter: @flpsrti
Florida Positive Behavioral Interventions & Support Project
– http://flpbs.fmhi.usf.edu/
– Email: [email protected]
– Facebook: www.facebook.com/flpbs
– FLPBIS on Twitter: www.twitter.com@flpbs
– YouTube: https://www.youtube.com/user/flpbs137
Contact Information
• Karen Childs– [email protected]
• Natalie Romer– [email protected]
• Kevin Stockslager– [email protected]
• Jose Castillo– [email protected]
• Amber Brundage– [email protected]
Including strands specifically designed for our Mental Health,
Juvenile Justice, and Family/Community Partners!
PBIS: Systems for Enhancing
Climate & CultureDonald Stephens Convention CenterRosemont, IL
Donald E. Stephens Convention Center
Rosemont, Illinois
This two-day forum for school, state, district and regional Leadership Teams and other professionals has been designed to increase the effectiveness of PBIS implementation.
Sessions are organized by strands that support initial through advanced implementation in elementary, middle, and high schools as well as juvenile justice facilities:
PBIS FoundationsClassroom Applications
Tier 2 Systems & Practices Tier 3 Systems & Practices
Aligning SystemsJuvenile Justice
Mental Health IntegrationEquity
Applied EvaluationSpecial Topics
Visit the Upcoming Events page at www.pbis.org for more
information
2016 National PBIS Leadership Forum