replacing “ready, aim, fire” with “research, inform, action”
TRANSCRIPT
Today’s Session
The Kansas City STEM Alliance and the Kansas City Area Education
Research Consortium collaborate with schools and nonprofits to recruit and
transition youth into STEM programs across the metropolitan area. Come
explore how research with local students, schools, out-of-school programs,
and community volunteers informs program decisions and illustrates a
return-on-investment of resources leveraged by businesses, K-12 schools,
higher education, and foundations. Also participate in discussions about
how what Kansas City is learning can be translated to inform efforts in your
region.
2
Our VisionThe vision of the KC STEM Alliance us is to see that
a diverse, innovative and sustainable STEM
workforce becomes a reality.
By developing an environment that leverages the
strengths of educators, STEM organizations, and
local industry we can create a collaborative network
to encourage and sustain STEM careers.
3
Working together will allow this…
Building on Our Success: 2006-2012
• More than 27 area school districts and higher education
partners to provide curriculum, professional development for
teachers
• Nationally recognized Project Lead the Way (PLTW) and
US FIRST - 15,000 students in the Kansas City Metro
area
• Since 2006 – well over 20,000 students have participated in
these two programs.
• Significant resources – private, public sectors and school
districts
5
Our strategy
• Ongoing program evaluation and data
collection
• Increase participation of existing initiatives
• Seek out new partnerships and opportunities
• Raise awareness about STEM in Kansas City
• Secure support from area companies and
organizations
6
Beyond Descriptive
• KC-AERC works with local education programs to
help create measures of program effects on
students
• Traditional quantifiable measures – Ex. Reading or Math gains pre and post tests
• Non-quantifiable skill measures– Ex. Socio-emotional skills, attitudes towards
science
Methods
• Discuss desired program goals and effects with
program leaders
• Research appropriate instruments for analyzing
effects on students
• Devise appropriate statistical model– Mixed methods of quantitative and qualitative
• Collaborate with program leaders on final design
Quantitative Methods
• Treatment and control groups to identify if effects can be
linked to program– “Gold Standard” in attempting to find causality– Many times difficult to find and engage a control group
• Regression analysis to factor a variety of background
characteristics into results
• Use of Pre and Post tests to identify growth through
program (T-tests, ANOVA)
Qualitative Methods
• Focus groups to gather thoughts from those
closest to subject area
• Use quantitative methods to identify strong and
weak areas of program, qualitative investigation
of why some aspects work
• Interviews with leaders for in depth details of
program implementation
13
• Lower participation rate among minorities (when
compared to overall school population – FIRST)
• Low female participation in STEM engineering
programs – but, high in biomedical programs• Higher participation of females in urban districts
• Urban programs are less robustenrollment, participation, funding and support
What did we learn in the first 6 months:
What are issues raised by these pilots?
Process Data for Program Improvement:
• Comprehensive Program
Implementation with School Districts
• Definitions of Program Participation
• Understanding Within-Group Variation
in Number of PLTW Courses Complete
• Understanding FIRST Robotics
Participation Systematically
• PLTW Course Outcome Data
Outcome Data for
Program Impact:
High School Data
Formats and
Availability
Data on College
Outcomes
Highlights of FIRST Robotics Surveys 2011-12
– FIRST Robotics, the majority of participants are White and Male.
– Majority have lived in the United States their whole life, come from households where English is spoken, and have highly educated parents (i.e., college degree or beyond) who in turn hold high expectations for their children.
– Slightly more than half of students surveyed (54%) were participating in FLL for the first time.
Highlights of FIRST Robotics Surveys
FIRST Coaches reported:
• 72% participating in a FIRST competition
• Of those participating, 44% participated in more than one competition.
• The number of mentors per team ranged from 0-15, with 24% of coaches
reporting their teams did not have even one mentor.
• Every single coach whose team had a mentor(s) reported their mentors
helped in “mechanical component design;” less than half of coaches
reported that their mentors helped with marketing, business plans,
computer applications, or website development.
Highlights of FIRST Robotics Surveys
FIRST Volunteers/Mentors (N = 77) reported:
• Majority of Event Volunteers were White and Male.
• Almost all volunteers lived in the Kansas City metropolitan area.
• Majority employed full-time and represented a variety of occupations.
• When asked about their satisfaction with their work in the FIRST Robotics
program, they gave responses that were generally very positive; however,
there were relatively less positive responses to questions about the
perceived effectiveness of information provided about the volunteer job.
Typically,
“EVALUATION” =
program level and
“ASSESSMENT” =
individual level
TYPES of DATA:
• Participant Level (interest,
engagement, achievement)
• Program Level (quality
activities, professional
development)
• Systems Level (Policy
changes)
Research, Inform, Action
• Two-Way Street
• Evaluation Planning to Implementation to
Utilization– Step 1: Program Purpose– Step 2: Data– Step 3: Plan to Obtain Data– Step 4: Communication/Share Data
Step 1: Document the nature and purpose of the program
• What is the program?
• What is the need?
• How is the program meeting the need?
• What are the goals of the program in terms of measurable
outcomes?– Short-term– Long-term
Note: Increases in knowledge, skills, and attitudes are
laudable goals.
Research, Inform, Action
• Inputs
• Outputs
• Measurable Outcomes/Goals
• Evidence/Possible Sources of Evidence
(Data)
Step 2: Identify data needed to measure progress against goals
• Choose just the important data; avoid collecting data
you will not use
• Look for data among the people you directly serve
• Consider:– Who, when, and how often participants attend – What participants know and can do
Note: Evidence of learning is more valuable than
what participants say they learned.
Step 3: Develop a plan to obtain the data with current resources
• Brainstorm with team: –What constitutes success?–What evidence can be collected?–What notes/data can frontline staff
record?–What data do you need assistance with
gathering and understanding?• RE – visit goals to see if they capture everything that is important
Step 4: Share how the data will be used
• Provide feedback to stakeholders about how to
improve what we do
• Information to different audiences:– Current and future funders– Schools/School districts– Parents/Students/Families– Participants
Research, Inform, Action
• Two-Way Street
• Evaluation Planning to Implementation to
Utilization– Step 1: Program Purpose– Step 2: Data– Step 3: Plan to Obtain Data– Step 4: Communication/Share Data
Replacing “Ready, Aim, Fire” with “Research, Inform, Action”
Dr. Leigh Anne Taylor Knight
Executive Director, KC-AERC
913-396-3214
Laura Loyac0no
Director, KC STEM Alliance
816-665-3823