evaluation in gifted programs
DESCRIPTION
Evaluation in Gifted Programs. June 14-15, 2013 University of Iowa. Contact Information. Mary Schmidt 9219 Lakewood Pointe Drive Norwalk, IA 50211 515.321.6090 [email protected]. Agenda Day 1. Welcome/Introductions Outcomes What is Program Evaluation? Guiding Principles - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/1.jpg)
Evaluation in Gifted Programs
June 14-15, 2013University of Iowa
![Page 3: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/3.jpg)
Agenda Day 1
Welcome/Introductions Outcomes What is Program Evaluation? Guiding Principles Connecting PE to Standards Program Evaluation as Continuous
Improvement Best Practices in Program Evaluation Framing the Question Data Skills
![Page 4: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/4.jpg)
Ten Things in Common
As a group– Identify ten things you all have in
common (no body parts)– 7 minutes– Discuss what you learned about
classmates
![Page 5: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/5.jpg)
Your Thoughts
What do you know about program evaluation?
What do you want/need to know?
![Page 6: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/6.jpg)
Outcomes
Develop an understanding of the purposes, procedures, and outcomes of program evaluation including formative and summative evaluation
Learn program evaluation models and procedures
Develop questions to drive local program evaluation
Explore data types, purposes, and skills
![Page 7: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/7.jpg)
Expectations
Grade A: Completion of all in-class assignments
Program brochureProgram evaluation plan
Final course reflection Grade B:
Completion of all in-class assignmentsProgram evaluation planFinal course reflectionOr, submissions for an A grade are all completed but
are not of sufficient quality to merit the higher grade.
![Page 8: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/8.jpg)
Managing Complex Change
= Change
Confusion=
=
=
=
=
+ + + +Vision Skills Incentives ResourcesGifted Plan
Adapted from Knoster, T., Villa R., & Thousand, J. (2000). A framework for thinking about systems change. In R. villa & J. Thousand (Eds.), Restructuring for caring and effective education: Piecing the puzzle together (pp. 93-128). Baltimore: Paul H. Brookes Publishing Co.
Anxiety
Resistance
False Starts
Frustration
+ + + +Skills Incentives ResourcesGifted Plan
+ + + +Vision Incentives ResourcesGifted Plan
+ + + +Vision Skills ResourcesGifted Plan
+ + + +Vision Skills IncentivesGifted Plan
+ + + +Vision Skills Incentives Resources
![Page 9: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/9.jpg)
Discuss in a small group…
How effective is your gifted program?
How do you know?
Record your thoughts on TodaysMeet.
http://todaysmeet.com/UI-ProgEval2013
![Page 10: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/10.jpg)
Close Enough…
“There is no such thing as professionalism without a commitment to evaluation of whatever it is that one supervises or produces—and to self-evaluation as well.”
—Michael Scriven, 2000
![Page 11: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/11.jpg)
IN LIGHT OF THIS…
NO MORE
CARDIAC
DATA!!
![Page 12: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/12.jpg)
![Page 13: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/13.jpg)
Program Evaluation Is…
…the systematic collection of information about the activities, characteristics, and outcomes of
programs to make judgments about the program, improve program
effectiveness, and/or inform decisions about future programming.
--Robinson, 2009
![Page 14: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/14.jpg)
What is Program Evaluation?
Chapter 21 “Evaluating Gifted Programs” by James Borland
Discuss the 5 A’s – What assumptions is the author
making?– What would you argue with?– What would you agree with?– What would you aspire to?– What would you ask?
![Page 15: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/15.jpg)
Best Practices
Based on information in Chapters 1 & 2, what are some of the best practices in gifted program evaluation?
![Page 16: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/16.jpg)
Guiding Principles & Attributes
Read p. 196-198 in Designing Services and Programs for High-Ability Learners
Discuss in a small group connections, insights, and questions
Complete the assessment on p. 203-4– Individually– Discuss with a partner– Large group sharing – one strength & one
growth area
![Page 17: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/17.jpg)
National Standards
Visit http://www.nagc.org/index.aspx?id=546
Peruse the revised standards, paying special attention to #2.
Discuss with your partner the guidance these standards provide.
Do the same for Chapter 59 of Iowa Code (handout)
![Page 18: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/18.jpg)
Making Connections
![Page 19: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/19.jpg)
Pro
gra
m E
valu
ati
on a
s C
onti
nuous
Impro
vem
ent
![Page 20: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/20.jpg)
Best Practices in Program Evaluation
Discuss in small groups:– What’s been the program evaluation
practice in your district?– How well do these practices align
with best practice?– What’s working?– What needs to be changed/improved?– How have you used the evaluation
results?
![Page 21: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/21.jpg)
Memory Mingle
1. How has the information you’ve engaged with so far “pushed” your thinking?
2. Share in triads3. Identify connections and/or
conclusions4. Share with large group
![Page 22: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/22.jpg)
Read “Asking the Right Questions: The Central Issue in Evaluating Programs for the Gifted and Talented” (ICON)
Read “General Evaluation Guidelines” found on ICON.
![Page 23: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/23.jpg)
Asking the Right Questions
Work with a partner Identify three big ideas and
discuss how these big ideas play out in your setting.
What questions does the article raise for you?
![Page 24: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/24.jpg)
“…one cannot evaluate what one cannot describe.”(Callahan, p.
41)
Describe your program in terms of its components.
Consider what’s common and different across grade levels/spans/buildings.
Identify audiences for this information
What questions naturally emerge?
![Page 25: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/25.jpg)
Connecting Questions to Data Sources
Consider one to three of your most pressing questions.
Do you have data already available that will help answer the question(s)?
What additional data will you need to collect?
![Page 26: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/26.jpg)
Developing a Program Brochure
Consider:– Purpose(s)– Audience(s)
Create the rough draft of a program brochure using the description you wrote earlier as the basis.
![Page 27: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/27.jpg)
We geek Data Consider the following questions related
to a framework for program evaluation:– What questions do we have about our program?– What data do we need to answer the questions?– What data do we already collect?– Who do we collect data on or from?– How often do we collect it?– How do we use the results?– How and to whom to we provide feedback?– What PD do we provide? How do we identify PD
needs?– Do we have an action plan?
![Page 28: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/28.jpg)
Data Skills
Find a partner Make a list of what you think
qualifies as data skills Identify the data skills you have.
IOW: What do you know how to do regarding data collection, organization, and use?
![Page 29: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/29.jpg)
Data Strengths and Needs
Complete the Data Skills Self-Assessment.
Discuss results with a partner. Set a data improvement goal. What does your district need to
focus on in terms of gifted program evaluation?
![Page 30: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/30.jpg)
Is your home energy efficient?
How would you know?
![Page 31: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/31.jpg)
Is y
our
hom
e e
nerg
y
effi
cient?
![Page 32: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/32.jpg)
![Page 33: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/33.jpg)
![Page 34: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/34.jpg)
Reflecting on the day…
What have you heard/learned today that reinforces your ideas about program evaluation?
What have you heard/learned today that’s changed your thinking about program evaluation?
What’s your most important take away from the day?
![Page 35: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/35.jpg)
Evaluation in Gifted Programs
Day 2June 15, 2014
![Page 36: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/36.jpg)
Agenda Welcome Back Types of Evaluations Evaluation Models Data Types, Purposes, and Uses Developing an Evaluation Plan
![Page 37: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/37.jpg)
Welcome Back!
Form triads Share
– What do you know about your program today that you didn’t know yesterday?
– How did you acquire this knowledge?– How can this information be used to
improve programming?
![Page 38: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/38.jpg)
Program Evaluation Is…
…the systematic collection of information about the activities, characteristics, and outcomes of
programs to make judgments about the program, improve program
effectiveness, and/or inform decisions about future programming.
--Robinson, 2009
![Page 39: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/39.jpg)
Types of Evaluations
Planning– Needs Assessment
Formative– “Miracle in the Middle”
Summative– Impact
![Page 40: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/40.jpg)
![Page 41: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/41.jpg)
![Page 42: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/42.jpg)
![Page 43: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/43.jpg)
![Page 44: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/44.jpg)
Program Performance Measures:
Questions about Program Service Delivery
Quantity(How much did we do?)
Quality(How well did we do
it?)
Input(Effort)
How much service did we deliver?
How well did we deliver service?
Output(Effect)
How much effect/change did
we produce?(#)
What Quality of effect/change did
we produce?(%)Is anyone better off?
![Page 45: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/45.jpg)
Black Box Evaluation
Glass Box Evaluation
Input Output
Input Output
![Page 46: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/46.jpg)
Worth
The extent to which a program or activity is essential to a school’s, district’s, agency’s, or individual’s mission. Worth is an indication of the program’s or activity’s perceived value to constituents or to a single individual.
--Assessing Impact training, 2003
![Page 47: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/47.jpg)
Merit
The value of the program is judged by comparing its performance against established standards of excellence in the profession.
--Assessing Impact training, 2003
![Page 48: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/48.jpg)
How Could a Program Have…
…worth but no merit?
…merit but no worth?
…both worth and merit?
![Page 49: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/49.jpg)
Thus
A program may have great merit yet be of little worth because it is not aligned with the organization’s mission or needs or it may have great worth and little merit…programs can be evaluated both on the basis of their worth and merit.
--Assessing Impact training, 2003
![Page 50: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/50.jpg)
Program Evaluation Models and Tools
Arkansas Evaluation Initiative Borland Evaluation Template Maker’s Responsive Model Self-Audit/Reflection Tool
![Page 51: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/51.jpg)
AEI
Designed to support in-house (formative) evaluation
Don’t need to use all data sources
![Page 52: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/52.jpg)
Borland Template
![Page 53: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/53.jpg)
![Page 54: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/54.jpg)
Types of Data
Outcome– What they got
Key Performance Indicators
Demographic– Who got it
Disaggregation
Process– How they got it
Change Agents (data that guides you toward change)
![Page 55: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/55.jpg)
![Page 56: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/56.jpg)
Which Type of Data?
Percentage of… students identified for gifted services. students scoring a 4 or 5 on the AP exam teachers employing curriculum
compacting the day students receive gifted services students who attain PEP goals identified minority students identified students who are accelerated
(in each acceleration category)
![Page 57: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/57.jpg)
Program Performance Measures:
Questions about Program Service Delivery
Quantity(How much did we do?)
Quality(How well did we do
it?)
Input(Effort)
How much service did we deliver?
How well did we deliver service?
Output(Effect)
How much effect/change did
we produce?(#)
What Quality of effect/change did
we produce?(%)Is anyone better off?
![Page 58: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/58.jpg)
Types of Evaluations
Planning– Needs Assessment
Formative– “Miracle in the Middle”
Summative– Impact
![Page 59: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/59.jpg)
Data Use and Purpose
Download the document Work as a team to complete Large group processing
![Page 60: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/60.jpg)
Data Analysis Questions
What process do we have in place to use data in the areas of curriculum, instruction, and assessment? How does this align with the general curriculum process in the district?
What types of data do we regularly collect? Are there other things we need to collect?
What processes and techniques do we have to organize and analyze our data? What do we need?
What data do we want/need to disaggregate? How are we using data to determine areas of
maintenance, improvement and focus? How effectively are we using data to make
decisions?
![Page 61: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/61.jpg)
Let your direction determine your decisions.Let your direction determine your decisions.
![Page 62: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/62.jpg)
Developing an Evaluation Plan
Consider the place of planning, formative, and summative evaluations in your plan.
What’s the timeline? What will be evaluated and how and by
whom will the evaluation questions be formulated?
Visit the Wiki (link on ICON) for samples and templates.
![Page 63: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/63.jpg)
Required Submissions Develop a program brochure.
– Visit ICON and review the link (under Course Information) that provides steps and tips to develop a brochure.
– Make sure you include Logo District name Clear purpose Distinct audience Layperson-friendly language
![Page 64: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/64.jpg)
Required Submissions Develop a program evaluation plan. Include formative and summative
components Use the template on p. 203-4 to help
you develop the framework Exact format is up to you, but make it
flexible enough that the evaluation question(s), data collection instruments and methods, and communication of results could change from year to year within the same framework.
![Page 65: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/65.jpg)
Required Submissions
Reflection on your learning in this class.– How does your evaluation plan align
with your written plan?– What do you still need to learn/do to
successfully evaluate your program?– How do you see program evaluation as
a tool and a process to improve programming?
![Page 66: Evaluation in Gifted Programs](https://reader036.vdocuments.net/reader036/viewer/2022081513/56813711550346895d9e9795/html5/thumbnails/66.jpg)
Submissions
A copy of your brochure Evaluation plan including at least
one survey Reflection Due June 29