developmental disabilities program independent evaluation (ddpie) project
DESCRIPTION
Developmental Disabilities Program Independent Evaluation (DDPIE) Project. Jennifer Johnson Lynn Elinson Cynthia Thomas AUCD Annual Meeting October 31, 2006. Purpose of the Independent Evaluation. Demonstrate impact of DD Network programs on: Individuals Families Service providers - PowerPoint PPT PresentationTRANSCRIPT
Developmental Disabilities Program Independent Evaluation (DDPIE) Project
Jennifer JohnsonLynn ElinsonCynthia ThomasAUCD Annual MeetingOctober 31, 2006
Purpose of the Independent Evaluation
Demonstrate impact of DD Network programs on:– Individuals – Families– Service providers– State systems
Provide feedback to ADD to help improve the effectiveness of its programs and policies
Promote positive achievements of DD Network programs
Promote accountability to the public
DDPIE Project
Independent evaluation 2 phases Phase 1 – development and testing of tools Phase 2 – full-scale evaluation Westat – contracted by ADD to implement
Phase 1
Evaluation Tools
Measurement matrices- standards- indicators (structures, processes, outputs,
outcomes) - performance levels
Data collection instruments
Evaluation
Comparison
Standards Indicators
What do we hope to achieve? What do we observe (measurement of indicators)?
Are there differences/discrepancies?
What is the nature and extent of the differences?
What action needs to be taken?
Framework of indicators (RFP)
To organize and guide the development of the performance standards and related measurement matrices, the following framework of indicators of program impact should be used:
- Structural indicators – adequate and appropriate settings and infrastructures, including staffing, facilities and equipment, financial resources, information
systems, governance and administrative structures, etc.- Process indicators – activities, procedures, methods,
and intervention supporting practices - Output indicators –results of the DD Network’s policies, procedures, and services - Outcome indicators – intermediate results
Open Systems Model
Structure (Input) Process Output
(Product) Outcome
Effectiveness
Efficiency
Measurement Matrices
Tools in the evaluation that will organize indicators, standards, and performance levels for each key function within each DD Network program and collaboration
Developed and pilot-tested in Phase I Used in Phase II
Basic Evaluation Approach
Performance-based approach – interested in outcomes
Development of standards and indicators Development of measurement matrices that contain
standards, indicators, and performance levels Collection of data Measurement of indicators to determine level at
which standards are being met Determination of overall performance at the national
level
Key Assumptions
State programs vary on their level of performance across the standards.
Consistently high performance across the standards is related to better outcomes.
Consistently low performance across the standards is related to poor outcomes.
Validation
Advisory Panel Working Groups Validation Panels Pilot Study Further analysis
Role of Advisory Panel
To provide balance, impartiality, and expertise.
To provide advice on: DDPIE process Standards, indicators, performance levels, and data
collection Measurement matrices Pilot study Synthesis of findings and recommendations
Composition of Advisory Panel
Individuals with expertise on: DD population Policies and services for the DD population Evaluation research DD Network programs Other evaluations
Advisory Panel
Self-advocates Family members Representatives from 3 programs – Richard
Carroll from Arizona UCEDD Child/disability advocates Evaluation expert Federal representative (for PAIMI evaluation)
Working Group Members: Criteria for Selection
Have broad overview of all aspects of UCEDD (Director, Associate Director)
Rural/urban state Geographic distribution Placement of UCEDD in University
Validation Panels
Role: To endorse the contents of the measurement matrices.
Composition: - Stakeholders (consumers, advocates)- DD Network program staff - DD Council members - ADD staff- Evaluation experts
Pre-test and Pilot Study
States randomly selected Pre-test – 1 state Pilot Study – 4 states
Progress to Date
Reviewed background materials. Conducted preliminary/background interviews. Established and met with Advisory Panel twice. Established and met with Working Groups. Identified key functions of each DD Network program. Discussed structures, processes, outputs and outcomes of
each key function. Conceptualized measurement matrices. Developed early drafts of standards and indicators for each key
function.
Project Tasks to Do
Complete draft matrices. Share process with state programs. Validate matrices (Validation Panels) Develop data collection instruments. Obtain OMB clearance (ADD). Conduct pilot study. Analyze and synthesize data. Write report and recommendations.
DDPIE Project Milestones
Draft measurement matrices completed. Feb., 2007
Measurement matrices validated by panels. May, 2007
Data collection instruments completed. June., 2007
OMB clearance obtained. Dec., 2007
Pilot study conducted (training, pilot) Jan.-Sept., 2008
Analysis and synthesis of data. Feb.-Nov., 2008
Report and final recommendations sent to ADD Oct.-Dec., 2008
Key Functions
A collection of activities that are intended to achieve particular results
Examples:- P&A – individual advocacy, outreach/public education- DD Councils – systemic advocacy, development of community capacity - UCEDDs – training, community service/technical assistance, research,dissemination (from DD Act)
UCEDD Working Group Members
Carl Calkins Kansas City, MOTawara Goode Washington, DCGloria Krahn* Portland, ORDavid Mank Bloomington, INFred Orelove* Richmond, VAFred Palmer Memphis, TNLucille Zeph Orono, ME
*Collaboration Working Group
Working Group Meetings
Orientation by telephone – March, 2006 Telephone meetings (full group) in spring In-person meeting in spring, 2006 –
coinciding with the national meeting Telephone focus groups (teams) summer,
2006 In-person meeting in fall, 2006
UCEDD Key Functions
Provision of Interdisciplinary Pre-Service Preparation and Continuing Education of Students and Fellows
Provision of Community Services Conduct of Basic or Applied Research Dissemination Governance and Management? Priority Setting?
Members of UCEDD Working Subgroups
Pre-Service Preparation Fred Palmer Lucille Zeph
Community Services Tawara Goode Fred Orelove
Basic or Applied Research David Mank Gloria Krahn
Dissemination Carl Calkins Gloria Krahn
Working Group Teams:
Described goals for each key function Explained the main activities Identified outcomes Discussed variability across UCEDDs Provided Examples Helped formulate SPOO tables, standards
Interdisciplinary Pre-Service Preparation...
Goal: Develop a cadre of individuals with knowledge, skills, attitudes and values to increase the capacity of states to provide services and supports in a culturally and linguistically competent manner for people with developmental disabilities and their families.
Interdisciplinary Pre-Service Preparation...(cont’d)
Function includes: Developing and teaching courses in the core curriculum Developing and/or teaching disability content for courses in other
departments Developing and teaching classes that offer continuing education
creditsOutcomes Include: “Graduates” from the preservice programs who demonstrate
knowledge, skills, and attitudes and values consistent with …. the principles of the DD Act.
Students who complete courses with disability content and demonstrate attitudes and values ….
Trainees who complete continuing education classes …..
Following Steps:
Development of written descriptions of each key function
Development of draft indicators Full working group reviews descriptions,
standard, indicators