sophia gatowski, ph.d., consultant national council of juvenile & family court judges sophia...
TRANSCRIPT
Sophia Gatowski, Ph.D., Consultant National Council of Juvenile & Family Court Judges
Sophia Gatowski, Ph.D., Consultant National Council of Juvenile & Family Court Judges
CIP Meeting, Aug 2009CIP Meeting, Aug 2009
Is Your Training Program Making a Difference? Is Your Training Program Making a Difference? A National Implementation Guide to
Conducting an Effective Training Evaluation
Why a Dependency Court Training Evaluation Guide?Why a Dependency Court Training Evaluation Guide?
• The Need
• The Vision– NCJFCJ, ABA, NCSC
• Funded by Children’s Bureau
© 2009, NCJFCJ
What we learned …What we learned …
Interviews with training administrators, and review of past CIP, Model Court and other Dependency Court training evaluation materials, found: – Constraints on evaluation activities
• Funding• Staff• Time
– Most evaluations limited to exit surveys• Numbers trained & roles trained• Reactions to training (faculty, content, venue)• Satisfaction with overall experience (expectations)
What we learned …What we learned …
– Behavior/ attitude change rarely measured • If measured mostly self-report
– Training impacts/ outcomes rarely measured• Few quasi-experimental or experimental designs• Lack of generalizability
– Measurement issues• What is being evaluated? • Multi-method approaches infrequent
Training Evaluation GuideTraining Evaluation Guide
Goals:• Define basic criteria for judicial training evaluation tools;• Provide evaluation guidance and tools that allow for
national data collection on training outcomes;• Help training administrators to design evaluation tools
that go beyond measuring “reactions” to training (more about this later);
• Provide guidance to CIP, Model Courts and others in designing and implementing effective training evaluations.
Training Evaluation GuideTraining Evaluation Guide
TRAINING DESIGN TRAINING IMPLEMENTATION IMMEDIATE
EVALUATION FOLLOW-UP EVALUATION
OUTCOMES
Guide Development ProcessGuide Development Process
• Conducted interviews with leading educators and training evaluators
• Extensive review of literature and resources on effective training models
• Obtained and reviewed past CIP, Model Court and other dependency court training evaluation materials
• Convened an Advisory Committee– CIP Directors, judicial educators, evaluation experts
Training Evaluation Basics Training Evaluation Basics
Chapter 1: Essential Components of Effective Evaluation– Designing and planning trainings with evaluation in
mind• Overview of effective training designs and strategies• Consideration of adult learning principles• Learning objectives, goal and outcome specification• Plan for using the evaluation data
Training Evaluation BasicsTraining Evaluation Basics
Chapter 2: The Basics of Training Evaluation– Purpose and goals for training evaluation
• Measurement issues (Constructs, operational definitions, validity, reliability)
• Logic Models
– 4 levels for training evaluation• Satisfaction• Learning/ knowledge acquisition• Behavior change• Impact of training on outcomes
Training Logic Model Example
The Levels of Training EvaluationThe Levels of Training Evaluation
Chapter 3: Level 1- Satisfaction Evaluation– Strategies (when to use them, how to use them)– Recommendations for implementation
• Question construction tips• Data collection strategies
– Template satisfaction evaluation• Reaction tools
– Analysis• Sample report formats• Taking satisfaction analyses to the next level
Reaction Tool Example
The Levels of Training EvaluationThe Levels of Training Evaluation
Chapter 4: Level 2- Learning/ Knowledge Acquisition Evaluation– Strategies (when to use, how to use)– Recommendations for implementation
• Question construction tips• Data collection strategies
– Using quasi-experimental designs• Advantages, challenges, and strategies
– Template learning/ knowledge evaluation– Analysis
• Sample report format
Skill Test Example
The Levels of Training EvaluationThe Levels of Training Evaluation
Chapter 5: Level 3- Behavior Change – Strategies (when to use, how to use)– Recommendations for implementation
• Question construction tips• Data collection strategies
– Using quasi-experimental designs• Advantages, challenges, and strategies
– Template learning/ knowledge evaluation– Analysis
• Sample report format
Self-report Example
Court Observation Example
The Levels of Training EvaluationThe Levels of Training Evaluation
Chapter 6: Level 4- Impact/ Outcomes – Strategies (when to use, how to use)– Recommendations for implementation
• Resource needs (time, staff)• Measurement issues
– Using quasi-experimental and experimental designs• Advantages, challenges, and strategies
– Sample outcome evaluation designs– Analysis
• Analytic approaches• Sample report formats
Putting it All TogetherPutting it All Together
Chapter 7: Combining Levels of Evaluation for a Multi-Tiered Approach– Integrating different evaluation levels into a complete
tool– Steps for creating an evaluation sequence
Evaluation Sequence Example
Appendix– Evaluation templates– Additional readings and resources
We want to hear from you …We want to hear from you …
• What advice do you have as we finalize the development of the Training Evaluation Guide?
• Are there particular areas we should focus more on?
• Do you have any “cool” and effective evaluation strategies or tips you would like to share?
• Other thoughts or recommendations?
Next Steps ….Next Steps ….
• Complete chapters and templates– Incorporate additional ideas, strategies
[Please complete our Follow-up Sheet]
– Review of Guide by advisory committee
– Finalize and disseminate to all CIP, Model Courts
Contact InfoContact Info
Nancy Miller, DirectorPermanency Planning for Children Dept.,
National Council of Juvenile & Family Court Judges
Sophia Gatowski, Ph.D., ConsultantNational Council of Juvenile & Family Court Judges