demonstrating the impact of careers guidance lester oakes president iaevg karen schober...
TRANSCRIPT
![Page 1: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/1.jpg)
Demonstrating the Impact of Careers Guidance
Lester Oakes
President IAEVG
Karen Schober
Vice-president IAEVG
Bryan Hiebert
Vice-president IAEVG
1
![Page 2: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/2.jpg)
A Challenge from Policy Makers
You say you are providing effective services
We believe you
BUT
Show us the evidence
2
![Page 3: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/3.jpg)
Policy-Practice-Research Dialogue
1. 1999: First International Symposium on Career Development and Public Policy
2. Establishment of International Centre for Career Development and Public Policy• IAEVG has been an active supporter of the
International Centre
3. International symposiums held in Canada, Australia, Scotland, New Zealand
4. 2009: Sixth International Symposium on Career Development and Public Policy
5. Predominating theme was Prove it Works3
![Page 4: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/4.jpg)
Overview
IAEVG has been an active participant in past symposiums and outcome-focused, evidence-based, practice has been an important part of IAEVG strategic planning• Update from European Lifelong Guidance
Policy Network • Update from Canadian Research working
Group on Evidence-based Practice in Career Development
4
![Page 5: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/5.jpg)
Karen Schober
will update us on what is happening in the European Lifelong Guidance Policy Network
5
![Page 6: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/6.jpg)
6
Outcome Focused Evidence-Based Practice
Input Process Outcome
Framework developed by the Canadian Research Working Group
on Evidence-Based Practice in Career Development
![Page 7: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/7.jpg)
7
Outcome-Focused Evidence-Based Practice
Input Process Outcome
Indicators of client change
1.Learning outcomes• Knowledge and skills linked to intervention
2.Personal attribute outcomes• Changes in attitudes,• Intrapersonal variables (self-esteem, motivation, independence)
3.Impact outcomes • Impact of #1 & #2 on client’s life, e.g., employment status, enrolled in training• Societal and relational impact• Economic impact
![Page 8: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/8.jpg)
8
Outcome-Focused Evidence-Based Practice
Input Process Outcome
Activities that link to outputs or deliverablesGeneric interventions• Working alliance, microskills, etc.Specific interventions1. Interventions used by service providers
• Skills used by service providers• Home practice completed by clients
2. Programs offered by agency3. Involvement by 3rd parties
![Page 9: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/9.jpg)
9
Evidence-based Outcome-focused Practice
Input Process Outcome
Resources available1. Staff
• Number of staff, level of training, type of training2. Funding
• Budget3. Service guidelines
• Agency mandate4. Facilities5. Infrastructure6. Community resources
![Page 10: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/10.jpg)
10
Outcome-Focused Evidence-Based Practice
Input Process Outcome
Intervention =
Process + Outcome
What will I do? + How is it working?
Professional Practitioner
![Page 11: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/11.jpg)
11
Evidence-based Practice
1. Research trials (traditional way in psychology)2. Professional Practitioner
• Purposeful intervention• Data (= evidence) on what was done• Evidence on client change• Look for patterns in data
linking intervention with outcome• Develop scientific attitude toward practice
Skepticism Curiosity Inquiry
![Page 12: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/12.jpg)
Intervention Planning & Intervention Evaluation
Client Outcomes• Knowledge• Skills• Attributes• Impact
Context:Client Needs
Client Goals
Counsellor Strategy
Client Strategy
Intervention Planning Framework
123
1212
![Page 13: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/13.jpg)
Intervention Planning & Intervention Evaluation
Client Outcomes• Knowledge• Skills• Attributes• Impact
Context:Client Needs
Client Goals
Counsellor Strategy
Client Strategy
OutcomesProcessesInputs
Intervention Planning Framework
Intervention Evaluation Framework
13
![Page 14: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/14.jpg)
14
Quality of Service Delivery
1. Accessibility• Regular hours
• Extended hours
• Physical accessibility
• Resources in alternate format
• Ease of access, who can access
2. Timeliness• % calls answered by 3rd ring
• Wait time for appointment
• Wait time in waiting room
3. Responsiveness• Respect from staff• Courteous service• Clear communication
4. Overall satisfaction• % rating service good
or excellent• % referrals from other
clients
Need to negotiate these with funders
![Page 15: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/15.jpg)
Comprehensive Service Evaluation
Quality Service frameworkService delivery
Client volumesClient presenting problemsNumber of sessions
Service standardsStaff credentials, competencies, resourcesEfficiency (at client needs being met?)
System requirementsAdherence to mandateCompletion of paper workCost-effectiveness
15
![Page 16: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/16.jpg)
Negotiated outcomes If you are lucky, funders might identify personal
attributes [client motivation, improved job satisfaction, increased self-confidence] or knowledge, or skills, as accountability indicators
BUT more likely funders will identify impact outcomes [employment status, enrolment in training, reduced # of sick days, increased productivity, etc.] or inputs [client flow, accessibility, timeliness of paper work, etc.]
So service providers need to identify the knowledge, skills, personal attributes that will produce the impacts and negotiate these as accountability indicators
Be careful what you promise to deliver BUT
deliver what you promisePromise small – DELIVER BIG
16
![Page 17: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/17.jpg)
17
Assessment as Decision Making (vs. Judgement)
Please use a two-step process
1. Would you say that your level of mastery of the attribute under considerations is
1. Then assign the appropriate rating 0 = really quite poor 1 = just about OK, but not quite 2 = OK, but just barely 4 = really very good 3 = in between barely OK and really good
acceptableunacceptable
0 1 2 3 440
![Page 18: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/18.jpg)
Problem with skill self-assessmentParticipants asked to rate their skill
(or knowledge) before and after a programOften, pre-workshop scores are high
and post-workshop scores are lower• People find out as a result of the workshop that they knew
less than they thought or had less skill than they thought• Based on the new awareness, post-scores are lower
People don’t know what they don’t knowHow can we get around this problem?
![Page 19: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/19.jpg)
19
Assessing Learning & Attribute Outcomes
Post-Pre Assessment
We would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now?
Please use a two-step process:• Decide whether the characteristic in question is
acceptable or unacceptable, then• assign the appropriate rating
acceptableunacceptable
0 1 2 3 440
![Page 20: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/20.jpg)
CRWG: Ongoing Projects
1. Validate framework and approach• ACT evaluation
2. Field test interventions • Develop evaluation component as part of
intervention• SME project
3. Field tests of practitioner use• LMI project
4. On the horizon
20
![Page 21: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/21.jpg)
Applied Career Transitions Program
Module 1 Building Career
Foundations
Module 2 Developing Career
Opportunities
Module 3Getting Experience:
The Internship
• On-line Program Curriculum
• Program Access Options
On-line only
Coached (four coaching appointments per module)
Coached with In-class Group Sessions (four coaching appointments and four group sessions per module)
21
![Page 22: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/22.jpg)
Results: Post-Pre AssessmentFor Module 1
• All together there were10 (items) x 29 (participants) = 290 ratings• Pre: 144 Unacceptable Ratings and 6 Exceptional Ratings • Post: 3 Unacceptable Ratings and 130 Exceptional Ratings • Exceptional Ratings increased from 2 to 44% of the participants• Pre: 50% Unacceptable Ratings; Post: 86% Acceptable Ratings
22
![Page 23: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/23.jpg)
Results: Impact outcomes
Module 1• 23 out of 29 had found a job• 10 of the jobs lined up well with career vision
Module 2• 4 out of 6 had found a job• 3 of the jobs lined up well with career vision
23
![Page 24: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/24.jpg)
24
Attribution for Change
To what extent would you say that the changes depicted above were the result of completing Module 1 of the ACT program, and to what extent were they a function of other factors in your life?
mostly other
factors
somewhat other
factorsuncertain
somewhat this
program
mostly this
program
0 0 0 10 19
![Page 25: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/25.jpg)
WSI ProjectCareer Development in SMEs
Career Development interventions for Small and Medium Enterprises to
promote personal ownership for career planningand within-organization career mobility
Three types of intervention Minimal: Web-based self-help intervention Moderate: On-the-job career conversations with managers,
supervisors, colleagues
Intensive: Bilan des competences
25
![Page 26: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/26.jpg)
WSI Project
General approach
1.Literature review• What is know already about the three areas
of intervention
2.Formal needs assessment
3.Design intervention AND evaluation plan
4.Field test intervention
5.Consolidate evaluation evidence
6.Final report
7.Market results
26
![Page 27: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/27.jpg)
WSI Project
General results
1.All 3 interventions worked
2.Different programs• designed for different purposes • requiring different resources• Chose the approach that best meets
organizational needs
3. It’s better to do something than nothing
27
![Page 28: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/28.jpg)
Practitioner Field Tests
1. Use and impact of Labour Market Information (LMI)• Isolate LMI from other interventions
2. Participant research approach• Normal clients seeking service• Agencies offering services
3. Use Post-Pre approach
28
![Page 29: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/29.jpg)
Research Design
29
Intervention
Delivery
Job Search
CDM
Independent
Time 2
Time 1
Assist
ed
![Page 30: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/30.jpg)
LMI: General Results
1. All intervention-delivery combinations produced significant change
• General ability to access and use LMI• Knowledge about how to use LMI• Skills for using LMI and taking action• Personal attributes, e.g., optimism, confidence
2. Assisted use produced greater change across time than independent use
3. 80% of clients attribute change to the program and not other factors
30
![Page 31: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/31.jpg)
On the Horizon
Common Indicators Project
1. Identify 3-5 indicators of success that all agencies will collect• Focus groups to identify outcomes and data
source
2.Aggregate results across agencies
3. Increase power of results
31
![Page 32: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/32.jpg)
Resources Available• Major report on evaluation practices• Special issue of Canadian Journal of Counselling• Sample tools and data gathering instruments• Evaluation workbook• Presentation notes, power point slides, etc.• Survey/needs assessment for self-help intervention
http://www.crwg-gdrc.ca32
![Page 33: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/33.jpg)
33
To demonstrate value, we need to develop
Culture of evaluation:We need to reach the state where
• Identification of outcomes is an integrated part of providing services Without efficacy data, career services are vulnerable It is in our best interest to gather evidence attesting to
the value of the services we provide• Measuring and reporting outcomes is integrated into
practice• Outcome assessment is a prominent part of counsellor
education• Reporting outcomes is a policy priority
This needs to be a priority in all sectors
![Page 34: Demonstrating the Impact of Careers Guidance Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG 1](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649ddb5503460f94ad2618/html5/thumbnails/34.jpg)
Demonstrating the Impact of Careers Guidance
Questions and Comments
Lester OakesPresident IAEVG
Karen SchoberVice-president IAEVG
Bryan HiebertVice-president IAEVG
34