Counseling & Psychological Services

MORE THAN SATISFACTION:ASSESSING CLINICAL

OUTCOMES IN COUNSELING CENTERS

International Assessment and Retention Conference

New Orleans, LAJune 13, 2009

Counseling & Psychological Services

Today’s PresentersJacqueline Alvarez, Ph.D.

Director of Counseling and Psychological Services, Oregon State UniversityCAPS Assessment Committee Member

Beth Dyer, B.A.Information Technology Expert at Counseling and Psychological Services, Oregon State University CAPS Assessment Committee Chair. Student Affairs Assessment Council Member

Michele Ribeiro, Ed.D.Staff Psychologist at Counseling and Psychological Services, Oregon State University. CAPS Assessment Committee Member. Student Affairs Assessment Council Member

Counseling & Psychological Services

Other ContributorsShailagh Clarke, Ph.D.

Staff Psychologist at Counseling and Psychological Services, Oregon State UniversityCAPS Assessment Committee Member

Kimberly Miller, Ph.D.Pre-doctoral Intern at Counseling and Psychological Services, Oregon State UniversityCAPS Assessment Committee Member

Today’s Objectives1. Explore how assessment can be used for learning

outcomes for clinical services.2. Identify ways to overcome barriers to clinical

assessment common to counseling centers.3. Become familiar with a successful model of

clinical outcome assessment that produces data that can influence client treatment.

4. Identify at least one learning outcome that you can assess at your home institution.

5. Identify strategies for implementing a comprehensive clinical outcomes assessment program.

How We Got StartedThe University Began to Require

Effective Assessment

It became important to demonstrate:1. How our work aligned with the mission of the

university

2. How counseling helps retain students

3. How counseling helps student matriculate

How did CAPS respond to the push to assess our services? Two staff members began developing assessment

instruments for Counseling and Psychological Services. We began with simple client satisfaction surveys.

OSU Student Affairs Assessment Council was established six years ago, offering a place for all offices to learn together, try new tools, and to work collaboratively.

CAPS staff members joined the Student Affairs Assessment Council, and established an Assessment Committee within CAPS.

This year, three new members joined the CAPS Assessment Committee, infusing new energy and ideas.

Barriers to Doing Clinical Assessment Well

Staff Resistance to Assessment

Lack of Expertise within Staff regarding Program Evaluation and the Assessment of Learning Outcomes

Limited Time and Resources

Overcoming Barriers to Clinical Assessment

Staff Resistance• Make assessment data meaningful to staff

members

• Link assessment measures to agreed upon mission and goals

• Facilitate staff participation

• Reward participation

Overcoming Barriers to Clinical Assessment, Continued

Lack of Expertise in Program Evaluation and Identification of Learning Outcomes• Use a committee model• Provide training and continuing education

opportunities• Invite members outside the agency who have

more expertise (e.g. a faculty member in your math/statistics department)

• Recognize that assessment is an ongoing process and that the tools will develop over time

Overcoming Barriers to Clinical Assessment, Continued

Limited Time and Resources• Create a standing time to do

assessment work• Add assessment responsibilities to

some staff members’ job descriptions, and take away other responsibilities

So, What Have We Done?

Methodology

Administered a 43 item self-report survey to students using CAPS services in the 6th and 7th weeks of each 10-week term for three terms.

Revised and updated survey regularly. Hence, the method does not change but the content does.

Information About Our Sample

Data from one term (Winter 2009), sample of convenience

N=126 with a 51% return rate In which session did you start to see

improvement/change occur? Count Percentage

First/Intake 12 13%2-5 60 67%6-10 7 8%11 or more 3 3%No change occurred/No change yet

8 9%

How the Content of Our Assessment Has Developed

Began with satisfaction surveys Moved to assessing learning outcomes Now we assess the relationship among:o Student Goalso Student Learning Outcomeso Content of Counseling Sessions

Three Areas of Assessment Stress and Mood Management

Personal and Relationship Skills

Academic Success

Stress and Mood Management

• Overall self-care• Positive sleep habits• Eating habits• Regular exercise• Overall mood• Coping with mental and emotional

difficulties

Personal and Relationship Skills

• Communication skills• Relationships with others• Identifying and expressing what I need• Relationships with my partner• Making relationship decisions• Insight into my personality• Self-esteem• My level of connection to the OSU

community

Academic Success

• Time management• Motivation• Test anxiety• Attending classes• Study skills• Concentration• Procrastination• Effectiveness as a student

The assessment revolves around three questions we ask about the previously identified areas

First Question

“It was important to me to experience change in…”

Rated on a 5-point Likert scale, ranging from “Very True” to “Very Untrue”

Second Question

“Please describe changes or growth, if any, since you began

counseling in…”

Rated on a 5-point Likert Scale, ranging from “Much Improved” to “Much Worse”

Third Question

“Which of the following issues did you discuss in counseling?”

(Check all that apply)

Rated as endorsed or not endorsed

ResultsSlide 1: Graph illustrating…% of clients who expressed a desire to

experience change in that particular areaAND

% of clients who said the area of focus was discussed in counseling

Slide 2: Graph illustrating…% of clients reporting each level of improvement

Slide 3: Results summary statements

Stress and Mood Management:Coping with Mental and Emotional Difficulties

Are topics that clients say are important to change being discussed in counseling?

Important to Experience

Change

Discussed in Counseling

0

40

80

94 90

Per-centage

Coping with Mental and Emotional Difficulties

Overall, how much improvement did clients report?

0

20

40

60

21

53

195 2

Per-centage

Results for Coping with Mental and Emotional Difficulties

94% indicated that it was important to experience change

74% indicated it was important to experience change in this area and reported improvement

20% wanted change, but did not experience change

Personal and Relationship Skills: Communication Skills

Are topics that clients say are important to change being discussed in

counseling?

Important to Ex-perience Change

Discussed in Counseling

0

20

40

60

80

10067

49

Percentage

Communication SkillsOverall, how much improvement did clients

report?

0204060

13

4837

2 0

Per-centage

Results for Concentration67% indicated that it was important

to experience change54% indicated it was important to

experience change in this area and reported improvement

13% wanted change but did not experience change

7% did not indicate that it was important to change, but did experience positive change

Academic Success: Concentration

Are topics that clients say are important to change being discussed in

counseling?

Important to Ex-perience Change

Discussed in Counseling

0

20

40

60

80

100 67

30

Percentage

ConcentrationOverall, how much improvement did clients

report?

-1010305070

1236 38

12 2

Percentage

Results for Communication67% indicated that it was important

to experience change45% indicated it was important to

experience change in this area and reported improvement

22% wanted change but did not experience change

3% did not indicate that it was important to change, but did experience positive change

Important to Change as related to Discussed in Counseling

Change desired and discussed Change desired, not discussedRelationships with

othersCoping with

mental/emotional difficulties

Expressing needsCommunication skillsOverall mood

Self-careSelf-esteemSelf-motivationEffectiveness as a

studentConcentration

General Conclusions

By looking at the relationships among the variables (client goals, counseling content, and client change), we get more meaningful information than simply asking, “Did you improve?” or “Did counseling help you stay in school?”

Some specific things we’ve learned through our assessment processStudents come to counseling with all kinds of hopes,

expectations and goalsStudents’ personal and relationship skills and ability to

manage mood improve at higher frequencies than academics

More students report greater change in mood management than academic skills

Students report that they talk about mood, personal and relationship skills at higher frequencies than academics

Clinical Uses of Assessment DataThe thing that makes assessment compelling is that, when done well, it can give us good information about our work.

If the data are meaningful, we are often moved to begin exploring how we can do things more effectively to better meet students’ needs.

Clinical Implications

We’re doing very good work in helping students positively change their mood, personal skills and relationship skills.

Clinical Implications Continued

Counselors may need to assess students’ goals and expectations more completely, especially regarding students’ hopes for academic change.

Clinical ImplicationsContinued

Counselors may need to more overtly discuss academic performance throughout counseling sessions.

Clinical ImplicationsContinued

Counselors may need to make explicit to student the relationships among the variables.

Next Steps Clinical Service Assessment

Outreach and Program Assessment

Training of pre-doctoral interns and other trainees

How to improve your own clinical assessment• Work with other people!

• Begin to identify learning objectives.

• Explore alternative factors that affect learning and may warrant assessment.

Let’s design one learning outcome that you could effectively assess at your institution.

Questions?

Contact Information

Oregon State UniversityCounseling and Psychological

Services500 Snell HallCorvallis, OR 97331541.737.2131http://oregonstate.edu/counsel/