evaluation team evaluator: nancy amodei, ph.d. – dept pediatrics evaluation coordinator: danielle...
TRANSCRIPT
![Page 1: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/1.jpg)
Evaluation Team•Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics•Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics•Data Manager: Kyle Kozlovsky, M.S. – Dept Pediatrics•Qualitative Expert: Suyen Schneegans, M.A. – Dept Pediatrics
®
Special thanks to: Rasheem BattleAlejandro BocanegraMeghan CrabtreeMerced DoriaDestiny RamosDrew Russell
![Page 2: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/2.jpg)
![Page 3: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/3.jpg)
Process Evaluation: How is the program being implemented?
S-START Process Goals
1. Train UTHSCSA medical residents and residents from other participating South Texas programs to use evidence based SBIRT procedures for patients who have or are at risk of substance abuse disorders.
2. Promote systems change in targeted residency programs by integrating the SBIRT model into the curriculum on a long-term basis.
![Page 4: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/4.jpg)
Process Goal #1: Train UTHSCSA medical residents and residents from other participating South Texas programs
Key Activities and Measures
1) Develop/implement a comprehensive curriculum
2) Train UTHSCSA and other Faculty Demographics, type of training, satisfaction (GRPA & qualitative
findings)
3)Train UTHSCSA residents & residents from other programs Demographics, type of training, satisfaction (GPRA)
![Page 5: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/5.jpg)
SBIRT Curricular Strategies by Specialty
Pediatrics
OB-GYN
Psychiatry
Internal
Medicine
IM - ERAH
C
FCM
FM –McAlle
n
FM -Fort Hood
FM -Santa Rosa
Surgery
Large Group Didactic Lecture
+ + + + + + + + +
Small Group Discussion
+ + + + + + +
Skill-Building Workshops
+ + + +
Reading Assignments
+ + + +
Screening Questions In Medical Records
+ + + + + + + + + + (likely but not in place
yet)
Reminder Pocket Cards
+ + + +
Screening Assignments*
+ + +
Inpatient Clinical Supervision
+ + + + + +
Outpatient Clinical Supervision
+ + + (child psych only)
+ + + + + +
Independent Study Module (Blackboard)
+ + + +
OSCE (Observed Standardized Clinical Exams)
+ +
![Page 6: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/6.jpg)
What is the core SBIRT event?
![Page 7: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/7.jpg)
Faculty Training
Who? How Many?
Method/Approach
Pediatrics, UTHSCSA 8 Train-the-trainer
Family & Community Medicine, UTHSCSA
9 Train-the-trainer
Family Medicine, CHRISTUS Santa Rosa
9 Train-the-trainer
Family Medicine, Fort Hood 5 Train-the-trainer
Internal Medicine, UTHSCSA 1 Chief Residency Immersion Training (CRIT)
Internal Medicine, ERAHC 1 Train-the-trainer; Chief Residency Immersion Training (CRIT)
Other UTHSCSA Departments and external Departments
18 Personal consultation, Sharing resources via website, Newsletter, Email
![Page 8: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/8.jpg)
Demographics of Faculty Completing GPRAs
Demographics or Respondents at Baseline
N = 17 (%)
UTHSCSA Family and Community Medicine Pediatrics
8 (47%)
9 (53%)
Male
9(52.9%)
Hispanic/Latino 7(41.2%)
White
12 (70.6%)
Asian 3(17.6%)
African-American 0(0%)
![Page 9: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/9.jpg)
* Wilcoxen Signed Ranks Test
Baseline vs. 30-day Faculty GPRA ratings (N=17)
GPRA Item Mean Baselin
e Rating
Mean 30-Day Rating
Z * P (2- tailed
)
Overall how satisfied with your training experience?
2.12 1.93 -.53 NS
Material presented useful to me in dealing with substance abuse
1.82 1.94 -.51 NS
Training enhanced my skills in topic area 2.00 1.94 .00 NS
Training relevant to my career 1.76 1.88 -.63 NS* Wilcoxon Signed Ranks Test
![Page 10: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/10.jpg)
Qualitative Study of Faculty Perceptions of S-START
Purpose: Gain an in-depth understanding of the experience and perceptions of S-START faculty
Methods
16 training faculty from 5 specialties invited
15 accepted (12 from UTHSCSA; 1 FM program in McAllen, 1 FM CHRISTUS Santa Rosa, 1 FM from Fort Hood)
Mean age (43.93 years); 72% female; 73% MDs, 1 Ph.D., 1 PsyD, 1 M.A.
Average yrs of experience = 15
![Page 11: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/11.jpg)
Qualitative Study of Faculty Perceptions of S-START
Data Collection
Data collection: ≈ 22 months after S-START began
45 to 60 minute interviews using scripted but open-ended questions
14 of the interviews taped to facilitate transcription
Topics: How S-START implemented in the program, barriers and challenges, impact of potential clinical service reimbursement in facilitating program; suggestions for improvement
![Page 12: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/12.jpg)
Qualitative Study of Faculty Perceptions of S-START
Data Analysis
Evaluation team hand-coded transcribed interviews
Thematically coded them to correspond to each question
Collapsed materials thematically into 10 emergent or preset categories
![Page 13: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/13.jpg)
Qualitative Study of Faculty Perceptions of S-START
Results3 Thematic categories accounted for > 50% of interview
responses Critical components Barriers Motivation
Critical Components Faculty training
Barriers Lack of Leadership
Motivation Buy-in from faculty and residents
![Page 14: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/14.jpg)
Resident TrainingDepartment Trained to
date Expected trained by Year 05 Old estimates
Total Expected Trained by Year 05
Pediatrics - UTHSCSA 80 107 112
Family & Community Medicine
•UTHSCSA•McAllen (South Texas)•CHRISTUS Santa Rosa•Fort Hood (Military)
71292517
8424----
95414031
Internal Medicine•UTHSCSA•ERAHC (South Texas)
11316
163--
16326
OB-GYN -UTHSCSA 29 41 41
Psychiatry - UTHSCSA 71 120 107
Surgery- UTHSCSA 58 --- 96
Nurse Practitioners-UTHSCSA
39* --- 39
Total 548 539 679
![Page 15: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/15.jpg)
Demographics of Residents Completing GPRAs
Demographics or Respondents at Baseline
N = 409 (%)
Family Medicine Pediatrics Internal Medicine OB/GYN Psychiatry
138 (33.7%)80 (19.6%)
121 (29.6%)27 (6.6%)43 (10.5%)
Male
155(37.9%)
Hispanic/Latino 133(33.8%)
White 259 (63.3%)
Asian 81(19.8%)
African-American 17(4.2%)
![Page 16: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/16.jpg)
* Wilcoxon Signed Ranks Test
Baseline vs. 30-day Resident GPRA ratings (N=409)
Item Mean Baselin
e Rating
Mean 30-day Rating
Z * P (2- tailed)
Overall how satisfied with your training experience?
1.75 1.71 -4.11 .00
Material presented useful to me in dealing with substance abuse
1.68 1.64 -3.98 .00
Training enhanced my skills in topic area
1.75 1.67 -3.09 .01
Training relevant to my career 1.59 1.50 -2.92 .01
![Page 17: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/17.jpg)
Process Goal #2: Promote Systems Change in Targeted Residency Programsby integrating SBIRT model into curriculum on long-term basis
Key ActivitiesCouncil of Residency SBIRT Trainers Meetings
Elicit support of key personnel
Changes to Electronic Medical Record
Pocket Cards
SBIRT resources (including key modules) on the S-START website
iPad Project
![Page 18: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/18.jpg)
Progress towards Goal 2:Council of Residency SBIRT Trainers Meetings
Date Pediatrics
BAMC
FCM
Internal Medicine
Psychiatry
OB-Gyn
FM – Fort Hood
FM – Santa Rosa
Trauma
Nursing
1.29.09 2 1 1 - 2 1 - - 1 -
1.30.09 2 - 2 2 - - - - - -
9.29.09 4 2 1 2 2 - - - 3 -
12.08.09
5 1 2 1 1 1 - - - 1
3.09.10 6 1 - - 1 - 1 1 - 1
8.17.10 4 2 2 2 1 1 - - - -
2.15.11 5 - 2 1 - - - - 1 -
6.28.11 2 2 - - - 1 - - - -
7.19.11 5 - 2 1 1 - - - 3 -
![Page 19: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/19.jpg)
Process Goal #2: Promote Systems Change in Targeted Residency Programsby integrating SPIRT model into curriculum on long-term basis
Progress re other ActivitiesSupport of change leaders- e.g. UTHSCSA President, Residency
Program Directors; PD and Co-PD have high profile positions
Changes to Electronic Medical Record- UTHSCSA – DFCM, Peds; (Psychiatry and Surgery planned)
Pocket cards McAllen FM: Part of every patient visit paperwork
SBIRT resources (including core modules, resource directory) on the S-START website
iPad Project-proposed for UTHSCSA Pediatrics
![Page 20: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/20.jpg)
![Page 21: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/21.jpg)
S-START Outcome Goals:
1. Enhance residents’ knowledge of evidence-based
SBIRT practices.2. Enhance residents’ readiness and perceived
confidence to implement SBIRT with their patients
3. Increase residents’ implementation of SBIRT practices with their patients
4. Enhance Faculty Participants’ knowledge and confidence in ability to teach SBIRT practices to future physicians
What is the program’s impact?
![Page 22: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/22.jpg)
3 x 2 Repeated MeasuresThree data collection methodsMeasurement Occasions for Surveys:
Pre-Test30-Day Follow-UpAnnually up to 36-month follow-up
Measurement Occasions for Pocket CardsVaries by department
Measurement Occasions for Chart Reviews12-month period prior to first core SBIRT module
implementation12-month period following the first year of SBIRT module
implementation12-month period following the third year of SBIRT module
implementation
Outcome Design
![Page 23: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/23.jpg)
Evaluation MeasuresDOMAIN
Measure Description Source
Knowledge
Core SBIRT knowledge every resident should know
Local
Residency-specific
Department-specific SBIRT knowledge
Attitudes Readiness to use SBIRT
Readiness to screen patients for use, assess readiness to change, perform intervention/referral, & documentation
Alcohol Education Survey (D’Onofrio et al., 2002)
Confidence to use SBIRT
Confidence to screen patients for use, assess readiness to change, perform intervention/referral, & documentation
Current practice
Self-reported current use of SBIRT
Self-reported current practice of screening patients for use, assessing readiness to change, performing intervention/referral, & documentation
Alcohol Education Survey (D’Onofrio et al., 2002)
Pocket cards Family Medicine programs documenting SBIRT behavior w/patients
ASSIST (WHO, 2002)
Chart review Review of charts in Ped. & FCM inpatient clinics to see change in use & documentation of SBIRT
Local
![Page 24: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/24.jpg)
Tool Pre-test
Baseline
30-Day F/U
12- mos F/U
24- mos F/U
36 –mos F/U
GPRA X XAlcohol
Education Survey
X X X X X
Brief Substance Abuse Attitude
Survey
X X X X X
Core knowledge
X X X X X
Residency-specific
knowledge
X X X X X
Incentive $20 $10 $10 $20
Timeline of Self-Administered Instruments & Incentives
![Page 25: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/25.jpg)
Methods of Survey Data Collection
Web-based surveys (i.e., SurveyMonkey)Emails to UTHSCSA and private email addressesUnique web links provided to residency
coordinators
Hard copy surveys Pass out at grand rounds and conference periodsIntra-office mail for fellows, facultyMail to home and clinic physical address
![Page 26: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/26.jpg)
Strategies for survey follow-upCollected contact information using a comprehensive tracking
formText reminders to cell phone numbersPhone calls to (1) cell, (2) home, (3) significant others, (4) clinicContact residency coordinators for updated contact informationEnlist authoritative support of facultyLook up information using White Pages, AMA DoctorFinder,
respective state medical board websites (usually Texas)Peer-to-peer contact updates
![Page 27: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/27.jpg)
Future strategies for survey follow-upReminder postcards sent twice before each
annual surveyBring surveys to end-of-year gatherings for
graduating residentsInclude surveys in residents’ exit processing
before graduation I pity the fool who doesn’t take
the survey.
![Page 28: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/28.jpg)
Resident survey ratesPre-
test**Baseline
**30-day 12-
month24-
month
Total residents & students
153.4% 128.7% 89.9% 56.6% 67.0%
Response rates for similar populations (e.g., students, medical professionals) tend be 60% or lower on follow-up surveys (Asch et al., 1997; Kaplowitz et al., 2004; Kaspryzyk et al., 2001; McMahon et al., 2003; Porter & Whitcomb, 2007)
![Page 29: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/29.jpg)
Analyses of resident survey dataDemographic data (pre-test)Measured changes from pre-test to 12-
month follow-up in:
Selected departments for analysis: Pediatrics, Family and Community Medicine, Internal Medicine
![Page 30: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/30.jpg)
Resident DemographicsPre-test results
TotalN=46
5
UT Ped
n= 97
UT OBn = 29
UT Psy
n = 83
UT Surn = 49a
FMn = 137
IMn = 130
Gender (freq. male)
38.3% 20.6% 10.7% 38.6% - - - 44.9% 51.6%
Race (freq.)
White44.1% 55.2% 67.9% 61.3% - - - 32.6% 31.5%
Black 3.7% 2.1% 3.6% 7.5% - - - 1.5% 4.8%
Hispanic 31.1% 28.1% 21.4% 18.8% - - - 35.6% 38.7%
Asian 17.9% 12.4% 7.1% 10.0% - - - 27.4% 19.4%
Other 3.2% 2.1% - - - 2.5% - - - 3.0% 5.6%
Age (mean; sd)
30.3(4.8)
28.2(2.6)28.8(2.4
)31.4(5.9) - - -
32.5(5.7)
29.1(3.6)Note. UT=University of Texas Health Science Center at San Antonio; Ped.=Pediatrics; FM=Family
Medicine; IM=Internal Medicine; OB=Obstetrics/Gynecology; Psy.=Psychiatry (adult & child); Sur=Surgery.aSurgery residents began the SBIRT curriculum on August 15, 2011.
![Page 31: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/31.jpg)
Resident Demographics (cont.)Pre-test results
UT FCM
n = 72
UT McAn = 23
SRn = 24
FHn = 18
UT IMn = 109
UT ERAHCn = 21
Gender (freq. male)
40.0% 47.8% 33.3% 90.9% 48.6% 66.7%
Race (freq.)
White22.5% 9.1% 54.2% 72.2% 35.0% 14.3%
Black 2.8% 0% 0% 0% 1.9% 19.0%
Hispanic 28.2% 77.3% 45.8% 0% 36.9% 47.6%
Asian 42.3% 13.6% 0% 22.2% 19.4% 19.0%
Other 4.2% 0% 0% 5.6% 6.8% 0%
Age (mean; sd)
33.5(6.2) 35.3(4.6) 28.2(2.4)30.5(4.4
)28.8(3.5) 30.8(3.9)
Note. UT=University of Texas Health Science Center at San Antonio; FCM=Family and Community Medicine; McA.=McAllen Family Medicine; SR=CHRISTUS Santa Rosa Family Medicine; FH=Fort Hood Family Medicine; IM=Internal Medicine; ERAHC=Edinburgh Regional Academic Health Center Internal Medicine.
![Page 32: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/32.jpg)
Outcome goal 1Enhance residents’ knowledge of evidence-based SBIRT practices.
Core SBIRT knowledge12 items developed locally by the SBIRT project
directorsKnowledge that residents across all
departments should know after training
Residency-specific SBIRT knowledge7-17 items developed locally by the SBIRT faculty
in the respective programsItems designed for specific residency program
SBIRT knowledge and patient populations
![Page 33: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/33.jpg)
Sample core knowledge item:“How many ‘standard drinks’ are considered at-risk alcohol
use by a healthy 40-year-old man?”
Sample Pediatrics knowledge item:“________ exposure is the leading known preventable cause of
mental retardation.”
Sample Family and Community Medicine knowledge item:“Hepatitis B, hepatitis C, HIV and AIDS are strongly
associated with abuse of…”
Sample Internal Medicine item:“Alcohol withdrawal treatment on the inpatient medical
service is best accomplished by…”
Outcome goal 1Enhance residents’ knowledge of evidence-based SBIRT practices.
![Page 34: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/34.jpg)
Outcome goal 1 cont.Enhance residents’ knowledge of evidence-based SBIRT practices.
Core SBIRT knowledgeAll residents increased
SBIRT knowledge, F(1, 167) = 32.1, p < .001.
No differences found between residency programs
![Page 35: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/35.jpg)
Outcome goal 1 cont.Enhance residents’ knowledge of evidence-based SBIRT practices.
Residency-specific SBIRT knowledge
![Page 36: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/36.jpg)
Outcome goal 2Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients.
Readiness to use SBIRT (D’Onofrio et al., 2002)Subscale of AES comprised of 7 10-point Likert scale itemsRange: 0-100Sample item: “How ready are you to change your practice
behavior to ask patients about quantity and frequency of their alcohol use?”
Confidence to use SBIRT (D’Onofrio et al., 2002)Subscale of AES comprised of 7 5-point Likert scale itemsRange: 0-100 Sample item: “I am confident in my ability to
discuss/advise patients to change their drinking behavior.”
![Page 37: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/37.jpg)
Outcome goal 2 cont.Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients.
Readiness to use SBIRT: No significant change in
readiness from pre-training to 12 months post-training, F(1, 161)
= .87, p = .353.
FCM reported higher readiness than IM overall, F(2, 161) = 4.7, p
= ..010.Pediatrics was not
significantly different than the other two programs
![Page 38: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/38.jpg)
Outcome goal 2 cont.Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients.
Confidence to use SBIRT: Residents overall
reported higher confidence at 12-month, F(1, 161) = 27.3, p < .001.
FCM reported higher confidence overall than IM and Pediatrics, F(2,
161) = 8.1, p < .001.Pediatrics was not
significantly different than the other programs
![Page 39: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/39.jpg)
Outcome goal 3Increase residents’ implementation of SBIRT practices with their patients.
Self-report of current SBIRT practice (D’Onofrio et al., 2002)Subscale of AES comprised of 7 5-point Likert
scale itemsRange: 0-100Sample item: “How often do you formally screen
patients for alcohol problems using brief screening tools (e.g., T-ACE, AUDIT, CAGE)?”
Pocket cardsChart reviews
![Page 40: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/40.jpg)
Current practice of SBIRT skills:
Residents overall reported higher current SBIRT practice at 12-month, F(1, 161) = 35.2, p
< .001. Significant interaction,
F(2, 161) = 19.7, p < .001.Both Pediatrics and
FCM improved self-reported current practice .
Internal Medicine declined in self-reported current practice.
Outcome goal 3 cont.Increase residents’ implementation of SBIRT practices with their patients.
![Page 41: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/41.jpg)
Summary of Resident Survey Data Findings
SBIRT core knowledge improved from pre-test to 12-month follow-up
Readiness to implement SBIRT did not change, but was high at pre-test
Confidence to use SBIRT improved from pre-test to 12-month follow-up
For self-report of SBIRT practice, residents overall improved from pre-test to follow-upHowever, when departments were analyzed
separately, Internal Medicine decreased from pre-test to 12-month
![Page 42: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/42.jpg)
Settings:Family Medicine inpatient service at University
Hospital in San Antonio, Texas
Subjects:285 adult patients, from July 2009 to May 2011.Average Age: 47Gender Distribution: 71.3% Male
Outcome goal 3 cont.UTHSCSA Family Medicine Pocket Cards
![Page 43: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/43.jpg)
Patients were interviewed with a 4-step pocket card
Step 1: Pre-screening questions for substance useStep 2: WHO ASSIST (Alcohol, Smoking, and
Substance Involvement Screening Test)Step 3: ASSIST score to assess the level of risk and
determine need for interventionStep 4: checklist describing the intervention,
patient response, and future plan.
Residents were asked to complete 12 per year26 out of 26 trained residents participatedResidents completed 11 total on average
UTHSCSA Family Medicine Procedures
![Page 44: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/44.jpg)
![Page 45: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/45.jpg)
Step 1: Pre-screening Results
![Page 46: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/46.jpg)
95.8% of patients screened positive for at least 1 substanceAvg. ASSIST Score was 19 indicating a moderate risk of substance abuse
Step 2-3: ASSIST Results
![Page 47: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/47.jpg)
When the ASSIST Score recommended a brief intervention, residents reported some form of brief intervention 69.4%(over two thirds ) of the time
Residents most likely to discuss consequences of use if ASSIST Score recommended brief intervention (79% of the time)
8% of patients declined to discuss their response to screening
Brief Interventions
![Page 48: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/48.jpg)
Brief Intervention Actions Taken
![Page 49: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/49.jpg)
When the ASSIST Score suggests a referral to treatment, residents referred a patient to treatment 71.8% of the time
Residents were most likely to contact an LCDC (Licensed chemical dependency counselor) when ASSIST Score recommended a referral to treatment (46.5% of the time)
Referrals to Treatment
Referrals to Tx
![Page 50: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/50.jpg)
Referrals to Treatment Actions Taken
![Page 51: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/51.jpg)
Patients were more likely to report some sort of action (cut back, quit or seek outside help) when the resident documented a brief intervention (80% of the time compared to 71.4%)
Patients were more likely to report some sort of action when the resident documented a referral to treatment (80% of the time compared to 68.8%)
Step 4: Patient Plans
![Page 52: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/52.jpg)
The ASSIST Pocket Card was distributed to other departments with some participation
Participation in Other Departments
Residency # of Cards Completed
# of Residents
Participating
# of Residents Trained
UTHSCSA McAllen Family Medicine
103 13 29
Fort Hood Family Medicine
25 6 11
![Page 53: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/53.jpg)
Outcome goal 3 cont.Overall chart review study design
Location 12-month period prior to first core SBIRT module implementation
12-month period following the first year of SBIRT module implementation
12-month period following the third year of SBIRT module implementation
Pediatrics May 4, 2008-May 4, 2009
May 4, 2010-May 4, 2011
May 4, 2012-May 4, 2013
Family & Community Medicine
April 22, 2008-April 22, 2009
April 22, 2010-April 22, 2011
April 22, 2012-April 22, 2013
![Page 54: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/54.jpg)
Goal 3 cont.Overall Chart Review Study Design
1st Chart Review*
# of cases for
sample1. Pediatrics outpatient continuity clinic 4001. Family & Community Medicine outpatient
continuity clinic 400SUBTOTAL 800
2nd Chart Review^
# of cases for
sample1. Pediatrics outpatient continuity clinic 4001. Family & Community Medicine outpatient
continuity clinic 400SUBTOTAL 800
3rd Chart Review
# of cases for
sample1. Pediatrics outpatient continuity clinic 4001. Family & Community Medicine outpatient
continuity clinic 400SUBTOTAL 800
TOTAL 2400
*Completed^In Progress
![Page 55: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/55.jpg)
Setting: Pediatrics Outpatient Continuity Clinic
Chart selection criteria: (1) seen at least once within the review period by a
resident(2) at least one visit within the appropriate review
period (e.g., one year prior to implementation of core SBIRT intervention and medical record changes, one year after, and three years after)
(3) age of patient 11 and upVisits included in the analysis are acute visits and
annual well child visits
Pediatrics Chart ReviewBaseline findings
![Page 56: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/56.jpg)
Patient Demographics
Mean age was 13 (1.98)
49.5% Female
Ethnicity known to be majority Hispanic, but documentation in charts is rare
![Page 57: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/57.jpg)
ScreeningOut of 967 visits, 511 (just over half) screenings
were documentedVisits included Annual Check ups and Acute Visits
Some acute visits were sports physicals
![Page 58: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/58.jpg)
Screening Cont.
9 positive screenings documented for tobacco0 positive screenings documented for alcohol
or other drugsThe HEADSS was documented as a
screening tool in 149 visitsOnly one CRAFFT
screening was documented
![Page 59: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/59.jpg)
Brief Interventions and Referrals to Treatment
Of 967 visits, 204 brief interventions were documented4 of 9 positive tobacco screenings were
followed by a documented brief intervention“Anticipatory guidance” for drug use is often
used with pediatric patients and was considered a BI in our design
Only 2 referrals to treatment were documented
![Page 60: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/60.jpg)
Conclusions from Pediatric Chart Review
Documentation is a likely contributor to the lack of SBIRT practices foundOngoing changes to the medical record and
routine SBIRT training will likely increase SBIRT practices
An increase in screening might lead to an increase in positive screenings (and, in turn, increase opportunities for brief interventions and referrals to treatment)
![Page 61: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/61.jpg)
Outcome goal 4Enhance faculty participants’ SBIRT knowledge and confidence in ability to teach SBIRT practices to future physicians.
SBIRT knowledge Developed locally by the S-START project directorsSame core knowledge that the residents receive
Confidence to teach SBIRT to residents Adaptation to the resident scale (D’Onofrio et al.,
2002)Range: 0-100Sample item: “I am confident in my ability to train
residents in advising patients to change drinking behavior. “
![Page 62: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/62.jpg)
Outcome goal 4 cont.Enhance faculty participants’ SBIRT knowledge and confidence in ability to teach SBIRT practices to future physicians.
![Page 63: Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky,](https://reader038.vdocuments.net/reader038/viewer/2022110208/56649ddb5503460f94ad1b50/html5/thumbnails/63.jpg)