comprehensive assessment: a data driven decision making process
TRANSCRIPT
Comprehensive Assessment:A Data Driven Decision Making Process
University of South Carolina Student Life
VISION
A caring community united for a safer Carolina.
MISSION
To create a campus-community environment that promotes healthy and safe behaviors among faculty, staff, students, and community organizations.
University of South Carolina Student Life
PURPOSE
To build capacity and to perform ongoing assessment, planning, implementation, and evaluation of substance abuse prevention strategies, and to make recommendations for policy change
GOAL
To contribute to the overall reduction of high risk substance use and its consequences at the University of South Carolina
The SPF process enables campus communities to:
• Plan and develop substance abuse prevention infrastructure
• Implement effective prevention strategies
• Environmental approaches and educational initiatives
• Achieve population level outcomes
A Model for Reducing High Risk Drinking Behaviors on Campus: The Strategic Prevention Framework
EVALUATIONMonitor, evaluate,
sustain, and improve or replace
those that fail
IMPLEMENTATIONImplement
evidence-based Prevention programs &
activities
PLANNINGDevelop a
Comprehensive Strategic Plan
ASSESSMENTProfile population
needs, resources, & readiness to
address needs and gaps CAPACITY
Mobilize and/or build capacity to address needs
Sustainability & Cultural
Competency
Our Coalition’s Assessment ProcessCreated the Data, Assessment, and Evaluation Subcommittee
Review all available data sources
Identify High Risk Priority Areas
Develop logic models
Form subcommittees to address each problem priority area
Set goals and objectives for each priority area
Identify gaps in research, data, and resources
Develop a comprehensive assessment toolkit
Develop assessment protocols and data collection tools
Collect comprehensive data
Identify emerging trends
Review environmental strategies to determine how best to address local conditionsSet goals and develop a strategic action plan for the year
Developed a comprehensive assessment toolkit
Developed assessment protocols and data collection tools
Collected comprehensive data
Identified emerging trends
Reviewed environmental strategies to determine how best to address local conditions
Set goals and developed a strategic action plan for the year
Assessment Timeline 2014-15August 2014: Review logic models and strategic plan
September 2014: Present strategic plan to coalition
October 2014: Implement strategies and monitor
November 2014: Implement strategies and monitor
December 2014: Implement strategies and monitor
January 2015: Implement strategies and monitor
February 2015: Review data for strategic planning session
March 2015: Identify problems, resources, readiness, and priority areasApril 2015: Annual Assessment report due
May 2015: Host Annual Strategic Session
June 2015: Review logic models
July 2015: Develop strategic plans
August 2015: Steering committee approves strategic plan
Developed a comprehensive assessment toolkit
Developed assessment protocols and data collection tools
Collected comprehensive data
Identified emerging trends
Reviewed environmental strategies to determine how best to address local conditions
Set goals and developed a strategic action plan for the year
Data, Assessment and Evaluation Subcommittee
To inform and guide the coalition in problem and asset identification, strategic planning and implementation, and outcome evaluation
Purpose
Goal
Objective
PURPOSE
To conduct ongoing, comprehensive community-campus assessment and process/outcome evaluation
GOAL
To To conduct assessment to determine the extent of: • Risk factors associated with the problem • Protective factors (assets and resource availability)• Community-campus readiness to address the problem and fill needs
and service gaps• To conduct process and outcome evaluation of prevention strategies
OBJECTIVE
Data, Assessment and Evaluation Subcommittee
Tasks of the Subcommittee
• Determine the assessment’s purpose and scope• Define the assessment’s goals and objectives• Select the approach and methods for collecting data• Design and pilot test the collection instruments and procedures• Prepare assessment timelines• Collect the data• Analyze the data• Prepare data in a way that describes the root causes and local
conditions that contribute to identified community problems• Develop a problem statement(s) that our coalition will address on a
community wide, environmental level
Source: The Fenway Institute
Comprehensive Assessment
Three Areas:• Problem Identification
and Prioritization• Resource Identification• Readiness for Change
Comprehensive Assessment: #1: Problem Identification and Prioritization
The subcommittee has identified many indicators that help to explain student substance use and patterns. Specifically we look at:
• prevalence of use• locations of use• drinking patterns• consumptions levels• student subpopulation(s) most affected• types of substances being used/abused• average age of use• environmental factors, such as availability, accessibility, and cultural
norms
Comprehensive Assessment: #1: Problem Identification and Prioritization
• Enterprise Risk Management
• Risk Assessment (probability, severity, velocity)
• Risk Control Assessment• Final Risk Score and
Ranking
Comprehensive Assessment:#2: Resource Identification
Identify past and current organizational and individual resources, and categorize those resources by key community sectors.
Identify future resources needed to achieve our desired outcomes
Ask the following questions: o Who has the knowledge we need? o Who has the time to assist? o Who has the human resources we need? o Who can hold us accountable for outcomes? o Who can provide financial support? o Who can encourage us and recognize our successes?o Who has helped with this issue in the past? o Who has an interest in our outcome? o Who has decision making power?o Who cares about this issue? o Who has other resources like space, food, giveaways, that we may need?
Comprehensive Assessment:#2: Resource Identification
Identify quantitative and qualitative data currently being gathered and analyzed.
• AlcoholEdu: Freshmen Student Survey• CORE: Upperclassmen Student Survey • NCHA• STIR: Indicated Student Population• OSC: Student violations and sanctions related to substance use• USCPD: crime data related to substance use • Health and Wellness Center: health consequences related to substance use• Key Informant Interviews• Focus Groups• Environmental Scanning
Comprehensive Assessment: #3: Readiness for Change
Determine important aspects to consider when assessing readiness to change. Considerations include:
• Examining multi-levels and multi-facets of campus and the surrounding community
• Determining political will for change• Measuring shared resolve/commitment to implement change • Determining if there is a shared belief in our capability to make change• Determining if there is a shared value that change needs to occur• Assessing the campus community’s ability to meet task and resource demands• Concluded that the higher the readiness for change, the more likely change will
be initiated, the greater the effort , persistence, and cooperation members will give
Drinking LocationFall 2013 vs. Fall 2014
Problem Identification and Prioritization 2014-15
1
2
3
4
Priority: Bars and nightclubs
Priority: Off campus residences
Priority: Other high risk substance use
Priority: Community organizing and advocacy
Conduct Strategic Prevention Planning
Steering Committee utilized assessment information to:• Identify and prioritize the problem(s)• Develop logic models to identify root causes and
contributing local factors
Coalition members then formed subcommittees to:• Create SMART Goals• Select the intended outcome(s) and measure(s)• Identify intermediate variables and measures• Identify strategies for changing intermediate
variables
USC Problem Priority : Prevalent Drinking in Off Campus Residences
Root Causes Contributing Local Factors
Perceived Risk
Social Access & Availability
Social Norms
Prevalent Drinking
Off CampusResidences
Problem
Social Hosting: Students over 21 provide alcohol to minors at
parties/get togethers
Social Hosting: Students over 21 are not award of social hosting laws
Property management rules vary and do not get enforced
“College” norm to drink alcohol
Multiple jurisdictions and enforcement varies
Students perceive low risk of harm in own apartment
Students perceive low risk of getting caught in own apartment
Underage drinking is acceptable
Driving after DrinkingSexual AssaultsPublic Nuisance
Property Damage
Negative Consequences
Inconsistent Enforcement
Logic Model: USC Problem Priority : Prevalent Drinking at Football Game Days
Root Causes Contributing Local Factors
Insufficient Laws and Policies
Social Access & Availability
Social Norms
Prevalent, High Risk Drinking
On football Game Days
Problem
Tailgate Environment, social hosting, alcohol available with little
monitoring
Excessive amounts of alcohol and time periods available for drinking
Large crowds vs. enforcement resources
Perception of “no rules, no consequences” on Game Days
Identification of illegal acts difficult to manage in large crowds
Policies on tailgating vary due to mix of university/privately owned
properties
Tailgating restrictions are inadequate
Alcohol central to tailgating culture
Underage DrinkingBinge Drinking
Driving after DrinkingBlack Outs
Alcohol PoisoningInjuries
Public IntoxicationProperty Damage
Negative Consequences
Inconsistent Enforcement
USC Problem Priority : High Risk Drinking in Bars
Root Causes Contributing Local Factors
Promotion
Access & Availability
Social Norms
High Risk Drinking in
Bars
Problem
Inconsistent Service Practices (ID checking, over-service)
Extended Drink Hours
Perceived safety: Safety risks are low with friends
Underage drinking is acceptable
Perceived getting caught: Easy to get in bars using fake ID and drink
underage
Social Networking and Social Media Advertising
Inconsistent enforcement of pricing, drink specials, and happy hour laws.
Drinking to intoxication is acceptable
Binge DrinkingUnderage Drinking
Black OutsAlcohol Poisoning
InjuriesSexual Assaults
Property DamageArrests
Negative Consequences
Bar Sales and Services Subcommittee
Long Term Goal By March 2016, reduce high risk drinking at bars by 10%.
Objectives Reduce access and availability, address alcohol promotion through social media, increase perception of risk (harm and getting caught), and change social norms.
Strategies/Activities Collect data on sales and service practices, promotion and advertising through social media, in order to develop and implement a responsibility toolkit and provide merchant education.
Off Campus Safety Committee
Long Term Goal By March 2016, reduce high risk drinking at off campus properties by 10%.
Objectives Decrease availability and accessibility, increase perception of harm and getting caught, consistently enforce policies and laws, improve referrals, and change social norms.
Strategies/Activities Collect data from off property managers, courtesy officers, and students to develop an incentive package, offer education and programming, and improve collaboration with properties, law enforcement and university.
Other High Risk Substance Use Subcommittee
Long Term Goal By March 2016, reduce high risk drinking at off campus properties by 10%.
Objectives Decrease availability and accessibility, increase perception of harm and getting caught, consistently enforce policies and laws, improve referrals, and change social norms.
Strategies/Activities Collect data from off property managers, courtesy officers, and students to develop an incentive package, offer education and programming, and improve collaboration with properties, law enforcement and university.
Community Organizing and Advocacy Subcommittee
Long Term Goal By March 2016, reduce high risk drinking at off campus properties by 10%.
Objectives Decrease availability and accessibility, increase perception of harm and getting caught, consistently enforce policies and laws, improve referrals, and change social norms.
Strategies/Activities Collect data from off property managers, courtesy officers, and students to develop an incentive package, offer education and programming, and improve collaboration with properties, law enforcement and university.
Game Day Task Force
Long Term Goal By March 2016, reduce prevalence of drinking on game day by 5%.
Objectives Reduce social access and availability, consistently enforce policies and laws, and change social norms.
Strategies Collect data on tailgate policies and student drinking patterns on game day to inform and improve law enforcement and collaborations.
Data, Assessment, and Evaluation Subcommittee
Developed the Comprehensive Assessment Tool Kit, a data collection protocol packet for each subcommittee
Purpose: to assist the subcommittees with conducting assessment specific to their priority area
Assessment Tool Kit included:o brief overview of assessment processo data collection guidelineso variety of data collection methods to considero general timeline template to assist with planningo tools and resources to use for data collection
University of South Carolina Student Life
Subcommittee Action Steps
• Review Assessment Tool Kit• Select most appropriate data collection
methods• Choose or develop data collection tools
and protocols• Plan the data collection process and
timeline• Implement the data collection process
Subcommittees considered the following:What is the focus of our assessment?
What concepts do we need to define to ensure clear understanding? (ie. How do we define “high risk drinking”?)
What methods of data collection are best for our assessment’s focus?
What resources do we need to collect the data? (financial, staff time, training, locations, etc.)
Who will collect the data?
How will we implement the data collection process?
How will we address confidentiality?
Who will our participants be? How will we recruit them?
How many participants do we need for our data collection?
Where will we conduct the assessment?
How much time is needed to administer the data collection process?
How will we measure the data?
How will we ensure the tools we use are culturally appropriate?
How will we pilot/adjust the tools with potential participants?
Review Assessment Toolkit
Consider Collection Methodsand Timelines
Subcommittees considered the following:
• Key Informant Interviews• Focus Groups• Written Surveys• Community Forums• Environmental Scans• Social Media Analysis• Policy Review• Benchmarking
Select or Create Data Collection Tools and Protocols
Data, Assessment and Evaluation Subcommittee developed the following tools:
• Written surveys• Focus group questionnaires and protocols• Key informant interview questionnaires and
protocols• Social Media Collection Tools• Environmental Scan Surveys
• Development of Tools/Instruments to assess Over-service & Effectiveness of Responsibility Toolkit: – Establishment Observation Tool, Volunteer Observer
Protocol, and Establishment Observation Supervisor Checklist;
– Utilization Assessment Tool; and – Patron Survey
Example:Bar Sales and Services Subcommittee
Responsibility Toolkit
• Raise awareness of negative consequences associated with drinking & driving
• Increase perception of risk of getting arrested for DUI• Raise awareness of SC law prohibiting over-service to
intoxicated patrons• Raise awareness of transportation options for intoxicated
patrons • Raise awareness of increased DUI enforcement
Toolkit Objectives: Patron
• Raise awareness of SC law prohibiting over-service to intoxicated patrons
• Raise awareness that law enforcement is tracking place of last drink
• Raise awareness of civil liability of establishment in event of tragedy
• Raise awareness of increased DUI enforcement • Raise awareness of transportation options for intoxicated
patrons • Increase self-efficacy to refuse service to intoxicated patron
Toolkit Objectives: Employees & Management
• Increase implementation of model policies governing alcohol sales
• Increase establishment accountability to prevent driving after drinking
• Increase partnerships between alcohol retail establishments & Task Force
• Increase participation in responsible beverage service training (PREP)
Toolkit Objectives: Management (behavior)
• Patron/staff ratio• Drink specials/promotions• Age verification systems• Inconsistent systems to designate underage patrons vs. 21
and older• Staff drinking with patrons• Patrons requesting non-alcohol drinks ridiculed, hassled• Retailers see over-service as a personal responsibility issue,
not a retailer issue
Observational Scanning Findings
Conclusions from the Assessment
Community Collaboration:• Challenging issue – multiple systems, sectors • Difficulty engaging alcohol retailers in substance use
prevention efforts• “No one size fits all” – community demographicsResponsibility Toolkit:• No good evidence of effectsObservation Tool: • Provided documentation for both (a) pre intervention
baseline and (b) post intervention • Manageable and easy to administer• Observer agreements (reliability) from 70-99%--
EXCELLENT
Implications for Future Coalition Prevention Efforts
• Continue assessment activities (bar sales and services observations, key informant interviews with bartenders and shuttle drivers; frequent, brief student surveys, etc.)
• Continue collaboration efforts, particularly with retail establishments• Establish a student substance abuse cost analysis for key stakeholders• Utilize science concerning key variables and effective interventions• Establish a city-college data monitoring system (MIS)—reflect USC plus
city.• Publish article “Observational Documentation in Evaluating Alcohol Over-
service in Bars and Restaurants” for professional prevention journals to share findings with other college communities