data analysis & interpretation - …€¢ build & practice data analysis &...
TRANSCRIPT
DATA ANALYSIS & INTERPRETATION
August 25, 2017 October 20, 2017
PERFORMANCE MANAGEMENT SERIES Data Management Focus
• Structure
o Workshops & Trainings
• Content
o Data Collection
o Data Analysis & Interpretation
o Data Communication
OBJECTIVES • Enthuse evaluative thinking
• Encourage performance managament
OBJECTIVES • Build & practice data analysis &
interpretation skills
o Cleaning data
o Quantitative & Qualitative analysis processes
o Finding meaning in analyzed data
• Work with existing program data
o Clean data to prepare for analysis
o Analyze program data
o Identify 2-3 key findings from program data
AGENDA
9:00 – 9:10 Intro & Overview
9:10 – 9:30 Cleaning/auditing Data Content
9:35 – 9:50 Cleaning/auditing Data Practice
9:50 – 10:00 Data Analysis Content
10:00 – 10:35 Data Analysis Practice
10:35 – 10:40 Data Interpretation Content
10:40 – 11:25 Data Interpretation Practice
11:25 – 11:30 Closing
BEST PRACTICES FOR
EFFECTIVE PERFORMANCE MANAGEMENT PERFORMANCE MANAGEMENT
PERFORMANCE MEASUREMENT & MANAGEMENT
Processes and systems to:
GATHER
MEANINGFUL DATA
MONITOR IMPLEMENTATION
& IMPACT
WHAT IS PERFORMANCE MANAGEMENT?
What went well?
What didn’t go well?
How do we improve?
Repeat
Good Intentions
Counting Outputs
Measuring Outcomes
Managing Performance
PERFORMANCE MANAGEMENT CONTINUUM WHAT IS PERFORMANCE MANAGEMENT? PERFORMANCE MANAGEMENT CONTINUUM
Good Intentions
Counting Outputs
Measuring Outcomes
Managing Performance
DATA ANALYSIS & INTERPRETATION
D R I V E S
BEST PRACTICES FOR
EFFECTIVE PERFORMANCE MANAGEMENT CLEANING DATA
Why collect data?
Why clean data?
DATA QUALITY & INTEGRITY
Cleaning data ensures it is
ACCURATE & FIT FOR ITS INTENDED USE
UNCLEAN DATA • Varying formats
o Jan 12, 2016 v. 1/2/16 v. 01/02/2016
o Caucasian v. white
• Errors in data
o Negative ages
• Blanks/missing information
• Large jumps in data
o Savings of $12,000 to $150,000 in 6 months
PREPARES DATA FOR ANALYSIS
Cleaning data prepares you for analysis
• Ensures data is free of errors
o Confirms or corrects discrepancies
• Identifies missing data
• Aligns formatting
• Determines process for unknowns/errors
How to clean data
DATA CLEANING PROCESS Reviewing and cleaning data
• Spot check random sample
• Sort/filter data
o Missing values
o Outliers (overly high/low values)
o Check feasibility (errors & discrepancies)
• Address abnormalities
EXAMPLES
Never modify the raw data.
• Use several worksheets when creating a spreadsheet.
• Always copy the raw data to a new worksheet and make the modifications on the new worksheet. If the new worksheet needs to be fixed or the process needs to start over, the data will not need to be extracted again.
• When finished, hide worksheets that don’t need to be seen by other users.
TIP
If your agency uses account numbers, clients numbers, program numbers, social security numbers, or zip codes with a leading zero, make sure the column is formatted as text. Always double check to make sure the leading zero is not dropped. If wrong, this may affect the data being analyzed.
PRACTICE 25-35 minutes
Clean your program data ensuring:
1. Aligned formatting is used
2. Identify & address missing data
a. Determine standard approach
3. Identify & address discrepancies/jumps in data
a. Are there obvious errors?
4. Write standard approach for
a. Missing data & errors
BEST PRACTICES FOR
EFFECTIVE PERFORMANCE MANAGEMENT DATA ANALYSIS
Why analyze data?
What variables do you analyze?
HOW TO “SLICE & DICE” DATA
• What question are you trying to answer?
o Examine variables that will answer or
influence that question
o Consider key variable relationships
o Identify appropriate calculations
How to analyze data
TYPES OF DATA
Quantitative Qualitative
• Numbers & statistics
• Often objective
• Often answers “what” questions
• Words/text & concepts
• Often subjective
• Often answers “why” questions
Qualitative Data Analysis
QUALITATIVE DATA ANALYSIS BASICS
• Less structured than quantitative analysis
• Not guided by universal rules
• Data reduction is key
o Leads to identifying key themes
QUALITATIVE DATA ANALYSIS BASICS
• Predefined codes
• Emergent codes
• Can make numerical
Categorize/ code
responses
• What themes emerge that answer evaluation questions?
Identify trends and
themes
Often helps answer “why” questions
QUALITATIVE DATA ANALYSIS EXAMPLES
Why didn’t clients enact healthy behaviors they
learned about?
1. Because my partner does it
2. I just need to smoke to calm down
3. I get too stressed without smoking
4. My friends said it was lame to not smoke
5. I freak out if I don’t smoke
QUALITATIVE DATA ANALYSIS EXAMPLES
Quantitative Data Analysis
QUANTITATIVE DATA ANALYSIS BASICS Key Quantitative Calculations
Mean Median Mode
Variability Frequency
Distributions
QUANTITATIVE ANALYSIS PROCEDURES
Data Tabulation • Frequency & Percent Distributions
Descriptives Data • Mean, median, mode, range, etc.
Data Disaggregation • Break down across subcategories
Moderate/Advanced Analysis • Correlation, Regression, ANOVA
OUTCOME COMPARISONS
Data disaggregation among the change or benefit clients experience
OVER TIME
AGAINST TARGETS
WITH BENCH-MARKS
BY CLIENT GROUPS
BY SERVICE
Examine findings across all indicators
QUANTITATIVE ANALYSIS APPROACHES
TIP
As soon as you start manipulating data there is the potential of inaccurate data.
Always! Always! Always! Double check your data
Have another person check your data
Lisa Emily Julie
PRACTICE 25-35 minutes
Analyze your program data
*You can use Analysis Decision Tree if helpful
1. Identify evaluation questions to answer
2. Identify variables that influence those questions
3. Identify appropriate analysis approach and
calculations
4. Conduct analysis using qualitative coding or
quantitative calculations
BEST PRACTICES FOR
EFFECTIVE PERFORMANCE MANAGEMENT DATA INTERPRETATION
If you torture the data long enough, it will confess to anything. Ronald Coarse, Economics Nobel Laureate
Why interpret data?
FINDING MEANING IN DATA
How to interpret data
INTERPRETING DATA Interpretation is where meaning is found; consider
o Patterns, themes & deviations
o If results make sense
o Surprising findings & potential causes
o Focus areas for improvement
o Additional questions that arise
INTERPRETING TIPS
• Start with analysis of clean data
• Ask
o So what?
o What does this mean/tell me about my program?
o How do we use these findings?
• Try visualizations
• Allot time for interpretation
DATA INTERPRETATION MEETINGS • Include key stakeholders
• Present data
o Data parties/placemats
o Consider key variables & relationships
• Pose key questions around findings
o What surprises you/stands out?
o What factors may explain findings?
o What action should we take?
o Any new questions?
REMEMBER:
What questions are you trying
to answer?
CAUSATION & CORRELATION
Mutual relationship between variables
(positive or negative)
Causation Correlation
One factor leads to/causes another
PRACTICE 15-20 minutes
Interpret your analyzed program data
*Use Data Interpretation Guide if helpful
1. Identify evaluation questions to answer
2. Identify 2-5 key findings/answers from analyzed
data
3. Identify other stakeholders to involve
4. Consider ideas for presenting data
Why are data analysis & interpretation important to performance management?
THANK YOU!!! • Thank you for your time
• Please share feedback
• Other opportunities
o Program Impact Reviews
• Feel free to contact us
Emily Uzzle Lisa Goodman
314-539-4256 314-539-4217