social media evaluation
TRANSCRIPT
Evaluating Social Media in Extension Programming
Sarah Baughman, Ph.D.eXtension
Sarah Baughman, Ph.D. & Brigitte Scott, Ph.D.Military Families Learning NetworkVirginia Tech
National Association of Extension Program and Staff Development Professionals October 21, 2014
Photo credit: Douglas Wrey on http://www.geek.com/wp-content/uploads/2012/02/social_media_donut.jpg
Ways to Use Social Media
Ways to Use Social Media
Annual Report from Calgary Zoo
Evaluating a “stand alone” social media campaign: Ohio State Kitchen & Campus Dairy Campaign, Jamie Seger
Theory of Change
Invest Resources
Program Activities
Increase consumption
of fruits & vegetables
Invest Resources
Program Activities
Increase consumption of fruits
& vegetables
Provide educational material on
nutrition
Maintain a food log
Enlist the support of
family
Nutritio
n Information
• Support F2F workshops with information on Facebook
• Use a FB page to encourage discussion on educational information presented F2F
Food
Log
• Provide FB incentives for person who increases the most or tries something new
• Have participants each take a different day to share one recipe they have tried or something from their food log that is working (or maybe not working)
Family
Suppor
t
• Invite family members to the FB page to encourage participants
• Highlight a different family member every week and how they have helped support healthier eating
Conversion
Engagement
Reach
Where’s the Data? Resources for Measuring and Evaluating Social Media
Facebook Insights
Twitter Tweetreach Hootsuite Tweepsmap Buffer Mentionmapp
Pinterest Tailwind
Multiplatform Sproutsocial SimplyMeasured SumAll Google Analytics
Evaluation Keys
Clear purpose
Clear goals
Consistent metrics and measurement over time
Use the measurement data to learn and improve
Contact Information
Sarah Baughman 540-231-7142 [email protected] @programeval Gplus.to/SarahBaughman Scoop it: Cooperative Extension Evaluation Pinterest.com/sarahbaughman
Getting to the Why behind the Numbers
CC by 3.0
You are already doing qualitative analysis.
CC by 3.0
From TweetReach….
From Facebook Insights….
Photo by LauraGilchrist4 - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/76060406@N07 Created with Haiku Deck
Basic Text Analysis
Basic Text Analysis: Inductive
Use data to discover concepts, themes, or models
Basic Text Analysis: Inductive
Use data to discover concepts, themes, or models
Evaluator as interpreter; highly involved
Basic Text Analysis: Inductive
Use data to discover concepts, themes, or models
Evaluator as interpreter; highly involved
Emergent, “bottom up”
Basic Text Analysis: Inductive
Use data to discover concepts, themes, or models
Evaluator as interpreter; highly involved
Emergent, “bottom up”
Qualitative outcome: key themes or categories relevant to evaluation/research questions
Application: Inductive Analysis
Facebook posts
Tweetchats or hashtags
Blog posts
LinkedIn discussions
Basic Inductive Analysis: 5 Steps
Step 1. Collect your raw data
Step 2. Read. And Read Again.
And:
Get organized!
Step 3. Create and Apply Codes (Repeat.)
Step 4. Refine Codes to Reduce Overlap
Step 5. Create Categories
Narrative Analysis (Step 6, Really.). .
Basic Text Analysis: Deductive
Data is analyzed according to prior assumptions
Basic Text Analysis: Deductive
Data is analyzed according to prior assumptions
Evaluator is “independent” from data
Basic Text Analysis: Deductive
Data is analyzed according to prior assumptions
Evaluator is “independent” from data
A-priori; “top down”
Basic Text Analysis: Deductive
Data is analyzed according to prior assumptions
Evaluator is “independent” from data
A-priori; “top down”
Quantitative outcome: metrics relevant to evaluation/research objectives
Application: Deductive Analysis
Category comparison, comparison over time Analyzing webinar chat pods Analyzing how a hashtag is leveraged in
Tweets Facebook/LinkedIn audience
engagement
Basic Deductive Analysis: 4 Steps
1.Develop data categories.
2.Clearly define those categories.
3.Read through all raw data and apply categories.
4.Count.
Chat Pod Engagement Metrics
Unique chat pod participants
Resources shared by participants
Resources shared by MFLN
Participant questions
Unique participant to participant exchanges
0 5 10 15 20 25
21
0
17
10
5
The fine print….
Only DCO viewers can participate in the chat pod; percentage of chat pod participants based on total number of DCO viewers and total number of unique participants.
Resources shared by participants include shared links, authors, studies, books, etc.; demonstrates high-level engagement because participants are contributing to the co-construction of knowledge during webinar.
Resources shared by MFLN include links, peer-reviewed studies and books, etc., from both MFLN and non-MFLN authors; demonstrates direct CA engagement with participants by further supporting and contextualizing knowledge construction by situating webinar presentation within the larger disciplinary area.
Participant questions are those listed in the chat pod; demonstrates intent to pursue two-way engagement in webinar and therefore high-level engagement.
Unique participant to participant exchanges are those in which chat pod participants respond directly to one another’s comments; demonstrates high-level engagement through realized reactive (two-way) and interactive (dependent) discourse patterns.
Chat pod text related to webinar content is not captured as an engagement measure due to its discursive category as declarative (one-way) communication. (It is noted, however, that declarative text is still understood to indicate webinar engagement, and MFLN encourages and values such participant engagement.)
Chat pod text related to technical issues and/or CEUs is not included in MFLN evaluation.
Storytelling
Storytelling
Identify narratives that connect to your evaluation aims
Storytelling
Identify narratives that connect to your evaluation aims
Be strategic and leverage stories for evaluation task at hand
Storytelling
Identify narratives that connect to your evaluation aims
Be strategic and leverage stories for evaluation task at hand
Contextualize your stories with other data to show a larger picture
Storytelling
Identify narratives that connect to your evaluation aims
Be strategic and leverage stories for evaluation task at hand
Contextualize your stories with other data to show a larger picture
Ethics, ethics, ethics
Storytelling: How?
Watch for stories
ASK for stories
Tell your own stories
From the Master Gardeners…
“On a Celebrex commercial a guy is shown bent over in some beets or chard and he raises up with a beautiful eggplant! The first time I laughed at it my wife thought I was crazy.”
Application: Storytelling and Evaluation
Use stories in your reports, and include an executive summary of those stories
Incorporate compelling stories with facts and figures
Include stories with direct quotes in press releases, on Web sites
Include stories and quotes in newsletters, brochures, annual reports
Larger Considerations
Larger Considerations
Reflexivity
Transparency
Credibility
Ethics
