Social Media Intelligence: Measuring Brand Sentiment from Online Conversations

David A. SchweidelGoizueta Business School

Emory University

October 2012

What’s Trending on Social Media?

Agenda• Social Dynamics in Social Media Behavior

– Why do people post a product opinion? What influences their posting behavior? (Moe and Schweidel 2012)

– What is the impact of these social dynamics on product sales? (Moe and Trusov 2011)

• Social Media Intelligence (work in progress with W. Moe)– What factors influence social media metrics?– How can we adjust our metrics for different sources of

data?

Why do people post?

• Opinion formation versus opinion expression (Berinsky 2005)

• Opinion formation– Pre/post purchase (Kuksov and Xie

2008)

– Customer satisfaction and word-of-mouth (Anderson and Sullivan 1993, Anderson 1998)

• Opinion expression– Opinion dynamics (Godes and Silva

2009, Li and Hitt 2008, Schlosser 2005, McAllister and Studlar 1991)

– Opinion polls and voter turnout (see for example McAllister and Studlar 1991)

Pre-Purchase EvaluationE[uij]

Purchase Decision and Product Experience

Post-Purchase EvaluationVij=f(uij, E[uij])

IncidenceDecision

EvaluationDecision

Posted Product Ratings

LATE

NT

EXPE

RIEN

CE M

ODE

LIN

CIDE

NCE

& E

VALU

ATIO

N M

ODE

LS

SELECTIONEFFECT

ADJUSTMENTEFFECT

Selection Effects: What influences participation?

• Extremely dissatisfied customers are more likely to engage in offline word-of-mouth (Anderson 1998)

• Online word-of-mouth is predominantly positive (e.g. Chevalier and Mayzlin 2006, Dellarocas and Narayan 2006)

• Subject to the opinions of others– Bandwagon effects (McAllister and Studlar 1991, Marsh 1984)

– Underdog effects (Gartner 1976, Straffin 1977)

– Effect of consensus (Epstein and Strom 1981, Dubois 1983, Jackson 1983, Delli Carpini 1984, Sudman 1986)

Adjustment effects:What influences posted ratings?

• Empirical evidence of opinion dynamics– Online opinions decline as product matures (Li and Hitt 2008)

– Online opinions decline with ordinality of rating (Godes and Silva 2009)

• Other behavioral explanations of opinion dynamics– “Experts” differentiate from the crowd by being more negative

(Schlosser 2005)

– “Multiple audience” effects when opinion variance is high (Fleming et al 1990)

Modeling Overview

• Online product ratings for bath, fragrance and home retailer over 6 months in 2007 (sample of 200 products with 3681 ratings)

• Two component model structure (Ying, Feinberg and Wedel 2006):– Incidence (Probit)– Evaluation (Ordered Probit)

• Product utility links incidence and evaluation models (non-linear)

• Bandwagon versus differentiation effects– Covariates of ratings environment– Separate but correlated effects in each model component

• Product and individual heterogeneity

Role of post-purchase evaluation (Vij)in rating incidence

-0.7

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

01.5 2 2.5 3 3.5 4 4.5 5

Util

ity E

ffect

on

Post

ing

Inci

denc

e

Individual-Product Utility Value

Classifying Opinion Contributors

• Groups based on frequency of posts (β0)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

Prop

ortio

n of

Pos

ters

Rating

% Activists % Low-Involvement

0

0.5

1

1.5

2

2.5

3

3.5

3

3.2

3.4

3.6

3.8

4

4.2

4.4

4.6

4.8

5

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

Varia

nce

Aver

age

Rat

ing

Rating

Average Variance

Empirical Trends• Posted ratings

– Average rating decreases over time

– Variance increases over time

• Poster composition– Community-builders are over-

represented in the posting population.

– As forum evolves, participation from community-builders increases while that of LI and BW decreases.

0

0.5

1

1.5

2

2.5

3

3.5

3

3.2

3.4

3.6

3.8

4

4.2

4.4

4.6

4.8

5

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

Varia

nce

Aver

age

Rat

ing

Rating

Average Variance

0

0.5

1

1.5

2

2.5

3

3.5

3

3.2

3.4

3.6

3.8

4

4.2

4.4

4.6

4.8

5

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

Varia

nce

Aver

age

Rat

ing

Rating

Average Variance

Effect of Opinion Variance

• Customer bases have same mean but different variance in opinions

• With a polarized customer base, ratings exhibit:– Lower average with a significant decreasing trend– Greater variance

• Negative ratings do not necessarily signal a lower average opinion among customers

MEDIAN CUSTOMER BASE POLARIZED CUSTOMER BASE

Conclusions: Individual-level Analysis• Empirical findings

– Heterogeneity in posting incidence and evaluation– Incidence and evaluation behavior are related and can result in a

systematic evolution of posting population

• General trends in the evolution of product forums:– Dominated by “activists””– Participation by activists tends to increase as forum evolves while

participation by low-involvement individuals tend to decrease

• Implications:– Ratings environment does not necessarily reflect the opinions of the

entire customer base or even the socially unbiased opinions of the posters

– Posting behavior is subject to venue effects…

Posting Decisions

Opinion / Brand Evaluation

Do I post? What do I post?

Sentiment

Product

Where do I post?

Attribute

Venue Format

Domain

SOCIAL MEDIA METRICS

The value of social media as a research tool

• Does it matter where (i.e., blogs, microblogs, forums, ratings/reviews, etc.) we listen?– Product/topic differences across venues?– Systematic sentiment differences across venues?– Venue specific trends and dynamics?

• How do social media metrics compare to other available measures?

Social media monitoringIN PRACTICE

• Early warning system: Kraft removed transfats from Oreos in 2003 after monitoring blogs

• Customer feedback: Land of Nod monitors reviews to help with product modifications and redesigns

• Measuring sentiment: Social media listen platforms collect comments across venues

IN RESEARCH• Twitter to predict sock prices

(Bollen, Mao and Zeng 2011)

• Twitter to predict movie sales (Rui, Whinston and Winkler 2009)

• Discussion forums to predict TV ratings (Godes and Mayzlin 2004)

• Ratings and Reviews to predict sales (Chevalier and Mayzlin 2006)

Does source of data matter?• Online venues (e.g., blogs, forums, social networks, micro-blogs) differ

in:– Extent of social interaction– Amount of information– Audience attracted– Focal product/attribute

• Venue is a choice– Consumers seek out brand communities (Muniz and O’Guinn 2001)

– Venue depends on posting motivation (Chen and Kirmani 2012)

• Social dynamics affects posting (Moe and Schweidel 2012, Moe and Trusov 2011)

Research Objective• Assess online social media as a listening tool

• Disentangle the following factors that can systematically influence posted sentiment– Venue differences– Product and attribute differences– Within venue trends and dynamics

• Examine differences across different venue types– Sentiment– Product and attribute differences– Implications for social media monitoring and metrics

Social Media Data• Provided by Converseon (leading online social

media listening platform and agency)• Sample of approximately 500 postings per month

pertaining to target brand• Comments manually coded for:

– Sentiment (positive, neutral, negative)– Venue and venue format– Focal product/attribute

• Categories: (1) enterprise software, (2) telecommunications, (3) credit card services, and (4) automobile manufacturing

Data for Enterprise Software Brand• 140 products within the brand portfolio• 59 brand attributes (e.g., compatibility, price, service, etc.)• Social Media data spanned a 15 month period

– June 2009 – August 2010– 7565 posted comments– Across 800+ domains

Venue Format Illustrative Website Frequency Direct ExperienceDiscussion Forum forums.adobe.com 2728 93%

Micro-blog twitter.com 2333 37%Blog wordpress.com 2274 23%

Social Network linkedin.com 155 40%Mainstream News cnbc.com 36 3%

Social News digg.com 19 47%Wiki adobe.wikia.com 10 50%

Video vimeo.com 6 0%Review Sites epinions.com 4 25%

Modeling Social Media Sentiment• Comments coded as “negative”, “neutral”,

or “positive”• Ordered probit regression

)Pr()Pr( 1)(

*)(

rivii

rivi Ury

1),(1),(*

iaipii VSU

venue-specific brand sentiment

product effect attribute effect

What affects venue-specific brand sentiment?

• General brand impression (GBI)• Domain and venue effects (including

dynamics)

)(),()()()( itivividiti GBIVS

domain effect

venue-format effect

venue-specific dynamics

Venue Attractiveness

• Model venue format as a choice made by the poster

• Multinomial logit model

effect of content on venue choice

product effect Attribute effect

Model Comparisons

• Baseline model– Independent sentiment and venue decisions– Controlling for product, topic and domain effects

DICSentiment

hit rateVenue choice

hit rateBaseline 32588.0 0.454 0.316+ Sentiment link 30040.7 0.451 0.424+ Venue specific sentiment 29677.2 0.465 0.424+ Venue specific dynamics 29529.1 0.473 0.424

Sentiment metrics vary depending on what you are measuring.

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

GBI

Observed AverageSentiment

GBIObs. Avg. Sentiment Blog Forum Microblogs

GBI 1

Obs. Avg. Sentiment -0.0346 1

Blog 0.678 0.496 1

Forum 0.00409 0.263 -0.0870 1

Microblogs 0.751 0.503 0.8196 -0.139 1

CORRELATIONS

Sentiment Differencesacross Venues (v(i))

0

0.5

1

1.5

2

2.5

3

3.5

Divergent Trendsacross Venues (v(i),t(i))

• Sentiment varies across venues• Venue-specific sentiment is subject to

venue-specific dynamics

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Blog

Forum

Micro-blog

* Blogs, forums and microblogs are the 3 most common venues

Attribute Effects on Sentiment (a(i),1)

• Provides attribute-specific sentiment metrics• Empirical measures are problematic due to data sparsity• Correlation between model-based effects and observed attribute-sentiment

metrics = -.276

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

Appl

icati

on

Com

patib

ility

Bran

d Re

puta

tion

Docu

men

tatio

n/Su

ppor

t

Ease

of I

nsta

llatio

n

Har

dwar

e Co

mpa

tibilit

y

Mob

ile C

ompa

tibili

ty

Pric

e

Qua

lity

of P

rodu

ct

Secu

rity

Size

of C

ompa

ny

Venue Attractiveness ResultsVenue Intercept (g)

Product/Attrib. Effects (l) Effect of GBI (k)

Blog 5.012 1.000 -1.610Forum 5.655 -2.713 5.002Mainstream Media 0.246 2.474 -2.632Microblogs 5.378 1.005 2.978Photoshare -4.509 -0.469 0.321Review site -1.229 -2.427 0.767Social network 2.349 1.771 1.086Social news aggregator 0.607 -0.058 1.848Video share -1.024 0.784 -1.105Wiki -- -- --

Posters with positive sentiments toward the brand are attracted to forums and microblogs.

Forums attract comments that focus on different products and attributes than microblogs.

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8In

dust

ry N

ews

Corp

orat

e Pa

rtne

rshi

pM

arke

ting/

Cam

paig

nM

arke

ting

Clou

d Co

mpu

ting

Mic

roso

ft P

artn

ersh

ipM

arke

t Sha

reFu

ture

of V

irtua

lizati

onPr

ivat

e/In

tern

al C

loud

Infr

astr

uctu

re C

ost

Serv

er V

irtua

lizati

on T

echn

ical

Issu

eI/

O P

erfo

rman

ceCu

stom

izatio

nEa

se o

f…Re

sour

ce o

ptim

izatio

nG

raph

ics s

uppo

rtHa

rdw

are

Com

patib

ility

Appl

icati

on C

ompa

t.Pr

otoc

ol C

ompa

tibili

ty

Attribute effects on venue choice (a(i),2)

TOP 10 BOTTOM 10

* For products with >5 mentions only

Predictive Value of GBI

• Offline brand tracking survey– Satisfaction surveys conducted from Nov 2009 to Aug 2010 in waves

(overlapping with last 10 months of social media data)– Approximately 100 surveys conducted per month– 7 questions re: overall sentiment toward brand

• Company stock price– Weekly and monthly closing prices for firm– Weekly and monthly closing S&P– June 2009 to September 2010 (extra month for lag)

GBI vs. Offline Survey• Potential for GBI as a lead

indicator

• Correlation with survey– GBI(t) = .375 [.277,.469]*– GBI(t-1) = .875 [.824,.919]*– Avg sentiment = -.0346– Blogs = .678– Forums = .00410– Microblogs = .751

8.75

8.8

8.85

8.9

8.95

9

9.05

9.1

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

Surv

ey Se

ntim

ent

GBI

Sen

timen

t

GBI at t Survey

8.75

8.8

8.85

8.9

8.95

9

9.05

9.1

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

Surv

ey Se

ntim

ent

GBI

Sen

timen

t

GBI at t-1 Survey

GBI and Stock Price(DV=monthly close)

Coeff StdErr p-val

Constant -69.045 34.044 0.070

S&P* 0.104 0.031 0.008

GBI(t) -16.695 10.324 0.137

GBI(t-1) 30.693 10.375 0.014

Adj R-sq .475

Posterior Means

* Closing price in month

Median Coeff

% p-val <.05

Constant -67.096 0.138

S&P* 0.102 1

GBI(t) -15.872 0.012

GBI(t-1) 29.921 0.9892

Adj R-sq .4635

Iteration level

Observed Social Media Metrics(DV=monthly close)

Average Blogs Forums Microblogs

Coeff p-val Coeff p-val Coeff p-val Coeff p-val

Constant -44.550 30.776 0.178 -93.215 31.675 0.015 222.605 52.477

S&P** 0.073 0.026 0.018 0.071 0.021 0.007 -0.014 0.021

SM(t) 34.995 17.181 0.069 6.059 8.055 0.469 -42.767 10.838

SM(t-1) 5.078 18.008 0.784 18.267 8.289 0.052 -39.252 10.840

Adj R-sq 0.404 0.547 0.753 0.390* Closing price in month

Conclusions• Social media behavior varies across venue formats

Need to account for the source of SM data

• Potential to use social media as market researchAdjusted measure (GBI) can serve as lead indicator

• Implications for academic research that use social media measures and for practitioners who monitor social media sentiment

• Next steps– Additional data sets– Simulations of different GBI scenarios and resulting metrics