Fron%ers of Computa%onal Journalism

Columbia Journalism School

Week 13: Tracking Effects December 12, 2014

Evolu%on of Media Effects Theory

1930s: "Hypodermic needle" or "magic bullet" theory. Whatever is communicated is taken up wholly. 1950s: "Two-step" theory. Media influences "opinion leaders" who then influence individuals through interpersonal contact. 1970s: Agenda-seSng. Media doesn't tell people what to think, but what to think about. 2000s: Online data makes first mass informa%on diffusion studies possible.

Media and vo%ng study, 1940 Elec%on

"One of the surprising results of The People's Choice study was that the impact of media exposure to campaign messages was rather negligible in terms of conversion changing voters from an inten%on to vote for one candidate to an inten%on to vote for the other. A more significant effect of media exposure was reinforcement assuring voters that their current inten%on was worthwhile and correct."

-- Glenn G. Sparks, Media Effects Research, A Basic Overview

Two-step model: Decatur Study, 1955

Surveyed 800 women in Decatur, Illinois, to determine how they made purchasing choices (study funded by an adver%ser.) Found personal contact much more influen%al than media messages. Solidified idea of "opinion leader" who is influenced by media, and then influences others personally.

Agenda-SeSng Theory "In choosing and displaying news, editors, newsroom staff, and broadcasters play an important part in shaping poli%cal reality. Readers learn not only about a given issue, but also how much importance to a_ach to that issue from the amount of informa%on in a news story and its posi%on. In reflec%ng what candidates are saying during a campaign, the mass media may well determine the important issues - that is, the media may set the "agenda" of the campaign."

-- McCombs and Shaw, The Agenda-SeBng FuncDon of Mass Media,1972

Agenda-SeSng Theory Points out that the media is how we perceive all of the world beyond our personal experience. Argues that, therefore, media defines the issues that we think about. Media produces salience. 1968 Chapel Hill elec%on study showed high correla%on (r>0.9) between news content and surveys asking undecided voters what their personal "issues" were.

Agenda-SeSng Research Methodology (1968)

1) Ask 100 undecided voters:

"What are you most concerned about these days? That is, regardless of what poli%cians say, what are the two or three main things which you think the government should concentrate on doing something about?"

2) Content analysis of media sources.

"For the Chapel Hill community almost all the mass media poli%cal informa%on was provided by the following sources: Durham Morning Herald, Durham Sun, Raleigh News and Observer, Raleigh Times, New York Times, Time, Newsweek, and NBC and CBS evening news broadcasts."

The metrics newsrooms have tradi%onally used tended to be fairly imprecise: Did a law change? Did the bad guy go to jail? Were dangers revealed? Were lives saved? Or least significant of all, did it win an award? But the math changes in the digital environment. We are awash in metrics, and we have the ability to engage with readers at scale in ways that would have been impossible (or impossibly expensive) in an analog world. The problem now is figuring out which data to pay a_en%on to and which to ignore. It is about seSng up frameworks for tes%ng, analysis and interpreta%on that are both scalable and replicable.

-- Aron Pilhofer (New York Times senior newsroom dev), Finding The Right Metric For News

"Effect"

If journalism has an effect, it must change belief

or behavior.

In principle, this should be detectable.

Key idea: Output vs. Outcome

To measure effects, you must measure something outside of your newsroom. Coun%ng the number of stories published is "output." Coun%ng the number of people who believed or acted aier viewing the story is "outcome."

Newsroom

Government

Industry

Stories

be_er life for user

Informa%on and Services

Products

Output Outcome

Why is measuring journalism effects hard?

Widely varying expecta%ons for different stories.

Different types of effects merely "informed" vs. took ac%on

Many causes for change in user, impossible to isolate journalism.

Long %me lags, indirect effects.

Different Types of Journalism

Entertainment Simple informa%on Buying decisions Drawing a_en%on to wrongdoing Explaining something complex

Voter Misinforma%on Survey, 2010

from MisinformaDon and the 2010 ElecDon: A Study of the US Electorate, WorldPublicOpinion.org and Knowledge Networks

Voter Misinforma%on Survey, 2010

Ideas from other fields: academic publishing

Public Library of Science "ar%cle-level metrics"

Ideas from other fields: social science research

From Donovan and Hanney, The "Payback Framework" Explained

Mixed-method effects tracking

Build a database which tracks, for each story:

1. Response: all tweets, status updates, comments, blog posts, links...

2. Claims of effect. Who said that this story changed something, and what they said.

3. Story metadata. Publica%on date, author, URL, category, etc.

AFAIK, no one has done this yet.