Estimating online ad effectiveness

Kay H. BrodersenHal R. VarianGoogle,Inc.

January 7, 2015

A note about the paper

• Created using knitr

• Allows you to merge LATEXand R commands

• Runs R each time you change paper and caches results

• Inserts R code and output into paper

• Voila: reproducible research

Estimating online ad effectiveness

1. Apply treatment: change ad spend, bid, budget, etc.

1.1 Treat in some places but not others1.2 Treat in some times but not others

2. Compare to counterfactual: what would have happenedwithout experiment?

3. Of course, counterfactual can not be observed, so it must beestimated

Procedure

Train a model to predict metric of interest such as websitevisits or revenue.

Test the model on out-of-sample data to evaluateperformance.

Treat Apply model to periods or places where treatmentwas applied to estimate counterfactual

Compare actual and (estimated) counterfactual outcomes.

Hypothetical example of train-test-treat

Jun Jul Aug Sep

5000

6000

7000

8000

9000

1000

011

000

date

visi

ts

Train Test Treat Test

Bayesian Structural Time Series

We will do this in a time series context using BSTS (available fromCRAN.) BSTS combines:

Kalman filter. Accounts for seasonality and trend

Spike-and-slab regression. Automated selection of predictors

Bayesian model averaging. Avoids overfitting.

Described in Scott-Varian [2013,2014], Brodersen et. al. [2013].Related to “interrupted regression”, “synthetic controls”.

Predictors selected by BSTS

Law_and_Government__Public_Safety

Health__Vision_Care

Internet_and_Telecom__Service_Providers

Shopping__Photo_and_Video_Services

Computers_and_Electronics__Electronics_and_Electrical

Inclusion Probability

0.0 0.4 0.8

Prediction from BSTS

Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01

5000

7000

9000

1100

0

cbin

d(y,

y.p

red)

ActualCounterfactualInterval

Research strategy?

1. Use a parametric model for impact of ad campaign?• Benefit: Can use all the data to estimate• Cost: Restrictive functional form: a parallel shift

2. Use alternative estimation technique?

3. Use different models for seasonality and trend

1. Use parallel shift for ad impact

Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01

−10

000

1000

2000

3000

Index

pred

err

or

Boxcar variable

Boxcar indicator variable for campaign

Computers_and_Electronics__Electronics_and_Electrical

Sports

Online_Communities__File_Sharing_and_Hosting

Shopping__Photo_and_Video_Services

campaign

Inclusion Probability

0.0 0.4 0.8

2. Alternative estimation: linear model

Drop Kalman filter, just use simple linear model

• July 4 holiday dummy

• Top two categories from Google Trends as regressors

Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01

5000

7000

9000

1100

0

cbin

d(y,

y.p

red,

y.c

ount

erfa

ctua

l)

ActualPredictedCounterfactual

Deseasonalized the data first

Deseasonalize by fitting model with holiday regressor +day-of-week dummies. (Explain spike.)

Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01

−20

00−

500

500

1500

y.ds

Alternative approaches

1. Make no adjustment for seasonality (since predictor alreadyhas appropriate seasonality)

2. Deseasonalize both predictor and outcome• Use boxcar regressor• Use extrapolation

3. Alternative seasonality: detrend first

Deseasonalize by fitting model with holiday regressor + day ofweek dummies. (Explain spike.)

Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01

−20

00−

500

500

1500

cbin

d(y.

ds, y

.pre

d)

ActualCounterfactual

Use weekly data

Jun 15 Jul 01 Jul 15 Aug 01 Aug 15

6500

7500

8500

9500

cbin

d(y.

wee

k, y

.pre

d1, y

.pre

d2)

actualboxcarphoto

Summary

method estimate

1 bsts-extrap 1830.432 bsts-boxcar 1362.883 bsts-boxcar-all-predictors 1279.054 bsts-boxcar-top-predictors 1327.065 lm-boxcar 1434.576 lm-extrap 1289.197 not deseasonalized 1393.418 deseasonalized-boxcar 1300.679 deseasonalized-extrap 1298.37

10 week-boxcar 1248.61

What about revenue?

• Ad clicks may cannibalize search clicks

• May want to look at total number of clicks (i.e., visitors)

• But ad clicks may be worth more or less than search clicks, soreally want revenue (or profit)

• Can model ad revenue, search revenue separately or together

Examine a different advertiser using a different set of possiblepredictors

BSTS: predictor selection for visits

Movie.Rentals...Sales.Queries

Movie.Rentals...Sales.Matched

Movie.Rentals...Sales.Impressions

Movie.Rentals...Sales.Clicks

campaign

Inclusion Probability

0.0 0.2 0.4 0.6 0.8 1.0

BSTS: Visits actual and counterfactual

Uses the BSTS extrapolation model and a linear regression

Jan 14 Jan 21 Jan 28 Feb 04 Feb 11 Feb 18

9000

011

0000

1300

00

Index

cbin

d(y,

y.p

red.

top

− b

oxca

r, y.

pred

.bst

s)

ActualLinear modelBSTS

BSTS: predictor selection for revenue

Movie.Rentals...Sales.Revenue

Music.Clicks

Movie.Rentals...Sales.Impressions

Movie.Rentals...Sales.Clicks

campaign

Inclusion Probability

0.0 0.2 0.4 0.6 0.8 1.0

BSTS: Revenue actual and counterfactual

Uses the BSTS extrapolation model and a linear regression

Jan 14 Jan 21 Jan 28 Feb 04 Feb 11 Feb 18

1000

0014

0000

1800

00

Index

cbin

d(y,

y.p

red.

top

− b

oxca

r, y.

pred

.bst

s)

ActualLinear modelStructural time series

Revenue cannibalization

When campaign begins ad revenue increases significantly, organicrevenue falls somewhat.

6080

100

120

sear

ch

Jan 14 Jan 21 Jan 28 Feb 04 Feb 11 Feb 18

Index

020

6010

0

ads

cbind(search, ads)