measuring the effectiveness of display adverting: a time...
TRANSCRIPT
Rel
ated
Wor
k
Met
hodo
logy
Res
ults
Mea
surin
g th
e Ef
fect
iven
ess
of D
ispl
ay A
dver
ting:
A
Tim
e Se
ries
App
roac
hJo
el B
araj
as*1
, Ram
Ake
lla1 ,
Mar
ius
Hol
tan2
, Jai
mie
Kw
on2 ,
Bra
d N
ull2
1 UC
San
ta C
ruz,
San
ta C
ruz
CA
, US
A2 A
OL
Res
earc
h, P
alo
Alto
CA
, US
A
Ref
eren
ces
Intr
oduc
tion
•O
nlin
e di
spla
y ad
verti
sing
is a
n ar
ea o
f rap
id g
row
th a
nd c
onse
quen
tly
of g
reat
inte
rest
as
a m
arke
ting
chan
nel.
•R
ecen
t stu
dies
sho
wth
at d
ispl
ay a
dver
tisin
g of
ten
trigg
ers
onlin
e us
ers
to s
earc
hfo
r inf
orm
atio
n ab
out c
omm
erci
al p
rodu
cts
[1].
•E
vent
ually
, man
y of
thes
e us
ers
perfo
rm e
ither
onlin
e co
nver
sion
s at
th
e ad
verti
ser's
web
site
or o
fflin
e co
nver
sion
s at
a p
hysi
cal s
tore
.
Met
hodo
logy
[1]A
tlas.
Eng
agem
ent m
appi
ng: A
new
mea
sure
men
t sta
ndar
d is
em
ergi
ngfo
r adv
ertis
ers,
200
8. W
hite
Pap
er.
[2]Z
oubi
n G
hahr
aman
i and
Geo
ffrey
E. H
into
n. P
aram
eter
est
imat
ion
for
linea
r dyn
amic
al s
yste
ms.
Tec
hnic
al re
port,
199
6.
[3]G
oogl
e. C
PApe
rfom
ance
tren
ds o
n th
e G
oogl
e di
spla
yne
twor
k, 2
010.
W
hite
Pap
er.
[4]R
anda
ll Le
wis
and
Dav
idR
eile
y. R
etai
ladv
ertis
ing
wor
ks!m
easu
ring
the
effe
cts
of a
dver
tisin
gon
sal
es v
ia a
con
trolle
dex
perim
ent o
nYa
hoo!
, 20
09. W
hite
Pap
er.
[5]M
ike
Wes
t and
Jef
f Har
rison
. Bay
esia
nfo
reca
stin
g an
d dy
nam
ic m
odel
s (2
nd e
d.).
Spr
inge
r-Ve
rlag
New
York
, Inc
., N
ewYo
rk, N
Y, U
SA
, 199
7.
[6]G
eorg
eC
asel
laan
d R
oger
L. B
erge
r. S
tatis
tical
Infe
renc
e(2
nd e
d.).
Dux
bury
, 200
2.
Onl
ine
disp
lay
ad s
how
n to
a us
er
•O
ur g
oal i
s to
mea
sure
the
effe
ctiv
enes
s of
dis
play
adv
ertis
ing
whe
n us
ers
are
expo
sed
to m
ultip
le a
dver
tisin
g ch
anne
ls.
Use
r is
exp
osed
tom
ultip
le a
dver
tisin
g ch
anne
ls in
tim
e
Eve
ntua
lly, t
he u
ser
perf
orm
s co
mm
erci
al a
ctio
ns
•If
a us
erpe
rform
s a
com
mer
cial
act
ion,
how
shou
ldth
e ad
verti
ser
attri
bute
cre
dit f
or th
e co
nver
sion
acr
oss
thes
e m
ultip
le c
hann
els
and
med
ia im
pres
sion
s?
•Th
is is
par
ticul
arly
impo
rtant
for t
he C
ost-
Per
-Act
ion
CPA
busi
ness
mod
el w
hen
deci
ding
how
to a
lloca
te
reso
urce
s ac
ross
adv
ertis
ing
chan
nels
.
•C
PApe
rform
ance
on
a m
onth
ly le
vel [
3].
Agg
rega
te s
tudy
bas
edju
st o
n th
e ob
serv
ed
aver
age
perfo
rman
ce
•A
stud
y ab
out u
ser
feat
ures
and
per
form
ance
on
com
mer
cial
act
ions
is
per
form
ed in
[4].
How
ever
, the
stu
dy w
as d
one
with
a s
mal
l set
of
user
s an
d ad
verti
sers
to h
ave
wel
l-def
ined
con
trol a
nd te
st g
roup
s.
•O
ur m
ain
goal
is to
mea
sure
the
effe
ctiv
enes
s of
cam
paig
ns o
n a
daily
leve
l. Th
is is
impo
rtant
for s
hort
cam
paig
ns a
nd to
pro
vide
dy
nam
ic o
nlin
e pe
rform
ance
est
imat
es.
•W
e fo
llow
a tim
e-se
ries
appr
oach
to d
ecom
pose
the
daily
num
ber
of
sale
s of
a g
iven
pro
duct
into
a w
eekl
y se
ason
al c
ompo
nent
and
a
trend
com
pone
nt u
sing
Dyn
amic
Lin
ear
Mod
els
base
don
Kal
man
fil
terin
g an
d sm
ooth
ing
idea
s [5
].
Cam
paig
n D
urat
ion
Day
s w
ith p
ositi
ve
sign
ifica
ntch
ange
WN
ww
G
VN
Fy
tt
tt
tt
tt
,0~
,0~
' 1
7/
2cos
/2
sin/
2sin
/2
cos
10
11
T
TT
TT
GG
Tp
2/,
,0,1,0,1,0,1
'T
Tp
GG
Gblockdiag
GF
Num
ber o
f com
mer
cial
act
ions
at t
imet
tyS
tate
evo
lutio
nof
the
mod
el c
ompo
nent
s (s
toch
astic
pro
cess
)t
•B
y fix
ing G
and F,
we
mod
el th
e se
ries
evol
utio
n as
a li
near
com
bina
tion
of a
sea
sona
l and
a tr
end
com
pone
nt [5
]•
Our
goa
l is
to d
esea
sona
lize
the
serie
s to
ass
ocia
te th
e tre
nd w
ith a
se
t of m
arke
ting
cam
paig
ns.
•W
e fin
dth
e M
LE o
f the
var
ianc
es Φ
= (V,W
)thr
ough
the
EM
alg
orith
m[2
] and
sm
ooth
the
serie
s to
ana
lyze
the
trend
com
pone
nt.
•W
e m
easu
re th
e ef
fect
iven
ess
of c
ampa
igns
by
com
parin
g sa
les
occu
rrin
gdu
ring
the
days
of t
he c
ampa
ign
fligh
t to
a ba
selin
e.
•W
e te
st s
tatis
tical
sig
nific
ance
of th
e ch
ange
in th
e tre
nd c
ompo
nent
ag
ains
t the
bas
elin
e.
•95
% c
onfid
ence
inte
rval
s ar
e us
ed to
det
ect a
n in
crea
se, d
ecre
ase,
orno
cha
nge
in th
e tre
nd.
E-step:
,
|log
|:1
:1,
|:1
:1T
Ty
iy
PE
Qi
TT
i
iQ
|
max
arg
1M-step:
Kal
man
Filt
erin
gan
d S
moo
thin
gO
ptim
izat
ion
to u
pdat
eth
e va
rian
ces (W
,V)
i
Q
|
1
i
*
0
iT
Ty
P,
|:1
:1
Cam
paig
n E
valu
atio
n•
We
use
the
prev
ious
day
of t
he s
tart
of a
cam
paig
nas
the
base
line.
•To
eva
luat
e th
e pe
rform
ance
of t
hese
cam
paig
ns, w
e de
fine
four
succ
ess
crite
ria.
1.A
vera
ge p
ositi
ve in
crea
se in
sal
es d
urin
g th
e ca
mpa
ign
2.In
crea
se o
r no
sta
tistic
al s
igni
fican
t cha
nge
insa
les
onev
ery
day
ofth
e ca
mpa
ign.
3.M
ore
days
with
incr
easi
ng s
ales
than
with
decr
easi
ng s
ales
.4.
Mor
e da
ys w
ithin
crea
sing
sal
es o
r no
sig
nific
ant c
hang
eth
an w
ithde
crea
sing
sal
es.
•W
e es
timat
e th
e av
erag
e ch
ange
of s
ales
dur
ing
the
cam
paig
ndu
ratio
n in
the
trend
com
pone
nt.
•W
e us
eth
e de
lta m
etho
dto
app
roxi
mat
e th
e di
strib
utio
nof
the
aver
age
chan
ge in
sal
es r
espe
ct to
the
base
line
•Th
e ap
prox
imat
ion
is b
ased
on th
e Ta
ylor
serie
s de
com
posi
tion
of th
e fu
nctio
n of
the
rand
omva
riabl
es [6
].
Dis
cuss
ion
•M
ain
goal
:–
Mea
sure
the
impa
ct o
f adv
ertis
ing
on s
ales
nea
ran
d lo
ng te
rm.
•D
iffic
ultie
s:–
Adv
ertis
ers
typi
cally
use
all
avai
labl
ech
anne
ls (
TV, r
adio
, prin
t, on
line)
and
ther
e ar
e m
any
vend
ors
with
in e
ach
chan
nel.
–D
ata
is ty
pica
lly n
ot in
tegr
ated
acro
ss c
hann
els
or e
ven
with
in
chan
nels
and
thus
the
idea
of f
ollo
win
g th
e on
line
clic
k-st
ream
beha
vior
of w
eb u
sers
is g
ener
ally
not
pos
sibl
e.•
Ach
ieve
men
tsan
d G
aps:
–H
igh
leve
l stu
dy a
cros
s se
vera
l tho
usan
ds o
f cam
paig
ns–
We
dono
t hav
e a
deta
iled
unde
rsta
ndin
g of
the
exac
t goa
ls o
f ea
ch c
ampa
ign.
–
Com
parin
g sa
les
agai
nst t
he s
ales
gen
erat
ed d
urin
g so
me
perio
d pr
ior
to th
e ca
mpa
ign
mig
ht n
ot a
lway
s be
app
ropr
iate
(th
e ca
mpa
ign
may
focu
s on
driv
ing
offli
ne s
ales
for i
nsta
nce
orla
unch
ed o
ff-se
ason
).–
Neg
ativ
esa
les
chan
ges
as m
easu
red
in th
is s
tudy
do
not
nece
ssar
ily m
ean
that
the
cam
paig
n fa
iled.
Res
ults
Cri
teri
on
1C
rite
rion
2
Cri
teri
on
3C
rite
rion
4
Suc
cesf
ul
Cam
paig
ns (%
)43
%65
%73
%84
%
Ave
rage
Incr
ease
of
Sal
es: S
ucce
ssfu
l C
ampa
igns
74%
42%
40%
31%
•D
istri
butio
n of
the
aver
age
chan
ge in
sal
es b
y ca
mpa
ign.
•
This
is b
roke
n do
wn
into
cam
paig
ns w
ith p
ositi
ve, n
egat
ive
and
no
stat
istic
ally
sig
nific
anta
vera
ge c
hang
e.
•W
e an
alyz
e 1,
635
cam
paig
nsfo
r429
pro
duct
sdu
ring
the
perio
d fro
m J
uly
1st, 2
009
to J
uly
31st
, 201
0.
•Th
e fo
llow
ing
tabl
es p
rese
nts
the
succ
essf
ul c
ampa
igns
ac
cord
ing
to th
e ab
ove
crite
ria.
Ong
oing
and
Fut
ure
Wor
k•
Inco
rpor
ate
the
num
ber
of im
pres
sion
s se
rved
as
adve
rtisi
ng a
ctio
ns in
a
mar
ketin
g ca
mpa
ign.
•D
oes
the
num
ber
of im
pres
sion
s ha
ve
an im
pact
to p
redi
ct th
e nu
mbe
rof
sa
les?
1
11
11
b
b
N tt
cc
c
N
1
11
11
b
b
N tt
cc
E
EE
NE
c
c
cc
c
N tt
bc
b
N tt
tbc
b
N tt
cb
N tj
tj
it
cN
E
E
NE
E
NE
11
12
31
11
24
1
2
11
22
1
11
11
,cov
2var
,cov
2var
var
cc
cE
N
var
,~
024681012141618
Percentage of Campaigns
[-50%
, -40
%]
[-40%
, -30
%]
[-30%
, -20
%]
[-20%
, -10
%]
[-10%
, 0%
][0%, 1
0%][10
%, 2
0%][20
%, 3
0%][30
%, 4
0%][40
%, 5
0%][50
%, 6
0%][60
%, 8
0%][80
%, 1
00%
][100%
, 150
%]
[150%
, 200
%]
[200%
, 250
%]
Sal
es P
erce
ntag
e C
hang
e
Dis
trib
utio
n of
Ave
rage
Cha
nge
in S
ales
Per
cent
age
for
All
Cam
paig
nsS
epar
ated
into
Pos
itive
, Neg
ativ
e an
d N
o-S
igni
fican
t Cha
nge
Com
pone
nts
Ori
gina
l Dis
trib
utio
nP
ositi
ve C
hang
eN
o S
igni
fican
t Cha
nge
Neg
ativ
e C
hang
e
* Con
tact
Aut
hor