how we use machine learning to push the boundaries of programmatic
TRANSCRIPT
How we use Machine Learning to push the
boundaries of Programmatic
Nicolas Beguin
P r o g r a m m a t i c c a m p a i g n s a r e u s i n g D S P ’ s a l g o r i t h m s
What happens if we create our own algorithms in addition to DSP’s ones?
We made the test on AppNexus thanks to APB
W h a t i s i t ?
APB stands for AppNexus Programmable Bidder
New AppNexus feature that allows you create your own custom algorithm to adjust your bids
W h a t i s i t ?
Strategy 1
Strategy 2
Strategy 3
Strategy 4
Without APB
$20
$15
$12
$30
W h a t i s i t ?
Strategy 1
Strategy 2
Strategy 3
Strategy 4
Without APB
$20
$15
$12
$30
With APB
$18 at 2pm
$14 at 4pm
$25 at 7pm
Strategy 1
E x a m p l e o f a d e c i s i o n t r e e
recency < 3h recency > 3h
day = monday = fri
day = tue
$10 $20 $15
day = thu
day = tue
creative size > 800
creative size = 800
$12
$35 $5
E x a m p l e o f a d e c i s i o n t r e e
Written in Bonsai
Handled by MFG Data Scientists
Uploaded to console via the API
Up to 300.000 lines!!
H o w a t r e e i s b u i l t
Step 1: CVR or CTR estimator what is the probability of a conversion or a click?
H o w a t r e e i s b u i l t
Step 1: CVR or CTR estimator what is the probability of a conversion or a click?
Step 2: Bid price optimization knowing the probability, how much should we bid?
H o w a t r e e i s b u i l t
Step 1: CVR or CTR estimator what is the probability of a conversion or a click?
Step 2: Bid price optimisation knowing the probability, how much should we bid?
Step 3: Transform into a tree once the model is known the tree can be created
I n s i g h t s f o r t h e m o d e l
W h a t w e d i d – Te s t 1 : S i m p l i f i c a t i o n
Combining individual strategies into one APB line:
Recency Desktop Bid Price
0-3 hours $9.17
3-12 hours $7.87
12-48 hours $10.47
2-7 days (48-168 hours) $6.57
7-15 days (168hours-360 hours) $9.17
Pixels Funnel Position Classification Bid Price
New Cars Models Upper $ 3.50
New Cars Models Offers Upper $ 4.50
New Cars Model BookTestDrive Middle $ 4.50
Dealer Locator Lower $ 6.10
Brochure Request Lower $ 7.10
Test Drive Conversion Lower $ 8.50
E n s u r i n g a f a i r t e s t - A / B C o n t r o l l e d G r o u p
Equal daily budget allocation to allow for fair test
W h a t w e d i d – Te s t 2 : G r a n u l a r i t y
Realised that pacing controls are sacrificed with current setup
Paired each individual non-APB strategy with a APB counterpart with far more granular optimisations and varying bidsExample: one APB line for every normal strategy
Results:
Steady performance ~25% reduction in CPA for APB lines overall and over 3 month period
C a s e S t u d y : K i a Ta c t i c a l
Budget: £30,000
Date: Oct 2016 – Dec 2016
KPI: £9 CPA
Retargeting campaign running on Mediamath and Appnexus
Appnexus split into APB and non-APB strategies:▪ Non APB: Dealer locator 0-12h, dealer locator 12h-7d, Models, and
Test drive request
▪ APB: Retargeting these pixels with optimal bid price
£- £1.000,00 £2.000,00 £3.000,00 £4.000,00 £5.000,00 £6.000,00 £7.000,00 £8.000,00
£- £2,00 £4,00 £6,00 £8,00
£10,00 £12,00 £14,00 £16,00
CPA Vs Gross Cost WoW
Gross Cost Non-APB CPA APB CPA
0,0%
20,0%
40,0%
60,0%
80,0%
100,0%
120,0%
-
50
100
150
200
250
10/10/2016 -10/16/2016
10/17/2016 -10/23/2016
10/24/2016 -10/30/2016
10/31/2016 -11/06/2016
11/07/2016 -11/13/2016
Conversions Volume Vs RR WoW
Non-APB Conversions APB Conversions Non-APB RR APB RR
£- £1.000,00 £2.000,00 £3.000,00 £4.000,00 £5.000,00 £6.000,00 £7.000,00 £8.000,00
£4,40 £4,60 £4,80 £5,00 £5,20 £5,40 £5,60 £5,80 £6,00
10/10/2016-
10/16/2016
10/17/2016-
10/23/2016
10/24/2016-
10/30/2016
10/31/2016-
11/06/2016
11/07/2016-
11/13/2016
CPM Vs Gross Cost WoW
Gross Cost Non APB CPM APB CPM
• APB drove a lower CPA WoW• Similar trend in conversion volume
and response rate • Non-APB had a lower CPM WoW
C a m p a i g n r e s u l t s
£-
£1,00
£2,00
£3,00
£4,00
£5,00
£6,00
£7,00
£8,00
£9,00
£-
£1.000,00
£2.000,00
£3.000,00
£4.000,00
£5.000,00
£6.000,00
£7.000,00
£8.000,00
10/10/2016 -10/16/2016
10/17/2016 -10/23/2016
10/24/2016 -10/30/2016
10/31/2016 -11/06/2016
11/07/2016 -11/13/2016
Retargeting CPA Vs Gross Cost WoW
Gross Cost Gross CPM Retargeting CPA
After Introduction of the APB line:
• Retargeting CPA decreased by 39%
• 13% increase in Response Rate
• 1,5x more conversions after introduction of APB
T H A N K Y O U