big data ideas and...big data deep-dive an inside look at the magic of programmatic adroll’s data...

BIGIDEASDATAUsing Programmaticto Drive Creative Thinking

AND

SUMMARY EXECUTIVE

From Adobe’s witty marketing cloud videos to Coke’s tender Share a Coke campaign, great ad-vertising grabs our attention and tugs on our emo-tions. But modern advertising needs more than just a great concept to connect with consumers and drive sales. Brands need to stand out across devices, platforms, and social media channels to stand a chance in a crowded marketplace.

Great campaigns aren’t driven by media buys alone– They’re fueled by big ideas AND data-driven strategies.

In the age of big data, predictive analytics, and real-time personalization, marketing has become both a left- and right-brain profession. Modern advertising needs a healthy balance of creativity and technological proficiency to prosper. Creative marketers have access to more data on their cus-tomers than ever before, plus a myriad of technol-ogy solutions to help make sense of this data—but connecting these dots can be daunting.

So in the age of mass media consumption, how are the biggest brands harnessing creativity that’s powered by data?

We’ve gathered insights from the hottest brands and top creative minds, dug deep into the latest innovations in ad tech, and revealed the nuts and bolts of one click and mortar’s break-out holiday campaign to find out.

HIGHLIGHTS INCLUDE:≥ Big Data Deep-Dive: An inside look at the magic of programmatic and RTB

≥ Data is the New Currency: An in-depth look at how Alex and Ani dominated the holiday shopping season with creative powered by retargeting

≥ Creative Tips from Original Thinkers: Perspec-tives from the industry’s most innovative thought leaders from Publicis Kaplan Thaler, Marketo,Zendesk, Jonathan Adler, and Alex and Ani

≥ The Art of A/B Testing: Expert tips on how to drive creative performance to the next level with user op-timization testing

INSIGHTS FROM THE HOTTEST BRANDSUnleash the power ofdata-driven creativity

01

TABLE OF CONTENTS :

BIG DATABig Data Deep-DivePages 2–3Data is the New CurrencyPages 4–11

BIG IDEASCreative Tips from Original ThinkersPages 12–17The Art of A/B TestingPages 18–19

BIG DATA DEEP-DIVEAn inside look at the magicof programmatic

AdRoll’s datavolume is 30xlarger thanthe NewYork StockExchange

02

Personalized marketing has always been the advertis-er’s dilemma. However, advertisers from the 1960s like-ly never thought this challenge would be solved by math and computer science. Today, marketers connect with customers based on their state of mind, and continually learn from every brand interaction with programmatic and real-time bidding (RTB) technology.

In real-time bidding (RTB), advertising inventory is bought and sold on a per-impression basis. Adver-tisers bid on an impression and, if the bid is won, the buyer’s ad is instantly displayed on the publisher’s site. Through web cookies, CRM data, or mobile ids, advertisers analyze the level of intent a user may have in a brand or product and bid on ad buys accordingly. This media marketplace relies on machine learning to make millions of auctions for millions of individual ad impressions every single second.

Across all the impressions around the world, the speed at which we can operate is physically con-strained by the time it takes for computers and net-works to send signals back and forth from the adver-tiser to the publisher.

There’s no easy way around this problem—physics simply gets in the way. Light in fiber moves at 200,000 km/second. So a ray of light going between Paris and New York will take 60 milliseconds or more to travel round-trip. With only 100 milliseconds available to bid on ad impressions for advertisers, we need all the information possible: every impression, click, and

purchase, to make the right bid on the right user. So we made the decision to invest in product and innova-tion, rather than setting up data centers or securing bandwidth contracts to solve for the latency problem. That’s why we turned to AWS Cloud Services to help us quickly scale globally, in order to spend our time fo-cusing on making RTB models smarter.

The next challenge was storage. Simply put, a lot of data collected very quickly needs a storage solution that is easily accessible at the speed we demand.

We needed a globally scaled database that could answer 1 million requests per second, and wouldn’t be slowed or hindered by the amount of data pour-ing in and being queried. That’s when we turned to DynamoDB—and the 1 million requests per second that we’re now processing are each returned in 5 mil-liseconds or less.

Ultimately, this means that our own infrastructure, machine learning jobs, MapReduce tasks, etc., gen-erate over 10B requests each month on our buckets. With every impression, click, or purchase, data is collected and run through our proprietary algorithm, which increases the accuracy of every bid thereafter.

In 5 milliseconds or less, we analyze hundreds of intent signals to determine which ad should be shown to which user—as well as how much an ad-vertiser should pay for that impression. It’s incredi-ble how much big data has powered the ability to reach the right user at the right time.

AROUND THE WORLD IN100 MILLISECONDS OR LESS

With over 20,000 customers in more than 150 countries, AdRoll processes over 60 billion bid requests per day. Each one of these requests must be responded to within 100 milliseconds or less, no matter where it occurs in the world.

Around theWorld in 100Milliseconds

or Less

03

AdRoll processes

150 terabytesof data every day.

That’s 30x more

than the New York

Stock Exchange.

AdRoll has

1.0 bi l l ionanonymous user profiles.

That’s equal to

1/7 of the world’s

population.

AdRoll handles

60 bi l l ion

bid requests per day.

The world

collectively sends

less than half as

many texts daily.

AdRoll takes only

20 mil l iseconds

to respond to each ad request.

That’s faster than a

ray of light traveling

from Paris to New

York—and back.

NYSE

CAMPAIGN RESULTS:

TECHNOLOGY SOLUTIONS:

PIXEL Collected valuable customer data by simply placing a JavaScript tag across their website

REAL-TIME BIDDING (RTB) Took advantage of advanced algorithms won them the best ad inventory across the web and mobile

WEB RETARGETING Served personalized ads as potential customers browsed their favorite sites on the web

FACEBOOK RETARGETING Served native ads to engage the most vibrantcommunity online

CROSS-DEVICE RETARGETING Drove engagement from site visitors and CRM lists on the mobile web and within mobile apps

DYNAMIC CREATIVE Delivered personalized creative using beautiful,custom designs tailored to each customer

≥ Black Friday: 6.11x ROI ≥ Overall holiday season: 2.72x ROI ≥ Y/Y holiday CTR increased by 2.52x ≥ Y/Y click revenue doubled

BRIEFAlex and Ani is not just a jewelry retailer; it’s a life-

style brand with a story to tell. For the 2014 holiday

shopping season, Alex and Ani wanted to differen-

tiate themselves from the competition, delight hol-

iday shoppers, and bring personalized marketing

to all their digital channels. Rather than setting a

specific reach and frequency for messaging against

a broad demographic target, Alex and Ani opted to

zero in on their core customers and prospects. They

needed a technology platform that could help them

build a cohesive user experience in a crowded and

competitive media landscape.

SOLUTIONAlex and Ani developed a unique approach to pro-

grammatic ad buying. They used innovative re-

targeting strategies to connect the dots between

offline branding, strategic content curation, and

first-party data to grow market share and massively

increase holiday sales. The key was unlocking the

power of every retailer’s most valuable asset: its

customers.

By placing a single JavaScript tag on their web-

site, Alex and Ani collected behavioral data from site

visitors to create contextual ad campaign segments.

Their digital team used an integrated platform to ex-

ecute highly personalized campaigns and measur-

able retargeting strategies that they could optimize

in real time.

Alex and Ani looked at past browsing behavior

to target individual customers with recommended

products or look-alike products they might like to

purchase. They set up campaigns to serve personal-

ized messaging and creative to users based on their

stage of the customer journey. For example, visitors

who browsed the bangle, necklace, and ring catego-

ries saw category-specific ad creative. Visitors who

had viewed a specific product saw dynamic ads fea-

turing similar and recommended products. Visitors

who had made a purchase in the past were enrolled

in a loyalty segment that served promotional ban-

ners featuring new products. Alex and Ani reached

all these segments across the inventory sources

they were most likely to visit, and avoided ad burn-

out by serving different ads to users based on their

on-site behavior.

RESULTSAlex and Ani’s innovative retargeting campaigns

allowed the brand to differentiate themselves from

the competition and connect on an emotional level

with their customers. Programmatic technology

helped the lifestyle brand:

≥ Implement sophisticated segmentation to target

users throughout the buy cycle and deliver contex-

tual creative and messaging

≥ Highlight the craftsmanship of their product cat-

alog through beautiful dynamic creative and share

content with users in an authentic way

≥ Achieve an unprecedented 6.11x ROI on Black

Friday and drive the highest holiday season revenue

in company history

≥ Enhance the customer experience with the latest

RTB technology and algorithmic personalization

How click-and-mortar retailer Alex and Ani used sophisticated retar-

geting to build brand, generate sales, and increase ROI

CURRENCY

DATA IS THE NEW

04

AdRoll’s 2015 CannesLions Innovation Entry

FromDiscovery

to Purchase Alex and Ani used first-party data and advanced segmentationto entice the shopper at every stage of the buy cycle.

05

For consumers with less brand

familiarity, Alex and Ani serves ads

that exemplify the elegance and

personality of the lifestyle brand.

These ads feature stylish models

and intricate photography.

For consumers who have ex-

pressed interest in the brand, Alex

and Ani serves dynamic ads that

evoke the spirit and essence of a

particular jewelry category. The

creative for this segment uses color

and strategic product placement.

For consumers who have demon-

strated intent to buy a specific

product, Alex and Ani shows

creative that highlights that product

and its unique attributes.

For consumers who have made

past purchases, Alex and Ani

re-engage their community with

a variety of creative messaging,

including new collections, loyalty

promotions, and curated lifestyle

and beauty content.

Awareness

Consideration

Purchase

Loyalty

CTR CPA

Over the course of the year, Alex and Ani used programmatic ad buying to maximize ROI and experiment with new strategies.

AnnualPerformance

A hyper-targeted social media campaign for the Carnivals and

Caravans Collection integrates paid and organic social messaging to

drive massive ROI, and sustains performance throughout the season.

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

Experimentation with mobile segmentation widens the

brand’s pool of users. Capitalizing on a low CPA, Alex

and Ani garners an astonishing 3% CTR.

A geo-targeted Super Bowl campaign invigorates Seahawks

and Patriots fans with pre- and post-game messaging. These

campaigns peak with a .24% CTR.

On Black Friday, the most competitive period in digital

advertising, Alex and Ani’s robust campaign roster

reaches an impressive all-time low CPA.

06

Alex and Ani’s holiday campaign performance was accelerated by the introduction of additional holiday-specific messaging and campaign optimizations.

HolidayPerformance

Clicks

Attributed revenue

Black Friday, traditionally a brick-and-mortar

revenue driver, produces massive online revenue

for Alex and Ani through time-sensitive promos

and holiday-themed creative.

Fueled by retargeting campaigns

that cut through the online clutter,

Cyber Monday becomes Alex and

Ani’s highest-grossing online

revenue day.

Alex and Ani launches a broad pro-

motional campaign on Facebook for

a new collection of leather totes.

After the Black Friday promotion ends, Alex and

Ani sees a drop in online orders but a high volume

of clicks. These high-intent users set the stage for

a massive buying frenzy on Cyber Monday.

Thanks to creative messaging around shipping

to get loved ones’ presents under the tree in

time, Alex and Ani sees a spike in sales less

than 10 days before Christmas.

01 NOVEMBER NOVEMBER DECEMBER DECEMBER15 1501 31

07

The popular 600x315 ad size is perfect for social media and mobile inventory. During a time of year when consumers were sofa surfing and comparing priceson the go, this native-friendly size provided maximum engagement.

HolidayAd Unit

Performance

Total conversions

Impressions

600x315 300x250 728x90 160x600

08

Alex and Ani ran an impressive 72 campaigns throughout the holiday season. While some campaigns reached millions of users, Alex and Ani also ran hyper-segmented campaigns that targeted a select few.

HolidayCampaigns

Alex and Ani’s awareness

campaign shows incredible

performance as interest in

holiday gifting accelerates.

Loyalty campaign performance spikes

as Alex and Ani’s holiday-specific ads

encourage past customers to shop for

their friends and family.

A collection-specific campaign

consistently brings visitors back

to the Alex and Ani site. While

overall volume is low, it delivers

a higher percentage of clicks

than any other campaign.

Campaign activity becomes more frenzied as

Alex and Ani uses a variety of shopper intent

signals, such as time on the site, to match con-

sumers with the right holiday ad campaigns.

01 NOVEMBER NOVEMBER DECEMBER DECEMBER15 1501 31

09

CTR

Alex and Ani took advantage of a variety of messages, creative concepts, and ad formats to entice holiday shoppers. The top performers capitalized on customer intent across devices and social networks.

CreativePerformance

Size: 600x315Inventory sources: mobile, desktop

Campaign length: 39 daysAd type: static

Size: 600x315Inventory source: desktop

Campaign length: 61 daysAd type: dynamic





10

12.6%

8.5%

7.1%

3.9%

67.9%Percentage of OnlineSales by Ad Creative

Alex and Ani’s 2014 holiday season broke sales records and digital barriers with programmatic innovation.

Alexand Ani

The holiday campaign was wildly successful. On Black Friday 2014 alone, Alex and Ani saw over 6x ROI, exceeding their expected goal by more than 400%. The entire 2014 holiday season represented nearly 3x ROI over the previous year.

Overall, Alex and Ani saw incredibleresults from their holiday retargeting campaigns:

≥ Delivered contextual creative and mes-saging with sophisticated segmentation that targets users throughout the buy cycle

≥ Highlighted the craftsmanship of their product catalog through beautiful dynamic creative and shared content with users in an authentic way

≥ Achieved an unprecedented 6.11x ROI on Black Friday and drove the highest holiday season revenue in company history

≥ Enhanced the customer experience with the latest RTB technology and algorithmic personalization

11

12

ETHANKANAT

RICOCIPRIASO

EMILYPANZER

RYANBONIFACINO

creative brandmanager

VP, directorof CRM

VP, demandgeneration

ecommerceweb designer

chief marketingofficer & seniorVP, digital

CREATIVE TIPS FROM ORIGINAL THINKERS

HEIDIBULLOCK

“The bestcreative in the world doesn’treally meananything if itdoesn’t servea purpose.”

13

How do you balance data-driven strategy andemotionally driven concepts?We tend to think of the concept as “the idea” and the data as the “reason for the idea.” The best creative in the world doesn’t really mean anything if it doesn’t serve a purpose. That’s not to say we don’t do things just to be weird or whimsical or whatever—we do. But even those projects come from a well-informed place.

Sometimes they actually work in concert with each other. The Zendesk Alternative project is a good example. Our SEO team came to us with some data about our performance as compared to competitors’ optimized landing pages. In other words, a lot of people were searching the phrase ”Zendesk Alternative” and we were the only ones not taking advantage of it. So we took that infor-mation and created an entire world around this fake alternative band—and it worked. We succeed-ed in charming our audience and landing the pole position in search results for that term.

What are the top three ingredients for creative that amazes and inspires?

1. Concept is king. If you don’t have a solid con-cept, then the end result is going to be shit.

2. Know your audience. It’s a lot easier to craft your message if you know who you’re talking to—and what they want to hear.

3. Always pick the weird idea. This doesn’t work for everybody. To be honest, sometimes it doesn’t even work for us. Still, we always try to surprise people with something unusual. It makes it more fun.

How does data help you tell stories and connect with customers?For us, data is the information behind the action. It helps us unearth the stories themselves. Data helps us figure out where our customers are, what they’re doing, and what they’re looking for. And

any time you have people doing things and looking for answers, you have stories.

What’s the number-one measure of an awesome marketing campaign? Qualified leads. Kidding! As long as we deliver on the brief, then I would say the best measure of an awesome campaign is whether or not we’re proud of it. To outsiders, creative teams and agencies of-ten look like a bunch of liberal arts clowns goofing off all day, but we work hard to do what we do. If, at the end of the day, we feel good enough about the work we produce to put our name on it, then I’d say that’s a win.

How important is personalization to your advertising campaigns? To be honest, it isn’t really. As I mentioned above, we need to know our audience so that we can adjust our tone and craft our message. But at the end of the day, we’re telling the story of our brand and our customers. We don’t want to dilute our message trying to be all things to all people. We want to tell a story that helps the right people relate to us.

Creative Tips from Original Thinkers

KANATETHAN

creative brandmanager,Zendesk

“Personalizationis absolutelycritical and non-negotiablein today’smodern business landscape.”

14

How do you balance data-driven strategy and emotionally driven concepts?It’s a blend of art and science. We always have to be sensitive to the creative process in developing language and visuals that speak to a variety of customers. However, when you add elements such as predictive analytics, segmentation, and the ability to affect the customer interaction in real time, you can test and optimize at scale. The blending of these processes allows creative teams across brands and agencies to experiment with different messages, while increasing margins and profitability. We can now leverage math and data to hit the right customer with the right message.

What are the top three ingredients for creative that amazes and inspires? I would rank the three key components to a great creative output in the following order:

1. Strategy. I’m a very strategic thinker and always pump strategy into the equation before putting boots on the ground. A well-defined, thoughtful strategy should be the backbone of every project, big or small.

2. Process. Process means many different things, but the key elements are project management and resource planning. It’s imperative to understand where you hit your limits with creative resources and that it’s clear who is responsible for what delivera-bles. Workflow, deadlines, production management, and asset gathering are all part of this process.

3. Creative optimization. At this point, it’s about taking what you have and being very intelligent about how you use the output to connect with customers and create that one-to-one brand experience. So-phisticated segmentation and dynamic capabilities allow for successful optimization.

How does data help you tell stories and connect with customers?At Alex and Ani, consumer data sits directly in the middle of the organization. We know so much about our customers, from demographics, to the data we source from our stores and at events, to the behav-

ioral data we collect across our online channels. The secret lies in marrying these different data sources. This strategy is critical not only in determining the allocation of media spend, but also understanding buying preferences, gift-giving patterns, and churn motivation across the customer journey. This data influences the way we design new products, ap-proach merchandising and packaging, and pick the influencers we partner with.

What’s the number one measure of a successful marketing campaign? It’s a matter of benchmarking. In the retail vertical, these measurements can be awareness- driven, engagement-driven, or performance-driven. On the product level, it’s about measuring per-formance in terms of incremental lift and scaling against audiences much larger than what we have. For content-driven campaigns, we look at how many people have an emotional experience with our brand, and then give us some bit of information about themselves.

One example is our partnership with singer/song-writer Ingrid Michaelson. We aim to get the word out, to our fans and hers, that we love each other. This allows both of our communities to discover each other, resulting in fans flocking to Ingrid’s YouTube channel or downloading her music, and more like-minded customers discovering our brand and products through our website and events. User acquisition management, specifically net new, is a major measurement of success for storytelling and branding campaigns.

How important is personalization to your advertis-ing campaigns? Personalization is absolutely critical and non-ne-gotiable in today’s modern business landscape. The question is how can we optimize personalization for the digital experience and translate that to our physical stores. You need centralized data to even have a discussion on this topic. For creative lead companies, product personalization will always lead the way. It’s an extremely high bar and we are always aiming to go beyond our expectations.


BONIFACINORYAN

chief marketingofficer & senior vice president,digitalAlex and Ani

“Peoplehave toget themessage FAST.”

How do you balance data-driven strategy andemotionally driven concepts?Emotionally driven concepts and a data-driven strategy can really go hand-in-hand. A richly designed concept is important—but how do you know if it achieved the marketing goal that it was intended to address? Marketers need the ability to listen and respond so they can understand which concepts work most effectively for the intended target audience. Marketers need data to under-stand if a) the campaign was really emotional for the target audience (i.e., not just the ad agency), and b) did it help achieve the business goal? A great ad concept can be leveraged across various chan-nels, and we now have the ability to understand the overall campaign performance, in addition to particular channel performance (e.g., mobile drove more conversions than social).

What are the top three ingredients for creative that amazes and inspires? It is really important to understand the perso-na first—because what amazes and inspires a 40-year-old woman in California may not be the same for a 40-year-old woman in New York. If the persona is well understood, the top three ingredi-ents for creative that amazes and inspires are:

1. Visually compelling. Does the imagery stand out and resonate with the audience?

2. Impactful. The copy AND visual has to hit a chord with the potential buyer that really speaks to them—meaning did it inspire, educate, or enter-tain?

3. Simple. People have to get the message FAST.

How does data help you tell stories and connect with customers?Data is critical because it helps you truly under-stand your customer. Today, it’s more important than ever before to have the ability to listen to your customer across all channels and understand behavior—not just what they tell you on a form or

survey. It is great if you can incorporate buyer in-tent data into your personas. You can also get more immediate feedback on what is working or not. For example, marketers can leverage marketing auto-mation solutions like Marketo to send communica-tions based on real buyer intent behavior. You can easily see what someone responds to. It is a great way to be targeted and personal and ensure you really know your buyer.

When you really understand what your customer likes, dislikes, worries about, gets excited about, you can develop more of a personal and relevant conversation with your buyer.

What’s the number one measure of an awesome marketing campaign? The number one measure of an awesome mar-keting campaign is if it hits or exceeds the goals you set out for it. Ideally you are clear about the goals from the start. For example, “by running this ad campaign, the team aims to increase average pageviews on our key product page from three to six by the end of the year, which will result in an X percent increase in conversion.” The goals can vary from increasing loyalty, re-activating buyers, etc., but a good campaign means it was effective in reaching those goals.

How important is personalization to youradvertising campaigns? Personalization is critical today—and buyers expect it. For example, just because I am a female in my early 30s does not mean I love clothes; my behavior may indicate that I like tools and building projects, and buyers want marketers to know that. It is also an exciting time for personalization. The idea is not a new one—marketers have talked about 1:1 mar-keting for 20 years—but now it is becoming real. For example, Marketo’s Ad Bridge makes digital advertising more personal by connecting Marke-to’s marketing insights with Facebook, Google, LinkedIn, and more. Now marketers can effectively target audiences based on who they are, their stage in the buyer journey, and their individual behavior over time to deliver more personalized digital ads.


15

BULLOCKHEIDI

vice president,demandgeneration,Marketo, Inc.

“You can’t rely onjust dataor justemotion.”

How do you balance data-driven strategy andemotionally driven concepts?There are two main elements: creativity and exe-cution. Many people overlook the fact that emotion is technically a data point itself. Data can help drive the creative concept, but only up to a point. To really make an impact you need an emotional component.

For execution, the behavioral data impacts ev-erything from the copy, the imagery, and the call to action. If the emotions are expressed appropri-ately, then you can use data to extract what you want from the individual viewer. The key is putting these elements into the appropriate context. You can’t rely on just data or just emotion.

What are the top three ingredients for creative that amazes and inspires?

1. Universal. It’s timeless. It’s a nugget of insight that works regardless of the device you are hold-ing, the websites you frequent, or how much mon-ey you make. The key is finding that message that resonates universally with your target audience.

2. Relevant. It’s an understanding that it’s not a one-size-fits-all world anymore. You have your key communication that is universal, but it needs to be relevant to the individual. It’s crucial to figure out how, where, and when to convey that message at every stage of the purchase cycle.

3. Appropriate. Success can only happen when you are cognizant of budget and tactics. You have to ask yourself: Given the universal truth and the relevance, what’s the appropriate form the creative should take? Do my budget allocations fit my universal truth?

How does data help you tell stories and connect with customers?The great thing about data now is that we are starting to connect digital activity to the individual. That activity provides a much better sense of the target, rather than the broad demographic re-search available. By leveraging the information we

have on each individual consumer, it makes every impression smarter. I don’t care about scale if it’s not delivering the right message to the right person.

There’s an old joke about how we don’t know where 50% of our advertising goes and if it’s working. The industry is moving to the point where we can actually identify 100% of impressions, and which ones are successful. That’s a terrific advantage of working with modern technology platforms.

What’s the number one measure of a successful marketing campaign? Incremental everything. We are no longer mea-suring campaigns by saying, “I got X many impres-sions.” It’s about, “How do I move the needle for the business?” Those measurements can span the gamut from awareness to sales dollars.

It’s important to get that differentiating mes-saging to the right people and reinforce it. It varies from client to client. For a retail client it’s about sales, or margin, or lifetime value, and so on. There’s so many different metrics, which is why I think incremental is the common thread.

How important is personalization to youradvertising campaigns? We may call it personalization, but it’s really clos-er to segmentation. We are able to leverage meta-data about an individual to create an experience that is relevant and contextual to them. The goal is to be able to access all the information available on an individual to provide a one-to-one marketing experience that aligns with the universal truth of the brand or product.

We aim to create as many segments as possible and then communicate that one-to-one message as best we can. To do that, you need to extract the most out of each marketing channel and the tech-nology tools associated with those channels.


16

CIPRIASORICO

VP, director of CRMPublicis KaplanThaler

“What we learn from the datais invaluable,but it has tobe balancedwith brand-right creative.”

How do you balance data-driven strategy andemotionally driven concepts?What we learn from the data is invaluable, but it has to be balanced with brand-right creative. I like to start with hard-working, on-brand cre-ative and tweak based on what the data tells us. There’s no magic formula for the right balance – there’s definitely a bit of intuition involved.

What are the top three ingredients for creative that amazes and inspires? Fantastic imagery will always evoke a response – it’s the presentation of it that is the movingtarget. So much goes into the photo shoots. I love to let the imagery really speak for itself and tell the brand story as much as possible with a supporting cast of a strong CTA and a few simple words or tagline to tell the story of the season or marketing initiative. How does data help you tell stories and connect with customers?Testing, testing, testing! It’s so interesting to learn about what our customers respond to. The more well-rounded view of your customer—their habits and how they interact with the prod-uct—the more you can tailor the product and advertising to them.

What’s the number one measure of anawesome marketing campaign? It really depends on the goal of the campaign. Some goals are brand awareness-focused and we won’t necessarily see that reflected in reve-nue. I’m lucky to work for a company that values great design above all, but money speaks really loud for any business. As a creative person I never thought numbers would be so critical to my job, but you have to be able to back up any decision with data.

How important is personalization to youradvertising campaigns? Dynamic ads are great to target specific prod-ucts a potential customer has been viewing, but otherwise, we don’t use a lot of personalization in our ads. For retargeting ads I keep in mind that the person viewing the ad is familiar with the brand on some level, but familiarity can range from coming to the site for 30 seconds to enter a sweepstakes or a VIP repeat customer, so the creative needs to be broad enough to speak to both.


17

PANZEREMILY

ecommerceweb designer,JonathanAdler

THE ARTOF A/BTESTING

Expert tips for improvingcreative performance

Continued on next page

Less than a decade ago, brands were limited to print, radio, and TV to connect with their tar-get markets. While marketers used tools like Nielsen to determine reach and demographic fit, there was no good way to know what media channels and concepts drove the bottom line for the business. With the proliferation of chan-nels, devices, and marketing automation tools, marketers have more insight in each individual campaign and can optimize the user experience incrementally.

In A/B testing you pit two different versions of the same piece of creative (whether it’s a web-site page, an email, or a lead gen form) against each other to measure the effectiveness of each approach. Parallel testing allows you to isolate the change you want to measure, and generate actionable results.

A/B testing lets you test everything from email subject lines, to landing page designs, to prod-uct positioning. And with tools like Optimizely, Hubspot and Marketo, it’s easier than ever.

But while these tools have made it extremely easy to implement these experiments, A/B test-ing can be a Pandora’s box. The following are tips and tricks for developing a successful frame-work, strategy, and execution plan.

Identify your hypothesis and how you willprove itThe scientific method is the heart of A/B testing, so it all starts with a hypothesis to test against. Your experiment should be structured to offer clear results for an actionable decision. Use qualitative inputs such as heat maps, surveys, user testing, and competitive analysis to help de-sign hypotheses, structure tests, and interpret the results.

If your hypothesis is, “I predict account signup conversion rate will increase if we provide customer proof on the landing page,” you will only change the customer proof variable. This experiment framework is targeted to produce concrete results. Use qualitative inputs such as heat maps, surveys, user testing, and competi-tive analysis to help design hypotheses, struc-ture tests, and interpret the results.

Avoid the pitfalls of testing assumptions It can be very difficult to admit when your ex-periment results don’t back up what has “always worked.” Split testing is about gathering action-

DRIVE CREATIVE PERFORMANCETHROUGH EXPERIENCE OPTIMIZATION

18

Parallel testingallows you toisolate the change you want tomeasure, andgenerateactionable results.

ADROLL TESTING CUSTOMER VALIDATION ON DEMO REQUEST PAGE

Original:

Test:

Spend, productadoptoin, churn

Account signups,conversions

Clicks, sales qualified leads, form fills

Opens,impressions, Traffic

Surface level metrics

Downstream results

able insight, not tallying up the wins. In fact, negative results are a good thing; they allow you to dive deeper into the elements of creative that actually matter, and those learnings help focus your future efforts.

Segment further to drill down on the “why,” and test if a smaller subset of your target re-sponds differently. Not every element of your creative is important to your conversion rate or success metric. Sometimes the headline, CTA button color, or landing page UX simply don’t affect the user experience in a significant way. Identifying this helps you address the things that are working and eliminate those areas from tests, so you can focus on results.

Analyze a variety of metrics to determine successThere are several variety of categories of metrics you can test against: surface-level metrics (opens, impressions, traffic); mid-funnel metrics (clicks, form fills, account signups, purchases); or down-stream results (customer spend, churn). Ideally, you can always tie test results to downstream busi-ness impact, but it may not always be practical.

Typically, you should balance the metrics for a particular experiment. Test and optimize using readily available surface metrics (like clicks), but also take a small control group at the same time to measure downstream results (like cus-tomer spend). You stay nimble and improve on the fly, but it also ensures you’re impacting the emotional experience of the user overall.

Iterative testing is keyApproach testing in small, digestible bites for best results. When crafting your hypothesis, it’s easy to get carried away with grand, high-level objectives that make it difficult to draw hard lines in the sand. For optimal results, strategize and then incrementally test. Testing small chang-es, one variable at a time, allows you to execute

targeted tests and easily implement the results. Don’t let your results go stale—customer behav-ior trends may change, and you should always be experimenting with new ideas and new variables.

Beware: You can’t test strategyNot everything is an optimization problem. Which products do your customers want? Who should your target audience be? Which customer behav-iors do you want to influence? A/B tests can’t de-finitively answer these questions.

For example, in product design you need re-search and customer feedback before you can start optimizing the packaging. Sometimes and A/B test may give you a piece of the puzzle, but the bigger questions should be answered and agreed upon first.

Make sure data is informing your optimization strategy, not dragging you down the rabbit hole. Creative thinkers should take risks and A/B test-ing will help refine the message, but it shouldn’t be the heart of your brief.

A/B testing has opened doors for marketers and created unprecedented transparency into what’s working and what can be improved. It’s now possible to optimize everything from the website home page design to drip email subject lines with a few clicks. However, it’s also easy to get caught up in the excitement of testing mania and over-rely on results that may not be statis-tically relevant or representative of your larger target market. Every marketing team should approach A/B testing a tool to help inform in-cremental optimizations as a tool that boost the bottom line. The key is finding the right balance between being open to new ways of approaching user experience and reliant on initial testing re-sults.

19

METRICS TO TEST THROUGHOUTTHE CUSTOMER JOURNEY

AdRoll is the global leader in retargeting with over 20,000 active advertisers worldwide.To reach beyond existing audiences, we’ve introduced AdRoll Prospecting to help busi-nesses attract new customers. Our innovative and easy-to-use marketing platform enables businesses of all sizes to create personalized ad campaigns based on their own website data, driving maximum return on online advertising spend. AdRoll provides a high de-gree of transparency and reach across the largest display inventory sources, including Google AdX and Facebook Exchange.

The company is backed by leading investors such as Foundation Capital, IVP, Accel Part-ners, Merus Capital, and Peter Thiel. For more information, please visit www.adroll.com.

AboutAdRoll

big data ideas and...big data deep-dive an inside look at the magic of programmatic adroll’s data...

Documents