A/B Testing on the Obama Campaign

Winning With Data

Winning with A/B Testing

What impact can testing have?

Testing = constant improvement Little improvements add up

Improving 1% here and 2% there isn’t a lot at first, but over time it adds up

Tests upon tests upon tests

Subject & draft tests Full-list tests Background personalization tests

Review: Every piece of communication was an opportunity to test

A single email often had many tests attached

Lessons

Click icon to add picture

Don’t Trust Your Gut

Lesson #1

Don’t trust your gut

We don’t have all the answers Conventional wisdom is often wrong Long-held best practices are often wrong You are not your audience

There was this thing called the Email Derby… If even the experts are bad at predicting a

winning message, it shows just how important testing is.

Foster a culture of testing

Lesson #2

The culture of testing

Check your ego at the door

Use every opportunity to test something

Compare against yourself, not against your competitors or “the industry” Are you doing better this month than last

month? Are you doing better than you would have

otherwise?

Use data to make the user experience more personal

Lesson #3

Big data ≠ big brother

Testing allows you to listen to your user base Let them tell you what they like Whether through A/B testing or behavioral

segmentation, optimization gives them a better experience

Usually, the interactions that are the most human are the ones that win

Be human!

In general, we founds shorter, less formal emails and subject lines did best. Classic example: “Hey”

When we dropped a mild curse word into a subject line, it usually won “Hell yes, I like Obamacare” “Let’s win the damn election” “Pretty damn cool”

Good segmentation: behavioral Behavioral segmentation was much more

effective than demographic segmentation Donor vs. non-donor High-dollar vs. low-dollar Volunteer status What issues do people say they care about?

After using A/B tests to create a winning message, we could tweak it slightly for various behavioral groups and get better results

Conclusions

Conclusions

Test everything, especially your gut instinct

Foster a culture of testing

Use data to make it personal