prove it! a/b testing for nonprofits

Prove It! A Practical Guide to A/B Testing for Nonprofits

@NetTuesday // @bradyjosephson // #ntvan

4 More Years.

Culture of Learning.


Tonight.

• 3 Forms of Testing• 4 Steps to Testing

– 6 Sources for Insights– 5 Things to Avoid

• 1 Case Studies & 3 Other Examples

• 7 Resources for Testing

• 1 Live Test• ? Questions

• Tweet– #ntvan– @NetTuesday– @bradyjosephson

• Poll– @poll– Text to: 37607

• Me– [email protected]– recharity.ca– recharity.ca/category/

testing/


3 FORMS OF TESTINGHacking. Tracking. Cracking.


Hack.


Track.


Crack.


4 STEPS TO TESTINGInsights. Hypothesis. Test. Analyze.


Insights.

Things that provoke curiosity and make you want to test.

6 Sources for Insights• Survey. • Feedback. • Analytics. • Focus Group. • Social Media. • Your Brain.


Insights. // Dissonance.

Text REDCROSS to 30333


Hypothesis.

Tries to answer an important question with an expected result.

• If True? – Would we do differently?

• If False? – What would we do differently?

• Needed?– Is this question already answered?


Test.

The execution to get the data/results you need to make a decision.

Choose the (right) metric.State what is significant.Prepare your data.Control your variables.


Analyze.

Looking at the unfiltered data for a winner, what you learned and what it means moving forward.

Which one won?What else did you learn?How confident are you in the result?What are the next steps?– Run another test?– Change something in your strategy?


5 THINGS TO AVOIDConfounding Variables. Small Sample Size. Lack of Purpose. Wrong Metric. Skewed Data.


Confounding Variables.

Too many variables making it hard to know what caused what.

• Try to change as few things as possible and iterate more

• The more moving pieces you have the less you can trust the outcome


Sample Size.

Smaller the size the less confident you can be in the result.

• If you only have small sample sizes, run more tests

• Smaller sample sizes allow for more custom/manual work


No Purpose.

What changes with the outcome of this test or finding?

• Think of a decision you recently made. Now think of what information you wish you had to make that decision. That’s purpose.


Wrong Metric.

Choosing to measure the wrong thing to “prove” the winner.

• Email open rates measure subject lines (firstly) and timing (secondly) so unless you are testing those things it is a useless metric for a test.


Skewed Data.

Bad data gets compounded. Start with good data and you have a chance.

• Start capturing good/clean data right now• Don’t try to capture everything all at once• Just because you can track it doesn’t mean

that you should


1 CASE STUDY & 3 OTHER EXAMPLES

Email Appeals. Email Opens. Email Signups. Facebook Ads.


Email Appeals.

• Insight. – Current site revolves around projects, we are suggesting

it should revolve around stories.

• Question. – If we can prove more connection to stories, not projects,

it will help with site and communications revision.

• Hypothesis. – Appeals centered around stories, not projects, will

produce a greater response rate measured by gifts made and greater than 20%.


Email Appeals.

• Test. – Send two similar emails to randomized group

with slightly different funding focus.

• Analyze. – See which email has the greater response rate

as determined by gifts made.


Email Appeals. // Test.

• Randomized Group.– Excel and random function

• Set Up Goals.– Connect MailChimp and Google analytics

• Complications.– Small sample size– Additional emails



Choose the (right) metric.– Number of gifts.

State what is significant.– 20% difference.

Prepare your data.– Randomized in Excel.

Control your variables.– Length, design, landing page, timing all the same.


Email Appeals. // Analyze.


Email Opens. // Analyze.


Email Signups. // Test.


Email Signups. // Analyze.


Facebook Ads. // Analyze.


7 RESOURCES FOR TESTINGUnBounce. MailChimp. Google. KISS Metrics. HubSpot. Wufoo. Excel.


Resources.

Formswufoo.com

Analytics (and everything)Search “Google” on bing.com Marketing

hubspot.com

DataYour Computer

LTV & Awesome Blogwww.wufoo.com

Landing Pagesunbounce.com

Emailsmailchimp.com


1 LIVE TESTOn You.


Live Test. // A.

If you sign up to get emails from me today, I’ll send you a free copy of our eBook “Tool Kit for Charities”.

To sign up:• Text 599154 to 37607

OR• Tweet @poll with 599154


Live Test. // B.

If you sign up to get emails from me today, you’ll be entered for a chance to win a $25 gift card to Starbucks.

To sign up:• Text 599156 to 37607

OR• Tweet @poll with 599156


Live Test. // Analyze.


? QUESTIONS & THANKSFrom You.


THANKS!From me. @bradyjosephson // recharity.ca


prove it! a/b testing for nonprofits

Documents