HOUSEKEEPING -  This presentation is being recording, a copy of the

presentation and slides will be distributed within 48 hours -  Please chat questions using the chat window, we will get to as

many as we can during Q & A -  Join the conversation on Twitter by using @Qualtrics #surveys

AGENDA -  Introductions -  The basics of online sample -  How can online sample help me -  How Credit Karma uses online sample to make

meaningful decisions.

INTRODUCTIONS

CHRIS CORRENTE Credit Karma Market Researcher

HARRISON TAYLOR Qualtrics Online Sample Pro

MATT SMITH Qualtrics Online Sample Pro

Introduction to Online Sample -  What is online sample?

- Large database of individuals who have either opted-in or been invited to participate in market research and be compensated for their time.

- Great resource to supplement internal customer lists or syndicated research.

- Find those who you don’t currently have access to

How can Online Sample Help Me? -  How can I use online sample?

-  Ad/Concept Testing -  IHUT studies (In-Home Usage Testing) -  Tracking Studies -  Competitive Benchmarking -  Pre/Post market campaign studies

Topics -  Accurate Aggregates

-  Marketing and product insights -  Demographic Drill Downs

-  Using drill downs to create dynamic, targeted experiences -  Product, marketing, and content decisions

-  Q&A

Credit Karma What do we do?  •  Most  well-­‐known  for  offering  free  credit  scores  and  reports  from  two  of  the  three  major  bureaus.    What are our goals? •  to  re-­‐engineer  consumer  finance  by  bringing  more  transparency  and  certainty  to  personal  finance  •  simplify  decision-­‐making  and  the  management  of  personal  credit  and  finances  

Who’s our target audience •  Every American citizen 18 or older

That all seems a bit ambitious, doesn’t it? •  Yes, it does. Which is why, from a research standpoint, we need…

Accurate Aggregates

Accurate Aggregates -  Qualtrics can match survey respondents to US census data

-  CK’s first touch point with sampling -  Prior to using Qualtrics panels, we were polling members through email.

-  We were able to receive feedback about the consumers we already engage -  Problems:

-  Demographic biases -  Behavioral and attitudinal biases

-  EX. “How much do you trust Credit karma?” -  Using sampling, can match research to US census data, get an accurate overview of brand perception

-  Minimizes bias -  Allows tracking of awareness and other non-member metrics -  Allows for competitive analyses

Gender:  50/50  split  Ages:    

11.3%  ages  18-­‐24    17.8%  ages  25-­‐34    17.3%  ages  35-­‐44    19.1%  ages  45-­‐54    16.5%  ages  55-­‐64  18%  ages  65+    

HHI:  ($0k-­‐$24k  25%    $25k-­‐$49k  25%  $50k-­‐$99k  30%  

                                   $100k+  20%)    

Accurate Aggregates

0  

500  

1000  

Male   Female  

Gender  

0  

100  

200  

300  

18-­‐24   25-­‐34   35-­‐44   45-­‐54   55-­‐64   65+  

Age  

0  200  400  

Income  

Quota    Respondents  

Accurate Aggregates -  Accurate measurement of public perception & preference

-  CK Marketing uses: -  Aided and Unaided Awareness metrics -  Competitive analysis -  Commercial spot testing -  Define and understand TAM -  Track goals of building trust and perception as a ‘helpful’ brand.

-  Product: -  Drive decisions around static content (define ‘defaults’)

-  New product initiatives based on data, not presuppositions

 

Accurate Aggregates •  New  Product  IniUaUves  – Hypothesis  – Research  –  ImplementaUon  – Research  – Adjustment  

 

–  Logic  +  Knowledge  

–  Panel  +  member  surveys,  focus  groups  

–  Engineering  

–  AB  tesUng,  consumer  surveys,  aggregate  behavioral  data,  find  pa]erns,  measure  success  

 –  Combine  all  research,  opUmize  

Accurate Aggregates

0%  

20%  

40%  

60%  

80%  

100%  

Very  important  to  criUcal   Important   Not  at  all  to  slightly  important  

“When evaluating a credit card, how important do you consider the following factors?”

Sometimes we are ‘too close’ to our industries, and overlook the obvious. •  What can seem like common knowledge in your industry can be largely unknown to consumers.

Credit Karma How many members do you currently have? •  Over 35mm How do you make money giving away free credit scores and reports? •  We try to match users with offers that can save them money or help them improve their credit. We try to

personalize the Credit Karma experience as much as possible. Does your member base, demographically speaking, currently index accurately with US census data? •  Definitely not So doesn’t using aggregate data mean that your product won’t be “right” for anyone in particular? •  It provides high level success metrics, good information about what a majority of people might find

useful or appealing, and lets us choose ‘default’ content. •  But yes, relying strictly on aggregate research data could mean our product isn’t optimized/

personalized for any individual, which is why we need…

Demographic Drilldowns

Demographic Drilldowns -  Sampling and quotas can guarantee statistical significance when drilling down

-  Calculate needed samples and build into quotas -  Ex. 1,500 person survey matched to gender and HHI can provide statistically significant insight into all income

brackets broken down by gender -  Risk tolerance, preferred confidence level and margin of error should be taken into account. -  Credit Karma aims for a 95% confidence level and ~3% confidence interval at the aggregate level.

-  This is strict by industry standards -  On drill downs, we look for results that truly stand out, due to the increased Margin of Error. -  NOTE: The variables in this presentation will not be variables Credit Karma actually uses, but examples of

what could be used. We are a bit protective of our secret sauces.

Demographic Drilldowns AT  a  95%  confidence  level:  

Completes:  1,500    Confidence  Interval:  >=~3%  Male:  750  ,  Female:  750  Confidence  Interval  :  >=~3.5%  

Male/Females  in  each  income  bracket:  Confidence  Interval  >=~8%    0-­‐24k:  187,  25-­‐49k:  187  50-­‐99k:  225  100k+  150  

Product: “When evaluating a credit card, how important do you consider the following factors? (By gender and income)

Demographic Drilldowns -  Marketing:

-  We (marketers) have entered the age of tailored (targeted) messaging -  How do you know who to target with any particular message? -  Classic question, changing environment: How to maximize the efficiency of

marketing spend? -  Consumer expectations have changed -  Available tools have changed

Demographic Drilldowns •  Demographic specific awareness metrics

are key to filling the ‘top of the funnel’ •  Aided and unaided awareness research

by demographic

+

•  Properly tailored ads are key to moving consumers through the funnel

•  Priorities –  Do we have a product that meets

your need? •  Preferences

–  Does our product meet that need in a way you find delightful?

•  Locations –  Where can we find you?

•  Ex. Aided Q: Below are a list of companies and website names

that…Please select the degree of familiarity that best fits your knowledge of that company or website.

•  I've been a customer of this company or website •  I've heard of this company or website •  I may have heard of this company or website, but I'm not sure •  I have never heard of this company or website before

Demographic Drilldowns For classification purposes, which of the following categories best describes your household total income before taxes last year? Select one response. (income brackets)

+

How do you watch television? Please read the list below and select the options that apply to you. (day parts) (cable/broadcast/online) +

(Embedded commercial) Please indicate how much you agree or disagree with the following statements about the commercial you just viewed. (CRITERIA) = Better marketing spend!

Demographic Drilldowns

Beyond  the  basics:  •  For  this  presentaUon,  focused  on  high  level,  universal  quota  targeUng  criteria  •  If  you  can  think  of  it,  you  can  sample  by  it  

•  EX.  “Have  you  ever  used  Credit  Karma?”  •  Screen  out  if  yes  

•  EX.  “When  was  the  last  Ume  you  checked  your  credit  report?”  •  Screen  out  if  <12  months  

•  CombinaUons  thereof  •  Looking  for  300  18-­‐24  year  olds  who  have  never  used  credit  karma  but  have  

seen  their  credit  report  in  the  last  12  months.  •  Looking  for  600  people  in  Scranton,  PA  with  children  named  Robert,  David,  Or  

Julia,  who  have  pets.  •  We  would  never  ask  this,  and  I’m  not  sure  how  Qualtrics  would  respond  if  you  did,  but  if  anyone  tries,  

I’d  love  to  know  the  cost  for  complete  and  how  long  it  took  to  fill  your  quota.  

Q & A