Market research without totally screwing it up

Colin Millerchip

18 April 2011

Agenda• Introduction• Before you start• Market segments• Looking for problems• General techniques• Discovery-validation cycle• Research tools (qualitative, quantitative)• Risk management• Financial modelling• Questions?

Before we start…• This presentation is about market research, primarily to

discover new product / service opportunities• Market research is not the only way to find such

opportunities– For eg, technology-driven innovation, customer funded

development• Apologies in advance if this is overly biased towards

“mass market B2B”• … and apologies that these are very verbose slides

– They’re intended to be a resource that can be used off-line

Introduction• Research isn’t a well-defined path you walk along

– Each piece of research will need its own path – If you follow a checklist, you still need to keep your brain

engaged– Discovery-validation cycle will help you find some of the

“unknown unknowns”• You’ll never get to 100% knowledge

– If you try to do this, you’re going to be way too late– There are no guarantees – deal with it!

Before you start

• What are you trying to find out?– What questions you want to answer– Who you want to ask (market segment)

• Make sure to think about the business justification– What will the people making the go/no-go decision need to

know?– For eg: What’s the channel to market? Do people tend to

spend money in this area? Can we capture the opportunity (core competence)?

• How long do you have?– Use common sense: plan the phases of your research,

track progress against plan, modify as you learn more

Market segments

(sorry, this is a bit “marketing-y”)

• What is a market segment? A group of people who…– can be reached via the same channel– respond to the same message– are in contact with each other (soft requirement?)

• What’s it for? Focus!

• How do you identify market segments?– Top-down: take the whole marketplace and slice– Bottom-up: pick a few representative customers and

build up a definition around them

Market segments (cont.)

• Good segmentation criteria– have clear boundaries– are easy to determine– are easy to estimate in the general population– don’t change much over time

• Example criteria– Geographic– Business demographic (industry, size, etc)– Adopter categories– Product usage– Purchasing approaches

Looking for problems

(sorry, this is a bit “product-y”)• Pragmatic Marketing call this “the 5th P”

– The traditional “four P’s of marketing”: product, price, place, promotion

• We’re trying to find a problem to solve for people…– … that’s sufficiently painful, pervasive, pressing (urgent), and

profitable (for us to solve)• Typically* people want to do one of 3 things: make more

money; reduce costs (time ~= money); or reduce risks. – Qu: what are you looking to do for people?

http://www.pragmaticmarketing.com/srpm

* … in B2B; this isn’t the case with B2C

General techniques

• Face-to-face meetings are invaluable– It’s costly, but you must do it – NIHITO

• Listening not talking– Don’t end up doing a pitch of your ideas

• You’ll learn more from -ve than +ve feedback– Thank them for the praise, but encourage criticism

• Don’t tell customers that they’re wrong– … even if they are– You’re researching, not selling– (if necessary, you can gentle correct their misunderstandings at the end)

General techniques (cont.)

• Contextualise– ie, understand the context of the comments– eg, their job title, level of experience, job title, size of

team, size of company, etc

• Will help you understand results– eg, testers are like X, developers are like Y

• Will help you find target market segments– eg, small companies don’t have this problem

General techniques (cont.)

• Open-ended questions• Leading questions

– Unprompted vs. prompted feedback (both are useful)

• How painful is the problem?– Frequency of use case– How ‘workaroundable’– Need vs. want

• Repeatability– Try to ask the same things, in the same way

• Document it (if you need to)– Write up customer interviews straight away– Try to record interview (dictaphone)

General techniques (cont.)

• Selection Bias– 3 segments within any market segment: customers; evaluators

(know of you, haven’t bought); prospects (don’t know about you).

• Make sure you’re not missing a major segment in your research• Much easier said than done for ‘prospects’…

– For eg, your poll shows that 100% of database users back up databases… but could it be that only database administrators answered the poll?

• People lie!– People tend to be nice to you– They love your product ideas… but won’t part with cash when

you ship the product– Qu: how can you test their propensity to spend money to solve

their problem?

Discovery-validation cycle

• Start with qualitative research– Customer interviews, open-ended, 2-way discussions– Learn more about your research topic

• Move to quantitative research– Polls/surveys, specific question / answers– Gather statistically-valid data– This phase isn’t pure validation, there’s still some discovery going on

“If you want to know what people think and do, choose quant;

If you want to know why they think and behave that way, choose qual”

Qualitative Quantitative

Root Quality (properties / characteristics) Quantity (number / magnitude)

Nature Learning Confirming (some learning)

Style Open questions Closed / specific questions

Interaction Two-way exchange One-way collection

Number of contacts Few (10’s) Many (100’s)

Effort per contact Large Small

Qualitative research tools

• Customer interviews• Usability tests• Forum research• Meet people at events (user groups, conferences, etc)• Feedback via sales, support

– careful with this…

• … etc etc…

Quantitative research tools: polls

• Normally (but not always), you want the answers to be mutually exclusive, and complete (ie, cover the total range of responses)

• Make then engaging – for eg, sacrifice perfect response data for amusement

• Yes and No isn’t enough: what about “Don’t know”, or “What’s this?” or “Other”

• Incentivise people: less critical than for surveys (minimal time commitment), sharing current results is normally enough

I’ve used:

http://www.polldaddy.com/

Quantitative research tools: surveys

• Keep the survey fairly short– Resist temptation to ask about secondary issues

• Find out about the respondent – Demographics, company, experience

• Get their permission to follow up (“if you want to enter the prize draw, or are happy to be contacted about your responses, please leave your email address here”)

• Interesting questions– “Spend 100 points”– Forced choices (“pick one”; ranking (beware this can annoy people))

• Incentivise people: freebees, prize draw, share results

I’ve used:

http://www.surveymonkey.com/

http://www.questionpro.com/

Quantitative research tools: gotcha’s

• Careful with your wording!• Test it out before going live

– Monitor initial early responses, be prepared to change survey if required (eg, loads of people select “Other” option to a question)

– Use a web re-director can be helpful here, since not all changes can be made to an active/open survey

• Think about how you’ll react to various findings– If survey response data to a question suggests X, will you believe it? Or

will you doubt that people understood the question / answered it properly?

Other research tools

• Google adwords• Alexa / compete.com traffic ranking?

– Question over whether these are biased• Forum traffic stats (ours, competitors)• Social networking sites• Customer feedback mechanisms• Usage data from your service / product (eg, download

stats; quote / invoice stats; “dial home” pings; etc)• … etc etc…

Analysts data

• Can cut down leg work• Beware of…

– “Hockey-stick” curves (especially ones that are continually adjusted)

– Correlation between vendor spend with analysts, and their ranking in analyst’s “magic quadrant”

Risk management

• You can’t wait until you know for sure• Identify the risks – technical, commercial,

personnel, etc• Try to quantify these

– Likelihood / Frequency– Consequences– Cost to mitigate

• Put in place a mitigation plan 

Financial modelling• FITA* may not be enough• Financial modelling, & related jargon

– Monte Carlo simulation / PDFs (eg, RiskAMP, @RISK)– Fully loaded costs, NPV, IRR, interest rate, cash flow forecast,

revenue model, Tornado chart, spider chartAre you scared yet? :o)

• Can drive the model backwards– “What revenue do I need to achieve to get my money back

in n years?”

* Finger in the air

Questions

?colin@millerchip.org

@millerchip