Experimenting Your

Way to MVP

Catherine ShyuPM @ FullContact

About

Product Manager at FullContact

Previously PM at SendGrid, Generalist at BandPage

Bay Area native

@cthrin

We all want to build this

Nobody wants to spend years building this

Our job is this

Address the right target market, with the right problem, with the right solution and marketing, at the right time… Easy right?

So how do we make sure we’re working on the right

things?

Shorten the Lean Startup cycle

Catch core problems before investing too much time/money

Validate your riskiest assumptions through experiments

Image from Google Ventures

What are experiments? We run experiments

to test hypotheses that...

1.Can be validated / invalidated

2.Address areas of risk in the product

No brainer, right? So why aren’t more companies doing it?

Top 5 Reasons for

not Experimenti

ng

1.I can’t justify taking time away from building the product.

2.I don’t have approval from management to do this.

3.We have a deeply complex product. There’s no way to simulate things on it.

4.Our product is too niche. I can’t find users to talk to.

5.I don’t want to damage our relationship with potential customers.

I can’t justify taking time away from building the product.

Excuse 1.

Time $ $$ $$$ $$$

$

Cost of Change Increases with Product Complexity

Example: Customer Development

Goal: Product/market fit

Assumption: [X] industry is in our target market

Methods: Customer development interviews with 5 people from each industry

Most powerful questions asked:

“What are the top three challenges you face in your industry?”

“How do you currently solve [x] problem?”

Results

● Findings helped me reorder our product roadmap● Saved us development time by cutting down v1

scope

Excuse 1.

I don’t have approval from management to do this.

Excuse 2.

Have a plan & explain it in cost/benefits

1.Make a list of big assumptions and costs associated with them if you get it wrong

a. Have a few experiments prepared to validate/invalidate

b. Lean towards experiments that are quick and easy to run but get you big insights

2.Are there easy tests you can run to mitigate big expensive risks?

3.Frame as “early risk mitigation”Results

● If you can validate/invalidate your most

important assumption, it will 1) inform your product direction and 2) pave the way for more experiments.

Excuse 2.

We have a deeply complex product. There’s no way to simulate things

on it.Niche industries, Hardware, AI, VR, etc.

Excuse 3.

Powerpoint is your friend

Take shortcuts however you canInvision: string together high-fidelity mockups to simulate interactivity

Powerpoint: when you need to mock something up very quickly

3D Printing: when you are testing the physical UX of a product

Results

● Ability to go through rounds of changes before engineering starts work

Excuse 3.

Our product is too niche. I can’t find users to talk to.

Excuse 4.

Use the power of LinkedIn stalking

LinkedIn has robust advanced search features (job title, company, industry)

Craigslist ads for local in-person interviews

Leverage existing customers

Results

● Once I got one person to talk to me, they would often introduce me to their peers in the industry. Always ask for intros to more people.

Excuse 4.

I don’t want to damage our relationship with potential

customers.

Excuse 5.

Frame the interviewsPick customers who will be sympathetic

Use framing techniques to prime the customer for giving feedback“I appreciate your time…”

“This is still early stages but we wanted to get your opinion at the start…”

“Since you are a long-time customer, I wanted to get your feedback on a change I’m considering…”

Make them feel specialResults

● Many of our private beta testers are still avid users of the app and respond with feedback when asked. They also have free access to the app.

Excuse 5.

Lean Experiments

: Basic Principles

Goal is to test your riskiest assumptions early in the game

Experiments run for 3-5 days

Multiple experiments can run at once

5 people = pattern

Don’t worry, it doesn’t have to be a huge ordeal

Team exercise: biggest risks in your productRisk Severity (1-5, 5

highest)Hypothesis 1 Experiment 1

People won’t like my Dilbert comics

5, obviously People hate the comics but are too nice to say so.

Distribute post-talk anonymous survey

What’d We Do at FullContact?Team of 5: two Sales, one PM, one PMM, one Customer Success, one

Engineer

Trello board for tracking experiments

Weekly check-ins to share learnings

Our prototype when we started.

Now: FullContact for Teams Beta - onwards to v1!

Results

De-scoped the feature set into two releases, by customer segment

Validated the need to invest in building a core set of features

Able to test out different pricing on potential customers

Discovered the problem we are solving is on a top 3 list of problems for multiple industries

Always Be Experimenting!

👋 Thanks for listening@cthrin