Everyone’s Customers Are Wrong andTheir Data Is Lying

Evan HamiltonHead of Community, UserVoice

DATA!Big Data! Social Data! Mobile Data!

It’s the future!

Actually...Data doesn’t tell the whole story.

Data doesn’t tell the whole story.

A/B test: will this carousel increase clicks?^ incredibly ugly

Data doesn’t tell the whole story.

“Hey, look! The version of the page with the carousel is making more people click through!”

Data doesn’t tell the whole story.

“Oh wait...it’s making more people click, not actually convert.”

Tricky data at work!

Data doesn’t tell the whole story.

We created confusion, not conversions.

Data doesn’t tell the whole story.

Data doesn’t tell the whole story.

Hey, that looks good!

Data doesn’t tell the whole story.

Oops. They “forgot” to report refunds (amongst other things.)

PEOPLE!Feelings! Empathy! Gut!

It’s the only way!

Actually...People don’t tell the whole story.

People don’t tell the whole story.

“Wow, maybe we should build this. Let’s ask for more feedback first...”

People don’t tell the whole story.

Oh. So they want this feature because they’re not responding to customers. Next.

There’s always more context.

• What is the pain point?• Are these paying users? Freeloaders?• Are these people using your product in the intended way?

• (We had someone using our forum to name their baby. Cute, but not our target market.)

• Are they using your main features?• Early “adopters”?

• (Robert Scoble is not your target market. He is fickle.)

People don’t tell the whole story.

“Makes sense, let’s build it!”

People don’t tell the whole story.

But customers found this confusing. (And it made us throw up in our mouths a little bit.)

People don’t tell the whole story.

“Yet another online identity” is what we should have focused on.

People don’t tell the whole story.

v2No password. No signup. No registration.

Solved the problem without a complicated sign-in process.

Customers often give you bad solutions to real problems.

(Don’t misinterpret this as good solutions or not real problems.)

Everything is wrong.Now what?

Data still has to be interpreted.

Data still has to be interpreted.

Requests can lead you astray.

Data still has to be interpreted.

Requests can lead you astray.

Combine data + customer stories.

Combine data + customer stories.

Back when we just had UserVoice Feedback, we’d lose lots of customers after 1 year. So we asked them why

they were leaving.

Turns out they were growing up. New staff, new department heads...& feedback forums didn’t have a

definitive place in the organization.

So we built a customer service tool into our product. Not what we necessarily planned to build...but what customers

wanted.

It increased retention immensely. We now have a place in the organization.

We got here because data alerted us to something, & user stories filled out the details.

Don’t get lost in the data rat-hole.

Don’t get lost in the data rat-hole.

Don’t chase 1% when you can get 15%.

Don’t get lost in the data rat-hole.

Don’t chase 1% when you can get 15%.

Don’t get lost in minor homepage optimization.

You’ll never know what you could get unless you try something crazy.

Tried this yet? If not, go do that before trying another shade of blue.

Tips, tricks, and tools

• (At least) Google Analytics• Hell, even Bit.ly links give you a basis to work from

• Mixpanel

• The ol’ fake button• How many people click a (fake) “calendar” button in your app? If 40% of

users do, you should build it!

• Test with segments• Try tests on nonpaying users if you don’t want to scare off paying users

• Talk to customers (constantly)

• UserVoice

Thank you!

Evan HamiltonHead of Community, UserVoice

@evanhamilton