how to lie with design research

Osaka, Japan

Several years ago, I did a research project in Japan for a major retailer who wanted to expand overseas to Japan. We went to Osaka in order to understand shopping.

Umeda Marketplace

Denden Town, Nipponbashi

Gunkan Apartments

On-The-Street Interviews

Signage and Iconography

Interesting Observations

Cultural Norms

Design PrinciplesUse colorful iconography to attract and comfort customers.

The street and store interiors are not as heavily separated in Japan as they are in the US. Bring the street into the store.

Display space should reflect those of a street market, with items tightly pressed together to create impressions of bounty and variety.

Here are several design principles we came up with.

1. Don’t do any design research. Make it all up.

You just got your first lesson and rule #1. All the images you just saw, I collected off Flickr in one afternoon. Voila, saved myself a trip to Osaka. I’ve never even been to Japan.

How to Liewith Design ResearchDan Saffer

If you are reading this note and you aren’t me, please understand that this presentation is meant to be delivered Stephen-Colbert style, as deadpan comedy. Please treat the following as entirely tongue in cheek.

TIP: Don’t go into the field unless you have to.

Why do research when you don't really need to? Most of the time companies are just looking to have their ideas validated. Why not give them what they want using carefully chosen photos and "stories" from the internet.

TIP: Wacky cultural practices always impress.

For "international" research, be sure to throw in a couple of unexpected cultural practices to make people feel that they've really taken the time to consider diverse perspectives.

The Jayson Blair Method

Named after the New York Times reporter who did his on-site reporting from his apartment in Brooklyn. I don’t recommend this method. Why?

TIP: Don’t lie about the easily (dis)provable.

Why? Because like Mr. Blair, you are eventually going to get caught! There are better--although not EASIER ways of lying with research. A really good lie is one that is untraceable and cannot be pinned on you or easily disproved.

But let’s back up a minute.

Why you’re really here

I know why you all came to this fine conference with such fantastic speakers like myself. It’s really a privilege for all of you, isn’t it? But even though I’m sure you sold this to your boss as “helping users” or “making better products” or even “user experience” (whatever that is), it’s really all about money isn’t it? It’s ok, you can admit it. You came because you wanted to jump on the design research bandwagon before we’re found out and it rolls away. And I’m here to help you do that. Lying makes it easier to cash in! We got paid 500 grand for that Osaka study!

Why would you ever lie about research?

Now I know some of you are saying, Why would I ever lie about design research? People lie for the same reason they design--to turn an undesirable situation into a better one. I have an idea I want you to act on. I don’t want to be uncomfortable or embarrassed or feel stupid. I want you to think that I really like that shirt. I want you to think that I’m really smart. I want you to do this project.

Research gives your ideas more credibility.

Suppose I have an idea for this cool hypnotic lamp. Wouldn’t it be great if I had some research that said, “Users want to be hypnotized at their desks?” That’s not really lying, is it? Distorting perhaps. But I bet right now, you are thinking, Dan, how (theoretically) might one go about doing that?

Let’s assume you have a good reason for lying.

But I’m not here to judge, but to teach and heal. I’m a uniter.

2. Skew the research subjects in your favor.

Rule Number 2. In design research, your data is only as good as your participants. Choose subjects that are going to bolster your case. The smaller and more alike the sample size, the better!

TIP: Diversity is overrated.

Research people who aren’t diverse. Diversity is overrated when stacking the deck in your favor. If you are making a website for an automobile manufacturer, why not interview all auto enthusiasts? Then you can say, without lying, “Everyone we talked to said they have spent hours online looking at car sites!”

Get bonus points for calling this not diverse group an “Unfocus Group.”

TIP: Make unconscious bias, conscious.

We do this anyway, right? If we have two people to choose from, we often unconsciously choose the person that we think is going to be the most cooperative, and astonishingly, that person often looks just like us! If we’re talking to people, why in the world would we even want to talk to people who are different from us?

What Would Johnny Do?

Pick your research subjects like the late defense attorney Johnny Cochrane picked juries. You want the right mix of people who are going to give you the outcome you want. No wild cards. Subjects who are going to give you the results that you want when you ask them for their stories and ideas.

3. Don’t be objective.

Rule number 3. Leave the objectivity to the scientists and reporters. We’re designers. Why is there this veneer that we need to be objective in our research? We’re not out there in our lab coats like some pinhead trying to discover a cure for cancer. Who is lying to who now?

Ask leading questions.My concept: really great or just great?

If we’re not objective, we can ask leading questions. [CLICK] Then you can put in your research findings that “Subject X thought this was a great concept.” It’s not lying if they said it, right?

TIP: Use carefully considered wording to lead.

“Wasn’t that a confusing experience?”

“Isn’t this a great conference?”

Remember that your research subjects are people in the unfamiliar situation of being interviewed and observed. They want to give you what you want and are looking to you for cues. Thus, you can easily influence their responses through the phrasing of your questions.

4. Toss out data you don’t like.

Don’t let a research subject ruin your design idea. When it comes time to analyze the data, it is essentially about culling your data, separating the wheat from the chaff. You have to use some criteria to make those calls. Some people use things like prevalence or, on the other hand, uniqueness. Why not use your own preferences? Don't make the mistake of letting rigor or a quest for objectivity lead you to keep data with implications you don't want to deal with.

TIP: Having data doesn’t mean using it.

Why do you have to use all the data? I mean, if you didn’t ask that question or if you weren’t sitting there observing, you might not know that piece of data at all, right? Just because something is known, doesn’t mean we have to use it!

"[A]s we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know we don't know." —Donald Rumsfeld

All you need to do is turn a known known in a known unknown. Simple enough.

TIP: See only what you want to see.

One way to avoid having to toss out data is to never observe it at all. Or at least not write it down. Only look in the places you think you are going to get the data you want.

5. Deliberately misrepresent the data.

Ok, once you have data, from one place or another, it’s time to present it. And it’s here that the best lies can be told all in the name of “storytelling.” Your charts, graphs, and other visualizations are just ways of telling a story about some data that was collected. Make sure it is YOUR story, not the story of your subjects or the story of the data!

0

10

20

30

40

50

60

70

80

90

100

2006 2007

How about this graph? Interest in adding a new feature that you want to add only went up 5 percent over a year. It’s informative, yes. Perhaps too informative. Different approaches allow you to highlight different aspects of the data. Don't miss this opportunity to emphasize how you were right and suppress evidence that you weren't.

0

10

20

30

40

50

60

70

80

90

100

2006 2007

0

55

Once you chop the top and bottom and the pesky numbers off of it, now we’re talking. That looks much better. The old saying, “A difference is only a difference if it makes a difference” doesn’t mean anything to me.

TIP: Bring the zazz!

Now this graph showing why people join record clubs has a little more zazz to it. If you make your charts colorful and fun, you pull attention away from the actual data. That way, you can spin it however you want!

[Graph courtesy of The Onion.]

TIP: Make the data appear meaningful even if it isn’t.

Prove your points through nifty diagrams that mean very little but sure look awesome. We’re designers! We’re good at making something pretty.

TIP: More specific = More credible

$311,294,070,513

The more specific the number you quote is, the more people think it must be correct. Anyone know what this number is? [Click] It’s the exact amount down to the dollar Jakob Nielsen said the whole world would save by implementing his usability recommendations. I’m surprised he didn’t put cents in there as well. Most businesses can’t get their budgets down to dollars, but here he does it for the entire world! Just masterful. My hat is off to you, sir.

TIP: Make the viewer feel good.☛ http://www.odannyboy.com ☚

This is from Jakob Nielsen again, from his article about how everyone should write essays, not blog posts. Very insightful, a must-read. I found it on a blog. But he says the bloggers on the left are bozos, and those on the right are clever. Wow, he must be reading MY blog. [CLICK]

TIP: Trend lines going up make for good drama.

If you can show everything is moving up, up, up you’ll make your clients feel better and you’ll probably get more work out of it! It doesn’t matter if you can’t really prove it. It’s the future, so no one knows what is going to happen!

6. Willfully confuse correlation and cause.

Rule number 6. Just because something happens in a sequence or is connected doesn’t mean the one thing causes the other. It might be a coincidence. But coincidences don’t make for good research presentations, do they?

Pick the reason you like the most.

If there are several reasonable explanations for a phenomenon, pick the one that suits you the most.

TIP: With a small sample, correlations are easy!

If there are several reasonable explanations for a phenomenon, pick the one that suits you the most. With a small enough sample, you can make a correlation between just about anything. In this case, more cars that are blue are hit by rocks every year.

7. Deliberately misinterpret data.

As anyone who has analyzed research data knows, you can take the exact same data and draw from it multiple conclusions. This is even more true with the qualitative research that designers most often do. Look at this image, taken on a research trip. You could look at this sign and say, Wow, the subway really cares about its passengers, giving them this helpful sign. You could also look at this and say, Wow, only two trips from this platform. Why is the subway so poorly marked it needs this sign to let us know this information??? It all depends on how you want to spin the data.

TIP: Make unlike things seem alike in comparisons.

Death rate in Chicago, 2006: 20 per 1000

Death rate in Iraq, 2006: 10 per 1000

Thus, Chicago is more dangerous than Iraq.

Now clearly, Chicago and Iraq aren’t quite the same, but by comparing apples to oranges, I can get you to believe that they are AND skew the data.

Oh, And I made those figures up, so don’t bother checking them.

“I really love using this product.”

If you divorce words from their context, they are ripe for misinterpretation. There is a big difference between this [CLICK] and this. But you can say, “Customers said they love using this product.”

“The subjects said this task was difficult.”“The subjects said this task was nearly impossible.”

Language is great for distorting data. Note the subtle difference. Makes it much more impactful, doesn’t it?

8. Answer questions you weren’t asked.

When you go out into the field (if you bother to go) you are likely to bump into something interesting but unrelated to the business problem. When that happens, simply change the scope of the research. If you can’t find what you are looking for, change the scope and then find what you need.

TIP: Call them“Emergent Patterns.”

If you can, call it an "emergent pattern."

TIP: Don’t get caught.

Amateurs like James Frey get caught because their lies are too obvious and too easily disproved. Be sure you follow my rules for distorting your research in your favor.

“It’s a subjective art!”

And if you do get caught, simply explain, Hey, it’s a subjective art! It’s a magic phrase that will get out out of all kinds of trouble and it is impossible to argue about.

It’s better to be a winner than to tell the truth.

And that, ladies and gentlemen, is...

How to Liewith Design ResearchThanks.Dan Safferhttp://www.odannyboy.comdan@odannyboy.com

Thanks to Flickr photographers for the images and to Darrell Huff’s classic book How to Lie with Statistics for inspiration.

Special thanks to the staff of Adaptive Path and especially Todd Wilkens and Brandon Schauer for their insights and contributions to this talk.