how to lie with design research

51
Osaka, Japan Several years ago, I did a research project in Japan for a major retailer who wanted to expand overseas to Japan. We went to Osaka in order to understand shopping.

Upload: dan-saffer

Post on 27-Jan-2015

114 views

Category:

Business


3 download

DESCRIPTION

Presented at the Design Research 2007 conference.

TRANSCRIPT

Page 1: How to Lie with Design Research

Osaka, Japan

Several years ago, I did a research project in Japan for a major retailer who wanted to expand overseas to Japan. We went to Osaka in order to understand shopping.

Page 2: How to Lie with Design Research

Umeda Marketplace

Page 3: How to Lie with Design Research

Denden Town, Nipponbashi

Page 4: How to Lie with Design Research

Gunkan Apartments

Page 5: How to Lie with Design Research

On-The-Street Interviews

Page 6: How to Lie with Design Research

Signage and Iconography

Page 7: How to Lie with Design Research

Interesting Observations

Page 8: How to Lie with Design Research

Cultural Norms

Page 9: How to Lie with Design Research

Design PrinciplesUse colorful iconography to attract and comfort customers.

The street and store interiors are not as heavily separated in Japan as they are in the US. Bring the street into the store.

Display space should reflect those of a street market, with items tightly pressed together to create impressions of bounty and variety.

Here are several design principles we came up with.

Page 10: How to Lie with Design Research

1. Don’t do any design research. Make it all up.

You just got your first lesson and rule #1. All the images you just saw, I collected off Flickr in one afternoon. Voila, saved myself a trip to Osaka. I’ve never even been to Japan.

Page 11: How to Lie with Design Research

How to Liewith Design ResearchDan Saffer

If you are reading this note and you aren’t me, please understand that this presentation is meant to be delivered Stephen-Colbert style, as deadpan comedy. Please treat the following as entirely tongue in cheek.

Page 12: How to Lie with Design Research

TIP: Don’t go into the field unless you have to.

Why do research when you don't really need to? Most of the time companies are just looking to have their ideas validated. Why not give them what they want using carefully chosen photos and "stories" from the internet.

Page 13: How to Lie with Design Research

TIP: Wacky cultural practices always impress.

For "international" research, be sure to throw in a couple of unexpected cultural practices to make people feel that they've really taken the time to consider diverse perspectives.

Page 14: How to Lie with Design Research

The Jayson Blair Method

Named after the New York Times reporter who did his on-site reporting from his apartment in Brooklyn. I don’t recommend this method. Why?

Page 15: How to Lie with Design Research

TIP: Don’t lie about the easily (dis)provable.

Why? Because like Mr. Blair, you are eventually going to get caught! There are better--although not EASIER ways of lying with research. A really good lie is one that is untraceable and cannot be pinned on you or easily disproved.

But let’s back up a minute.

Page 16: How to Lie with Design Research

Why you’re really here

I know why you all came to this fine conference with such fantastic speakers like myself. It’s really a privilege for all of you, isn’t it? But even though I’m sure you sold this to your boss as “helping users” or “making better products” or even “user experience” (whatever that is), it’s really all about money isn’t it? It’s ok, you can admit it. You came because you wanted to jump on the design research bandwagon before we’re found out and it rolls away. And I’m here to help you do that. Lying makes it easier to cash in! We got paid 500 grand for that Osaka study!

Page 17: How to Lie with Design Research

Why would you ever lie about research?

Now I know some of you are saying, Why would I ever lie about design research? People lie for the same reason they design--to turn an undesirable situation into a better one. I have an idea I want you to act on. I don’t want to be uncomfortable or embarrassed or feel stupid. I want you to think that I really like that shirt. I want you to think that I’m really smart. I want you to do this project.

Page 18: How to Lie with Design Research

Research gives your ideas more credibility.

Suppose I have an idea for this cool hypnotic lamp. Wouldn’t it be great if I had some research that said, “Users want to be hypnotized at their desks?” That’s not really lying, is it? Distorting perhaps. But I bet right now, you are thinking, Dan, how (theoretically) might one go about doing that?

Page 19: How to Lie with Design Research

Let’s assume you have a good reason for lying.

But I’m not here to judge, but to teach and heal. I’m a uniter.

Page 20: How to Lie with Design Research

2. Skew the research subjects in your favor.

Rule Number 2. In design research, your data is only as good as your participants. Choose subjects that are going to bolster your case. The smaller and more alike the sample size, the better!

Page 21: How to Lie with Design Research

TIP: Diversity is overrated.

Research people who aren’t diverse. Diversity is overrated when stacking the deck in your favor. If you are making a website for an automobile manufacturer, why not interview all auto enthusiasts? Then you can say, without lying, “Everyone we talked to said they have spent hours online looking at car sites!”

Get bonus points for calling this not diverse group an “Unfocus Group.”

Page 22: How to Lie with Design Research

TIP: Make unconscious bias, conscious.

We do this anyway, right? If we have two people to choose from, we often unconsciously choose the person that we think is going to be the most cooperative, and astonishingly, that person often looks just like us! If we’re talking to people, why in the world would we even want to talk to people who are different from us?

Page 23: How to Lie with Design Research

What Would Johnny Do?

Pick your research subjects like the late defense attorney Johnny Cochrane picked juries. You want the right mix of people who are going to give you the outcome you want. No wild cards. Subjects who are going to give you the results that you want when you ask them for their stories and ideas.

Page 24: How to Lie with Design Research

3. Don’t be objective.

Rule number 3. Leave the objectivity to the scientists and reporters. We’re designers. Why is there this veneer that we need to be objective in our research? We’re not out there in our lab coats like some pinhead trying to discover a cure for cancer. Who is lying to who now?

Page 25: How to Lie with Design Research

Ask leading questions.My concept: really great or just great?

If we’re not objective, we can ask leading questions. [CLICK] Then you can put in your research findings that “Subject X thought this was a great concept.” It’s not lying if they said it, right?

Page 26: How to Lie with Design Research

TIP: Use carefully considered wording to lead.

“Wasn’t that a confusing experience?”

“Isn’t this a great conference?”

Remember that your research subjects are people in the unfamiliar situation of being interviewed and observed. They want to give you what you want and are looking to you for cues. Thus, you can easily influence their responses through the phrasing of your questions.

Page 27: How to Lie with Design Research

4. Toss out data you don’t like.

Don’t let a research subject ruin your design idea. When it comes time to analyze the data, it is essentially about culling your data, separating the wheat from the chaff. You have to use some criteria to make those calls. Some people use things like prevalence or, on the other hand, uniqueness. Why not use your own preferences? Don't make the mistake of letting rigor or a quest for objectivity lead you to keep data with implications you don't want to deal with.

Page 28: How to Lie with Design Research

TIP: Having data doesn’t mean using it.

Why do you have to use all the data? I mean, if you didn’t ask that question or if you weren’t sitting there observing, you might not know that piece of data at all, right? Just because something is known, doesn’t mean we have to use it!

Page 29: How to Lie with Design Research

"[A]s we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know we don't know." —Donald Rumsfeld

All you need to do is turn a known known in a known unknown. Simple enough.

Page 30: How to Lie with Design Research

TIP: See only what you want to see.

One way to avoid having to toss out data is to never observe it at all. Or at least not write it down. Only look in the places you think you are going to get the data you want.

Page 31: How to Lie with Design Research

5. Deliberately misrepresent the data.

Ok, once you have data, from one place or another, it’s time to present it. And it’s here that the best lies can be told all in the name of “storytelling.” Your charts, graphs, and other visualizations are just ways of telling a story about some data that was collected. Make sure it is YOUR story, not the story of your subjects or the story of the data!

Page 32: How to Lie with Design Research

0

10

20

30

40

50

60

70

80

90

100

2006 2007

How about this graph? Interest in adding a new feature that you want to add only went up 5 percent over a year. It’s informative, yes. Perhaps too informative. Different approaches allow you to highlight different aspects of the data. Don't miss this opportunity to emphasize how you were right and suppress evidence that you weren't.

Page 33: How to Lie with Design Research

0

10

20

30

40

50

60

70

80

90

100

2006 2007

0

55

Once you chop the top and bottom and the pesky numbers off of it, now we’re talking. That looks much better. The old saying, “A difference is only a difference if it makes a difference” doesn’t mean anything to me.

Page 34: How to Lie with Design Research

TIP: Bring the zazz!

Now this graph showing why people join record clubs has a little more zazz to it. If you make your charts colorful and fun, you pull attention away from the actual data. That way, you can spin it however you want!

[Graph courtesy of The Onion.]

Page 35: How to Lie with Design Research

TIP: Make the data appear meaningful even if it isn’t.

Prove your points through nifty diagrams that mean very little but sure look awesome. We’re designers! We’re good at making something pretty.

Page 36: How to Lie with Design Research

TIP: More specific = More credible

$311,294,070,513

The more specific the number you quote is, the more people think it must be correct. Anyone know what this number is? [Click] It’s the exact amount down to the dollar Jakob Nielsen said the whole world would save by implementing his usability recommendations. I’m surprised he didn’t put cents in there as well. Most businesses can’t get their budgets down to dollars, but here he does it for the entire world! Just masterful. My hat is off to you, sir.

Page 37: How to Lie with Design Research

TIP: Make the viewer feel good.☛ http://www.odannyboy.com ☚

This is from Jakob Nielsen again, from his article about how everyone should write essays, not blog posts. Very insightful, a must-read. I found it on a blog. But he says the bloggers on the left are bozos, and those on the right are clever. Wow, he must be reading MY blog. [CLICK]

Page 38: How to Lie with Design Research

TIP: Trend lines going up make for good drama.

If you can show everything is moving up, up, up you’ll make your clients feel better and you’ll probably get more work out of it! It doesn’t matter if you can’t really prove it. It’s the future, so no one knows what is going to happen!

Page 39: How to Lie with Design Research

6. Willfully confuse correlation and cause.

Rule number 6. Just because something happens in a sequence or is connected doesn’t mean the one thing causes the other. It might be a coincidence. But coincidences don’t make for good research presentations, do they?

Page 40: How to Lie with Design Research

Pick the reason you like the most.

If there are several reasonable explanations for a phenomenon, pick the one that suits you the most.

Page 41: How to Lie with Design Research

TIP: With a small sample, correlations are easy!

If there are several reasonable explanations for a phenomenon, pick the one that suits you the most. With a small enough sample, you can make a correlation between just about anything. In this case, more cars that are blue are hit by rocks every year.

Page 42: How to Lie with Design Research

7. Deliberately misinterpret data.

As anyone who has analyzed research data knows, you can take the exact same data and draw from it multiple conclusions. This is even more true with the qualitative research that designers most often do. Look at this image, taken on a research trip. You could look at this sign and say, Wow, the subway really cares about its passengers, giving them this helpful sign. You could also look at this and say, Wow, only two trips from this platform. Why is the subway so poorly marked it needs this sign to let us know this information??? It all depends on how you want to spin the data.

Page 43: How to Lie with Design Research

TIP: Make unlike things seem alike in comparisons.

Death rate in Chicago, 2006: 20 per 1000

Death rate in Iraq, 2006: 10 per 1000

Thus, Chicago is more dangerous than Iraq.

Now clearly, Chicago and Iraq aren’t quite the same, but by comparing apples to oranges, I can get you to believe that they are AND skew the data.

Oh, And I made those figures up, so don’t bother checking them.

Page 44: How to Lie with Design Research

“I really love using this product.”

If you divorce words from their context, they are ripe for misinterpretation. There is a big difference between this [CLICK] and this. But you can say, “Customers said they love using this product.”

Page 45: How to Lie with Design Research

“The subjects said this task was difficult.”“The subjects said this task was nearly impossible.”

Language is great for distorting data. Note the subtle difference. Makes it much more impactful, doesn’t it?

Page 46: How to Lie with Design Research

8. Answer questions you weren’t asked.

When you go out into the field (if you bother to go) you are likely to bump into something interesting but unrelated to the business problem. When that happens, simply change the scope of the research. If you can’t find what you are looking for, change the scope and then find what you need.

Page 47: How to Lie with Design Research

TIP: Call them“Emergent Patterns.”

If you can, call it an "emergent pattern."

Page 48: How to Lie with Design Research

TIP: Don’t get caught.

Amateurs like James Frey get caught because their lies are too obvious and too easily disproved. Be sure you follow my rules for distorting your research in your favor.

Page 49: How to Lie with Design Research

“It’s a subjective art!”

And if you do get caught, simply explain, Hey, it’s a subjective art! It’s a magic phrase that will get out out of all kinds of trouble and it is impossible to argue about.

Page 50: How to Lie with Design Research

It’s better to be a winner than to tell the truth.

And that, ladies and gentlemen, is...

Page 51: How to Lie with Design Research

How to Liewith Design ResearchThanks.Dan Safferhttp://[email protected]

Thanks to Flickr photographers for the images and to Darrell Huff’s classic book How to Lie with Statistics for inspiration.

Special thanks to the staff of Adaptive Path and especially Todd Wilkens and Brandon Schauer for their insights and contributions to this talk.