intro to rapid experimentation - intel july 2015
TRANSCRIPT
Handicapping & Dark Horses
What Design Thinking Can Learn from Horse Racing
2 Techniques
#1 Handicapping
handicapping:the art of evaluating a horse’s performance in order to determine the outcome of a race
handicap the ideas
running a rapid experiment
rapid experimentation:using prototypes to evoke behaviors that quickly tell us whether our ideas meet users’ needs
rapid
experiments
usability test
survey
feedback methods
what will someoneactually do
what they might do
rapid experiments separate what customers say from what they do in the real world
what is an experiment?• a scientific procedure—a controlled empirical test of a
hypothesis
• hypotheses include (never null!):-A causes B-A is better, bigger, faster than A’-A changes B more when we do/provide X
• requirements:– independent variable that can be manipulated– dependent variable that can be measured– random assignment to condition (conservatively)
+
What can design experiments test?• are people interested?
• will they do/use this given all other choices/demands available?
• does it meet the need you designed for?• does it have the desired (and not
undesired) effects?• can people figure out how to use/do it?• which design is better (at any of the
above)?
How do you do this at home?1. Pick an idea 2. Make a list of all the questions you have
about the efficacy of the idea3. Select the most critical question to
success4. Generate a hypothesis5. Design an experiment to test your
hypothesis6. Create the prototype to support the
experiment
Before you move into prototyping, make a plan:• What questions do you have about the
effectiveness of the ideas?• What is my hypothesis about what will happen?• What kinds of observations would validate my
hypothesis?• What prototype(s) do I need to create for these
observations?• How should I run this experiment?
Use the prototype to• Create an experience that evokes the desired
behaviors• Evaluate your working hypotheses
Don’t create the prototype before designing your experiment
Make a list of questions
Concepts we generate often have implicit assumptions about how people will respond and what people will doWhat are the crucial questions that could make or break the success of your idea?
Select the most critical question
Will people be more likely to select a rental if it has a nice photo?Will homeowners be willing to let photographers photograph their space?How expensive is it create this service?Will the photos really look better?
Generate a hypothesisA hypothesis is your prediction about how people will behave.
A lot more people will rent an apartment that has been professionally photographed than one that has not.
Design an experiment• the goal is to evoke “real”
behaviors in “real” situations • the scenario must be
– believable– immersive– natural
• and, allow you to test/measure what you need to
• a word on “confederates”
Plan for measurement
people are notoriously bad at predicting (and also remembering/reporting) their own behaviors
Plan for analysis• before conducting your experiment, figure out
how you’re going to analyze your data• what conclusions will you be able to draw from the
data?
normal photo
pro photo
4 5Did theyrent theapartment?
Design your experiment– A/B Test – Show ½ the people the a rental with
normal photos, ½ the same rental with professional photos.
– Constrain to one geographic area– Select 2 popular and 2 unpopular
apartments.–Measurement: What percentage of people
who view the apartment rent it in condition A vs. B.
Create the prototype
run the experiment&
analyze your results
Tip: You can follow up with qualitative interviewing
#1 Handicapping
Dark Horse#2 Dark Horses
dark horse:a contestant that seems unlikelyto succeed, but emerges to prominence against all odds
dark horse idea1. dark: explores a space that is risky,
difficult, radical, or orthogonal
2. brainstormed after more traditional ideas
3. refined enough that can be prototyped and tested with a rapid experiment (can’t be infeasible)
rational1. Designers need to preserve ambiguity to
leave themselves open to new ideas
2. Taking the dark horse idea to the point you can evaluate it keeps the design space from shrinking too fast
3. Even if the dark horse doesn’t work, you will gain insights that will make your final design better
Example 1
Design the next generation cinema/broadcastingimaging system
Original idea: Immersive movie watching experience where viewer’s head and eye position affect the perspective and focus of a sceneDark Horse: What if the camera was everywhere?Rigs of multiple Kinects and software capture 3D info of a scene for post capture scene manipulation.
Example 2
Design a system to improve street navigation forpeople on vacation
Original idea: Hotter/ColderPhone vibrates differently as you are getting closer or farther from the direction you should be going.Dark Horse: Stalker NavigationNo maps, crowdsourced directions, follow the stranger.
Dark Horse#2 Dark Horses
Handicapping & Dark Horses
What Design Thinking Can Learn from Horse Racing