Research Design and Evaluation

Colin Ware

Goals for empirical research

Uncover fundamental truths and test theories (late stage science).

Discover the nature of the world. Find out about how people interact with information artifacts. Find out how people respond to a particular display. (survey of recreational boaters).

Does theory apply in practice. E.g. good continuity applied to flow vis.

Compare two (or more) display or interaction methods. Is this method adequate?

Continued

Compare two entire systems. (GIS 1, vs GIS 2) Measure task performance (e.g how fast can we find a

certain class of patterns). Determine user preferences._________________________________

Non-empirical contributions

Invent a better interface for some particular domain (Social network analysis, interactive flow visualization) Proof of concept prototype Constructivist analysis

Invent a better evaluation method Invent a new “theory” (information foraging

theory)

Methods for evaluation Measure errors (how accurate are we)

Target detection (rapid response) Binary data Method of adjustment

Measure time to perform a task How much work can we do per unit time.

Rating scales (Likert scales) Turn judgments into numbers.

Task analysis + Structural analysis. A cognitive theory is constructed with observation as evidence

(constructivism). (e.g. narrative). Goes hand-in-hand with proor of concept applications

Semi-stuctured interviews. Expose the user to a problem + solution in a

systematic way. (+questions) Wizard of Oz Cognitive walkthroughs.

Have the user perform a task and talk aloud as they do it.

Spiral design method

Must documentthe design process

Spiral design

One way of proceeding is though a spiral design method.

Construct a rapid prototyping environment (tool kit) – demo or die

Construct a prototype Evaluate (and

document findings) Refine prototype

Evaluate (and document findings)

Methodological Issues

If you want to measure time/error data Best to have a simple representative task Justify it.

If you want to get opinions (ratings, interviews). Domain experts often ideal. But they may have established ideas – they like the status quo.

Information interface design

Determine data and problem

Determine tasks

Create mappings from data to representation such that important patterns are “transparent”.

Create interfaces for rapid task performance

Create support for hypothesis

Things to consider

Aim for important targets. Keep it simple Maximize payoffs for (your) time expended Be sure to have control conditions.

Defacto standard Alternative display methods.

Choose dependent variable ranges. Remove all un-necessary variables (e.g.

background). Have sufficient observations.

Statistics

Means. Standard deviations. Standard error of the mean = /sqrt(N);

2se ~ 95% confidence intervals.

ANOVA e.g. color and motion.

The End

Velocity glyphs

CurvedCompactShow direction clearlyEasily readable: more looks “greater”Capable of 100 levels

Visualization design pattern

Patterns of common low-level tasks

Relating patterns- e.g. bioinformatics.

Pattern searching Finding paths Finding groups Finding target symbols