Selecting, recruiting and informing users

Alfred KobsaUniversity of California, Irvine

Recruiting users• How many?

– Using 5-8 users for a set of tasks will already reveal most major UI problems related to those tasks

– Use 3-5 users for each subgroup (☛ potentially exponential growth of subject size)

– Considerably more if statistically significant differences are at issue

• Inhouse or external?– Inhouse: confidentiality preserved, good motivation, inexpensive(?), fewer logistics, can be rushed

– External: users often more representative, neutral

• Recruitment channels:• announcements (posters, flyers, [online] bulletin boards,

[online] classified ads, Internet forums, email, …), word-of-mouth, crowdsourcing, recruiting agencies, sales / distribution channels (?)

Selecting and scheduling users

• “Screener”: set of criteria to help filter participants – Positive criteria: specific demographics, skills

• Go for facts, avoid self-assessment

– Negative criteria: contacts to competitors, “odd people”, “professional test users”

• Screen potential participants on first contact (preferably via phone or in person, 10-15 mins)

• Send out invitation in writing, and an email reminder on the day before (with directions, phone number for last-minute changes)

• Over-recruit (?), or better have backup• Create an inhouse database of potential subjects (also for tax purposes)

Participant incentives

• Prime motivation to participate should be – being interested in your product (but not too much)– being curious about user studies– feeling flattered– wanting to help develop a better product– wanting to help you and others

• Incentives can also be a motivator, but generally should not be the only one– “Freebies”, gift coupons, cash, charitable donation, …– Thoughtful selection can be very important– Incentive should not be too high

• Give everyone the same amount, or make sure that participants are not aware of discrimination

Legal requirements

• Company policies often require subjects to sign liability waivers, video release forms, non-disclosure agreements,…

• Federal agencies and colleges receiving federal funding:Federal regulations for the protection of human subjects may require informed consent of subjects if scientific research is performed (whose purpose is to enhance human knowledge, and not “only” to improve a product).

• Codes of conducts of professional societies (such as the American Psychological Association) may regulate how their members have to carry out human subjects studies.

Informed consent

• Purpose (in general terms; deception permitted, specifically if disclosure at the end of experiment)

• Procedures• Risks• Benefits, Incentive/Compensation• Confidentiality• Right to withdraw• Consent to being video/audiotaped• Retention and “secondary use” of data• Point of contact