cheap& quick internal user testing
DESCRIPTION
Hwo soes one do user-testing on a shoe-string when incrementally improving a live service? Here's some pointersTRANSCRIPT
![Page 1: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/1.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 1Mobile Design UK
3 Dec 2008
Jaeseung Chang ([email protected])Usability Architect @ Handmade Mobile Entertainment Ltd.
Cheap & Quick Internal User Testing
- Flirtomatic’s Recent Challenge -
![Page 2: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/2.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 2Mobile Design UK
Quick Introduction
- BA in Education from SNU
- MSc in Information Technology from QMUL
- PgDip in HCI from UCLIC
- MSc by Research in Computer Science from QMUL
- Open Education (Korea) / Usability Designer
- Samsung Electronics (Headquarter in Korea)
/ Mobile Usability Designer + Researcher
- Handmade Mobile (Flirtomatic) / Mobile Usability Architect
![Page 3: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/3.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 3Mobile Design UK
Cheap & Quick Internal User Testing
– Flirtomatic’s Recent Challenge
1. Why Cheap & Quick?
2. Experiment Design
3. Good Lessons – Dos and Donts
![Page 4: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/4.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 4Mobile Design UK
1. Why Cheap & Quick?1.1. Evaluation Needs
Requirement
Analysis
Fast
Prototyping
SpecificationImplementation
EvaluationEvaluation
- We all know the necessity of usability evaluation
- At least 2 times per 1 iteration cycle
- But the reality sometimes doesn’t help
![Page 5: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/5.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 5Mobile Design UK
1. Why Cheap & Quick?1.2. Usability Firms
- Hiring professional research firms?
- Excellent facilities, professional experiment design,
reliable recruitment, & nice report (maybe?)
- But costs a fortune!
![Page 6: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/6.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 6Mobile Design UK
1. Why Cheap & Quick?1.3. Self-Evaluation
- Then we should go DIY
- Venue: How about using our meeting room?
- Experiment design: We can do that maybe?
- Recruitment: Find some volunteers from our users?
- Observation and analysis?: 15 day trial version MORAE!
![Page 7: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/7.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 7Mobile Design UK
![Page 8: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/8.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 8Mobile Design UK
Cheap & Quick Internal User Testing
– Flirtomatic’s Recent Challenge
1. Why Cheap & Quick?
2. Experiment Design
3. Good Lessons – Dos and Donts
![Page 9: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/9.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 9Mobile Design UK
2. Experiment Design2.1. Overview
- The 1st iteration of our ‘Redesign Project’
- Based on wireframes (workflows) produced so far
- Evaluation on our first redesign UI/IA works
- To discover good/bad points for more future works
- Included three parts: UT + GUI Preference + FGD
User TestingGUI
Preference
Focus Group
Discussion+ +
![Page 10: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/10.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 10Mobile Design UK
2. Experiment Design2.2. User Testing
- Normal user testing techniques (think-aloud)
- Went through 6 sample tasks with 8 participants
- Used a hi-fi prototype to simulate our new UI
- Controlled experiment with video recording (MORAE)
- 1 moderator and multiple observers
CONFIDENTIALCONFIDENTIAL
![Page 11: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/11.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 11Mobile Design UK
2. Experiment Design2.3. GUI Preference
- Asked the same participants to choose our new GUI designs
- A sort of multiple-choice survey questions
- Asked to choose the best 2 & the worst 1 design
CONFIDENTIAL
![Page 12: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/12.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 12Mobile Design UK
2. Experiment Design2.4. Focus Group Discussion
- To collect real users’ preferences and behaviours
- In terms of using the current version of Flirtomatic
- Focused on gathering additional information
- To help redesign the details of our WAP service
- Went through 5 topics with 3 groups of users (m, f, & gay)
![Page 13: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/13.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 13Mobile Design UK
Cheap & Quick Internal User Testing
– Flirtomatic’s Recent Challenge
1. Why Cheap & Quick?
2. Experiment Design
3. Good Lessons – Dos and Donts
![Page 14: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/14.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 14Mobile Design UK
3. Good Lessons – Dos and Donts3.1. Outcomes
- Were able to conduct evaluation ourselves – DIY!
- Captured some important findings to validate our prototype
- Less than 2 wks (Experiment design + experiment + analysis)
- Less than 500.00 GBP
- Much quicker and cheaper than hiring research firms
- Were able to share the importance of usability evaluation
with the management level and all other departments
- Self-learn about the process of usability evaluation
![Page 15: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/15.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 15Mobile Design UK
3. Good Lessons – Dos and Donts3.2. Difficulties
- Recruitment error: No show up, recruited friends, etc.
- MORAE error (due to the inexperienced operation):
No video recording, low quality audio recording, etc.
- Some mistakes of our colleagues:
intruding into the UT room to connect to wireless network,
interrupting UT sessions to ask questions to the moderator,
giving money to participants during the session, etc.
![Page 16: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/16.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 16Mobile Design UK
3. Good Lessons – Dos and Donts3.3. Dos
- Good preparation in every way:
1. Documentation on task scenario, forms, analysis
2. Train yourself to master any SW tools (like MORAE)
3. Clean up the venue in advance
- Share the importance of usability evaluation with everyone
1. With the management level
2. Especially with colleagues nearby the UT room
(should politely ask silence during each session)
- Share the formal process of evaluation with team members
- Educate yourself how to run each session professionally
![Page 17: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/17.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 17Mobile Design UK
3. Good Lessons – Dos and Donts3.4. Donts
- Do not rely on the quick list of recruited users
(importance of having some back-up participants)
- Do not expect everyone else in office will behave as you do
1. They don’t know how important it is
2. Actually they don’t care what you do
- Do not expect your team members understand everything
1. They don’t know how to formally run each session
2. Importance of pre-training for usability evaluation
![Page 18: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/18.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 18Mobile Design UK
3. Good Lessons – Dos and Donts3.5. Resources
- MORAE: http://www.techsmith.com/morae.asp
- The Usability Methods Toolbox by James Hom:
http://jthom.best.vwh.net/usability
- Usability First: http://www.usabilityfirst.com/
- Jakob Nielsen: http://www.useit.com/
![Page 19: Cheap& Quick Internal user testing](https://reader036.vdocuments.net/reader036/viewer/2022081403/5561b9e6d8b42a9f2f8b4c53/html5/thumbnails/19.jpg)
© 2008 Handmade Mobile. All Rights Reserved. 19Mobile Design UK
Thank you
Jaeseung Chang