user experience research report: usability testingruofanzhang.com/ux/file/collage/usability.pdf ·...

User Experience Research Report:

Usability Testing SI 622 003 Group 3: Yunqi Hu, Diane Pham, Chieh-Lin Wu, Ruofan Zhang Date: April 14, 2016 Word Count: 3,080

Table of Contents Executive Summary............................................................................................................ 2 Introduction........................................................................................................................... 3 Methods .................................................................................................................................. 4 Findings and Recommendations.................................................................................... 6

Summary of Results.......................................................................................... 6

Finding 1............................................................................................................ 6 Recommendation 1....................................................................................... 6 Finding 2............................................................................................................ 7 Recommendation 2....................................................................................... 7 Finding 3............................................................................................................ 8 Recommendation 3....................................................................................... 8 Finding 4............................................................................................................ 9 Recommendation 4....................................................................................... 10 Finding 5............................................................................................................ 10 Recommendation 5....................................................................................... 11

Discussion.................................................................................................................................12 Conclusion ...............................................................................................................................14 Appendices...............................................................................................................................15

Appendix A: Usability Script................................................................................. 15 Appendix B: Pre-Questionnaire Answers and Observation Notes........ 17 Appendix C: Post-Debrief Results………………………………………………….21

1

Executive Summary For this report, we conducted five usability tests in order to discover and analyze usability problems when using Collage.com. Usability testing allows us to actually visualize how users would use the site in real life and better understand how users use the site. Tasks were centered on product creation with one photo and product customization. After conducting the tests, we analyzed our observations and developed the following series of findings and recommendations. Finding 1: People do not understand what the cut area on the photo means. Recommendation: We recommend locking the photo once the user confirms the photo placement instead of still allowing the photo to be dragged and resized. This will allow users to take notice of what parts of their photo have been cut out. Finding 2: Users have trouble using the top function panel and finding certain functions. Recommendations: We provide several recommendations for improving this problem, such as by grouping features by category, implementing a customized feature panel that corresponds to the product choice of number of photos, and adding the photo editing feature button to the top to make it more noticeable. Finding 3: People get confused when trying to change or add the background or border to the product because there is no feedback from the system. Recommendation: We recommend providing the user with a feedback message of “Background is beneath the photo and may not be visible” when the user is trying to add a background to the product with a photo already existing on the product. Finding 4: People cannot find the “back” button to customize the back of the products. Recommendations: We recommend redesigning the “back” button to be more noticeable and having the system provide users with a reminder if they try to finish customizing the product without customizing the back. Finding 5: The user’s understanding of “Next” differs from the system’s. Recommendation: The “Next” button on Collage.com means “Add to Cart.” Because our participants thought the “Next” button meant moving on to the next step of the product customization, we recommend re-labeling the “Next” button to “Add to Cart” so it accurately reflects what clicking on that button will lead to.

2

Introduction Collage.com is a website that allows users to create customized products using their own photos. Collage.com offers its users with a large variety of product options from blankets to puzzles and photo customization and editing tools. Our team will be reviewing and analyzing the usability of Collage.com. Because Collage.com was initially intended for creating products with multiple photos, but a significant number of users are using only one photo, our overall research question through the entire assessment is:

● How can the customer’s experience of creating products using only one photo as opposed to the site’s standard multiple photo option be improved?

In order to better understand how users use Collage.com, we conducted usability tests with five users. Usability testing allows us to confirm existing usability problems discovered in earlier assessments and discover new usability issues we have not yet encountered. The usability test consisted of five tasks that were focused on understanding how users create and customize Collage.com products using only one photo. Our research questions for this assignment are:

● How easily and successfully can users create a product on Collage.com using one photo?

● How easily and successfully can users edit their product and photo on Collage.com? ● What features, if any, are difficult or confusing to use?

After conducting the usability tests, we highlighted any common problems from the tests, ranked the severity of each problem, and provided recommendations.

3

Methods Our group conducted five usability tests. Usability testing allows us to better understand how users use Collage.com by having participants perform common tasks and provide us with their feedback on the website. Usability testing can also help us confirm the usability issues we have identified throughout the semester through surveys and heuristic evaluations and potentially discover additional usability issues we had not encountered before. After conducting the usability tests, we gained valuable user input that helped us create well-informed recommendations for the client.

Script Design Our first step was creating our usability test script. We started by deciding on our problem scope and the tasks that we wanted participants to test. Our tasks were focused around the following steps:

● Product selection ● Uploading a photo to the product ● Adding text to the product ● Changing the layout of the photo on the product ● Editing the photo

After agreeing on our tasks, we created our usability script, which included an introduction, informed consent form, a pre-questionnaire form focused on learning about the participant’s online shopping habits and experience with customized products, the tasks, and a post-debrief form to fully understand the participant’s thoughts on using Collage.com (see Appendix for full script).

Test Setup All tests were conducted in group study rooms in North Quad to provide a big enough space and privacy for our group members and the participants. Tests were conducted on a laptop and recorded using Morae. Morae allows us to simultaneously record the laptop screen and the actual user so we can visualize what the user is doing and record their spoken words and facial expressions as well. There were at least two group members present at each usability testing. One group member would play the role as the moderator, sit next to the user, read the script, and instruct the user on what to do. A second group member would take notes on what the user was doing. Additional members were there for observation and assistance. We would also occasionally take pictures of the participant during the testing session with a camera.

Pilot Test Before conducting any formal usability testing, we administered a pilot test to ensure that our script and tasks were understandable and that our testing software would record properly. We used the feedback we got from the pilot test to improve our script and tasks before formally testing our participants.

4

Participant Recruitment and Selection The majority of Collage.com users are female, and the majority of those females tend to be middle-aged. However, because of time constraints, we aimed towards getting female users of any age who were unfamiliar with Collage.com. Therefore, we mainly recruited classmates and acquaintances. A brief breakdown of our participants is listed below:

User No.

Gender Age Educational Background

Familiarity with online product customization

1 Female 23 1st year Master’s student No experience with online product customization

2 Female 23 1st year Master’s student Made a photobook before




Analysis Methods After conducting the five usability tests, our group reconvened to discuss our notes and review the recordings. We noted any problems that arose in multiple test sessions and common comments our participants made during the actual testing or post-debrief. We then gave a severity rating score to each key finding using the same Nielsen severity scale from our Heuristic Evaluations assessment:

● 0 = don’t agree that this is a usability problem ● 1 = cosmetic problem ● 2 = minor usability problem ● 3 = major usability problem; important to fix ● 4 = usability catastrophe; imperative to fix

5

Findings and Recommendations

Summary Results The main mission of our usability testing is to discover existing and potential issues of customizing products with one photo in Collage.com. Overall, our five participants completed all the tasks successfully. However, we discovered some usability issues related to not only the single photo customization process, but also to the information architecture of the website. In addition, we gained a lot of valuable feedback from the post-debrief, which helped us generate more concrete recommendations for Collage.com. Finding 1: People do not understand what the cut area on the photo means.

● Severity: 3 Cropping is a frequently used function when people edit their photos on Collage.com. However, instead of directly providing a cropping tool for users, Collage.com provides a cut area to crop out areas that will lie outside the product (Figure 1). Through our usability tests, we found 2 of 5 participants cannot directly crop the part as they please. Instead, they have to zoom in or zoom out as well as drag the photo to leave the parts they want within the cut area while the other parts outside the cut area will be discarded. Our participants said that the final product preview was different from their expectations due to the misunderstanding of the cutting area. This problem was also mentioned in our initial client meeting and our survey results.

Figure 1 Recommendation 1 To address this issue, we suggest locking the selected area of the photo after a user has confirmed it instead of still allowing the photo to be dragged. Based on our participants’ feedback, we believe the reason users are confused with the cut area is because people can

6

always drag the photo to locate its different parts within the cut area. This creates the illusion that other parts outside the cut area will still be shown on the final product. If the site could lock the photo within the cut area once it has been selected, then people would notice that the parts outside cutting area have been cut. If people want to redo the cut area selection, there will be a button for them to reset the cut areas. In doing so, the cropping function will be improved. Finding 2: Users have difficulty using the top function panel and finding certain functions.

● Severity: 3 We discovered that users have trouble understanding the top function panel (Figure 2) and finding the proper functions they want. Our participants thought the top function panel is not presented in an organized way. In addition to this, on the top function panel, there is a “shuffle” function, which allows the user to shuffle multiple photos on the product. However when users are customizing with only one photo, the feature is still shown and is usable. There is no reminder to inform users of the inefficacy of that feature. This increases user’s confusion and becomes a usability issue. The photo editing feature button, which is shown only after clicking on the photo, is also difficult to find. For 3 of 5 participants, even though they clicked on the photo and saw the photo editing icon (Figure 3), they failed to realize that the icon was a signifier for photo editing.

Figure 2

7

Figure 3 Recommendation 2 To solve the problems found, we propose three recommendations:

● Organize the functions in the top panel based on their category instead of showing them directly on the panel. For example, the website Canva.com does a good job of implementing this as they group their customization features in a sensible order (Figure 4). Layout goes first because it is the most common step after choosing a product to customize.

● Implement a customized feature panel that corresponds to the product of choice and number of photos. For example, after users choose the product and photo(s), customization features only relevant to the product and photo choices will be shown on the panel to reduce confusion and information overload.

● Include a button in the top feature panel for photo editing with a more instructive icon to guide users to make this feature more noticeable.

Figure 4

8

Finding 3: People get confused when there is no feedback when they try to change background or border

● Severity: 2 Adding a background to a customized product is an important task in our usability test, but some common problems occurred among participants when trying to accomplish the task. 4 of 5 participants were confused by the background and border features. When they added a background to a product in the “edge to edge” mode, there was not any noticeable change on the product or any feedback from the system. This confused our participants who tried to add the background to the product multiple times but without any apparent change. The truth is that the background was already successfully added, but it was beneath the photo and unable to be seen by participants. Recommendation 3 The key reason causing this problem is the lack of feedback from the system. To address this problem, we recommend providing the user with a feedback message of “Background is beneath the photo and may not be visible” when the user is trying to add a background to the product with a photo already existing on the product in the “edge to edge” mode. Finding 4: People cannot find the “back” button

● Severity: 2 For some products on Collage.com, the user can customize both sides of the product, such as a blanket (the most popular product on Collage.com). When our group conducted heuristic evaluations, we also initially missed the “back” button (Figure 5), which will show the back side of the product for users to customize. In order to see if this was a common problem, we designed a task of customizing the back of a blanket for our usability test. One participant spent a lot of time finding the “back” button. 2 of 5 participants could not find the “back” button to design for the back of the product. Only one participant succeeded in this task. This means that the design of “back” button for some products is not obvious and noticeable enough for most of the users.

Figure 5

9

Recommendation 4 We provide two suggestions to improving this problem:

● Redesign the back button to make it more salient. ● Give a reminder to users if they finish the customization process without adding

anything to the back of a product. Finding 5: The user’s understanding of “Next” is different from the system’s.

● Severity: 2 During our usability test, we found 2 of 5 participants accidentally clicked the “Next” button (Figure 6), while they finished one of the tasks. The “Next” button would lead them to a pop out message to add their product to the cart (Figure 7), which confused participants because they are not expected to go to the checkout page after clicking the “Next” button. Participants thought clicking the “Next” button meant moving onto the next step of the customization process. Participants perceived the meaning of the “Next” button to be different from the actual system setting.

Figure 6

Figure 7

10

Recommendation 5: In order not to confuse the users with the “Next” button, we suggest that the label could be changed. The word “Next” might not be intuitive enough for users to connect it with “Add to Cart” or “Checkout”. Because the participants were in the customized product process, the “Next” label lead them to think of “save and go to next editing step”. The “Next” label could change to “Add to cart”, which might reduce the chance of misleading users. For example, Shutterfly uses two icons to present each feature, avoiding icon to mislead users (Figure 8). The other recommendation is that if the customization page could add a progress bar to let users know what step they are now, such as choose a product→ customize it→ check out, which might reduce the misclicking of “Next” button as well.

Figure 8

11

Discussion Limitation 1: Sample Bias The demographic information from our client and survey results revealed the primary users of Collage.com are middle-aged females. However, due to time constraints, our participants are mainly female graduate students, who may inherently hold a stronger technical background than middle age females. They may have a higher tolerance of usability issues and more capable of finishing our usability tasks, which prevents some usability issues from being uncovered. All of our subjects are also female. Even though that more than 95% of Collage.com users are female, we may still need to consider male users in our usability test. Despite the sample bias listed above, the results of our usability test are still valuable because all of our participants are all first time users with little knowledge about the website. It is safe to assume that all problems they encounter are typical. Moreover, if Collage.com would like to expand the market, it would be of significant importance to collect reflection from the principle consuming group of e-commerce. Limitation 2: Task Scope Our main focus on Collage.com is the customization process. Accordingly, all of the tasks in our usability test are customization functions. This set of tasks helps us to fully concentrate on the key functions and take a closer look at specific functions. However, the tasks lacked testing a general overview of the whole website. If we want to figure out what are the most popular functions used, more general tasks like “customize a gift for you mother” should be involved. In this case, the following tasks could focus on other specific functions that are not involved in this process. However, the drawback of this setting is that the tasks will be inconsistent between different participants according to their performance in the first general task, which may lead to the difficulty in keep consistent in the tested contents. Limitation 3: Task Ambiguity One of our tasks asked users to explore more layout options for the photo they chose. Our initial intention was to test the usability of the functions including adding shapes, switching from portrait to landscape, and so forth. However, many of the participants were confused with the task instruction of changing the layout. Therefore, our task instruction was not clear and instructive enough, which we should improve for future usability tests. Limitation 4: Pre Questionnaire Since our usability tests are done before the in-class one, we were unaware of the limitation of short-answered questions in the pre-questionnaire. Some of our questions were close-ended, which could be improved by adding more open-ended questions to get more information from our users. Limitation 5: Post-Debrief Our post-debrief questions are all pre-set and the same for all subjects. However, based on our observations during the usability tests, we could have implemented a dynamic

12

post-debrief for different participants based on their different performance to make the results we got from post-debrief questions more useful and meaningful than a universal set of post-debrief questions.

13

Conclusion We conducted five usability tests in order to understand how users use Collage.com and discover usability issues while using the site. While our users were able to successfully complete our designed tasks, we noticed common problems users faced that arose from the website’s information architecture, lack of system feedback, and wording. We provided recommendations aimed towards improving the website’s hierarchy of features, highlighting the system status, and changing the wording of certain labels in order to decrease user confusion and improve user experience. Throughout the semester, we have created and conducted an interaction map, interviews, comparative analysis, surveys, heuristic evaluations, and usability testing to fully understand how Collage.com can improve its usability. We hope that the findings and recommendations we have provided can help Collage.com in the near future.

14

Appendices

Appendix A: Usability Script Introduction Thank you for agreeing to participate in our usability study today. During our session, I’ll be using this script to ensure that, as far as possible, my instructions to everyone who participates are the same. Our objective today is to observe you using Collage.com to create a photo product. We’ve brought you here to see what you think of the site. This evaluation should take about 40 minutes. During the session, you’ll be working on your own while I observe you. I’ll be taking notes and we’re recording the session. All of the equipment in the room is to make sure that my notes are accurate. In this session, we’ll have you do typical tasks, to learn how the site work for people like you. Please keep in mind that we’re not testing you—it’s you who are helping us evaluate the site. Since it’s designed for people like you, we really want to know exactly what you think. Also, while you’re using the site, I’d like you to say your thoughts aloud. That gives us an idea of what you’re thinking when you’re doing something. Just narrate what you’re doing, sort of as a play-by-play, telling me what you’re doing and why you’re doing it. Here’s how the session will work:

1. You’ll do the tasks one at a time. Please don’t look ahead at the other tasks and don’t skip any tasks. 2. I’ll direct you about what to do and when to do it. 3. Start working on the task only once I have said, “Please begin.” I’ll start recording time as soon as you

touch the keyboard or the mouse—so don’t touch either one until you’re ready to start the task. 4. When you have completed the task, let me know. 5. After each task, you’ll answer a couple questions about your experience doing the task on that software. 6. At the end of the session, you’ll answer one more questionnaire. Do you have any questions before we

begin?

Informed Consent Form This is a usability study of Collage.com. It is intended to improve users’ experience with the site. In this study, we will ask you to perform specific tasks using the website and tell us what you think. We would like to video record this usability test and share the results with our team for the purposes of improving the site.We will not share your name or any other identifying information with anyone else. Please check the box if [ ] you give us permission to record video and share it with the team. [ ] you give us permission to quote any verbal statements you make during the test in our reports. You are free to leave at any time. If you end participation, we will delete any video we have captured and notes that we have taken. Please let us know as soon as possible if you have any questions or concerns I understand and agree. Name _________________

15

Signature ______________ Date__________________

Pre-Questionnaire

● Online shopping habits and experience: ○ How often have you shopped online the past month? ○ What types of products do you usually buy online?

● Customized product experience: ○ Have you ever bought a customized product before?

■ If so, what type? ○ Have you received a customized product before?

■ If so, what type? ■ If you haven’t, what type of product would you like?

○ Would you be interested in creating customized products? Tasks Task 1: Please imagine that you are going to select a gift for your mother’s upcoming birthday. Your mother particularly likes nice items to add to her bedroom. You would like to make it unique so you decide to go to make a customized product with a selected photo/image on it. Please go to Collage.com to choose a product and then upload a photo or image from your local drive (computer→ picture→ testphoto.jpg ) . Task 2: Now imagine that you’d like that the product would remind your mother of you as she sees the gift, so you decide to add “Happy Birthday! Love, + (Your name)” to your product. Task 3: Imagine you are not satisfied with the default layout and decide to modify it. You can change the layout with the top functional panel until you are satisfied with your layout. Task 4: Imagine you are not satisfied with the default effect of the photo and decide to edit it. (for example by increasing the lightness, adding a frame, or adding stickers etc.) Task 5: - If you are working on a blanket: Imagine you are good with the front of the product and want to add the background to its back.

16

- If you are not working on a blanket: You suddenly remember your mother mentioned she wanted a blanket the other day, so you decide to change your product to blanket, and then design the back of it. Post-Debrief

● Overall website Please indicate the extent to which you agree or disagree with the following statements. Scale from 1 (strongly disagree) to 5 (strongly agree).

1. The site is easy to use 2. I always know where I am in the site 3. The site is easy to learn 4. Please indicate your overall level of satisfaction for "select product to customize"

● Specific function 5. Please indicate your overall level of satisfaction for "upload photo" 6. Please indicate your overall level of satisfaction for "add text" 7. Please indicate your overall level of satisfaction for "layout" 8. Please indicate your overall level of satisfaction for "add background" 9. Please indicate your overall level of satisfaction for "add effect" of photo 10. Please indicate your overall level of satisfaction for "crop and resize photo"

● If there were 3 things you could change in the site you tried today, what would they be? ● Do you have any other comments or suggestions?

Appendix B: Pre-Questionnaire Results and Observation Notes User No.1 [ Pre-questionnaire ]

● Online shopping habits and experience: ○ How often have you shopped online the past month?

Two ○ What types of products do you usually buy online?

Cosmetics

● Customized product experience: ○ Have you ever bought a customized product before? No

■ If so, what type? ○ Have you received a customized product before? No

■ If so, what type? ■ If you haven’t, what type of product would you like? Mug with photos

○ Would you be interested in creating customized products? Yes, for others [ Observation ]

- Size makes no sense - “Back” function of the blanket is hard to figure out - Photos are not stable - Stickers not allowed to rotate - Not used help page - Change product: time consuming

17

[ Post feedback ] “Next” button: confusing Cropping & moving around the image

- “Family tree” only one picture is Multi-functional

● Confusion over product size ● Used the photo editing features ● Border is confusing ● Should have a customized function panel ● Dragging and resizing is annoying and difficult to do ● Back button is not easily noticeable

____________________________ User No.2 [ Pre-questionnaire ]


■ twice ○ What types of products do you usually buy online?

■ Vitamin, pair of glasses


■ If so, what type? ● Photobook, frames

○ Have you received a customized product before? ■ If so, what type? ■ If you haven’t, what type of product would you like?

○ Would you be interested in creating customized products? Yes ● Observation

○ Picks canvas because she’s made one before ■ Takes a decent amount of time to look through products and their

information ■ Concerned about the size - need to convert

○ Likes to explore all the options before making a decision ○ Takes to customize font even though it wasn’t specified in the task ○ Is the layout task ambiguous? ○ Understood that some features only made sense if you had multiple photos ○ Task 4: looked through left side panel and top function panel to edit photos

instead of clicking on the photo

18

○ Unsatisfied with photo dragging capabilities ○ Blanket back option only available to fleece and plush fleece ○ Website should tell where I am in the process. I am worried if I click on next or

change something, that I will have to restart. Should inform if I can save it as a draft

○ Font size is too big ○ Background and text style choices seem directed for middle-aged women ○ Familiar with photo customization sites ○ Expressed confusion and frustration often

_____________________________________________ User No.3 [ Pre-questionnaire ]



■ books




● Cups, basketball, plates ○ Would you be interested in creating customized products? Yes - for others as a

gift Observation

● Editing font before it is displayed on the product ● Customized layout by dragging objects (photo, font) ● Rug seems to have better customization function regarding layout, can easily resize size of

objects ● Asked about clicking the photo to use photo editor but ended up using top function panel ● Task 5 about switching to the blanket may be a little confusing ● Understood cut area well ● Thoughts:

○ Easy for people who have photoshop experience ○ Functions are simple ○ Would like to see what the real product and its color would look like

19

___________________________________________________________ User No. 4 [ Pre-questionnaire ]



■ books




○ Would you be interested in creating customized products? Yeah Observation

● Automatically choose blanket, didn’t explore other options ● Tried dragging the text ● Difficulty editing text ● Cut area makes it difficult to see border ● Expresses frustration if she can’t figure out how to do the task immediately ● Didn’t experiment with layout a lot ● For task 4, uses the top function panel ● Didn’t realize she already picked a blanket ● Back of blanket is hard to find ● If your photo takes up the whole product, then adding a background or pattern does not

make sense ● Thinks background, text, pattern should be in the top navigation bar ● Didn’t know how to change the layout - asked if you can even change the layout if you use

just one photo ● Didn’t allow you to drag text

_______________________________________________ User No. 5 [ Pre-questionnaire ]


■ Just look Amazon every week but didn’t buy anything this month ○ What types of products do you usually buy online?

20

■ Electronic devices, books


■ maybe, but don’t really buy customized product ■ If so, what type?

○ Have you received a customized product before? ■ If so, what type? ■ If you haven’t, what type of product would you like?

● Ipod with the name ○ Would you be interested in creating customized products?

■ yes Observation:

● Choose blanket, and choose the product successfully ● Finished adding test smoothly ● Didn’t know the text box and photo could be moved until task 3 ● Easily find the way to edit the sharpness of the photo ● Easily find the “back” of the product ● Overall, she finished all the tasks without difficulties

Appendix C: Post-Debrief Results Please indicate the extent to which you agree or disagree with the following statements. Scale from 1 (strongly disagree) to 5 (strongly agree).

1. The site is easy to use 2. I always know where I am in the site 3. The site is easy to learn 4. Please indicate your overall level of satisfaction for "select product to customize" 5. Please indicate your overall level of satisfaction for "upload photo" 6. Please indicate your overall level of satisfaction for "add text" 7. Please indicate your overall level of satisfaction for "layout" 8. Please indicate your overall level of satisfaction for "add background" 9. Please indicate your overall level of satisfaction for "add effect" of photo 10. Please indicate your overall level of satisfaction for "crop and resize photo"

21

If there were 3 things you could change in the site you tried today, what would they be?

● User 1: crop and resize-- smarter and more flexible; it would be better if it is separated into steps/tasks; next button is confusing

● User 2: display all choices of product in one page;more flexibility of text and layout;display where I am in the website

● User 3: Flexibility of the text layout; Partial effect on photo; Freedom of layer hierarchy ● User 4: change the background; Text ; Layout ● User 5: I think they should give more options for changing the layout; i also think they

could provide labels to indicate whether the photo is being made sharper or softer. ; iI also think it could be great if we could customize the size of the photo on the blanket

22

user experience research report: usability testingruofanzhang.com/ux/file/collage/usability.pdf ·...

Documents