Heuristic EvaluationCollections on Beta on Android device

Overall Impressions● no glaring and few overall issues that prevent users from

using/navigating collections○ there are 3 bugs that need to be fixed asap, though

● however, collections pages, interactions, and workflows are rife with opportunities for users to experience the following:○ contextual confusion○ lack of consistency○ layout, icons used, or other interaction elements that may

not follow users’ mental models○ performing inadvertent actions

GuideFlows tested● logged out user creates a collection● logged in user adds a collection● logged in user adds to existing

collection from article● logged in user finds collection they

have created● logged in user adds to existing

collection from the collection page● logged in user adds description to

collection● logged in user shares collection● logged out user follows shared link

and finds collection (stricken flows not available)

Scoring for Severity

high = Show stopper. User can not accomplish task because it takes too long to figure out, or is non-functioning. No work-around.

medium = Takes a while, or is hard to find or learn how to do, but can still accomplish the task with a work around. Some users may give up.

low = Irritating or annoying though learnable.

none = no usability concerns.__________________________________________

other = Observations that don’t impact usability, but call out consistency, language, and other considerations.

High Severity - BugsNo high severity issues with workflows, yay!

High Severity - BugWatchlist landing page (Collections)● if watchlist list is empty, user is

prompted to add by clicking edit○ there is no edit button, to add

pages (or to edit existing pages, if watchlist is populated)

High Severity - RecommendationAdd an edit button.

High Severity - BugWatchlist and non-watchlist landing pages (Collections)● users are supposed to be able to

share collections via a shareable linka. there is no method to share on

this page

High Severity - RecommendationAdd a share button/functionality.

High Severity - BugNon-watchlist edit page● Delete button doesn’t work

High Severity - RecommendationMake sure the delete button does what it’s supposed to (surface warning dialog and delete collection when requested).

Medium Severity - Issues

Medium Severity - IssueAdd to existing collection action● When performing more than one

interaction with star/collections menu, the pop-up from the previous action briefly blocks the ability to interact with text lower on the window. This can potentially lead to accidental unintended actions and/or cause confusion.

Medium Severity - RecommendationConsider an (x) for the pop-up so that they can be manually removed when desired.

Medium Severity - IssueAdd to existing collection action● pages are now starred (in a green

color) only when they’ve been added to watchlist

● users may not understand this and add pages to new or existing collections, but go back to the page and think they hadn’t because they didn’t add it to the watchlist collection specifically and the star is not green-activated

Medium Severity - RecommendationUncertain what would be the best overall solution at this point, but the green star, green checkbox, + icons etc. all in my opinion should be re-aligned and made more consistent.

Re: this particular issue, I recommend starring an article when it has been added to any collection a user owns, not just specifically the watchlist collection, as an indicator that it has indeed been ‘activated’ in terms of collections.

Low Severity - Issues

Low Severity - IssueNormal navigation to pages on beta● pop-up about star/collections will

continue to occur on all pages visited

Tap elsewhere on page dismisses pop-up on that page● navigating to another page shows

pop-up again

Low Severity - RecommendationIs it accepted/understood that if a user taps off of a pop-up, that it continues to appear upon further navigation? It seems that a dismissive action like tapping off the pop-up (in addition to tapping ‘no thanks’, which is more directly dismissive) should both dismiss pop-ups during further navigation.

Low Severity - IssueStar/Collections Menu● if user scrolls down the

star/collections menu, when scrolling back up in the menu (dragging downward), user may inadvertently reload the page

Low Severity - RecommendationIf there is a way to separate the star menu from the typical browser actions to minimize this potential inadvertent action, that solution would be ideal.

Low Severity - IssueAdd to new public collection● no contextual understanding of

what a ‘new public collection’ is.○ is there a private collection?○ am i creating something? where

exactly am i adding this? am i somehow doing both?

○ what happened to my watchlist? is it still somewhere?

○ is it mine, but publicly viewable, or will it be a collection anyone can view and add to?

Low Severity - Issue‘Enter a new collection’ and ‘Create’ button (Add to new public collection)● keyboard can hide the field on

some devices● box for text entry doesn’t always

load fully, and it looks like a [] next to the ‘create’ button

Low Severity - RecommendationMinimizing the white space on the top section of the star menu may help with this issue. If this is not possible, it would be helpful to ‘jump’ the user to that field, where the field is then positioned immediately above the keyboard ‘fold’.

Make sure the box for text entry always loads correctly/fully.

Low Severity - IssueAfter creating a new collection (Add to new public collection)● tapping ‘back’ or ‘save’ from edit

collection page goes back to the wikipedia page. however, the star is not ‘activated’ on the page and it doesn’t look like anything was done, even though it is listed in the collections with a green checkbox

● also, both actions (back/save) do the same thing. this is potentially confusing.

Low Severity - RecommendationI believe the star-’activation’ has since been changed to the following condition, that it is only ‘activated-looking’ if the article is added to watchlist collection.

Again, would recommend that article is ‘green-starred’ when it has been added to any collection a user owns.

Low Severity - IssueAfter creating a new collection (Add to new public collection)● even though users are being directed

to the ‘new’ collection (blocked by volume in screenshot), it does not necessarily eliminate contextual confusion○ + icon for collections and star

for watchlist in the sidebar, star on article page, watchlist within collections

Low Severity - IssueAdd to existing collection● no contextual understanding of

what an ‘existing collection’ is.○ why is watchlist under existing

collection?○ if there are existing collections,

why is watchlist the only one?○ are these existing collections in

wikipedia or are they just mine?

Low Severity - IssueAdd to existing collection● watchlist entry is ‘below the fold’

on some devices (Abbey’s iPhone exhibits this issue)○ could be larger typeface setting

by user● ‘Test’ collection I made is under the

fold. Users may not even know that their newly created collection(s) are below the fold and can be added to. Perhaps contributes to the numerous one-item collections.

Low Severity - RecommendationAdding a scroll bar would make the UI inconsistent, but there needs to be a more apparent way for users to realize that there is content below the fold.

Low Severity - IssueAdd to existing collection● when users add to existing

collections, a checkmark is green-activated. This is not a problem in and of itself, but potentially creates more general contextual confusion due to the star on the page, the star in the menu, the checkmark in ‘adding to existing collections’, then the plus in the hamburger menu

Low Severity - RecommendationSuggest rethinking of the gray and the green ‘activated’ checkbox as an indicator of status and action: ‘added’ vs ‘not added’ or ‘added’ vs ‘removed’.

Generally, the icons involved with watchlists and collections need to be streamlined.

Low Severity - IssueWatchlist● since the star/collections menu will

now be the first interaction from article pages for adding to lists/collections, the following confusion may result:○ on the watchlist page itself,

there is no indication of it being a collection■ there is no indication that

it is public or private

Low Severity - RecommendationOn beta, I think it would make more experimental sense to do a full absorption of the watchlist by the new collections, and collect feedback from that. Having both is confusing, especially when the star watchlist page looks totally different than its corresponding page within collections, and it is not marked public/private as it is in collections, among other inconsistencies.

Low Severity - IssueCollections main landing page● if watchlist is the only collection

listed and it is labeled ‘private’, this may cause confusion back on the article pages (with the star/collections menu) when users are prompted to ‘add to new public collection’

Low Severity - RecommendationMake collections all public or all private by default, and users can edit that setting after the fact.

Low Severity - IssueCollections main landing page● cannot tap on the gray rectangle

area to access the collection; must click directly on the name area within the box○ unless the photographic

representation from a page in the collection shows up, in which case the photo part can also be tapped to access

Low Severity - RecommendationEntire area should be tap-able to access that particular collection (entire gray box and photo area, if applicable).

Low Severity - IssueNon-watchlist edit page● when the list is empty or populated,

the buttons ‘edit title and description’ (which is also too long for translations) and ‘delete’ are present

● potentially confusing what exactly is being edited and deleted

Low Severity - RecommendationThe buttons could just be ‘Edit’ and ‘Delete’, if it is clearly associated to the collection itself and not the pages listed within it.

However, these buttons (and adding/deleting pages to a collection) could all be functionalities surfaced more directly from the collection main landing page and the collection landing page.

Low Severity - IssueNon-watchlist edit page, ‘Edit title and description’● why is the public/private collection

toggle buried here?○ after unchecking public

collection, going back to the landing page requires a refresh to see the ‘private’ designation up top

Low Severity - RecommendationThis is a bit of commentary on both existing flow and potential for additional features.

Again, public/private tag on collections pages can be tap-able toggles.

Certain pages require distinct ‘confirmation’ actions, like the save button on this page. Others, like adding to existing collection on star menu, do not. Should we be consistent about these?

Low Severity - IssueNon-watchlist edit page, ‘Delete’● many users may hesitate just to click

on this for fear of what this will delete○ is it deleting one page, if there’s

one page in the collections?○ is it deleting all the pages, if

there’s more than one page?○ is it deleting the collection?

● only clicking it surfaces the warning dialog (and this button is sometimes non-functional)

Low Severity - RecommendationThe button does not need to be such a bright red, which is potentially very deterring.

Secondly, it and the ‘edit’ buttons should both be (as previously mentioned) more closely associated to the collection itself, and not the pages listed within it.

Low Severity - IssueNon-watchlist edit page, search bar● tapping brings user to new page.

existing pages in collections is there, but searching covers those pages.

● user needs to exit out of search results by clicking back or the ‘NO’ icon, which is a confusing icon

● if user taps ‘back’ again, user will realize that user needs to save even though the page(s) (‘Saucony’, in screenshot) already appears added

Low Severity - RecommendationIt is confusing when on an article page, tapping on an existing collection green activates it and requires no extra confirmation step, but an extra confirmation step is required here. Making that consistent could be helpful for users.

The ‘no’ icon is very confusing; is there any reason this is used instead of a standard x for cancel search/clear search bar?

Other Issues (in list form)Observations that don’t impact usability, but call out consistency, language, and other considerations.

Other IssuesObservations that don’t impact usability, but call out consistency, language, and other considerations.

Star/Collections Menu● there is a lot of horizontal white space

Star/Collections Menu || Add to new public collection● long; translation may become an issue down the line

Star/Collections Menu || Add to existing collection● long; translation may become an issue down the line

Watchlist● for first-time users, there may be confusion around the tabs ‘list’ and ‘modified’

○ the tabs under modified (all, pages, talk, other) may add to the confusion● other than the star in the hamburger menu, there is no star (or + icon) on the watchlist page

Collections || Main landing page● does not reinforce the plus icon ‘re/branding’, icon not present● no photographic representation of a page and/or pages from the collections

○ sometimes populated collections don’t have a photo background, other times there are. buggy, or is there a rule around this determination?

Other IssuesObservations that don’t impact usability, but call out consistency, language, and other considerations.

Collections || Watchlist landing page● Watchlist from hamburger menu and watchlist collections landing page look totally different -

more contextual confusion○ still no reinforcement of collections icon here

● headings are awkward, spacing between lines is awkward, font sizes look randomCollections || Non-watchlist landing page● still no reinforcement of collections icon here● though there was an indication on watchlist landing page that it is private, there is no parallel

indication on non-watchlist landing pages that these collections are public● headings are awkward, spacing between lines is awkward, font sizes look random

○ photos from pages are very large. potentially a lot of scrolling will be involved for collections with 2+ pages■ suggest a photo view and a listing view, similar to etsy storefronts, and a text listing

view. photo view is more of an organized pinterest view, listing view offers an avatar photo view but a full page name, and text listing offers a quick list of just the names of the pages

Other IssuesObservations that don’t impact usability, but call out consistency, language, and other considerations.

Collections || Non-watchlist edit page● menu bar (where the collections title is located) looks too much like a search bar● why is the ‘edit title and description’ button blue?

○ why is the edit capability for toggling between public and private collection setting buried here■ why is there a ‘tag’ for private, but not public?■ why is this edit capability not surfaced on the actual tag on the collection landing

page● why is the ‘delete’ button red?● why is there red, blue, and green on this page?

Collections || Non-watchlist edit page, search● cta in the search box needs shortening

Issues/Future feature considerationsLack of current functionality that potentially affects usability; and recs that can help steer future product iterations.

Medium Severity - IssueAdd to existing collection action● users can only add to/remove from

one existing collection at a time; inconvenient if trying to add to/remove from more than one existing collection

Medium Severity - RecommendationConsider whether this menu should be a single-action and minimized menu, or if more than one action can be taken.

Medium Severity - IssueCollections main landing page● no option to add a new collection

‘bucket’ from this page● user can only work with existing

collections here and create new ‘buckets’ from wiki pages

● no centralized way to create new collections

Medium Severity - RecommendationAdd functionality to surface actions like adding a new collection bucket (and possibly deleting current buckets, if not too destructive) from the main collections landing page, and directing the add to a modified version of the current collections ‘edit’ page.

Making users go to an article page to start a new bucket every time seems excessive.

Medium Severity - IssueWatchlist and non-watchlist landing pages (Collections)● when users do not access this page

from the main collections landing page, they will be unable to ‘go back’ or ‘jump’ there and see all collectionsa. two taps from here back to the

main collections landing page*android devices have a ‘return’ button, but again that is only usable if user accesses the page from main landing page

Medium Severity - RecommendationFrom watchlist and non-watchlist landing pages in collections, allow users a quick navigation option to return to the main collections landing page, regardless of their click path getting there.

Low Severity - IssueCollections main landing page● cannot access edit collection

directly from main landing page

Low Severity - RecommendationFrom the collections main page, users should be able to quickly access add collection, delete collection, and edit collection options directly instead of having to click twice (or more).

Low Severity - IssueWatchlist landing page● cannot tap to change setting to

public (or vice versa)

Low Severity - RecommendationThe ability to change public/private collections setting should be surfaced where it appears (and there should be a ‘tag’ for public as well as private), not after 4 taps (nav to edit, nav to edit title/description, toggle public on/off, save).

Low Severity - IssueNon-watchlist landing page● there is no quick way to remove

pages on this landing page when populated

Low Severity - RecommendationWhen on a collections page, user should have the ability to quickly access delete, add and edit lists (as mentioned previously).

Low Severity - IssueNon-watchlist landing page● if a user is brought to this page after

adding to an existing collection from an article page, user may not immediately realize that the pages are organized according to descending alphabetical order, and may have to scroll a lot to confirm the page is added

Low Severity - RecommendationWhen a collection has more than one article, there should perhaps be a ‘sort by’ feature (alphabetical, date added, date modified, etc).

Low Severity - IssueGeneral● users have no ability to add to

watchlist/existing collections and to a newly created one simultaneously

Low Severity - RecommendationIf users can perform multiple actions like this within the star menu, then it could be made more consistent with the issue on the last two pages; within the star menu, there can be a confirmation button after one or more actions are made.

This potentially can make one more page interaction ‘confirmation-based’, as in requiring a user to tap a ‘save’ button after one or more action.