samanthamannportfolio.files.wordpress.com€¦ · web viewthe focus of these usability test...

Mann, Samantha

Sogeti USA

11/25/2015

2015Usability Findings Report, Iteration 1

DSS Usability Findings Report

Table of ContentsIntroduction.....................................................................................................3Executive Summary.........................................................................................3Methodology....................................................................................................4

Sessions........................................................................................................4Participants...................................................................................................5Evaluation Scenarios.....................................................................................7

Results.............................................................................................................7Task Completion Success Rate.....................................................................7Time on Task (ToT).......................................................................................8Errors............................................................................................................8Summary of Data..........................................................................................9Overall Metrics..............................................................................................9

Overall Survey Responses.........................................................................9Overall Questionnaire Responses............................................................12Liked Most................................................................................................13Liked Least...............................................................................................14

Recommendations.........................................................................................17Conclusion.....................................................................................................23Appendix........................................................................................................24

1


Table of FiguresTable 1: Total Test Session Time.....................................................................3Table 2: Background Question Responses.......................................................6Table 3: Test Scenario Completion Rates........................................................7Table 4: Time on Task......................................................................................8Table 5: Number of Errors per Test Scenario...................................................9Table 6: Summary of Data...............................................................................9Table 7: Participant Survey Responses..........................................................11Table 8: Participant Questionnaire Responses...............................................13Table 9: Participant A Liked Most...................................................................13Table 10: Participant B Liked Most.................................................................14Table 11: Participant C Liked Most.................................................................14Table 12: Participant A Liked Least................................................................16Table 13: Participant B Liked Least................................................................16Table 14: Participant C Liked Least................................................................17Table 15: Recommendations Severity Rating Scale......................................17Table 16: Test Scenario 1 Recommendations................................................18Table 17: Test Scenario 2 Recommendation.................................................19Table 18: Test Scenario 3 Recommendations................................................19Table 19: Test Scenario 4 Recommendations................................................21Table 20: Test Scenario 5 Recommendations................................................22Table 21: General Recommendations............................................................23

Attachment A: Participant Consent Form.......................................................24Attachment B: Test Scenario 1, Participant Handout.....................................25Attachment C: Test Scenario 2, Participant Handout.....................................25Attachment D: Test Scenario 3, Participant Handout....................................26Attachment E: Test Scenario 4 Part 1, Participant Handout..........................26Attachment F: Test Scenario 4 Part 2, Participant Handout...........................27Attachment G: Test Scenario 5, Participant Handout....................................27Attachment H: Post-test Survey.....................................................................28Attachment I: Post-test Questionnaire...........................................................29Attachment J: Error Log Template.................................................................30Attachment K: Error Log Code Definitions.....................................................31Attachment L: Affinity Matching, Bottom-up Analysis Method.......................32

2


Introduction Medical client Decision Support System (DSS) stores hospitals’ expense and revenue data and supplies the user a means to evaluate the hospital’s money flow. DSS has several modules available to users—the newest is the “ABC” module. Analysts using the ABC module will be able to build an ABC model and assign an ABC model to a population of encounters.The focus of these usability test sessions is to evaluate the DSS ABC module to determine how easy it is for users to complete routine tasks. The ABC module is currently in full functional prototyping; therefore, this usability testing is formative. Formative usability testing simply means testing is taking place during the product’s design phase with the aim of discovering insight into user’s expectations and areas of struggle and delight, as well as providing recommendations to shape the direction of the product’s design. The first iteration of ABC Usability Testing was conducted onsite at the client’s place of business using portable equipment. One Medical client laptop was used as the testing device. The ABC prototype was accessed through the DSS Sales demo environment for testing purposes. The usability expert acted as the test facilitator, data logger, observer, and data analyst. The usability expert and the participant were the only individuals present in the testing room during individual test sessions. All three individual test sessions were recorded using Camtasia and analyzed to identify potential areas for improvement to the ABC module. The sessions capture participant’s navigational choices, task completion rates, comments, questions, and feedback.

Executive SummaryThe usability expert conducted the first iteration of ABC usability testing on-site at Seton in Austin, Texas on Wednesday, November 11, 2015 with three potential users. The purpose of the test sessions was to assess the usability of the ABC module’s interface design, information flow, and information architecture.The following table displays the duration of each test session, per participant:

Participant A Participant B Participant CTotal Test Session Time

58 minutes 1 hour, 1 minute 30 minutes

3


Table 1: Total Test Session Time

In general, all participants understood the purpose of the DSS system by looking at the homepage, and were also able to quickly identify Medical client as the site vendor. Two out of the three participants stated that they use the DSS system on a daily basis and that it is integral to their work.The test identified a few usability problems:

1) User confusion of where to create a new ABC model2) Confusion with creating folders3) Confusion with operators in Advanced Criteria4) Users are unfamiliar with the term “model” in ABC model5) Users find the Costing bubble on the Homepage hard to read, due to

the green color contrast with the white text6) Users expect to use a single-click action over a double-click action to

populate or open dataThis document contains participant feedback, overall satisfaction ratings, task completion rates, ease or difficulty of completion of task, time on task, errors, and recommendations for improvements. A copy of the test scenarios, survey, and questionnaire are included in this report’s Appendix section.

Methodology Sessions Medical client was responsible for contacting and recruiting participants for the first iteration of usability testing, as well as all iterations of testing to follow. The main criterion for the target user group, as defined by Medical client, consisted of two parts: 1) The target user will be familiar with the concept of an ABC of Care, and 2) The target user will have a working knowledge of the DSS. Medical client e-mailed potential participants informing them of the need to conduct usability test sessions and the potential participants responded with their availability. The usability expert scheduled each on-site test session according to the participant’s availability. During the test sessions the usability expert explained the purpose and the agenda of the test session and asked the participant to sign a consent form, allowing the usability expert to record audio, video, and screen capture of the test session (See Attachment A). The usability expert then asked participants brief background questions, recorded their responses, and read each test scenario to the participant, as well as provided the participant a

4


paper copy to read along (See Attachments B-G). The Usability expert recorded the time it took the participant to complete each test scenario, encouraged the participant to think-out-loud about what he or she was doing, guided users when they struggled with test scenarios, and took observational notes. Test Participants were encouraged to ask questions and provide any positive or negative feedback throughout the test session.After the participant completed the last test scenario, the Usability Expert asked the participant to fill out a survey using a 5-point Likert scale (Strongly Disagree to Strongly Agree) to rate 15 subjective measures, including the following:

1) It is easy to find my way around the site.2) I can quickly enter information.3) It is easy to remember where to find things.4) Screens present content in a manner that is easy to understand.5) Information is written in a style that suits me.6) Screens contain the right amount of information.7) It is clear how screen elements (e.g., pop-ups, scrolling lists, menu

options, etc.) work.8) The site is designed with me in mind.9) The site is relevant to my professional needs.10) I would use the site to create an ABC.11) I would recommend my colleagues to use the site to create an

ABC.12) The web application is well-suited to first-time visitors.13) The web application is well-suited to repeat visitors.14) I always felt I knew what was possible to do next.15) My mistakes were easy to correct.

The participant was also administered an open-ended questionnaire to elicit honest feedback on their first impressions of the ABC module after the completion of the final test scenario. The questionnaire included the following:

1) What are your overall impressions of the ABC module?2) What are the three things you like best about this web application?3) What are the three things you like the least about the ABC module?4) Is building an ABC model easier, or more difficult, than building a

Contracting Model? (only for DSS users)5) Is the ABC module a tool that your company might invest in? 6) Would you prefer to see the ABC module made more flexible?

5


a) If yes, which components seemed too rigid?7) Would you prefer to see the ABC module made more pre-

configured?a) If yes, which components seemed too flexible?

8) Is there anything else you would like added to the ABC module?Please see Attachments H and I for the survey and questionnaire documents.Participants All usability test session participants are employees of Client in Austin, Texas.Three participants were scheduled for on-site test sessions from 11 a.m. – 4 p.m., Wednesday, November 11, 2015. The following table displays each participant’s responses to background questions:

Participant A Participant B Participant COccupation Lead Business

AnalystDirector of Clinical Effectiveness

Business Systems Analyst

Type of Work Outcomes Reporting

Helps clinical stakeholders evaluate the success of service lines and initiatives

Costing analysis

Internet Use (hr./wk.)

84 hr./wk. 50 hr./wk. 100 hr./wk.

E-mail:Browsing %

50:50 80:20 20:80

ABC Familiarity Basic understanding of the concept of an ABC

Intermediate understanding of the concept of an ABC

Basic understanding of the concept of an ABC

DSS Familiarity HIGH

Uses DSS almost every day; “Integral part of work”; mainly uses Analytics

LOW

Never uses DSS; Asks others to pull data from DSS when needed; Does

HIGH

Uses DSS all day, every day; Mainly uses Costing and Reporting

6


and Reporting modules

not have a DSS login; Did not know until the day of testing what the DSS acronym represented (Decision Support System)

modules

Table 2: Background Question Responses

The data in the table above shows that the users are comfortable to advanced regarding working on the Internet, that the users have a basic to intermediate understanding of the concept of an ABC, and that two out of three users have a high familiarity level with the operation of DSS. Medical client stated that the target user group should fit two main criteria: 1) The target user will be familiar with the concept of an ABC, and 2) The target user will have a working knowledge of the DSS. The participant’s responses to background questions show that Participants A and C fall into the target user category, while Participant B is not the ideal target user, due to a very low familiarity with using DSS. However, all three participant’s data and responses will be reported as valuable information for the purpose of this usability study.Evaluation ScenariosThe three test participants attempted the completion of the following tasks (Please see Attachments B-G for complete test scenarios):

1) Locate the button that allows you to create a new ABC and name and save a new ABC

2) Create a new [parent] ABC folder3) Create a new [child] ABC folder (Scenario asks: How does the user

change folder hierarchy?)4) Build a new ABC: Create new services, and specify inclusions and

exclusions5) Add a new service and remove an existing service from an existing

ABC

ResultsTask Completion Success RateThe usability expert was responsible for recording each participant’s ability to complete the test scenarios.

7


All participants (100%) successfully completed Test Scenario 1 (Locate the button that allows you to create a new ABC and name and save a new ABC). One out of three participants (33%) wrongly assumed they had successfully completed Test Scenario 2 (Create a new [parent] ABC folder). Two out of three participants (66%) were able to complete Scenario 3 (Create a new [child] ABC folder). Two out of three participants (66%) completed Test Scenario 4 (Build a new ABC: create new services and specify inclusions and exclusions). Participant C was able to complete the adding of new services to an ABC for Test Scenario 4, but a bug in the system caused his services not to save, and therefore the remainder of Test Scenario 4, as well as Test Scenario 5, could not be completed by Participant C. All three participants required guidance from the Usability Expert on Test Scenario 4. Two out of three participants (66%) successfully completed Test Scenario 5 with minimal guidance from the Usability Expert. The following table displays the test scenario completion rates (the number of successes divided by the number of participants):

Participant

Test Scenario

1

Test Scenario

2

Test Scenario

3

Test Scenario

4

Test Scenario

5A √ √ √ √ √B √ √ √ √ √C √ - - - -Success 3 2 2 2 2Completion Rates

100% 66% 66% 66% 66%

Table 3: Test Scenario Completion Rates

Time on Task (ToT)The usability expert, acting as the data logger, recorded the time on task for each test scenario and each participant. Some test scenarios (AKA: Tasks) were inherently more difficult to complete than others, as reflected by the average time on task.Test Scenario 4 required participants to build a new ABC model from scratch and took the longest time to complete (mean = 867 seconds). Test Scenario 2 instructed participants to create a new ABC folder and took the least amount of time to complete (mean = 25 seconds). Test Scenario 1 is the only task that has a 100% completion rate and took the second least amount of time for participants to complete (mean = 55.6 seconds). Overall, completion times ranged from 10 to 1,245 seconds, with most tasks taking less than 250 seconds to complete. The following table displays the Time on

8


Task (in seconds) per participant for each test scenario, as well as a mean Time on Task (in seconds):

Test Scenario

Participant A Participant B Participant C Mean ToT

1 45 74 48 55.62 17 10 48 253 230 14 - 1224 1,245 (over

20 mins.)769 (over 12

mins.)587 (over 9

mins.)867

5 520 (over 8 mins.)

168 - 344

Table 4: Time on Task

ErrorsThe usability expert captured the number of errors participants made while trying to complete the test scenarios during the analysis phase of the usability study, using error logs (See Attachments J and K) while reviewing test session recordings.Participants made the largest number of errors in Test Scenario 4, building a new ABC (Total # of Errors = 27). The other four test scenarios each individually totaled four or less errors. However, Participant C wrongly assumed he correctly completed Test Scenario 2 and therefore could not complete Test Scenario 3. Participants A and B each made two non-critical errors in Test Scenario 1. A non-critical error is an error that does not prevent the successful completion of a test scenario. In Test Scenario 3 Participant A made three non-critical errors. In Test Scenario 5 Participant B made two non-critical errors. None of the participants were able to complete Test Scenario 4 without multiple assists from the usability expert (acting as the Test Facilitator). The test facilitator made several assists to participants directly following what would have been a critical error had the test facilitator not offered guidance to the participant while they attempted to complete Test Scenario 4.The following table displays the total number of errors made by the participants in each test scenario:

Test Scenario

Participant A Participant B Participant C Total Errors

1 2 2 0 42 0 0 3 33 3 0 - 3

9


4 10 11 6 275 0 2 - 2

Table 5: Number of Errors per Test Scenario

Summary of DataThe table below displays a summary of the test data. High errors and high time on tasks are highlighted in red.

Test Scenario Test Scenario Completion

Total Errors Mean Time on Task

1 3 4 55.62 2 3 253 2 3 1224 2 27 8675 2 2 344

Table 6: Summary of Data

Overall MetricsOverall Survey ResponsesThe usability expert (acting as the Test Facilitator) administered a post-test survey (See Attachment H) upon completion of the final test scenario. The survey asked participants to use the Likert scale (1 = Strongly Disagree and 5 =Strongly Agree) to rate their agreement with statements regarding the general usability of the ABC module in DSS. The participant’s responses to the survey provided beneficial insight into their individual user experiences with the ABC module. Participant A’s satisfaction responses are all low (a 3 or less). Ten out of 15 of Participant B’s satisfaction ratings are high. Participant C’s satisfaction ratings are high, minus one. However, it must be noted that Participant C only successfully completed one out of five test scenarios, so he did not have the full experience of working in ABC, like the other two participants. The following statements were generally agreed with (4 = Agree 5 = Strongly Agree) by participants:

It is easy to find my way around the site. It is easy to remember where to find things. Information is written in a style that suits me. Screens contain the right amount of information. It is clear how screen elements (e.g., pop-ups, scrolling lists, menu options, etc.) work. I would use this site to create ABCs of care. I would recommend my colleagues to use this web application. The web application is well-suited to repeat visitors.

10


My mistakes were easy to correct.

The following statements were generally disagreed with (1=Strongly Disagree and 2 = Disagree) by participants:

The web application is designed with me in mind. The web application is relevant to my professional needs. I can quickly enter information. Screens present content in a manner that is easy to understand. I always felt I knew what was possible to do next.

The table below displays the participants’ satisfaction ratings regarding the overall usability of ABC. Low satisfaction ratings are highlighted in red.

Survey Question

Participant A Participant B Participant C

1It is easy to find my way around the site.

2 4 4

2I can quickly enter information.

2 3 4

3It is easy to remember where to find things.

2 4 5

4Screens present content in a manner that is easy to understand.

1 3 5

5Information is written in a style that suits me.

2 4 5

6Screens contain the right amount

2 5 5

11


of information.7It is clear how screen elements (e.g., pop-ups, scrolling lists, menu options, etc.) work.

1 5 4

8The site is designed with me in mind.

1 2 4

9The site is relevant to my professional needs.

1 2 5

10I would use this site to create an ABC.

1 5 5

11I would recommend my colleagues to use this web app.

N/A 4 5

12The web app is well-suited to first time visitors.

1 4 3

13The web app is well-suited to repeat visitors.

N/A 4 5

14I always felt I knew what was possible to do next.

1 3 4

15My mistakes were easy to

3 4 5

12


correct.Table 7: Participant Survey Responses

Overall Questionnaire ResponsesThe usability expert (acting as the Test Facilitator) administered an open-ended post-test questionnaire regarding the participant’s overall impressions of ABC upon the completion of the final test scenario. The post-test questionnaire (See Attachment I) asks the following questions:

1) What are your overall impressions of the ABC module?2) What are the three things you like best about this website?3) What are the three things you like the least about the ABC module?4) Is the ABC module a tool that your company might invest in? 5) Would you prefer to see the ABC module made more flexible?

If yes, which components seemed too rigid?6) Would you prefer to see the ABC module made more pre-configured?

If yes, which components seemed too flexible?7) Is there anything else you would like added to the ABC module?

The following table displays the participant’s responses to the questionnaire:Question Participant A Participant B Participant C

1 The tool is not intuitive or easy to use. In depth step-by-step training would be mandatory, A data dictionary would be helpful.

I want to see the costing aspect of the program. I want to know how it applies to a real life scenario that we have now. (DRG 469/470-need to identify patients when their joint replacement is scheduled.)

Laid out pretty clean

2 Organized well, easy to read.

Being able to quickly apply inclusion and exclusion criteria to identify a patient population.

Allows for a good definition for this group of patients not available in the normal reporting tools

3 Too much clicking required,

Want more info about costs. Not

Could quickly become

13


not intuitive, too much navigation required to input one ABC.

sure how Seton will have access to post-acute care not available in DSS.

cluttered with the multiple definitions of different types of ABCs

4 I can’t think of a department/FTE that would currently be responsible for inputting the data. I would be interested in seeing the analytics and reporting features.

Yes, if the cost elements are connected. Need connection to post-acute care in DSS.

Possible

5 Creating new inputs/ABCs.

We only looked at model ABC, so that wasn’t enough overview to answer this question.

Skip

6 No. No! Give us the flexibility to design reports.

Skip

7 Code descriptors. Not sure, I didn’t get a full tour.

Skip

Table 8: Participant Questionnaire Responses

The Usability Expert analyzed written observational notes from each usability test session and the session video clips through a Bottom-up method called affinity matching. Affinity matching is the process of reviewing test data to identify findings, then similar findings are grouped together, and then groupings are given category names. The primary advantage of using an affinity matching process to organize findings is that the usability expert stays open to whatever the process will uncover. The affinity matching process also allows the usability expert to realize trends across the data of all participants. Please see Attachment L for a photograph of the affinity matching results for this usability study.The use of the affinity matching process and error logs in the analysis phase of the usability study led the usability expert to identify what participants

14


liked the most and least about the ABC module. The following sections will outline these findings.Liked MostThe following comments capture what the participants liked most:

Participant A Explanation Comment “That [creating a new

parent ABC folder] was more intuitive.”

The participant is comparing the creation of a folder to the naming and saving of a new ABC.

Table 9: Participant A Liked Most

Participant B Explanation Comment “Medical client is very clear

[as the DSS vendor} …”The participant is evaluating the homepage.

Comment “Oh, I know how to do this now—you want to create an ABC folder.”

The user creates the ABC folder with no errors because she had tried to create a new ABC model using the new folder button in the previous activity (create a new ABC model) and she recalls the button’s purpose.

Comment “Inclusion and exclusion [terminology] definitely makes sense.”

The participant understand the terminology.

Table 10: Participant B Liked Most

Participant C Explanation Comment “I probably could have done

these as save and put in a new one.”

The participant realizes, after the fact, that the “Save & Create New” button would have been beneficial for him to use while creating new services.

Table 11: Participant C Liked Most

Liked LeastThe following comments capture what the participants liked least:

Participant A Explanation Comment “So that wasn’t intuitive at

all…I wouldn’t know what that terminology—ABC Model—meant…just leave

The user is unfamiliar with the terminology of an ABC Model.

15


out model altogether and just have ‘New ABC’.”

Comment “It’s still not giving me the option…it doesn’t say ‘Create New Folder” or anything like that, right?”

The participant is trying to create a child folder.

Comment “Okay, so right clicking doesn’t work with this [creating a child folder].”

The participant expects a right click function to allow her the option of creating a child folder.

Comment “Can you rename folders?” The participant wants the user control and freedom to be able to rename ABC folders.

Comment “…Instead of having to search through the dropdown and then have to choose it [the correct service code]…if you could just type it in…I don’t know which one I’m supposed to choose.”

The user does not find the list of service codes generated in a drop-down to be intuitive.

Comment “Usually if you click once on these it will show up.”

The participant clicked once on the appropriate MS-DRG code in the Advanced Criteria, but the code did not populate because the single click action is not supported. The user prefers a single click to a double click.

Comment “So, there’s no in between [operator]? I was going to look for that.”

The participant is trying to enter an MS-DRG range in Advanced Criteria.

Comment “And how do I know these [services] are all being saved to the same ABC?”

The system does not inform users that their services have been successfully saved.

Comment “I don’t like that it [the ABC module] doesn’t really give you any more insight other than that number [The Service Code that displays as users add services to an ABC]. I want to know what that number is associated to, so it should have a description as well—where

The participant is reviewing services she added to the ABC and is unsure which service is which.

16


you’re selecting.”Comment “So they’re not there?” The participant is clicking

through the service categories under general information, expecting her newly created services to already be populated.

Comment “Yeah, I have no idea what I’m supposed to be looking for here.”

The participant is scrolling through the list of available services, needing to add the trigger she created to her ABC.

Comment “I’m completely lost.” The participant continues to try to locate the services she created to add them to the ABC and is very confused.

Comment “It [the non-trigger service] wouldn’t go here…That’s not intuitive at all and it doesn’t make sense.”

The usability expert assisted the participant in locating services to add to the ABC, now the participant is attempting to add all services under the Trigger category, even if they are not triggers. The usability expert explains that each service must be added under its corresponding category. The participant does not understand why the system is setup this way.

Comment “It’s a lot of clicking and going through different sections.”

The participant is referring to the process of creating a new ABC in the ABC model.

Table 12: Participant A Liked Least

Participant B Explanation Comment “So are we creating a new

folder?”The test facilitator just reminded the participant that she is being asked to create a new ABC model and the participant clicks the New button in the screen’s left pane. The New Folder pop-up is displayed.

Comment “I am not a coder…This is not fair—this is not my area

The participant does not like the idea of working with operators

17


of expertise.” in creating a service with a range—this language is unfamiliar to her.

Comment “in between” or “min max” The participant states she is looking for these types of operators to create the MS-DRG range.

Comment “Okay, you’re gonna have to help me…I do not code. I do not know SQL.”

The participant is very frustrated with trying to create new services to add to an ABC model. The participant’s tone of voice harshens and she begin shaking her hands and head.

Comment “The trigger part did not make sense to me.”

Trigger terminology is unfamiliar to the participant.

Comment “It’s weird that you can’t see it…Well I wanna review ‘X’…and there’s no way to double click…to be able to drill in and see which one [full name of service has been selected]…”

The participant is adding services to the ABC model.

Table 13: Participant B Liked Least

Participant C Explanation Comment “Where it says Costing—I

don’t like that shade of green—it’s hard to read…The Reporting could also use a little more contrast.”

The participant is referring to the homepage module bubbles.

Comment “Alright, where are you hiding the DRG code?”

The participant is attempting to create new services to add to his ABC model.

Comment “I don’t see Ken [services].” The participant is in General Information of his ABC model searching through services to locate the services he just created and add them to his ABC model, however the services are not available—the services were not saved by the system properly.

Table 14: Participant C Liked Least

18


RecommendationsThe Recommendations section provides recommended changes and justifications driven by the participant success rate, behaviors, and comments. Every recommendation includes a severity rating. The severity rating scale will be as follows:Critical If we do not fix this, users will not be able to complete the

scenario.Serious Many users will be frustrated if we do not fix this; they may

give up.Minor Users are annoyed, but this does not keep them from

completing the scenario. This should be revisited later.Table 15: Recommendations Severity Rating Scale

The following recommendations will improve the ABC module’s overall ease of use and address the areas where participants experienced problems or found the interface, information architecture, or language to be unclear.Test Scenario 1: Create a New ABC Test Scenario 1 requires participants to locate the button to create a new ABC (Test 1) and to name and save the new ABC (Test 2).Change Justification SeverityClearly label the New button used to create new ABCs as “New ABC.”

The New buttons for creating a new ABC and creating a new folder look exactly the same and are displayed on the same screen, causing user confusion.

The following comment captures the participant’s confusion with the New buttons and is made by Participant B while attempting to create a new ABC model:

“So are we creating a new folder?”

Serious

Change the primary navigation label from ABC Model to “ABC Library.”

All three participants looked through ABC Data Maintenance or Data Maintenance first, when asked to create a new ABC model.

Serious

19


The terminology of ABC model is not familiar to the users.

Participant A stated, “So that wasn’t intuitive at all…I wouldn’t know what that terminology—ABC Model—meant…just leave out model altogether and just have ‘New ABC’.”

Table 16: Test Scenario 1 Recommendations

Test Scenario 2: Create a New Parent ABC Folder Test Scenario 2 requires participants to locate the button to create a new ABC folder (Test 1) and name and save the new ABC folder (Test 2).Change Justification SeverityClearly label the New button used to create new ABCs as “New Folder.”

The New buttons for creating a new folder and creating a new ABC look exactly the same and are displayed on the same screen, causing user confusion.

The following comment captures the participants’ confusion with creating new folders:

“It’s still not giving me the option…it doesn’t say ‘Create New Folder” or anything like that, right?” - Participant A

Participant C was not able to complete this Test Scenario. Participant C wrongly assumed he successfully completed this test scenario by creating a new ABC model.

Serious

Table 17: Test Scenario 2 Recommendation

Test Scenario 3: Create a New Child ABC FolderTest Scenario 3 requires participants to locate the parent folder they created in Test Scenario 2 (Test 1) and create a new folder and indicate it as the child folder (Test 2).

20


Change Justification SeverityRemove the blue arrows in the left pane used to change folder hierarchy.

Two out of three participants intuitively used the drag and drop feature to rearrange folders and change folder hierarchy.

Minor

Provide user control and freedom by allowing users to rename existing folders.

Participant A specifically asks if she can rename a folder: “Can you rename folders?”

Minor


Test Scenario 4: Build a New ABC Test Scenario 4 requires participants to create new trigger, post-admission, acute, and post-discharge services (Test 1), then add the services to the ABC created in Test Scenario 1 (Test 2), and specify each service as inclusions or exclusions (Test 3).Change Justification SeverityAdd an “in between” operator for users to use to create a range in Advanced Criteria.

Two out of three participants were confused and frustrated by the operators in Advanced Criteria.

“I am not a coder…This is not fair—this is not my area of expertise.” – Participant B

Two out of three participants stated that they were searching for an in between operator to allow them to create a range.

“So, there’s no in between [operator]? I was going to look for that.” –Participant A

The Test Facilitator asks Participant B what type of operator she is searching for and she responds, “in between”

Serious

21


or “min max.”Improve visibility of system status by displaying a message informing users that the service they created has successfully saved.

Participants were confused about if and where their newly created services were being saved.

Serious

Improve user control and freedom by allowing the user to type in the service code, rather than having a drop-down with an overwhelming number of options to choose from display.

All three participants required assistance when selecting which service code to choose from the drop-down.

The following comment illustrates the user experience with this feature:

“…Instead of having to search through the dropdown and then have to choose it [the correct service code]…if you could just type it in…I don’t know which one I’m supposed to choose.” – Participant A

Serious

Improve flexibility and efficiency of use in the Filter function. Allow users the option of typing in only partial information into the search to generate all options with matching partial data.

Participant A attempted to use the Filter when adding services to the ABC. She knew all of her service names contained her first name, so she set the Filter to “Name” “is” “equal to” and typed in her name in “Value.” No results were returned because she did not enter the entire service name. The participant became frustrated with the Filter and required assistance in understanding how to locate her services through the paging tool.

Minor

Allow users the option to save new services directly to their ABC upon the creation of

All three participants required guidance regarding where to add services to an ABC model.

Serious

22


the service. Two out of three participants assumed the services were being saved to the ABC model as they were created.

Two out of three participants expressed that they were unsure of where each new ABC was being stored upon creation.

Only generate a list of services that the specific user has created when creating services and building ABCs.

All three participants scrolled through the list of existing services, searching for the service they needed, before attempting to create a new service.

All three users were overwhelmed by the large list of services to scroll through when searching for the services they needed to add to the ABC model.

The following survey response and comment express the users’ experiences with the long list of services:

“Could quickly become cluttered…” – Participant C

“So do I really have to go through all of these?” – Participant A

Minor


Test Scenario 5: Add and Remove ServicesTest Scenario 5 requires participants to remove an existing inclusion service from the ABC built in Test Scenario 4 (Test 1) and add a new inclusion service to the ABC built in Test Scenario 4 (Test 2). Change Justification SeverityAllow users the ability Two participants assumed that Minor

23


to specify if a service is an inclusion or an exclusion upon its creation.

services would be pre-populated in the ABC’s General Information.

The following comments capture the users’ experiences with adding services to the ABC:

“So they’re not there?” – Participant A

“It’s a lot of clicking and going through different sections.” – Participant A

Provide a full description of each service when the user adds it to an ABC.

Participants wanted more information on each service they added to an ABC to be displayed so they can easily identify which service is which.

The following comments capture the user’s need for more information when adding services to ABCs:

“I don’t like that it [the ABC module] doesn’t really give you any more insight other than that number [The Service Code that displays as users add services to an ABC]. I want to know what that number is associated to, so it should have a description as well—where you’re selecting.” – Participant A

“It’s weird that you can’t see it…Well I wanna review ‘X’…and there’s no way to double click…to be able to drill in and see which one [full name of service has been selected]…” – Participant B

Minor


24


General Recommendations Change Justification SeverityChoose a darker shade of green or completely different color for the Costing module bubble on the homepage.

Two out of three participants stated that the text in the Costing bubble was not readable due to the light green background color and the white text.

Minor

Choose a darker shade of blue or completely different color for the Reporting module bubble on the homepage.

Participant C stated the Reporting module bubble was difficult to read due to the light blue background and white text color contrast.

Minor

Employ a single click functionality in instances that require a double click, such as creating a new service in Advanced Criteria when selecting a service code.

Two out of three participants attempted to open items or populate data by using a single click.

The following comment shows the users’ expectations of the single click function:

“Usually if you click once on these it will show up.”

Minor

Table 21: General Recommendations

Conclusion In general, the participants found the homepage design and layout to be aesthetically pleasing and well organized. Participants had little difficulty using the primary DSS navigation. Changes to specific language used in the ABC module, such as ABC Model, triggers, and operators in Advanced Criteria of Services will create a more intuitive user experience. Ridding the ABC module of identical buttons performing different tasks (New folder button and New ABC button) and clearly labeling buttons will eliminate user confusion. Focusing on creating a more fluid experience when building an ABC model will limit user frustration and foster a pleasant and intuitive user experience. Implementing the recommendations of this usability report and continuing to work directly with ABC model target users will ensure a continued user-centered system.

25


Appendix

Attachment A: Participant Consent Form

26


Attachment B: Test Scenario 1, Participant Handout

Attachment C: Test Scenario 2, Participant Handout

27


Attachment D: Test Scenario 3, Participant Handout

Attachment E: Test Scenario 4 Part 1, Participant Handout

28


Attachment F: Test Scenario 4 Part 2, Participant Handout

Attachment G: Test Scenario 5, Participant Handout

29


Attachment H: Post-test Survey

30


Attachment I: Post-test Questionnaire

31


Attachment J: Error Log Template

32


Attachment K: Error Log Code Definitions

33


Attachment L: Affinity Matching, Bottom-up Analysis Method

34

samanthamannportfolio.files.wordpress.com€¦ · web viewthe focus of these usability test...

Documents