kick start on accessibility and usability testing

8/3/2019 Kick Start on Accessibility and Usability Testing

1/26

Kick Start on Accessibility and Usability

Testing

Abstract: This course gives the fundamentals of accessibility, how to evaluate accessibility(Importance

of Comprehensive Accessibility Evaluation, Standards Review, Heuristic Evaluation, Design

Walkthroughs, Screening Techniques and Usability Testing) ,brief focus on usability testing and how to

understand the difference between accessibility and general usability.

CONTENTS:

1. Accessibility

1.1. What is Accessibility?

1.2. Evaluating for Accessibility

1.2.1. Importance of Comprehensive Accessibility Evaluation

1.2.2. Standards Review

1.2.3. Heuristic Evaluation

1.2.4. Design Walkthroughs

1.2.5. Screening Techniques

1.2.6. Usability Testing

2. Usability Testing

2.1. Planning Usability Testing

2.1.1. Determining Participant Characteristics

2.1.2. Recruiting Participants with Disabilities

2.1.3. Choosing the Best Location

2.1.4. Scheduling the Right Amount of Time

2.2. Preparing for Usability Testing

2.2.1. Ensuring the Facility is Accessible

2.2.2. Preparing Test Materials

2.2.3. Setting Up and Testing the Participants' Configurations

2.2.4. Becoming Familiar with the Assistive Technology

2.2.5. Conducting Pilot Testing

2.3. Conducting Usability Testing

2.3.1. Setting up the Room

2.3.2. Orienting the Participant


2/26

2.3.3. Completing Paperwork

2.3.4. Completing the Tasks

2.3.5. Collecting Data

2.3.6. Providing Compensation

2.4. Reporting Usability Testing

2.5. Checklist for Usability Testing

2.6. Recruiting Screener

3. Understand the difference between accessibility and general usability

3.1.1. Usability Problems

3.1.2. Accessibility Problems

3.2. Distinguish between usability and accessibility issues, as appropriate


3/26

1.1. What is Accessibility?

Accessibility basically means that people with disabilities can use a product. More specifically,

accessibility is making user interfaces perceivable, operable, and understandable for people with a wide

range of abilities. It encompasses all disabilities, or functional limitations, including visual, auditory,

physical, speech, cognitive, and neurological disabilities. This includes temporary conditions, such as

when you break your arm, or lose your glasses.

Accessibility also makes products more usable by people in a wide range of situations. Situational

limitations come from circumstances, environments, and conditions, and can affect anybodythat is,

people withoutdisabilities as well. For example, situational limitations include using the Web on a mobile

phone when your eyes are busy (such as driving), in bright sunlight, in a dark room, when your hands are

full, in a quiet environment (where you don't want it to make noise), in a noisy environment (where you

can't hear well), and in an emergency (when you may not be thinking clearly).

Thus, while access to people with disabilities is the primary focus of accessibility, it also benefits

people without disabilities and organizations that develop accessible products because designing for

functional limitations overlaps with designing for situational limitations.

1.2. Evaluating for Accessibility

Information on incorporating accessibility into the following evaluation methods:



1.2.3. Heuristic Evaluation





Accessibility evaluation is often limited to assessing conformance to accessibility standards.

Conformance to accessibility standards is important: in some cases it's a legal requirement and in others


4/26

it's just a good way to help check that you've adequately covered the range of accessibility issues.

However, when the focus is only on the technical aspects of accessibility, the human interaction aspect

can be lost. Usability evaluation methods can assess usable accessibility to ensure that your accessibility

solutions are usable by people with disabilities.

Effective accessibility evaluation includes both evaluation expertise and the experience of people

with disabilities. If we have people with disabilities easily available to help with evaluation, such as

employees in the same building, we probably want to do lots of informal evaluation with them on early

design prototypes. In the more common case where it takes more effort to get people with disabilities for

evaluation, we probably want to employ the other evaluation methods first.

If we have limited budget, we might need to do the evaluations ourself, or we might be able to

afford an accessibility specialist. An accessibility expert with first-hand experience of how people with

different disabilities interact with products can:

Evaluate accessibility issues for a broad range of users, which might not be found

by a few individual users in usability testing;

Help fix any known accessibility barriers before bringing in users; and

Focus usability testing or informal evaluation with users on potential areas of

concern.

While each evaluation plan will be different based on resources and other factors, ensure that you

employ comprehensive evaluation that includes at least a little of the methods described next: standardsreview, heuristic evaluation, design walkthroughs, screening techniques, and usability testing.


A standards review in the User-Centered Design process assesses whether a product conforms to

specified interface design standard. Sometimes the standards are internal style guide recommendations,

and other times they are external standards.

Accessibility standards and guidelines are available from international standards organizations;

national, state and local governments; industry groups; and individual organizations. Accessibility

standards reviews are often more rigorous than typical user interface reviews, especially when

conformance to a standard is a legal requirement. Furthermore, user interface issues often overlap with

technical issues in accessibility standards reviews.

1.2.3.Heuristic Evaluation


5/26

In a heuristic evaluation, specialists judge whether each design element conforms to established

usability principles. To conduct a heuristic evaluation for accessibility, accessibility specialists judge

whether design elements conform to accessibility principles.


The purpose of a design walkthrough is to find potential usability problems by envisioning the

user's route through an early concept or prototype. Typically, a person acts as a representative user while a

design team member guides her through actual tasks with early prototypes. Sometimes another team

member plays the computer or device, changing paper mockups of windows, drop-down menus, pop-up

dialog boxes, and other interface elements.

Ways to incorporate accessibility into design walkthroughs include:

Focus on specific accessibility issues during regular walkthroughs.

Conduct walkthroughs specifically for accessibility.

An example of focusing on specific accessibility issues during regular software walkthroughs is

device-independent interaction. The design team listens for the acting user to say, I would click on this,

indicating an action that is completed with a mouse. The team then checks that all actions triggered with a

mouse are also available through the keyboard for people who don't use pointing devices. Another

example of a specific accessibility issue to evaluate during design walkthroughs is use of sound. When

walking through use of a consumer product, the design team listens for the team member playing the

device to indicate feedback or interaction provided via sound.

To conduct walkthroughs specifically for accessibility, use persons with disabilities and scenarios

that include adaptive strategies to complete the task. For example, the acting user would be blind and

another design team member would play the role of the screen reader.

For design walkthroughs of high-fidelity prototypes we can also use Screening Techniques.


Screening Techniques are simple, inexpensive activities to help identify potential accessibility

barriers in product designs. Design teams use screening techniques to learn about accessibility issues and

to evaluate prototypes or existing products. Screening techniques save time and money by finding barriers

early, when it is less expensive to make changes to the product, and by focusing later usability testing

with people with disabilities.


6/26


Usability testing provides quantitative and qualitative data from real users performing real tasks

with a product. Usability professionals can evaluate some aspects of accessibility by using standard

usability testing protocols, with a few modifications for including participants with disabilities. While

usability testing is useful for learning how people use our products and assessing the usability of

accessibility solutions, it does not evaluate conformance to accessibility standards.

We don't have to be usability professional and we don't have to follow formal usability testing

protocols to include people with disabilities in evaluation. Short informal evaluation can gather valuable

feedback from people with disabilities without the rigor of formal usability testing. In most cases,

including users in evaluation involves:

Finding a few people with disabilities,

Asking them to complete tasks on prototypes,

Observing them interact with the prototype,

Discussing accessibility issues with them.

While usability testing evaluates how usable accessibility solutions are by some people with

disabilities, stability testing can't address all accessibility issues and doesn't evaluate conformance to

accessibility standards.

The following sections discuss usability testing with participants with disabilities:

2. Usability Testing

2.1 Planning Usability Testing

2.2 Preparing for Usability Testing

2.3 Conducting Usability Testing

2.4 Reporting Usability Testing

2.5 Checklist for Usability Testing

2.6 Recruiting Screener

Planning Usability Testing covers determining participant characteristics, recruiting participants, choosing

the best location, and scheduling the right amount of time.

Preparing for Usability Testing covers ensuring the facility is accessible, preparing test materials, setting

up and testing participants' configurations, becoming familiar with the assistive technology, and

conducting pilot testing.


7/26

Conducting Usability Testing covers setting up the room, orienting the participant, completing paperwork,

completing the tasks, collecting data, providing compensation, and specific considerations for people with

different disabilities.

Reporting Usability Testing covers distinguishing between accessibility and usability issues, including

relevant study parameters, being careful about categorizations and comparisons, clarifying conclusions,

and writing about people with disabilities.

Checklist for Usability Testing summarizes the tasks and considerations involved in planning, preparing

for, conducting, and reporting usability studies with participants with disabilities.

Recruiting Screener lists question to ask during recruiting.

2.1. Planning Usability Testing

Planning a usability study that includes people with disabilities involves the following

considerations:



2.1.3. Choosing the Best Location2.1.4. Scheduling the Right Amount of Time


There is no definitive answer to the questions, "How many participants with disabilities should be

included in usability testing?" and "What characteristics should they have?" It depends. This section

explains factors that can help answer those questions for our particular situation.

Understanding the challenge

The number of participants to include in any usability test is a subject of debate among usability

professionals. Some studies have led to the conclusion that testing a large number of participants does not

yield significantly more information than testing fewer participants, since the first participants will find

most of the usability problems. Some research shows that five participants is sufficient to find 85% of the

usability issues when you have comparable users who will be using the product in fairly similar ways.

When we have several highly distinct groups of users, we need to test additional users.

People with disabilities use products differently, and so a thorough usability test for accessibility


8/26

requires more than five users. (The next section includes guidance for using fewer than five participants

with disabilities.) When we have different categories of users, one recommendation is to include three

users from each category. However, people with disabilities do not fit easily into categories in terms of

product interaction.

Careful with categorization

Disabilities are sometimes grouped into four high-level categories: visual, auditory, physical,

cognitive, yet there is vast variability within each category. For example, people with visual disabilities

include a middle-aged woman who has low vision since birth and is very experienced with screen

magnification software, a young man who recently went totally blind from retinitis pimentos and is a

new screen reader user, an elderly woman whose sight is deteriorating from macular degeneration yet she

doesn't use any assistive technology, and a young boy whose color blindness has not yet been diagnosed.

This variability within categories is significant in product design and evaluation. For example,

although visual disabilities are often categorized together, a product can be accessible and usable for a

person who is blind, and yet totally inaccessible and unusable for a person with low vision, and vice

versa.

How a person interacts with products is also impacted by differences such as: whether the person

was born with the disability, acquired the disability at a young age, or acquired it when older; whether the

disability is temporary, permanent, static, progressive, or regressive; and what adaptive strategies and

assistive technology the person uses. Also note that it is common for people to have multiple disabilities

across more than one category.

Because of this variability within a category, the four common categories of disability (visual,

auditory, physical, cognitive) are not sufficient for categorizing usability study participant characteristics.

Identifying a realistic range of participants

Projects rarely have the time and money resources to do thorough usability testing with a wide

range of participants with disabilities. The number of usability test participants with disabilities included

in a given usability test is usually determined by limited project resources. The following considerationscan help us make the most of limited resources and focus on which participant characteristics to include

in our usability tests.

Understand the overlap between accessibility and usability.

In deciding the number of participants with disabilities to include, consider that users with


9/26

disabilities also address general usability issues that impact all users, including users withoutdisabilities.

Therefore, usability testing with participants with disabilities will identify both accessibility issues and

general usability issues. In many cases general usability issues are amplified when testing with

participants with disabilities, making it easier to find issues that impact all users.

Explaining how users with disabilities also address general usability can help us get more time

and money budgeted to include users with disabilities in usability testing and throughout our project.

Screening techniques can effectively supplement testing with people with disabilities in some

cases. For example, anyone can test using a cell phone with one hand; we dont have to have a disability

to do that. However, some aspects cant be effectively evaluated with screening techniques. For example,

if we dont have much experience with screen readers, we wont be able to effectively test our product

with one.

Include different characteristics in different usability tests. Ideally, product designs go

through several iterations of usability testing. In some cases we can include participants with different

characteristics in each round of testing, rather than trying to fit participants representing all characteristics

into every test.

Focus on the target users. If our target users include a higher percentage of people with a certain

disability, focus on those relevant characteristics. For example, a usability test for a product that is

marketed primarily to seniors should include seniors who have age-related disabilities; and a test for a

website containing information on diabetes should include people with visual disabilities.

Focus on highest impact. Some products will impact people with a specific disability more than people

with another disability. You might not need to include participants with certain characteristics because of

the nature of the product or the situation. For example, if we are designing a product that produces no

sound, we might choose not to actively recruit participants who are hard of hearing; if we are designing

an intranet (internal website) and the company standard is a specific screen reader, we might choose not

to include participants who use other screen readers.


Often usability specialists think it will be difficult to recruit people with disabilities, yet after a

little bit of effort, it turns out to be fairly easy.

Plan for additional recruiting With any usability test, the more specific the participant

requirements, the longer it takes to recruit. For example, recruiting doctors age 35-55 who use Macs takes


10/26

longer than if the only requirement is that participants are age 35-55. The same is true of recruiting people

with disabilities. Adding specific participant requirements, such as type of disability and use of assistive

technology, will likely add time and effort for recruiting.

On the other hand, our recruiting could take less time than usual because of "viral marketing"within disability communities.

Make key contacts. Places to look for participants with disabilities include the following:

Organizations for specific disabilities or conditions

Cross-disability organizations

Mailing lists

College and university programs for students with disabilities

Local disability-related support groups Local or regional government rehabilitation or disability services departments

Seniors organizations and local senior centers

Independent living organizations

Consider pilot tests as a recruiting tool. Some people with disabilities are part of networks that actively

share accessibility information. Word of a positive usability testing experience can spread rapidly and

result in potential participants contacting you.

To take advantage of this word-of-mouth recruiting, consider conducting pilot tests early in your

recruiting efforts and seek out participants who are well-respected, vocal members of a community, such

as a student organization, senior center, or disability support group. Encourage them to spread the word of

our recruiting.

Add relevant information to recruiting screener. When screening participants with disabilities for a

usability test, use the same parameters as we would for participants without disabilities. Additional

information to cover in the recruiting screener includes technology use, alternative formats for printed

materials, etc. Specific questions to add to the screener are included in theRecruiting Screenersection.

Arrange for interpreters as needed. If we are conducting usability studies with participants who are

deaf, it's customary for us to find, schedule, and pay for sign language interpreters. (This is usually not the

case when a person with a physical disability has a personal assistant; they will make the arrangements.)

Plan to reimburse participants for necessary expenses. Although the basic compensation for people

with disabilities is the same as for any participant in your test, there might be some additional cost
http://www.uiaccess.com/accessucd/ut_ppt-screen.htmlhttp://www.uiaccess.com/accessucd/ut_ppt-screen.htmlhttp://www.uiaccess.com/accessucd/ut_ppt-screen.htmlhttp://www.uiaccess.com/accessucd/ut_ppt-screen.html


11/26

considerations. Transportation might be more complicated and costly for participants with disabilities. We

should usually reimburse for additional transportation expenses, such as an accessible taxi van.

For some participants we might need to pay for the time of a personal assistant or interpreter. As

part of the recruitment process, find out about participant expenses and confirm with participants whichyou will reimburse.

2.1.3.Choosing the Best Location

With any usability test there are advantages and disadvantages of conducting the sessions in a

usability lab or "in the field" at the participant's work, home, or other location. Some people use remote

evaluation where the facilitators and participants are in different locations. When conducting usability

tests with people with disabilities, there are additional factors to consider when making the decision of

location.

Consider the goals of the usability test. Where the best location is depends partly on the goals and

potential additional benefits of the test. Most designers don't know how people with disabilities use their

products, and many people are uncomfortable around people with disabilities.

Conducting usability tests in your lab usually provides the opportunity for more of the project team to

observe, and for you to record the sessions. Informal sessions with lots of interaction between designers

and participants are especially useful, and can be conducted wherever is most convenient for the project

team, rather than in a formal lab.

When the goal is for a couple of people to learn more about how people use the product in their

own environment, field studies are best.

When we want face-to-face interaction, remote evaluation is not an option. Additionally, remote

evaluation doesn't provide equivalent results as in-person usability testing. However, there are cases

where remote evaluation is best. For example, if we already have an established relationship with a

person who helps us evaluate for accessibility, it may be equally effective and much easier to do some

evaluation remotely rather than traveling to the same location.

Consider assistive technology needs. Assistive technologies are a significant factor in where to conduct

usability tests.

Providing the required assistive technologies in a lab can be costly, complicated, and time-

consuming. When testing software or web-based products, participants might require different versions of

assistive technologies with different configurations. Changing system configurations between participants


12/26

can be difficult.

It is common for an information technology specialist, rehabilitation specialist, or family member

to set up a person's home or office computer system, including assistive technologies. Some assistive

technologies have many settings; for example, screen magnification software provides multiple optionsfor zoom, color, contrast, cursor, and more. Participants might not know how to configure assistive

technologies in your lab to work as they are used to them working in their regular system.

Take into account transportation. Transportation may be more difficult for some participants with

disabilities. Also, accessible transportation is often unreliable and participants using such transportation

might not be able to meet a tight schedule in a usability lab.

Evaluate the accessibility of potential locations. Many buildings that claim to be accessible could have

barriers that make it difficult for people with disabilities to participate in the test. For example, one sitethat was technically wheelchair accessible had very thick carpeting that made access with a manual

wheelchair difficult. "Ensuring the Facility is Accessible" in the Preparing for Usability Testingsection

provides specific points to consider regarding the location's accessibility.

2.1.4. Scheduling the Right Amount of Time

Use pilot tests to work out timing. Plan extra time for the pilot tests and record how long each step

takes.

In one of our the studies with people with disabilities, they planned to have participants complete the

consent form, web expertise survey, financial knowledge assessment, and post-study survey online.

However, their first pilot participant with low vision took almost 1 hour of the planned 1.5 hour session to

complete the online forms. Thus, they dropped the web and financial surveys from the study.

Plan based on specific disability considerations. In some cases, the time for each usability test

participant session will be impacted by a participant's disability and longer or shorter sessions may be

more effective. For example, people with some physical disabilities, such as tremor and poor fine motor

control, might take longer to do basic task steps such as activate a button, and a person with a cognitive

disability might need longer to process textual information and instructions. It will take some people with

disabilities longer to complete tasks, especially if the product is not highly accessible or if they haven't

used products like it before.

You may need to schedule additional time between sessions as well. It may take some participants

longer to complete pre-test paperwork, get in and out of the testing room, and take breaks. You might also


13/26

want to plan for an additional staff person to be available, for example, to help escort participants around

the facility.

Be aware of energy level considerations. Fatigue tends to be more of an issue with some participants

with disabilities due to factors such as the disability itself, medications, and the extra effort required to

use assistive technology. Participants might need breaks and might not be comfortable or effective in long

sessions. Ask participants during recruiting about any time and fatigue considerations, and plan session

timing accordingly. A sample question is included in theRecruiting Screener.

In one of the studies, a participant using a screen magnifier told that he gets nauseous after a half-hour

of watching the screen and would need a short break every 30 minutes.

On the other hand, some participants with disabilities have high energy levels and will be

effective longer than some people withoutdisabilities. Some are used to taking longer to accomplish tasks

and have more patience and more determination to complete a task successfully. They might want extra

time.

Schedule time to confirm assistive technology setup. Plan time at the beginning of each test session for

participants to check that any assistive technologies are set up and configured as they want.

Plan time for the participant to become familiar with the product. When the usability goals are for

people somewhat familiar with the product, the usability testing schedule might include time for the

participant to interact with the product before usability testing begins.

Preparing for Usability Testingcovers ensuring the facility is accessible, preparing test materials, setting

up and testing participants' configurations, becoming familiar with the assistive technology, and

conducting pilot testing.

2.2. Preparing for Usability Testing

The Evaluating for Accessibility page provides guidance on incorporating accessibility into

common evaluation methods, including standards review, heuristic evaluation, design walkthroughs, and

informal evaluation with users with disabilities. The Usability Testingsection is an overview of usability

testing with participants with disabilities.

Preparing for a usability test that includes people with disabilities involves the following considerations:
http://www.uiaccess.com/accessucd/ut_ppt-screen.htmlhttp://www.uiaccess.com/accessucd/ut_prep.htmlhttp://www.uiaccess.com/accessucd/ut_prep.htmlhttp://www.uiaccess.com/accessucd/ut_ppt-screen.htmlhttp://www.uiaccess.com/accessucd/ut_prep.html


14/26







Make sure that the usability testing facility is accessible to participants. Depending on your

participants, check things such as wheelchair access into the building, to the test room, and to the

bathroom; parking space for a van with a side wheelchair lift; and space in the room for an interpreter to

be in the best position. To help ensure that nothing is overlooked, consider the following:

Use a checklist to ensure that you have anticipated any potential barriers.

Schedule a walkthrough by a person with similar accessibility needs; for example, someone from

participant screening who didn't meet the participant criteria.

Provide enough space for a wheelchair, assistive technology, interpreter, personal attendant, or guide

dog, as needed. Consider using modular tables so that the room can be rearranged.


Write usability test materials in clear and simple language. This will be especially important for

participants with some types of cognitive disabilities who have difficulty processing information orinstructions.

Be prepared to provide all materials in alternative formats. Materials include directions to the facility,

consent form, release form, non-disclosure agreement and instructions for the participant, and tasks to be

completed during the usability test. Alternative formats include:

Large print

Braille

Electronic (HTML, plain text, or other format; via e-mail, CD, etc.)

Audio (read live, electronic file, cassette tape)

Note that some people with disabilities don't like PDF format. There are things you can do to increase the

accessibility of PDF documents for screen reader users. People with low vision cannot increase the font

and print PDF documents, so you will need to provide the PDF document with the font already in the

user's preferred font face and font size, or provide the document in a format where the user can set the


15/26

font themselves.

Plan time to have your documents "brailed" if necessary and remember that you can't make last minute

changes to your test materials if you have them brailed elsewhere. Label your Braille pages (if they get

mixed up you won't be able to sort them out unless you read Braille).

Only a small percentage of people who are blind read Braille. People who are born blind often learn

Braille people who go blind later in life usually don't.

Include alternate format questions in the recruiting screener. Questions about alternative formats to

ask participants during recruiting are included in theRecruiting Screenersection.

Send consent forms and other documents ahead of time. Sending the consent form, non-disclosure

agreement, and any other documents to participants before the test lets them read the documents ahead of

time in their own environment, with their own assistive technologies, and at their own pace. Some people

may take a long time to read documents and it helps your scheduling if this is done ahead of time. Explain

to the participant that some materials, such as the tasks, cannot be sent ahead of time and will be available

in the participant's preferred format.

Provide consent forms and other documents for interpreters and attendants. In most cases, you will

need sign language interpreters, personal attendants, and any others who accompany the participant in the

actual test to also sign consent forms, non-disclosure agreements, and other such documents.

Send materials to interpreters ahead of time. Sign language interpreters can be better prepared if they

have at least a rough script of what you will be saying. For formal studies where close translation is

important, consider scheduling a little time to work with the interpreter before the test to discuss precise

meanings.

Test materials in different formats across participants

For almost all usability tests, it is acceptable to have participants use test materials in different formats,

for example, some read tasks themselves and others hear tasks read to them. Just because one participant

chooses to have tasks read to them, doesn't mean that the facilitator should read the tasks to allparticipants. Instead, provide the tasks and other test materials in the format that is best for each

participant.


As previously mentioned in "Consider assistive technology needs in the Planning Usability
http://www.uiaccess.com/accessucd/ut_ppt-screen.htmlhttp://www.uiaccess.com/accessucd/ut_ppt-screen.html


16/26

Testingsection, providing the required configurations can be complicated. When testing software or web-

based products, participants might require different versions of different assistive technologies with

different configurations. People who don't use assistive technologies might use different system settings,

such as large fonts and alternative color schemes.

Acquire, set up, and test the assistive technologies to the participants' configurations, well in

advance of the usability test. Starting early leaves time to address complications, such as acquiring older

versions of software, getting different configurations to work on a single computer, or figuring out that

different configuration will not work on a single computer and we will need multiple computers for back-

to-back participant scheduling.


When participants will use assistive technology (AT) in a usability test, it will be more effective if

the facilitator is somewhat familiar with the AT. Otherwise, the facilitator might not be able to understand

the interaction between the participant, the AT, and the product being tested (as well as being distracted by

the novelty of the AT). Usability test observers, data analyzers, and others will also benefit from having

some familiarity with the AT used in the test.

Get experience with the assistive technology as appropriate. We can gain varying levels of experience

with AT in the following ways:

Get an introduction to and demonstration of the AT from an experienced

AT trainer or someone who regularly uses the assistive technology

perhaps someone from participant screening who didn't meet the

participant criteria. When possible, have them use the AT with a product

similar to what you'll be testing.

Practice using the assistive technology with the product.

After you are somewhat familiar with the AT, ask a regular AT user to

answer your questions and help you learn more about how the AT is used

with the product.


Pilot testing is especially important when testing with participants with disabilities. There are

more things that could go wrong or be new to the test designers and facilitators, such as problems with

assistive technologies.

Conduct pilot tests early. If we are new to usability testing with people with disabilities and if you are


17/26

planning a formal study, conduct at least a couple of pilot tests fairly early in the project so you have

plenty of time to work out any issues. If we don't have enough extra participants from our recruiting we

might be able to use someone who didn't meet all of the recruiting criteria.

As mentioned previously, we can use pilot tests to help with recruiting because often when participantswith disabilities have a good experience, they will tell others. Also, we can use pilot tests to work out

timing.

Use pilot tests to work out issues with assistive technologies. Assistive technologies behave differently

in different configurations, and users use them in many different ways. Additionally, assistive

technologies may require adaptations in our data collection. For example, some screen recording software

may conflict with some assistive technology.

Use pilot tests to work out logistics. Pilot tests help ensure that there are no unforeseen barriers ordifficulties for participants in the study. For example, in pilot test you may learn that our non-visual

directions to our lab can be improved, that we need to find a good place for a guide dog to go outside, or

that there is a step we hadn't noticed that makes the main path to the lab inaccessible to wheelchairs.

Use pilot tests to work out facilitation. We may need to adjust our facilitation, observation, and data

recording. For example, with participants who are deaf we need to mix the interpreter, and with

participants who use screen readers we probably want to record audio output.

During a test with participants using screen magnification, some observers susceptible to motion sickness

were unable to stay through the entire test because of the motion on the screen.

2.3. Conducting Usability Testing

Conducting Usability Testing covers setting up the room, orienting the participant, completing

paperwork, completing the tasks, collecting data, providing compensation, and specific considerations for

people with different disabilities.





2.3.6. Providing Compensation
http://www.uiaccess.com/accessucd/ut_conduct.htmlhttp://www.uiaccess.com/accessucd/ut_conduct.html


18/26


19/26

taking additional lighting, especially if we are videotaping.

One of the studies says that one time they were testing at a participant's house, and when the test started

there was enough daylight in the room. As the sun set it became too dark to see our test papers and notes.

Eventually they asked the participant if they could turn on the room light, at which time they discoveredthat the light bulb was burned out. Because the participant was blind, he didn't know the light wasn't

working.

Specific considerations for some participants who are deaf or hearing impaired

Provide seating for an interpreter. If an interpreter will be present, provide

room for the interpreter to sit in different positions, such as near the facilitator or

across from the participant.

Position seating for a direct line of sight between the participant and the

facilitator, for participants who speech read (also known as lip-reading).

Encourage the participant and interpreter to adjust their seating to make it easy

for them to work together.

Be sure the room is well-lit so that a participant who speech reads can easily see

the facilitator's lips and facial expressions.

Record both participant and interpreter audio, depending on the situation. If

the participant doesn't speak at all, we might want to mix only the interpreter. Ifthe participant speaks some, we probably want to mix both of them. It is usually

not necessary to include the interpreter in the videotape.

Specific considerations for some participants who have physical impairments

Allow enough room for a wheelchair to get in, move around, and be positioned at the computer,

table, or other area where the participant will interact with the product.


Encourage the participant to become familiar with the setup of any hardware that will be used in the

test, such as keyboard, mouse, and speakers. Encourage them to adjust the equipment, chair, etc. so they

are comfortable. If assistive technology will be used in the usability test, allow time for the participant to

check the settings and to change them if necessary.


20/26

Specific considerations for some participants who are blind or visually impaired

Introduce our self as we approach the participant. Don't assume that the

participant will recognize our voice. Introduce others who are with us so that the

participant knows who is in the room.

Describe the setting to the participant briefly , including the location of doors

and position of the video camera, especially if people will be coming in and out

and the video camera will be making noise.

Explain unusual noises and your activities, such as when we are beginning to

record or changing the videotape. Most video cameras make a sound when

recording starts or stops and the participant might be distracted wondering about

the sound. Tell the participant when you or others enter or leave the room.

Offer your elbow to lead the participant. Don't grab the participant's arm,

hand, or cane.

Give directions about where to be seated. Generally it is best to give verbal

directions and not physically direct a person. We can ask the participant if he

would like we to put his hand on the chair.

Specific considerations for some participants with physical impairments

Don't move mobility aids without asking. Some people who use a mobility aid,

such as a walker, might be uncomfortable if it is out of reach.

Remember seating for a personal attendant .


Provide documents in the participant's preferred format, as requested during participant

recruiting. Some people read Braille very quickly, and some read it very slowly. To help manage time,

you may want to ask participants if we can read the documents aloud and provide the Braille version for

reference. Be prepared for some people to want to use the Braille themselves and not have you read it.

Be prepared to indicate the place for signature. Some participants who are blind will have a

signature stamp. Some participants will have a signature guide, a small device that guides them when

signing a document. Be prepared to position a signature guide or straight-edged object so that a

participant who is blind or has low vision knows where to sign.


21/26

Specific considerations for some participants with physical impairments

Have a clipboard available to hold documents to be signed . Be ready to hold the consent form

and other documents in a position where the participant can easily sign it; for example, a participant with

limited upper body movement might want us to hold it up so that she can sign it with a pen in her mouth.


Some people with disabilities will be particularly eager to complete tasks without help, and might

be bothered by being stopped before they are done if they aren't expecting it. Tell participants at the

beginning of the session that you might stop tasks before they are complete. For example, we could say

something like, "When we have enough information for a task, we might go to another one, even if we

haven't completed the task." Also consider what to say when interrupting a task. Instead of, In the

interest of time I'm going to stop you there, consider something more like, Let's stop there and I'll give

you something new to do.

Be prepared to use alternative techniques for facilitating. Facilitating usability tests and focus groups

often involves subtle communication. Some facilitation techniques might not work with participants with

disabilities. For example, body language won't work with participants who are blind and with some

people with autism; and if a participant is deaf, the technique of just not answering a participant's question

during a low-interaction session won't work.

When facilitating a focus group with blind participants, it became apparent that my practiced use of eye

contact and body language to regulate the flow of discussion was of no use. I had to devise strategies for

communicating the same cues verbally.


Request screen reader speech rate according to usability test protocol. Screen reader users

usually set the reading rate fast and most people who aren't used to listening to a screen reader can't

understand it that fast. For an example of fast screen reading rate, listen to the end of the Introduction to

the Screen Readervideo. When you want to understand specific interaction with the product, you will

likely need to ask the participant to slow down the reading rate. Note that while most screen reader users

will be happy to work at a slower pace; some might be frustrated working at the slow pace for a longtime. For usability tests measuring time-on-task, the screen reader should be set to the participant's normal

rate.

Specific considerations for some participants who are deaf or hard of hearing

Face the participant while speaking, and speak at eye level. Many people who


22/26

are deaf or hard of hearing rely at least partially on speech reading and thus need

to see your lips. Remember not to look down or cover your mouth while

speaking. If the participant is seated, sit to speak to him whenever possible.

Speak clearly. Don't speak too fast.

Don't just repeat louder. Some people who use hearing aids or cochlear

implants are especially sensitive to loudness. If the participant is having trouble

hearing your words:

Try rewording what you are saying. Sometimes a person with a hearing loss

might be partially dependent on speech reading because some sounds might not

be easily heard even with a hearing aid. Since some words are easier to speech

read than others, rephrasing what you said might make it easier for the participantto understand.

Offer to write down what you are saying.

Move closer to the participant, while still observing personal space.


Consider debriefing after each taskinstead of after the entire test. This is helpful for participants who

have spent a significant amount of time on the task, seniors who might have short-term memory loss, and

participants with cognitive disabilities who have difficulty processing a large amount of information.

Specific considerations for some participants with speech impairments

Listen carefully, and ask the participant to repeat for clarification if needed . If you don't understand

the participant, ask him to repeat what he said or ask for clarification. Don't pretend that you understand if

we don't understand. Wait for the participant to finish, rather than interrupting or finishing his sentence. If

it's difficult for the participant to speak, we can repeat the part that we did understand so he only needs to

restate part of it.

Be prepared to offer to use written communication. Some participants might prefer to communicate by

typing or writing.

Be very clear that an attendant should repeat exactly what the participant said . Sometimes an

attendant or other person who is used to the participant's speech will clarify what the participant says.

That person might be inclined to expound on the participant's comments. Make sure that person knows


23/26

the importance of saying just what the participant said.

Specific considerations for some participants who are deaf or hard of hearing

Be very clear that an interpreter should repeat just what the participant signs . Clarify to the

interpreter the importance of translating as accurately as possible and not adding clarifications.

2.3.6. Providing compensation


State what the currency is as you hand it to the participant if we are paying

in cash. People with vision impairments often have a particular way to fold each

denomination so they can identify it.

Verify the spelling of the participant's name

Reporting Usability Testingcovers distinguishing between accessibility and

usability issues, including relevant study parameters, being careful about

categorizations and comparisons, clarifying conclusions, and writing about

people with disabilities.

2.4. Reporting Usability Testing

The Evaluating for Accessibility page provides guidance on incorporating accessibility into

common evaluation methods, including standards review, heuristic evaluation, design walkthroughs, and

informal evaluation with users with disabilities. The Usability Testingsection is an overview of usability

testing with participants with disabilities.

2.5. Checklist for Usability Testing

One of the most effective forms of inspection-based user testing involves the use of a "usability

checklist." Checklist-based user testing is extremely inexpensive to implement, and requires a

surprisingly small number of testers to be effective. It's also easy to schedule; it can be used at virtually

any time throughout the development cycle, from the earliest prototype screens to a full-blown Website.

Here's the basic method for employing a checklist-based user test.

Step 1: Preliminary Self-Appraisal

No author can view his or her own work with dispassion. Still, there are certain things that
http://www.uiaccess.com/accessucd/ut_report.htmlhttp://www.uiaccess.com/accessucd/ut_report.html


24/26

inevitably make for an unfriendly Web page. We can save considerable time, both for yourself and for

your evaluators, if we start with a basic sweep of our site for known usability problems. View this self-

appraisal as a preliminary step, however, and not as a substitute for user testing methods.

Step 2: Provide checklists to our testers

The more independent and autonomous your testers are, the more valuable the feedback they can

provide. A topical site will probably want to enlist the aid of volunteer testers with some interest in the

subject of the site. Corporate sites should strongly consider using agency-based temporary employees for

user testing.

One important consideration is: how many testers are enough? There's no hard-and-fast rule, but

inspection-based testing methods provide a surprisingly quick payback, even with a small number of

evaluators. Even a single tester can probably uncover the most common usability problems on your site.

And a handful (4 or 5) is more than adequate to ensure a generally reader-friendly Website.

Step 3: Provide some brief instructions

Understand that your evaluators will naturally assume that the problems they encounter in using

our site are the result of some fault on their part, rather than a flaw in the design of the site itself. It is

therefore vitally important for us to explain to your testers that we need them to make note of any

problems they encounter, regardless of what they believe the underlying cause to be.

Step 4: Leave

In formal usability experiments, the experimenter typically remains in the room to observe and

record testers' behavior. But unless we are a trained usability professional, our presence will likely as not

serve to inhibit your evaluators, and thus compromise their ability to test your site. Unless we are

planning on providing a personally-supervised guided tour of our site to all our readers, leave the tester to

the business of testing your site. Allow the tester sufficient time to test . 2.6. Recruiting Screener

A usability test participant recruiting screener is used to determine if a potential participant

matches the user characteristics defined in the usability test protocol. When recruiting participants with

disabilities, ask the usual questions about demographics, frequency of use, experience level, etc.Additionally, include questions that address the characteristics related to disability and accessibility

defined in the usability test protocol.

Distinguishing Between Accessibility and Usability Issues

In addition to finding accessibility problems, usability testing with participants with disabilities


25/26

will find general usability problems that impact all users, that is, including users withoutdisabilities.

3. Understand the difference between accessibility and general usability. There is not a clear

distinction between accessibility for people with disabilities and general usability for all. Some things are

clearly accessibility; some are clearly usability; and many things are in a gray area where accessibility

and usability overlap.

One way to start looking at the distinction between the two is to categorize interface problems:

3.1.1. Usability problems impact all users equally, regardless of ability; that is, a person with a

disability is not disadvantaged to a greater extent by usability issues than a person without a disability.

3.1.2. Accessibility problems decrease access to a product by people with disabilities. When a

person with a disability is at a disadvantage relative to a person without a disability, it is an accessibility

issue.

The distinction between usability and accessibility is especially difficult to define when

considering cognitive and language disabilities. Many of the accessibility guidelines to improve

accessibility for people with cognitive disabilities are the same as general usability guidelines. The

distinction is further blurred by the fact that features for people with disabilities benefit people without

disabilities because of situational limitations (that is, limitations from circumstance, environment, or

devicesuch as using a mobile phone in bright sunlight with one hand because you're holding a sleeping

baby with the other), and accessibility increases general usability.

Another point to cloud the distinction is usable accessibilityhow usable are accessibility

solutions. For example, if a website uses images for navigation and there's no alt text, the site is clearly

not accessible. If the site has frustratingly verbose alt text (such as "This image is a line art drawing of a

dark green magnifying glass. If we click on it, it will take us to the Search page for this Acme Company

website" instead of just "Search"), one might say that the site is technically accessible because there is alt

text; however, the alt text is so bad that the usability of the site is awful for anyone who relies on alt text.

3.2. Distinguish between usability and accessibility issues, as appropriate.

When designing products, it's rarely useful to differentiate between usability and accessibility.

However, there are times when such a distinction is important, such as when looking at discrimination

against people with disabilities and when defining specific accessibility standards. In some usability test


26/26

reporting it may be important to distinguish between accessibility and usability problems.

When usability test reports are used internally to improve the usability of the product for all users, it is

usually not necessary to distinguish between usability and accessibility issues. However, when usability

test reports make statements about accessibility, it can be vital to distinguish between usability andaccessibility issues.

There can be problems when people don't understand the issues around the distinction between usability

and accessibility.

A research study reported results on website accessibility without clearly separating general usability

issues not related to accessibility. Because usability issues were mixed with accessibility issues, the study

reported incorrect conclusions about web accessibility guidelines.

----------------------------------------------------End----------------------------------------------------------

kick start on accessibility and usability testing

Documents