ICGI Content Standards

Usability Test Results

Content Managers’ Forum

July 15, 2004

The Tests:

Tested eleven content standards Terminology

Provide description—”What would you call this?” Rank list of options by preference

Placement on the web page Identify placement on generic web page

Categorization Card sort exercise

Reaction to “typical” federal page

Generic Federal Homepage

The Test Participants:

Eleven participants Mix of races/ethnicities Variety of web experience Wide age spectrum Both genders Range of formal education

Sponsorship of Gov Web Sites

Participants shown a homepage (HHS/HUD)

What tells you it's an official govt. site?

.gov in URL 54.5%

"U.S. Dept .of…" visible 45.5%

"Dept. of/agency name" visible 18.2%

graphic logo/seal 18.2%

other graphics 9.1%

Comments

“ “Mainly checks for the .gov URL such as dot.gov and trusts it more than other sites like .com.”

“Definitely goes to .gov for government information and checks for the .gov URL.”

Link to Gov’t-wide Portal

Which label most accurately depicts: An official gateway to all government information

in the United States A portal containing links to federal and state

government sites

Ranking:

Unadjusted Adjusted

www.us.gov 1.90 2.05www.fedinfo.gov 1.91 1.91www.usa.gov 2.10 2.23www.firstgov.gov 3.89 3.82

Participants thought that “U.S.” and “fed” meant federal, and therefore, were looking for a more encompassing term that did not seem to exclude state governments.

Location

Users expected the link to be at the top of the page:

3 participants- top left menu bar 2 participants- top right menu bar 2 participants- top nav bar 2 participants- bottom of page/

footer

Comments

“Regarding the site name www.firstgov.gov – doesn’t know what this means at all.”

“Regarding the site name www.fedinfo.gov – said this doesn’t apply to states because of the fed term.”

“Suggested govinfo.gov” “Said firstgov.gov is ridiculous and looks like a typo” “Said to be consistent and put all links on the left. I

don’t want to look all over the page for it. There should be one spot where I can expect to see everything.”

About Us

Which label most accurately depicts a place where you can:

Learn who operates this web site Learn what this organization does Find the organizational structure of the group

that runs this site

Ranking

Unadjusted Adjusted

About Us 2.25 2.50

About [Organization] 2.31 2.56

Learn About Us 2.50 2.50

Who We Are 2.94 2.94

For More Information 4.50 4.50

Location

Users were split as to expecting the About Us link to be at the top or the bottom of the page.

3 participants- top left menu bar 1 participant- bottom of page 1 participant- bottom/ middle of

page 1 participant- bottom/ right

Comments

“Expects basic stuff near the top of the page and lower links to be more explanatory.”

“Said the test page looked as expected- it has links to specific organizations where you can find more. Expects to find jobs and then sees employment.”

“Said the test page looked as expected. Expects to find email not the mailing address.”

“A guide to the organization structure is great” “Would like to see contact us closer to the top. At

the bottom it makes me think they don’t want me to contact them.”

Laws and RegulationsCard LabelsGuidance Policy Current Legislation in

Congress Major Laws & Statutes Executive Orders Proposed Rules Regulatory Actions U.S. Code Federal Register Documents Preambles to Final Rules Dockets Testimony

Tester Recommendations Legislative Rulings Legal/ Legislation Building a Law from Start to

Finish Steps to Creating a Law Information Reference/

Resources Rules and Regulations Governmental Procedures

Ranking

Laws and Regulations 1.71

Policies and Regulations 2.14

Policy 3.00

Laws and Regulatory Assistance 3.14

The majority of the participants (90%) thought “passed and signed laws are different than proposals, so they would not be grouped together” or “these terms should be arranged in a developmental way.”

Location

The majority though the Laws and Regs link belonged at the top of the page

4 participants- top left nav bar 2 participants- top right nav bar 2 participants- middle left menu

bar 2 participants- bottom of page/

footer

Comments “Divided cards into two groups. From the two groups in the card

sort would call them Preamble (for future laws) and Statutes (for current laws).”

“Said they should be arranged in the order of the process of becoming a law.”

“Thought these looked like the steps to creating a law.” “Sees the list in a developmental sense such that proposed rules

come before laws. Would arrange the list into three categories and didn’t think “Guidance” fit in the group.”

“Said terms were related but didn’t all fit together. Saw a difference between the concrete laws and other terms.”

“Said the information did not fit together and should be broken into categories: 1. Agency Legislation in Process, 2. Historical Source of Information for Legislation, 3. Executive Orders, 4. Policy & Guidance.”

Search

Which label most accurately depicts a place where you can: Enter terms in order to find content most relevant to your needs Scan a website using keywords to find matching information Use single words or phrases to quickly locate information within

an agency's web site

6 out of 10 (60%) of the participants came up with at least one label suggestion that included the word “Search” (search, keyword search, quick search, etc.).

Ranking

Users were provided the following options: Find It Ask Search Where do I… Select a Topic

Clear preference here: 80% of the participants tested ranked the label “Search” #1.

For most participants "Ask" ranked very low, and some commented it sounded like Ask Jeeves or like one could actually ask questions online. Many participants would like a "Select a Topic" feature and would expect to see a drop-down menu, but said this was obviously different than Search.

Location

90% expected Search to be somewhere in the top menu bar Right hand side of the bar 4 Left hand side of the bar 2 Center of the bar 0 No real preference within bar 3

Search Preferences

Participants were given color printouts of three home pages: example with search box empty (ED.gov, box in left hand

menu) example with search box filled in (DOT.gov, filled box in

header graphic) example with no search box, just a link to "Search" (DOD,

link in menu bar)

During this test, 60% of the participants expressed a definite liking for a “search box” (2 with text inside, 2 empty, 2 no preference or didn’t matter).

Contact Us

Which label most accurately depicts a place where you can: Email the owners of the site Get the phone number of the site owners Get the address of the site owners Report a technical issue or a question about the site

Suggestions for labels included Public Information (3), Request… (3), For More Info (3), FOIA (2 – one user had worked for govt. and recognized acronym).

Ranking

Assistance

Comments

Help

Contact Us

For More Information

There was a strong preference for Contact Us- eight of the nine participants listed it as #1.

Location

70% expected to find the Contact Us link at the bottom of the page

3 participants- top right menu bar 3 participants- bottom left menu bar 2 participants- bottom of page 1 participant- bottom right menu bar 1 participant- bottom center of page

Comments

“Said this should be a link to a phone directory for agencies, divisions, and bureaus . . . you can never find anybody anymore!”

“Thinks agencies should send at least a boiler plate response in an email.”

“Contact Us should be a way to email an agency.”

FOIA

Which label most accurately depicts a place where you can:

Learn about your rights to request from a Government Agency public information that in the past had not been available to the public.

Request information – such as books, charts, emails, and records - on a specific Government Agency’s activities.

Ranking

Unadjusted Adjusted

Public Rights to Gov Info 2.43 3.18

Public Access to Gov Info 2.50 2.82

Freedom to Request 2.86 3.59

Request Gov Info 3.44 3.55

Freedom of Information Act 3.67 3.77

FOIA 4.07 4.23

Location

Participants were mixed on expected location:

5 participants left menu bar 3 participants page center 4 participants bottom of page 2 participants right menu bar

Comments “Recognized "Freedom of Info" Thinks media/big wigs have this,

not people like her… Would not recognize FOIA, Definitely prefers the term "public" Tells her she, a citizen, has access to it.”

“[says] If she didn't know FOIA, she would pick public rights/access.”

“"Rights" suggests these may be problems or roadblocks in getting the info.”

“… people want to go right to submitting a request. The example page [USDA FOIA page] is what's expected but info about law is not useful…. Links on the page should be moved up.”

“Example page: does not like to have to go to another page to find out more about filing a request.”

Current Information

Test participants were shown two pages: Today’s Date at top of page (DOL) Last Modified Date at bottom of page (BLS)

Is this page up to date? What does the date indicate? Is this date useful?

Ranking

Participants were then asked to rank, in terms of usefulness, the following types of dates: Today’s Date Posted Date Last Modified Date Last Reviewed Date Last Certified Date Last Updated Date Creation Date

Results

Participants either loved or hated Today’s Date: “Today's date doesn't tell whether info is current.” “Likes both Today's Date and Last Modified.. Depends on

info he's looking for.” “Date up on top left corner is not updated date, not useful.” “Date by itself could mean today. Many people may not

know what "modified" means - but there has to be a label with a date.”

“Today's date is important. Indicates that page is kept current all day. Stats should be current!”

Results

Participants seemed to like either “Last Modified” and “Last Updated” to indicate the currency of content on a particular page. Several ranked them equally or close to equal. “Stats and Statements of fact- wants a last

modified date. If no date no way of knowing how accurate or up to date is the info.”

“Last modified-content updated. TP likes the last modified date info.”

Location

Most participants expect the date to be at the top of the page:

5 participants- top left of page

2 participants gave an expected location vs. actual (desired) location--“prefer top of page, expect upper right”

FAQs or Common Questions

Which label most accurately depicts a place where you can find information requested most often from the agency?

4 out of 11 (36%)- Frequently Asked Questions 3 out of 11 (27%)- FAQs Other suggestions: Frequently Requested…, Most Often

Requested….

Strong preference for “standard” labels of Frequently Asked Questions or FAQs. Note: Throughout testing, users often expressed dislike for, or confusion from, acronyms such as FAQ, FOIA, etc.

Location

Participants were mixed on expected location:

3 participants- top left of page 2 participants- bottom of page 3 participants- right menu bar (1

bottom, 2 unspecified) 2 participants- middle of left

menu bar

Comments

“Doesn't like to hunt. Uses it as a last resort. Looking for questions/answers. On test page [EPA FAQ on Radon]…a lot on one page.”

“Want to reach a human being. Not machine.” “Usually get the answers I want” “Would like contact info or help if you can't find what

you need. “Does use FAQs depending on what he is looking

for, if he doesn't see info on first page and he thinks it is common information.”

No Fear Act

Which label most accurately depicts a place where you can:

See a summary of any grievances related to unfair treatment of a person or group on the basis of prejudice filed against an agency

Find statistical information about any acts of reprisal against government workers who reported unfair treatment at an agency

Suggestions for labels included: Grievances, Government Workers Disputes, Report on Grievances, Discrimination Against Government Employees, Employee Actions & Grievances, Reprisal, Discriminatory Site, Unfair Employment…

Ranking

Discrimination and Retaliation Complaints 2.20 2.41 Equal Employment Opportunity Complaints 2.44 2.82 Equal Employment Opportunity Data 3.25 4.45 Employee Complaint Data 3.56 3.73 Federal Employee Antidiscrimination 4.29 4.64 and Retaliation Act EEO Data 4.75 5.05 Antidiscrimination & Whistleblower 4.88 4.91

Protection Reporting

Location

Top locations selected on generic page layout:

5 of 11 (45%) of participants- look for it at bottom of the page, (no predominance for left/right/center).

4 of 11 (36%) of participants- look for it in top nav bar, (2 left side, 2 right side)

Comments

“Didn't like Federal Employee Antidiscrimination & Retaliation Act and Antidiscrimination & Whistleblower Protection Reporting because they were "very wordy"”

“Employee Complaint Data is too generic and vague, not very familiar with EEO or EEOC”

Web Site Policies and Important NoticesCard Sort Items

Privacy & Security Statement Freedom of Information Act (FOIA) Information Quality Guidelines Portable Document Format (PDF) Accessibility Policy Linking to Outside Websites Disclaimer of Endorsement Copyright Information No Fear Act Photo Credits Schedule for Publishing Information Nondiscrimination Statement Legal Notices Free Commercial Downloads

Participant Suggestions

Policies Legal Overall Info Site Info Additional Info Information Guidelines Information Statement

Card Sort Results

Users had a lot of difficulty creating one group that covered all topics. Some users suggested two groupings of “legal” and “information”.

The following content items were singled out by multiple (3+) users as either irrelevant or confusing with relationship to grouping the other items: PDF, Linking To Outside Websites, Free Commercial Downloads, and Photo Credits.

Participant comment: “stuff you don’t even know what it is. PDF? Photo Credits?”

Ranking

Unadjusted Adjusted

Notices and Disclaimers 1.50 1.55

Site Notices and Links 2.44 2.50

Site Policies and Links 2.71 2.85

Important Notices 3.00 3.09

Note: 54% of the participants ranked “Important Notices” at or near the bottom.

Location

Whatever we call it, users expect to find it at the bottom of the page:

7 of 11 (63%) of participants- look for it at bottom of the page (0 left side, 2 right side, 1 center, 4 not specified)

No other location got more than 1 suggestion.

Comments “Important Notices (this doesn't say anything)” “Does not know what "Accessibility Policy" is. Accessibility does

not fit.” “Keep policies/legal info separate from free stuff .” “Card Sort: PDF, Linking to Outside Websites, Free Commercial

Downloads -these three don't fit with others” “Free Commercial Downloads - What does that mean! Nothing to

do with govt.” “"I like policies more than notices. " "They are legal policies not

all are site policies." Why is "Photo Credits" here.” “No to [suggested label] Important Notices (Not important)”

What testing could be done next

Two tests were already designed by the testing subcommittee but were dropped from this round of testing because of time constraints

Label, location and expected content for “Jobs”, “Privacy”, “Site Index” and “Site Map”.

Best practices for Linking to Outside Websites.

Possible additional tests

Label, location and content for “Information Quality Guidelines”

Appropriate content and best practices for cross-government portal implementation

Plain language (writing content for the web) Limited English Proficiency (translation

services / cross-language content delivery)