quantifying the impact of transparency · several agencies have embraced the concept of...

ForeSee Results’ E-Government Transparency Index(2010 Year in Review)

Quantifying the Impact of Transparency

February 22, 2011

by Larry Freed President and CEO of ForeSee Results

© 2011 ForeSee Results

2 • ForeSee Results’ E-Government Transparency Index (2010 Year in Review)

www.ForeSeeResults.com

Introduction

“Transparency and Open Government” has been one of the buzz-phrases of President Barack Obama’s White House, an ambitious and sweeping objective to transform government departments and agen-cies into entities that reflect the democratic ideals of openness, democratic participation, and col-laboration. Now, two years after a memorandum was disseminated to the federal government insti-tuting such an organizational renovation, many are wondering if the goal has taken root and, if so, how successful it has been. In the meantime, political power has changed hands, we’re in the middle of a contentious budget season, and the focus on transparency as a government-wide priority is un-certain.

This report, the fifth of its kind, gives us the opportunity to look back over 2010 and assess a full year of transparency. In the first quarter of 2010, 23 federal sites participated in the E-Government Transparency Index. By the end of 2010, participation had increased 35%, bringing the total number of participating websites to 32. Online Transparency scores have been improving for the past two quarters, after a drop in the second quarter of 2010. A chart with scores by quarter for the 32 federal websites currently participating in this Index can be found on page 5.

How to Measure Transparency

At the request of government agencies interested in clear and concise progress data, ForeSee Results developed the E-Government Transparency Index in 2009. The goal was twofold:

• To create an accurate, actionable and precise measurement of citizens’ opinions on government transparency

• To quantify the relationships among online transparency, citizen satisfaction, trust, and the likeli-hood to participate and collaborate with government agencies

The ForeSee Results E-Government Transparency Index has grown into a vital tool for measuring the success, failure, or progress of government departments and agencies online, providing a clear direction for improvement, and forming a benchmark from which all other non-participating entities can draw instruction and inspiration. After all, you cannot manage what you do not measure. The ForeSee Results E-Government Transparency Index does not count the number of documents that each agency makes public or check which information is available and which isn’t. Using a standard methodology that has been in use in government since 1999 (the American Customer Satisfaction Index), the E-Government Transparency Index measures online transparency according to citizen perceptions. How easy is the information to find? How quickly is it made available? How thorough is it? In a democracy, citizen perceptions define reality, and transparency’s success or failure should be first and foremost defined by citizen opinions.

http://www.whitehouse.gov/the_press_office/TransparencyandOpenGovernment/



The E-Gov Satisfaction Model

Quantifying the relationship between citizen satisfaction, transparency, and trust is a remarkable step toward making the government accountable for measuring and improving performance. Research has defined the link between online transparency, satisfaction, and trust, giving government agencies the tools they require to measure their success in meeting the open government directives, identify where and how to improve citizens’ view of transparency, and drive citizen satisfaction higher. Im-proved citizen satisfaction is a key result. A targeted focus on what matters most increases citizens’ trust in the agency and increases usage of the most cost-effective channel available (the web), all of which leads to a more democratic and cost-effective government.

This model was created and perfected over the previous year by academics and experts who tested the relationships shown in the following diagram. Following a rigorous and extensive proof period, a model emerged that clearly defines and quantifies the relationships between online transparency and overall trust, with online satisfaction as the mediator between the two (as show in Fig. 1).

Figure 1: E-Gov Satisfaction Model

WebsiteSatisfaction

Website Elements Future Behaviors

Functionality

Content

Online Transparency

Site Performance

Look and Feel

Navigation

Site Search

E-Gov: Online Transparency and Overall Trust

Trust

Future Participation

Primary Resource

Recommend

Return



How it Actually Works

Surveys are conducted with randomly-selected visitors to the 32 federal websites included in this Index. About 320,000 visitors who completed surveys over the course of 2010 were asked a series of questions about their experience with and perceptions of various elements of the website, including three aspects of online transparency: how thorough the information on the website is, how accessible it is, and how quickly it is made available. They were also asked about other elements of the online experience such as look and feel, navigation, site performance, etc.

Respondents are asked to rate their perceptions of various aspects of the site experience on a ten-point scale. Citizens’ responses are then processed through the ACSI statistical engine, and one result is a set of priorities for improvement. These priorities are not determined by the agencies or even by the lowest-scoring elements, but by the citizens and the ACSI methodology. Thus, we are able to calculate whether transparency is a top priority that could have a large impact on improving satisfac-tion with an individual website. For some, it is; for others, it isn’t.

The 2010 Year-End E-Government Transparency Index

The agencies and departments listed here have agreed to measure and report on transparency as part of an effort to meet the Obama administration’s open-government objectives. The scores are calculated within a sophisticated structural equation model based on responses to a set of relevant questions, as is the ACSI E-Government Satisfaction Index (which is also released quarterly but in-cludes more than 100 websites since satisfaction is a common e-government performance metric). All online transparency scores are reported on a 100-point scale.

Figure 2: Online Transparency Index Over Time

Time Period Online Transparency Over Time

Q4 2009 75.4

Q1 2010 76.2

Q2 2010 75.0

Q3 2010 75.8

Q4 2010 76.2

The aggregate online transparency score for the 32 federal sites included is 76.2 on the study’s 100-point scale, up nearly half a point since the previous quarter and a full 1.2 points since its low-est point in the second quarter. While it’s good news that citizen ratings of online transparency have rebounded, little progress has been made over the last year.

74

76

78

80

82

84

Customer Satisfaction with Online Retail Over Time

Cust

omer

Sat

isfa

ctio

n (1

00-p

oint

sca

le)

78

77

8384

80

81

83 83

82

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

83

80

84



It is important to remember that all agencies on this list have voluntarily submitted their scores. These results measure only 32 federal websites among thousands, although most of the federal gov-ernment’s department sites are represented. Sites that find themselves at the bottom of this Index would certainly score higher than many others in a comprehensive Index of federal government on-line transparency. As such, each of the entities should be commended for their efforts.

In Figure 2, we have reported online transparency scores over the last year. We’ve also shown Q4 sat-isfaction scores for reference because of the close and causal relationship between transparency and satisfaction. The most recent quarterly online transparency score is highlighted.

Figure 3: ForeSee Results 2010 E-Government Transparency Index: Department and Agencies Scores Over the Last Year

Dept. WebsiteQ1 2010 Online

Transparency

Q2 2010 Online

Transparency

Q3 2010 Online

Transparency

Q4 2010 Online

TransparencyQ4 2010

Satisfaction

DHSU.S. Citizenship and Immigration Services Español-- www.uscis.gov/portal/site/uscis-es

- 85 85 86 85

DOCNational Oceanic and Atmospheric Administration main website -- www.noaa.gov

84 83 85

DHSU.S. Citizenship and Immigration Services -- www.uscis.gov/portal/site/uscis

80 82 82 83 81

DOD DoD Navy -- www.navy.mil 82 82

HHSNational Human Genome Research Institute -- www.genome.gov

83 82 84 82 80

DOD DoD Air Force -- www.af.mil 80 79 80 81 80

HHS SAMHSA -- www.mentalhealth.samhsa.gov 81 82 80 80 77

DOS Bureau of Consular Affairs -- http://travel.state.gov 80 80 81 80 77

DOD Department of Defense portal - - www.defense.gov 79 75 76 78 76

GSA GSA main website -- www.gsa.gov 78 78 73 78 75

NRCU.S. Nuclear Regulatory Commission website -- www.nrc.gov

76 77 73




Transparency

Q2 2010 Online

Transparency

Q3 2010 Online

Transparency

Q4 2010 Online

TransparencyQ4 2010

Satisfaction

DOS Department of State main website -- www.state.gov 74 73 75 77 75

FDIC FDIC main website -- www.fdic.gov - 77 75

DOSDepartment of State blog website -- www.blogs.state.gov

76 75 75

DOC BEA main website -- www.bea.gov 74 74 75 75 71

Treasury IRS main website --www.irs.gov 75 72

NARA NARA main public website -- www.archives.gov 74 75 75 75 74

HHSAgency for Healthcare Research and Quality -- www.ahrq.gov

77 79 78 75 72

HHS SAMHSA website -- www.samhsa.gov 76 74 76 74 70

HHS Heath Information Technology -- healthit.hhs.gov 71 70 71 74 70

USDA FAS main website -- www.fas.usda.gov - 72 72 74 72

PBGC U.S. PBGC main website -- www.pbgc.gov 77 70 71 74 71

DHSDepartment of Homeland Security main website -- www.dhs.gov

75 72 73 74 70

DOJOffice of Community Oriented Policing Services -- www.cops.usdoj.gov

77 76 75 74 76

DODForce Health Protection & Readiness Policy and Programs -- http://fhp.osd.mil/

- 73 74 73 69

DOI U.S. Geological Survey -- www.usgs.gov 76 75 73 73 68

Figure 3: ForeSee Results 2010 E-Government Transparency Index: Department and Agencies Scores Over the Last Year (continued from the previous page)




Transparency

Q2 2010 Online

Transparency

Q3 2010 Online

Transparency

Q4 2010 Online

TransparencyQ4 2010

Satisfaction

EPAU.S. Environmental Protection Agency -- www.epa.gov

- 71 72 73 68

DODMilitary Health System main website -- www.health.mil

77 72 73 73 67

DOTDOT Research and Innovative Technology Administration website -- www.rita.dot.gov

73 72 67

HHSU.S. Food and Drug Administration main website -- www.fda.gov

71 71 72 72 68

FDIC FDIC Applications -- www2.fdic.gov 71 69

USDA USDA Farm Service Agency - www.fsa.usda.gov 69 68 69 69 68

Several agencies have embraced the concept of transparency at a high level and have three or more websites measuring transparency, including the Department of Health and Human Services (measur-ing transparency on six sites), Department of Defense (five sites), Department of State (three sites), and Department of Homeland Security (three sites). As more and more agencies and departments make online transparency an organizational priority, we should see greater participation in this Index, and higher scores.

Quantifying the Impact of Transparency

The need for transparency in government is not a fad. As much as it is admittedly the current political flavor-of-the-month, our research indicates it shouldn’t be: not only does transparency have a signifi-cant and quantifiable impact on making government more democratic and cost-effective, it is highly prized by citizens.

Figure 3: ForeSee Results 2010 E-Government Transparency Index: Department and Agencies Scores Over the Last Year (continued from the previous page)

74

76

78

80

82

84


Cust

omer

Sat

isfa

ctio

n (1

00-p

oint

sca

le)

78

77

8384

80

81

83 83

82

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

83

80

84

74

76

78

80

82

84


Cust

omer

Sat

isfa

ctio

n (1

00-p

oint

sca

le)

78

77

8384

80

81

83 83

82

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

83

80

84



As the following chart shows, when citizens find a website highly transparent (80 or higher on a 100-point index), they give 85% higher satisfaction ratings than citizens who rate a federal website’s transparency poorly (70 or lower). Moreover, based on score differences, citizens who perceive a federal website to be highly transparent also report they are:

• 48% more likely to participate by expressing their thoughts and ideas with that agency or depart-ment in the future, offline or online;

• 38% more likely to return to the website in the future;

• 62% more likely to recommend the website;

• 55% more likely to use the website as a primary resource; • And they express 57% higher levels of trust in the government entity.

Figure 4: The Impact of Online Transparency on Cost-Saving Future Behaviors

Likely Future Behavior Transparency Scores 80+

Transparency Scores < 70

Point Difference % Difference

Future Participation 68 46 22 48%

Likelihood to Return 94 68 26 38%

Likelihood to Recommend the Website 94 58 36 62%

Likelihood to Use the Site as a Primary Resource 90 58 32 55%

Trust in the Agency/Department 88 56 32 57%

If citizens perceive an e-government site to be highly transparent, the returns for that agency are sig-nificant—visitors are more likely to return to the site, recommend it, use it rather than a more costly channel, and even have higher levels of trust in the government entity.

Trust is one attribute that departments and agencies can never take for granted. Trust implies that a department has proven itself worthy of confidence and reliance. It is one of the few qualities for which citizens are willing to give an agency a second or third chance after a mistake has been made. It involves a responsibility borne from integrity and, as such, both its origins and its reach cannot be underestimated.

These behaviors and attitudes represent the rewards for open, cost-effective, efficient government. Every federal agency and department has access to these sorts of returns—all that is required is a baseline measure (both of transparency and the behaviors and attitudes mentioned in the previous chart) to capture data and point the way toward palpable, effective improvements.

74

76

78

80

82

84


Cust

omer

Sat

isfa

ctio

n (1

00-p

oint

sca

le)

78

77

8384

80

81

83 83

82

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

83

80

84



How Much Does Transparency Matter?

Figure 3 reveals a clear and robust relationship between transparency and citizens’ attitudes and behaviors. While these figures represent aggregate analysis, it is equally important to examine the role transparency plays for individual websites, and the impact that it can play—great or small—on a website’s, and hence an agency’s, success.

ForeSee Results does not measure online transparency in isolation, but rather as one of a group of key drivers influencing a citizen’s experience with a federal government website, often including naviga-tion, functionality, content, look and feel, and site performance. While different sites naturally have different key drivers of satisfaction, the list below represents the crux of a methodology with the ability to quantify which elements have the largest impact on satisfaction, trust, and a host of desired future behaviors, as listed in Figure 3. The questions in the ACSI model ask citizens about their per-ceptions of aspects of their experiences on a federal website as shown in Figure 3, and to rate their perceptions on a 10-point scale. Citizens answer multiple questions for each measured element, and our statistical engine allows us to quantify improvement priorities. These priorities are determined not by the agencies themselves or by aggregate benchmarks, but by the citizens’ own evaluations of their experience while visiting the site. The ACSI methodology determines the impact of the evalua-tions on citizens’ website satisfaction and likelihood to engage in key behaviors in the future. It would be difficult to know which element has the largest impact on citizens’ trust without utilizing the ACSI methodology to identify the critical opportunities for improvement from a citizen’s perspec-tive. As mentioned earlier, the elements vary according to organization.

Some of the website elements measured as part of this study are: online transparency, site search, navigation, functionality, look and feel, site content, and site performance. Sites may have more than one priority that has a large impact on satisfaction, so each site’s top two priorities were examined in this analysis. Clearly, for a site to be considered transparent, other elements of the website experi-ence must be performing at an adequate level. Critical content that can contribute to a site’s percep-tion as transparent must first be found and accessed before it can be evaluated by citizens to deter-mine what type of information is available on a given site and whether or not it is enough. Without good functionality, navigation, and search mechanisms, a website’s efforts at transparency (through sheer availability of content) may remain unnoticed.



Figure 5: Critical Website Elements and Their Importance

Driver of Website Satisfaction What It Measures Priority Analysis

Site Search The utility and effectiveness of the site’s search tool

Improving search is a first or second priority for 74% of sites that are measuring this element (27 of 31 sites represented in this research do so). Focusing on improving transparency (in terms of information availability) alone at the expense of improving search may not produce the desired ROI for sites where search is also a priority, if it is an impediment to visitors’ success in finding information and, ultimately, being perceived as transparent.

Functionality The usefulness, convenience, and variety of online features available to citizens

75% of sites in this Index that measure this element (18 sites) register website functionality as a first or second priority. Individual sites may look to balance efforts in more than one area if they have no clear number one priority. Also, for a site to be viewed as transparent, it must have functionality that supports and provides access to the information citizens feel should be available.

Online Transparency The thoroughness and accessibility of information made available on the website

This research shows that more than half the sites included in the Transparency Index register improving online transparency as a first or second priority. However, for 20% of sites, transparency registers as a low priority for improvement, highlighting the importance of each site measuring its own priorities.

Navigation The organization of the site and how easy it is to navigate

Navigation is often a challenge for information-rich sites, and this element is often lower-scoring and higher-impact for federal government sites tasked with offering access to broad and deep information resources. One-third of the measured sites register navigation as a number one or two priority.

Look and Feel The visual appeal of the site and its consistency throughout the site

This element is rarely a priority for individual federal sites, once a certain threshold of consistency in design and removal of any potentially distracting visual aspects is achieved.

Site Content The accuracy, quality, and freshness of news, information, and content on the website

This element is rarely a priority for the individual federal sites, because the content that is available is often considered adequate. When this element is a low priority, it is usually easier and less inexpensive to address than issues with many other elements. (As compared to transparency, this element focuses on the quality of current content, while transparency evaluates whether what is available is sufficient and accessible.)

Site Performance The speed, consistency, and reliability of loading pages on the website

Federal government sites generally provide adequate site performance for visitors. If transparency efforts fuel needs for advanced database access on federal sites, this element may become a higher priority.

74

76

78

80

82

84


Cust

omer

Sat

isfa

ctio

n (1

00-p

oint

sca

le)

78

77

8384

80

81

83 83

82

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

83

80

84



Often, what an organization does not value is every bit as informative as what it does value. While online transparency represents a top priority for a majority of sites that measure it, there are still those who, despite the administration’s initiatives, have higher priorities. The priorities highlighted in this research are determined through citizen surveys and the ACSI methodology, not by the agen-cies themselves or the Administration. For example, search is a critical priority for most sites that also measure online transparency, but is closely followed by online transparency itself. For sites who register search or some other element as a top priority, focusing on transparency in terms of infor-mation availability alone at the expense of improving other aspects of the site experience will not produce the desired ROI, regardless of executive orders and memorandums.

Without the kind of critical information found here, federal websites will have a hard time making the sorts of improvements that will enhance the value and usefulness to citizens. If every website follows the Obama Administration’s mandate to focus on transparency by providing additional content alone, many sites will be misplacing some of their resources. As this sampling of federal websites shows, the diagnosis and prescription for increasing trust (and a whole host of other attitudes and behaviors) differ from site to site.

About The Author

Larry Freed is an expert on government web effectiveness and web customer satisfaction. He is Pres-ident and CEO of ForeSee Results, a market leader in customer satisfaction measurement on the web, which utilizes the methodology of the American Customer Satisfaction Index.

About the Research Team

Rhonda Berg, Research Manager at ForeSee Results, leads the research team that produces the quar-terly E-Government Satisfaction Indices and the quarterly E-Government Transparency Indices. She also serves as an internal consultant regarding statistics, methodology, and survey design. Rhonda has been a research professional for 20 years in a number of industries and holds advanced degrees in business and sociology.

About ForeSee Results

As the leader in customer satisfaction measurement, ForeSee Results captures and analyzes voice-of-customer data to help both private-sector and public-sector organizations increase loyalty, rec-ommendations and marketing value. Using the methodology of the American Customer Satisfaction Index (ACSI), ForeSee Results identifies improvements across all channels and touch points that drive satisfaction. With over 58 million survey responses collected to date and benchmarks across dozens of industries, ForeSee Results offers unparalleled expertise in citizen satisfaction measurement and management for the federal government.

ForeSee Results, a privately held company, is headquartered in Ann Arbor, Michigan and on the web at www.ForeSeeResults.com. Connect with ForeSee Results at www.ForeSeeResults.com/connect.html.

http://www.ForeSeeResults.com

http://www.ForeSeeResults.com/connect.html

quantifying the impact of transparency · several agencies have embraced the concept of...

Documents