test

Open Source Science Journal Vol. 2, No. 2, 2010

212

An objective approach to evaluating the quality of Romanian central public

authorities’ web sites: Are they citizen-orientated?

Claudia ICONARU, PhD Candidate Academy of Economic Studies, Bucharest, Romania

[email protected]

Abstract: The purpose of this paper is to frame specific metrics for evaluating the

quality of public authorities’ web sites and to employ them in a case study measuring the

quality of Romanian central public authorities’ web sites. The proposed metrics are built-up

based upon previous researches in web sites’ evaluation field and Romanian legislation in

force, governing the citizens’ right to information and decisional transparency. The sample

used in this study, namely the Romanian ministries’ web sites, is rated against the proposed

metrics which measure the technical characteristics, e-content and e-participation, factors

influencing the overall quality of the web sites. The findings of the case study capture the

peculiarities, the shortcomings and the best practices of the above mentioned central public

authorities’ web sites. The paper also provides an objective assessment of the metrics used in

this study, identifying the limitation of these measurements and the study itself.

Keywords: metrics, quality, web sites, public authorities, citizens.

1. Introduction

The internet, the rapid development of the technology and the globalization are all

factors contributing to the rise of the new economy, namely the digital economy which favors the emergence of new capabilities and mentalities among businesses and consumers [11]. The private sector is not the only one facing new challenges and opportunities, the public sector also. It is found in a situation where it must change its orientation towards meeting the citizens’ needs and wants.

The citizen is the central element of the new informatics applications: the citizen-orientated web applications. Their purpose is to maximize the degree of citizens’ satisfaction by solving its particular needs. They come as a solution to citizens real problems, not as a solution to the own organization needs and wants. The information they provide is correlated with the citizens’ interest, not the organization’s interest [9].

Subsequent to the provisions of Law no. 544/2001 regarding free access to information of public interest and Law no. 52/2003 regarding the decisional transparency in public administration [1], the Romanian public authorities are determined to reinforce their online presence through citizen orientated web sites. The implementation of high quality web sites, directed towards citizens’ needs, has the ability to strengthen the bi-directional communication between public authorities and Romanian citizens. This way, citizens are involved in the decision making process of adopting legislative acts (art.1, para.2, Law no. 52/2003) and they are granted a higher accessibility and awareness of public interest information and their rights conferred by law. Thus, I outline the first objective of this study: the investigation of the availability of public interest information and decisional transparency within the web pages of Romanian central public authorities’ web sites. The implications of Law no. 544/2001 and Law no. 52/2003 to this study will be quantified in terms of the web sites’ content information (e-content), more specifically the relevance and completeness of


213

public interest information published with-in web pages, and e-participation, the degree of decisional transparency and the ability of certain web sites’ features to stimulate an interactive relationship between public authorities and citizens. Besides the specific content and the degree of interactivity with its audience, the quality of public authorities’ web sites should be also reflected in their technical capabilities of providing a facile accessibility, a good navigation through the web pages and increased aesthetics features. Thus, I outline the second objective of this paper, evaluating the quality of Romanian central public authorities’ web sites through their technical characteristics. [17] implies that governmental web sites should be interactive, accountable, and usable and that they should demonstrate sufficient technical expertise.

For reaching both objectives, I shall propose an evaluation grid containing a series of metrics based upon a comprehensive literature review regarding the evaluation of web sites, both public and commercial, and current legislation in force. The evaluation grid will quantify three aspects of central public authorities’ web sites, as noted above: technical characteristics of the web sites, e-content and e-participation. Focusing on all three aspects, this paper will be able to provide an objective assessment of the overall quality of Romanian central public authorities’ web sites and their ability to be orientated towards the citizens and their needs and legal rights.

2. Literature review

This section provides a comprehensive review of previous research work related to

web sites evaluation. Different types of web sites were evaluated against various criteria: - Commercial web sites: [5] analyzed Spanish commercial sites, focusing on

accessibility, speed, navigability and content, using metrics such as: search engine presence and link popularity for measuring accessibility; the size of the home page and the loading speed; permanent site menus and search function for measuring navigability; informational factors, transactional factors and communicational factors for measuring content. - Educational web sites: [23] evaluate the content quality, general readability and

usability characteristics of nutrition web sites, using metrics such as: clear organizational scheme, table of contents, internal linking structure, ease of navigation, vertical navigation, help system for measuring navigate on; clarity, relevant and active internal links; high resolution images, appealing pictures and graphics, formatting text to measure aesthetics; content accuracy and content currency for measuring content quality [23]. - E-government web sites: [26] developed a measurement model consisting of nine

items that could evaluate the user satisfaction: awareness, technical aspects, usability, content, security/privacy, infrastructure, availability, customer friendliness; [15] proposed a framework consisting of four axes: two for evaluating general characteristics and content of the public authorities’ web sites and two for assessing e-services and e-participation, using various metrics to assess accessibility, navigation, privacy, multilingualism, public outreach, general and specific content, news and updating and services offered.

Following previous literature and taking into consideration the particular needs of this study, I grouped the various factors affecting web sites’ quality into three building blocks: technical characteristics, e-content and e-participation (e.g. Table 1).


214

Table 1. Main building blocks of factors influencing the quality of central public authorities’ web sites

Factors

Technical characteristics Accessibility, Navigability, Privacy/Security, Aesthetics E-content Public interest Information, General Information E-participation Decisional transparency, Interactivity

Accessibility: [15] identified several approaches to the concept of accessibility in previous literature: the most commonly addressed is the issue of providing access to disabled people, followed by technical accessibility, hardware and software, internet connection speed and multiple language support. With regard to multilingualism support [17] includes among government public not only the nation’s citizens, but also the stakeholders outside a nation, outlining the potential of web sites to build the nation brand and consolidate diplomatic objectives.

[5] addresses accessibility in terms of identification and ease of relocation in www, proposing two evaluation metrics: search engines presence and link popularity. According to World Wide Web Consortium, web accessibility gives the possibility of disabled people to perceive, understand, navigate and interact with the web site (w3.org). Besides people with disabilities, the W3C community also includes older people, whose senses, such as vision and hearing decrease due to aging.

There are a plenitude of studies and researches regarding web accessibility: [19] reveals usability problems faced by Chinese e-government web sites and the need for implementing accessibility laws and guidelines, after analyzing the home pages of 339 Chinese local government web sites using Bobby Online Free Portal; [10] analyzes the compliance with section 508 of the Rehabilitation Act which requires U.S. federal e-government web sites to be accessible to disable people, using a complex multi-method: user testing, expert testing, automated testing and questionnaire testing, showing that different results are obtained when different methods are used, and different agencies place different emphasize on Web accessibility, some made web accessibility a priority and some resisted compliance with the Rehabilitation Act; [12] used an online accessibility testing tool to test 130 sites of UK members of Parliament against the UK’s Disability Discrimination Act, summarizing that a preponderance of UK e-government MP sites is incapable of meeting all legal mandates and community recommended guidelines. The same accessibility issues were identified in other areas, other than e-government web sites: education field: [20] tested Web accessibility of U.K. higher education web sites, using a automatic validation testing tool, manual testing, evaluating the web site with different browsers and assistive technologies (PwWebspeak, JAWS for Windows, Braille Display), summarizing that the overall level of accessibility was reasonable, although there are many accessibility barriers; e-commerce: [GUTI05] evaluated the accessibility of airline online reservation sites in US for disabled people, concluding that most airlines are not addressing the issue of web accessibility thus ignoring a significant proportion of potential customers.

The assessment of web accessibility is even more important for public authorities’ web sites considering their role in offering information of public interest and involving all citizens in the administrative decision making process, despite their disabilities. Not being able to meet accessibility requirements means depriving a considerable percent of the total population from their right to public information and decisional transparency: there are 681.558 disabled people in Romania, both non-institutionalized and institutionalized, among which 142.922 suffer from auditory and visual disabilities [3]. Although there are no legal requirements abiding Romanian public authorities web sites’ to meet certain accessibility for disabled people, The Romanian Ministry of Communication and Information Technology


215

offers general guidelines regarding the implementation of public web pages, such as compliance with the latest W3C specifications, WAI and WCAG [4].

Navigability: A good navigation structure which can facilitate the citizens’ movement through the information architecture is a must for public authorities’ web sites. Navigability can be defined as the sequencing of pages, well organized layout and consistency of navigation protocols [14]. The poor quality of navigability results in disorientation, intimidation and frustration [4].

Relevant literature in the field of improving within-site navigation focuses on: - Site maps or A-Z Index presenting the hierarchical organization of a web site: [4]

refers to site maps as a common combatant of the problem of disorientation, [27] reinforces the fact that site maps are the most commonly used solution to address both the issue of too much information and relevancy. - Search Tool within the web pages: [1] points out the importance of search systems in

academic web sites for a better retrieval of large amounts of information. [27] considers in-site search facilities as the second most popular tool for overcoming information overload. - Average number of clicks: [18] assessed the quality of the web site navigation

structure by computing the average number o clicks (path length divided by the number of nodes in the tree navigation structure) to reach a desired web page. - Intelligent navigation aid tools: term coined by [27], refers to recommendation

systems and collaborative filters. According to the guidelines provided by the Ministry of Communication and Information Society, public authorities’ web sites should have a understandable and accessible hierarchical structure of web pages, and they should all have in common a drop-down menu on the left side of the web page, a site map, a search facility, a help facility, a calendar date indicator etc. [4]

Privacy and Security: Online trust, or the confidence users have in a particular web site, the information it contains and its ability to meet users’ expectations [2], has gained a lot of importance as a topic for many studies, especially with the increase number of e-transactions. Two of the most critical issues to online trust are privacy and security, although online trust depends on other variables as well [25]. A recent study of [28] assessing the security level of e-government web sites, shows that although most of e-government web sites contained a privacy and security policy statements, less than half were clear about the implementation of key security measures. The Romanian Ministry of Communication and Information Society recommends the implementation of measures to protect personal data and security measures to protect information [4]

Aesthetics: The first impression of a web site is strongly correlated with its aesthetics and highly aesthetics web sites can positively influence the online consumer behavior [24]. Moreover, there is a direct and positive relationship between web site aesthetics and its credibility, the content of a web site with a higher aesthetic treatment was perceived to be more credible than the same content with less aesthetic treatment [16].

There are two types of factors that can predict the positive aesthetic impression and user preference of a web site, objective factors like symmetry, order, complexity and subjective factors like familiarity and novelty [13]. A study of [13] demonstrates that users appreciate both qualities like discipline, clarity, control offered by classical design and qualities like innovation, creativity and expressivity. Still, the classical design of a website is more related to perceived usability [13].

The Romanian Ministry of Communication and Information Technology recommends decent aesthetics, in accordance with the web sites belonging to senior institutions: a brightness contrast between the text and background color, avoidance of tacky strong contrast, standard color palette and good quality images and graphics [4].


216

Another aesthetic factor and still a usability principle is the palette of colours used when designing a web site. Subsequent to an experiment, [7] highlights the fact that web sites having a greater contrast between the text color and the background color are easier to read and that blue shades and chromatic colors were the most preferred among the interviewed people, leading to higher ratings of aesthetics quality.

E-content: Precise and current content is an indispensable part of a successful web site [15]. Considering the public nature of the information published with-in public authorities’ web sites, the importance of quality content is even higher for central administration. The public participation of citizens to government decision making process depends upon the availability and accessibility of information [17]. In the guidelines regarding the implementation of public web pages, provided by the Romanian Ministry of Communication and Information Technology [4] there are several types of information that public authorities’ web sites should contain: - General public information: contact coordinates, operating schedules, schedule

hearings, news and events, useful links and telephone numbers etc. - Specific public interest information, related to the transparency of public interest

information: organizational structure of the public authority, the functions of the departments, wealth declarations and declarations of interest, subordinated institutions, vacancies and employment competition, list of programs and strategies, financial information (financial sources, budget and balance sheet), the regulatory provisions governing the organization and functioning of the public authority or institution, list of public documents and public standard forms etc.

[21] implies that the existent information should be reviewed and updated with regularity. Regarding the currency of information, the Romanian Ministry of Communication and Information Technology recommends updates of information in every month or whenever necessary [4].

E-participation: Public authorities’ web sites should foster a multi-directional communication between the citizens plus other stakeholders and public administration and between the members of the stakeholders groups, for each other benefits [17]. The decisional transparency of Romanian local public authorities has been previously studied in a monitoring report regarding the implementation and enforcement of Law 52/2003, by “Asociatia Pro Democratia”, a non-governmental, non profit and non-party affiliated organization, aiming to strengthen democracy at national and international level (adp.ro). The content of the local public authorities’ web pages were monitored and tested against an evaluation grid containing a number of indicators regarding the transparency of decision and information of public interest to citizens. Romanian public authorities are encouraged to involve the citizens in the decision making process by displaying on their websites the normative acts subject to public debate and by inviting the citizens to make proposals to the draft (Article 6 § 1 of Law 52/2003). Citizens can also be involved in the decision making process through the interactive nature of the web site, more specifically through the existence of e-forums, blogs, e-surveys, spot polls, social media and e-petition [17], [15].

3. Research methodology

This section provides an overview of the methodology used in conducting the

research. First, I define the sample used in the research; second, I provide a full description of the metrics utilized for evaluating the three aspects of central public authorities’ web sites: metrics for evaluating the technical characteristics, metrics for evaluating e-content and e-participation; third, I evaluate the proposed metrics against different criteria and last I assign


217

weights to the metrics, resulting the evaluation grid used for testing the sample.

3.1 Sampling

As defined by the law-in-force, the Romanian central public administration is formed by the Ministries, other specialized agencies which may be held in subordination of the government or the ministries and autonomous administrative authorities (msci.ro). All the ministries’ web sites were selected for this research: the Ministry of Administration and Interior (mai.gov.ro), the Ministry of Foreign Affairs (mae.ro), the Ministry of Agriculture and Rural Development (madr.ro), the Ministry of National Defense (mapn.ro), the Ministry of Culture and National Patrimony (cultura.ro), the Ministry of Regional Development and Tourism (mdrl.ro), the Ministry of Economy, Commerce and Business Environment (minind.ro), the Ministry of Education, Research, Youth and Sport (edu.ro), the Ministry of Public Finance (mfinante.ro), the Ministry of Justice (just.ro), the Ministry of Environment and Forests (mmediu.ro), the Ministry of Labor, Family and Social Protection (mmuncii.ro), the Health Ministry (ms.ro), the Ministry of Transports (mt.ro). The above mentioned sample was monitored between the 17th and 29th of May, 2010.

3.2 Measures

In order to measure the level of the first quality characteristic, namely the technical

characteristics of a web site, I propose the following aggregate indicator: I= 0.4 I1 + 0.35 I2 + 0.35 I3 where: I1 is the level of technical quality; I2 is the level of e-content quality; I3 is the level of e-participation quality. Following the literature review of section 2 of this paper, I identified metrics for

evaluating the level of each characteristic correspondent to each partial indicator. I propose the following six metrics for evaluating the public authorities’ web sites accessibility: a) Metrics for evaluating the accessibility for disabled people: the web sites’ ability to meet priority level 1 of Web Content Accessibility Guidelines 1.0

b) Metrics for evaluating the accessibility for foreign citizens: Multilingualism: the existence of at least one foreign language

c) Metrics for evaluating technical accessibility: the web sites’ average downloading and browser compatibility

d) Metrics for evaluating ease of identification on the web: the web sites’ position in Google SERP and The number of incoming links Although there are over one hundred tools with different degrees of automation for

evaluating Web accessibility according to the “Complete List of Web Accessibility Evaluation Tools” published by W3C [2] the results they offer vary among them. [3] studied three different tools: Bobby, Tawdis and WebXACT, concluding that accessibility evaluation tools provide a poor automated coverage of WCAG checkpoints. Even though similar comparison studies between different tools were made, there is no general agreement upon the most suitable software for assessing web accessibility [19]. Due to this fact, I have selected the following accessibility evaluation tools: - SortSite version 2.0, for evaluating browser compatibility issues in the most common web browsers, such as Internet Explorer 6.0, Internet Explorer 7.0, Internet Explorer 8.0, Firefox 2.0, Firefox 3.0, Firefox 3.6, Safari 4.0, Opera 10.0 and Chrome 4.0


218

(powermapper.com); - Web Accessibility Inspector version 5.1 (Fujitsu Ltd.) for evaluating the ability the website to be accessed by elderly and disable people, a tool based on W3C WCAG 1.0 guidelines;

- Website Speed Check for evaluating the average download speed computed as a mean of modem load time, UMTS load time, DSL 768 load time, T1 and T3 load time (rapid.searchmetrics.com);

- Back link Checker for evaluating the total number of incoming links toward the referred site (seo-portal.ro). Taking into account previous literature and the guidelines of the Romanian Ministry of

Communication and Information Society for implementing public authorities’ web sites, I propose the following five metrics for evaluating navigability: a) Metrics for evaluating the navigation structure: availability of a site map or a A-Z index, availability of a left quick navigation menu, availability of a home link on each page

b) Metrics for evaluating the capacity of information retrieval: availability of an internal search engine, the level of complexity of the internal search engine For evaluating the critical issues of privacy and security of the information, I propose

the following metric: availability of a privacy / security statement or terms of use. Due to the nature and purpose of public authorities’ web sites, this study will focus

upon the classical aesthetic dimension, “where beauty is determined by the extent to which form follows function” [13]. In consequence, the following three metrics for assessing aesthetics are proposed: a) Metrics for evaluating layout familiarity: the existence of a 2 or 3 columns layout, including a left sidebar, header and footer

b) Metrics for testing the color contrast and color brightness between text and background color: the ability to pass the W3C standard for color difference and the ability to pass the W3C standard for brightness difference For evaluating the ability to pass W3C recommended standard for color and brightness

difference, the AccessColor version 1.0 (accesskeys.org) was used. E-content was analyzed subsequent to provisions of Law no. 544/2001 regarding free

access to information of public interest and Law no. 52/2003 regarding the decisional transparency in public administration: a) Metrics for evaluating free access to public interest information: availability of public information department consisting of official (s) responsible for disseminating public information (full name, function, contact information, schedule); availability of public information request form and complaint form; availability of public information content such as: the laws governing the functioning and organization of the PA, the organizational structure, the Regulation of organization and functioning (ROF), the list of Programs and Strategies, the list of subordinated institutions, vacancies; availability of financial information such as the budget for the period 2005-2010 and the balance sheet for the period 2005-2009, availability of Wealth Declaration for 2009 and 2010

b) Metrics for evaluating other types of information: availability of contact information, meaning contact details such as phone number(s), fax, e-mail(s), address(es), general schedule, address map and audience schedule; availability of news and press releases and their updating frequency. E-participation: For assessing the level of decisional transparency and interactivity

provided by the public authorities’ web sites, the following nine metrics are proposed: a) Metrics for evaluating decisional transparency: availability of normative acts subject to public debate and their background notes; availability of the Annual Report


219

regarding decisional transparency for 2006-2009 b) Metrics for evaluating the web site’s interactivity: availability of online forms, e-forums, e-petitions, newsletters, spot polls, and social media.

3.3 The evaluation of the metrics

The metrics proposed for testing the quality of web sites are evaluated according to

[22] who proposed a model for evaluating metrics against measurement scale (nominal, ordinal, interval, ratio and absolute), measurement independence (the ability of a metric to offer the same result when measured by different people), automation (not requiring human measurement), simplicity (clarity of the metric’s definition) and the cost associating with using the metrics.

Table 2. The evaluation of metrics

Measurement Scale (MS)

Measurement Independence

(MI)

Automation (AU)

Simplicity (SI)

Cost (CO)

Ability to meet WCAG 1.0 Dichotomous + + - - Multilingualism Dichotomous + - + -

Average Downloading speed Ratio + + + - Browser compatibility Dichotomous + + + - Google SERP Position Nominal + - + - Number of incoming links Ratio + + + - Availability of a site map or a

A-Z index Dichotomous + - + -

Availability of a left quick navigation menu

Dichotomous + - + -

Availability of a home link on each page

Dichotomous + - + -

Availability of an internal search engine

Dichotomous + - + -

The number of filters of the internal search engine

Ratio + - + -

Availability of a privacy /security statement or terms of

use statement

Dichotomous + - + -

Existence of a 2 or 3 columns layout, including a left sidebar, header and footer

Dichotomous + - + -

Ability to pass the W3C standard for color difference

Dichotomous + + - -

Ability to pass the W3C standard for brightness

difference

Dichotomous + + - -

Availability of public information department

Dichotomous + - + -

Availability of public information request and

complaint form

Dichotomous + - + -

Availability of laws governing the functioning and organization of the PA

Dichotomous + - + -

Availability of organizational structure

Dichotomous + - + -

Availability of Regulation of organization and functioning

(ROF)

Dichotomous + - + -


220

Availability of Programs and Strategies

Dichotomous + - + -

Availability of list of subordinated institutions

Dichotomous + - + -

Availability of vacancies Dichotomous + - + - Availability of Financial

information Dichotomous + - + -

Availability of Wealth / Interest declarations

Dichotomous + - + -

Availability of contact information

Dichotomous + - + -

Availability of News and press releases

Dichotomous + - + -

Updating frequency of news or press releases

Interval + - + -

Availability of normative acts subject to public debate

Dichotomous + - + -

Availability of background notes

Dichotomous + - + -

Availability of the Annual Report regarding decisional transparency for 2006-2009

Dichotomous + - + -

Availability of online forms Dichotomous + - + - Availability of e-forums Dichotomous + - + - Availability of e-petitions Dichotomous + - + - Availability of newsletters Dichotomous + - + - Availability of spot polls Dichotomous + - + - Availability of social media Dichotomous + - + -

Most of the proposed metrics are dichotomous, three metrics are measured on ratio

scale and one is measured on interval scale. The metrics offer a high level of independence when measured by different people but they cannot provide a high level of independence when measured in different periods of time, due to the fact that web sites are improving continuously. Six metrics are measured using specialized software, thus resulting a high degree of automation. Except the automatically measured metrics, the majority of them are considered to be simple and clear. The cost of the metrics is low.

3.4 The evaluation grid

In order to evaluate the central public authorities’ web sites, I designed an evaluation

grid to cover all the above mentioned metrics. The dichotomous metrics were awarded with 0 or 10 points each, and the metrics measured on interval or ratio scale were awarded between 0 and 10 points, depending on each rationale:

- Previous researches, Akamai and Jupiter Research [5] established the “4 seconds” as an acceptable response time of retail web sites. In consequence, an average speed above 4 seconds is awarded 0 points, an average speed between 2 and 4 seconds is awarded 5 points and an average speed below 2 seconds is awarded 10 points.

- A web site should be in the first positions in Google SERP for queries containing as keywords the name of the authority. As demonstrated by [8] less than half of the search engines users look beyond the first page of results and the majority of the search engines’ users first click on organic results shown in the visible area of the search engine results page (SERP) (pp.1799-1800).

- The number of the incoming links is the most important internet positioning factor, influencing the web visibility of a web site [SERR09]. I have computed the average


221

number of incoming links of the entire sample as being 23450 and awarded 10 points for a number of incoming links greater than 23450, 5 points for a number of incoming links between 23450 and 2345 and 0 points for less than 2345.

- The level of complexity of the internal search engine was computed based on the average number of available filters among the sample of analyzed web sites. For more than one filter 10 points were awarded, for 1 filter, 5 points were awarded and for the absence of filters, 0 points were awarded.

- The average updating frequency of news and releases among the sample was found to be 23 posts per month. Thus a web site having an updating frequency of news and releases greater than 23 was awarded 10 points, a web site having an updating frequency of news and releases between 7 than 23 was awarded 5 points and 0 points were given for less than 7 posts per month. In order to compute the average of news and press releases, the whole April month was analyzed. The metrics were selected in such a way as to measure only the quantitative aspects of

various elements of the quality of public authorities’ web sites, thus eliminating the subjectivity of the person employed to complete the evaluation grid.

Table 3. The Evaluation Grid

1. Technical characteristics

1. 1 Accessibility metrics:

Points Weighting

scheme

(per cent)

1. Ability to meet WCAG 1.0 – 1st level priority 0 / 10 4 2. Multilingualism 0 / 10 2.5 3. Downloading speed 0 - 10 2 4. Browser compatibility 0 / 10 2 5. The web sites’ position in Google SERP 0 - 10 1 6. The number of incoming links 0 - 10 1

Total 1.1 12.5%

1.2 Navigability metrics: 7. Availability of a site map or a A-Z index 0 / 10 2 8. Availability of a left quick navigation menu 0 / 10 3 9. Availability of a home link on each page 0 / 10 2 10. Availability of an internal search engine 0 / 10 4 11. The level of complexity of the internal search engine 0 - 10 1

Total 1.2 12%

1.3 Security / Privacy metric: 12. Availability of a security/privacy statement/terms of use 0 / 10 5

Total 1.3 5.5%

1.4 Aesthetics metrics: 13. Existence of a 2 or 3 columns layout, including a left sidebar, header and footer

0 / 10 4

14. Ability to pass the W3C standard for color difference 0 / 10 3 15. Ability to pass the W3C standard for brightness difference 0 / 10 3

Total 1.4 10%

Total (1.1+1.2+1.3+1.4)

40%

2. E-content metrics 2.1 Free access to public interest information metrics: Availability of public information department consisting of official (s) responsible for disseminating public information

16. Full name 0 / 10 1 17. Function 0 / 10 1

18. Contact information 0 / 10 1


222

19. Schedule 0 / 10 1 Availability of public information request and complaint form 20. Request Form 0 / 10 2 21. Complaint Form 0 / 10 2 Availability of public information content 22. Availability of laws governing the functioning and organization of the PA 0 / 10 2 23. Availability of organizational structure 0 / 10 2 24. Availability of Regulation of organization and functioning (ROF) 0 / 10 2 25. Availability of Programs and Strategies 0 / 10 2 26. Availability of list of subordinated institutions 0 / 10 2 27. Availability of vacancies 0 / 10 2 Availability of Financial information 28.Budget for 2005-2010 0 / 10 2.5 29. Balance Sheet for 2005-2009 0 / 10 2.5

Total 2.1 25%

2.2 Other types of information metrics Availability of contact information

30. Contact details such as phone number(s), fax, e-mail(s), address(es) 0 / 10 1 31. Schedule 0 / 10 1 32. Address Map 0 / 10 1

33. Audience Schedule 0 / 10 1 34. Availability of News and press releases 0 / 10 3 35. Updating frequency of news or press releases 0 - 10 3

Total 2.2 10%

Total (2.1 + 2.2 )

35%

3. E-participation metrics

3.1 Decisional transparency metrics

36. Availability of normative acts subject to public debate 0 / 10 10 37. Availability of background notes 0 / 10 3 38. Availability of the Annual Report regarding decisional transparency for 2006-2009

0 / 10 16

Total 3.1 29%

3.1 Interactivity metrics 39. Availability of online forms 0 / 10 1 40. Availability of e-forums 0 / 10 1 41. Availability of e-petitions 0 / 10 1 42. Availability of newsletters 0 / 10 1 43. Availability of spot polls 0 / 10 1 44. Availability of social media 0 / 10 1

Total 3.2 6%

Total (3.1 + 3.2)

35%

TOTAL (1 +2 +3) 100%

4. Results and interpretation

This section of the paper presents the descriptive analysis of data and the overall

results of the evaluation grid applied to the analyzed sample.

4.1 Descriptive analysis

Descriptive analysis is used to present the following data, as a way of summarizing

and aggregating the results subsequent to the evaluation of the sample used in this study.


223

4.1.1 Descriptive analysis of data regarding the technical characteristics of web sites

a) Data for evaluating the level of accessibility

Ability to meet WCAG 1.0 (priority level 1):

73.33% of the ministries’ web sites met the priority level 1 of Web Content Accessibility Guidelines 1.0, meaning that approximately three quarters of the Romanian ministries’ web sites are accessible to people with disabilities. More than a quarter of Romanian ministries’ web sites cannot meet the necessities of all users, operating under different conditions.

Multilingualism: More than half of the Romanian ministries’ web sites offer the possibility of viewing part of the content in English Language. Only the Ministry of Foreign Affairs provides content in French and German. 46.67% of the Romanian ministries’ web sites provide content only in native language.

Fig. 1. Multilingualism (www.mae.ro)

Average downloading speed: The average response time of the Romanian ministries’ web sites is 2.5 seconds. 60% of the Romanian ministries’ web sites average downloading speed is below 2 seconds.

Browser compatibility: 85.72% of the Romanian ministries’ web sites support the most popular browsers, such as: Explorer, Mozilla, Opera, Safari and Chrome. The rest of the Romanian ministries’ web sites face difficulties of browsers compatibility.

Fig. 2. Browser incompatibility (Google Chrome) (www.edu.ro)


224

Google SERP position: 86.67% of the Romanian ministries’ web sites are placed on the first position when a query formed by the name of the authority is formulated. Thus, citizens can easily find the web address of the Romanian ministries when formulating queries to search engines. The number of incoming links: The average number of back links or the incoming links to the Romanian ministries’ web sites is 23450, resulting a high visibility on the world wide web.

b) Data for evaluating the level of navigability

Availability of a site map or an A-Z index: Two thirds of the Romanian ministries’ web sites, more specifically 66.67%, include a functional site map or A-Z index. A third of the Romanian ministries’ web sites do not encompass this facility for their users.

Fig. 3. Site map www.mcsi.ro

Availability of a left quick navigation menu: All Romanian ministries’ web sites have a left quick navigation menu.

Availability of a home link on each page: All Romanian ministries’ web sites have a


225

“home” link on every page, pointing to the index

Availability of an internal search

engine: 80% of the Romanian ministries’ web sites encompass an internal search engine, usually located in the right column of the layout. 58.33% of internal search engines have no filters for retrieving information. 33.33% of the internal search engines have between one and three filters and only 8.34% of the search engines contain more than three filters.

Fig. 4. Example of an advanced internal search engine (www.mapn.ro)

c) Data for evaluating the level of security and privacy

Availability of a privacy / security statement or terms of use agreement: Only 40% of the Romanian ministries’ web sites display privacy/ security statement or terms of use agreement on their web sites.

d) Data for evaluating aesthetics

Aesthetics metrics: 93.33% of the Romanian ministries’ web sites have a standard layout design, made of a 2 or 3 columns layout, including a left sidebar, the header and the footer. Only 46.67% of the Romanian ministries’ web sites have passed the W3C standard for color difference and brightness difference between the text color and background color.


226

Fig. 5. Standard layout design (www.madr.ro)


227

e) Data for measuring free access to public interest information metrics

Availability of public information department consisting

of official (s) responsible for disseminating public

information: 66.66% of the Romanian Ministries introduced a special public information department for a better compliance with Law no. 544/2001. 53.33% of the Romanian Ministries published on their web sites, the name and the function of the appointed persons with responsibilities in this area. Only 26.66% of the Romanian Ministries published on their web pages a schedule of the particular department.

Availability of public information request and

complaint form: 53.33% of the Romanian Ministries’ web sites provide special forms for requesting public information. They are available for download and print and they can be sent online, via the e-mail or printed and sent to the physical address of the Ministry. An administrative complaint form can be found on 46.66% of the Romanian Ministries’ web sites. It is also available for download and print.

Availability of public information content: All Romanian Ministries’ web sites contain programs and strategies for the current mandate. The laws governing the functioning and the organization of the ministries are available on all web sites. All Romanian Ministries published their organizational structure on their web sites but only 46.66% have published a Regulation of organization and functioning (ROF). 93.33% of the ministries’ websites include a list of subordinated institutions.

Availability of vacancies: All Romanian Ministries have implemented a special page within their web sites for displaying information on public vacancies in the organizations, but only 60% of the ministries had actually published job vacancies. The rest of the 40% web pages were empty


228

Availability of financial information: Almost a quarter of the Romanian Ministries (26.66%) published their budget for 2010 on their web sites. The same percentage of ministries published their budget for 2007 and 2008 and 20% of the ministries published their budget for 2005 and 2006. Citizens can find the ministries’ balance sheet for the financial year 2008 and 2009 in 2 out of 15 cases. 26.66% of the ministries published their balance sheet on 2007.

Availability of wealth declarations: 86.67 % of the Romanian Ministries’ web sites contain the declarations of wealth and declarations of interest for 2009. 66.67% of the Romanian Ministries’ web sites contain the declarations of wealth and interest for 2010.

f) Data for measuring other types of information

Availability of contact information: All Romanian Ministries’ web sites display contact information. In 93.33% cases, citizens can find contact information on a special “contact” page. A percentage of 33.33 of the contact web pages display also the general schedule of the public authority. The hearings schedule can be found in 6 out of 15 cases. The address map of the physical location of the ministries is available in 3 out of 15 cases.

Availability of News and press releases: All Romanian Ministries’ web sites display press releases and news on their home page. The frequency of posts varies between 1-7 posts per month to over 31 posts per month. Almost half of the ministries’ web sites (46.67%) update their press releases at least once a day.


229

g) Data for measuring decisional transparency metrics

Availability of normative acts subject to

public debate: Normative acts subject to public debate are displayed within all the ministries’ web sites, but just in 10 out of 15 cases (66.67%), these are accompanied by background notes.

Availability of the Annual Report

regarding decisional transparency: The Annual Report regarding the decisional transparency for 2006 could be found in 3 out of 15 cases (20%), the annual report regarding the decisional transparency for 2007 and 2008 could be found in 4 out of 15 cases (26.67%). Only two ministries published their annual report regarding the decisional transparency for 2009.

h) Data for measuring the web sites’ interactivity

Interactivity metrics: Only one of fifteen Romanian Ministries’ web sites offers the possibility of online consultancy through an online form. 20% of the Romanian Ministries’ web sites contain a visible link to a personal e-forum. Only two out of fifteen web sites have a special web page for submitting online petitions and offer the possibility of signing up for a newsletter. Only one Romanian ministry web site offers the citizens the possibility to answer spot polls of public interest information issues. More than a quarter of the Romanian Ministries’ web sites communicate with the citizens through social media (RSS, YouTube)

Fig. 6. Interactivity (www.mcsi.ro)


230

4.2 The evaluation grid

The overall results of the evaluation grid are presented in the table below:

Table 4. Overall Results of the Evaluation Grid Average Score Maximum Minimum 1. Technical characteristics 1. 1Accessibility metrics (out of 12.5 points) 8.83 71% 12.5 100% 5 40% 1.2 Navigability metrics (out of 12 points) 9.93 83% 12 100% 5 42% 1.3 Security / Privacy metric (out of 5 points) 2.20 40% 5.5 100% 0 0% 1.4 Aesthetics metrics (out of 10 points) 6.53 65% 10 100% 4 40%

Total 1. (out of 40 points) 28.33 (71%) 2. E-content 2.1 Free access to public interest information metrics (out of 25 points)

14.97 60% 18.5 74% 8.5 34%

2.2 Other types of information metrics (out of 10 points)

6.83 68% 9 90% 4 40%

Total 2. (out of 35 points) 23.66 (68%) 3. E-participation 3.1 Decisional transparency metrics (out of 29 points)

15.73 54% 29 100% 10 34%

3.1 Interactivity metrics (out of 6 points) 0.73 12% 2 33% 0 0% Total 3. (out of 35 points) 16.46 (47%)

TOTAL: 68.46 (68%)

4.2.1 Technical characteristics evaluation

Romanian ministries web sites score relatively high (8.83 average score out of 12.5 maximum points) for the accessibility metrics, meaning that:

- They can be accessed by disabled people and foreign stakeholders, - Their response time is below the “acceptable 4 seconds”, - They are compatible with the most popular browsers - They can easily be found on the internet using popular search engines and they have a good visibility on the web, materialized in thousands of incoming links pointing to their web pages.

- Romanian ministries web sites recorded the highest score for navigability (9.93 average score out of 12 maximum points), concluding that:

- Citizens can easily navigate through the information structure due to an organized layout which also includes a left quick navigation menu.

- Navigation facilities such as site maps, A-Z indexes and internal search engines are provided. In respect to the level of security and privacy, Romanian ministries web sites recorded

a below average score (2.20 out of 5.5 maximum points). Romanian ministries web sites score above the average for the analyzed aesthetics metrics (6.53 out of 10 maximum points), providing citizens a familiar layout and readable information.

Summarizing the average score registered for technical characteristics metrics, Romanian ministries web sites put a great emphasis on factors such as: accessibility, navigability and aesthetics, all capable of influencing the citizens’ online experience with the web site.


231

Romanian ministries web sites score above the average at displaying public interest information to citizens (14.97 average score out of 25 maximum points) although Law no. 544/2001 stipulates free and unimpeded access of citizens to any information of public interest. There is a lack of transparency especially when it comes to displaying financial information such as budgets and balance sheets (Graph 12).

Romanian ministries web sites score below average for decisional transparency (15.73 average score out of 29 maximum points). Although normative acts subject to public debate are displayed within all web sites, there is a lack of transparency regarding the feedback of the ministers to citizens’ recommendations. Public authorities failed at publishing the Annual Report regarding decisional transparency, which contains among various elements, the number of recommendations received and their inclusion in final normative acts and decisions made. Moreover, the Romanian ministries web sites score very low for the interactivity metrics analyzed in this study, such as the availability of online forms, e-forums, e-petitions, newsletters, polls and social media (0.73 average score out of 6 maximum points).

5. Conclusions

This paper studies the quality of the Romanian central public authorities’ web sites,

materialized in their capacity to serve all Romanian citizens and other stakeholders, despite their possible disabilities, their level of computer literacy or different technology barriers that may interfere in the proper functioning of a web site.

The first factor analyzed and considered to exercise a huge influence upon the quality of websites, was formed by the technical characteristics of the websites, such as accessibility, navigability, security/privacy and aesthetics.

The other two factors discussed in this paper are: e-content, the availability of public interest information and other types of useful information, and e-participation, the level of decisional transparency and the level of citizens’ involvement in the decision making process.

The three factors influencing the overall quality of a website were measured through a series of 44 proposed metrics: metrics for measuring the accessibility of web sites (the ability to meet WCAG 1.0 first level priority, the existence of multilingualism, the web sites’ downloading speed, browser compatibility, the web site’ ease of relocation on the web, the website’s visibility on the web), metrics for measuring navigability (the availability of a site map or a A-Z index, the availability of a quick navigation menu, the availability of a home link on every page, the existence of an internal search engine and its level of complexity), metrics for measuring the privacy and security level (the availability of a privacy/security statement or a terms of use agreement), metrics for evaluating aesthetics (the level of familiarity layout, information readability), metrics for measuring free access to public interest information (20 metrics analyzing the provisions of Law no. 544/2001 regarding free access to information of public interest), metrics for measuring decisional transparency (3 metrics analyzing the provisions of Law no. 52/2003 regarding the decisional transparency in public administration) and finally, metrics for evaluating the web site’s level of interactivity with its audience.

The sample used for evaluating the Romanian central public authorities’ web sites was formed by the total population of Romanian ministries.

The results of the study highlighted that Romanian central public authorities put a great emphasize on the accessibility characteristics of their web sites, making the accessible for disabled citizens and even foreign citizens, delivering a rapid response time, being compatible with the most popular browsers and being easy to find and relocate on the web. However, there is still room for improvement in regards to navigability, mainly the


232

complexity of the internal search engine for retrieving the huge amounts of information. Romanian central public authorities implement a similar layout design, making them familiar to the general public and thus increasing usability.

When it comes to free access of information and decisional transparency, Romanian central public authorities’ web sites could have performed better, especially when there are laws governing these two principles. Nevertheless, Romanian central public authorities’ web sites scored low for interactivity, being unable to perform a multi-directional communication between the authority and citizens, between citizens and authority and between citizens themselves.

Although this study increases the awareness of the quality shortcomings of Romanian central public authorities’ websites, I would like to acknowledge its limitations. First of all, the proposed metrics were designed in such a way as to offer objective evaluation of the web sites’ characteristics, thus the attitudes of citizens were not involved. Second of all, the evaluating grid was filled in by an experienced internet user, thus the results might have been lower in case of an inexperienced internet user. Third, the sample used in this study was formed by Romanian ministries’ web sites, thus excluding other other specialized agencies held in the subordination of the government or the ministries. Last, more complex metrics could be developed such as: the ability to meet WCAG 1.0 second and third level priority, the degree of completeness of web sites’ content in foreign language, the degree of information retrieval of internal search engines, citizens’ perceptions towards the web sites’ aesthetics etc.

Despite its limitations, this paper manages to offer an objective framework of metrics for evaluating the quality of public authorities and the presented study-case provides considerable insights for overall quality issues faced by Romanian central public authorities’ web sites. Its importance lays in the fact that quality repercussions are redounded upon the final users of the web sites, the citizen and other stakeholders, whereas the objective of implementing a public web site is to serve and meet citizens needs.

References

[1] A. Salim and A. Bouazza, “An evaluation study of the navigation and search systems on two academic websites and Google,” The International Information & LibraryReview, Vol. 2, No. 1, pp. 50-61, March 2010. [2] B. Y. Iakov, V. Shankar, F. Sultan and G. L. Urban, “Are drivers and role of Online Trust the same for all the web sites and consumers? A large scale exploratory empirical study”, Journal of Marketing, Vol. 69, No. 4, pp. 133-52, April, 2005. [3] C. V. Luque, C. D. Kloos, J. A. Fisteus, L. Á. Álvarez, “Web Accessibility Evaluation Tools: A survey and some improvements,” Electronic Notes in Theoretical Computer Science, Vol. 157, No. 2, pp. 87-100, May 2006. [4] D. R. Danielson, “Web navigation and the behavioral effects of constantly visible site maps,” Interacting with Computers, Vol. 14, No. 5, pp. 601-618, October 2002. [5] F. J. M. Gonzalez and T. M. B. Palacios, “Quantitative evaluation of commercial web sites: an empirical study of Spanish firms,” International Journal of Information Management, Vol. 24, No. 4, pp. 313-328, August 2004.


233

[6] F. C. Gutierrez, C. Loucopoulos and R. W. Reinsch, “Disability-accessibility of airlines’ Web sites for US reservations online,” Journal of Air Transport Management, Vol. 11, No. 4, pp. 313-325, July 2002. [7] H. R. Hall and P. Hanna, “The impact of web page text-background color combinations on readability, retention, aesthetics, and behavioral intention,” Behaviour & Information Technology, Vol. 23, No. 3, pp. 183-195, 2004. [8] N. Hochstotter and D. Lewandowski, “What users see – Structures in search engine results pages,” Information Sciences, Vol. 179, No. 12, pp. 1796-1812, 2009. [9] I. Ivan and B. Vintila, “Managementul aplicatiilor informatice orientate spre cetatean,” Economie teoretica si aplicata, Vol. 17, No. 4, pp. 45-69, 2010. [10] P. T. Jaeger, “Assessing Section 508 compliance on federal e-government web sites: A multi-method, user-centered evaluation of accessibility for persons with disabilities,” Government Information Quarterly, Vol. 23, No. 2, pp. 169-190, 2006. [11] P. Kotler, D. Jain and S. Maesincee, Marketingul in era digitala: O noua viziune despre profit, crestere si innoire, Meteor Business, 2009, initial published as “Marketing moves. A new approach to Profits, Growth and Renewal”, Harvard Business Press, 2002. [12] J. M. Kuzma, “Accessibility design issues with UK e-government sites,” Government Information Quarterly, Vol. 27, No. 2, pp. 141-146, March 2010. [13] T. Lavie and N. Tractinsky, “Assessing dimensions of perceived visual aesthetics of a web site,” International Journal of Human-Computer Studies, Vol. 60, No. 3, pp. 269-298, March 2004. [14] J. Palmer, “Web site usability, design and performance metrics,” Information System Research, Vol. 13, No. 2, pp. 151-167, 2003. [15] E. Panopoulou, E. Tambouris and K. Tarabanis, “A framework for evaluating web sites of public authorities,” in Aslib Proceedings New Information Perspectives, Vol. 60, No. 5, 2008, pp. 517-546, 2008. [16] D. Robins and J. Holmes, “Aesthetics and credibility in web site design,” Information Processing & Management, Vol. 44, No. 1, pp. 386-399, January 2008. [17] M. E. Searson and M. A. Johnson, “Transparency laws and interactive public relations: An analysis of Latin American government Web sites,” Public Relations Review, Vol. 36, No. 2, pp. 120-126, 2010. [18] G. Sreedhar, A. A. Chari, V. V. Venkata Ramana, “Measuring quality of web site navigation,” Journal of Theoretical and Applied Information Technology, Vol. 14, No. 2, pp. 80-86, 2010. [19] S. Yuquan, “The accessibility of Chinese local government Web sites: An exploratory study,” Government Information Quarterly, Vol. 24, No. 2, pp. 377-403, April 2007.


234

[20] D. Sloan, P. Gregor, P. Booth and L. Gibson, “Auditing accessibility of UK Higher Education web sites,” Interacting with Computers, No. 14, Issue 4, pp. 313-325, July 2002. [21] G. A. Smith, “Applying evaluation criteria to New Zealand government websites,” International Journal of Information Management, Vol. 21, No. 2, pp. 137-49, 2001. [22] A. Stefani and M. Xenos, “Meta-metric evaluation of e-commerce-related metrics,” Electronic notes in Theoretical Computer Science, Vol. 233, pp. 59-72, 27 March 2009. [23] A. L. Sutherland, B. Wildemuth, M. K. Campbell and P. S. Haines, “Unraveling the Web: An Evaluation of the Content Quality, Usability and Readability of Nutrition Web Sites,” Journal of Nutrition Education and Behavior, Vol. 37, No. 6, pp. 300-305, November-December 2005. [24] N. Tractinsky and O. Lowengart, “Web-store aesthetics in e-retailing: A conceptual framework and some theoretical implications,” Academy of Marketing Science Review, Vol. 11, No. 1, pp. 1-18, 2007. [25] L. G. Urban, C. Amyx and A. Lorenzo, “Online trust: State of art, new frontiers, and research potential,” Journal of Interactive Marketing, Vol. 23, Issue 2, pp. 179-190, May 2009. [26] P. Verdegem and G. Verleye, “User-centered E-Government in practice: A comprehensive model for measuring user satisfaction,” Government Information Quarterly, Vol. 26, No. 3, pp. 487-497, July 2009. [27] Y. Wang, W. Dai and Y. Yuan, “Website browsing aid: A navigation graph-based recommendation system,” Decision Support Systems, Vol. 45, No. 3, pp. 387-400, June 2008. [28] J. J. Zhao, S. Y. Zhao, “Opportunities and threats: A security assessment of state e-government websites,” Government Information Quarterly, Vol. 27, No. 1, pp. 49-56, January 2010. [29] “Transparenta decizionala in Romanian”, Raport de monitorizare cu privire la aplicarea si respectarea Legii 52/2003 privind transparent actului decisional la nivelul administratiei publice, (Accessed on 12th May 2010), Available online at: http://www.apd.ro/files/publicatii/ Raport_de_monitorizare_cu_privire_la_aplicarea_si_respectarea_Legii_52_2003.pdf [30] “Complete List of Web Accessibility Evaluation Tools”, (Accessed on 14th May 2010), Available online at: http://www.w3.org/WAI/RC/tools/complete [31] “Numarul persoanelor cu handicap la data de 31 decembrie 2009”, (Accessed on 16th May 2010), Available online at: http://www.anph.ro/tematica.php?idt=13&idss=41 [32] “Ghid privind realizarea paginilor web pentru autoritatile si institutiile administratiei publice centrale si locale” (Accessed on 12th May 2010), Available online at: http://www.mcsi.ro/Minister/Domenii-de-activitate-ale-MCSI/Tehnologia-Informatiei/Ghiduri-IT-(1)/Realizarea-paginilor-web-pentru-autoritatile-si-in


235

[33] Akamai and Jupiter Research Study: http://www.akamai.com/html/about/press/releases/ 2006/press_110606.html

Author

Claudia ICONARU has graduated the Faculty of Business Administration (in foreign languages) from the Bucharest Academy of Economic Studies in 2007. She obtained her Master degree in ‘Business Administration’ in 2009. Currently, she is a student in final year at the Faculty of Political Studies from The National School of Political Studies and Public Administration and a PhD Candidate in the field of Marketing at the Bucharest Academy of Economic Studies. Her research areas include: online consumer behaviour, e-

commerce and e-government.

test

Documents