Measuring the impact of ICT for development

CSTD 2010-2011 Inter-sessional Panel, Theme 116 December 2010, Geneva

Sheridan Roberts, consultant to UNCTAD

sheridanroberts@infosocietystats.com

This presentation will cover…

Why measuring the impact of ICT for development is important

… and why it is difficult

The complexity of ICT impacts

The role of the Partnership on Measuring ICT for Development

Measurement approaches

Empirical evidence for the impacts of ICT

Issues for consideration

Why measuring the impact of ICT for development is important

Arguably, ICT has the potential to directly and indirectly change the lives of much of the world’s population

how individuals live, learn, work and interact

processes of business and government

the quality of the natural and built environment

poverty alleviation via direct and indirect effects.

The Tunis Commitment from the second phase of WSIS expressed a strong belief in the benefits that ICT can bring to humanity and linked them to fulfilment of the MDGs.

Why measuring the impact of ICT for development is important

From the 10 WSIS targets, important impact areas can be inferred

ICT access for poor and rural communities

ICT in education

ICT networks in health

availability of e-government services

electronic access to information and knowledge.

Comparable and reliable ICT statistics are required for designing, implementing, monitoring and evaluating ICT policies.

Measures of ICT impact are possibly the most important … but also the most challenging.

Why measuring the impact of ICT is difficult

Any impact is hard to measure

even with an obvious relationship between variables, causality can rarely be proven.

The impact of ICT is even more difficult to measure because

there is a variety of ICTs and they have different impacts

… in a variety of contexts

indirect impacts of ICT as a general purpose technology.

What do we mean by impact? There are qualitative and quantitative differences between macro-level and micro-level impacts, direct and indirect impacts, short and long term impacts etc.

Why measuring the impact of ICT is difficult

It is not possible to categorise most impacts as simply social, economic or environmental

there is a complex web of relationships.

There are important but varying roles of context factors such as level of ICT infrastructure, human capital, government regulation, e-government.

Lack of data and data collection infrastructure.

In an international context, there are extra challenges in achieving comparable measures across countries.

individuals & communities

economic performance

citizen participation

privacy & security

innovation & research

health

employment

Existing ICT infrastructure Level of education, skills, income

Government ICT policy & regulation, e-government

Economy Society

Environment

poverty alleviation

education

The Partnership on Measuring ICT for Development

The Partnership was formed after the Geneva phase of WSIS to further international evaluation and benchmarking. Its members are 11 international organisations (8 UN including UNCTAD; plus Eurostat, OECD and the World Bank).

Partnership members are involved in various activities directed towards achieving internationally comparable and reliable ICT statistics, including

developing and maintaining a core list of ICT indicators

compiling and disseminating ICT data

providing technical assistance to developing countries through manuals and workshops.

Role of the Partnership in measuring the impact of ICT

The Tunis Agenda specifically noted the Partnership and its role in the measurement of ICT impact.

To date, the Partnership has not developed either ICT impact indicators nor measurement methodologies

that reflects the difficulties involved in such tasks.

However, the Partnership has established a Task Group on Impacts, led by the OECD. Its objective is

“… to give an overview of the economic and social impacts of ICTs, how these impacts can be measured and what the data requirements are … and to provide countries with a set of statistical and analytical tools … to carry out their own assessment of ICT impact.”

Its terms of reference recognise both economic and non- economic impacts and a variety of methodologies and data sources.

Measurement – Analytical techniques

Can be performed at the macro-economic, sectoral and micro-economic (firm) level

econometric modelling using regression, growth accounting and input-output analysis.

The usual objective of an ICT impact analysis is to examine the relationship between ICT and productivity, economic growth or employment

ICT includes the ICT sector and ICT demand.

Strengths include objectivity, use of existing data.

Main weaknesses are model assumptions and dependence on the reliability and availability of input data.

Measurement – Statistical surveys

Can be a direct source of data on ICT impacts; inputs to other studies e.g. analytical.

Surveys cover households, individuals, businesses, other entities e.g. government organisations.

Strengths

strong international statistical infrastructure

reliability of output is reasonably subject to control

data can be cross-classified by various characteristics

perceptions questions provide data on causal links.

Weaknesses

Expensive to do properly

perceptions questions lack objectivity.

Measurement – Other main approaches

Case studies

source of much of the data on ICT impacts

a wide range of approaches and data sources

flexibility of scope but findings not generalisable.

Panel studies

longitudinal data, may be survey-based

can provide good baseline data and account for time lags

expensive and may suffer from unit attrition over time.

Controlled experiments

can show causality but difficult for complex topics.

Use of administrative data

some data e.g. trade readily available for many countries

not usually designed for statistical purposes.

Empirical evidence for the impacts of ICT – economic performance

Increased economic performance provides potential for poverty alleviation.

Macro-economic and sectoral impacts

increase in size and productivity of ICT sector and related industries; appears to be the main source of benefit for developing countries

ICT investment across the economy contributes to capital deepening and leads to increase in economic growth and labour productivity; there may be a ‘critical mass’ limitation for many developing countries

multi-factor productivity growth across the economy, via role of ICT in helping firms innovate and increase overall efficiency, there is also likely to be a ‘critical mass’ limitation for many developing countries.

Contribution of ICT investment to GDP growth, percentage points

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Australia

United States

Sweden

Denm

ark

United Kingdom

Belgium

Canada

Japan

New

Zealand

Spain

Netherlands

Portugal

Finland

Ireland

Greece

Italy

Germ

any

France

Austria

%

1990-95 1995-2003 (1)

Source: OECD

Empirical evidence – economic performance

Firm level studies

most studies are for developed countries; they have generally found a positive relationship between use of ICT and labour productivity

importance of complementary factors such as skills and organisational change

impacts may be different for developing countries, with greater impact possible from less sophisticated ICTs, such as computers.

Case study evidence

especially on the benefits of mobile phones to small and micro businesses in developing countries.

Firm level study in Thailand

A 2007 study by UNCTAD and the Thai National Statistical Office analysed the impact of ICT on labour productivity in urban manufacturing firms with 10 or more employees in Thailand.

An econometric analysis, controlling for non-ICT factors, showed that firms with a combined use of computers, the Internet and the Web had on average 21% higher sales per employee than firms without any of these ICTs.

The ICT with the greatest single association with increased sales per employee was computers (14%).

Empirical evidence – economic performance

Various studies on the impact of broadband

have found a relationship between broadband penetration and economic growth

but broadband is not as available in developing countries …

therefore critical mass may be insufficient.

There has been relatively little focus on negative economic impacts

privacy and security impacts

systems failures, data loss, disclosure of data

loss of productivity through use of ICT in work time.

Empirical evidence – employment

ICT’s role in the creation of employment and self- employment opportunities for poverty alleviation

direct impacts through growth of the ICT sector and ICT- using industries, especially telecommunications services for developing countries, ICT manufacturing and IT-enabled services for a small number of developing countries

indirect impacts through multiplier effects.

Case study evidence in developing countries indicates positive impact of ICT skills on employment prospects.

Possible negative impacts through loss of employment as tasks are automated.

Empirical evidence – education

There is considerable policy interest in the benefits that ICT can bring to education

however, impact studies show mixed results

measurement is complicated.

Potential benefits from ICT include

tools for teaching and learning processes

provision of skills needed in an information society

improved attitudes/motivation to learning

development of teachers’ technology skills

increased access of the community to adult education and literacy.

Empirical evidence – education

Highly controlled and specific studies of the impact between ICT use and educational outcomes show positive (causal) impacts in certain conditions.

PISA studies of performance of 15-year olds, co- ordinated by OECD and conducted triennially show positive impacts of ICT use in some conditions

home use of computers but not necessarily use at school

quality of ICT use (confidence, length of experience) more important than quantity

other factors important – capital (social, economic, cultural and technological).

Students’ use of ICT and average science scores, PISA 2006

470

480

490

500

510

520

530

Bottom quarter Secondquarter

Third quarter Top quarter

Index of ICTInternet/entertainment useIndex of ICTprogram/software use

Source: OECD

Empirical evidence – health

Could expect positive impacts from

use of electronic health records (EHR)

telemedicine

m-health (mobile health)

ICT as enabling complex and networked medical equipment

Internet as a source of information.

Evidence is generally from case studies and indicates positive impacts

benefits to individuals in efficacy of health care, convenience

cost-saving benefits to health systems

interoperability (between EHR and other clinical and non- clinical systems) appears to be a prime driver of benefits

WHO has plans to establish indicators for monitoring e-health and assessing the impact on health systems.

Empirical evidence – citizen participation, individuals, communities

ICT can facilitate democratic processes and increase participation by citizens

by various means including facilitation by e-government.

It is clear that ICT can change many aspects of individuals’ lives

there are measurable ‘intermediate’ impacts (e.g. Internet commerce, time spent using ICT)

benefits may be greater for the poor and other disadvantaged.

There are obvious negative as well as positive impacts e.g. on children, Internet fraud.

Perceived impacts (positive and negative) on individuals as measured by household surveys.

Case study evidence of impacts on communities e.g. information and communication, social capital.

Empirical evidence – environment

There are positive and negative links between ICT and the environment

potential to improve efficiency of energy-using processes and equipment

facilitation of dematerialization (e.g. teleworking)

essential role in climate change monitoring and modelling, dissemination of information, and administration of carbon pollution reduction schemes

BUT negative impacts of ICT from energy usage and greenhouse gas (GHG) emissions from ICT use, manufacturing and transport of ICT products

pollution from disposal of e-waste.

Some impacts from scientific measurement and modelling with inputs from a range of sources.

Contribution of ICT to GHGs

Estimates from 2007 and 2008 indicate that the ICT sector and ICT products are currently responsible for about 2% of global GHG emissions.

High growth in ICT penetration and increases in processing power indicate that this contribution will grow further.

Issues for consideration – prioritisation

Given the range of ICT impacts and the fairly low availability of evidence on impacts, should the measurement of impact in certain areas be given higher priority than others in the years leading up to 2015?

This question may be considered in conjunction with considerations of feasible and affordable data collection work.

Issues for consideration – what can be done to extend indicators of ICT impact?

Given the importance of relevant and internationally comparable data for impact studies, what can governments, development partners and international organisations – especially those that are members of the Partnership – do to extend ICT impact indicators?

Examples may include setting statistical standards, accelerating the building of capacity for the production of relevant statistics and allocating sufficient funds to undertake surveys.

Issues for consideration – policy imperatives

What types of impact studies are the most useful, for example:

extend macro-economic analysis to developing countries using methodologies applied by the OECD

extend the measurement of firm level impacts to more developing countries

consider the use of perceptions questions on surveys of business and household use of ICT

extend the PISA programme to more developing countries to shed light on the impact of ICT on learning outcomes of 15-year olds

attempt to standardise and extend case studies and controlled experiments.

Issues for consideration – policy awareness raising

What can be done to raise awareness among different stakeholders about the need for the measurement of impact of ICT?

How can the CSTD contribute?