Evaluation and Impact Estimation at the OFT and its applicability to CREWNatalie Timan

Deputy Director, Office of Fair Trading

13 March 2013

1

Introduction

●Contents:

- Evaluation objectives: why do we measure impact?

- Types of evaluation within the OFT

- Essential features of an Impact Assessment

- Methodology

- Two worked examples

- Application to CREW

- Ex-post evaluation

2

Evaluation objectives

●The aim of our evaluation work is two-fold:

- External accountability: to evaluate whether the OFT delivers its objectives and does so cost effectively and to demonstrate this externally;

- Internal management and prioritisation: to inform future OFT work regarding how we prioritise, conduct and follow-up our work to ensure we maximise our impact. This is not an internal evaluation of our procedures or efficiency, but an assessment of the accuracy of our predictions about markets and the effectiveness of our interventions.

3

Types of evaluation

● Impact estimation: ex-ante estimation of anticipated impact usually undertaken shortly after the OFT’s intervention.

- Direct financial savings for consumers only – no wider benefits.

●Ex-Post evaluations: ex-post estimation of actual impact usually undertaken some years after the OFT’s intervention.

●Application to CREW: ex-ante estimation may be the more useful type of evaluation given:

- Cost effectiveness;

- Data limitations;

- Time constraints.

4

Welfare standard

●Both types of evaluation look at benefits to consumers as the OFT has a Consumer Welfare standard.

- Typically estimate effect on prices and output as a proxy for consumer welfare

●OFT does not estimate potential impact of its decisions on business as part of positive impact estimation.

● Impact estimation looks at quantifiable direct financial benefits to consumers:

- Decrease in price;

- Monetized improvements in quality, range or service;

- Monetized time savings;

- Benefits that consumers gain from making better informed choices about what goods to purchase.

5

Essential features of OFT’s Impact Estimation I*

●Assessed on regular (annual) basis during next year.

●Relatively undemanding in cost and time, utilising data from the original interventions and/or simple default assumptions.

●Estimates are performed using ex-ante data.

●Assumed that no intervention can have a negative impact.

●Estimates are deliberately ‘conservative’.

* These features were presented by Professor Steve Davies to the OECD on 25 February 2013

6

Essential features of OFT’s Impact Estimation II

●Estimates in terms of static consumer benefits.

●Annual moving averages employed.

● ‘Point’ estimates, rather than a range of plausible values.

●Covers mergers, cartels, abuse, market studies and consumer enforcement. Does not cover advocacy of competition to other government departments.

●Deterrence impact not included in estimate.

●Possible beneficial effect of competition policy of productivity and innovation (and growth) not estimated.

7

Methodology: assumptions

●To estimate the impact of each individual case intervention, information is required on:

- The size of affected turnover;

- The price increase removed or avoided;

- The length of time the increased price would have prevailed absent the intervention.

●An adjustment could be made for the deadweight loss (i.e. Surplus lost by consumers who are deterred from making any consumption).

● Ideally use any case-specific estimates from the case team; if not default assumptions required:

8

Methodology: default assumptions

 

Mergers Cartels Abuse

Affected turnoverTurnover of

affected goodsTurnover of

affected goodsTurnover of

affected goods

Price effectGUPPI (or 3-5%) plus deadweight

loss estimate

10-15% 10-15%

Duration (yrs) 2 6 6

9

Two recent examples: An Abuse of dominance caseSummary of total consumer detriment two year period Product A

£mFirm 1 50

Firm 2 20

Firm 3 30

Firm 4 40

Total harm over 2 year period 140

Total harm over 2nd year alone 93

Assumed duration (years) into future absent OFT intervention ie we think it would have continued for further 2 years

2

Total consumer detriment averted in following 2 years (deflated and discounted to prices at decision date) 291.96

10

● The figures represent the extra turnover derived from the implementation of various initiatives over a two year period, borne directly by the UK consumer.

 

● This is calculated by multiplying the volumes sold by the amount of increment, bearing in mind that the price increases were maintained and carried over into the following years. So, for example, if the price of one product was increased by £5 in year one and £5 in year two, then the total benefit would be £15.

Two recent examples: A cartel case

Turnover and Profit

(£m) Firm 1 Firm 2 Firm 3 Firm 4 Firm 5 Firm 6 Total £mNet Fees £5 £2 £3 £10 £20 £10 £50Net fees

converted from year of cartel to prices at year of decision £5.1 £2.0 £3.1 £10.0 £20.4 £10.2 £51

Saving in net fees from OFT interventi

on 15% £0.8 £0.3 £0.5 £1.5 £3.1 £1.5 £8Savings in net fees from OFT intervention over 3

years 3 £2.2 £0.9 £1.3 £4.4 £8.9 £4.4 £22

11

• Attribution of impact is justified by:

• Leniency programme was used

• Evidence that cartel met after application for leniency

• Assume 15% reduction in net fees

• Assume 3 years duration. Duration is scaled down from rule of thumb of 6 years on account of:

• Instability of cartel

• Some participants were not adhering to rules of cartel

• Total impact: £22m

Impact estimation

2009-10 Benefits

2010-11 Benefits

2011-12 Benefits

Competition enforcement

£84m £83m £151m

Merger control £125m £90m £8m

Market studies and reviews

£107m £117m £207m

Consumer protection enforcement

£42m £36m £35m

Total estimated benefits

£359m £326m £402m

Benefit/Cost ratio

7 7 8

12

● The figures presented above are 3-year rolling averages.

Application to CREW I

●Rules of thumb

- Affected turnover

- Discount rate

- Cartel overcharges in developing countries

- Price rise in abuse cases

- Alternatives to merger simulation

- Duration

13

Application to CREW II

●Deterrence

●Producer surplus: barriers to entry

●Competition, innovation and growth

- Output = f [natural resources, capital, labour] technology, management

- Prioritise interventions that lift restrictions on e.g.:

• Land (natural resource);

• Finance (capital)

• Markets associated with labour participation and higher education (labour)

• Innovation intensive sectors e.g. Pharmaceuticals

• Tackling public restrictions on competition -> managers not sheltered from impact of competition

14

Ex-Post Evaluation

● What is ex-post evaluation?

- An ex-post assessment of the actual impact of our interventions in enforcement cases.

- Primary purpose: measure the net benefits to consumers of any intervention. Secondary purpose: examine wider benefits (usually unquantifiable), such as the effect on consumer confidence and deterrence.

- Typically compares the quantified net benefit to the cost incurred by the OFT in the initial intervention.

● How do we evaluate?

- Can involve: (i) desk-based research on prices and other market data, (ii) surveys of consumers, businesses and other stakeholders.

- Usually conduct two to three evaluations a year. Can be conducted immediately after a case is completed, but typically there is a delay, potentially up to several years, whilst recommendations are implemented.

15

Methodology:Ex-post evaluation

● Formal techniques:

- Simulations and structural models

- Event studies

- Difference in Difference techniques

● Reality:

- Identify changes in market characteristics (price, quality, choice etc)

- Identify changes in consumer behaviours (shopping habits, awareness and understanding, product selection)

- Link those changes to actions or recommendations – are the changes attributable to us?

- If possible monetise the changes that are attributable to authority

16