Developing approaches to ex-post assessment of regulatory change impacts

at the farm level: prototype study

John Powell

Jane mills

CCRI

University of Gloucestershire

Aims and objectives

Overall aim: • develop a methodology to improve understanding of

the systemic and other factors influencing impacts arising from the implementation of regulatory change

Objectives:• Understand the form and magnitude of errors in

current impact assessments• Identify potential improvements in the general

methodology in order to reduce errors• Develop a framework methodology for the conduct of

ex-post assessment of regulations

Role of IA in the policy process

1. What does the Literature say?

2. Selection of regulations

3. Collection of empirical data

4. Comparison of ex-ante and ex-post impact assessments

5. Identification of causal factors accounting for differences

6. A framework methodology for undertaking ex-post impact assessment of regulatory change.

Research Design

Evaluating regulatory impact assessments

Reasons for ex-ante /ex-post differences– Use of worst case

scenarios

– Strategic behaviour

– Compliance issues might alter costs

– Incorrect assumption

Recent studies• Significant variation in

quality• Limited policy options

considered• Limited impacts

explored• Limited utility

Selection of Regulations

•Criteria for selection

•Nitrate Pollution Control regulations 2008

•Agricultural waste Regulations 2006

Comparison of ex-ante and ex-post impacts: Nitrate Regulations 2008

Costs ex-ante:– £655 - 1,009 million

Total present value discounted over 20yrs at 3.5%

• Benefits– £28 - 274 million TVP

over 20 years

• Comparisons between measures

Costs ex-post:– £300 - 900 million

concentrated in the short-term period after 2008

• Benefits– No monetary estimate,

potentially long-term and small

• Main differences related to behavioural changes

Ex-ante/ex-post differences – Nitrate Regulations

Category Magnitude of differenceTotal costsTotal benefits

LowUnclear - Low

Compliance costs: Storage capacityCovered yards/run-off reduction measuresReduction in stocking rate Additional spreading costsSpreading techniques

MediumLowUnclearUnclearLow

Compliance costs:Moving and making more efficient use of slurryReduction in fertiliser use

HighUnclear

Compliance costs High

Increased farm labour/time costs High

Dairy Derogation Unclear – possibly high.

Planning costs Low

Implementation and enforcement Costs Unclear – possibly low

Small firms impact test Low

Competitiveness Unclear – possibly high

Comparison of ex-ante and ex-post impacts: Agricultural Waste Regulations 2006

Ex-ante:

• Costs:

– Haz. Waste £28.7 – 69.8 m/yr

– On-farm £177-430/yr

– Off-farm 1 – 12% income

• Benefits – high, linked to improvements in soil and water quality

• Comparisons with ex-post measures

Ex-post:

• Costs:

– Haz. Waste £35.4 m/yr

– On + off farm £124-750 or ave. £219/farm/yr.

• Benefits – not estimated – Environmental possibly lower; indirect benefits higher

• Main differences due to high implementation costs; farmer behaviour; level of re-use and recycling

Ex-ante/ex-post differences – Waste Regulations

Category Magnitude Reason for differenceEnvironmental BenefitsAmmonia; Loss of fishery value; Bathing waters pollution; Direct impact of faecal pathogens to soil/water

HighEnvironmental impacts overstatedMany practices altered to comply with farm assurance schemes.No mention of emissions from burning plastic and other wastes.

ComplianceOn farm hazardous waste storageOn-farm disposal/recovery Off-farm disposal/recovery

LowMost farmers already in complianceLargest waste stream (plastic) is recycled. Farmers not disposing of all wastes.

Implementation costsThe Environment Agency (EA) will face increased costs in administration, monitoring and enforcement.

HighCommunication costs not considered – in particular those of associated stakeholders outside of government.

Costs to the court systemEstimated 11 court prosecutions involving the agricultural sector.

HighLack of waste focused inspections. EA prefers light touch approach

Small firms Unclear No consideration of significance of plastics recycling.

Exploring reasons for the differences in estimates

Causal Category Description

Assumptions Differences caused by assumptions

Methodological Difference caused by ineffective or inaccurate methods

Technological Unforeseen changes in technology; anticipated changes that did not develop, or not had an impact

Sector Knowledge Erroneous or imperfect understanding of the sector

External drivers Failure to account for the impact of external forces

Implementation support Erroneous accounting

Systemic Failure to carry out actions or undertake studies required; or, specific ‘ways of doing’ that cause errors.

Exploring the differences in estimates: Nitrate regulations

Assumptions•Derogations•Reduction in stocking rates•Spreading of slurry

Methodological•Costs annualised over 20 yrs•Increased time to manage nitrates

Sector knowledge•Renting land•Variable impacts

Technological•Anaerobic digestion

External drivers•Fertiliser prices

Implementation support•Provision of advice/guidance•Mapping and appeals•monitoring

Systemic•Timing•Options explored•Methodological

Exploring the differences in estimates: Waste regulations

Assumptions•Recycling behaviour

Methodological•Estimating on farm costs

Sector knowledge•Effects on other stakeholders

External drivers•Value of waste materials

Implementation support•Provision of advice/guidance•Communicating complex definitions•IT system - registration

Systemic•Limited options explored

Suggestions for improving ex-ante approaches

• Workshops/case study methods

• Scenario modelling

• Guidelines for implementation impact measures

• Market forces

• Cost-benefit estimation

A framework for the conduct of ex-post assessment of regulations

Proposed approach: key issues

•IA documentation•Identification of impacts•Collecting data•small sample size•Qualitative•Validating data•Accounting for differences

Evaluation of the proposed approach: Strengths

•Impacts considered from the perspective of the business unit affected

•Business and external influences are brought together

•Focus on identifying causal

•Pattern matching approach

•Qualitative assessment of costs and benefits

•Feeds recommendations into the next round of policy review

Evaluation of the proposed approach: Weaknesses

•Level of compliance

•Capturing variability

•Systemic differences

•Benefit measures

•Timing

•Resource requirements

Resource implications of the modified approaches

Ex-ante IA•70-90 person days

PIR•35-75 person days

•Size/complexity of legislation

•Potential benefits