best practices for the submission of economic … · 2019-10-21 · european commission, 1 dg...
TRANSCRIPT
1European Commission, DG Competition, Chief Economist
BEST PRACTICES FOR THE SUBMISSION OF ECONOMIC EVIDENCE AND DATA COLLECTION
Damien Neven*Chief competition economist, European Commission
Conference in honour of B. Hawk
Brussels, March 17, 2010
*The views expressed are those of the speakers and do not necessarily reflect those of DG Competition or the European Commission
2European Commission, DG Competition, Chief Economist
Introduction
• More analysis of effects (more focused objectives, better technology, increased scrutiny by the courts)
• Identify the consequences of mergers, agreements or single firm conduct (by formulating a theory of harm and/or understanding of efficiencies)
• Assess the direction and magnitude of these effects empirically
• Trade-off between precision and administrability
3European Commission, DG Competition, Chief Economist
Introduction • Popular scepticism arises from the observation that apparently
sound but contradictory pieces of evidence can be submitted
• Understandable but incorrect belief that professional application of rigorous methods should produce unambiguous and consistent results
• However, apparent divergences can be due to
– Intrinsic differences in quality. Some studies may be methodologically superior
– Differences in data, modelling approaches, empirical techniques (and unintentional mistakes)
– Understanding the sources of the differences (data, models) may provide important insights
4European Commission, DG Competition, Chief Economist
Goals of the Best practices• Develop some quality standard
– To help those involved in the production and presentation of economic evidence
– Most importantly, to help those with responsibilities to make decisions
• Ensure that economic analysis meets certain minimum standards at the outset,
• Facilitate the efficient gathering and exchange of facts and evidence, in particular any underlying quantitative data, and
• Use in an efficient way reliable and relevant evidence obtained during the administrative procedure, whether quantitative or qualitative
5European Commission, DG Competition, Chief Economist
Content of the Best Practices
• Provides recommendations regarding the content and presentation of economic or econometric analysis. To facilitate its assessment and replication
• Provides guidance to respond to Commission requests for quantitative data to ensure that timely and relevant input for the investigation can be provided.
• The desire to ensure transparency and accountability that guide these best practices apply to all parties involved in proceedings
6European Commission, DG Competition, Chief Economist
Some basic principles
• An economic analysis may support an assessment of the anticompetitive or pro-competitive effects of a merger.
• Counterfactual• Assumptions need to be consistent with the facts of the industry under
consideration.– These assumptions should be carefully laid out and– the sensitivity of its predictions to changes to the assumptions should be made explicit.
• It is not necessary for economic submissions to actually formalize verbal arguments in a model, but this may help to:– clearly spell out the assumptions underlying an argument,– to check its logic consistency,– to assess effects of a high degree of complexity, or– to use the model as the theoretical basis for an empirical estimation
7European Commission, DG Competition, Chief Economist
Criteria for assessing the quality of economic analysis
• The relevant question• Data• Methodology• Results• Robustness
8European Commission, DG Competition, Chief Economist
A. The relevant question
• Needs to be formulated unambiguously• Properly motivated, taking into account:
– the nature of the competition issue– the institutional features of the markets and industries
under consideration– the relevant economic theory.
• The hypothesis to be tested (or null hypothesis) must be clearly spelled out as well as the alternative hypothesis or hypotheses under consideration.
9European Commission, DG Competition, Chief Economist
B. The data• The intrinsic quality of an economic theory depends on the
extent to which the underlying assumptions match the corresponding economic facts.
• Likewise, empirical analysis depends on the relevance and the reliability of the underlying data.
• Not all facts can be observed or measured with high accuracy and most datasets are incomplete or otherwise imperfect. Hence, parties and/or DG Competition should become familiar with the facts and data and acknowledge its limitations explicitly.
10European Commission, DG Competition, Chief Economist
• Describe data sources. Provide the data!• Report how the data were gathered, the sample selection
process, the measurement of the variables
• Explain/Describe data handling and data management
– Provide software code employed to generate the final dataset.
• Describe final data set (including variables!!)
• Explain anomalies in the data and efforts made to correct them.
11European Commission, DG Competition, Chief Economist
C. Methodology• Motivate the choice of empirical methodology• Discuss choices in light of (a) their data limitations, (b) the
features of the market under investigation, and (c) the economic issues under consideration (the relevant question).
• Identification!! can be understood as:– clarifying the basis upon which one theory can be preferred to
another.– More than one set of parameter of the econometric model generates
the same distribution of observations
• Alternative methodologies should also be discussed
12European Commission, DG Competition, Chief Economist
D. Results and implications
• Explain the details of the models• Share any documentation needed to allow timely
replication (e.g. the programming code used to run the analysis).
• Clear reporting understandable to non-economists• Discuss not only the statistical significance of their
results but also their practical relevance.• Discuss results with reference to relevant economic
theory
13European Commission, DG Competition, Chief Economist
E. Robustness tests
• Check whether the empirical results are sensitive to changes in the data, the choice of empirical method, and the precise modelling assumptions;
• Test whether the results of the analysis can be generalised; and
• Compare the results of the empirical work in question with previous results in the relevant literature.
• In the case of econometric or simulation models: always provide a sensitivity analysis with respect to the key variables– (All results from the sensitivity analysis conducted should also be
reported and not only those that support the argument)
14European Commission, DG Competition, Chief Economist
Practical recommendations for all parties
• Credibility is enhanced when limitations are addressed and acknowledged
• Facts that could not be audited/observed should not be endorsed
• Opinions should not disguised as statements of facts
• Dialogue between economists is useful, but needs to happen early and should not interfere with the process of developing the arguments
15European Commission, DG Competition, Chief Economist
Responding to data requests• Parties should provide the data and codes timely, in an
appropriate format to allow immediate replication– Note the Commission may not be able to take the analysis further into
consideration if replication is not possible• The Commission shall also provide the data and codes (upon
request).– A data room may be necessary to protect the confidentiality of other
parties– Third parties are expected to allow access to the data they submit
• Third parties or complainants are equally expected to submit all the underlying data used in the analysis and allow access to a data room
16European Commission, DG Competition, Chief Economist
• Dual objective:– Reduce the burden on the involved parties– Ensure data gathered is (i) Accurate and
Complete and (ii) that there is sufficient time to analyze it
• Ask for the adequate amount of data• Completeness, correctness, timely submission• Cooperation in good faith, early consultation
17European Commission, DG Competition, Chief Economist
Final advice…Quantitative analysis needs to…
• Be based on clear economic theory that generates • …testable propositions (prices in the input market will go up
in vertical mergers?)– Which distinguish between alternative theories (estimating the
elasticity of demand of a firm in a dominance?)
• Be intuitive (be suspicious of price effects of 400%)• and replicable (track down aggregation in variables)• Use simple, well tested methods whenever possible
– Complexity of the analysis can lead to courts simply choosing to ignore the evidence
18European Commission, DG Competition, Chief Economist
Examples of Poor Economic Submissions• Irrelevant submissions
– Example: report comparing financial multiples in different industries, which was unrelated to the issue under investigation (input foreclosure).
• Pseudo data analysis– Example: report which argues that a chain of substitution exists
because the price difference across end points of segments was smaller than the price difference between end points within segments
• Results hidden in legal submissions– Example: results of win/loss analysis inserted in form CO without any
data description or rigorous interpretation
19European Commission, DG Competition, Chief Economist
• Theoretical claims that do not fit the facts of the case– Example: report arguing that the merger under review would take
place in an idealised bidding market and hence could not have any anticompetitive effect (without any data analysis supporting this claim)
• Analysis of unreliable data– Example: report analysing an unrepresentative sample of bidding data
• No data or methodology description– Example: results of econometric estimation initially submitted without
proper methodological explanation and data description
20European Commission, DG Competition, Chief Economist
• Results poorly reported or interpreted– Example: econometric estimation presented without reporting standard errors
• Lack of sensitivity analysis– Example: report presenting one single result without any indication of its robustness to
underlying assumptions
• Replication issues– Example: data and computers codes not submitted, or submitted in such a way that
they do not allow timely replication and assessment
⇒ Dealing with some of these submissions may capture precious resources from DG COMP
⇒ Importance for submission to meet minimum standards given short deadlines
21European Commission, DG Competition, Chief Economist
Conclusion
• Complementarity with best practice on procedures
• Economic evidence in courts
• Comparison with US