experiments and quasi-experiments
DESCRIPTION
Experiments and Quasi-Experiments. Josh Lerner Empirical Methods in Corporate Finance. The problem. Corporate finance has not traditionally carefully thought about: Reverse causation. Impact of unobserved third factors. Even when acknowledge, often responses are inadequate: - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/1.jpg)
Experiments and Quasi-Experiments and Quasi-ExperimentsExperiments
Josh LernerEmpirical Methods in
Corporate Finance
![Page 2: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/2.jpg)
The problemThe problem
Corporate finance has not traditionally carefully thought about:Reverse causation.Impact of unobserved third factors.
Even when acknowledge, often responses are inadequate:
E.g., Kaplan-Ruback [1995].
![Page 3: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/3.jpg)
UnderstandableUnderstandable
In education and labor, there are:Lots of decisions.Relatively minor stakes.Desire to appear innovative:
At least in education.
Larger stakes, greater risk aversion in finance:Hard to get CEOs, I-bankers to agree to
randomized IPOs.
![Page 4: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/4.jpg)
ConsequencesConsequences
Lots of papers about correlation…Relatively limited number that decisively
show causation.As a result…
Will focus here on experiments to address the issues.
Somewhat different approach.
![Page 5: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/5.jpg)
Strengths of experimentsStrengths of experiments
Allows careful design of choices.True randomization of participants.Limited worries about sample
selection and other issues.
![Page 6: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/6.jpg)
Limitations of experimentsLimitations of experiments
Short time frame and little study compared to real-world choice.
Are student participants representative?
Modest, low-powered financial stakes.Role of human subjects committee:
The cancelled pencil orders.
![Page 7: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/7.jpg)
AgendaAgenda
Will examine variety of approaches:From less to more practical in finance
context.
![Page 8: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/8.jpg)
Using Randomization in Using Randomization in Development Economics Development Economics
ResearchResearch
Duflo, Glennerster, and KremerWorking Paper
![Page 9: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/9.jpg)
Case for experimentationCase for experimentation
Many things may be different:Legal regime.Economic conditions.Skill level.
These are likely to be correlated with “treatment.”
A study of treated and untreated may thus reflect other influences.
![Page 10: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/10.jpg)
Case for experimentation (2)Case for experimentation (2)
With experiment, have otherwise identical people:Allows one to capture treatment effect…While minimizing bias associated with
selection effect.
![Page 11: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/11.jpg)
Other biasesOther biases
In traditional studies, likely to have data snooping issues:Many regressions run, but only a few
reported.Playing with controls and sub-samples.
Less opportunities with experiments:But still sub-sample issues.
FDA bans in clinical trials.
![Page 12: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/12.jpg)
ImplementationImplementation
In development, can be done fairly cheaply:Groups are often searching for solutions.Many programs encounter excess
demand.In many cases, programs are phased in
over timeMore challenging in corporate finance.
![Page 13: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/13.jpg)
Practical issuesPractical issues
Changes are often done in packages: Makes it harder to assess impact of any change.
May deviate from perfect experiment: Selection may be partially non-random. Participation may not be universal.
Statistical approaches may partially adjust here.
Knowledge spillovers to others.
![Page 14: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/14.jpg)
GeneralityGenerality
Experiments by definition are micro in scope:Overall effects may be different:
Shifts in pricing, externalities, etc. “Hawthorne” effect:
Those in experiment may react to being selected, as do those not selected.
Specificity of particular context.
![Page 15: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/15.jpg)
Can experiments work in Can experiments work in finance?finance?
In many cases, no:Size of stakes.Limited number of players.Differences in needs across firms.Risk aversion.Cost of implementation.
![Page 16: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/16.jpg)
Possible settingsPossible settings
Public programs:More emphasis on evaluation.
Angel groups, and other young firm financiers:Mixture of motivations?!
Laboratory experiments that capture essence of problem:Next paper is an example.
![Page 17: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/17.jpg)
Is Pay-for-Performance Is Pay-for-Performance Detrimental to Innovation?Detrimental to Innovation?
Ederer and MansoWorking Paper, 2008
![Page 18: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/18.jpg)
Split view on incentives and Split view on incentives and innovationinnovation
Number of economics papers showing power of incentive schemes:Typically, basic production activity.Experimental methodology.
Negative view in psychology literature of impact on creativity.
![Page 19: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/19.jpg)
MethodologyMethodology
Use HBS’s CLER laboratory.379 subjects.60 minute experiment:
20 periods.Students paid between $11 and $40,
depending on success.
![Page 20: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/20.jpg)
Methodology (2)Methodology (2)
Make choices regarding running lemonade stand in each period.
Choose one of three locations (school—most profitable, stadium, and business district), and sugar level, lemon level, color and price.
Different optimum for each location. Suggest initially a non-optimal strategy
(business district): Participants can fine-tune or radically alter.
![Page 21: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/21.jpg)
Methodology (3)Methodology (3)
Three compensation schemes:Fixed wage per period.
Expect will explore the least.50% of profits throughout.50% of profits in last 10 periods
(exploration).Expect will explore more and get closer to
optimum.
![Page 22: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/22.jpg)
ResultsResults
Most likely to sell in school if exploration contract.Figure 1.
Most likely to explore in early periods if exploration contract.Figure 2.
![Page 23: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/23.jpg)
Searching performanceSearching performance
Look at for those exploring (leave the business district), when stop (return or converge to a narrow band):Longer for exploration contract.
Especially for first ten periods.Especially when also if punitive feature of
insufficient profits. Table 1.
![Page 24: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/24.jpg)
More resultsMore results
Better record keeping with incentive pay:Figure 3.
More profits with exploration contract:Figure 4.
![Page 25: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/25.jpg)
Assessment and concernsAssessment and concerns
Suggests contracts can effect innovation:Truth in both views.
But remaining questions:Would real stakes introduce more risk
aversion?Do incentives really work in real world like
this?Then why are researcher contracts so flat?
What about joint production functions?
![Page 26: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/26.jpg)
The Importance of Holdup in The Importance of Holdup in Contracting: Evidence from Contracting: Evidence from
a Field Experimenta Field Experiment
Iyer and SchoarWorking Paper, 2008
![Page 27: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/27.jpg)
Here, real stakesHere, real stakes
Send entrepreneurs to pen market in Chennai, India (>100 shops).
Ordered either generic or custom pens.
Looked at levels of deposit required, as well as reaction to cancellation.Seeks to test theories of hold-up and
renegotiation.
![Page 28: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/28.jpg)
MethodologyMethodology
Real entrepreneurs used, so familiar with bargaining:Control for ethnicity, etc.Push to complete a deal, according to a
script (e..g, deposit required).Randomization in order size and other
variables.Average about $25.
Cancellations done via phone.
![Page 29: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/29.jpg)
Key findingsKey findings
25% larger deposit required for customized pens.
If cancel order ex post, more willingness to renegotiate if…Lower deposit.Customized pens.
![Page 30: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/30.jpg)
CommentsComments
Closer to real business situation:Developing country setting allows to
replicate real interactions.Though pressure to get real bargain for
analysis.But may wonder whether more
sophisticated parties, bigger stakes, would affect results.
![Page 31: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/31.jpg)
Does Microfinance Really Help Does Microfinance Really Help the Poor?the Poor?
MorduchWorking Paper
![Page 32: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/32.jpg)
Regression discontinuity designRegression discontinuity design
Units are assigned to conditions based on a cutoff score.
The effect is measured as the discontinuity between treatment and control regression lines at the cutoff: It is not the group mean difference.
![Page 33: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/33.jpg)
RD design, no effectRD design, no effect
1009080706050403020100
80
70
60
50
40
30
20
10
Pre
Postnull
Cutoff
![Page 34: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/34.jpg)
RD design with effectRD design with effect
1009080706050403020100
80
70
60
50
40
30
20
Pre
Posteff
Cutoff
![Page 35: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/35.jpg)
RD design with effectRD design with effect
1009080706050403020100
80
70
60
50
40
30
20
Pre
posteff
Controlgroup
regressionline
Treatmentgroup
regressionline
Cutoff
Differencebetweengroups
![Page 36: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/36.jpg)
Trade-offTrade-off
When properly implemented and analyzed, RD yields an unbiased estimate of treatment effect:it is reasonable to assume that in the
absence of the treatment, there would be no discontinuities that naturally occurs between the two groups.
![Page 37: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/37.jpg)
Trade-off (2)Trade-off (2)
Statistical power is considerably less than a randomized experiment of the same size.
Effects are unbiased only if the functional form of the relationship between the assignment variable and the outcome variable is correctly modeled: E.g., non-linearities.
![Page 38: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/38.jpg)
![Page 39: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/39.jpg)
AssessmentAssessment
Cook, Shaidsh and Wong [2006]: Regression discontinuity studies in education do
as well as experiments. But other approaches do not:
Differences-in-differences. Careful controls.
Other studies suggest dramatic advantages to experiments (e.g., of peer effects).
![Page 40: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/40.jpg)
Application to financeApplication to finance
In many cases, seem like appropriate design, e.g.:Securities regulation cut-offs.Subsidies for small firms.
Avoidance of many problems illustrated above.
![Page 41: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/41.jpg)
Paper illustrates remaining Paper illustrates remaining difficultiesdifficulties
Seek to understand impact of microfinance:
In Bangladesh, half-acre cut-off for participation.
But rules appear to be broken:See Table 1, Figure 2.
![Page 42: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/42.jpg)
Paper illustrates remaining Paper illustrates remaining difficulties (2)difficulties (2)
As a result, need to divide villages by whether microfinance was available there.
Ends up more like a differences-in-differences approach.
![Page 43: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/43.jpg)
LessonsLessons
This approach seems very reasonable:Better fit for corporate finance.But messy reality has to be grappled
with.And limited power of tests may be
problematic.
![Page 44: Experiments and Quasi-Experiments](https://reader035.vdocuments.net/reader035/viewer/2022062217/568152eb550346895dc10a87/html5/thumbnails/44.jpg)
Final thoughtsFinal thoughts
Corporate finance has been surprisingly casual about these issues.
Addressing causality issues is quite important.
Practical limitations of many approaches used elsewhere…But more can be done!