Meta-Analysis and Strategy Research

Dan R. Dalton

Kelley School of Business

Indiana University

2

A [Very] Brief History of Research Synthesis

• Averaging Correlations?• Combining Significance Levels?• The Narrative Review (aka “Counting”

Review)• Gene Glass (1976) “Invents” Meta-Analysis• Early Critics – “An Exercise in

Mega-Silliness”

3

An Example of Meta-Analysis(Data Are Simulated)

• Research Question: The Extent to which Equity Holdings by CEOs Are Related to Firms’ Financial Performance

• Proposed Moderator: Expected that this Relationship Will be Moderated by the “Maturity of the Firm” (i.e., Firms that Are Five or Less Years Post-IPO vs. Other)

• Studies Available for Meta-Analysis = 30 (10 are not significant, 10 are positive and

significant, 10 are negative and significant)

4

An Example of Meta-Analysis(Data Are Simulated)

• R = Reliability• RR = Range Restriction• M = Moderator (1 = ≤ 5 yrs. Post-IPO; 2 = > 5 yrs. Post-IPO)

r n Ry Rx RRy RRx M

.26

.39

.37

.29

.23

.11

56

225

192

146

70

325

.8

.8

.8

.8

.8

.8

.8

.8

.8

.8

.8

.8

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

2

5

“r” - A Bivariate Correlation

• “r” vs. “d”• R-square• Deriving “r” from “d,” “t,” “F-score,” “Z,”

“Chi-Square” …• “r” from Incomplete Information

r = Z/sqrt n

if “n” = 120 and Z = 1.96 with “r” unknown

then r = +/ - .179 (i.e., 1.96/10.95)

6

“r” – A Bivariate Correlation, cntd.

• -17 to +17 and Enter What?

• Discard the Study?

• “r” and the Z-transformation?

• “r” and Statistical Significance

• And, a “Surprise” About Multiple Non-Significant Results

7

“r” – A Bivariate Correlationand “n”

• “r” As an Independent Variable, a Dependent Variable, a Control Variable, a Moderating Variable, a Mediating Variable…

• “n” – The Sample Size from which the “r” Was Calculated

• To Weight the Observed Correlation in Order to Calculate the Mean Weighted Correlation Across All of the Studies

• “n” and the Correlation Matrix

8

Ry (Reliability of y); Rx (Reliability of x)

• Constructs vs. Observed Variables

• Strategic Management Meta-Analyses with Ry = 1 and Rx = 1

• Strategic Management Variables Are Not That Good

• The Choice of Ry and Rx Levels Is Counterintuitive – Lower Ry’s and Rx’s Will Improve the “Corrected r”

• Ry and Rx at .8

9

RRy & RRx (Range Restriction of y and x)

• Analytical Issues of Range Restriction Have Become Increasingly Complex

• In Strategic Management – RRy and RRx as Deliberate Selectivity in the Sample

• Strategic Management and “Survival” Issues

10

Moderation in Meta-Analysis

• In Meta-Analysis a “Moderator” Is a Subgroup

• Profligate Testing for ModeratorsCapitalization on ChanceLoss of Statistical Power

• Moderators Need Not Always Be Operationalized as a Dichotomy

11

Meta-Analytic Procedures and Results

PART 1: # of

Correlations

CombinedSample

Size

MeanTrue ScoreCorrelation

Std Dev:Mean True

ScoreCorrelation

EntireSample

30 9,685 -.026 .283

Moderation: ≤ 5 Yrs. from IPO

16 2,032 .417 .048

Moderation: > 5 Yrs.

from IPO14 7,653 -.144 .188

12

Meta-Analytic Procedures and Results

PART 2: MeanTrue Score

Correlation

80%Credibility

Interval

90%Confidence

Interval

% VarianceAttributableTo Artifacts

EntireSample

-.026 - .389 : .336 - .112 : .059 5.74

Moderation:≤ 5 Yrs. from IPO

.417 .354 : .479 .396 : .437 80.49

Moderation:> 5 Yrs.from IPO

-.144 - .386 : .098 -.061 : -.227 7.26

13

Meta-Analytic Results:Some Diagnostics

• The Magnitude of the Mean True Score Correlation• Does the 90% Confidence Interval Include Zero?

Suggests that the Mean True Score Is Not Significant• Does the 80% Credibility Interval (Difference

between Low and High Estimates) Exceed .11? Suggests the Existence of a Moderator

• Does the % Variance Attributable to Artifacts Exceed 75%? Suggests that a Moderator Is Unlikely

• And, If the Tests Had Relied on Different Rx and Ry Values? [ .7 = .48; .8 = .417; .9 = .37 ]

14

Results Summary

• There is no simple relationship ( -.026, ns) between CEO equity holdings and firm financial performance. There is, however, some evidence of the existence of a moderating variable.

• There is evidence of a moderating effect for time since IPO. The relationship between CEO equity holdings and firm financial performance for firms 5 years or less from IPO is .417, a significant relationship. The diagnostics suggest that a further moderating effect of this result is unlikely.

15

Results Summary, cntd.

• The relationship between CEO equity holdings and firm financial performance for firms more than 5 years from the IPO is -.144, a significant relationship. The diagnostics suggest that a further moderating effect of this result is likely.

16

Other Issues in Meta-Analysis

• Fixed vs. Random Effects ModelsRandom Effects Models – Population Parameters

May Vary Across StudiesFixed Effects Models – Population Parameters Are

Invariant• “File Drawer” Problem• Unreported Null Results• “Fail Safe” Approach• The Issue Is Less a Matter of Fail Safe

Algorithms than of Reliance on Too Few Studies

17

Other Issues in Meta-Analysis, cntd.

• Quality of Data

• Outliers Statistical OutliersEntry Error Outliers

• Sensitivity to Outliers

• The General Question of Discarding Data

• Disclosure and Replicability

18

Other Issues in Meta-Analysis, cntd.

• The Independence of Data• Entering Data that Are Clearly Not

Independent• A Random Selection, Pooling, a Weighted

“r”, a Weighted “n”• An Interesting Catch-22• “Clearly Reflect the Same Construct”• Independence of Samples

Constructive Replication

19

General Guidelines for Meta-Analysis

• There is no need to transform the input values of “r”s.

• When it is necessary to impute the value of “r,” set “r” = 0.

• For observed variables, rely on .8 for the reliability of the dependent and independent variables.

• With observed variables, it will rarely be necessary to assign a range restriction score.

20

General Guidelines for Meta-Analysis, cntd.

• Use a conservative 90% confidence interval for the meta-analysis diagnostics (for these data, 95% would be an interval of -.128 to .075, much wider than the -.112 to .059 reported).

• Use a conservative 80% credibility interval for the meta-analysis diagnostics (for these data, the 90% would have been an interval of -.493 to .448, much wider than the -.389 to .336 reported).

• Where the meta-analysis software provides an option, rely on a “Random Effects Model.”

21

General Guidelines for Meta-Analysis, cntd.

• Assuming every effort has been made for an exhaustive search for meta-analysis input data, you need not be concerned about “file drawer” issues

• Neither weight nor exclude data on the basis of the quality of the study. Instead, run two meta-analyses and compare the results for the entire data set and a reduced data set without the troublesome data

22

General Guidelines for Meta-Analysis, cntd.

• Only under extremely rare conditions would there be any concerns about the independence of the data; accordingly, there is no need to combine data from separate “r”s in any manner.

• No need to exclude outliers. Instead, run two meta-analyses and compare the results for the entire data set and a data set without the outliers.