mima tutorial

MiMa Function Tutorial (version 1.4) 1

MiMa: An S-Plus/R Function to fit Meta-Analytic Mixed-,Random-, and Fixed-Effects Models

Wolfgang ViechtbauerUniversity of Maastricht

http://www.wvbauer.com/

March 8, 2006

Contents

1 Meta-Analytic Models 21.1 Mixed-Effects Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Random-Effects Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Fixed-Effects with Moderators Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Fixed-Effects Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Hypothesis Tests in the Mixed-Effects Model Context 3

3 MiMa Function for Fitting the Meta-Analytic Models 4

4 Example 4

5 Predicted/Estimated Effect Sizes 7

6 Categorical Moderators 8

7 Fitting a Random-Effects Model 8

8 Negative Heterogeneity Estimates 9

9 Possible Problems when Fitting the Mixed-Effects Model 10

10 Fitting Fixed-Effects Models 10

11 How to Cite the Function (APA style) 11

12 References 11


1 Meta-Analytic Models

1.1 Mixed-Effects Model

Given are i = 1, . . . , k independent effect size estimates, each estimating a corresponding effectsize. This can be expressed algebraically by

Yi = θi + εi, (1)

where εi is the sampling error for the ith study. The sampling errors are assumed to be nor-mally distributed with mean zero and variance vi. Now assume that p moderator variables areinfluencing the effect sizes in a linear fashion. Let Xij denote the value of the jth moderatorvariable for the ith effect size. We then write

θi = β0 + β1Xi1 + . . .+ βpXip +ui, (2)

where ui is assumed to follow a normal distribution with mean zero and variance τ2. Theaverage effect size is given by β0+β1Xi1+. . .+βpXip, while the study-specific effect size is givenby (2). Therefore, τ2 denotes the amount of residual heterogeneity, that is, the amount of excessor unexplained variability in the effect sizes after accounting for the amount of heterogeneitywe would expect based on the moderator variables that are included in the model. CombiningEquations 1 and 2 yields the mixed-effects model, which is given by

Yi = β0 + β1Xi1 + . . .+ βpXip +ui + εi. (3)

Therefore, βj denotes by how much the average effect size changes for a one unit increase inthe corresponding moderator variable when holding all other moderator values constant, whileβ0 denotes the average effect size when all moderator variables are equal to zero.

1.2 Random-Effects Model

When no moderators are included in the model, then the mixed-effects model simplifies to

Yi = µ +ui + εi, (4)

where β0 has been replaced with µ. According to the random-effects model, all of the hetero-geneity among the effect sizes is random and cannot be accounted for by moderators. There-fore, τ2 in the random-effects model simply denotes the amount of total heterogeneity, that is,the amount of variability in the effect sizes.

1.3 Fixed-Effects with Moderators Model

When moderators are included in the model and the moderators can account for all of theheterogeneity among the effect sizes, then the fixed-effects with moderators model applies,which is given by

Yi = β0 + β1Xi1 + . . .+ βpXip + εi. (5)

Therefore, this model is again just a special case of the mixed-effects model, where τ2 = 0.


1.4 Fixed-Effects Model

When all of the effect sizes are homogeneous (i.e., no moderators are operating and there is norandom heterogeneity among the effect sizes), then θ1 = θ2 = . . . = θk = θ and the appropriatemodel is given by

Yi = θ + εi, (6)

using θ in place of β0.

2 Hypothesis Tests in the Mixed-Effects Model Context

Once estimates of β0 through βp and τ2 have been obtained (i.e., once the model has beenfitted), we can conduct a variety of hypothesis tests:

1. We can test whether the amount of residual heterogeneity (τ2) is significantly differentfrom zero (i.e., H0: τ2 = 0). For this, we use the QE statistic, which follows a chi-squaredistribution under the null hypothesis with degrees of freedom equal to k− p − 1.

2. We can also test whether any of the β1, . . . , βp parameters in the model are significantlydifferent from zero, or in other words, whether at least one moderator is influencing theeffect sizes (i.e., H0: β1 = β2 = . . . = βp = 0). For this, we use the QME statistic, which isdistributed chi-squared withp degrees of freedom (i.e., equal to the number of parameterstested) under the null hypothesis.

3. We can also test individual parameters, by dividing the parameter estimate by its stan-dard error. This ratio follows a standard normal distribution under the null hypothesis.Confidence intervals for each parameter are also easily obtained by inverting these tests.

Notes: (1) The sampling variances (i.e., the vi values) are usually not exactly known. In-stead, the vi values are replaced with corresponding estimates (v̂i). Therefore, the distribu-tions of the test statistics above are only exact when the within-study sample sizes are large(i.e., when the error in the sampling variance estimates is negligible). (2) Moreover, when fit-ting a mixed/random-effects model, then τ2 must be estimated (τ̂2). The estimate is thensimply treated as a known constant. Therefore, the QME statistic and the individual param-eter tests assume that k is sufficiently large, such that τ2 is estimated with high precisionand the error in the estimate is negligible. (3) Finally, most effect size measures are not ex-actly normally distributed, as assumed under the meta-analytic models. However, the normaldistribution approximation usually becomes more accurate as the within-study sample sizesincrease. Therefore, large within-study sample sizes are needed to be certain that the testsperform as desired (specifically, that the Type I error rate is controlled). Refined moderatortests that provide better control of the Type I error rate when these assumptions are violatedhave been developed and will be included in the MiMa function later on.


3 MiMa Function for Fitting the Meta-Analytic Models

The S-PLUS/R function “mima” can be used for fitting the various meta-analytic models.

Usage:

mima(yi, vi, mods, method="REML", threshold=0.00001, maxiter=100, alpha=0.05,verbose="no", digits=4, fe="no", out="no")

Required Arguments:

yi -- a vector of effect size estimatesvi -- a vector of sampling variancesmods -- a vector/matrix of moderator values

Optional Arguments:

method -- method for estimating the amount of (residual) heterogeneity;possible methods are: "REML", "ML", "HE", "DL", "SH", or "EB";the default is "REML" (see below for more details)

threshold -- precision of the Fisher scoring algorithm when using eitherML, REML, or EB estimation; the default is 0.00001

maxiter -- maximum number of iterations of the algorithm when using eitherML, REML, or EB estimation; the default is 100

alpha -- to set the width of the confidence interval for individualparameters (default is .05, which yields a 95% CI)

verbose -- set to "yes" to obtain information on the evolution of theiterative algorithm when using either ML, REML, or EB estimation

digits -- number of decimals for rounding of results; the default is 4fe -- set to "yes" to force fitting a fixed-effects modelout -- see below for explanation

Several methods for estimating the amount of (residual) heterogeneity are available. The de-fault is restricted maximum-likelihood (REML) estimation and this is the recommended method(Viechtbauer, 2005). Regular maximum-likelihood (ML) estimation is also available, but it isnot recommended, because the estimate of τ2 tends to be too small (negatively biased). Twomethod of moment estimators are also available (HE and DL, respectively); the first is an ex-tension of the estimator suggested by Hedges (1983, 1989) and the other an extension of theestimator suggested by DerSimonian and Laird (1986). More details on these two estimatorscan be found in Raudenbush (1994). Finally, the estimator suggested by Sidik and Jonkman(2005a, 2005b) and the empirical Bayes estimator (Berkey, Hoaglin, Mosteller, & Colditz, 1995;Morris, 1983) are also available (SH and EB, respectively). The ML, REML, and EB estimationmethods make use of the Fisher scoring algorithm, which is robust to poor starting values andusually converges quickly (Harville, 1977; Jennrich & Sampson, 1976).

4 Example

Table 1 provides the results for k = 16 studies examining the effectiveness of massage therapyfor relieving state anxiety. Listed are the sample sizes of each study (nEi and nCi ), the effect size


estimate (Yi), the estimated sampling variance (v̂i), the minutes per session of massage therapyprovided in each study, whether a fully trained massage therapist or a layperson provided thetherapy, and the mean age of the sample. The data were adopted from Moyer, Rounds, andHannum (2004), leaving out three studies with missing data and two studies where the durationof massage therapy provided was less than 10 minutes.

Table 1Results for 16 Studies on the Effectiveness of Massage Therapy for Reducing State Anxiety

Sample Sizes Effect Size Sampling Minutes/ Trained MeanStudy nEi nCi Estimate (Yi) Variance (v̂i) Session Therapist Age

1 30 30 0.444 0.068 30 0 282 46 39 −0.495 0.049 10 0 423 15 15 0.195 0.134 20 1 314 10 10 0.546 0.207 40 1 395 12 12 0.840 0.181 20 1 176 10 10 0.105 0.200 30 1 517 26 24 0.472 0.082 15 1 268 18 14 −0.205 0.128 10 0 649 12 12 1.284 0.201 45 1 48

10 12 12 0.068 0.167 30 1 4011 15 15 0.234 0.134 30 1 5212 12 12 0.811 0.180 30 1 3313 15 15 0.204 0.134 30 1 2014 18 18 1.271 0.134 60 1 2715 15 15 1.090 0.153 45 1 5216 43 35 −0.059 0.052 10 1 61

Note. Adopted from Moyer, Rounds, and Hannum (2004). Minutes/Session = minutes of therapy provided persession; Trained Therapist: 0 for a layperson providing the therapy and 1 for a trained therapist; Mean Age =mean age of the sample.

The data are available in the S-PLUS dataset file “dat.sdd”. The names of the variables are:n1, n2, yi, vi, minutes, trained, and meanage. In order to use the mima function, we mustfirst create a matrix with the moderator values that we want to include in the model. For this,load the dataset, open up the command window (click on “Window” in the menu bar and choose“Command Window”), and type:

> attach(dat)> mods <- cbind(minutes, trained, meanage)

The attach command makes the variables in the dataset available (technically, it adds thedataset to the S-PLUS search path). The second command creates a matrix, called “mods”,where the columns are composed of the minutes, trained, and meanage variables.

Next, load the mima function by double-clicking on the file called “mima.ssc” (if askedwhether to overwrite the existing script, then this means that the function is already loaded,but to be certain that the newest version of the function is available, select ok). Once the script


is loaded, run it (i.e., make it available) by hitting F10 (you can then close the script window).Finally, to fit the mixed-effects model to the massage therapy data, simply type:

> mima(yi, vi, mods)

The output should then look like this:

Estimate of (Residual) Heterogeneity: 0

Test for (Residual) Heterogeneity:

QE = 9.1278df = 12p-value = 0.692

Parameter Estimates:

[,1]intrcpt -0.2971minutes 0.0250trained 0.3013meanage -0.0059

Variance-Covariance Matrix of Parameter Estimates:

intrcpt minutes trained meanageintrcpt 0.1218 -0.0013 -0.0112 -0.0018minutes -0.0013 0.0000 -0.0005 0.0000trained -0.0112 -0.0005 0.0382 -0.0001meanage -0.0018 0.0000 -0.0001 0.0000

Omnibus Test of all Moderators:

QME = 28.9176df = 3p-value = 0

Individual Moderator Tests:

estimate SE zval pval CI_L CI_Uintrcpt -0.2971 0.3490 -0.8513 0.3946 -0.9811 0.3869minutes 0.0250 0.0067 3.7172 0.0002 0.0118 0.0381trained 0.3013 0.1954 1.5422 0.1230 -0.0816 0.6842meanage -0.0059 0.0063 -0.9488 0.3427 -0.0182 0.0063


The output shows that no residual heterogeneity remains when all three moderators areincluded in the model (the estimate of residual heterogeneity is zero). The test for residualheterogeneity is also far from significant (QE = 9.13,df = 12, p = .692). The QME statistic ishighly significant (QME = 28.92,df = 3, p < .0001), indicating that at least one moderator vari-able is related to the effect sizes. The individual parameter tests suggest that it is the minutesmoderator that is significant (95% confidence interval bounds for this parameter: (0.01, 0.04),z = 3.72, p = .0002).

One can also obtain a confidence interval for the amount of (residual) heterogeneity in themodel with the tau2cime.r function available also on the author’s website. Simply open thatscript in S-Plus and source it with F10. Then type:

> tau2cime(yi, vi, mods)95% CI: (0, 0.144)

which shows that the bounds for a 95% confidence interval for τ2 are given by (0, 0.144). Formore information about this function, see the author’s website.

5 Predicted/Estimated Effect Sizes

Once a model has been fitted, it is easy to obtain predicted/estimated effect sizes from theoutput. Recall that the output for the mixed-effects model fitted earlier included the followingparameter estimates:

[,1]intercept -0.2971minutes 0.0250trained 0.3013meanage -0.0059

Therefore, the predicted/estimated effect size for 30 minutes of therapy provided by a fullytrained therapist to a group with an average age of 40 is:

−0.2971+ 0.0250(30)+ 0.3013(1)− 0.0059(40) ≈ 0.52.

We can use S-PLUS/R to carry out these calculations. We first fit the model again, using theout="yes" option. With this argument set to yes, the estimate of residual heterogeneity, theparameter estimates, and the variance-covariance matrix of the parameter estimates are givenin list form for further processing. Specifically, if we write:

> res <- mima(yi, vi, mods, out="yes")

then the output is put into the object “res”. Next, we select some moderator values and thenmultiply the moderator values with the parameter estimates to obtain a predicted effect size.For example, for minutes = 30, trained = 1, and meanage = 40, we use:

> mvals <- cbind( c(1, 30, 1, 40) )> t(mvals) %*% res$b

[,1][1,] 0.5157254


The “1” in front of the “30” on the line starting with mvals is needed, because of the interceptparameter in the model. The variance/standard error of this estimate can also be obtained. Toobtain the variance, we type:

> t(mvals) %*% res$vb %*% mvals[,1]

[1,] 0.0103143

and for the standard error (the square root of the variance), we use:

> sqrt( t(mvals) %*% res$vb %*% mvals )[,1]

[1,] 0.1015593

An approximate 95% confidence interval is then given by adding and subtracting 1.96 times thestandard error from the predicted effect size:

0.5157± 1.96(0.1016) = (0.32,0.72).

6 Categorical Moderators

Categorical moderators can be included in the model, in the same way that linear models caninclude appropriately coded categorical independent variables (in the example above, the vari-able specifying whether a layperson or a trained therapist provided the treatment was dummycoded). See, for example, Neter, Kutner, Nachtsheim, and Wasserman (1996) for more detailson how to code categorical independent variables in the context of linear models.

7 Fitting a Random-Effects Model

One can also fit the random-effects model with the mima function. The random-effects modelis simply a special case of the mixed-effects model, which does not include any moderatorvariables. For this, create an empty moderator matrix and then run the mima function asfollows:

> mods <- c()> mima(yi, vi, mods)

Estimate of (Residual) Heterogeneity: 0.1648

Test for (Residual) Heterogeneity:

QE = 38.0456 df = 15 p-value = 0.0009


Parameter Estimates:

[,1]intrcpt 0.3772

Variance-Covariance Matrix of Parameter Estimates:

intrcptintrcpt 0.0183

Omnibus Test of all Moderators:

QME = NA df = 0 p-value = NA

Individual Moderator Tests:

estimate SE zval pval CI_L CI_Uintrcpt 0.3772 0.1354 2.7855 0.0053 0.1118 0.6425

Note that the estimate of residual heterogeneity now indicates the amount of (total) het-erogeneity in the effect size estimates over and beyond that which we would expect based onsampling variability alone. The estimate is quite large (τ̂2 = 0.1648). The test for residualheterogeneity is now equivalent for the usual Q-test for heterogeneity and also reveals thatthe effect sizes are heterogeneous (QE = 38.05,df = 15, p = .0009). The QME statistic isnot needed here (since there are no moderators in the model), but we can test whether theaverage effect size (i.e., µ) is significantly different from zero, which turns out to be the case(z = 2.79, p = .0053). The bounds for a 95% confidence interval for µ are (0.11, 0.64). However,given that at least one moderator appears to be influencing the effect sizes, the results fromthe random-effects model are incomplete at best. Again, a confidence interval for τ2 can alsobe obtained with the tau2cime.r function available on the author’s website:

> tau2cime(yi, vi, mods)95% CI: (0.035, 0.524)

which shows that the bounds for a 95% confidence interval for τ2 are given by (0.035, 0.524).

8 Negative Heterogeneity Estimates

Except when using the Sidik and Jonkman estimator (SH), it is possible that the heterogeneityestimate turns out to be negative when using any of the estimation methods. However, negativeestimates of τ2 are outside of the parameter space. For the method of moments estimators,negative estimates are therefore simply truncated to zero. For ML, REML, and EB estimation,the Fisher scoring algorithm used in the mima function was modified using step halving toguarantee a non-negative estimate (see Harville, 1977, and Jennrich & Sampson, 1976, for moredetails on ML and REML estimation and penalty techniques).


9 Possible Problems when Fitting the Mixed-Effects Model

Two problems can occur when fitting the mixed-effects model with the mima function:

1. First of all, there is a chance that the Fisher scoring algorithm used for ML, REML, andEB estimation does not converge. One can try increasing “maxiter” (the default is 100iterations), but there are cases where the algorithm will cycle between several estimatesand never converge, no matter how many iterations are run (by setting verbose = "yes",one can see how the iterative algorithm is progressing). In this case, one can try a differentestimation method thatn the default, which is REML estimation.

2. Another problem that can occur is the following error message when running the function:“Problem in solve.qr(a): apparently singular matrix”. This problem can occur for at leastthree reasons.

(a) When there are linear relationships between the moderator variables (i.e., the valuesof one moderator variable can be given as a linear function of the other moderatorvariables), then this error will occur. In this case, one must remove moderatorsvariables from the model until such linear relationships are no longer present.

(b) When some of the moderators are highly correlated, then this error can also occur(a correlation of exactly ±1 indicates a linear relationship between two moderatorsand we are back to the first case). Highly correlated moderators could be argued tobe redundant anyway and therefore one should probably remove one of them.

(c) The error can also occur even when the moderators are not highly correlated, butthe values of the moderators are scaled very differently (e.g., the values of one mod-erator are between 0 and 1, while the values of another moderator are between 1and 10,000). Such differential scaling can cause numerical problems with the matrixinversions used as part of the algorithm (right now, “brute force inversion” is usedin the iterative procedures – in the future, this may be replaced with more elegantmethods). The solution here is to make sure that the scaling is not too dissimilar (itdoes not have to be exactly the same). One could try dividing the values of a moder-ator with very large values by 10 or 100 and re-running the model. For example, tore-scale the meanage moderator, use:

> meanage <- meanage/10> mods <- cbind(minutes, trained, meanage)> mima(yi, vi, mods)

The re-scaling will change the parameter estimate and variance for this moderator,but the z-value and p-value should remain unchanged. The other parameter esti-mates should not be affected by the re-scaling.

10 Fitting Fixed-Effects Models

One can also force the mima function to fit a fixed-effects model (with or without moderators).To do this, set fe="yes". The estimate of (residual) heterogeneity will automatically be set


to zero. This is not recommended, because the resulting moderator tests can have severelyinflated Type I error rates (i.e., moderators that are not actually significant may appear to besignificant). This option is available only for demonstration purposes and for conducting asensitivity analysis, but should not be used to actually test the moderators.

11 How to Cite the Function (APA style)

The function and this manual/tutorial should be cited (APA style) as follows:

Viechtbauer, W. (2006). MiMa: An S-Plus/R function to fit meta-analytic mixed-, random-, andfixed-effects models [Computer software and manual]. Retrieved from http://www.wvbauer.com/.

12 References

Berkey, C. S., Hoaglin, D. C., Mosteller, F., & Colditz, G. A. (1995). A random-effects regressionmodel for meta-analysis. Statistics in Medicine, 14, 395-411.

DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials,7, 177–188.

Harville, D. A. (1977). Maximum likelihood approaches to variance component estimationand to related problems. Journal of the American Statistical Association, 72, 320–338.

Hedges, L. V. (1983). A random effects model for effect sizes. Psychological Bulletin, 93,388–395.

Hedges, L. V. (1989). An unbiased correction for sampling error in validity generalizationstudies. Journal of Applied Psychology, 74, 469–477.

Jennrich, R. I., & Sampson, P. F. (1976). Newton-Raphson and related algorithms for maxi-mum likelihood variance component estimation. Technometrics, 18, 11–17.

Morris, C. N. (1983). Parametric Empirical Bayes inference: Theory and applications (withdiscussion). Journal of the American Statistical Association, 78, 47-65.

Moyer, C. A., Rounds, J., & Hannum, J. W. (2004). A meta-analysis of massage therapyresearch. Psychological Bulletin, 130, 3–18.

Neter, J., Kutner, M. H., Nachtsheim, C. J., & Wasserman, W. (1996). Applied linear statisticalmodels (4th. ed.). Irwin: Chicago.

Raudenbush, S. W. (1994). Random effects models. In H. M. Cooper & L. V. Hedges (Eds.),The handbook of research synthesis (pp. 301–321). New York: Russell Sage.

Sidik, K., & Jonkman, J. N. (2005a). A note on variance estimation in random effects meta-regression. Journal of Biopharmaceutical Statistics, 15, 823-838.

Sidik, K., & Jonkman, J. N. (2005b). Simple heterogeneity variance estimation for meta-analysis. Journal of the Royal Statistical Society, Series C, 54, 367-384.

Viechtbauer, W. (2005). Bias and efficiency of meta-analytic variance estimators in therandom-effects model. Journal of Educational and Behavioral Statistics, 30, 261-293

mima tutorial

Documents