KJM The Korean Journal of MedicineKorean J Med 1738-9364 2289-0769 The Korean Journal of Medicine 10.3904/kjm.2019.94.5.391 kjm-94-5-391 What's hot? 메타분석의 명과 암 Strengths and Limitations of Meta-Analysis LeeYoung Ho 이영호 Department of Internal Medicine, Korea University College of Medicine, Seoul, Korea 고려대학교 의과대학 내과학교실 Correspondence to Young Ho Lee, M.D., Ph.D. Division of Rheumatology, Department of Internal Medicine, Korea University Anam Hospital, Korea University College of Medicine, 73 Goryeodae-ro, Seongbuk-gu, Seoul 02841, Korea Tel: +82-2-920-5645, Fax: +82-2-922-5974, E-mail: lyhcgh@korea.ac.kr 1 10 2019 1 10 2019 94 5 391 395 27 7 2019 5 8 2019 Copyright © 2019 The Korean Association of Internal Medicine 2019 This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Meta-analysis is a statistical method that combines and synthesizes multiple studies and integrates their results. Meta-analysis increases the sample size, and in turn, the power to study the effects of interest by combining primary studies and providing a precise estimate of the effects. Data synthesized from meta-analyses are usually more beneficial than the results of narrative reviews. In a meta-analysis, the decisions are transparent, and statistical analysis yields an objective measure of the integrated quantitative evidence. The biases of narrative reviews can be limited or overcome by conducting a meta-analysis. The systematic approach and transparency in meta-analysis help to resolve conflicts and uncertainties between studies, while leading to significant conclusions. However, this method is controversial and may not always be the best tool. Moreover, meta-analysis has several shortcomings, and in some cases, it may not be appropriate. Although meta-analysis has been criticized due to its limitations, there are solutions to such problems. The aim of this review is to describe and discuss the strengths and weaknesses of meta-analysis.

 메타분석 자료분석 편견 Meta-analysis Data analysis Bias INTRODUCTION

Meta-analysis is a statistical method that combines and synthesizes different studies and integrates their results into a single conclusion [1,2]. The word “meta” comes from Greek and means “after” or “beyond,” with meta-analysis meaning an “analysis of analyses” [3]. A meta-analysis increases the sample size, and consequently, the power to study the effects of interest by combining primary studies while considering the sizes of the studies included [2]. Thus, meta-analysis provides a more precise estimate of the effect. Meta-analysis is based on mathematical and statistical rules [4]; therefore, it is more objective than other methods, such as narrative reviews, and is less influenced by the author’s personal opinions. Thus, the use of meta-analysis is becoming increasingly popular [5-7]. Meta-analysis can be a powerful tool when it is well designed and performed appropriately. However, it may not always be the best tool, and is controversial [8]. Meta-analysis has several limitations, which can lead to misleading results; furthermore, it may not be appropriate in some cases. This review aims to describe and discuss the strengths and weaknesses of meta-analysis.

Meta-analysis vs. narrative review

Data synthesized from meta-analyses are usually more beneficial than the results of narrative reviews, because the latter have numerous disadvantages that can be overcome by meta-analysis [9]. Narrative reviews are based on subjective selection of publications; thus, study selection can be arbitrary [10]. In a narrative review, the lack of a specific search strategy increases the risk of failing to identify relevant studies on a given topic. In addition, the selected studies may not be critically appraised, the strength of the evidence may not be weighted, and no quantitative synthesis of the data may have been performed [10]. In narrative reviews, it may be difficult to compare the effects of studies that use varying metrics to measure a similar outcome. The reviewer in a narrative review qualitatively addresses a question by summarizing the findings of other studies and drawing a conclusion. Therefore, narrative reviews are prone to biases and errors, as different reviewers may draw different conclusions from the same evidence [11]. Therefore, narrative reviews should be considered opinion pieces with a low level of evidence. Hence, more transparent and reproducible systematic research syntheses may enable the resolution of any uncertainties when there are disagreements between the conclusions of studies. Meta-analysis minimizes bias by employing a methodological approach [12]. In meta-analysis, the decisions are transparent, and the statistical analysis results in an objective measure of the integrated quantitative evidence, which can then be repeated and verified. Moreover, meta-analysis converts the results of primary studies into a common metric as the effect size, thus different measures from primary studies can be compared against each other and yield conclusions that are more significant. By conducting a metaanalysis, the biases of narrative reviews can be limited or overcome. Therefore, meta-analysis is considered to provide evidence with the highest level of accuracy.

 Strengths of meta-analysis

The methodological strengths of meta-analysis are listed in Table 1. It is more efficient to present a combined result than to describe the results of each individual study [8,13,14]. Meta-analysis can summarize and quantify the results from individual studies [15,16]. Additionally, it can clarify heterogeneity between the results of different studies and analyze differences in the results. An individual study may contain too few subjects in a particular subgroup. However, meta-analytic data from several individual studies may provide a clearer picture of the subgroup. Meta-analysis increases statistical power by increasing the sample size, and can determine small but clinically significant effects by combining data from numerous studies.

The precision of a study’s findings largely depends on the number of subjects. Statistically combining data from individual studies can provide a more precise estimate of the underlying effects than a single study. Thus, meta-analysis overcomes the limitation of small sample sizes of individual studies, detects the effects of interest, and reduces the risk of false-negative results. Moreover, meta-analysis can settle controversies resulting from studies with conflicting results. In addition, combining primary studies with varying sample sizes and patient populations can increase the generalizability of the results of individual studies; this allows the results of the meta-analysis to be generalized to a wider population. Appropriately examining the heterogeneity between individual studies allows the testing of novel hypotheses that have not been proposed in previous studies [17]. As meta-analysis summarizes currently existing knowledge, it may help in identifying areas that lack adequate evidence, thereby producing new research questions. Meta-analysis overcomes the problems and biases of traditional narrative reviews through a more transparent and subjective process that includes a systematic methodological approach.

 Weaknesses of meta-analysis

The methodological weaknesses of meta-analysis are listed in Table 2. In addition, the limitations of meta-analysis as well as suggestions for addressing these have been described [8,13,14,18].

One number cannot summarize a research field

Summarizing large amounts of varying information using a single number is a controversial aspect of meta-analysis [19], as it ignores the fact that treatment effects may vary from study to study. However, a meta-analysis generalizes results despite differences in primary research and does not simply report a summary effect. If there is substantial heterogeneity, then the focus should shift from the summary effect to the heterogeneity itself. Meta-analysis provides a variety of tools to assess the pattern of heterogeneity, and to possibly explain it.

 Mixing apples and oranges

Two main criticisms of meta-analysis are that it combines different types of studies (“mixing apples and oranges”) [8], and that the summary effect may ignore important differences between studies. Meta-analysis should be avoided if studies are too heterogeneous to be comparable, as the metaanalytical results may be meaningless and true effects may be obscured. However, meta-analyses, by their very nature, address broader questions than individual studies. Therefore, it can be said that a meta-analysis is similar to asking a question about fruits, for which both apples and oranges can contribute valuable information.

 Garbage in, garbage out

The phrase “garbage in, garbage out” means that if a meta-analysis includes low-quality studies, its results will be biased and incorrect [20]. Meta-analysis includes a set of criteria for determining which studies to analyze. Hence, meta-analysis should be based on stricter criteria regarding the quality of studies to be included. When the available studies are flawed, a meta-analysis may employ sensitivity analyses to identify the influence of study biases.

 Heterogeneity

In meta-analysis, heterogeneity refers to the degree of dissimilarity in the results of individual studies [2]. The main assumption for performing meta-analysis is that studies are homogenous in terms of populations, interventions, controls, and outcomes. Assessing the heterogeneity between primary studies is an important step in conducting a meta-analysis [21]. If there is substantial heterogeneity, the focus of the analysis should be on exploring and understanding the sources of the variation. Meta-analysis examines the existence of heterogeneity among primary studies and analyzes the variance in their results [2]. Subgroup analyses and meta-regression are used to explore the sources of heterogeneity. However, if there is a considerable amount of heterogeneity, it may not be appropriate to pool data in a meta-analysis.

 Publication bias

Studies that report positive effects tend to be published more frequently than those that do not, and studies that report no significant results usually remain unpublished [22]. As meta-analysis includes only published studies, it may overestimate the actual magnitude of an effect [22]. This outcome is termed “publication bias.” To reduce the effect of publication bias on meta-analysis, serious efforts should be made to identify all relevant studies, because the outcome of the meta-analysis depends on the studies included. If the presence of publication bias is suspected, it can sometimes be detected by means of funnel plots and corresponding statistics, such as Egger’s linear regression test [23] for measuring funnel plot asymmetry or the “trim and fill” method to adjust summary estimates for the observed bias [24].

 Not all variables are comparable

Some variables have no comparable measure for meta-analysis. Therefore, it may sometimes be necessary to construct new variables that present comparable concepts or restrict the analyses to common elements.

 Meta-analysis can disagree with randomized trials

The main reason for discrepancies in meta-analysis is that it is based on heterogeneous and often small studies. The subjects in the individual studies may substantially differ with respect to diagnostic criteria, comorbidities, severity of disease, and geographic region. In contrast, in large randomized controlled trials, the target population is more limited. However, meta-analysis that is conducted appropriately may provide complementary valuable information.

 Meta-analysis cannot overcome subjectivity

Meta-analysis relies on shared subjectivity, rather than objectivity. There is often a certain amount of subjectivity when deciding how similar studies should be before it is appropriate to combine them. Every form of analysis, including narrative reviews, requires certain subjective decisions. However, such decisions are always explicitly stated in a meta-analysis.

 Meta-analysis deals only with the main effects

Meta-analysis deals with the main effects, and its results can be generalized to the target population. However, the effects of interactions may also be examined by moderator analysis.

 CONCLUSION

Meta-analysis is a powerful tool when used judiciously to objectively summarize existing evidence regarding a specific issue. Moreover, meta-analysis provides a more objective appraisal of the evidence than narrative review, and attempts to minimize bias by utilizing a methodological approach. Meta-analysis provides a more precise estimate of the effect size and increases the generalizability of the results of individual studies. Therefore, it may enable the resolution of conflicts between studies, and yield conclusive results when individual studies are inconclusive. However, there are many caveats in the application of meta-analysis. Conclusions derived from meta-analysis are susceptible to the methodological quality of included studies, as well as to publication bias and the formulation of eligibility criteria. Although combining the data from independent studies using meta-analytical methods can improve statistical precision, it cannot altogether prevent bias. However, many criticisms of meta-analysis are true for narrative reviews as well [22]. Although meta-analysis is criticized for its limitations, solutions to these problems exist. A systematic approach and transparency in conducting meta-analysis help to resolve conflicts and uncertainties between studies and to derive meaningful conclusions. The use and value of metaanalysis is likely to increase in the future based on its power to reveal new findings.

 REFERENCES EsterhuizenTM ThabaneL Con: meta-analysis: some key limitations and potential solutions Nephrol Dial Transplant 2016 31 882 885 WalkerE HernandezAV KattanMW Meta-analysis: its strengths and limitations Cleve Clin J Med 2008 75 431 439 GlassGV Primary, secondary, and meta-analysis of research Educational Researcher 1976 5 3 8 EggerM SmithGD PhillipsAN Meta-analysis: principles and procedures BMJ 1997 315 1533 1537 LeeYH Meta-analysis of genetic association studies Ann Lab Med 2015 35 283 287 LeeYH BaeSC SongGG Hepatitis B virus (HBV) reactivation in rheumatic patients with hepatitis core antigen (HBV occult carriers) undergoing anti-tumor necrosis factor therapy Clin Exp Rheumatol 2013 31 118 121 LeeYH ChoiSJ JiJD SongGG Overall and cause-specific mortality in systemic lupus erythematosus: an updated meta-analysis Lupus 2016 25 727 734 FinckhA TramèrMR Primer: strengths and weaknesses of meta-analysis Nat Clin Pract Rheumatol 2008 4 146 152 CooperHM RosenthalR Statistical versus traditional procedures for summarizing research findings Psychol Bull 1980 87 442 449 PaeCU Why systematic review rather than narrative review? Psychiatry Investig 2015 12 417 419 CiprianiA GeddesJ Comparison of systematic and narrative reviews: the example of the atypical antipsychotics Epidemiol Psichiatr Soc 2003 12 146 153 BailarJC3rd The promise and problems of meta-analysis N Engl J Med 1997 337 559 561 IoannidisJP LauJ Pooling research results: benefits and limitations of meta-analysis Jt Comm J Qual Improv 1999 25 462 469 NobleJHJr Meta-analysis: methods, strengths, weaknesses, and political uses J Lab Clin Med 2006 147 7 20 LeeYH Association between the neutrophil-to-lymphocyte ratio, and platelet-to-lymphocyte ratio and rheumatoid arthritis and their correlations with the disease activity: a meta-analysis J Rheum Dis 2018 25 169 178 LeeYH SongGG Comparative efficacy and safety of secukinumab and adalimumab in patients with active ankylosing spondylitis: a Bayesian network meta-analysis of randomized controlled trials J Rheum Dis 2017 24 211 219 GotzschePC Why we need a broad perspective on metaanalysis. It may be crucially important for patients BMJ 2000 321 585 586 TranfieldD DenyerD SmartP Towards a methodology for developing evidence?informed management knowledge by means of systematic review British Journal of Management 2003 14 207 22 GargAX HackamD TonelliM Systematic review and meta-analysis: when one study is just not enough Clin J Am Soc Nephrol 2008 3 253 260 SharpeD Of apples and oranges, file drawers and garbage: why validity issues in meta-analysis will not go away Clin Psychol Rev 1997 17 881 901 HigginsJP ThompsonSG Quantifying heterogeneity in a meta-analysis Stat Med 2002 21 1539 1558 DickersinK MinYI Publication bias: the problem that won't go away Ann N Y Acad Sci 1993 703 135 146 discussion 146-148 EggerM Davey SmithG SchneiderM MinderC Bias in meta-analysis detected by a simple, graphical test BMJ 1997 315 629 634 DuvalS TweedieR Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis Biometrics 2000 56 455 463 Tables

Strengths of meta-analysis

One number cannot summarize a research field
Mixing apples and oranges
Garbage in, garbage out
Heterogeneity
Publication bias
Not all variables are comparable
Potential disagreement with findings of randomized trials
Cannot overcome subjectivity
Only deals with main effects

Weaknesses of meta-analysis

One number cannot summarize a research field
Mixing apples and oranges
Garbage in, garbage out
Heterogeneity
Publication bias
Not all variables are comparable
Potential disagreement with findings of randomized trials
Cannot overcome subjectivity
Only deals with main effects