A detailed description of the criteria and the process of literary search is not the subject of this review, but it is important to note that any creation of a systematic review consists of a set of items. These items are combined into a standard scheme called the PRISMA flow diagram (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), by the name of the relevant recommendations [2] (Fig. 1).

In reality, meta-analysis begins after the inclusion is completed and the studies from which the data extraction will be performed are available.

The main guidelines for the initial and further steps of practical interest to the reader are:

These guidelines will help to maintain high standards of writing systematic reviews and reduce the number of possible errors and inaccuracies that may complicate the continuation of the work. Metaanalyses are widely used by drug companies, as well as other commercial entities, and therefore observance of guidelines standards is very carefully checked by reviewers both when reviewing and when publishing articles. Strict observance of regulation and guidelines is the key to a successful publication.

After all suitable publications have been collected for inclusion in a systematic review or meta-analysis, it is necessary to evaluate them in terms of potential biases.

Unfortunately, there are many sources of potential bias, so special tools have been developed to allow researchers to conduct a potential evaluation of publications. Such tools are called risk of bias plots.

These plots can be built for studies with different design types, primarily for randomized controlled trials (RCT) (Fig. 2) and non-randomized trials (NRT) [3] [4] (Fig. 3). The plots below are called “traffic lights”. The basic idea is that researchers conducting a metaanalysis with the inclusion of RCTs (Fig. 2) review each study separately and assess the risks associated with the following five domains:

The assessed risk is “high”, “some concerns” and “low”.

If NRTs are included in the meta-analysis, we assess the risks associated with the following seven domains:

The assessed risk is interpreted as “critical”, “serious”, “moderate” and “low”.

Specified tools enable critical approaches to the results obtained in the meta-analysis and consider studies with a high risk of bias as less reliable. A detailed description of the capabilities of this tool is provided on a specialized website2.

The results of studies combined in a metaanalysis are measured by an identical endpoint. This measurement is an effect that has been achieved, or an observed effect (abbreviated as θ or θk for each of the k studies). The definitions of the effect, effect size and effect size measuring are described in our publication on statistical hypotheses testing [6].

There are two main concepts that allow us to describe and measure the effect of several studies. Both concepts relate to certain statistical models, with fixed and random effects, respectively.

This is a model in which the studies included in the meta-analysis are very similar to each other in terms of design, number of patients, methodology, evaluation of results and other items, and their results or effects θ1,2,3,...k are considered a single sample from one general population of all possible similar studies.

The probability distribution of such a population has an expected value (the mean weighted by the probabilities of possible values), which represents a certain true effect size θ̂. Each study is a part of the population, and several studies randomly taken from such distribution (meta-analysis) is an ordinary sample, respectively.

The observed effect of each study k will differ from the true one by the error:

θ̂ = θk + ϵk.

We believe that among several studies, the studies with the smallest sample error ϵ are the most accurate (Fig. 4).

A sample from several studies {1,2,3,...k} must have some central tendency or expected value indicating the true effect size. Thus, using the definition of expected value as a mean weighted, we get:

where • θ̂ – weighted effect size for k studies resulting from meta-analysis;

However, it is still unclear how to achieve the weight of each study. We know that the observed effect obtained in study k is a point estimate. Study k includes a certain set of patients, n. The standard error is a measure of the variability of the effect θk and is calculated as:

where

In a fixed effects model, the inverse variance method is one of the ways to calculate weights:

where

sk 2 – the square of the standard error of the effect θk in the study k;

• wk – weight of the study k.

Thus, in a model with fixed effects, the weight of any study is inversely related to the inverse effect error in the study and directly related to the number of patients included in the study. The model implies that only the number of patients can affect the weight of the study.

This concept seems to be overly simplified, because in the real world, there are a huge number of different factors apart from the sample size.

If we combine different designs (RCTs, cohorts, etc.) in a meta-analysis studies conducted in different periods of time, in different countries, in hospitals with different standards of medical care, with intervention protocols according to different clinical guidelines, we will need a model that will take into account not only differences in sample size, but also the abovementioned factors.

In this type of model, the effect for each study included in the meta-analysis is a sample from its own set of effect sizes and differs from the expected value of its own set by the ϵk (Fig. 5).

If we included k studies in the meta-analysis, there are k samples from k different general populations. At the same time, k populations have their own distribution with the expected value represented by weighted effect size, and each general population differs from the expected value of its distribution by ξk. This overall distribution has a point estimate, which corresponds to the meta-analysis weighted effect, θ̂, and variance τ2.

Thus, the point estimate of the effect in each study among k differs from the weighted effect the following way:

θ̂ = θk + ϵk + ξk.

The parameter ξk combines the differences that are not related to the sampling error.

The model that uses this logic is called the random effects model. The individual studies weights in such a model are calculated the following way:

where

And the weighted effect is calculated in the same way as for the fixed effects model:

where

Various mathematical approaches are used to calculate parameter τ2, most often DerSimonian- Laird, Restricted Maximum Likelihood, Maximum Likelihood, Paule-Mandel estimators but there are others [7–9]. The choice of a specific method depends on the type of measurement of endpoint and on the specific situation, therefore it requires consultation with a biostatistician.

We discovered that the studies included in the metaanalysis can vary significantly, moreover, depending on these differences, one or another analysis model is chosen. Is there any measure that can assess the degree of differences? Can we somehow explain the degree of differences and is it necessary? How to determine which studies are more different from others? The concept of heterogeneity can answer all these questions.

Heterogeneity depends on many things, with the most common being:

When a researcher encounters excessive heterogeneity, such as in a situation where a number of studies have opposite effect direction, or where effect sizes are substantially different, it is necessary to understand whether there is an erroneous inclusion of studies in the meta-analysis. An attempt to combine studies in which completely different parameters were evaluated will lead to huge heterogeneity and will not answer the meta-analysis question. The results of such a meta-analysis will be highly doubtful.

If we believe that there is no error at this stage, it is necessary to look further for the cause of high heterogeneity. For example, when non-standardized parameters are used, it is necessary to try to use standardized instead. If several parameters are measured on different scales or differ significantly on inclusion between studies, standardization makes it possible to smooth out such differences.

The next reason for the high heterogeneity is the presence of covert and overt moderators or confounders. For example, when assessing the prevalence of cardiovascular diseases, sex and age group of patients are obvious moderators. The inclusion of a moderator and the assessment of its impact on the effect and heterogeneity is carried out using meta-regression analysis or meta-regression. Further analysis of subgroups can significantly reduce heterogeneity.

Finally, a small number of studies in a meta-analysis can result in high heterogeneity.

There are 2 main types of heterogeneity by Rücker [10]:

Cochran's Q

We looked at two types of models and realized that there is an observable effect θ̂ k of a certain study k, as well as a weighted effect that we calculate θ̂ for all metaanalysis studies. We also remember that each study has its own weight wk. The deviation of the observed effect from the weighted one can be directed in both the direction with a plus sign or a minus sign. If we square the deviation, it will not depend on the direction.

The sum of the weighted squares of such deviations is called Cochran’s Q:

We can calculate the deviation of the observed effect from the weighted one for all studies.

The Cochran’s Q is distributed as a χ2 statistic with K – 1 degrees of freedom, where K is the number of studies in the meta-analysis.

Cochran’s Q will grow with an increase in the number of studies in the meta-analysis, as well as with the presence of large studies with many patients in it.

Higgin’s & Thompson’s I2

I2-statistic

I2 is calculated from Cochran’s Q and describes the percentage of heterogeneity which is not caused by sampling error ξk. A null hypothesis occurs when there is no heterogeneity, and Cochran’s Q follows a distribution of χ2 with K – 1 degrees of freedom (expected heterogeneity). But we also have the observed heterogeneity of Q. Then the deviation of the observed heterogeneity from the expected one is:

and is expressed in unit fractions or a percentage.

Heterogeneity can be qualified as low, moderate, and high, with upper limits of 25%, 50% and 75%, respectively [11].

H2-statistic

H2-statistic calculates the ratio of Q-statistics to K – 1. If there is no heterogeneity, then the value tends to 1; higher values indicate the presence of heterogeneity between studies.

Heterogeneity of variance τ2

The true weighted effect has its own variance τ2 and standard deviation τ. This parameter is used to evaluate the measure of heterogeneity and has the same dimension as the effect in studies in meta-analysis. If we know the calculated weighted effect size θ̂ in the meta-analysis, we can estimate the 95% confidence interval (CI) of the true effect as θ̂ ± 1,96 × τ.

Sensitivity analysis shows how individual studies can influence the weighted effect, and how stable the results of the meta-analysis are. The leave-one-out is one of the most used methods for the evaluation of sensitivity. Each study is excluded from the metaanalysis on an individual basis, then the weighted effect size and heterogeneity are recalculated. Serious changes in the effect size and a decrease in heterogeneity indicate that the excluded study has a significant impact on the overall result. If at the first stage in the “traffic light” plot the study has a high/some concern risk of bias, then at the stage of using the leave-one-out method it may be an outlier, and it will be necessary to consider the feasibility of its presence in the meta-analysis.

Graphically, the results of the sensitivity analysis are presented in Figure 6.

In the example from Figure 6, the weighted effect obtained in the meta-analysis is the mean difference and amounted to 0.16 [0.1; 0.23]. We see that when studies are excluded from meta-analysis on an individual basis, the weighted effect does not change significantly. At the same time, according to I2 statistics, the exclusion of the Protocol 162A study significantly reduces heterogeneity to 5%. This study requires close attention since its presence causes high heterogeneity in the whole meta-analysis.

Sensitivity analysis evaluates how the weighted effect changes when excluding studies that received “high” and “very high” risk levels when assessing potential risks of bias. If, when one excludes one of these studies, they significantly influence the effect (for example, when, after excluding high-risk studies, 95% CI of the new weighted effect ceases to include the point estimate of the weighted effect before excluding studies), it is necessary to reconsider the need to include high-risk studies in the meta-analysis.

Forest plot is the most common way to summarize the results of a meta-analysis in a single image. It shows the studies included in the analysis, the effect of each one of them, the weighted effect, as well as a set of additional parameters, for example, the weights of each study, parameters of heterogeneity, the type of chosen statistical model (fixed or random effects). By using forest plots, you can separately duplicate the effect sizes and their 95% CI, as well as the weighted effect and their 95% CI.

The size of the points on the plot that characterize a particular study is often associated with the weight of the study (the points of the largest size are associated with studies with the highest weight, respectively). A common forest plot is shown in Figure 7. The research data are taken from the materials accompanying the meta library of the R programming language [12].

From Figure 7, we can see the names of the studies and the year of results publication, the characteristics of the test and control groups (quantities, means and standard deviations required to calculate the standard error), the effect size (in this case, mean difference, MD) and its 95% CI, weights in fixed and random effects models, weighted effect for both types of models, as well as heterogeneity parameters.

We also see a chart showing all the same effect sizes in studies (represented by squares, the size of which proportional to weights) and the weighted effects for fixed and random effects models (represented by diamond).

A solid vertical line is known as the “line of null effect”. If 95% of the CI of individual studies or weighted effects pass through the “line of null effect”, the study data is said to be statistically insignificant, and there is a high probability that the observed point estimates are random.

In this example, Floral 1971 was the only study where a significant effect was observed; its weight was the highest in both fixed and random effects models (38.6% and 33.3%, respectively). The fixed effects model showed a significant effect, the means difference was -0.71 [-1.26; -0.16], while the random effects model was insignificant, since the point estimate was -0.75, and 95% CI included zero [-1.53; 0.03] according to the results of the meta-analysis

In this section of the article, we have introduced the reader to the stages of including studies in metaanalysis, reviewed the existing guidelines that you need to familiarize with when writing a meta-analysis, analyzed in detail the process of creating weights, various types of models used in meta-analyses. We also became acquainted with the definition of heterogeneity and understood how it is calculated as well as the main plot published in meta-analyses such as the forest plot and the leave-one-out plot for sensitivity analysis were presented.

In the next article, we will consider how to analyze subgroups, familiarize with meta-regression, and learn how to evaluate the publication bias, visually and mathematically. Additionally, we will recall the most common ways of evaluating the meta-analysis effect and focus on standardized indicators and the evaluation of the standard error for them.

Alexander Yu. Suvorov, Irina V. Latushkina and Kseniya А. Gulyaeva equally contributed to this work and should be considered the first co-authors. Alexander Yu. Suvorov, Irina V. Latushkina, Kseniya А. Gulyaeva, Nikolay M. Bulanov, Maria Yu. Nadinskaia and Alexey A. Zaikin participated in writing the text of the manuscript. Alexander Yu. Suvorov, Irina V. Latushkina, Nikolay M. Bulanov and Alexey A. Zaikin searched and analyzed the literature on the topic of the review. Alexander Yu. Suvorov developed the general concept of the article, Alexey A. Zaikin supervised writing the text of the manuscript. All authors participated in the discussion and editing of the work. All authors approved the final version of the publication.

Conflict of interests. The authors declare that there is no conflict of interests.

Financial support. This article was supported by the Academic leadership program Priority 2030 proposed by Sechenov First Moscow State Medical University (Sechenov University)

1 Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from https://training.cochrane.org/handbook/current 2 https://www.riskofbias.info/ (access date 01.11.2022).