Unsystematic Versus Systematic Approaches

When an unsystematic approach is applied in research synthesis, there are typically no strict criteria for the inclusion or exclusion of studies and the weight placed on the findings of single studies. For example, one researcher may determine the weight that is placed on studies based on sample size while another researcher may determine the weight that is placed on studies based on the quality of studies. As a result, unsystematic reviews tend to be biased because these rely heavily on the subjective decisions of researchers (Borenstein et al., 2009). Moreover, due to the lack of decision-making rules, inconsistent decisions can be made that make it impossible to determine the nature of the bias within an unsystematic research synthesis.

Research syntheses that use a systematic approach differ from unsystematic research syntheses by applying strict rules for making decisions regarding the search for studies, inclusion criteria, and the weight that is put on the findings of single studies (e.g., Eysenbach, Powell, Kuss, & Sa, 2002). The rules enable researchers to review studies in a systematic way that can be replicated by other researchers (Borenstein et al., 2009). Although rule specification still includes some degree of subjective judgments with a systematic approach such subjectivity can be taken into account as long as the rules are thoroughly reported and consistently applied. Therefore, research syntheses using a systematic approach (i.e., systematic reviews and meta-analyses) are considered superior to research syntheses using an unsystematic approach (i.e., narrative reviews).

The Use of Meta-analytic Techniques

Literature on meta-analyses suggests that the main difference between meta-analyses and the two other types of research syntheses outlined above is the use of vote counting instead of meta-analytic techniques. In vote counting, researchers simply count the number of studies that did find statistically significant results and the number of studies that did not find statistically significant results based on a specified cut-off value (typically p < .05). Based on the number of votes, the researcher judges whether there is an effect or not. However, vote-counting can yield misleading results, particularly when the number of included studies in a review increases (Hedges & Olkin, 1980).

The issue with vote-counting is that it is based on p-values of single studies. The p-value of a single study is the product of the observed effect, variance, and the sample size. However, without an adequate sample size, the p-value will not indicate that an effect is statistically significant, even if the magnitude of the observed effect is substantial. As such, if a large proportion of studies have inadequate sample sizes, then a researcher will draw the erroneous conclusion that no effect exists.

Meta-analyses are unlikely to suffer from this problem, because they combine the information of a series of single studies. This helps meta-analyses provide not only a more accurate summary of effects across studies but also more powerful tests of summary effects than the individual tests of single studies (Borenstein et al., 2009). Consequently, small effects may be revealed that would not be found through vote counting.

The lack of adequate sample sizes poses a serious threat to the validity of vote-counting studies in communication science. This can be illustrated using a meta-analysis by Shen, Sheer, and Li (2015) comparing the persuasiveness of narrative versus nonnarrative health messages. The meta-analysis revealed a significant overall mean effect size of r = .063 in favor of narrative health messages. However, if these authors would have resorted to a vote-counting method instead, the findings could have been very different due to a lack of power in the included studies. To better illustrate this contention, a post-hoc power analysis was conducted for 19 of the 33 included studies (the remaining 14 did not provide sufficient information) based on the mean effect size found in the meta-analysis and the sample sizes of the single studies. For all these 19 studies, power was below .20, indicating that each study had less than 20% chance of finding a persuasive advantage of narrative over non-narrative messages. Because a minimum of 80% power is typically desired (Cohen, 1992), these 19 studies had severely inadequate sample sizes. This issue can further be demonstrated by the significance tests of the 19 studies. Only 7 of the 19 studies showed statistically significant p-values (p < .05). Therefore, a vote-counting study would have resulted in the erroneous conclusion that no persuasive advantage of narrative (over nonnarrative) messages exists.

Database Search for Peer-Reviewed Literature

The search for peer-reviewed typically starts with database searches, which may be followed by a search through the reference lists of retrieved articles and a cited reference search for the retrieved articles. The first step of the database search is to select a set of appropriate bibliographic databases in which the search will be conducted. Bibliographic databases index articles that are published in journals from a certain domain. Many databases are available, covering a wide range of domains. Although databases may overlap, each database indexes an unique set of journals (Lipsey & Wilson, 2001). For example, PsycINFO indexes journals within the domain of psychology and MEDLINE journals within the domain of health-related topics. Overall, there are many differences between these databases. However, they also overlap because both databases index journals on topics like health communication. When conducting a literature search, it is important to select a set of databases that covers the journals in which relevant studies are published.

After selecting the databases, a string needs to be composed of search terms that results in the retrieval of as many relevant and as few irrelevant hits as possible. Often, this takes a lot of effort. On one hand, a search string that is too broad may result in too many hits to scan through. On the other hand, a search string that is too narrow may miss many relevant articles. Therefore, a suitable search string requires testing and fine tuning.

Various operators help to fine tune the search string and specify which fields need to be included in the search. Thus, it is possible to limit the search to only the title or to include fields such as the abstract and key words if desired. In addition, it is possible to search for combinations of words or to specify words that should not be included (e.g., “Introduction to Boolean Logic,” n.d.). For example, a search can specify that the words “health” and “education” need to appear together when the meta-analysis focuses on health education. Furthermore, many databases offer the opportunity to use truncations (e.g., “Truncation,” n.d.). These are special operators that allow a search for various variations of a word. Truncations are often symbols like “*” or “?”. If a researcher, for example, wants to retrieve items that include both the words “story” and “stories,” the search term “stor*” could be used. This search term will match any word that begins with “stor,” including the words “story” and “stories.” However, it will also match words like “store” and “storage.” Therefore, when using truncations, the possibility of irrelevant hits and their extent should be considered.

After the search string is composed, it needs to be run on the set of databases; however, the syntax and search options vary between databases. For example, the operator for a truncation may be “*” in one database and “?” in another. Hence, the search string needs to be adapted for the various databases. Both the selection of databases and the composition of a search string are complicated tasks for which the assistance of an experienced librarian may be preferred, depending on the experience and skills of the researchers who are conducting the meta-analysis. The quality of the database search is an important determinant of the quality of a meta-analysis. Therefore, the need for the assistance of a librarian should not be underestimated.

Reference List Search and Cited Reference Search for Peer-Reviewed Literature

Following the database search, researchers can choose to scan the reference lists of articles that have been retrieved. Using this procedure, researchers can ensure that relevant articles not retrieved during the database search will eventually be included. Similarly, a cited reference search can be conducted using Web of Knowledge, which allows researchers to search for articles that have cited the articles that were retrieved during the database search. The important difference between the reference list search and the cited reference search is that the former focuses on articles that were published prior to an article and the latter focuses on articles that were published following an article (Lipsey & Wilson, 2001).

Grey Literature Searches

A grey literature search is a search for studies that have not been published in any commercial outlets and, thus, have not been published in a peer-reviewed journal (“What is Grey Literature?,” n.d.). This procedure is often less structured than the search for peer-reviewed articles because few resources such as databases and reference lists can be searched to retrieve nonpublished studies. Instead, researchers have to initiate calls for papers through professional organizations and websites or email scholars who are known to work on related topics to ask for unpublished data. Another possibility is to scan the programs of relevant conferences for relevant presentations and to ask the presenters whether they would like to share their data (Lipsey & Wilson, 2001). The success of this approach relies heavily on the success of researchers to approach the right people and their willingness to collaborate. Therefore, a grey literature search can be very challenging.

Searching through grey literature both has advantages and disadvantages. The first advantage is that it allows the opportunity to retrieve studies that are still in the process of getting published. This process can take many months, but by searching for grey literature, researchers might be able to retrieve the manuscripts before they are published. The second advantage of a grey literature search is that it is more likely to retrieve studies with statistically nonsignificant findings. Peer-reviewed scientific literature is likely to contain a significance bias, because journals are more likely to publish studies that show statically significant findings (Rosenthal, 1979). Moreover, researchers who are aware of this practice often do not even submit reports of studies with non-significant findings, because they perceive it to be unlikely for these reports to be accepted by any journal (Easterbrook, Gopalan, Berlin, & Matthews, 1991). Hence, a grey literature search may provide access to studies that would not be retrieved by searching through peer-reviewed articles.

A potential disadvantage of grey literature may be a lack of quality control. The peer-review process of scientific journals is intended to guaranty the quality of studies that are published. Grey literature is often not subjected to any peer-review process (e.g., papers from file drawers) or subject to a peer-review process of lower quality (e.g., conference papers). However, this may only be a problem when studies are included in a meta-analysis without conducting a quality check. Another disadvantage of grey literature concerns the possibility to replicate the search. It is difficult to document a search through informal channels. This makes it unlikely that the search can be replicated identically by other researchers or would result in the same outcomes.

Computing Effect Sizes

After the literature search is complete, effect sizes need to be computed for all the included studies. This first requires researchers to determine the type of effect size that will be used, which should be the effect size that best fits the type of data reported in the majority of the included studies. There are several options to choose from, but the most commonly used effect sizes in communication meta-analyses are the standardized mean difference (Cohen’s d) and the correlation between two variables (Pearson’s r). The standardized mean difference is most appropriate when comparing two groups for which subgroup mean values, standard deviations, and group sizes have been reported in the single studies (Borenstein et al., 2009). For example, standardized mean difference was applied in our recent meta-analysis on experiments that compared the persuasive effects of statistical and narrative evidence. This meta-analysis strictly focused on studies that compared separate groups that were either exposed to narrative evidence or statistical evidence. To compute effect sizes, the separate group mean scores were used for the outcome variables: beliefs, attitude, and intention (Zebregs, van den Putte, Neijens, & De Graaf, 2015). Pearson’s r is most appropriate for examining the relation between two observed (measured) variables for which correlation coefficients sample sizes have been reported in the majority of the primary studies (Borenstein et al., 2009). For example, Pearson’s r has been used to examine the relation between exercise intentions and behavior (Rhodes & Dickau, 2012). Although most researchers logically choose between d and r based on their research question, it is possible to convert d into r and r into d. Therefore, r is sometimes used as an effect size in meta-analyses that compare two groups (Borenstein et al., 2009).

After selecting which effect size to use, the necessary data needs to be extracted for all the studies that have been deemed relevant for inclusion. However, frequently not all necessary data are reported. In such cases, there may be two possibilities. First, alternative formulas may be used that require alternative statistics to be reported. If the alternative data are available, then this may be an appropriate solution (Lipsey & Wilson, 2001). Second, authors could be contacted with a request to provide the data (Lipsey & Wilson, 2001). However, such attempts are often unsuccessful. A recent meta-analysis by Tannenbaum and colleagues (2015), for example, reported that 39 of the 163 eligible studies they retrieved did not provide all the necessary data. After multiple contact attempts, they were able to retrieve the necessary data for only three studies. Six authors responded that they were unable to provide the relevant data, while all other authors did not respond at all.

Mean Effect Size

After computing the effect sizes for the single studies, a mean effect size can be computed to summarize the effects of the single studies. There are two approaches for computing a mean effect size: fixed effects and random effects models (Borenstein et al., 2009). Fixed effect models assume that a true effect size exists that is identical for all studies and does not take into account variance between studies, whereas random effect models assume that the true effect size differs between studies and does take into account the variance between studies. This difference translates into the weight that is assigned to the effects sizes of single studies. In fixed effects models, the weight is fully determined through sample sizes, assuming that larger studies provide better information about the true effect size. Therefore, in fixed effect models, studies with larger samples receive more weight than smaller studies. In random effect models, it is assumed that each study provides information about a different effect size. As such, it is important that the information from each single study is sufficiently represented in the mean effect size. Hence, in random effect models, there is less difference between the weights that is assigned to larger and smaller studies (Borenstein et al., 2009).

The decision to use fixed effects or random effects models depends on whether it is assumed that a true effect size exists that is the same for all studies. If such an true effect size is assumed to exist, then a fixed effect models should be used, whereas random effects models should be used when it is assumed that such a true effect size does not exist (Borenstein et al., 2009). For meta-analyses on experimental communication studies, it can generally not be assumed that a true effect size exists that is the same for all studies. Manipulations differ between studies, which inevitably cause differences between the true effect sizes of studies. For correlational studies, the choice between fixed effects and random effects models is somewhat more complicated. If relations between reliable validated measures are used, then a true effect size may exist that is the same for all studies. In any other case, differences may exist between studies, and random effects models may be preferred.

In addition to the distinction between fixed and random effects models, variations of random effects models exist that make statistical corrections for methodological factors of individual studies. These variations are referred to as the Hunter and Schmidt approach and are commonly applied in meta-analyses on health and risk communication. Following the Hunter and Schmidt approach, effect sizes of single studies are corrected for sources of bias (e.g., sampling error, attenuation, and reliability of independent and dependent variables) before mean effect sizes are computed. However, this approach is seldom fully applied because most studies do not fully report all sources of error (Hunter & Schmidt, 2004). Currently, no evidence exists that the Hunter and Schmidt approach is superior to approaches that do not correct for methodological sources of bias or the other way around (Johnson et al., 1995).

Heterogeneity

After a mean effect size is computed, a heterogeneity test should be conducted in which the null hypothesis is tested that all studies share the same true effect size, which would be the case if the heterogeneity is zero. Although this does not assume that all effect sizes are identical (each single study would still include random error), it would be expected that each effect size falls within some range of the common effect. However, if the true effect does differ across the single studies, then the variations between studies would include more that within-study errors. Additional variance would be included that reflects the real differences between effect sizes. This variance is called heterogeneity (Borenstein et al., 2009).

The existence of heterogeneity is examined by conducting a Q-test. If this test returns a statistically significant value, then there may not be one true effect size that is shared by all studies. However, the Q-test only examines whether or not there is heterogeneity; it does not examine the magnitude. Thus, it is important to compute I², which indicates the percentage of real variance between studies (the total amount of variance minus random within-study error; Borenstein et al., 2009). If the value of I² is low, then it may not be necessary to examine explanatory factors because the amount of variance that could be explained is too low. It is suggested that an I²-value below 40% could perceived as an indication that there is insufficient variance between studies that could be explained (Higgins & Green, 2011). Nevertheless, this decision always remains arbitrary.

Notably, when the Q-test returns a statistically nonsignificant value, this does not necessarily imply that no heterogeneity exists. There could be various causes of such a non-significant effect such as a lack of statistical power or a large amount of within study error caused by imprecise studies. For this reason, the Q-test should not be perceived as a decision-making test to choose between fixed effects and random effects models. The Q-test and the decision between models both focus on the existence of a true effect size that is the same across studies, but the alternative explanations for nonsignificant Q-test results make it inappropriate to use this test for choosing between fixed effects and random effects models (Borenstein et al., 2009).

Moderators

When I² (e.g., I² > 40%) indicates the existence of a substantial amount of real variance between studies, then it could be warranted to examine factors that may theoretically explain the differences between studies. Several techniques can be applied for this purpose, varying from subgroups analyses for examining the influence of a single factor with a limited number of values to meta-regression analyses in which multiple factors can be examined. The most appropriate test should be selected based on the characteristics of the study. It is very important to apply a statistical test before conclusions are drawn. Approaches that do not involve statistical tests are likely to result in misleading conclusions because differences between mean effect sizes will always emerge due to random within-study error, and differences between significance tests could be the result of variations in statistical power between subgroups (Borenstein et al., 2009).

Statistical Tools

Several software packages and tools can be used for conducting meta-analyses. Software packages like DSTAT, RevMan, and Comprehensive Meta-Analysis are specifically developed for this purpose. Macros and packages are available for programs like SPSS, R, and STATA (e.g., Sterne, Bradburn, & Egger, 2008). For the calculation of effect sizes from single studies, online tools available; however, these do not allow researchers to compute a mean effect size or additional analyses (e.g., “Practical Meta-Analysis Effect Size Calculator,” n.d.). In addition, it is possible to use a spreadsheet program like Microsoft Excel to perform the calculations involved in a meta-analysis. A common approach is to compute the effect sizes of single studies online and then to conduct all further analyses in a spreadsheet program. The decision to use a specific tool depends on the personal preferences of the researcher conducting the meta-analysis. In all cases, the decision should not influence the outcome.