Table 1 Overview of the Different Inclusion Approaches

Description

Primary effect sizes obtained from ILSA and non-ILSA data are meta-analyzed separately and considered as different meta-analytic data sets

Primary effect sizes obtained from ILSA and non-ILSA data are directly included in the meta-analyses, allowing for multiple effect sizes per study, cycle, or country

Stage 1: Primary effect sizes obtained from ILSA data are pooled

Stage 2: The pooled ILSA effect size(s) and the primary effect sizes of the non-ILSA data are meta-analyzed

Primary effect sizes obtained from ILSA data are meta-analyzed in the first step, and the resultant overall effect size and/or variance estimates inform the meta-analysis of non-ILSA data as priors

Analytic steps

1. Compute the primary effect sizes for ILSA and non-ILSA data, taking into account the (complex) study designs

2. Identify the meta-analytic data structures for the ILSA and non-ILSA data sets

3. Synthesize the primary effect sizes of the ILSA data via (multilevel) random-effects meta-analysis and obtain the overall effect size(s) and variance component(s) for the level of studies or cycles (e.g., PISA or PISA 2015)

4. Synthesize the primary effect sizes of the non-ILSA data via (multilevel) random-effects meta-analysis and obtain the overall effect size(s) and variance component(s)

5. Conduct moderator analyses to identify possible sources of heterogeneity for ILSA and non-ILSA data separately

6. Examine publication bias and file-drawer issues for ILSA and non-ILSA data separately

1. Compute the primary effect sizes for ILSA and non-ILSA data, taking into account the (complex) study designs

2. Combine the primary effect-size ILSA and non-ILSA data

3. Identify the meta-analytic data structure

4. Quantify the overall effect size and the respective variance component(s) via (multilevel) random-effects meta-analysis

5. Examine the possible differences in effect sizes between ILSA and non-ILSA data

6. Conduct moderator analyses to identify other possible sources of heterogeneity

7. Examine publication bias and file-drawer issues

8. Conduct sensitivity analyses with respect to ignoring the hierarchical data structure vs. modeling this structure

1. Compute the primary effect sizes for ILSA and non-ILSA data, taking into account the (complex) study designs

2. Identify the meta-analytic data structures for the ILSA and non-ILSA data sets

3. Synthesize the primary effect sizes of the ILSA data via (multilevel) random-effects meta-analysis or other aggregation approaches and obtain the overall effect size(s) and variance component(s) for the level of studies or cycles (e.g., PISA or PISA 2015)

4. Combine the synthesized ILSA effect sizes with the primary effect-size non-ILSA data

5. Quantify the overall effect size and the respective variance component(s) via random-effects meta-analysis

6. Conduct moderator analyses to identify possible sources of heterogeneity

7. Examine publication bias and file-drawer issues

8. Conduct sensitivity analyses with respect to excluding vs. including the synthesized effect size of the ILSA data

1. Compute the primary effect sizes for ILSA and non-ILSA data, taking into account the (complex) study designs

2. Identify the meta-analytic data structures for the ILSA and non-ILSA data sets

3. Synthesize the primary effect sizes of the ILSA data via (multilevel) random-effects meta-analysis and obtain the overall effect size(s) and variance component(s) for the level of studies or cycles (e.g., PISA or PISA 2015)

4. Meta-analyze the primary non-ILSA effect sizes via Bayesian random-effects meta-analysis using priors for the overall effect size and/or the variance component(s) that are informed by the overall effect size and variance component(s) of the ILSA data

5. Conduct moderator analyses to identify possible sources of heterogeneity

6. Examine publication bias and file-drawer issues

7. Conduct sensitivity analyses with respect to the choice of priors

Primary analytic approaches

Random-effects meta-analysis, meta-analysis with robust variance estimation

Multilevel random-effects meta-analysis, meta-analysis with robust variance estimation

Random-effects meta-analysis, meta-analysis with robust variance estimation

Bayesian meta-analysis

Key advantages

▪ Less complex meta-analytic models are required to synthesize the data (e.g., within-ILSA variation is no longer estimated)—standard meta-analytic models available

▪ Full information on the ILSA characteristics at the level of effect sizes can be incorporated

▪ Variation between and within studies or other entities can be quantified

▪ Moderator analyses at different levels of analysis are possible

▪ Comparisons of overall effect sizes between ILSA and non-ILSA data possible via mixed-effects models

▪ Less complex meta-analytic models are required to synthesize the data (e.g., within-ILSA variation is no longer estimated)—standard meta-analytic models available

▪ Less complex meta-analytic models are required to synthesize the data (e.g., within-ILSA variation is no longer estimated)

▪ Overall effect size and variance component(s) based on non-ILSA data are informed by ILSA data without their direct inclusion

Key challenges

▪ Separate meta-analyses of ILSA and non-ILSA data do not inform each other

▪ Fewer studies available for each of the separate meta-analyses

▪ Advanced meta-analytic methods are needed (e.g., multilevel random- and mixed-effects meta-analysis)

▪ Sensitivity analyses are required to examine the influence of including one or more synthesized ILSA effect sizes based on very large primary samples

▪ Fewer studies available for meta-analysis

▪ Limited possibilities to examine variation within ILSAs or ILSA cycles

▪ Sensitivity analyses are required to examine the influence of the priors

▪ Only non-ILSA data are meta-analyzed in the main analyses