The potential of international large-scale assessments for meta-analyses in education

Large-scale Assessments in Education

An IERI – International Educational Research Institute Journal

Table 5 Results of the mixed-effects meta-regression analyses of the standardized mean differences across gender including study- and country-level moderators

	Separate meta-analyses				Direct inclusion			Indirect inclusion
	ICILS 2013 data	ICILS 2018 data	ICILS 2013 & 2018 data	Non-ILSA data	One-stage inclusion	One-stage inclusion	Two-stage inclusion	Bayesian meta-analysis
Regression coefficients \(B\) (95% CI)
Intercept	− 0.03 [− 0.37, 0.30]	− 0.54[− 0.90, − .018]	− 0.17 [− 0.49, 0.14]	− 0.22 [− 1.49, 1.05]	− 0.30 [− 0.58, -0.02]	− 0.28 [− 0.57, 0.02]	0.12 [0.01, 0.22]	− 0.21 [− 1.57, 1.15]
Test fairness	–	–	–	0.20* [0.01, 0.39]	0.17 [− 0.01, 0.36]	0.18* [0.00, 0.36]	0.17* [0.00, 0.34]	0.20 [− 0.01, 0.41]
Publication type	–	–	–	− 0.24* [− 0.43, − 0.05]	− 0.21* [− 0.40, -0.01]	− 0.21* [− 0.41, -0.01]	− 0.22* [− 0.39, -0.05]	− 0.24* [− 0.45, − 0.04]
Power distance index (PDI)	0.00 [0.00, 0.01]	0.00 [0.00, 0.01]	0.00 [0.00, 0.00]	0.00 [0.00, 0.01]	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	–	0.00 [0.00, 0.01]
Global innovation index (GII)	0.00 [0.00, 0.01]	0.01* [0.01, 0.02]	0.01* [0.00, 0.01]	0.00 [− 0.02, 0.02]	0.01* [0.00, 0.01]	0.01* [0.00, 0.01]	–	0.00 [− 0.02, 0.02]
Baseline model	MEM2	MEM2	MEM3	MEM2	MEM3	CCMEM4	MEM2	MEM2

ILSA = International large-scale assessment. The results of the two-stage direct inclusion are based on the pooling of the ICILS 2013 and 2018 data via random-effects meta-analyses in the first stage. The GII was derived as the country average of the global innovation indices across publication years. PDI and GII are country-level variables, while test fairness and publication type are study- level variables. MEM2 = Two-level mixed-effects meta-regression model, MEM3 = Three-level mixed-effects meta-regression model, CCMEM4 = Four-level cross-classified mixed-effects meta-regression model (with 31 countries). Positive standardized mean differences (Hedges’ g) suggested that girls performed better than boys. * p < .05