Sampling weights in multilevel modelling: an investigation using PISA sampling structures

Mang, Julia; Küchenhoff, Helmut; Meinck, Sabine; Prenzel, Manfred

doi:10.1186/s40536-021-00099-0

Large-scale Assessments in Education

An IERI – International Educational Research Institute Journal

Table 5 Coverage Rates of PISA simulated data

From: Sampling weights in multilevel modelling: an investigation using PISA sampling structures

Software	Weighting approach	CR \(\widehat{{\beta }_{0}}\)	CR \(\widehat{{\beta }_{1}}\)	CR \(\widehat{{\beta }_{2}}\)	CR \(\widehat{{\sigma }_{\varepsilon }^{2}}\)	CR \(\widehat{{\sigma }_{\tau }^{2}}\)
A: Coverage rates—PISA simulated data—Model 1
SAS	No weights	0.00			0.94	0.98
	Unscaled weights	1.00			0.51	0.99
	Only school weights	1.00			0.94	0.95
	Only student weights	0.00			0.32	0.96
	Withincluster weights	1.00			0.62	1.00
	Scaled weights: cluster	1.00			0.95	0.95
	Scaled Weights: ECluster	1.00			0.95	0.95
	Clustersum	0.00			0.95	0.99
	House weights	0.00			0.94	0.99
Mplus	No weights	0.00			0.92	1.00
	Unscaled weights	1.00			0.97	0.97
	Only school weights	1.00			0.97	0.97
	Only student weights	0.00			0.92	1.00
	Withincluster weights	1.00			0.97	0.97
	Scaled weights: cluster	1.00			0.96	0.96
	Scaled weights: ECluster	1.00			0.96	0.96
	Clustersum	0.00			0.97	1.00
	House weights	0.00			0.97	1.00
B: Coverage rates—PISA simulated data—Model 2
SAS	No weights	0.00	0.83		0.94	0.00
	Unscaled weights	0.99	0.90		0.53	0.00
	Only school weights	0.98	0.91		0.95	0.84
	Only student weights	0.00	0.85		0.37	0.00
	Withincluster weights	0.98	0.94		0.68	0.59
	Scaled weights: cluster	0.99	0.90		0.96	0.81
	Scaled weights: ECluster	0.99	0.90		0.96	0.81
	Clustersum	0.00	0.92		0.96	0.39
	House weights	0.00	0.85		0.94	0.00
Mplus	No weights	0.00	0.91		0.95	0.61
	Unscaled weights	0.98	0.92		0.94	0.96
	Only school weights	0.98	0.92		0.95	0.96
	Only student weights	0.00	0.91		0.95	0.61
	Withincluster weights	0.98	0.92		0.94	0.96
	Scaled weights: cluster	0.99	0.91		0.95	0.97
	Scaled weights: ECluster	0.99	0.91		0.95	0.97
	Clustersum	0.00	0.92		0.96	0.39
	House weights	0.00	0.92		0.95	0.65
C: Coverage rates—PISA simulated data—Model 3
SAS	No weights	0.01	0.93	0.96	0.94	0.44
	Unscaled weights	0.96	0.94	0.93	0.53	0.45
	Only school weights	0.95	0.94	0.93	0.94	0.87
	Only student weights	0.09	0.94	0.93	0.36	0.23
	Withincluster weights	0.94	0.94	0.92	0.68	0.82
	Scaled weights: Cluster	0.96	0.93	0.93	0.95	0.86
	Scaled weights: ECluster	0.96	0.93	0.93	0.95	0.86
	Clustersum	0.01	0.92	0.96	0.96	0.38
	House weights	0.03	0.92	0.95	0.95	0.47
Mplus	No weights	0.01	0.93	0.96	0.94	0.52
	Unscaled weights	0.95	0.94	0.93	0.95	0.91
	Only school weights	0.95	0.94	0.92	0.95	0.91
	Only student weights	0.01	0.93	0.96	0.94	0.51
	Withincluster weights	0.95	0.94	0.93	0.95	0.91
	Scaled weights: cluster	0.96	0.93	0.93	0.96	0.9
	Scaled weights: ECluster	0.96	0.93	0.93	0.96	0.9
	Clustersum	0.01	0.92	0.96	0.96	0.38
	House Weights	0.03	0.92	0.95	0.95	0.51

The CR represents the compliance rate of the estimators within its 95% confidence interval of three hierarchical models. Model 1 is declared as \({y}_{ij}={\beta }_{0}+ {\tau }_{i}+ {\varepsilon }_{ij}\), Model 2 as \({y}_{ij}={\beta }_{0}+ {\beta }_{1}*{x}_{ij}+ {\tau }_{i}+ {\varepsilon }_{ij}\) and Model 3 as \({y}_{ij}={\beta }_{0}+ {\beta }_{1}*{x}_{ij}+{\beta }_{2}*{x}_{i}+ {\tau }_{i}+ {\varepsilon }_{ij}\). PISA simulated data serves as scenario template. Simulation variation is displayed with the different weighting approaches combined with different estimation algorithms implemented in the two examined software packages

Back to article page