Incorporating test-taking engagement into the item selection algorithm in low-stakes computerized adaptive tests

Large-scale Assessments in Education

An IERI – International Educational Research Institute Journal

Table 1 Results for Simulation Study 1 and 2

Test length	Evaluation criteria	Conventional MFI	Effort-guided CAT	Engagement-ranked MFI	Unconditional MFI-LE	Conditional MFI-LE (θ = 0)	Conditional MFI-LE (θ = − 1)
20	r	0.889	0.889	0.911	0.903	0.901	0.906
	Bias	− 0.011	− 0.015	− 0.001	0.004	0.026	0.036
	RMSE	0.450	0.451	0.339	0.426	0.433	0.422
	Mean cSEM	0.388	0.390	0.393	0.389	0.391	0.391
	n	19.71	19.71	19.74	19.72	19.57	19.57
30	r	0.931	0.930	0.945	0.933	0.932	0.932
	Bias	0.030	0.025	0.036	0.035	0.058	0.058
	RMSE	0.357	0.357	0.316	0.355	0.364	0.364
	Mean cSEM	0.347	0.349	0.353	0.349	0.350	0.349
	n	27.95	27.95	28.26	28.09	27.84	27.84
50	r	0.952	0.952	0.958	0.955	0.953	0.953
	Bias	− 0.079	0.061	− 0.069	0.073	0.091	0.092
	RMSE	0.303	0.302	0.282	0.296	0.313	0.313
	Mean cSEM	0.316	0.317	0.322	0.355	0.316	0.317
	n	40.80	40.80	42.06	40.95	40.63	40.63

Bold values indicate the best results for each evaluation criterion
MFI: Maximum Fisher Information, Effort-Guided CAT: Removing disengaged items when estimating ability levels after the CAT is completed, Engagement-Ranked MFI: MFI that considers item engagement ranking in item selection, Unconditional MFI-LE: MFI that considers both ability and latent engagement in item selection, Conditional MFI-LE: MFI that considers both ability and latent engagement in item selection only when latent engagement is below a certain threshold, \(r:\) Correlations between true and estimated theta values, RMSE: Root-mean-squared error, cSEM: Conditional standard error of measurement, n: The average number of items administered