The standard setting process: validating interpretations of stakeholders

Large-scale Assessments in Education

An IERI – International Educational Research Institute Journal

Table 4 Science ability cut score VCs and G coefficients of the reliability analyses

Effect	df	MC items				Effect	df	CMC items
		Round 1		Round 2				Round 1		Round 2
		VC	%	VC	%			VC	%	VC	%
\({\hat{\sigma }}^{2} \left( \text{i} \right)\)	16	0.179	47.4	0.063	48.1	\({\hat{\sigma }}^{2} \left( \text{i} \right)\)	12	0.045	41.7	0.043	41.7
\({\hat{\sigma }}^{2} \left( \text{e} \right)\)	8	0.105	27.8	0.069	1.5	\({\hat{\sigma }}^{2} \left( \text{e} \right)\)	8	0.000	0.0	0.000	0.0
\({\hat{\sigma }}^{2} \left( {\text{ie}} \right)\)	128	0.094	24.8	0.066	50.4	\({\hat{\sigma }}^{2} \left( {\text{ie}} \right)\)	96	0.026	24.1	0.000	0.0
						\({\hat{\sigma }}^{2} \left( {\text{c}:\text{e}} \right)\)	27	0.012	11.1	0.017	16.6
						\({\hat{\sigma }}^{2} \left( {{\text{ic}}:{\text{e}}} \right)\)	324	0.025	23.1	0.043	41.7
\({\hat{\sigma }}^{2} \left( {{\text{X}}_{\text{ie}} } \right)\)		0.378	100	0.198	100	\({\hat{\sigma }}^{2} \left( {{\text{X}}_{{{\text{ie}}\left( {\text{ce}} \right)}} } \right)\)		0.108	100	0.103	100
Φ		0.63		0.48				0.42		0.42

\({\hat{\sigma }}^{2} \left( {{\text{X}}_{\text{ie}} } \right)\) and \({\hat{\sigma }}^{2} \left( {{\text{X}}_{{{\text{ie}}\left( {\text{ce}} \right)}} } \right)\) are total scores of variance, \(\hat{\sigma }^{2} \left( e \right)\) = variance of experts, \(\hat{\sigma }^{2} \left( i \right)\) = variance of items, \(\hat{\sigma }^{2} \left( {ie} \right)\) = variance of the interaction between experts and items, \(\hat{\sigma }^{2} \left( {ce} \right)\) = variance of the cut scores that are nested in experts and \(\hat{\sigma }^{2} \left( {{\text{ic}}:{\text{e}}} \right)\) = variance of interaction between the items and the cut scores that are nested in experts
Φ index of dependability, VC variance components, df degrees of freedom