Automatically analyzing text responses for exploring gender-specific cognitions in PISA reading

Large-scale Assessments in Education

An IERI – International Educational Research Institute Journal

Table 1 Item characteristics

Item	Cat^a	Aspect^b	Correct (%)	n	♀^c (%)	Words^d
1. explain protagonist’s feeling	WHY<s>	B	83	4152	49	12.3 (4.6)
2. evaluate statement	ENABLE<a>	C	43	4234	48	15.6 (9.0)
3. interpret the author’s intention	SIG<a>	B	10	4234	48	12.5 (6.3)
4. list recall	CON<s>	A	59	4223	50	5.6 (3.0)
5. evaluate stylistic element	HOW<s>	C	56	4234	48	14.7 (6.2)
6. verbal production	WHN<a>	B	80	4152	49	12.4 (6.9)
7. select and judge	ENABLE<s>	C	68	4152	49	13.6 (7.0)
8. explain story element	CON<e>	B	69	4223	50	14.4 (5.5)
Total			59	33,604	49	12.6 (6.1)

The items refer to seven different stimulus texts. Only items 1 and 6 ask students about the same text
^aQuestion category according to QUEST (Graesser and Clark 1985; Graesser and Murachver 1985); in the form FUNCTION<T> with FUNCTION\(\in\){WHY, HOW, ENABLE, CONS, WHEN, WHERE, SIG, WHN} and T\(\in\){state, action, event}
^bAccording to PISA framework (OECD 2013), A = Access and Retrieve, B = Integrate and Interpret, C = Reflect and Evaluate
^cPercentage of girls (note that for items 1, 4, 6, 7, and 8 there was one case with missing information about the student’s gender each, referring to two students)
^dWord count in non-empty responses on average (with SD)