No substantive effects of school socioeconomic composition on student achievement in Australia: a response to Sciffer, Perry and McConney

Abstract

In this journal, Sciffer et al. (Large-scale Assessments in Education 10:1–22, 2022), hereafter SP&M, conclude that school socioeconomic compositional (SEC) or school socioeconomic status (school-SES) effects in Australia are substantial and substantively important for research and policy. This paper demonstrates that these claims are unwarranted. Their SEC estimates are much larger than estimates from comparable studies and a metastudy. Despite plausible theoretical reasons and empirical evidence, SP&M do not consider that school academic composition is a significant predictor of student achievement independent of SEC. SEC effects are confounded by academic composition and are typically trivial when considering academic composition. The second part of this paper compares SP&M’s estimates with analysis of the same data, from the Australian National Assessments in Performance—Literacy and Numeracy (NAPLAN). In a model corresponding to SP&M analyses comprising demographics, SES, school-SES, and student-level prior achievement, the effects of school-SES are small, with standardized effects mostly less than 0.10. With the addition of academic composition measured by school-level prior achievement, school-SES effects are effectively zero. In contrast, academic composition has significant, albeit small, impacts on student achievement. Therefore, contrary to SP&M’s (2022) conclusion, school-SES effects on student achievement in NAPLAN are negligible, whereas school-level prior achievement has small effects. That is not to say that school-SES is always irrelevant, but any assessment of its importance must consider both student- and school-level prior achievement.

Introduction

In this journal, Sciffer et. al. (2022, p. 1) hereafter SP&M, claim that school socioeconomic composition¹ (SEC) “is of practical significance to policy makers and educational researchers” based on analyses of Australian national assessment data. They highlight extraordinarily large school-SES effects and propose a raft of policies to address the supposedly large SEC effects. These policies include, requiring private schools to enroll a diverse academic mix of students, prohibiting private schools from allowing lower achievers to drop out of school, undefined amelioratory education policies that address school compositional effects, funding reforms, and adjusting students’ tertiary entrance scores for university entry based on school demographic factors (2022, p. 17).

This paper demonstrates that SP&M’s conclusions on SEC effects are unwarranted; thus, their policy recommendations are irrelevant. Their highlighted estimates for SEC or school socioeconomic status (SES) are several times larger than comparable studies. Furthermore, they do not consider school academic composition as influencing student achievement independent of school-SES.² There are sound theoretical reasons why academic composition may be significant, and the empirical literature indicates that it has more substantial effects than school-SES. Typically, studies find trivial or no effects of school-SES when considering academic composition. The first part of this paper discusses these issues.

The second part of this paper reports analyses of the same data as SP&M analysed, the 2017 data from the Australian National Assessments in Performance—Literacy and Numeracy (NAPLAN). It shows that school-SES effects are minor when considering student-level prior achievement (model 1). The standardized effects for school-SES are most often below 0.10. In contrast, SP&M (2022, p. 12) claim that standardized school-SES effects average 0.32 among primary school students and 0.52 among secondary school students. With the addition of school-level prior achievement, school-SES effects are effectively zero (model 2). Model 3 assesses the importance of school-SES for student achievement by removing it from the analyses. Its deletion does not reduce the model’s explanatory power and the parameter estimates are mostly unchanged, providing further evidence that according to Australian NAPLAN data, school-SES is irrelevant to student achievement.

Statistical and methodological issues

Extraordinary large school-SES effects

SP&M’s (2022, p. 9) equation 1 specifies student achievement as a function of demographic factors, student-level SES, school-SES, and student-level prior achievement, but not academic composition. They report substantial indirect standardized school-SES effects of 0.32 for Year 5 and 0.48 for Year 9.

Published studies report much smaller SEC effects than SP&M (2022) from analyses using the same model specification, that is controlling for student-, but not school-level, prior achievement. Marks (2015, p. 15) reports standardized school-SES effects between zero and 0.05 for Year 5 NAPLAN in Victoria. Analysing NAPLAN and tertiary entry scores for students in New South Wales, Lu and Rickard (2014, p. 32) report standardized school-SES effects of 0.08 for Years 3 to 5, 0.03 for Years 7 to 9 and 0.13 for Years 11 to 12. Armor et. al. (2018) report standardized school-SES effects around 0.05 for math and reading scores in population data from three US states. In a meta-analysis of school-SES effects, van Ewijk and Sleegers (2010, p. 145) report that school-SES effects are, on average, 0.15 standard deviations higher in the absence of student-level prior achievement. Since their average standardized effect of school-SES is 0.32, the average school-SES effect is 0.07 for studies that include prior achievement. So, SP&M (2022, p. 12) estimates for school-SES are at least four times larger than estimates from comparable studies.

School-SES effect confounded by schools’ academic composition

SP&M (2022, p. 12) specify academic composition as only mediating the effects of school-SES on student achievement. They do not consider that schools’ academic composition may directly affect student achievement independently of school-SES. There are strong theoretical reasons why school-level prior achievement may influence student performance, net of student-level prior achievement. In schools catering to higher achieving students, schools pitch the curriculum at a higher level, teachers demand more from their students, and competition among peers raises standards and expectations. Students have more positive attitudes toward learning, their school, and their teachers. There are fewer disruptions to learning. Overall, the school’s environment is more academic. These social processes cannot be construed as simply reflections of school-SES.

School-SES effects are typically trivial or are not statistically significant, net of school-level prior achievement. In a review of school effectiveness research, Scheerens et al. (2001, p. 136) conclude that “it is contextual IQ effects (or contextual effects of previous achievement) rather than contextual SES effects that seem to predominate.” Snidjers and Bosker’s (2012, pp. 84–86, 121–122) graduate-level textbook on multilevel modeling presents several examples where the effect of school IQ is positive and non-trivial and estimates for school-SES are much smaller, often negative and mostly not statistically significant. SP&M (2022, p. 4) are aware that school-SES effects are seriously confounded by school-level prior achievement and cite several studies that find small-to-zero effects for school-SES (Dumay & Dupriez, 2008; Lauder et al., 2010; Marks, 2015). Although SP&M (2022, p. 4) acknowledge that school-SES effects on achievement are often negligible, net of academic context, they perversely interpret the finding of small-to-zero SEC effects as indicating that “academic context may mediate socioeconomic context” rather than socioeconomic context is unimportant.

SP&M’s (2022, p. 9) equation 1 does not specify that academic composition has a direct effect on student achievement. In addition, academic composition is specified as mediating the effects of school-SES. There is no reason to specify direct effects for school-SES but not for school academic composition. Their specification means that the effects of school-SES are exaggerated because they incorporate the stronger effects of academic composition. A more appropriate specification would be their model 1 equation with the addition of academic composition.

Tan et. al. (2023, p. 17) conclude from their meta-analysis that school-SES is not associated with several school processes: instructional programs, educational resources, and parental involvement. If school-SES is as important to research and policy as SP&M (2022) contend, it should be associated with at least one of these school processes.

Reanalysis of the 2017 NAPLAN data

The data analysed is from the 2017 NAPLAN tests administered by the Australian Curriculum, Assessment and Reporting Authority (ACARA).

The Australian educational system comprises three types of schools: government, Catholic, and private. Government schools have the largest share of enrolments (65%), followed by Catholic schools (20%), and about 15% of students enrol in private schools (Australian Bureau of Statistics, 2021).

Following SP&M, these analyses are of Year 5 (primary school) and Year 9 (secondary school) students. Appropriate prior achievement measures are only available for Years 5 and 9.

Measures

NAPLAN measures student achievement in Years 3, 5, 7, and 9. Multiple-choice and constructed response test items assess student performance in numeracy, reading, spelling, and grammar. A ten-criteria rubric assesses students’ writing (ACARA, 2018, pp. 7–9, 31–32). For each domain students’ scores range from 0 to 1000 across all four Year-levels, with a mean of 500 and a standard deviation of 100. NAPLAN scores are “conditioned” using several predictor variables to increase the precision of the estimates (ACARA, 2018, p. 32). Conditioning has little impact on the overall and subgroup means but reduces the standard deviations and standard errors (von Davier et al., 2009).

The models analysed include students’ socioeconomic status (SES), sex, Indigenous status, language background, school-SES, school sector, and student- and school-level prior achievement.

Student SES is measured by the number of years of parents’ formal education, averaged if data from both parents are available. The measures are centred around the respective calendar year means. This SES measure differs from SP&M’s (2022) SES measure, a composite of parental education and occupation. However, the NAPLAN data on parental occupation is crude, with only four very heterogenous occupational categories, and there is considerable missing data.³ The standardized estimates for student-level parental education estimated in this study are almost identical to SP&M’s (2022) standardized estimates for SES (discussed in “Results” section).

For this study, school-SES is measured by school means for students’ parents’ education.

For the dichotomous categorical variables, female, Aboriginal and Torres Strait Islander (ATSI), Non-English-speaking background (NESB) students, and students attending non-government (Catholic or independent) schools are contrasted (scored one) with male, non-indigenous, English language background students and students attending government schools (scored zero). In the NAPLAN data, it is not possible to distinguish students attending Catholic and independent schools.

Same-domain prior achievement is measured by students’ NAPLAN scores two years earlier, Year 3 for Year 5 and Year 7 for Year 9. School-level prior achievement was constructed by first standardizing, within domains, Year 3 (and Year 7) achievement scores, then averaging the non-missing scores for each student and then calculating mean school prior achievement for each school. In contrast, SP&M measure school prior achievement as a latent variable so that student scores are manifestations of an underlying latent (school-level) construct.

Univariate statistics

Table 1 presents summary statistics for the analysed variables and Table 2 the bivariate correlations.

Table 1 Univariate statistics for study variables

Table 2 Correlations among variables

Table 1 shows that the data comprises about 300,000 Year 5 students and around 280,000 Year 9 students. These are population data, so sampling statistics are not required. About one-quarter of students are from non-English speaking backgrounds, and 6% are Indigenous. About 30% of Year 5 students attend non-government schools and about 40% in Year 9.

Differences between prior achievement and achievement are larger for primary school students than for secondary school students. The average difference between Year 7 and Year 9 student achievement scores is 39 score points, substantially smaller than the average 77 score point difference between Years 3 and 5.

Correlations

Table 2 presents the bivariate correlations among the study variables.⁴ For numeracy, the mean scores for female students are slightly lower than those for male students. For the other four domains, girls exhibit higher mean scores, especially for writing and grammar. NESB students generally exhibit higher mean scores than English-speaking background students, except for reading. The mean scores of ATSI students are consistently and substantially lower than those of non-ATSI students. Students attending non-government schools exhibit higher average scores than students attending government schools, with larger differences at Year 9.

The correlations between parents’ education and achievement range from about 0.3 to 0.4, higher than that obtained from metastudies in which the average correlations of parents’ education and achievement are generally between 0.2 and 0.3 (Harwell et al., 2017, p. 207; Liu et al., 2022, p. 2879; Sirin, 2005, p. 433; White, 1982, p. 470).

Table 2 shows that school-level prior achievement has stronger correlations with achievement than school-SES for all achievement measures. This finding undermines SP&M’s (2022) contention that academic composition only mediates the relationship between school-SES and student achievement. Stronger correlates do not usually mediate the effects of a weaker correlate.

School-SES and academic composition are highly correlated: 0.78 for Year 5 and 0.85 for Year 9. The strong correlations are important to the debate about SEC effects. If academic composition is not controlled for, then SEC effects on student achievement are spurious because they incorporate the effects of academic composition.

The high correlation between school-SES and academic composition does not preclude obtaining estimates for both measures in model 2. There is no indication of problematic high multicollinearity. For example, for Year 5 numeracy the tolerance statistics for these variables are about 0.35, well above the cut-off of 0.10 below which indicates unacceptably high multicollinearity. The variance inflation factors are around 2.8, well below the cut-off of 10. For Year 9, the corresponding multicollinearity statistics are 0.23 and 4.2 (for details on the multicollinearity statistics see Schreiber-Gregor, n. d.).

The most striking feature of Table 2 is the strong same-domain over-time correlations, that is, between achievement and prior achievement (italicized). They range from about 0.60 for writing to 0.87 for secondary school numeracy and spelling, which are as large as some test–retest correlations. Only for numeracy is the prior achievement-achievement correlation appreciably larger in Year 9 (0.87) than in Year 5 (0.79). Large intradomain correlations are routinely found in longitudinal studies of student achievement (Marks, 2022). These sizable correlations indicate that student achievement is highly stable, especially for spelling and secondary school numeracy.

What accounts for the stability of student achievement? An obvious candidate is socioeconomic background. However, the relationship between SES and achievement are too weak for SES to account for the stability of student achievement. Prior achievement has much stronger correlations with achievement. For these NAPLAN data, prior achievement accounts for about 36% of the variance in writing and between 50 and 75% in the other domains.

Behavioural genetics provides the most plausible explanation for the high stability of student achievement. Between 40 and 80% of the variance in NAPLAN scores is attributable to genetics (Grasby et al., 2016). Grasby and Coventry (2016, p. 649) conclude that “stability in performance was primary due to genes”.

Another notable feature of Table 2 is the sizable interdomain correlations. Because they all tap skills relating to English literacy, the intercorrelations for reading, writing, spelling, and grammar should be greater than their correlations with numeracy, which involves quite different skills. However, the correlations of numeracy with the four literacy domains are comparable to the intercorrelations for the literacy domains. The high interdomain correlations can also be accounted for by shared genes. Grasby et. al. (2016, p. 644) conclude that genetics accounts for about three‑quarters of the observed interdomain correlations in NAPLAN.

Student and school effects on student achievement

Multilevel modelling is the most common statistical procedure for isolating school effects. The standard two-level model comprises random effects for schools and fixed effects for student-level and school-level predictors (see Hox, 2010; Snijders & Bosker, 2012). For this study, the coefficients and other parameters were estimated using PROC MIXED in SAS (Singer, 1998). In contrast, SP&M (2022, p. 13) used multilevel path models with student and school components.

Estimates are from three multilevel models. The first model comprises student-level demographic and socioeconomic variables, school-SES (measured by school-level parental education), and same-domain prior achievement, equivalent to SP&M’s (2022, p. 9) model summarized by their equation 1. Model 2 adds school-level prior achievement so that SEC effects do not incorporate school differences in prior achievement. Model 3 removes SEC to assess its importance to the parameter estimates and the models’ explanatory power. In other words, is SEC important in accounting for variation in student achievement?

The equation for the general model is:

$$Ach_{ijt = 2} = b_{0} + b_{1} Sex_{ij} + b_{2} NESB_{ij} + ATSI_{ij} + b_{4} SES_{ij} + b_{5} Sector_{j} + b_{6} SES_{.j} + b_{7} Ach_{ijt = 1} + b_{8} Ach_{.jt = 1} + \delta_{0j} + \varepsilon_{ij} ,$$

where $Ach_{ijt = 2}$ is the achievement score of student i in school j at time 2, $b_{0}$ is the intercept, $b_{1}$ to $b_{4}$ are the coefficients for the respective student-level predictor variables, $b_{5}$ is the effect for attending a non-government school, $b_{7}$ is the coefficient for prior achievement, $b_{6} \;{\text{and}}\;b_{8}$ are the coefficients for school-SES and school-level prior achievement, $\delta_{0j}$ is the school-level residual variance and $\varepsilon_{ij}$ is the student-level residual variance.

Tables 3 and 4 report estimates for the student- and school-level predictors, the percentages of variance accounted for by schools, the percentages of the total variance in student achievement accounted for in each model, and the numbers of students and schools. The tables include the standardized coefficients (β) for parents’ education, SEC, and school-level prior achievement to allow for comparisons between and within models (Lewis-Beck et al., 2004).

Table 3 Student- and school levels on year 5 achievement

Table 4 Student- and school levels on year 9 achievement

Cohen (1988, pp. 79–81) classified correlations, which are equivalent to standardized coefficients in bivariate regression, as small (≈ 0.1), medium (≈ 0.3), and large (≈ 0.5). In this study, “very large” refers to standardized coefficients greater than 0.7 and “very small” to standardized coefficients around 0.05 or less.

The interclass correlation (ICC) is the between-school variance as a proportion of the total variation in students’ scores (Hox, 2010, p. 15; Snijders & Bosker, 2012, pp. 18, 52). In Tables 2 and 3, the ICCs are the school-level residual variance $\left( {\delta_{0j} } \right)$ as a percentage of the total student-level variance for each domain. The pseudo R square is the square of the correlation between the dependent variable and the predicted values from the model [r(Y, Ŷ)]² converted to percentages. It is a measure of model fit and allows comparison of the predictive power of the models. It is equivalent to the unadjusted R square in Ordinary Least Squares regression. In this context, it encompasses variance accounted for by schools as well as the variance accounted for by the predictor variables.

Results

Student-level effects

Model 1 in Tables 3 and 4 shows that boys outperform girls in numeracy but underperform compared to girls in writing, spelling, and grammar. For reading, there is little difference. The differences in favor of girls tend to be larger in Year 9 than Year 5. In Year 9 writing, girls score, on average, 18 score points higher than boys.

NESB students outperform English-speaking background students in numeracy, writing, and especially spelling but not in reading. Differences in language background are slightly smaller in Year 9 than in Year 5.

Differences between Indigenous (ATSI) and non-Indigenous students are larger than those for the other binary contrasts examined, especially for writing and grammar. Differences in writing by Indigenous status is larger in Year 9 than in Year 5, but not for the other domains.

In Year 5, a one-year increase in parents’ education is associated with a rise of only four score points in numeracy and five score points in reading. The Year 5 standardized coefficients for parents’ education are around 0.10 but lower for spelling (β = 0.05). The effects of parental education are smaller in Year 9 than in Year 5. These estimates indicate that SES is only a weak predictor of student achievement when considering prior achievement.

Prior achievement is a powerful predictor. The estimates are for a one-unit change in the respective prior achievement measure on achievement. Since achievement measures range from zero to 1000, the effects of prior achievement appear small, always less than one (model 2, Table 3). However, the magnitudes of standardized coefficients for prior achievement are large or very large. For Year 5, the standardized coefficients range from 0.50 for writing to 0.81 for spelling. For Year 9 numeracy, the standardized coefficients for prior achievement range from 0.51 for writing to 0.80 and 0.84 for numeracy and spelling. For comparison, the standardized coefficients for parental education are between 0.04 and 0.09.

The coefficients for attending a non-government school on Year 5 achievement hover around zero, and sometimes they are negative. For Year 9, school sector coefficients are positive but small. The small coefficients are not because school sector and school-SES are strongly correlated. The coefficients for attending a non-government school are only slightly larger after removing school-SES from the model. The absence of school sector differences in NAPLAN achievement is consistent with the literature (Larsen et al., 2023).

Table 5 compares the standardized estimates for prior achievement and SES with the standardized effects from SP&M’s (2022) analyses. The two sets of estimates are similar despite different methodological approaches and slightly different measures of SES. Both analyses show small effects of SES and large effects for prior achievement. The magnitudes of the standardized coefficients for prior achievement are 5 to 20 times those for student SES.

Table 5 Comparison of the standardized coefficients for SP&M’s study and the present study

Between school differences

Model 1 indicates that little of the variation in student performance in NAPLAN is between schools. For Year 5, the between-school variation as a percentage of the total variance ranges from 4% (reading) to 7% (numeracy, writing, and grammar). For Year 9, the between-school variation is largest for numeracy at around 8%, which may reflect differences in mathematics teaching between secondary schools. For the literacy domains in Year 9 the between-school variation in is only 3 or 4%. These estimates indicate that schools vary little in average student achievement when considering students’ prior achievement and other variables in model 2.

Unstandardized school-SES effects

The effects of school-SES are small, even without including prior achievement (model 1). For Year 5, a one-standard-deviation increase in SEC, on average, increases student achievement by between 2 (spelling) and 7 score points (writing). For Year 9, school-SES effects are slightly larger, ranging from 3 (spelling) to 12 (writing). The larger coefficients for writing are likely due to its weaker correlation with prior achievement.

SP&M (2022) converted the small estimates from Figure 6 to percentages of achievement growth. They claim that the difference in student achievement growth between low and middle SEC schools is 11% for primary schools and 31% for secondary schools. These are misleadingly high percentages. It is very unusual to convert an estimate to a percentage of average achievement growth. The model analysed already incorporates achievement growth because it includes achievement measures at two time points. The main reason that SEC appears more important in secondary schools than primary schools is because in NAPLAN there is less achievement growth in secondary schools.

Standardized school-SES effects

For Year 5, the standardized coefficients for school-SES are small or very small, ranging from 0.02 to 0.07. For Year 9, the respective standardized coefficients are also small or very small, ranging from 0.03 to 0.12.

SP&M’s (2022, p. 12) standardized school-SES (SEC) estimates can be approximated from Figure 3 which summarizes their simulation analyses. The estimates are similar to the estimates from the present study. The first panel of Figure 3 includes SEC estimates (black dots) with no additional measurement error for Year 5. The estimates are not labeled by their NAPLAN domain nor discussed in the text, but it is possible to match coefficients to domains by their relative magnitudes. The smallest Year 5 SEC estimate is about 0.025, close to the present study’s estimate of 0.02 for spelling. The largest estimate is 0.12, probably for writing, larger than this study’s Year 5 estimate of 0.06. The estimates for numeracy, reading, and grammar are between 0.06 and 0.08, corresponding to estimates between 0.05 and 0.07 in the current study.

The standardized estimates for Year 9 are in the third panel of Figure 3. The smallest estimate (0.035) corresponds to the present study’s estimate of 0.03 for spelling. The largest estimate is about 0.14, corresponding to the current study’s estimate of 0.12 for writing. SP&M’s (2022, p. 12) three other SEC estimates for Year 9 are about 0.08 and 0.10, not too different from the school-SES estimates estimated in this study of 0.07, for numeracy and reading and 0.08 for grammar.

So, SP&M (2022, p. 12) standardized estimates for school-SES effects discerned from Figure 3 are small and similar to the estimates in this study. They are not around 0.32 for Year 5 and 0.48 for Year 9.

Academic composition

Model 2 provides the most realistic parameter estimates since it includes academic composition (i.e., school-level prior achievement). With the addition school-level prior achievement, the school-SES coefficients are trivial. The standardized effects are all close to zero indicating that school-SES is of no substantive importance. In contrast, the effects of school-level prior achievement are non-trivial. A one-standard-deviation difference in school-level prior achievement increases achievement scores by between 5 score points (spelling) and 18 school points (writing) in Year 5 and between 6 and 20 score points for Year 9. The standardized effects for school-level prior achievement range from 0.04 to 0.15, so they are small or very small.

Model 3 removes school-SES from the analysis. The effects of school-level prior achievement tend to be slightly larger, although the respective standardized coefficients remain small: 10.5 (β = 0.08) for numeracy, 10.4 (β = 0.07) for reading, 17.2 (β = 0.14) for writing, 4.7 (β = 0.03) for spelling and 14.7 (β = 0.08) for grammar. So, there are small effects in the expected direction for academic composition. The removal of SEC did not substantially change the coefficients for the predictor variables, nor did its removal reduce model fit, reiterating the conclusion from Model 2 that school-SES has little or no impact on student achievement.

Discussion

SP&M’s (2022, p. 17) proposed reforms to Australian education are based on purportedly large school-SES effects. However, school-SES effects are effectively zero in correctly specified models that include student- and school-level prior achievement. Therefore, their policy recommendations are not credible. Any assessment of the importance of school-SES or any other school-level factor must consider student- and school-level prior achievement.

SP&M (2022, p. 16) contend that parental choice of government-subsidized private schools is a “key driver” of school-SES effects in Australia, although they were unable to distinguish private school students from other non-government school students. Private schools enrol only about 15% of students, so cannot be the “key driver” for school-SES effects. Even if the definition of ‘private schools’ is extended to include Catholic schools, most students attend government schools. SP&M (2022) provide no evidence that SEC effects are absent from government schools. The key driver for the school-SES effects estimated by SP&M, and in model 1 in the current study, is school-level prior achievement. SEC and school-level prior achievement are highly correlated and SEC effects disappear with the addition of school-level prior achievement.

Australia’s research and policymaking communities are wedded to the sociodemographic-schools paradigm, which assumes that student outcomes differ because of SES, demographic factors, and schools. This paradigm dominates educational research and policy in Australia, and in other OECD countries (OECD, 2019; Thomson, 2018). SP&M’s (2022) paper belongs to the sociodemographic-schools paradigm, focusing on the supposed importance of school-SES.

This sociodemographic-schools paradigm cannot account for the empirical realities of student achievement: the large interdomain correlations, the even larger same-domain over-time correlations, the weak contemporaneous effects of SES, the very much larger effects of prior achievement, the small proportions of variation in student performance attributable to schools, the negligible impact of school-SES and the small effects of academic composition. However, the educational research and policymaking communities, almost without exception, ignore these empirical realities and continue to focus on SES.