The substantiveness of socioeconomic school compositional effects in Australia: measurement error and the relationship with academic composition

Abstract

This study examines the effect of school socioeconomic composition on student achievement growth in Australian schooling, and its relationship with academic composition utilising the National Assessment Program—Literacy and Numeracy (NAPLAN) dataset. Previous research has found that school composition predicts a range of schooling outcomes. A critique of school compositional research has been that measurement error may have biased findings of compositional effects. Prior studies have found that socioeconomic compositional effect sizes are small when models include academic composition. The relationship between socioeconomic and academic compositions has yet to be fully determined. Multi-level regressions and structural equation models were compared to estimate the degree of bias in socioeconomic compositional effects due to measurement error. Multi-level path models were used to test if academic composition mediated the relationship between socioeconomic composition and achievement growth. The results showed that measurement error did not bias compositional effects in the dataset, and academic composition mediates the relationship between socioeconomic composition and achievement growth. We argue that school value-add research should include academic composition to account for contextual effects. The socioeconomic compositional effect is of practical significance to policy makers and educational researchers due to its relative size compared to average student achievement growth. Potential reforms include ensuring public subsidies to private schools in Australia do not increase school segregation and the amelioration of the effects of residential segregation through school funding reforms.

Introduction

The predictive relationship between student socioeconomic status (SES) and academic outcomes has been well established in educational research. Sirin’s (2005) meta-analysis of the relationship between SES and academic achievement found a mean effect size of r = 0.29. National and international testing programs have consistently found SES achievement gaps. Australia’s National Assessment Program—Literacy and Numeracy (NAPLAN) 2018 administration found that in all assessed academic domains across each year group, higher levels of parental education and occupation were associated with higher levels of academic achievement (ACARA, 2018). The OECD’s Programme for International Student Assessment (PISA) most recent administration found that on average across OECD countries student SES accounted for 12% of the variation in reading achievement (OECD, 2019).

Less well established in the literature are the effects of school socioeconomic composition (SEC) and its relationship with other contextual factors. SEC effects are the predictive relationships between the aggregate socioeconomic status of a student body and individual-level student outcomes (Willms, 2010). Student bodies may consist of classes, cohorts, age groups or schools. SEC effects are conceptualised as the difference in outcomes of students who have the same individual SES associated with attending schools with differing socioeconomic compositions (Raudenbush & Bryk, 2002). That is, school socioeconomic composition has an effect distinct from individual student SES.

Prior research has found that socioeconomic composition has a substantial predictive relationship with academic achievement (Perry & McConney, 2010a, 2010b, 2013; McConney & Perry, 2010a, 2010b; Chesters & Daly, 2015, 2017; Lamb & Fullarton, 2002) and achievement growth (Rumberger & Palardy, 2005), but many gaps remain in the literature. No research has established how SEC interrelates with academic composition, or average school-level prior achievement, in predicting achievement growth. Research findings into the linearity or consistency in the strength of the SEC effect have diverged (Benito et al., 2014; Chiu & Khoo, 2005; Rumberger & Palardy, 2005). Methodological questions also remain as to whether measurement error and selection effects have biased past findings of SEC effects or if SEC effects are of a size to warrant policy interest (Armor et al., 2018; Lauen & Gaddis, 2013; Marks, 2015).

The present study aims to address some of these gaps in our understanding of the socioeconomic compositional effect with the following research questions within the Australian context:

• (1) How substantial is the influence of measurement error on socioeconomic school compositional effects?

• (2) Does academic composition mediate the relationship between socioeconomic composition and achievement growth?

• (3) Is the SEC effect of practical significance?

These research questions aim to locate the potential role of the socioeconomic compositional effect in predicting the equity of the Australian school system. Addressing the issues of measurement error and selection effects indicates whether school effectiveness research should be informed by compositional effects, including the development of value-added measures (Marks, 2021). Testing the nature of the relationship between academic and socioeconomic compositional effects may improve the conceptualisation of why SEC predicts schooling outcomes. Exploring the substantiveness of socioeconomic composition on achievement growth, suggests whether it should be a focus of school policy reforms. Australia offers insights into the role of the socioeconomic compositional effect as a comprehensive or non-tracked schooling system that has engaged in significant market-based reforms (Lubienski et al., 2021), namely substantial increases in public funding to private schools, alongside having one of the most socioeconomically segregated schooling systems within the OECD (2018). As such, it offers potential insights into the segregational effects of market-based reforms on schooling outcomes free from the segregational effects of between-school curriculum tracking.

Background

School compositional effects came to prominence in educational literature following the seminal Coleman Report (Coleman et al., 1966). David Coleman and colleagues investigated the quality of US public education afforded to minority ethnic groups compared to European–American students through relating school characteristics to academic achievement (Coleman et al., 1966). They found that school composition was the largest school-level effect on academic performance. These effects have consistently been found over time across international schooling systems predicting a diverse range of educational outcomes.

International large-scale studies have consistently found that socioeconomic composition is a substantial predictor of student academic achievement. Each cycle of PISA, for example, has found that school-level socioeconomic factors substantively predict academic achievement separate from individual-level SES in most participating countries (OECD, 2003, 2004, 2007, 2010, 2013, 2016, 2019). Secondary analyses of PISA datasets have shown an inverse correlation between the degree of socioeconomic school segregation and national levels of academic achievement (Chiu & Khoo, 2005; Willms, 2010) and factors that may mediate the relationship between composition and achievement (Liu et al., 2015).

Socioeconomic composition has been shown to predict a range of schooling outcomes. Both academic achievement (Lamb & Fullarton, 2002) and achievement growth (Rumberger & Palardy, 2005) across a broad range of subject domains are related to SEC. Academic achievement is predicted by SEC at both primary (Sofroniou et al., 2004) and secondary school levels (Chesters & Daly, 2017). Other school outcomes that have been associated with SEC have been high school graduation and college attendance (Chesters, 2019; Palardy, 2013) and college choice (Palardy, 2015). Boonen et al. (2014), however, found no statistically significant main effects of a range of compositional effects on achievement growth in early primary school.

Reported effect sizes for socioeconomic composition vary widely in the literature. Van Ewijk and Sleegers’ (2010) meta-analysis of cross-sectional studies found effect sizes ranged 0.03 to 0.59 standard deviations with an average of 0.32. The 2015 PISA administration reported that on average, SEC accounted for 62.6% of variation in academic achievement due to schools (OECD, 2016). Longitudinal studies have tended to find smaller effect sizes for SEC as they control for selection effects where prior achievement is positively correlated with SEC. That is, students with initial higher achievement tend to self-select into schools with higher SEC in systems where such selection is possible. Rumberger and Palardy’s (2005) analysis of the National Education Longitudinal Survey of 1988 found SEC effect sizes ranged 0.05 to 0.21 standard deviations depending on the academic domain. Marks’ (2015) study with primary school samples found smaller SEC effects ranging from 0.00 to 0.05 standard deviations, arguing they were of insignificant size to warrant policy reforms.

The relationship with academic composition

Academic composition has been another school-level factor in school effectiveness research. Its conceptualisation, or what it represents, has not been well-defined by the literature, but it is usually measured as average or school-level prior achievement (Marks, 2021). It has been argued that academic composition predicts student-level achievement outcomes due to influencing teacher expectations of student abilities and students’ expectations of themselves (Scheerens et al., 2001). Best practice in modelling school compositional effects includes joint modelling of academic and socioeconomic composition and the relationship between them (Thrupp et al., 2002).

Prior research has found a strong correlation between academic and socioeconomic composition, but the nature of the relationship remains an open question in the literature (Harker & Tymms, 2004; Thrupp et al., 2002). Models that have included academic and socioeconomic composition as simultaneous predictors of academic achievement or growth have found small-to-zero effect sizes for the SEC effect (Dumay & Dupriez, 2008; Lauder et al., 2010; Marks, 2015). This suggests that academic context may mediate socioeconomic context. That is, SEC captures the sorting of students into schools according to prior achievement due to socioeconomic segregation. This sorting process varies the academic composition of schools, which in turn varies academic outcomes.

Compositional effects are a distal explanation of school effectiveness, providing a broad explanation of why school contexts predict school effectiveness. Jencks and Mayer’s (1990) theoretical analysis outlined three types of models that explained how children and young people’s academic performance might be influenced by the socioeconomic composition of their neighbourhoods and schools. Collective socialisation models posit that non-paternal adults from a child’s neighbourhood influence the attitudes, beliefs and behaviours of young people (Jencks & Mayer, 1990). Socioeconomically advantaged adults provide a social milieu in which young people learn to value academic performance and the social skills needed for success at school (Jencks & Mayer, 1990; Sui-Chu & Willms, 1996). Epidemic or peer effects models posit that peers influence each other’s beliefs, values, attitudes and behaviours (Jencks & Mayer, 1990). Exposure to middle class peers who value academic performance and model the skills for its success may induce similar values and skills in their peers in a form of social contagion (Bankston & Caldas, 1996; Jencks & Mayer, 1990). Institutional models posit that the key social institutions within a neighbourhood, such as schools, differ in their quality of service depending on the socioeconomic composition of a neighbourhood (Jencks & Mayer, 1990). When researching schools, institutional models look to differences in the experiences young people have of social institutions in differing socioeconomic contexts (Jencks & Mayer, 1990). Institutional and peer effects models have dominated mediational models of school compositional effects research (Liu et al., 2015; Palardy, 2008, 2013; Rumberger & Palardy, 2005; Willms, 2010).

Appropriately specifying models of socioeconomic composition

Limitations of school compositional research have been the potential for selection effects (Lauen & Gaddis, 2013) and measurement error (Marks, 2015) to upwardly bias compositional effects. Selection bias occurs when unmeasured systematic differences between groups confound the relationship between an independent variable and a dependent variable. For example, families with high educational goals for their children may seek to enrol them in schools with good reputations. Due to demand constraints, it is more likely such schools will have above average socioeconomic composition, such as public schools in expensive suburbs or high-fee private schools. Thus, SEC may act as a proxy for unmeasured factors that explain higher achievement in higher SEC schools. Controlling for prior achievement may partially address selection effects by accounting for the clustering of higher ability, or more motivated, students into higher SEC schools (Hallberg et al., 2018). While longitudinal studies account for biases in prior achievement, achievement growth may also be biased by selection effects as growth rates may differ according to selection effects. One method to mitigate selection bias is to include variables that may capture the probability of school selection (Palardy, 2013; Rangvid, 2007). Controlling for school sector in Australian samples likely captures much of the potential selection effect because of the high degree of school choice in Australia (ABS, 2017; Rowe, 2020). Family selection of private schools is an indicator of family academic aspiration in Australia (Warren, 2016).

A critique of school compositional effects research has been that measurement error in SES indicators may lead to “phantom” or spurious socioeconomic compositional effects in multilevel models. Two similar mechanisms have been posited for phantom compositional effects. Firstly, measurement error may deflate individual-level effect sizes while minimally influencing group-level effects (Gorard, 2006; Harker & Tymms, 2004). As compositional effects are the difference between student and aggregated effects (Raudenbush & Bryk, 2002, pp. 139–141), the compositional effect is inflated. Secondly, it has been found that measurement error deflates individual-level effects alongside inflating group-level effects in common modelling scenarios (Pokropek, 2015).

Measurement models derived from factor analytic approaches are an effective method to address measurement errors (Fan, 2003; Muthén, 1991). Factor analysis derives latent variables from the common variance of indicators, partitioning out error and unique variance. The doubly-latent structural equation modelling (SEM) approach to modelling compositional effects (Marsh et al., 2009) affords a means by which compositional effects research can be conducted free from potential measurement error biases. Simulation (Pokropek, 2015) and applied (Televantou et al., 2015) studies have shown the doubly-latent method reliably estimates compositional effects free from biases due to individual-level measurement errors.

A theoretical difficulty with applying factor analytic approaches to the development of SES composite measures arises from the assumed direction of causality between composite measures of SES and its indicators. Factor analysis assumes that latent variables cause variation in the associated reflective indicator variables (Bollen & Bauldry, 2011). That would mean SES causes changes in its underlying indicators such as parental education and occupation. This is inconsistent with how SES is operationalised in the majority of educational research as a convenient summary of measures associated with SES (Avvisati, 2020).

Principal components analysis (PCA) is a common method for constructing composite measures of SES (NCES, 2012, pp. 22–24; OECD, 2017, pp. 339–340). Unlike factor analysis that derives latent factors from the shared variance of indicators, PCA derives components from the total variance of underlying variables (Dunteman, 1989, p. 55). As such, composite measures derived from PCA capture the measurement error of the underlying variables, thus potentially biasing findings when used as predictors in models.

The standard factor analytic practice of testing model fit is not applicable to SES composite measures. As a convenient summary of SES indicators, it is not expected that SES composite measures exhibit a unitary factor structure. Instead, such measures are developed to simplify the modelling and reporting of SES effects (NCES, 2012, p. 22) through capturing multiple independent social processes that determine a person’s place in a social hierarchy. Thus, poor model fit would indicate SES composite measures represent an intended set of independent constructs.

The non-unitary assumption of SES composite measures is partly due to the lack of agreement and progress in conceptualising SES (Marks & O’Connell, 2021). Researchers and policy makers are often dependent on pre-existing indicators of SES that are conveniently accessible from school administrative processes, including the current paper. Whilst outside of the scope of this paper, much more work is required to develop both the theory and the measurement of socioeconomic status.

This study applies a novel approach to address the potential biases associated with measurement errors in SES measures. It evaluates the substantiveness of the influence of measurement error on SES and SEC effects in models applied to the same sample. Simulations of increasing measurement error are compared across PCA and the doubly-latent structural equation approaches. If the SES and SEC effect sizes are comparable between the two approaches at low levels of measurement error, it can be assumed that measurement error does not bias models using PCA scores for composite measures of SES. Therefore, such models can be applied to research questions.

Method

Participants

The research questions were examined through a secondary analysis of the 2017 student-level de-identified NAPLAN dataset. NAPLAN is Australia’s annual population assessment of academic achievement and growth in reading, writing, spelling, grammar and numeracy for students in grades three (313,807 students from 7820 schools), five (311,412 students from 7809 schools), seven (288,946 students from 3422 schools) and nine (281,280 students from 3064 schools). Being a population assessment, it is representative of the diversity of school contexts in Australia.

The Australian commonwealth is a federation of six states and two territories. The funding and administration of compulsory schooling is shared across federal and state levels of government. Australia’s schooling system consists of primary, kindergarten to grade six, and secondary, grade seven to grade twelve, school levels. It is a comprehensive system with no between-school tracking at any age. The majority of students attend state-funded public schools administered by state government education departments. Thirty-five percent of students attend private schools which are administered by independent bodies such as Catholic dioceses, individual churches or parent boards. A national curriculum exists with variations of implementation across each state. Within each state, the same curriculum and teaching standards exist across public and private schools. Public schools do not charge compulsory fees, and offer open enrolment based on catchment areas, except for a very small subset of selective public schools whose enrolments are based on academic test scores. Private schools determine their own enrolment and exclusion policies and charge fees. As such, the majority of students from low and middle-income families attend public schools whereas private schools tend to limit their student bodies to those from middle-to-high income families.

We used 2017 grade five (and the same cohort’s grade three scores from 2015), and 2017 grade nine (and the same cohort’s grade seven scores from 2015) scores in our analysis. In this way, we were able to examine both socioeconomic and academic compositional effects in relation to achievement growth over 2 years of primary and secondary school levels. Students who changed schools between the two measurement occasions were excluded from the analysis to avoid confounding SEC across differing schools. NAPLAN is administered by the Australian Curriculum, Assessment and Reporting Authority (ACARA). The NAPLAN dataset allows for the examination of socioeconomic and academic compositional effects on achievement growth in primary and secondary schools. As a population dataset, NAPLAN also enables the use of descriptive statistical methods to make generalisable comparisons among differing demographic groups.

Measures

The dependent variables were reading, writing, spelling, grammar and punctuation, and numeracy in the NAPLAN dataset. NAPLAN tests are designed to “broadly reflect aspects of literacy and numeracy within the curriculum in all jurisdictions” (ACARA, 2018, p. iv). The knowledge and skills assessed are drawn from the Australian Curriculum: English and Mathematics and the literacy and numeracy general capabilities of the Australian curriculum (ACARA, 2017). The Australian curriculum defines literacy as “the capacity to interpret and use language features, forms, conventions and text structures in imaginative, informative, and persuasive texts” and numeracy as “the knowledge and skills to use mathematics confidently across all learning areas at school and in their lives more broadly” (ACARA, 2017, pp. 6–7).

SES was operationalised as PCA composite scores for each research question. To evaluate potential measurement error in research question one, SES was also operationalised through SEM. The SES composite was derived from four indicator variables of parental self-reports of occupation and education which were extracted from school enrolment forms at the time of enrolment in Kindergarten in primary school and grade seven in secondary school. A time gap of up to 5 years in primary school and 2 years in secondary school exists between enrolment data collection and the NAPLAN assessments. Very few parents are likely to change their highest years of schooling or occupation within those time periods, and the occupational groups are broad, thus little measurement error would be introduced by time lags between SES indicator data collection and academic assessment.

The raw measures were ordinal scales of mothers’ and fathers’ education, and occupation. Table 1 shows how we mapped the ordinal measures of SES onto interval scales. Highest education was re-coded into years of schooling. Occupation was re-coded onto the average Australian Socioeconomic Index 2006 (AUSEI06) for the occupational category. AUSEI06 is an interval occupational status scale for the occupational classification categories of the Australian Bureau of Statistics (ABS) (McMillan et al., 2009). Each NAPLAN occupational category consisted of multiple ABS occupational categories (ACARA, 2019). The average AUSEI06 score for each of ABS occupational categories was assigned to the four NAPLAN occupational codes. In cases where parents reported as being outside of the workforce, AUSEI06 scores were imputed from parental education (McMillan et al., 2009, p. 132).

Table 1 Mapping of ordinal NAPLAN SES measures onto interval scales

SEC was operationalised as school-average SES. Academic composition was operationalised as a school-level latent factor indicated by school average scores in all five NAPLAN academic domains of prior achievement in grades three or seven (see Fig. 1). Average, or manifest aggregate, scores were utilised for compositional variables for research questions 2 and 3 as NAPLAN is a population dataset and there is thus no need to adjust for aggregation error through latent aggregation (Lüdtke et al., 2008).

Fig. 1

Multilevel path model of SEC mediated by academic composition

Potential selection effects were addressed with prior achievement scores and control variables for sex, indigeneity, language background other than English, and school type (public or private), which is a key mechanism of school choice in Australia (Rowe, 2020). Descriptive statistics for all variables are in the Additional file 1: Tables S1 and S2.

Procedure

Multilevel residualised-change regressions and SEMs were compared to answer research question 1. Residualised-change models are two-occasion growth models where the prior score on a dependent variable is included as a covariate of the dependent variable (Gollwitzer et al., 2014). When academic achievement is the dependent variable, the resulting coefficients of the predictor variables provide estimates of their influence on achievement growth. Equation (1) represents the residualised-change model this study used in research question 1 with PCA scores.

$${Y}_{i2j2}={\beta }_{0}{+{\beta }_{1}{Y}_{i1j1}+\beta }_{2}{X}_{ij}+{{\beta }_{3}\overline{X}.}_{j}+{\delta }_{0j}+{\varepsilon }_{ij}$$

(1)

For student i in school j, ${Y}_{i2j2}$ was academic achievement, ${\beta }_{0}$ was the intercept, ${\beta }_{1}$ was the coefficient for prior achievement, ${\beta }_{2}$ was the coefficient of student SES, ${\beta }_{3}$ was the coefficient of school-average SES, ${\delta }_{0j}$ was the school-level residual variance and ${\varepsilon }_{ij}$ was the student-level residual variance. Academic growth was modelled over 2 years of secondary school, grades seven to nine, and 2 years of primary school, grades three to five.

Multilevel residualised-change SEMs were constructed to answer research question 1. Such models had the same structural relations as Eq. (1) but modelled SES and SEC as latent factors and aggregated SEC through latent aggregation (Marsh et al., 2009). Equation (2) represents the residualised-change SEM this study used in research question 1. In this case ${U}_{ij}$ is the latent variable of SES and ${U}_{j}$ is the latent variable of SEC derived from latent aggregation.

$${Y}_{i2j2}={\beta }_{0}{+{\beta }_{1}{Y}_{i1j1}+\beta }_{2}{U}_{ij}+{{\beta }_{3}U}_{j}+{\delta }_{0j}+{\varepsilon }_{ij}$$

(2)

The comparison of the substantiveness of the influence of measurement error on SEC effects in PCA score and SEM models was performed by adding increasing levels of random variation to the indicators of SES. The proportion of additional error in each indicator ranged from 0 to 90%, increasing in increments of 10%. If the comparison revealed little-to-no differences between the effect sizes of SES and SEC of PCA score and SEMs at 0% introduced error, we argue it is appropriate to use PCA scores in multilevel models of SEC effects. In such a scenario we moved on to the rest of the research questions.

The mediation of SEC by academic composition was tested through a multilevel path analysis as shown in Fig. 1, to test research question 2. The path analysis was an extension of Eq. (1) with the addition of academic composition mediating the SEC effect. Prior achievement at the student-level was the same domain of achievement as the dependent variable. Academic composition was measured as a latent construct to capture the error-free shared variance of all school-average prior achievement scores. Such a factor represents the academic composition of each grade five or nine cohort within each school.

The substantiveness of the SEC effect was assessed by comparing the unstandardised coefficients of the indirect path of SEC via academic composition to population average achievement growth over 2 years in the same academic domain. This effect size measure enables policy makers and the broad educational community who may not have a background in educational statistics to readily judge the practical importance of school compositional effects. A one unit change, equivalent to one standard deviation, in SEC was used as the basis of the SEC effect as it represents the difference between low and middle, or middle and high, SEC schools. That is, what would be expected with a substantially different degree of school segregation in the population.

Statistical modelling was performed with MPlus. Missing data were handled with the full information maximum likelihood (FIML) method with robust standard errors. It was assumed data were missing at random (MAR) as NAPLAN is a compulsory assessment of all Australian students in grades 3, 5, 7 and 9, excluding a small set of students with intellectual disabilities or less than 1 year’s exposure to the English language. Missingness due to SES can be handled using FIML modelling under the MAR assumption as SES is included in the model. Students missing data on the dependent variable were due to either being absent or withdrawn. Absent students were those who do not participate in a test due to non-attendance at school on the test date and catch-up periods. Causes of absences likely include extended student illness, family travel or movement, and chronic absenteeism. These factors are likely random except for chronic absenteeism, which likely covaries with SES or Indigenous status, both of which are modelled. Withdrawn students are those whose parents apply to withdraw their child from the test due to religious or philosophical reasons, representing 2.3% of grade 5 students and 2.0% of grade 9 students. This low level of missingness at the student level is safe to ignore in a large population-level dataset where the focus of analysis is on group-level relationships. Sample sizes and missing data rates are in Additional file 1: Tables S1 and S2.

FIML has advantages over listwise deletion in that it produces unbiased parameter estimates if data are MAR and has greater power (Enders, 2010, p. 87). An additional advantage of the FIML estimator implemented in MPlus is that it calculates standard errors that are robust to deviations from normality (Muthén & Muthén, 2017, p. 668). PCA scores were calculated with the ‘psych’ package (Revelle, 2021) in R.

Results

We found that measurement error does not appear to inflate SEC effects in the NAPLAN sample, the SEC effect was mediated by academic composition, and the SEC effect is of a size as to warrant further research and policy responses.

Figure 2 graphs the grade 9 2017 NAPLAN dataset. It was observed that both SES and SEC predicted the likelihood of students’ achievement of national minimum benchmarks across all academic domains. For each student SES category, SEC demonstrated a positive relationship with academic achievement. Low SES students in high SEC schools were 2.05 times more likely to achieve minimum benchmarks than low SES students in low SEC schools. The same relationship for middle and high SES students were 1.83 and 1.30 respectively. This indicates that low SES students may be more sensitive to the effects of SEC than high SES students. Non-linearity is also apparent in Fig. 2. The predictive strength of SEC increases as SEC increases with low SES students, whereas it weakens with high SES students. It should be noted that Fig. 2 is the raw descriptive statistics of academic achievement, thus are likely biased by selection effects.

Fig. 2

Proportion of low, middle and high SES students that achieved minimum benchmarks in low, middle and high SEC schools. Low was defined as 1 SD below average, high as 1 SD above average

Figure 3 compares the influence of increasing measurement error in SES indicators in PCA and SEMs in grades 5 and 9. Results are reported as standardised coefficients based on total model variance. In PCA models the lines of best fit tended to show that the inflation of the SEC effect peaked when the proportion of random variance in SES indicators was 70%. Indicator error variance greater than 70% would be very unlikely in educational research. As such, we will use this as a comparative point with 0% random variance for the two modelling methods. At 70% error variance, on average the SEC effect was inflated by 37% with PCA scores and − 8% with SEM. Respective average SES deflation effects were 63% and − 1%. This is consistent with prior research showing that SEM exhibits much lower bias from measurement error in school compositional effects models (Pokropek, 2015; Televantou et al., 2015). With our sample, SEC effects were slightly downwardly biased by measurement error in SEM models at 70% error variance.

Fig. 3

Simulation results of increasing measurement error in multilevel PCA score and SEM. Dots are individual coefficients for each modelled academic domain and lines are best fit of modelled academic domains

The average absolute difference between SES and SEC standardised coefficients in PCA and SEMs at 0% error was 0.002 SD. It can also be observed in Fig. 3 that the curve of the lines of best fit of PCA coefficients levels off as it approaches 0% measurement error. Thus, in the NAPLAN sample, measurement error does not appear to bias coefficients of SES and SEC in multilevel models. As such, we utilise PCA scores in the rest of our modelling.

Figures 4 and 5 show that academic composition mediated the relationship between SEC and achievement growth across all domains in grades five and nine. All indirect paths from SEC to achievement growth were statistically significant with standardised coefficients averaging 0.32 SD at grade five and 0.48 SD at grade nine.

Fig. 4

Subset of grade 5 path analyses results of SEC effect mediated by academic composition. Note. All p < 0.001 unless *p < 0.05 or **p < 0.01 or^NS (not significant)

Fig. 5

Subset of grade 9 path analyses results of SEC effect mediated by academic composition. Note. All p < 0.001 unless *p < 0.05 or **p < 0.01 or^NS (not significant)

Tables 2, 3, 4 and 5 reports the results of the paths not in Figs. 4 and 5 and overall model fit indices.

Table 2 Grade five standardised regression coefficients of control variables

Table 3 Grade five model fit indices

Table 4 Grade nine standardised regression coefficients of control variables

Table 5 Grade nine model fit indices

Figure 6 presents the unstandardised coefficients (indirect effects) of SEC and average achievement growth across each academic domain. It compares the effect of a 1 SD change in SEC compared to average achievement growth. It shows that SEC accounted for a larger proportion of achievement growth in secondary than primary schools. This was due to average achievement growth being larger in primary schools and the SEC effect tending to be larger in secondary schools. On average, SEC effects were equivalent to 11% of average achievement growth in primary schools and 31% of average achievement growth in secondary schools.

Fig. 6

Unstandardised average achievement growth and SEC effect sizes (indirect path)

The SEC effect may be larger in secondary schools due to Australian secondary schools having a greater degree of socioeconomic segregation than primary schools. In the 2017 NAPLAN dataset, 31% of grade five students attended private schools whereas 43% of grade nine students attended private schools, consistent with the historical pattern of Australian parents being more likely to exercise school choice at the secondary than primary school levels. The difference between the average scores of SEC of public and private school students was 42% larger in grade nine than it was in grade five. That is, the systemic socioeconomic differences between public and private schools in Australia are greater in secondary schools than primary schools.

Discussion

Our simulations of compositional effects models shows that measurement error does not bias the coefficients of SES and SEC in multilevel models in the NAPLAN dataset. This is potentially the case in many other international large-scale assessment datasets as well, suggesting that prior critiques of bias in school compositional research may have been misplaced. While increasing levels of measurement error does bias multilevel models of compositional effects, only a comparison of standardised coefficient sizes between methods that do, and do not, account for measurement error can determine the level of potential bias in compositional effects. Our method of comparing SEMs with multilevel models utilising measures derived from PCA suggests a way for researchers and the administrators of large-scale assessments to test for the degree of measurement error in composite SES variables. Comparing such models indicates the degree to which components scores may differ from true scores.

Our finding that academic composition mediates the SEC effect provides some clarity to the relationship between the two compositional effects in schooling systems highly influenced by parental choice policies. One explanation may be that SEC represents the selection of students of differing prior academic histories into differing schools. That is, socioeconomically advantaged students who enjoy greater academic opportunities tend to self-select into schools with other students of the same background. The composition of such schools may be more conducive to learning due to teachers having higher expectations of student achievement, and/or due to peer effects where students develop higher academic expectations of themselves. Another explanation may be that academic composition captures a range of compositional effects, including SEC, which are highly correlated, yet offer at least partial explanations for school contextual effects. Thus, more research is needed to determine the relationship among a range of compositional effects and why they may jointly predict schooling outcomes. Additionally, in schooling systems where segregation is driven more by curriculum tracking, such as in many European countries, the relationship between academic composition and SEC may differ.

If academic composition mediates the SEC effect due to the sorting of students by differing academic histories, then the academic compositional effect is likely a stronger and more reliable explanation for school effects in the Australian context due to it being more proximal to school-level effects than the social contexts of students in our models. At the same time, the role of SEC in predicting academic segregation cannot be ignored as it indicates the policy and social causes of academic segregation. School value-add research that seeks to identify explanations for teacher, school and systemic effects should control for selection and compositional effects by including measurement of academic composition. This would mitigate the confounding of school value-add with compositional effects over which many schools have little influence.

Parental choice of government subsidised private schools is a key driver of the SEC effect in Australia and internationally (Alegre & Ferrer, 2010; Sciffer et al., 2022). Australia has the fourth largest proportion of private school enrolments within the OECD (OECD, 2020), while the capacity of Australian private schools to exclude and expel students with no parliamentary oversight is associated with one of the most segregated schooling systems within the OECD (OECD, 2018). Recent Australian research showing that private schools add no value to the trajectories of student learning (Larsen et al., 2022) indicates that the taxation burden of private schools has added no value to overall student achievement while at the same time increasing social inequality. While calls have been made to require private schools to be socially representative of the community (Bonnor et al., 2021; Greenwell & Bonnor, 2022), a more politically feasible approach may be to require all schools to be academically representative of the communities from which they draw students. That is, publicly funded private schools could be required to enrol a diverse academic mix of students and be barred from expelling students based on academic achievement.

Beyond school choice polices, spatial segregation also contributes to socioeconomic school segregation (Smith et al., 2018). The residential clustering of families according to SES has multiple causes, many beyond the influence of education policies. This suggests that amelioratory education policies are also needed to address school compositional effects. For example, funding reforms may partially address the deleterious effects of concentrations of disadvantage (Lafortune et al., 2018). Adjustments to tertiary entrance student rankings based on school demographic factors may improve the enrolment representativeness of elite universities.

The socioeconomic compositional effect is of substantive practical significance in Australia. On average across all assessed academic domains, the difference in student achievement growth between low and middle SEC schools predicted by socioeconomic composition, as mediated by academic composition, is equivalent to 11% of average achievement growth in primary schools and 31% of achievement growth in secondary schools. Assuming a constant rate of learning, this is equivalent to almost one term of learning in grades 3–5 and over two terms of learning in grades 7–9. School-level reforms that ignore socioeconomic composition are unlikely to succeed, especially given that research (Palardy, 2008) has found differential effectiveness of school practices according to SEC.

A limitation of our study was its sole consideration of socioeconomic and academic compositional effects. Other compositional effects that have been found include intellectual ability (Opdenakker & Damme, 2001), ethnicity (Caldas & Bankston, 1998) and migration status (Peetsma et al., 2006). Some of these compositional effects may overlap or explain some of the variance due to SEC. A second limitation of our study was our measure of SES in the NAPLAN dataset. Both parental occupation and education had limited categories of differentiation compared to other large-scale studies such as PISA, while there was no measure of family wealth. As such, the predictive relationships of SES and SEC with achievement growth may have been underestimated due to limiting the variance SES and excluding an important socioeconomic predictor of educational outcomes (Chesters, 2019; Marks et al., 2006). A third limitation of our study design was that it did not measure class-level effects. The NAPLAN dataset allows the measurement of cohort effects while the PISA dataset allows the measurement of school effects. A three-level model of achievement growth measuring variance at student, classroom and school levels may find that a proportion of the student-level variance found in our residualised-change models may be better explained by classroom composition (Lamb & Fullarton, 2002).

Many avenues of future research would advance our understanding of socioeconomic compositional effects. As mentioned, our understanding of the mediators of SEC and achievement growth is not fully developed (Marks, 2015). Such knowledge may be able to guide interventions aimed at ameliorating SEC effects. Palardy’s (2008, p. 26) model of school effects may provide a framework for future studies of mediators of SEC effects. SEM and path analysis may provide researchers a flexible means to identify mediators of compositional effects (Preacher et al., 2010, 2011).

Conclusion

This study has applied a novel simulation strategy to show that measurement error is minuscule in SES indicators in the NAPLAN dataset and it does not bias coefficients of school composition in multilevel models. This provides confidence in the use of PCA in the development of composite measures of SES and SEC in datasets where indicators are drawn from parental self-reports. The study also found that academic composition mediates the relationship between socioeconomic composition and achievement growth. This may be due to socioeconomic compositional measures representing the selection process of academically advantaged students tending to enrol in schools with similar students, or academic composition capturing a range of correlated compositional effects. In either case, school value-add researchers should consider including academic compositional effects to control for contextual effects over which many schools have little influence. The socioeconomic compositional effect is of practical significance to policy makers and educational researchers as it is of a substantial size compared to average achievement growth. This suggests that reforms to public policies that contribute to socioeconomic school segregation, and the amelioration of the deleterious effects of residential segregation, are needed to improve the equity of the Australian schooling system.