Socioeconomic status and beyond: a multilevel analysis of TIMSS mathematics achievement given student and school context in Turkey

Abstract

Socioeconomic status (SES) is considered a major predictor of student and school achievement. In most cases, SES is not malleable or available for manipulation to improve students’ learning and achievement. Therefore, we explored other student and school-related factors that may be malleable to reduce achievement differences between students and schools in Turkey. We used the TIMSS 2015 fourth-grade mathematics data and analyzed it using hierarchical linear modeling. We found that SES at both student and school levels is a dominant factor related to mathematics achievement and a much stronger predictor at the school level. Early literacy and numeracy activities, preschool education, intrinsic motivation, and engagement in instruction were found to be significantly and positively associated with higher achievement within schools. Similarly, among the school-level variables, we found significant and positive associations between schools’ mathematics scores and instruction quality and school readiness, in the presence of SES. We recommend that preschool education and early literacy and numeracy activities at home should be promoted. Additionally, although the findings of this study indicate possible SES-based school segregation; parents, teachers, and school leaders, as well as students, should work toward a more positive school climate to reduce achievement disparities due to SES.

Introduction

In Turkey, public education is fully financed by the government, including textbooks and educational technologies used in classrooms. Nevertheless, family socioeconomic status (SES) explains a great amount of the variation in achievement differences between students (Oral & McGivney, 2014; Özgürlük et al., 2016; Yıldırım et al., 2016). Furthermore, SES based residential segregation¹ accompanied with an address-based school enrollment system² potentially leads to students with similar family socioeconomic backgrounds gathering in the same school which increases the opportunity and achievement gaps not only between students but also between schools (Owens et al., 2016; Owens, 2018; Ünal et al., 2010).

In this study we investigated several student and school-related factors that may explain achievement differences between students and schools, above and beyond SES. Thus, our purpose was to identify plausibly malleable variables for informing educators, school leaders, researchers, and policymakers to contribute to reducing educational inequities due to the student and school-level SES differences in Turkey.

In this study, mathematics data for Turkish fourth graders from the Trends in Mathematics and Science Study (TIMSS) were used. The first reason for studying fourth-grade data is that early mathematics learning and achievement are essential in later learning and school dropout prevention (Rumberger & Lim, 2008; Siegler et al., 2012). Therefore, it is crucial to understand the educational state of Turkish students at an early stage of education. In addition, Turkey’s participation in international large-scale assessments at the fourth-grade level is relatively recent; therefore, items regarding family SES and school readiness have become available in the TIMSS 2015 assessment for the first time allowing researchers to build more comprehensive models (IEA, 2013, 2017). Finally, another motivation for this study was to investigate the association between school environment and achievement differences for students and schools, beyond SES. However, the effect of schools on student and school achievement above and beyond SES may be masked by other factors in higher grades. For instance, students in higher grades prepare for national exams by receiving additional support such as private tutoring and private classes (Tansel, 20132016). Likewise, among the students who have been admitted to one of the selective high schools, their schools’ academic achievement levels differ based on the type of high school due to the process of selection and placement of students (Berberoğlu & Kalendar, 2005; Oral & McGivney, 2014; Özdemir, ; Özgürlük et al., 2016; Şahin et al., 2012).

Understanding the context: an overview of the education system in Turkey

In this section, a brief summary of the Turkish education system was presented to provide the reader background on its unique characteristics and some justification for the study variables.

Today’s Turkish education system began taking shape in 1923, during the establishment of Turkey as a modern democratic country. It has transformed during the past 100 years, influenced by economic, cultural, sociological, and political dynamics.

The Turkish education system is heavily centralized, a hallmark characteristic (Gerschber, 2005). In such a centralized education system, the Turkey Ministry of National Education (Turkey MoNE) is responsible for maintaining all educational activities across the country (Gerschber, 2005; Turkey Ministry of National Education, 2018). These activities include developing educational curriculum and textbooks, deciding on instructional subjects, specifying instructional hours per week to cover these subjects, and determining the hiring process of teachers. In addition, all schools are required to follow the same academic calendar. Public education is fully financed by the government, and public schools are the most common type of schools in Turkey.³

In the last three decades, there have been two main changes in the education system. In 1997, compulsory education was extended from 5 to 8 years, and the 5 + 3 system became an 8 year uninterrupted primary education program. In 2012, compulsory education was extended to 12 years, formed into three compulsory levels also known as the 4 + 4 + 4 system: primary + lower secondary + upper secondary education (Gürkan et al., 2014; OECD, 2019; Turkey Ministry of National Education, 2018). Pre-primary education is not compulsory (Turkey Ministry of National Education, 2018); however, it is being discussed.

As of 2019, about 16 million students were enrolled in compulsory education, almost evenly distributed in each four-year education level (Turkey Ministry of National Education, 2019, p. 17). According to educational statistics for the 2018–2019 academic year (Turkey Ministry of National Education, 2019, p. 1), the net school enrollment ratio for primary education was 92%, for lower secondary education it was 93%, and for upper secondary education it was 84%. On the other hand, it was 68% for pre-primary education. The school enrollment ratio was similar between girls and boys in Turkey for all educational levels (Turkey Ministry of National Education, 2019, p. 17); and there was no significant difference between Turkish male and female students’ mathematics scores in TIMSS (Mullis, Martin, Foy et al., 2016).

International large-scale assessments

International large-scale assessments in education (ILSA) date back to the 1960s, to the first endeavors of developing a collaboration to gain an understanding of associations between the inputs and outcomes of education systems through cross-national comparisons of student achievement conducted by the International Association for the Evaluation of Educational Achievement ([IEA], 2013; Wagemaker, 2013). Since then, there has been growing interest in ILSAs, and an increasing number of countries have participated in these assessments in each cycle (Wagemaker, 2013). The influences of ILSAs include, but are not limited to, policy changes regarding teacher training, classroom instruction, the use of technology, hours of instruction, and class size (Heyneman & Lee, 2013). Similarly, there has been increasing attention to the results of ILSAs, research findings of secondary analyses by using ILSA data, and technical/methodological advancements to improve large-scale survey research. Some examples of the most well-known ILSAs in education which have a large number of participants from international countries include the Programme for International Student Assessment (PISA) conducted by the Organisation for Economic Co-operation and Development (OECD), Progress in International Reading Literacy Study (PIRLS), and TIMSS conducted by the IEA.

TIMSS is a large-scale international assessment used to monitor trends of fourth- and eighth-grade students’ mathematics and science achievement, administered every four years since 1995 (Mullis & Martin, 2013). The TIMSS mathematics achievement score distribution was centered at an international mean of 500 with a standard deviation of 100. This scaling approach was determined with the TIMSS 1995 mathematics and science achievement assessment tests and has been sustained to monitor trends over time (Martin et al., 2013). In total, 49 countries and seven benchmarking entities participated in TIMSS 2015 (the focus of this study), and national mathematics average scale scores ranged from 353 to 618 (Mullis, Martin, Foy et al., 2016).

Turkish participation in TIMSS

Turkey participated in the TIMSS in 1999 for the first time at the eighth-grade level. Turkey then participated in 2007 solely at the eighth-grade level and in 2011 and 2015 at the fourth- and eighth-grade levels (Mullis, Martin, Goh et al., 2016). The Turkish students’ mathematics average scale score has been below the TIMSS international mean of 500 for each cycle, despite an increasing trend. For eighth graders, national mathematics average scale scores increased from 429 in 1999 to 432 in 2007, 452 in 2011, and 458 in 2015. For fourth graders, the national mathematics average scale scores increased from 469 in 2011 to 483 in 2015 (Mullis, Martin, Foy et al., 2016; Yildirim et al., 2016).

Even though Turkey was part of 2011 TIMSS at the fourth-grade level for the first time, student background data were not fully available from this cycle. The home resources for learning scale for 2011 was created using responses to the student questionnaire as well as responses in the PIRLS home questionnaire. Since Turkey was not part of the PIRLS in 2011, the home resources for learning scale, which can be used to represent SES (Caponera & Losito, 2016), was not available for the TIMSS in 2011 for Turkish fourth graders (Martin & Mullis, 2012; IEA, 2013). In addition to the items regarding family SES, items regarding students’ early literacy and numeracy activities were included in the home questionnaire for fourth graders; therefore, the availability of these measures was limited for the 2011 TIMSS (IEA, 2013). Fortunately, home questionnaire data were available for the 2015 cycle (IEA, 2017). These extended datasets have allowed researchers to build and test more comprehensive statistical models.

Beginning with its initial participation, Turkey has put great emphasis on TIMSS results by preparing national reports with the support of the Turkey MoNE (TIMSS Türkiye, n.d.). Also, there have been numerous studies conducted using TIMSS data to investigate factors associated with Turkish students’ mathematics and science achievement (e.g., Akyüz, 2014; Arıkan et al., 2016; Oral & McGivney, 2014; Yetkiner-Özel et al., 2013). A common finding from these studies was that family SES was a prominent factor in explaining achievement differences between students.

To understand the family SES-achievement association, we examined the role of home resources for learning. In this study, SES was measured with the home resources for learning scale, from which a three-level index was created (IEA, 2017): many resources, some resources, and few resources. Only 5% of Turkish fourth-graders fell into the many resources category where the international average was 17%. Although the national average mathematics achievement scale score was 483, and Turkey ranked 36th among the 49 participating countries, Turkish students with many resources had a mathematics average scale score of 590, which ranked 9th, sharing the same scale score and rank with Hungarian students (Mullis, Martin, Foy et al., 2016). Moreover, the correlation between the home resources for learning scale and mathematics achievement was 0.51, and this was the third-largest correlation among countries, following Hungary and Slovak Republic in the 2015 TIMSS (Martin et al., 2016).

In summary, although SES typically explains substantial variation in students’ mathematics scores in general, it is not malleable or available for manipulation to improve students’ learning and achievement. Therefore, other potentially malleable student and school characteristics were examined to understand their association with mathematics achievement in the presence of SES, which could hold promise for reducing the effect of SES on mathematics achievement. The findings of this research can contribute to the Turkish and international educational literature on educational equity. The findings will also provide insight into countries sharing similar characteristics with Turkey on possible routes to reduce achievement gaps, acknowledging the presence of SES.

Background

Socioeconomic status

Since the Coleman report (1966), SES measures have become a core focus of educational research as well as a relevant and important covariate in statistical models. Meta-analysis researchers revealed a modest to moderate association between SES and achievement (Harwell, Maeda, Bishop, & Xie, 2017; Sirin, 2005; White, 1982), with a stronger association when the data were aggregated to the school-level (Sirin, 2005; White, 1982). The role of SES has been examined in the Turkish educational system framework within an educational equity context, revealing a significant positive association between SES and mathematics achievement (Akyüz, 2014; Bellibaş, 2016; Kalaycıoğlu, 2015; Özdemir, 2016; Tomul & Savaşçı, 2012; Yetkiner-Özel et al., 2013).

A high association between SES and academic achievement at both individual and school levels is a serious threat to educational equity and opportunities to learn. According to a report published by the Education Reform Initiative (Oral & McGivney, 2014), there was no difference in the net school enrollment rate between families with different educational backgrounds. This finding is a positive outcome of the compulsory education system in Turkey; however, private (family) spending on education dramatically differs based on the parents’ educational background. For instance, parents who have a college degree spend three times more for education than parents whose highest degree is a primary school degree. In a society such as Turkey, where national exams and future job opportunities are highly competitive, the privilege of spending extra resources for education (e.g., private tutors, test preparation courses, educational technologies) is one of the major contributors to future success (Bakış et al., 2009; Education Reform Initiative, 2009).

School effectiveness

SES is an inseparable part of school effectiveness research since students are not randomly assigned to schools, but often attend schools based on family and neighborhood socioeconomic characteristics, particularly in early grades of education. Therefore, to examine school effectiveness by using achievement data, it was suitable to use SES as a covariate variable (Raudenbush & Bryk, 1986). Similarly, one of the purposes of this study was to examine how school effectiveness variables are associated with mathematics achievement when controlling SES to explain achievement differences between schools.

In a school effectiveness meta-analysis, effectiveness enhancing factors were summarized as orderly school climate, regular monitoring/evaluation, curriculum quality, homework, parental involvement, school emphasis on academic success, cooperation among educators, differentiated instruction, and opportunity to learn (Scheerens et al., 2013). In such a centralized education system as Turkey’s, teachers, assessments, and curriculum qualities of a school do not fully represent the quality of the school. Therefore, other school climate-related variables such as a safe and orderly school environment, emphasis on academic success with cooperation between teachers and parents, the school’s physical condition and its educational resources, as well as the quality of educational instruction were taken into consideration as indicators of school effectiveness.

The role of a school’s quality versus family SES in a student’s academic achievement and how they interact is an ongoing research topic. This association can be complicated and may vary depending on a country’s level of economic development. Heyneman and Loxley (1983) reported that SES had a greater effect on achievement in economically developed nations, whereas schools in low-income countries had more of an effect. Nevertheless, this claim, known as the Heyneman–Loxley effect, was disconfirmed (Baker et al., 2002; Bouhlila, 2015; Ilie & Lietz, 2010) and partially confirmed by later researchers. For instance, Chudgar and Luschei (2009) showed that the school’s effect relative to family SES on academic achievement was more substantial in low-income countries, but weaker than what was found in earlier findings.

Gustafsson et al. (2018) examined the SES-mathematics achievement association with respect to the development level of a country using TIMSS data. According to their findings, there is more slope variance in developing countries, suggesting that the SES-achievement association within schools varies more in developing countries. Also, they showed that safe and orderly school environments, school emphasis on academic success, and instructional quality could be used to reduce the SES-achievement slope within a school of mostly developed countries. On the other hand, in South Africa, Tunisia, Morocco, Botswana, Iran, Indonesia, Thailand, Ghana, Honduras, and Turkey, there was a positive correlation between school-SES and within-school SES-achievement slope, which indicates these school systems are anti-compensatory for educational equity.

This anti-compensatory effect of positive school characteristics can be explained by the tendency that parents who have a higher SES may be in pursuit of a school where the reputation and school quality are high, which eventually adds more SES-based segregation across schools. In a policy brief produced by using the TIMSS 2011, authors examined the role of instructional quality, school emphasis on academic success, and a safe and orderly school environment in terms of reducing the association between SES and achievement. Accordingly, they showed that for Turkish students, safe and orderly school environments had anti-compensatory effects, where it strengthened the association between SES and mathematics achievement (Nilsen, Blömeke et al., 2016).

In a research synthesis reviewing the association between SES, inequality, and school climate, researchers reported that a positive school climate was associated with academic achievement and reduced the negative impact of low SES on academic achievement (Berkowitz, 2017). Even though some school characteristics may have an anti-compensatory effect on educational equity, a positive school climate can be influential in reducing the impact of a family’s SES. In a qualitative study, four Turkish urban elementary schools were examined where school resources and technology used for instructional purposes were limited, the classes were crowded, and students came from low or middle-income families; however, at the same time, the students’ academic achievement was high (Şahin, 2008). Accordingly, what made these schools and students academically resilient was the positive school climate that included parent involvement, teachers’ high academic expectations, and positive and strong school principal-teacher-parent-student relationships. Similarly, the IEA published a policy brief using the TIMSS 2011, reporting that Turkey, Hong Kong Sar, Kazakhstan, and Thailand have academically resilient students. According to this policy brief, educational aspirations and school emphasis on academic success were statistically significant factors for Turkish students with high mathematics scores who come from economically-disadvantaged families (Erberber et al., 2015).

In conclusion, the role of school effectiveness on academic achievement and the impact of family SES still needs further research. For instance, for Turkish students, positive school environment variables may have anti-compensatory as well as compensatory effects on reducing SES-based achievement gaps. Therefore, the association between SES and school effectiveness variables and mathematics achievement were studied where SES was used as a covariate. As an OECD member and developing country, studying Turkish fourth-grade data can contribute to the literature on school effectiveness and the further development of the Turkish education system.

School readiness

Pre-primary education is an essential component of early childhood development in which cognitive, social, emotional, and physical developments are fast and crucial for later development (Barnett, 1995, 2011; Gorey, 2001; Kagitcibasi et al., 2009). Pre-primary education plays a significant role in future academic achievement by improving school readiness (Erkan & Kırca, 2010). It is associated with higher mathematics achievement during primary school (Altun & Çakan, 2008; Berlinski et al., 2009; Sandoval-Hernandez et al., 2013; Tramontana et al., 1988), even after controlling for family SES (Melhuish et al., 2008; Waldfogel & Zhai, 2008). The positive effect of pre-primary education is observed in upper-secondary school level students as well (Ağırdağ et al., 2015; Altun & Çakan, 2008).

Preschool education benefits early literacy and numeracy skill development, which in turn benefits later mathematics achievement (Aubrey et al., 2006; Bodovski & Farkas, 2007; Claessens et al., 2009; Claessens & Engel, 2013; Manfra et al., 2017), even when controlling for family SES (Duncan et al., 2007; Niklas & Schneider, 2017). However, early literacy and numeracy activities should not be limited to activities in the preschool and kindergarten classes but should be supported by parents at home as well.

Preschool attendance can reduce achievement gaps, specifically in economically disadvantaged populations (Duncan & Magnuson, 2013; Engle et al., 2011; Kagitcibasi et al., 2009; Tucker-Drob, 2012). However, considering that a child’s enrollment in preschool is associated with the family’s economic situation (Batyra, 2017; Education Reform Initiative, 2016), greater emphasis should be placed on early literacy and numeracy activities at home in order to reduce the later SES-based achievement differences (Arıkan et al., 2016; Meinck et al., 2018).

Considering the positive impact of pre-primary education on a student’s later academic life, it is evident that the classroom and school environment can benefit from its students’ school readiness in what can be described as a domino effect. For instance, 60% and 36% of fourth graders’ teachers in the TIMSS from 2015 stated that classroom instruction was limited at some level and a lot, respectively, due to a lack of previous knowledge.

Attitudinal variables

In addition to SES and school-related variables, attitudes towards mathematics were associated with mathematics achievement (Geesa et al., 2019). For the fourth-grade level, students confident in mathematics, and students like learning mathematics (measures of intrinsic motivation) scales were available in the database (IEA, 2017). Although in the current study we did not examine a causal association, a model was proposed to examine the associations of some student- and school-level factors with mathematics achievement above and beyond the influence of SES. Researchers, educators, and policymakers need to investigate the role of malleable factors in improving national-level mathematics achievement and reducing achievement gaps. Therefore, variables that could be manipulated to improve achievement and reduce gaps were included in this study.

Previous researchers provided evidence that students’ confidence in mathematics was a significant predictor of mathematics achievement even when controlling SES (Akyüz, 2014; Arıkan et al., 2016; Kalaycıoğlu, 2015). However, there has been an ongoing debate regarding the causal order of academic self-concept and academic achievement (Guay et al., 2003; Helmke & van Aken, 1995; Marsh et al., 1999; Marsh, 1990). Accordingly, the self-enhancement model suggests self-concept precedes achievement, whereas the skill-development model suggests the opposite. In older ages, self-concept becomes a relatively stable causal influence on achievement due to the cumulative effect of prior academic achievement-related successes and failures. On the other hand, in earlier grades, they have a reciprocal association, and therefore, self-concept interventions may not make significant contributions to improve achievement (Guay et al., 2003; Helmke & van Aken, 1995). Based on these findings, subject-specific self-concept measures, such as self-confidence in mathematics, were not included in this study; however, intrinsic motivation for mathematics was included as a motivation measure.

Student engagement in mathematics instruction and instruction quality

Students’ engagement in instruction is another factor in explaining achievement differences (Akyüz, 2014; Kahraman, 2014), particularly for the earlier grades in Turkey. Younger students may have less private tutoring or after-school programs compared to older students who prepare for national examinations for high school or university entrance and placement (Tansel, 2013).

Similarly, instructional quality is another contributor to academic achievement (Nilsen, Gustafsson et al., 2016), and instructional quality is linked to teacher quality (Blömeke et al., 2016). Some indicators of teacher quality include educational background (e.g., education major, years of experience, other professional development activities), attitudinal characteristics, and students’ classroom experiences (Akiba et al., 2007; Blömeke et al., 2016).

In Turkey, in terms of educational equity, teacher characteristics significantly differ between regions. For a new teacher who wants to work in public schools, it is common (and almost compulsory) to start service in a public school located in rural, economically disadvantaged neighborhoods, or eastern provinces where SES levels are relatively low⁴ (Özoğlu, 2015). Therefore, it is very likely that novice teachers serve in schools where students come from families with low SES (Yetkiner Özel & Özel, 2013). On the other hand, the majority of these teachers need to hold relevant university degrees, and as a result, they will be armed with new skills in educational technologies and trained with more current pedagogical knowledge. Therefore, rather than focusing on teacher quality, students’ views on engaging teaching were used for instructional quality in the study.

Purpose of the study

Even though SES is a strong predictor of student achievement, other student-level and school-level factors were examined regarding their association with mathematics achievement, controlling for SES. These variables were represented by the scales available in the TIMSS dataset (IEA, 2017). Student level factors included students’ intrinsic motivation for mathematics, their engagement in mathematics instructions, preschool education, and early literacy and numeracy activities. School effectiveness factors included a safe and orderly school environment, school emphasis on academic success, instructional quality, school conditions and resources, and school readiness for learning.

Therefore, the purpose of this study was first to reveal the magnitude of the association between SES and mathematics achievement, and second to examine the role of student- and school-level factors above and beyond SES.

In light of the previous and current study findings, recommendations were made to parents, policymakers, school leaders, educators, and educational researchers. Moreover, we provide insight for other countries with similar characteristics as Turkey, which is an OECD member, and a developing country with upper-middle-income (The World Bank, n.d.).

Research questions

To expand on previous literature, the proposed research questions were:

• 1. How much of the variance in mathematics achievement is within and between schools?

• 2. How much of the variance in mathematics achievement is explained by SES at both the student and school levels?

• 3. How much of the variance in mathematics achievement is explained, without controlling for SES, by students’ intrinsic motivation for mathematics, student engagement in mathematics instructions, preschool education, and early literacy and numeracy activities at the student level; and safe and orderly school environment, school emphasis on academic success, instruction quality, school conditions and resources, and school readiness for learning at the school level?

• 4. How much of the variance in mathematics achievement is explained, when controlling for SES, by student and school-level factors (from question 3)?

Method

Here, we describe the TIMSS 2015, including research design, sampling procedure, the fourth-grade Turkish sample, data sources and measures, and data analyses used to address the research questions.

Research design

Our study had a cross-sectional research design where associations between students’ mathematics achievement and several student- and school-related variables were examined. In the TIMSS, a stratified two-stage cluster sample design was employed. Accordingly, in the first stage, schools were randomly selected from the population of schools, and in the second stage, classrooms were sampled at the target level (fourth grade) within selected schools. For the Turkish fourth-grade sample, from a majority of the schools where the school size was not so large, only one classroom was selected, and all students in the selected classrooms participated in the study. The stratification variables for the Turkish sample were urbanization and region (Martin et al., 2013).

As a result of the stratified two-stage cluster sample design, in order for sample statistics to represent the population correctly, several sampling weights were calculated (Martin et al., 2013), and were used during data analysis (Rutkowski et al., 2010). The weights included a total student weight (TOTWGHT), student house weight, student senate weight, and school weight (SCHWGT). Except for SCHWGT, other weights are a linear transformation of the total student weight (Rutkowski et al., 2010). Total student weight and school weight are raw weights; their sums equal the target populations. Since the current study research design is a one-country cross-sectional design, TOTWGHT and SCHWGT were used to calculate proper level-1 (student) and level-2 (school) sampling weights to use in two-level hierarchical linear models. Being a total student weight, TOTWGT includes a school weight factor that is the inverse of the joint probability of being selected from a school and class. In this study, student weights within-schools were used for the level-1 of the multilevel models which were obtained by dividing TOTWGT by SCHWGT. Accordingly, level-1 weights represent the inverse of the probability of a student being selected within a school, given the school that was selected (Rutkowski et al., 2010). SCHWGT weights were used for the level-2 of the multilevel models.

Sample

In the TIMSS 2015 for Turkey, 6456 fourth-grade students from 242 schools were sampled to represent 1,189,025 students and 21,154 schools in the target population. The average national age at the time of testing was 9.9 (LaRoche et al., 2016). In total, 49.2% of the sample was female, 50.8% was male.

Data sources and measures

The purpose of the TIMSS is to assess fourth- and eighth-grade students’ mathematics and science achievement with contextual information that may be used to improve teaching and learning including information about curriculum and curriculum implementation, instructional practices, and school resources. In addition to cognitive assessments, the TIMSS collects noncognitive data from students, teachers, school principals, and parents by context questionnaires. Assessment and context measures were created using item response theory models.

In the TIMSS, mathematics assessment items were given to the students through a matrix sampling approach. Accordingly, the fourth-grade level item pool consisted of 175 items divided into 14 booklets. Each item appeared in two booklets that allowed linking between booklets. This matrix sampling approach eases the time and resource burden ensuring estimation of mathematics proficiencies on the same item pool even though each student answered a subset of items. Due to planned missingness resulting from the matrix sampling approach, five plausible values were computed for mathematics proficiencies. Later, mathematics scores were scaled to a distribution with an international mean of 500 and a standard deviation of 100. In the current study, all five plausible values were used as dependent variables (Martin et al., 2016).

For each context variable, both continuous scales and index variables were created and made available to the public by the TIMSS researchers. The technical details of these measures and related validity evidence for each country were reported in Methods and Procedures in TIMSS 2015 (Martin et al., 2016). Principle components analysis results, coefficient alpha reliabilities, and data sources of these scales for the current sample are provided in Table 1 (Martin et al., 2016).

Table 1 Principle components analysis results and coefficient alpha of study scales

Based on validity evidence for scales, each context measure was created with the partial credit model (Masters & Wright, 1997) where scale scores generally ranged between − 5 and 5. Later, these measures were scaled to a distribution with an international mean of 10 and a standard deviation of 2 (Martin et al., 2016). Accordingly, any scale score above 10 represents a high attribute level, whereas below 10 represents a low attribute level.

Variables

In this section, general information on variables is presented. More information on the items used to create the variables, their response choices, and data sources are presented in the Appendix.

Socioeconomic status

Although SES is a widely-used variable in many educational research studies, there is no consensus in terms of representations of SES, and generally, atheoretical approaches are adopted when conceptualizing SES (Harwell, 2018). Even though it is typical to use only economic facets of SES, researchers have shown that SES measure have a multidimensional structure, including economic and cultural aspects (Yang & Gustafsson, 2004). Similarly, Harwell (2018) suggested a hypothetical multidimensional framework for SES that combines human, material, and social capital.

In the TIMSS, the scale of home resources for learning included information about the educational and occupational background of parents, number of books at home, internet access, and whether the student has their own room at home. Since this scale included different aspects of SES accommodating a broader definition (Gustafsson et al., 2018), the home resources for learning scale was used to represent student SES. In addition, this scale’s scores were aggregated to the school level and used as a school-level SES covariate.

School effectiveness

An effective school has a positive school climate indicating cooperation of parents with teachers and school principals, high achievement expectations from students, safe and orderly school environment, suitable physical school conditions, educational resources, and high instructional quality (Berkowitz, 2017; Erberber et al., 2015; Gustafsson et al., 2018; Şahin, 2008; Ünal et al., 2010).

In this study, to cover different aspects of an effective school, scales created by the TIMSS were used, including problems with school conditions and resources, school emphasis on academic success, and safe and orderly school. These measures were created by data collected from teachers and school principals. Lastly, the instructional quality variable represents the perceptions of students regarding the extent to which they were engaged in instruction. Accordingly, it was obtained by using students’ views on engaging teaching in mathematics lessons, aggregating it to the school level.

School readiness

School readiness at the student level was addressed by preschool education attendance (dichotomous variable) and how parents were involved in the student’s early literacy and numeracy activities before primary school. For the latter variable, early literacy and numeracy activities before beginning primary school scale was used.

Another school readiness variable used at the school level comprises information about student performance on several literacy and numeracy activities when they started primary school. Students might have gained such skills at pre-primary school or at home, or both. Either way, school readiness included the extent to which a school’s students could do literacy and numeracy tasks. For this purpose, could do literacy and numeracy tasks when began primary school scale was aggregated to schools. All of these measures were created using item responses collected from home questionnaires.

Attitude towards mathematics

To address students’ intrinsic motivation towards mathematics, we used the students like learning mathematics scale. With this scale, information regarding the extent to which students enjoy learning mathematics was covered.

Student engagement in mathematics instruction

Students’ engagement with the instruction variable was represented by students’ views on engaging teaching in mathematics lessons scale. This scale included information about how students perceived their teachers’ classroom instructional practices and the extent to which they found mathematics instruction engaging.

Analyses

As a result of two-stage cluster sampling, TIMSS data have a hierarchical structure where students are nested within schools. When such a hierarchical data structure is held, observations within the same cluster tend to share similar characteristics more so than observations randomly sampled from the entire population, jeopardizing the independence of observations (residuals). When the independence assumption is violated, standard errors become smaller than they should be, leading to improper result and interpretation of the significance of regression coefficients (Osborne, 2000). Since TIMSS data have a nested structure, hierarchical linear models (HLM) were employed for the analyses (Raudenbush & Byrk, 2002). Our models have two levels: students at level-1 are nested in schools at level-2. In addition, descriptive statistics of the variables at both student and school levels were estimated.

In this study, for data manipulation, descriptive analyses, and visualization of the associations between SES and mathematics achievement scores, dplyr (Wickham et al., 2019; v.0.8.0.1), psych (Revelle, 2018; v.1.8.12), and ggplot2 (Wickham et al., 2018; v.3.1.0) packages in R software environment (R Core Team, 2018, v.3.5.2) were used. For hierarchical linear modeling, HLM 8 (Raudenbush et al., 2019) was used.

There were no missing data at level-2; however, there were some missing values at the student level. Cases with missing data constituted about 6% of the cases in the sample data, and before and after listwise deletion, descriptive statistics and correlations between variables remained the same. Therefore, cases with missing data were deleted when creating the data files for use in HLM. In addition, predictor variables at the student level were group-mean centered, and at the school level they were grand-mean centered.

Moreover, after running the analyses, model assumptions were examined. It was found that linearity, normality, and independence assumptions were held. Final fixed-effects and random-effects variances were obtained by combining five plausible values. The main purpose of the analyses was to estimate the variance explained by the additive effects of student and school variables above and beyond SES. Therefore, since robust standard errors of fixed effects do not impact random-effect variances and unequal level-1 residual variances across groups were observed to some degree, fixed effects with robust standard errors were examined.

HLM models constructed to address the research questions are given in Table 2, where the model number represented the research question number. In the models, Y_ij represents the mathematics achievement score (Y) of student i in school j.

Table 2 Hierarchical linear models used to answer research questions

Results

Descriptive analyses

According to the original TIMSS 2015 Turkish fourth-grade mathematics datasets, there were 6456 students and 242 schools in the sample; however, cases with missing values were excluded from this sample. Therefore, the final sample had 6051 students and 242 schools. Descriptive statistics of the variables used at both the school and student levels are provided in Table 3. Accordingly, there were no extremely skewed variables at the student level. At the school level, the school readiness variable we created had a negatively skewed distribution.

Table 3 Descriptive statistics of the sample at the student and school levels

As shown in Table 3, considering mathematics scores were scaled with an international average of 500 and a standard deviation of 100, Turkish fourth graders’ mathematics scores (all plausible values) were below the international average. Similarly, all the noncognitive measures were scaled to distribution with a mean of 10 and a standard deviation of 2. Therefore, Turkish fourth graders were below the international average scale score for SES and early literacy and numeracy activities. Students’ intrinsic motivation towards mathematics and their engagement in the instruction were above the average scale score. For the school level, only instruction quality was above the average scale score. For all other student and school variables, Turkish fourth graders were below the international average scale score.

Further, the associations between students’ SES and their mathematics scores were estimated for each of the 242 schools. For the mathematics scores, the five plausible values were used and the results are illustrated in Fig. 1.

Fig. 1

Associations between SES and mathematics achievement within schools by plausible value

Accordingly, within-school slopes were randomly varying across schools, and as indicated in Fig. 1, this association can be defined as strong or weak, as well as negative or positive for different schools. However, between schools, there was a clear increasing trend in the schools’ average mathematics scores when students’ SES increased. Further demonstration is given in Fig. 2.

Fig. 2

Association between school-level SES and mathematics achievement by plausible value

The associations of school-SES and school-mathematics achievement are plotted in Fig. 2. Horizontal and vertical lines on the plots indicate international average scale scores. In Fig. 2, regression lines represent the association between school-SES and school-mathematics achievement. Therefore, we observed a strong association between school-SES and school-mathematics achievement. Also, the upper left corner includes schools that have academic resiliency, which means that students in these schools came from families with low SES (below the international average SES); however, the schools' mathematics scores were above the international average. Moreover, correlations between variables at the student level and school level are reported in Table 4.

Table 4 Inter-item correlations at student and school levels

Although not reported in Table 4, the correlations among the mathematics plausible values at the student level were .91 to .92 and at the school level were .99. The correlations between mathematics achievement (plausible values) and SES were around .52 at the student level (Table 4). Even though this correlation suggested a moderate-level association, this was the third largest among the 64 participant countries, after Hungary and Slovakia. The next largest correlations with mathematics scores were with the early literacy and numeracy activities. For school-level variables, school-SES and school-mathematics achievement values had large correlations, around .80 (Table 4). This finding confirmed the association illustrated in Figs. 1 and 2. Therefore, the association between SES and mathematics was stronger at the school level. This reflects the fact that generally, group scores are more stable and precise, leading to higher correlations. There were also low-to-moderate correlations among other variables.

Hierarchical linear models

In order to answer the research questions, four HLM models were fitted, and the results are provided in Table 5. We summarized fixed and random effects with standard errors, as well as estimates of variance explained relative to the unconditional model in this table.

Table 5 A summary of the fixed and random effects estimated from HLM models

For the first research question, to examine how much of the variance was due to school differences, the unconditional linear model (model 1) was fitted, and results are given in Table 5. Accordingly, the intraclass correlation is .39, which is the ratio of ${\upgamma}_{00}$ to the total variance (${\upsigma }^{2}$+${\upgamma}_{00}$). This means 39% of the variation in mathematics achievement scores was due to school differences.

For the second research question, the proportion of variance explained by SES at both the student and school levels was examined, and the results are given in Table 5. Accordingly, 7% of the student level variation was explained by the inclusion of SES, whereas 67% of the school level variance was explained by the inclusion of school-SES. A one unit change in SES (corresponding to a one-half standard deviation change) was associated with about a 14 point change on within-school mathematics scores, and a one unit change in school-SES (about 0.7 SDs) was associated with a 34 point change on the school-mathematics scores.

For the third research question, student-level and school effectiveness variables (other than SES) and their associations with mathematics achievement were studied. Student-level variables explained 14% of the student-level mathematics score variance, and school-level variables explained 39% of the school-level mathematics score variance (Table 5). According to model 3, at the student level, pre-primary education and intrinsic motivation were more associated with change in mathematics scores than other variables. For instance, mathematics scores of students who attended pre-primary school were about 19 points higher than scores of students who did not attend pre-primary school on average, controlling for the other predictor variables. In addition, at the school level, instruction quality and school readiness explained mathematics performance more than other variables. Accordingly, a one-unit change in instruction quality (about 1.3 SDs) was associated with a 22-point school mathematics score change on average.

For the fourth research question, all of the variables at both levels were included in the model. Accordingly, all of the student-level variables explained 18% of within-school variance, and the school-level variables explained 72% of between-school variance in mathematics scores. In addition, the ratios of explained variance by model 4, compared to model 2 and model 3, are provided in Table 5. Considering the variables in model 2 (SES only) and model 3 (all variables except SES), for within-school variation, the additive effects of individual attitudes, engagement, exposure to early literacy and numeracy activities, and preschool education explained more variance compared to student-level SES; whereas for between-school variation, school-SES explained more variance compared to the additive effects of other school-related variables. More specifically, comparing the explained variances of models 3 and 4, relative to the additive effects of student and school level variables; SES explained 4 percentage points more at the student level and 54 percentage points more at the school level. Similarly, comparing the explained variances of models 2 and 4, relative to the effects of SES at both student and school levels; the additive effects of student and school variables explained 12 and 16 percentage points more at student and school levels respectively.

In model 4, at the school level, along with school SES, instruction quality and school readiness were found positively and significantly associated with mathematics achievement, after controlling for school SES and other variables. Accordingly, a one unit change in school SES (about 0.7 SDs), instruction quality, and school readiness were associated with about a 29, 18, and 7 percentage points increase in a schools’ mathematics scores respectively, all else constant. At the student level, preschool education and students’ intrinsic motivation towards mathematics were more associated with change in mathematics scores, even after controlling for SES and other student variables. For instance, attending pre-primary school was associated with a 14-point change in student mathematics scores within a school, on average, when controlling the effect of other variables at the student level.

We conducted additional analysis to examine the partial correlations between school-level variables and a school’s average mathematics scores, when controlling for SES.

The associations between instruction quality and mathematics scores were significant even after the effect of SES was controlled (Table 6). However, the associations between other school effectiveness variables and mathematics scores were insignificant (p > 0.01). This may indicate a potential mediational role of school climate variables on the association between school-SES and mathematics achievement. Such an examination was not among the research questions of the current study, and additional research is needed to fully examine the mediational role of SES.

Table 6 Partial correlations between variables when controlling school-SES

Discussion

Our main purpose in this study was to examine factors, additional to SES, associated with mathematics achievement to propose a different perspective that may inform policy action and practice. Accordingly, we highlight important findings regarding achievement gaps and inequity in a developing country, Turkey, where SES is the dominant determinant.

Previous researchers showed that SES is one among the most strongly correlated factors with academic achievement. When separating the effect of SES within and between schools, we show that the influence of SES is stronger at the school level. In other words, in terms of SES, we observe a high degree of heterogeneity between schools. Previous researchers showed that variables that indicate a positive school environment, including a safe and orderly school environment, school emphasis on academic success, and instructional quality, could be used to reduce the effect of SES within schools (Gustafsson et al., 2018; Nilsen, Blömeke et al., 2016). Our research did not examine their effect on student achievement within schools; however, we find that these variables are positively linked with school-SES rather than being effective in reducing achievement gaps between schools.

Ünal et al. (2010) showed that students enrolled at low-achieving schools also came from families with low income and low education levels. In such schools, students’ school readiness and motivation for academic success were low. In these schools, safety and discipline problems were more frequent. On the other hand, parents who had high SES levels also had a greater active role in their children’s homework, their social and cultural development, and their future educational preparation. Likewise, these parents collaborated with teachers more effectively. Similar to research findings conducted by Ünal et al. (2010), our findings confirm that school segregation in Turkey is based on SES. However, due to the country’s social dynamics and accompanying residence-based school enrollment system, improving school climate to reduce the achievement gap and educational inequity between schools is a multilayered process beyond educational policymaking.

To summarize, a positive school environment is associated with higher mathematics achievement for Turkish fourth graders. Therefore, parents, teachers, school principals, and students should collaborate to improve their school climate and create a positive learning environment. On the other hand, a positive school climate is mostly observed in schools where the students come from families with high SES that is potentially a result of school segregation based on SES. This scenario also poses a danger in that high performing schools will be appealing to high SES students, and that will increase SES-based school segregation. Nevertheless, the effects of quality of instruction, created by aggregating student reports on their mathematics instruction engagement, are significant, even after controlling SES. This is a positive finding concerning equity in teacher training and job placement, although there is still an unequal distribution of experienced teachers in the country (Özoğlu, 2015).

Moreover, exposure to early literacy and numeracy activities can reduce future SES-based inequities (Meinck et al., 2018). Parallelly, we find that preschool education and early literacy and numeracy activities are important predictors of achievement differences between students, even when controlling the impact of SES and other variables. However, currently, preprimary education is more available to children who come from economically advantaged families (Education Reform Initiative, 2017). Therefore, we provide evidence on the importance of fully funded preschool education, mainly since compulsory preprimary education is under consideration in Turkey. In addition, economically disadvantaged families should be supported financially to invest in their children’s preschool education and at-home learning activities (Education Reform Initiative, 2017).

The economic burden of preschool education also negatively impacts the participation of women in the Turkish labor market. Moreover, if the mother works, the grandparents or other relatives tend to take care of the children (Education Reform Initiative, 2017). Therefore, parents, grandparents, or other caregivers should be educated on how early childhood education and home activities are essential for a child’s cognitive development at an early age. For this purpose, educators and policymakers can raise awareness of the importance of early childhood education and offer formal or informal education programs on how to create such activities at home. When considering the fact that positive intrinsic motivation towards mathematics is also related to higher mathematics achievement, exposure to early literacy and numeracy activities at home or preprimary school can help students develop better attitudes towards mathematics as well (Colliver, 2018; Fisher et al., 2012).

Conclusion

To conclude, SES is a strong predictor of mathematics achievement between schools. Even though previous researchers indicated there might be more influence of school variables on achievement than SES in low-income countries, also known as the Heyneman–Loxley effect, we did not observe such an effect in Turkey. Instead, we see that schools whose students come from economically and socially wealthier families also have a more positive school environment and higher achievement that increase educational inequity in the country. Through this study, we hope to contribute to the literature on the SES–school effectiveness association, providing an example of an upper middle-income developing country.

Another important contribution of this study is showing how school readiness is important at both the student and school levels. We found significant roles of preschool education and early literacy and numeracy activities before primary school in relation to fourth grade students’ mathematics achievements even when controlling the effect of SES. Furthermore, we showed positive attitude towards Mathematics and engagement in Mathematics instruction are important factors in explaining achievement differences within schools. Given these findings, we can imagine deeper studies of academic resilience by exemplifying the role of student-level characteristics above and beyond SES.

Limitations

In this study, we encountered the potential mediational role of school climate variables; however, we did not hypothesize nor test them. In future research, the impact of SES on achievement through other school-related variables could be studied.

Lastly, even though we showed that SES-mathematics slopes vary between schools, we did not examine if any of the school-variables explain the variation of these slopes. Therefore, future researchers may extend HLM models presented here to intercepts-and-slopes as outcomes models (Raudenbush & Bryk, 2002). Such investigations will help researchers further understand the role of other context variables on SES-achievement associations within schools at the fourth-grade level (Gustafsson et al., 2018; Nilsen, Blömeke et al., 2016).

Appendix

See Table 7.

Table 7 TIMSS 2015 public data variables, noncognitive scales, and associated items