 Research
 Open Access
 Published:
Trend analyses of TIMSS 2015 and 2019: school factors related to declining performance in mathematics
Largescale Assessments in Education volume 10, Article number: 15 (2022)
Abstract
Gaining knowledge of what contextual factors may contribute to changes in student achievement across cycles of international largescale assessments (ILSA), is important for educational policy and practice. Addressing this necessitates advanced methodology that utilizes the trend design of the ILSAs. The Trends in International Mathematics and Science Study (TIMSS) data is suitable for such analyses as it measures students’ competence based on the participating countries curricula. In Norway, students’ performance in mathematics decreased from 2015 to 2019 as evidenced by TIMSS. During this timeperiod, there are indications that also school climate, student motivation and selfconcept decreased. This study investigates whether school climate (including bullying, a safe environment, school emphasis on academic success, and students’ sense of school belonging and wellbeing), student motivation (including interestenjoyment and utility value), and selfconcept declined from 2015 to 2019, and whether this possible decline is related to the decline in mathematics performance in Norway. The present study utilized a trend approach with mediation structural equation modelling. The results showed that school climate, the utility value of the subject and students’ selfconcept declined during this period, and that a safe environment and student selfconcept mediated the changes in achievement from TIMSS 2015 to 2019. Hence, declines in a safe school climate and student selfconcept were associated with declining achievements. While the study cannot prove a causal relation, it is discussed whether this could indicate that the decline in these predictors may explain the decline in mathematics achievement. The usefulness of the methodological approach for other countries, as well as the implications of the results for policy, practice, and research are discussed.
Countries may experience changes in learning outcomes between two cycles of international largescale assessments (ILSAs) such as PISA or TIMSS. In order to be able to make correct, evidencebased decisions, it is crucial that educational policy and practice gain knowledge on what school factors are related to these changes. TIMSS data provides a unique opportunity for such investigations. TIMSS, based on the participating countries’ curricula, is a trend study with representative samples of students, and includes contextual information provided by students, their teachers, and principals (LaRoche et al., 2020).
In Norway, grade 9 students’ performance in mathematics declined by 9 points in lower secondary school from TIMSS 2015 to TIMSS 2019 (Kaarstein et al., 2020; Mullis et al., 2020). Learning gain per year in lower secondary school is smaller than for instance in primary school, and generally decreases with students’ age (Luyten et al., 2017). Olsen and Bjørnsson (2018) found that one year of learning gain from grade 8 to grade 9 in Norway is about 22 points, which is in line with the Norwegian TIMSS report (Kaarstein et al., 2020). The 9 points decline from 2015 to 2020 would hence reflect approximately a third of a year of schooling (Bergem et al., 2016; Olsen & Bjørnsson, 2018). At the same time, there are indications that several aspects of school climate, student motivation, and selfconcept may have declined over the same time period (Mullis et al., 2020; Nilsen & Kaarstein, 2021; Wendelborg et al., 2020). According to previous research, school climate, student motivation, and selfconcept are key predictors of student performance (e.g. Eccles & Wigfield, 2002; Thapa et al., 2013; Wang & Degol, 2015). School climate is a broad concept encompassing academic aspects such as emphasis on academic success, school belongingness, a safe environment, and absence of bullying (Wang & Degol, 2015).
However, it is an open question whether the noticed declines in school climate, student motivation, and selfconcept contributed to the decline in performance in mathematics. It is important to gain insight into this, as stakeholders within educational policy and practice need this knowledge for evidencebased decisions. Students’ competence in numeracy is especially important in Norway, as this is one of the five basic competencies that are prioritized in the national curriculum along with reading, writing, oral and digital competencies (UDIR, 2022).
While a randomized control trial design with representative samples would be the best approach to answer such a question, it is unethical to provide a very good school climate and motivate only one group of students. ILSAs have representative samples and trend designs (meaning it is possible to compare achievements across time at the national level), yet their crosssectional design (i.e. the same individuals do not participate across time, the data are not longitudinal at the student level) does not allow for casual inferences. However, there are advanced methods that exploit ILSAs trend design to enable robust inferences (Gustafsson & Nilsen, 2022; Murnane & Willett, 2011; Nilsen & Gustafsson, 2014). The present study utilizes such methodology to exploit the trend design of TIMSS and investigate whether the noticed declines in school climate, student motivation, and selfconcept are related to the decline in mathematics performance. The methodology provides a technique to investigate whether contextual factors may be related to changes in countries’ performance and should hence be of interest for other countries that seek explanations for changes in performance on ILSAs.
Theoretical framework. School climate, student motivation, and selfconcept
School climate
School climate is a broad concept, and therefore it is common for researchers to focus on either one or a few of its aspects (Wang & Degol, 2015). Moreover, school climate is often defined differently from one study to the next (Grazia & Molinari, 2021; Nilsen & Teig, 2022). There are two major reviews of school climate (Thapa et al., 2013; Wang & Degol, 2015) and one systematic review of studies using ILSA data to investigate school climate (Nilsen & Teig, 2022). These three have reviewed a vast number of studies and concluded that a good school climate includes four aspects: 1. a safe environment, 2. sound relationships within the community, 3. a good academic environment, and 4. a good institutional environment.
A safe environment
A safe environment is reflected by a high degree of safety at school among students and employees and characterized by mutual respect, adherence to school rules and discipline, and absence of bullying (Thapa et al., 2013; Wang & Degol, 2015). Of all the participating TIMSS countries in 2015, Norway was one of the countries with the safest environment (Mullis et al., 2016a).
Safety is one of the aspects of school climate with the strongest relation to student outcomes (Mullis et al., 2016b), also in the Nordic countries (Gustafsson & Nilsen, 2016; Scherer & Nilsen, 2016). Absence of bullying is widely investigated in educational research (Nilsen & Teig, 2022) and is of great importance to student learning outcomes (Engel et al., 2009). Absence of bullying has been prioritized in Norway. Many Norwegian schools have special programs to reduce bullying such as the Olweus program (Havik & Ertesvåg, 2020; Olweus, 2010). Still, a largescale national study has shown that bullying, especially digital bullying through social media, has increased over the last five years (Wendelborg et al., 2020).
Relationships within the community
This aspect of school climate is characterized by sound relationships and connectedness between all members of the school community (students, parents, teachers, and the leadership). High quality of interpersonal relationships is often characterized by trust, respect, and support (Hoy et al., 2006; Wang & Degol, 2015), where a positive student–teacher relationship is especially important for student achievement (Kincade et al., 2020).
Common indicators of this aspect of school climate in ILSAs are students’ sense of school belongingness and their wellbeing (Borgonovi, 2022; Nilsen & Teig, 2022). These are important for student outcomes (Borgonovi, 2022; Nilsen & Teig, 2022; Scherer & Nilsen, 2016; Wang & Degol, 2015), also in Norway (Nilsen, 2016).
Academic environment
Schools with a good academic climate are characterized by a priority and focus on the academic environment or emphasis on academic success (Nilsen & Gustafsson, 2014; Scherer & Nilsen, 2016; Wang & Degol, 2015). In such schools, all members of the school community, including students, parents, teachers, and the leadership, prioritize learning and have common ambitions and visions (Hoy et al., 2006; Nilsen & Gustafsson, 2014).
In 2006, a curriculum reform took place in Norway. The main goals and components of this new reform included a shift from the knowledge students were supposed to learn towards competencies they were expected to demonstrate (Sivesind, 2013). Previous research in Norway found that the implementation of the 2006 curriculum, may have led to a larger emphasis on learning and success (Sivesind, 2013) and improved learning outcome (Nilsen & Gustafsson, 2014). The pressure, enthusiasm, and attention from educational policy and society regarding the implementation of the curriculum, remained high for many years (Sivesind, 2013). The work with the implementation led to a higher degree of professional development for the teachers (Mullis et al., 2016a). However, from 2015 to 2019 a decline in teachers’ participation in professional development, and less emphasis on academic success, were seen (Mullis et al., 2020). It could be that these declines took place as the enthusiasm with the implementation of the curriculum of 2006 faded when attention and pressure from educational policy and society diminished.
Institutional environment
A good institutional environment is reflected by adequate access to resources (Thapa et al., 2013; Wang & Degol, 2015). In Norway, access to resources is not problematic (Mullis et al., 2020) and the difference between schools’ performance is small (Nilsen et al., 2018; OECD, 2019a).. Therefore, this aspect of school climate is of little interest in this study. Although it may explain variation across countries, it is not expected to explain variances within a resourceful, egalitarian country (Buchholtz et al., 2020).
Student motivation and selfconcept
It is well known that student motivation and selfconcept are important for performance (Bandura, 1997; Deci & Ryan, 1985; Eccles & Wigfield, 2002, 2020; Marsh et al., 2019). A relation between student motivation and selfconcept and mathematics achievement has also been found in Norway (Jensen & Nortvedt, 2013; Kaarstein & Nilsen, 2016; Skaalvik et al., 2015).
Eccles and Wigfield (2002, 2020) developed a model called the expectancyvalue model, which is the theoretical foundation behind the questions on students’ motivation and selfconcept in TIMSS (Mullis et al., 2012). The model is rather complex and includes the interaction between a number of factors. However, there are three factors commonly included in educational research because these are strong predictors of student performance. The first is the interestenjoyment value Eccles and Wigfield (2002) describes this as “the enjoyment the individual gets from performing the activity or the subjective interest the individual has in the subject” (p.120).
The second factor is the utility value, which is about “how well a particular task fits into an individual's present or future plans” (Eccles & Wigfield, 2020, p. 5). This factor, toghether with the interestenjoyment value, is herin referred to as student motivation.
The third factor is what Eccles and Wigfield (2002) referred to as selfconcept. According to Eccles and Wigfield (2002, 2020), and further elaborated by Marsh et al. (2019), selfconcept is about faith in how well one will perform on a task based on past experiences and circumstances. In mathematics, selfconcept is especially important for performance (Marsh et al., 2006).
There has been an increased attention to the importance of students’ motivation (interestenjoyment value,utility value) and selfconcept in the STEM (science, technology, engineering and mathematics) field as these aspects are important for individual success, longterm performance, recruitment to future education and careers in STEM, and hence for a sustainable society (Marsh et al., 2006; Regan & DeWitt, 2015; Young et al., 2018). It is therefore a concern that a largescale national survey in Norway (“Elevundersøkelsen” or the student survey) identifies a decline in student motivation and selfconcept from 2016 to 2019 for students in lower secondary school (Wendelborg et al., 2020). This survey points to declines in students’ wellbeing, students feeling more stressed about schoolwork and feeling more depressed as possible explanations for the decline in students’ motivation and selfconcept (even if no causal evidence could be established). The decline in student motivation and selfconcept is supported by findings in TIMSS (Mullis et al., 2016a, 2020).
The present study and research questions
The above review indicates that both school climate, student motivation, and selfconcept may have declined from 2015 to 2019. At the same time, students’ achievement in mathematics has declined, and we hence ask the following research questions, using merged TIMSS 2015 and TIMSS 2019 data:

1.
How has students’ motivation and selfconcept in mathematics and the different aspects of school climate changed from 2015 to 2019?

2.
What is the relation between the changes in the predictors (i.e. students’ motivation, selfconcept, and the different aspects of school climate) and the changes in students’ mathematics achievement from 2015 to 2019? More specifically, we investigate whether changes in school climate, students’ motivation, and selfconcept may mediate the changes in mathematics performance over time.
Methodology
Data and samples
TIMSS is an international largescale assessment study that is repeated every fourth year. Norway has participated since the first cycle in 1995. TIMSS assesses the mathematics and science competence of students in fourth and eighth grade (fifth and ninth in Norway). Questionnaires to students, teachers and principals (and parents in grade 5) ensure collection of contextual information. About half of the more than 200 items in mathematics are trend items which are publicly unavailable and that allows for linking one cycle to the next (the same goes for science). Both these trend items, as well as the new items included in 2019, were included in the analyzes. Furthermore, the scaling of the achievements contributes to the linking of the cycles: the mean score was set to 500 points and the standard deviation (SD) to 100 points in 1995. After 1995, data from each cycle is scaled for comparability, always using the original scale with 500 as the scale midpoint and an SD of 100 (Martin et al., 2020). This is possible because many of the same countries participate in adjacent cycles, and, as mentioned above, about half of the test items are the same across two cycles. Having the same scale enables comparisons across cycles.
The sampling design is referred to as a twostage cluster sample design, where schools are randomly picked with probability proportional to their size in the first stage, and in the second stage, two classes are picked within each school (LaRoche et al., 2020). If the school is small and only has one class, then one class is selected.
The present study includes students in grade 9 who participated in TIMSS 2015 (N = 4697) and TIMSS 2019 (N = 4575) and their mathematics teachers (N = 239 in 2015 and 277 in 2019). The mean age of the students was 14,7 in both cycles with equal percentages of girls and boys.
Measures
To cover the wide range of topics specified in the frameworks (Mullis & Martin, 2017), TIMSS has a rotating matrix design of the mathematics (and science) items. This means that each student receives only a subset of all the items in the assessment pool. However, the results from the students are sewed together to provide results at the national level in terms of five plausible values (PV). These plausible values have to be taken into account in the analyses in the correct manner, for instance through imputation in the software Mplus (see e.g. Rutkowski et al., 2013).
All variables included in the present study are described in Table 1. The variables are sorted by their function (i.e. whether they are predictors, outcomes, or control variables), and according to what they measure (e.g. aspect of school climate or aspect of student motivation). These variables are measured as latent variables including only items used in both cycles. The items covering the underlying concepts should provide valid measures of the concepts according to our theoretical framework. The variables are retrieved from student and teacher questionnaires in 2015 and 2019, available on the webpage of TIMSS (https://timssandpirls.bc.edu/).
SES was included as latent control variable measured by the number of books at home and mother’s and father’s educational level.
Methods of analyses
The student and teacher data were merged using the software IDB analyzer (https://www.iea.nl/datatools/tools) to ensure that the links between teachers and their students is correct. Next, the data from 2015 and 2019 were merged to one SPSS file. This data file also contains a dummy variable called Time coded to 0 for the 2015 cycle and 1 for the 2019 cycle.
The methodological approach is a mediation structural equation model (SEM) with trend data (Murnane & Willett, 2011; Nilsen & Gustafsson, 2014). Mplus version 8 (Muthén & Muthén, 1998–2017) was used to estimate a mediation SEMmodel. The hierarchical design of the data, where students are nested within schools, was taken into account by using the Mplus option “TYPE = COMPLEX”, where the clustering variable is IDSCHOOL (the unique school identification). Robust maximum likelihood was used to account for missing data. The school weights and plausible values for mathematics were included according to recommended procedures (LaRoche et al., 2020; Rutkowski et al., 2013).
According to the international report, Norwegian 9th graders’ decline in mathematics achievement was about 9 points. Therefore, we expect the unstandardized regression coefficient of the effect of Time on achievement to be negative and somewhat close to 9 points (see Fig. 1 where c denotes the regression coefficient). The hypothesis is that there are other explanations for the decline in achievement than that of Time passing. Our question is whether school climate and student motivation and selfconcept may mediate the effect of Time on mathematics achievement (see Fig. 2). In other words, we examine whether changes in school climate, motivation, and selfconcept are related to changes in achievement.
The first step of the modelling was to do a confirmatory factor analysis to examine the reliability and validity of the latent variables through inspections of factor loadings, significance of indicators, and model fit (Hox & Roberts, 2011).
The second step was to estimate a correlation matrix between all the latent variables, to check whether some were highly correlated.
Having established the reliability of the latent variables, and inspected correlations and other descriptive statistics, created a foundation for the modelling. The third step of the modelling was to estimate the effect of Time on achievement (Fig. 1, the nullmodel). In the fourth step, the mediation model is built. All predictors as well as the dummy variable “Time” and mathematics achievement were included in one and the same model. Including all predictors in one model means that each predictor is controlled for by all other predictors and hence allows for evaluation of which predictor is the strongest. We therefore included all predictors in one model to evaluate which was the strongest mediator (i.e. the one with the strongest indirect effect). To be able to compare the indirect effects, one needs to compare the standardized estimates.
Figure 2 shows how we examine whether the aspects of school climate, the aspects of motivation, and selfconcept have changed over time (denoted by a in the figure), and whether these predictors have an effect on achievement (denoted by b). The direct effect of time on achievement (c`) is expected to decrease if the predictor mediates the time changes in achievement. The indirect effect of the predictor (a times b, not shown in Fig. 2), reflects how many points of the decrease in achievement is explained, or mediated, by the predictor. The indirect effect is the focus of research question 2. Conceptually, this reflects whether changes in predictors are related to changes in achievement. The crosssectional data cannot tell us whether the predictors caused the decline in achievement. However, the terminology of mediation models, such as “indirect effect” is used for clarity. We would like to point out though, that we investigate relations, not effects, as these are crosssectional data. We would further like to point out that there is a difference between: (1) merging the 2015 and 2019 data and merely estimate the regression coefficient between averaged predictor (e.g. utility value) and averaged achievement, and (2) our model that investigates whether the predictor mediates the time changes in achievement. Our model enables more robust and valid inferences than the former, even if no causal inferences may be drawn. The model does not deviate from that of a classical mediation approach, although it may be uncommon to use the Time variable as a predictor. Moreover, trenddata, rather than longitudinal data, is utilized. The trend analysis is based on countrylevel achievement, school climate, motivation, and selfconcept data from one country (i.e., one unit of observation) and two time points. This means that a representative sample of grade 9 students in 2015 and a representative sample of grade 9 students in 2019 were included in the analyses. Hence, these are not the same students followed over time. However, TIMSS is by definition a trendstudy because both achievements and contextual variables are comparable across time due to both the sampling of representative samples and the equated scales (Martin et al., 2020). This means that a significant decline in for instance achievement or school climate is not an artefact of different samples in two different cycles.
To control for SES, achievement was regressed on SES, and SES was modelled to correlate with all predictors.
Results
In the present section, we first present descriptive statistics and results from the null model. We then answer research question 1 and 2.
Table 2 shows the correlation matrix between all latent variables including all aspects of school climate, aspects of motivation, and selfconcept as well as SES. Weights were applied also in this analysis. As expected, the interestenjoyment value, utility value, and selfconcept showed high correlations, and the highest was that between selfconcept and the interestenjoyment value. The only variables for which there were no significant correlation, were between the utility value and a safe environment. It should be pointed out, however, that many of the correlations were small even if they were significant. The studentreported variables wellbeing and bullying were correlated with the two aspects of motivation and selfconcept, and they also correlated with each other. In addition, the teacher reported variables reflecting a safe environment and school emphasis on academic success were highly correlated. These correlations were expected as they were in line with previous theory (see e.g. Wang & Degol, 2016; Eccles & Wigfield, 2002).
The results for the nullmodel (Fig. 1) showed that the unstandardized regression coefficient of the relation between Time and mathematics achievement was 8.62 (standardized: 0.06) with a significance level of p < 0.05. This result is in line with the international TIMSS report, in which an average decline of 9 points is reported for the Norwegian students from 2015 to 2019 (Mullis et al., 2020). According to previous Norwegian results, this decline reflects about a third of a school year (Kaarstein et al., 2020; Olsen & Björnsson, 2018).
Research question 1
For research question 1, we found a significant decline from 2015 to 2019 in all variables, except for school emphasis on academic success and student interestenjoyment value. In Table 3, the decline of the variables are shown in the second colum, with the heading «regression coefficient a». A safe environment declined significantly by 0.10 (standardized regression coefficient: 0.31). This is a moderate to large decline. The other variables exhibited small, but significant declines. Seeing how these are all latent variables, it is hard to directly interpret this in a more conceptual or practical manner. Results from descriptive statistics provided in the international TIMSS reports () are easier to interpret. To exemplify, the results from the 2015 report showed that 72% (SE = 4.0) of the Norwegian teachers felt very safe (international average was 46% with an SE = 0.5). Norway then had the second most safe environment of all participating countries. In 2019, only 53% (SE = 4.0) of the Norwegian teachers felt very safe (international average was 48% with an SE = 0.5), with 17 countries out of 39 participating countries ranking higher in terms of saftey. Safety had declined by 19%.The principals answered the same questions about safety, and the decline in their report are in line with that of the teachers.This illustrates that the decline was substantial.
With regards to the relations between the predictors and mathematics achievement (regression coefficient b), some were significant and some were not. The variables with the largest regression regression coefficients with the highest proportion of the variance explained, were students’ selfconcept (b = 48.02, standardized coefficient = 0.54) and a safe environment (b = 33.62, standardized coefficient = 0.07), which together explained about 10% of the variance. Using the example of students’ selfconcept, the interpretation of the unstandardized regression coeffcient is as follows: an increase by 1 unit on the self concept scale is related to an increase of about 48 score points in mathematics achievement.
School emphasis on academic success and bullying were also significant, although the regression coefficient for bullying was small (b = 5.28, standardized coefficient = 0.03). Note that the variable bullying is reverse coded such that the decline shown in regression coefficient a, implies that bullying has increased and the regression coefficient b reflects the positive relation between absence of bullying and achievement.
School belonging and motivation were not significant.
The model presented in Table 3 had good model fit (CFI and TLI > 0.95, and RMSEA < 0.05). The results are also presented in Fig. 3 which shows the paths but not the indirect effects.
For the latent variables, the standardized factor loadings ranged between 0.62 and 0.93 and were all significant. The model fit and factor loadings confirmed the reliability and validity of the results (Hox & Roberts, 2011).
Research question 2
For research question 2, we examined whether the different aspects of school climate, the aspects of student motivation, and selfconcept could mediate the timechanges in achievement (i.e. the effect of Time on achievement). If the indirect effect (see Table 3, last column) is significant and negative, it means that the variable in question mediates the changes in achievement over time. The unstandardized indirect effect indicates how many points out of the total 8.6 points of declined achievement is explained, or mediated in the model.
Students’ self concept declined significantly from 2015 to 2019, had a significant association with achievement, and mediated the relation between Time and achievement. The indirect effect for selfconcept is 5.62 (unstandardized, p < 0.05). This indicates that out of the 8.6 points of decline in mathematics achievement, 5.6 of these were mediated by student selfconcept. In other words, the results indicate that students’ declining self concept is related to the declining achievement in mathematics from 2015 to 2019.
Also a safe environment declined significantly from 2015 to 2019, had a significant association with achievement, and mediated the relation between Time and achievement. The indirect effect for a safe environment is 3.24 (unstandardized, p < 0.05). This indicates that out of the 8.6 points of decline in mathematics achievement, 3.2 of these were mediated by a safe environment.
There were no significant indirect effects from the other variables. However, there is a clear pattern that all variables had negative indirect effects.
Discussion
To summarize, the key findings indicate that a safe environment and student selfconcept declined from 2015 to 2019 and mediated the changes in achievement over time.
School climate
The four aspects of school climate included in the analyses were school emphasis on academic success, school belonging and wellbeing, bullying, and safe environment. Apart from school emphasis on academic success, the other three aspects of school climate have been found to correlate in previous studies (OECD, 2019b) which is in line with our correlation matrix (see Table 2). All three of these aspects declined from 2015 to 2019. School belonging and wellbeing is a latent variable that includes students’ sense of safety at school, whether the teachers are fair, whether the students feel proud of their school, like being at their school, and feel a belonging to their school. The decline in this variable is in line with findings from a largescale national student survey that found that students’ wellbeing declined from 2016 to 2019 (Wendelborg et al., 2020). The decline could be related to the increased bullying (reported by students) and decrease of a safe environment (reported by teachers), especially considering the high correlation between bullying and wellbeing (see Table 2). Students’ sense of school belonging and wellbeing naturally decline in a less safe environment with increased bullying. These findings are also in line with findings from PISA (Jensen et al., 2019) and the aforementioned largescale national student survey (Wendelborg et al., 2020). It is further in line with the national TIMSS 2019 report, although the analyses used science achievement as the outcome (Nilsen & Kaarstein, 2021).
Of the four aspects of school climate, only a safe environment mediated the changes in achievement over time. However, there is a clear pattern for the school climate variables, where all the variables decrease and exhibit negative indirect effects, albeit not significant. The reason these other variables do not quite reach significance could be the relation to other variables. Including several predictors in a model is always risky, because if the predictors are highly correlated (i.e. multicollinearity), the significance of the relation between the predictor and outcome may be underestimated (Grewal et al., 2004). Such correlations could perhaps explain why modelling the school climate aspects one at a time, resulted in significant indirect effects (except for school emphasis), even after controlling for SES (see Additional file 1).
Whether a decline in a safe environment partly caused the declining achievements, remains an open question We found that a decline in a safe environment is related to the declining achievements. There could be omitted or confounding variables, and hence further research on this is required.
Student motivation and selfconcept
Students’ utility value and selfconcept in mathematics have declined from 2015 to 2019, while there was no significant change in their interestenjoyment value. Declining motivation in general is supported by the national largescale student survey (Elevundersøkelsen) from 2016 to 2019 (Wendelborg et al., 2020).
We did not investigate why students’ utility value and selfconcept declined, while there were no significant changes in student interestenjoyment value. It is positive that students still enjoy and like mathematics, but it is worrisome that they seem to value mathematics less and feel less confident in this subject. One explanation may be that the student composition has changed over time. Students participating in TIMSS 2019 report feeling hungrier at school, eating breakfast less often, and getting less sleep. At the same time, teachers report struggling more with students due to students increased lack of prerequisite knowledge or skills, a higher level of disruption of lessons, more uninterested students, and increased numbers of students with physical or mental disabilities (Kaarstein & Nilsen, 2021; Vik et al., 2022). Moreover, the number of students with minority background has increased extensively over the years (Lehre & Nilsen, 2021; SSB, 2021). Future studies are needed to investigate why utility value of mathematics and student selfconcept have decreased and whether this is related to student characteristics and home background.
When investigating interestenjoyment value, utility value, and selfconcept, only the latter was significantly related to mathematics achievement in Norway. This is in agreement with national findings from PISA (Kjærnslie & Olsen, 2013), showing the importance of selfconcept for mathematics achievement. The strong relation between selfconcept and achievement, together with the high correlation between selfconcept and interestenjoyment value and utility value, may explain why only selfconcept has a significant relation to achievement. Usually the values of interestenjoyment and of utility are related to achievement (e.g. Eccles & Wigfield, 2002) too.
The findings pertaining to the second research question indicate that student selfconcept not only declined, but also explained a substantial part of the declining mathematics performance from 2015 to 2019. Whenever there is a decline in a contextual variable like motivation and in achievement, it is not given that these changes are related to one another. It is hence interesting and important to survey whether these declines are in fact related, as this may provide indications on how to increase achievement. Increasing students’ selfconcept may result in increased achievement. While the relationship between student motivation and achievement is known to be reciprocal (Seaton et al., 2014), the fact that student motivation decreased and affected the decreased achievement, may indicate that motivation caused decreased achievement. However, it could also be that the declining achievements caused a decline in student motivation, and hence more evidence is needed to confirm this, preferably with longitudinal data.
A mediation would require that both the relation between time and predictor, and between predictor and achievement, are significant. Only selfconcept was related to achievement, and so this could explain why only selfconcept mediated the time changes in achievement. Previous research has found that selfconcept is more important to student achievement than utility value and interestenjoyment value (e.g. Kjærnslie & Olsen, 2013), and this could be the reason why only selfconcept (and not ulility value or interestenjoyment value) was related to the decline. More research is needed to further investigate why there only were declines in selfconcept and utility value and not in interestenjoyment value, and why only selfconcept was related to a decline in achievement.
Our findings are in line with findings from the national TIMSS report using science performance as the outcome, however, the utility value for science did not change from 2015 to 2019 (Nilsen & Kaarstein, 2021). Selfconcept on the other hand, declined in both mathematics and science, and could explain substantial parts of the declines in both subject domains. This supports the finding that students’ declining selfconcept could have contributed to the decline in achievement. However, it is impossible to prove a causal relation with crosssectional data that does not have the design of a randomized controlled trial, and so further research is required to confirm these findings.
Limitations
ILSAs have crosssectional designs, and hence no causal inferences may be drawn. However, utilizing the trend design with equal contextual scales and equal scaling of performance (mean = 500, SD = 100) across cycles, allows for more robust and valid inferences than those using crosssectional data from one time point (Gustafsson & Nilsen, 2022; Nilsen & Gustafsson, 2014). At the same time, including more timepoints in a trend analysis or using data from a true longitudinal study, would have increased the reliability and validity of the findings.
There could be other explanations for why Norway’s achievement declined from 2015 to 2019 that are not included in our analyses, such as students eating breakfast less often, and getting less sleep as discussed earlier. Furthermore, there could be confounding variables not included in the analyses that threatens causal inferences. Confounding variables is a general limitation by mediation analyses (MacKinnon et al., 2007). For instance, it could be that a decline in school safety leads to more depressed students, which again may lead to a decline in achievement. Here the confounding variable is depressed students. This was just an example, and further research would be needed to establish such causal links.
Another threat to causal inference besides that of confounding or omitted variables, are reciprocal relationships. It could for instance be that a decrease in student motivation and selfconcept leads to a decline in achievement. Our data cannot establish sufficient evidence for the direction of this relationship. RCT studies are needed to confirm evidence and establish causality.
The data obtained for measuring school climate, student motivation, and selfconcept are selfreported and may hence be biased. Such bias is most often problematic when comparing means across countries, since different response patterns and understandings of the concepts may result in incomparable measures (Van de Vijver & Tanzer, 1997). However, comparisons across two cycles, within the same culture or country, should reduce this bias.
Contributions, implications, and concluding remarks
A number of studies have found school climate, student motivation, and selfconcept to be related to achievement in ILSAs, internationally (e.g. Eccles & Wigfield, 2020; Eklöf, 2021) and in Norway (e.g. Kaarstein & Nilsen, 2016). However, few studies have investigated how school climate, student motivation, and selfconcept relate to changes in achievement. The study herein may contribute to research, both in terms of supporting findings from previous studies (e.g. Wang & Degol, 2015) and in terms of methodology; other researchers may find the method useful for investigating school factors related to change in performance. These types of models produce robust, reliable, and valid results (Gustafsson & Nilsen, 2022; Nilsen & Gustafson, 2014), something that was confirmed through model fit and the alignment between our findings and IRT analyses performed by TIMSS concerning decline in performance (Mullis et al., 2020).
Our study has shown that declines in aspects of school climate and student selfconcept were related to the declines in achievements from 2015 to 2019. While evidence of a declining achievement, school climate, student motivation and selfconcept in Norway is found in the literature, no study has examined whether the decline in achievement is related to a decline in school climate, student motivation and selfconcept. As such, the present study adds to the literature but also opens for further research. Future research is needed to investigate whether this is a causal relationship, or whether there may be omitted variables and reciprocal relations. Moreover, other variables than the ones we identified may also have contributed to the decline.
Our findings support previous research that policy and practice should take measures to ensure a good and safe school climate, and to improve students’ selfconcept, because these are prerequisites for learning (Gustafsson & Nilsen, 2016; Scherer & Nilsen, 2016; Wang & Degol, 2015).
Availability of data and materials
The data used for the study is openly avaiable at: https://timssandpirls.bc.edu/.
References
Bandura, A. (1997). Selfefficacy: The exercise of control. WH Freeman and Co.
Borgonovi, F. (2022). WellBeing in International LargeScale Assessments. In T. Nilsen, A. StancelPiątak, & J. E. Gustafsson (Eds.), International handbook of comparative largescale studies in education. Springer International Handbooks of Education.
Borgonovi, F. (2022). WellBeing in International LargeScale Assessments. In: Nilsen, T., StancelPiątak, A., Gustafsson, JE. (eds) International Handbook of Comparative LargeScale Studies in Education. Springer International Handbooks of Education. Springer, Cham. https://doi.org/10.1007/9783030382988_451
Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and selfdetermination in human behavior. Springer Science & Business Media.
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53(1), 109–132.
Eccles, J. S., & Wigfield, A. (2020). From expectancyvalue theory to situated expectancyvalue theory: A developmental, social cognitive, and sociocultural perspective on motivation. Contemporary Educational Psychology, 61, 101859.
Eklöf, H. (2021). Student motivation and selfbeliefs. In T. Nilsen, A. StancelPiątak, & J. E. Gustafsson (Eds.), International handbook of comparative largescale studies in education. Springer International Handbooks of Education.
Engel, L. C., Rutkowski, D., & Rutkowski, L. (2009). The harsher side of globalisation: violent conflict and academic achievement. Globalisation, Societies and Education, 7(4), 433–456.
Grazia, V., & Molinari, L. (2021). School climate multidimensionality and measurement: A systematic literature review. Research Papers in Education, 36(5), 561–587.
Grewal, R., Cote, J. A., & Baumgartner, H. (2004). Multicollinearity and measurement error in structural equation models: Implications for theory testing. Marketing Science, 23(4), 519–529.
Gustafsson, J. E., & Nilsen, T. (2016). The impact of school climate and teacher quality on mathematics achievement: a differenceindifferences approach. In T. Nilsen & J. E. Gustafsson (Eds.), Teacher Quality, Instructional Quality and Student Outcomes (pp. 81–95). Springer.
Gustafsson, J. E., & Nilsen, T. (2022). Methods of causal analysis with ILSA data. In T. Nilsen, A. StancelPiątak, & J. E. Gustafsson (Eds.), International handbook of comparative largescale studies in education. Springer International Handbooks of Education.
Havik, T., & Ertesvåg, S. (2020). Olweus programmet (OBPP) – et skoleomfattende program mot mobbing og antisosial atferd [The Olweus program a schoolwide program against bullying and antisocial behavior] (2nd edition.) Ungsinn. Retrieved from. https://ungsinn.no/post_tiltak_arkiv/olweusprogrammetobpp2utg/
Hox, J., & Roberts, J. K. (2011). Handbook of advanced multilevel analysis. Psychology Press.
Hoy, W. K., Tarter, C. J., & Hoy, A. W. (2006). Academic optimism of schools: A force for student achievement. American Educational Research Journal, 43(3), 425–446.
Jensen, F., & Nortvedt, G. A. (2013). Holdninger til matematikk. In M. Kjærnsli & R. V. Olsen (Eds.), Fortsatt en vei å gå. Norske elevers kompetanse i matematikk, naturfag og lesing i PISA 2012 (pp. 97–120). Universitetsforlaget.
Jensen, F., Pettersen, A., Frønes, T. S., Kjærnsli, M., Rohatgi, A., Eriksen, A., & Narvhus, E. K. (2019). PISA 2018. Norske elevers kompetanse i lesing, matematikk og naturfag. Oslo: Universitetsforlaget. ISBN 9788215037356.
Kincade, L., Cook, C., & Goerdt, A. (2020). Metaanalysis and common practice elements of universal approaches to improving studentteacher relationships. Review of Educational Research, 90(5), 710–748.
Kjærnslie, M., & Olsen, R. V. (Eds.). (2013). Still a way to go. Norwegian students’ competence in mathematics, science and reading in PISA 2012. [Fortsatt en vei gå. Norske elevers kompetanse i matematikk, naturfag og lesing i PISA 2012.]. Universitetsforlaget.
Kaarstein, H., & Nilsen, T. (2015). Motivasjon [Motivation]. In O. K. Bergem, H. Kaarstein, & T. Nilsen (Eds.), Vi kan lykkes i realfag‐Resultater og analyser fra TIMSS [We can succeed in science nad mathematics. Results and analyses from TIMSS] (pp. 6377). Oslo: Universitetsforlaget.
Kaarstein, H., & Nilsen, T. (2021). Lærerkompetanse, undervisningskvalitet og naturfagprestasjoner fra TIMSS 2015 til TIMSS 2019. In T. Nilsen & H. Kaarstein (Eds.), Med blikket mot naturfag: Nye analyser av TIMSS 2019data og trender (pp. 2015–2019). Universitetsforlaget.
Kaarstein, H., Radišić, J., Lehre, A. C. W., Nilsen, T., & Bergem, O. K. (2020). TIMSS 2919 Kortrapport [short report]. Universitetsforlaget.
LaRoche, S., Joncas, M., & Foy, P. (2020). Sample design in TIMSS 2019. In M. O. Martin, M. von Davier, & I. V. S. Mullis (Eds.), Methods and procedures: TIMSS 2019 technical report (pp. 31–333). TIMSS & PIRLS International Study Center.
Lehre, A.C.W., & Nilsen, T. (2021). Språk i hjemmet og naturfagprestasjoner fra TIMSS 2015 til TIMSS 2019. In T. Nilsen & H. Kaarstein (Eds.), Med blikket mot naturfag: Nye analyser av TIMSS 2019data og trender (pp. 2015–2019). Oslo: Universitetsforlaget.
Luyten, H., Merrell, C., & Tymms, P. (2017). The contribution of schooling to learning gains of pupils in Years 1 to 6. School Effectiveness and School Improvement, 28(3), 374–405.
Marsh, H. W., Hau, K.T., Artelt, C., Baumert, J., & Peschar, J. L. (2006). OECD’s brief selfreport measure of educational psychology’s most useful affective constructs: Crosscultural, psychometric comparisons across 25 countries. International Journal of Testing, 6(4), 311–360.
Marsh, H. W., Pekrun, R., Parker, P. D., Murayama, K., Guo, J., Dicke, T., & Arens, A. K. (2019). The murky distinction between selfconcept and selfefficacy: Beware of lurking jinglejangle fallacies. Journal of Educational Psychology, 111(2), 331–353. https://doi.org/10.1037/edu0000281
Martin, M. O., von Davier, M., & Mullis, I. V. S. (Eds.). (2020). Methods and Procedures: TIMSS 2019 Technical Report. Retrieved from Boston College, TIMSS & PIRLS International Study Center. https://timssandpirls.bc.edu/timss2019/methods.
MacKinnon, D. P., Fairchild, A. J., & Fritz, M. S. (2007). Mediation analysis. Annual Review of Psychology, 58, 593. https://doi.org/10.1146/annurev.psych.58.110405.085542
Mullis, I. V. S., & Martin, M. O. (2017). TIMSS 2019 assessment frameworks. UK: ERIC.
Mullis, I. V. S., Martin, M. O., Foy, P., & Arora, A. (2012). TIMSS 2011 international results in mathematics. ERIC.
Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2016a). TIMSS 2015 international results in mathematics. T. P. I. S. C. Boston College.
Mullis, I. V. S., Martin, M. O., & Hooper, M. (2016b). 20 years of TIMSS international trends in mathematics and science achievement, curriculum, and instruction. TIMSS & PIRLS International Study Cente, and IEA.
Mullis, I. V. S., Martin, M. O., Foy, P., Kelly, D. L., & Fishbein, B. (2020). TIMSS 2019 international results in mathematics and science. https://timssandpirls.bc.edu/timss2019/internationalresults
Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.
Muthén, L. K., & Muthén, B. O. (1998–2017). Mplus user´s guide, 8th edition. Muthén & Muthén.
Nilsen, T. (2016). Skolemiljø [School climate]. In O. K. Bergem, H. Kaarstein, & T. Nilsen (Eds.), V kan lykkes i realfag. Resultater og analyser fra TIMSS 2015 [We can succeed in mathematics and science. Results and analyses from TIMSS 2015]. Universitetsforlaget.
Nilsen, T., & Gustafsson, J.E. (2014). School emphasis on academic success: Exploring changes in science performance in Norway between 2007 and 2011 employing twolevel SEM. Educational Research and Evaluation, 20(4), 308–327.
Nilsen, T., & Kaarstein, H. (2021). Skolemiljø, motivasjon og naturfagprestasjoner fra TIMSS 2015 til TIMSS 2019 [School climate, motivation and student science performance from TIMSS 2015 to TIMSS 2019]. In T. Nilsen & H. Kaarstein (Eds.), Med blikket mot naturfag. Nye analyser av TIMSS 2019data og trender 2015–2019 [Focus on science. New analyses of TIMSS 2019 data and trends 2015–2019]. Universitetsforlag.
Nilsen, T., & Teig, N. (2022). A systematic review of studies investigating the relationships between school climate and student outcomes in TIMSS, PISA, and PIRLS. In T. Nilsen, A. StancelPiątak, & J. E. Gustafsson (Eds.), International handbook of comparative largescale studies in education. Springer International Handbooks of Education.
Nilsen, T., Bjørnsson, J. K., & Olsen, R. V. (2018). Hvordan har likeverd i norsk skole endret seg de siste 20 årene? [How has educational equality changed over the last two decades?] In Björnsson, J. K., & Olsen, R. V. (Eds.), Tjue år med TIMSS og PISA i Norge: Trender og nye analyser. [Twenty years of TIMSS and PISA: Trends and analyses]. Universitetsforlaget. ISSN 9788215030067. p. 150–172. https://doi.org/10.18261/9788215030067201808.
OECD. (2019a). PISA 2018 results (Volume II). Where all students can succeed. OECD.
OECD (2019b). PISA 2018 Results (Volume II): Where All Students Can Succeed. PISA, OECD Publishing, Paris. https://doi.org/10.1787/b5fd1b8fen.
Olsen, R. V., & Bjørnsson, J. K. (2018). Fødselsmåned og skoleprestasjoner [Birth month and performance]. Tjue år med TIMSS og PISA i Norge [Twenty years of TIMSS and PISA in Norway] (pp. 76–93). Universitetsforlaget.
Olweus, D. (Ed.). (2010). Understanding and researching bullying: Some critical issues. Taylor & Francis Group.
Regan, E., DeWitt, J. (2015). Attitudes, interest and factors influencing STEM enrolment behaviour: an overview of relevant literature. In: Henriksen, E., Dillon, J., Ryder, J. (eds) Understanding Student Participation and Choice in Science and Technology Education. Springer, Dordrecht. https://doi.org/10.1007/9789400777934_5.
Rutkowski, L., von Davier, M., & Rutkowski, D. (2013). Handbook of international largescale assessment: Background, technical issues, and methods of data analysis. CRC Press.
Scherer, R., & Nilsen, T. (2016). The relations among school climate, instructional quality, and achievement motivation in mathematics. In T. Nilsen & J. E. Gustafsson (Eds.), Teacher quality, instructional quality and student outcomes relationships across countries, cohorts and time (Vol. 2, pp. 51–79). IEA Springer.
Seaton, M., Parker, P., Marsh, H. W., Craven, R. G., & Yeung, A. S. (2014). The reciprocal relations between selfconcept, motivation and achievement: Juxtaposing academic selfconcept and achievement goal orientations for mathematics success. Educational Psychology, 34(1), 49–72.
Sivesind, K. (2013). Mixed images and merging semantics in European curricula. Journal of Curriculum Studies, 45(1), 52–66.
Skaalvik, E. M., Federici, R. A., & Klassen, R. M. (2015). Mathematics achievement and selfefficacy: Relations with motivation for mathematics. International Journal of Educational Research, 72, 129–136.
SSB. (2021). Statistikkbanken. Retrieved from https://www.ssb.no/statbank/table/10516/tableViewLayout1/
Thapa, A., Cohen, J., Guffey, S., & HigginsD’Alessandro, A. (2013). A review of school climate research. Review of Educational Research, 83(3), 357–385.
UDIR. (2022). Overordnet del. Grunnleggende ferdigheter. Retrieved from https://www.udir.no/lk20/overordnetdel/prinsipperforlaringutviklingogdanning/grunnleggendeferdigheter/?lang=nob
Van de Vijver, F., & Tanzer, N. (1997). Bias and equivalence in crosscultural assessment. European Review of Applied Psychology, 47(4), 263–279.
Vik, F. N., Nilsen, T., & Øverby, N. C. (2022). Aspects of nutritional deficits and cognitive outcomes–Triangulation across time and subject domains among students and teachers in TIMSS. International Journal of Educational Development, 89, 102553.
Wang, M.T., & Degol, J. L. (2015). School climate: A review of the construct, measurement, and impact on student outcomes. Educational Psychology Review. https://doi.org/10.1007/s1064801593191
Wendelborg, C., Dahl, T., Røe, M., & Buland, T. H. (2020). Elevundersøkelsen 2019: Analyse av Utdanningsdirektoratets brukerundersøkelser. Trondheim: NTNU Samfunnsforskning. ISBN 9788275706223.
Young, A. M., Wendel, P. J., Esson, J. M., & Plank, K. M. (2018). Motivational decline and recovery in higher education STEM courses. International Journal of Science Education, 40(9), 1016–1033. https://doi.org/10.1080/09500693.2018.1460773
Acknowledgements
We would like to thank JanEric Gustafsson for developing the methodology behind our analyses.
Funding
Not applicable.
Author information
Authors and Affiliations
Contributions
All three authors, TN, HK, and AL, contributed to the writing of the article. HK and AL prepared the data for the analyses, and TN did the analyses. TN, HK, and AL contributed to the interpretation of the data. All three authors, TN, HK, and AL have approved the submitted version, and agreed both to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, are appropriately investigated, resolved, and the resolution documented in the literature. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
All aspects of ethics related to the TIMSS study has been approved by the Directorate of Education.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Additional file 1.
Separate models.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Nilsen, T., Kaarstein, H. & Lehre, AC. Trend analyses of TIMSS 2015 and 2019: school factors related to declining performance in mathematics. Largescale Assess Educ 10, 15 (2022). https://doi.org/10.1186/s40536022001348
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s40536022001348
Keywords
 TIMSS
 School climate
 Interestenjoyment value
 Utility value
 Selfconcept
 Trend analyses
 Norway