The influence of ICT use and related attitudes on students’ math and science performance: multilevel analyses of the last decade’s PISA surveys

Abstract

This study analyzed the latest four PISA surveys, 2009, 2012, 2015, and 2018, to explore the association between students’ ICT-related use and math and science performance. Using ICT Engagement Theory as a theoretical framework and a three-level hierarchical linear modeling approach, while controlling for confounding effects, ICT-related independent variables of interest were added to the models at the student, school, and country levels. The series of models revealed that, in general, an increase in ICT availability and ICT use both inside and outside school had a negative association with learning outcomes, while students’ positive attitude toward ICT demonstrated a strong positive relationship. However, students’ perceived autonomy related to ICT use had the strongest association with academic performance, which is consistent with the changing nature of the modern learning environments. Findings revealed that virtually all forms of student ICT use, both inside and outside of school and whether subject related or not, had no substantive positive relationship with student performance in math or science. Conversely, higher student attitude toward, confidence in, belief in utility of, and autonomous use of ICT was associated with higher math and science performance for each of the four years of the study. Incidentally, we also found that while country GDP per capita had no consistent association with student performance, a school’s provision of extra-curricula activities did. Recommendations for educational leaders, teachers, and parents are offered.

Introduction

The use of Information and Communication Technology (ICT) have been a hot topic in education research since the beginning of the 1990s. ICT usage in vocational training, primary and secondary education is rapidly growing all around the world, but it remains unequally distributed across countries (OECD, 34). Schools are looking for new ways to integrate ICT skills into their policies and curriculum to foster the teaching and learning process in the context of “twenty-first-century skills” (Anderson, 1).

There is a rich research collection investigating the vital role that school ICT infrastructure and student ICT-related behaviour plays in students’ academic development, with much of this research country-specific, based on Programme for International Student Assessment (PISA) data, and focusing on only one cycle of PISA with some selection of variables (Biagi & Loi, 7; Bulut & Cutumisu, 9; Carrasco & Torrecilla, 9; Erdogdu & Erdogdu, 9; Hu et al., 21; Luu & Freeman, 24; Petko et al., 34; Wittwer et al., 51). We believe that researching the trends throughout the last decade of PISA cycles and making use of all key ICT-related variables can provide a more holistic picture of how school ICT infrastructure, ICT use and availability, and attitudes toward ICT is associated with academic performance over time. Therefore, the current study aims to explore the relationship between of ICT infrastructure, ICT use and availability inside and outside school, and students’ attitudes toward ICT and students’ math and science abilities measured in all the PISA surveys within the last decade (2009, 2012, 2015, and 2018).

Theoretical framework

This study uses Self-Determination Theory (SDT) to explain the associations between ICT-related variables and students’ academic performance. We bring together a set of environmental factors, individual differences, ICT use and availability inside and outside school, and attitudes toward ICT to explain the differences in students’ math and science performance. SDT asserts that self-motivation and determination are the main drivers of an individual’s learning (Deci & Ryan, 9). Competence (mastery and control over outcomes), relatedness (the drive to communicate with others), and autonomy (the desire to make their own choices) are the three basic facets in SDT used to explain mastery in learning (Deci & Ryan, 9). Based on SDT, Goldhammer et al. (9) introduced the ICT engagement concept with the dimensions of perceived autonomy related to ICT use, perceived ICT competence, ICT interest, and ICT as a topic in social interaction. Goldhammer et al. (9) assert that it is not only the use and availability of ICT inside and outside school but the underlying attitudes toward ICT that predict students’ academic achievement. Based on SDT, ICT Engagement Theory suggests that students’ interest, positive social interactions, autonomy, and competence related to ICT increase their intrinsic motivation, enabling them to challenge themselves with self-driven technology use, which can generate conditions conducive to optimal academic performance (Goldhammer et al., 9). Based on ICT Engagement Theory (Cristoph et al., 9; Kunina-Habenicht & Goldhammer, 24), student attitudes toward ICT were partially covered in the 2009 and 2012 cycles of PISA, while they were more closely reflected in the 2015 and 2018 cycles in the “ICT Familiarity Questionnaire” (OECD, 24).

In addition to our focus on student-related ICT variables, we also explore the role of background and ICT-related variables on student science and math performance. Though researched rarely (see Hu et al, 21), we explore the association between GDP per capita and Math and Science performance for each of the four cycles. Under SDT, it is important to consider the role of such contextual effects (Deci & Ryan, 9) and report on the results to educational stakeholders (Skyrabin et al., 34).

Considering that some schools can be considered digital frontrunners (Novak et al., 24) we are also interested in the role of school ICT infrastructure for the study period. Specifically, while controlling for important covariates (Zhang & Liu, 21) we look at the role of the number of available computers per student and the proportion of available computers connected to the internet in schools on the math and science performance of schools.

Students’ attitudes toward ICT

The empirical evidence suggests that students’ positive attitudes toward ICT are positively associated with their mathematics and science performance (Petko et al., 34; Tourón et al., 34). Areepattamannil and Santos (2) found that students who perceived themselves as autonomous and competent in ICT use develop positive views and feelings towards science, such as self-efficacy, enjoyment, and interest in science. Numerous studies have supported the notion that students’ mathematics and science achievement is associated with autonomous use of ICT (Hu et al., 21; Juhaňák et al., 9; Kunina-Habenicht & Goldhammer, 24; Meng et al., 21), interest in ICT use (Christoph et al., 9; Hu et al., 21; Koğar, 21; Kunina-Habenicht & Goldhammer, 24; Meng et al., 21), perceived self-confidence in ICT use (Guzeller & Akin, 9; Luu & Freeman, 24), and perceived self-competence in ICT use (Hu et al., 21; Koğar, 21; Kunina-Habenicht & Goldhammer, 24; Luu & Freeman, 24; Papanastasiou et al., 21; Srijamdee & Pholphirul, 44).

Although most of the studies reported positive relations between those attitudes and mathematics and science performance, Meng et al. (21) and Juhaňák et al. (9) reported controversial results for some of those attitudes. Meng et al. (21) found that the association between interest in ICT and mathematics and science performance was positive for the Chinese students while negative for the German students. Meng et al. (21) also reported a negative relationship between perceived self-competence and mathematics and science performance for the Chinese students, while there is no relation for the German students. In addition, Juhaňák et al. (9) found no associations between mathematics and science achievement with either interest in ICT or perceived self-competence (Czech students). On the other hand, most of the studies found negative relations between ICT use in social interaction and mathematics and science performance (Hu et al., 21; Juhaňák et al., 9; Meng et al., 21). Conversely, Martínez-Abad, Gamazo, and Rodríguez-Conde (9) reported positive associations between ICT use in social interaction and mathematics and science achievement on a sample of Spanish students. Given the conflicting results pertaining to students’ attitude toward ICT and academic performance, more substantial research in this area is in order.

ICT use and availability inside and outside of school

Research has suggested that ICT can add value to the learning process (UNESCO, 44). ICT use in educational settings with academic purposes has been shown to be useful in improving students’ performance in science (Erdogdu & Erdogdu, 9; Luu & Freeman, 24; Skryabin et al., 34) and mathematics (Carrasco & Torrecilla, 9; Erdogdu & Erdogdu, 9; Skryabin et al., 34).

The research on the impact of technology on learning outcomes, especially in mathematics and science, revealed the importance of technology use in education (Luu & Freeman, 24; Rutten et al., 24; Tamim et al., 21; Wittwer & Senkbeil, 51). Further, several meta-analysis studies suggested that ICT use in education has a small but positive impact on student performance (Bayraktar, 5; Cheung & Slavin, 9; Torgerson & Zhu, 51). However, a substantive number of research studies using large-scale international databases investigated how forms of ICT availability, use, and engagement has a positive association with student performance in mathematics and science (i.e., databases such as PISA, the Trends in International Mathematics and Science Study, TIMSS; and the Progress in International Reading Literacy Study, PIRLS). Importantly, the majority of these studies suggested that increased use of ICT at school had a negative association with mathematics and science performance (Bulut & Cutumisu, 9; Erdogdu & Erdogdu, 9; Hu et al., 21; Petko et al., 34; Skryabin et al., 34; Wittwer & Senkbeil, 51). The summary of the findings of a number of these key studies is now provided.

Using the PISA 2012 data, Petko et al. (34) investigated the role of the frequency of educational technology use on student achievement. They found that while ICT use at home for school purposes had a positive relationship with achievement, ICT use for entertainment purposes and the magnitude of use at school had a negative relationship with achievement. They also found that students’ positive attitudes towards educational technology were associated with higher test scores in most countries. They concluded that the moderate use of educational technology could be related to higher achievement, though both low and intensive use of educational technology in school appears to have a negative association. To explain this finding, the authors inferred that students’ lower academic achievement could be the result of ineffective pedagogy while they used technology and low-quality educational software that is used in the teaching process. However, these results of the study were not conclusive and there were a limited number of control variables used in the analysis.

Skryabin et al. (34) investigated how country-level ICT development and individual ICT usage was related to 4th- and 8th-grade student achievement in reading, mathematics, and science based on the data from TIMSS 2011, PIRLS 2011, and PISA 2012. The analysis revealed that country-level ICT development was a significant positive predictor for individual academic performance in all three subjects for both 4th- and 8th-grade students. After controlling for students’ gender and socioeconomic status, they found that country-level ICT development and student ICT use at home had a positive relationship with students’ academic performance; however, the ICT rate of change (measured by country’s recent shift in the ICT development index; International Telecomunications Union, 2012) had a negative association with students’ academic performance, but this link was not always significant for all subjects.

Early research by Wittwer and Senkbeil (51) investigated the role of using computers at home and school on student academic performance (based on PISA 2003 data). Their results suggested that, for the majority of students, the use of the computer at home or at school had no substantial influence on their academic achievement. However, more recently, Hu et al. (21) conducted research on how national ICT skills affected students’ academic performance (using PISA 2015 data). They found that ICT skills had a positive relationship with student academic performance and that ICT availability at school also had a positive relationship with students’ academic performance. In addition, the researchers found that student use of ICT for academic purposes had a positive relationship with student performance, whereas student use of ICT for entertainment purposes had a negative relationship. However, the study did not control for school SES and only focused on one year so could not draw conclusions across multiple cycles.

Wainer et al. (56) analysed the 2001 Brazilian Basic Education Evaluation System (SAEB) achievement exam for 4th-, 8th-, and 11th-grade students in mathematics and reading (Portuguese). The results suggested that the frequency of computer use had (1) a negative association with test results, and (2) a particularly high negative association with the test results of younger and lower-ability students. The researchers also identified that having internet access had a negative relationship with the academic performance of younger students, whilst this relationship was positive for older students. More recent research has explored the association between internet availability at school and home and student academic performance. Erdogdu and Erdogdu (9) explored the associations between access to ICT, student background, and school/home environment and students’ academic performance based on PISA 2012 data. While controlling for parental education level and socio-economic conditions (e.g., students’ having their own room), findings suggested that internet availability at home and at school was positively associated with students’ academic performance. Though, the specific relationships between availability and types of ICT use across the last decade have yet to be explored. Moreover, the nuanced associations between outside-of-school ICT use for leisure and social interaction for all countries has yet to be examined comprehensively in the literature.

Carrasco and Torrecilla (9), drawing upon PISA 2006 data, researched how computer access and use affected students’ academic performance. They found that computer access and use had a positive association with student performance. The researchers found that having a computer at home had a significant positive association with students’ reading and mathematics performance. Furthermore, Bulut, and Cutumisu (9) examined whether the use and availability of ICT at home and school was related to students’ academic success in the PISA 2012 mathematics and science-based assessments in Finland and Turkey. In both countries, they found that the use of ICT for mathematics lessons had a negative association with mathematics success; however, the general use of ICT at school had no substantive relationship with student performance in both mathematics and science. Finally, findings suggested that the use of ICT for entertainment had a positive association with students’ academic successes in Turkey while at the same time a negative association with students’ academic performances in Finland. Though, nuanced relationships for outside-of-school ICT use for leisure and social interaction for all countries has yet to be examined comprehensively. In another related study, Luu and Freeman (24) analysed the relationship between ICT use and scientific literacy across Canada and Australia based on PISA 2006 data. Their results suggested that students who browse the Internet more frequently and those who were more confident with basic ICT tasks earned higher scientific literacy scores. Though, more recent work in this area appears to be lacking.

Controversial findings in the associations between ICT related variables and students’ academic performance may have stemmed from the variety of PISA results across different nationalities, cycles, subjects, the combination of the variables chosen by the researchers, or the statistical approaches adopted by the researchers. For instance, ICT availability and use had a positive relationship with mathematics and science performance of Turkish students in PISA 2012 data, while it has either negative or no association with the performance of the Finnish student sample in the same study Bulut & Cutumisu 10). Similarly, using PISA 2015 data, Meng et al. (21) observed negative associations between mathematics and science performance of the Chinese and German students and their self-competence and interest in ICT, as opposed to the PISA results of the other countries. Juhaňák et al. (9) and Luu and Freeman (24) took into account the moderation effect of the frequency of ICT use on academic performance and found divergent results regarding the subgroups of students who used ICT at low, moderate, and high levels. They found that very low and very high usage of ICT had a negative association with academic performance. Biagi and Loi (7) found a positive association between ICT use for gaming and students’ academic performance. Petko et al. (34) argued that the controversial positive relationship could have resulted from an artefact of the method of analysis that Biagi and Loi (7) used. Rodrigues and Biagi’s (21) findings varied substantially by the combinations of type of school, frequency of ICT use, and ESCS (student economic, social, and cultural status) regarding the subgroups of the chosen variables. Through the econometric specification method they adopted, they regressed the students' performance on the different frequencies of ICT uses, while controlling for other variables that could be simultaneously associated with the dependent and independent variables. They found that low-frequency ICT users with mid to high ESCS benefit the most from an increased ICT use at school. They also reported that the positive association between ICT use at home for schoolwork and students’ science performance is stronger than those with low ESCS in private schools. In the current study, rather than comparing specific countries or testing any moderation or mediation effects, we use all the ICT related predictors of students’ mathematics and science performance using the complete data sets from the latest four PISA cycles to provide a comprehensive view of the subject matter.

The rationale for the current study

There is an increasing trend in the amount of research based on PISA data with interest in ICT skills and how these skills affect our students’ performances and other related constructs. Based on the rich research evidence, it was evident that ICT use can have a positive (small to moderate generally) association with students’ academic successes whilst it does depend on students’ purpose of using ICT, attitudes toward ICT, and the availability of ICT at both home and school.

Of the PISA studies reviewed, common independent variables pertained to ICT availability and use at school and home, with the strength of relationship between these variables and student academic performance sometimes dependent on the student sample and year of study. To date, little research has focused on the role of student competence in, attitude towards, interest in, and autonomous use of ICT. Moreover, to date, many studies have focussed on examining the role of ICT using a single PISA (and other single cycle large-scale assessment datasets such as TIMSS) and by taking a limited number of covariates into account (Luu & Freeman, 24; Erdogdu & Erdogdu, 9; Meng et al., 21; Odell, Galovan, & Cutumisu, 34; for TIMSS and PIRLS, see, for example, Grilli et al., 9). It should be noted that the cross-sectional nature of the PISA surveys makes longitudinal research impossible: i.e., the same cohort of students are not tracked longitudinally across time. However, for each cycle, attempts are made to ensure that samples are representative of the student group of interest, 15-year-olds, and questions pertaining to ICT are repeated opening the possibility for reasonable comparisons to be made across administrations.

We could only identify one example of research that focussed on five cycles of PISA. Zhang and Liu (21) investigated the role of ICT use on student performances for PISA cycles spanning 2000 to 2012. Research based on multiple PISA cycles over time provides a more holistic approach to highlighting and identifying the general situation of ICT use and attitude and its role in student learning. Therefore, the current research focuses on the last decade on PISA administrations and makes use of all ICT-related variables. Therefore, this study aims to explore the relationship between (1) ICT use and ICT related attitudes and (2) students’ math and science abilities measured in all the PISA surveys within the last decade. Besides, this study accounts for a wide range of covariates while undertaking the analyses at the student, school, and country levels. This was done to adjust for the confounding of associations of variables possibly related to both ICT-related use and students’ math and science performance. To note, Zhang and Liu (21) analysed the PISA surveys between 2000 and 2012 with a similar research question. However, in the 2015 and 2018 PISA cycles, several essential variables were added to the ICT surveys. To this point, in their scoping review, Odell, Galovan, and Cutumisu (34) noted that ICT as a topic in social interactions, interest in ICT, and autonomy in using ICT—variables added to the ICT survey in the latest two cycles—have been less studied concepts in the relevant literature. The current study makes further use of data from these two more recent cycles with the intention to provide updated and more comprehensive insights into the role of ICT use on student academic performance. Accordingly, the following three research questions are proposed for the current study:

RQ1: Can reasonable comparisons between ICT-related variables and control variables be made year-to-year for PISA 2009, 2012, 2015, and 2018? If not, what type of variable transformations might be usefully be applied to ensure this?

RQ2: What proportion of the variance in Math and Science can be attributed to within-school, between-school, and between-country effects?

RQ3: While controlling for student-, school-, and country-level confounding factors, what forms of student ICT-related attitude, accessibility, and school ICT-related infrastructure are associated with student performance in PISA Science and Math across PISA cycles?

Methodology

Participants

The data for the current study was compiled from the previous four PISA cycles, which were made available from the OECD website. PISA is an international survey that has been conducted every three years since 2000. PISA aims to assess 15-year-old students’ science, math, and reading achievement scores, their various attitudes, behaviors, demographics, and other relevant contextual data from their parents and schools. For each of the four cycles, 2009, 2012, 2015, and 2018, both student and school data were merged. Each country had the option to have students and schools complete questions that measured the student- and school-level utility of, familiarity with, and attitude toward ICT. Because this survey was not obligatory, different numbers of countries opted to be involved in the ICT survey year-to-year. Accounting for this missing data, and after removing schools with fewer than ten students (Lai & Kwok, 9), total student sample sizes across the four cycles amounted to 247,352, 243.060, 194,399, and 212,652, respectively. The total number of schools was 9,123, 9,923, 7,726, and 8,261, respectively, while the total number of countries was 44, 43, 45, and 49, respectively. On average, there were 27.1, 24.5, 25.2, and 25.7 students in each school, respectively; and an average of 207.3, 230.8, 171.7, and 168.6 schools were sampled from each country, respectively.

Variables

In this study, a series of three-level models were used to examine the relationship between ICT-related variables and students’ academic performance. The plausible values of students’ math and science achievement scores were used as dependent variables in the models. The control and independent variables used at the country, school, and student levels are described below.

Country-level variables

There are inequalities in computer and internet use between countries, and this has been found to be related to countries’ socio-economic characteristics (Montagnier & Wirthmann, 24). As a prominent indicator of a country’s socio-economic level, each country’s GDP per capita score was taken from World Bank (34) and included in the model as an independent variable at the country-level. Therefore, in the current study, GDP per capita was considered an important independent variable of interest.

School-level variables

School-level ICT development indices were used as independent variables and several educational variables related to school infrastructure were also included as control variables at the between-school level.

For school-level ICT development, we included the ratio of available computers per student at modal grade (RATCMP1 in 2015 and 2018; RATCMP15 in 2012; IRATCMP in 2009), and the proportion of available computers that are connected to the Internet (RATCMP2 in 2015 and 2018; COMPWEB in 2009 and 2012; 0 = no computers in school online, 1 = all computers in school online).

We included the following six control variables: (1) “Shortage of educational material” (EDUSHORT in 2018 and 2015), (2) “Quality of educational resources” (SCMATEDU in 2012 and 2009), (3) School-level economic, social, and cultural status (ESCS) (aggregated from students’ ESCS scores), (4) School type (SCHLTYPE; 1 = Private; 2 = Public), (5) Creative extra-curricular activities (EXCURACT in 2009; CREACTIV in 2012, 2015, 2018, and (6) Shortage of educational staff (STAFFSHORT in 2015 and 2018; TCSHORT in 2009 and 2012).

Student-level variables

Like at the school-level, multiple independent and control variables of interest were included in all models.

For control variables, students’ economic, social, and cultural status (ESCS) and gender (1 = female; 2 = male) were used. ESCS is a composite score computed by three indices (OECD, 24): home possessions including books at home (HOMEPOS), highest parental education (PARED), and highest parental occupation (HISEI).

The independent variables related to ICT use can be classified into three categories: ICT use outside school, ICT use in school, and students’ attitudes toward ICT.

ICT availability at home (ICTHOME in all cycles), ICT use outside of school [leisure] (ENTUSE in all cycles), use of ICT outside of school [for schoolwork activities] (HOMESCH in all cycles), subject-related ICT use outside of lessons (ICTOUTSIDE in only 2018 PISA), and ICT as a topic in social interaction (SOIAICT in only 2015 and 2018 cycles) were the variables related to “ICT use outside school.”

ICT availability at school (ICTSCH in all cycles), use of ICT at school in general (USESCH in all cycles), and subject-related ICT use during lessons (ICTCLASS only in 2018 PISA) were the variables related to “ICT use in school.”

Self-confidence in ICT high-level tasks (HIGHCONF only in 2009 PISA), attitude towards computers (ATTCOMP only in 2009 PISA), limitations of a computer as a tool for school learning (ICTATTNEG only in 2012 PISA), attitudes towards computer as a tool for school learning (ICTATTPOS only in 2012 PISA), interest in ICT (INTICT only in 2015 and 2018 cycles), perceived ICT competence (COMPICT only in 2015 and 2018 cycles), and perceived autonomy related to ICT use (AUTICT only in 2015 and 2018 cycles) were the variables pertaining to “studies attitudes toward ICI”¹

Table 1 Means and Standard Deviations of the Sample Data

Data adjustments

Dichotomous variables were dummy coded as follows: school type (SCHLTYPE: private = 1, public = 2) and student gender (GENDER: female = 1, male = 2). The variance for (1) GDP per capita, (2) the ratio of computers to students (RATCMP1), ICT available at home (ICTHOME), and ICT available in school (ICTSCH) was not consistent across the four cycles. For this reason, these variables were each also standardized prior to MLM analyses (see Table 1). In addition, the variable specifying the proportion of computers connected to the Internet (COMPWEB: none = 0, all = 1) was highly negatively skewed each cycle, so normalization procedures were undertaken in accordance with Courtney and Chang (9) (see Table 1 for selected descriptive statistics) prior to analysis. Decisions concerning the centering of predictor variables were made in accordance with Brincks et al. (8) and Lüdtke et al., 21). Specifically, we group mean center variables at the individual or school level when (1) student perception of the school environment was measured (e.g., perceived ICT use and availability inside schools, and (2) in the special case when the predictor has been computed by averaging the responses for all cases in each group (herein, ESCS). Further, because the school-level variables, STAFFSHORT, SCMATEDU, and EDUSHORT pertain to school principal perception (likely bound by comparative in-country perceptions), country-mean centering was applied to these variables.

To note, it was decided that the coefficients reported in the final linear mixed-effects models would be unstandardized. This decision was made so that the size of the coefficients would reflect the commonly understood metric in PISA, i.e., with the mean of approximately 500 and SDs of 100. While this is not exactly the case (see Table 1, means of all PVs), means and standard deviations are approximately the same. It should also be noted that the Supplementary Materials (Additional file 1: Table A1) provide definitions for each of the variables included in the study.

Use of sample weights

To ensure that each of the participating countries made an equal contribution to the study and to make the results of the study more generalizable internationally, we decided to make use of “senate weights” for all models. Because of missing data, the resultant sum of all student senate weights did not reach 5000. Therefore, the student senate weights for each country were multiplied by a constant such that the resultant sum of all student senate weight for the respective country came to 5000. The constant for each country was estimated in accordance with Eq. 1:

$$Country\,Senate\,Weight\,Constant= \frac{5000}{\sum_{i=1}^{N}SENW{T}_{i}}$$

(1)

where N is the total number of students included in the final analyses for each country after accounting for missing data.

Analysis

The analysis was undertaken with the assistance of the open-source software, R (R Core Team, 44). The means and standard deviations for all variables are reported based on the observed sample data. The null and linear mixed effects modes made use of the lme4 (linear mixed-effects) package (Bates et al., 4) and lmer function. Analyses accounted for the three-level hierarchical structure of the data with students nested schools and schools nested in countries. All multilevel modeling analyses incorporated normalized weights so that the contribution from each of the countries in the analysis could be considered equal, regardless of their population or sample size (for PISA 2009, W_FSTUWT; for 2012, SENWGT_STU; for 2015 and 2018, SENWT were used). This way results of the study could be considered applicable to all participating countries. For each cycle, an initial exploration of the intra-class correlations (ICCs) for students’ Math and Science was followed by analyses of the aforementioned country-, school-, and student-level variables as fixed effects.

In accordance with Wu (44), analyses for each year and associated subjects were run with all available plausible values (PV1-5 for 2009–2012, and PV1-10 for 2015–2018). After implementing optimization algorithms in accordance with Nash and Ravi (34) and Bates et al. (3), all models converged successfully. All models used the maximum likelihood (ML) estimation.

Based on these results, mean coefficients, t values, and p values for each year-subject combination were then calculated for the models for all four years. With the trend toward more strict assessments of statistical significance (Benjamin et al., 6), and the large sample sizes associated with the PISA studies, a threshold of p < 0.001 and b = 2.00 (unstandardized shift in achievement/scale scores) was deemed as substantive at the student and school levels, while a threshold of p < 0.05 was deemed of interest at the country level. Given the inclusion of multiple control variables in the models, we set the minimum association at 2 scale score points, though recognize that other researcher may propose different substantive limits depending on their study.

Results

RQ1 asks whether or not reasonable comparisons between ICT-related variables and control variables can be made year-to-year. Results suggest that, after standardizing the three variables, namely RATCMP1, ICTHOME, and ICTSCH, the variance in each variable does not change substantially year-to-year. Therefore, it is argued that reasonable comparisons can be made across the four administrations (see Table 1).

RQ2 asks what proportion of the variance in Math and Science can be attributed to within-school, between-school, and between-country effects. The null models were run for both math and science achievement scores, using the available plausible values for each analysis. This was done to examine the extent to which student achievement differed significantly between schools and countries. Table 2 shows the intercepts, residuals, and intraclass correlation coefficients (ICCs) at school and country levels. For Math, country-level ICCs were quite stable across all cycles ranging from 0.199 to 0.214, while school-level ICCs dropped from levels slightly higher than 0.300 to slightly higher than 0.200 for the latter two cycles. Similarly, for science, country-level ICCs were quite stable with values ranging from 0.155 to 0.183 across all cycles while school-level ICCs dropped from approximately 0.300 and 0.310 in the first two cycles to approximately 0.220 and 0.210 for the last two cycles.

Table 2 Intraclass correlation coefficients for math and science from 2009 to 2018

RQ3 asks, what forms of student ICT-related attitude, accessibility, and school ICT-related infrastructure are associated with student performance in PISA Science and Math across PISA cycles. After establishing substantive school- and country-level effects in RQ2, a series of three-level linear mixed-effect models, inclusive of the independent variables at the student-, school-, and country-levels, were run. A review of the final models in Tables 3 and 4 reveal that the independent variables explained up to 9.5% of the residual variance at the student-level, 61.8% of the residual variance at the school-level, and 34.8% of the residual variance at the country-level variance.

Table 3 Mixed Effects Model for Students’ Mathematics Ability

Table 4 Mixed Effects Model for Students’ Science Ability

At the country-level, the GDP per capita of the countries involved in the ICT survey only had a statistically significant relationship with math and science achievement in 2012 (b = 13.21, p < 0.01; b = 10.98, respectively, p < 0.05).

For both mathematics and science performance, results revealed that the overall variance explained at the school level tended to increase in the last two cycles (2015 and 2018). For math, variance explained at the school level grew from 50.7% in 2009 to 60.9% in 2018. Similarly, for science, variance explained grew from 50.5% in 2009 to 61.8% in 2018, with the variance explained due to school type appearing to become more substantive for both subjects.

Results revealed that the overall variance explained in science performance, at the within-school (student) level, increased from 6.5% in 2009 to 9.5% in 2018. It appears that the inclusion of variables in 2015 and 2018 pertaining to students’ perceived interest, competence, and autonomy in ICT provided substantive explanatory power for science performance. However, in comparison, the level of variance explained for math performance remained more constant across PISA cycles.

Tables 3 and 4 reveal that the direction of the relationships between the variables and the coefficients at all three levels appear to be quite consistent over the four cycles for both math and science ability. At the student level, two covariates, ESCS, and gender have strong positive association with both math and science achievement across all cycles. The ESCS effect indicates that the economic, social, and cultural advantages have a substantial relationship with students’ math and science achievement levels. The results for the gender variable means that, with females as the reference group, males have higher academic performance consistently across all the cycles.

ICT availability both at home and at school, and ICT use both inside and outside school—no matter the purpose of the students; for general, leisure, for schoolwork activities, or social interaction—was virtually always associated with either neutral or lower math and science performance for all cycles (with the single exception being Science, 2018, “ICT use outside of school, leisure”). For student use of ICT outside of school, substantive associations (b > 2.00; p < 0.001) were quite consistently negative across all cycles with no instances of substantive positive associations. Similarly, for ICT use inside school, relationships were generally negative or neutral with no substantive positive relationships for either students’ math or science performance.

Students’ positive attitudes and beliefs toward ICT use have a substantive positive relationship with both their math and science performance for all cycles. The findings indicate that the more successful students have higher self-confidence in ICT high-level tasks, have more positive attitudes towards computers, more strongly believe in the usefulness of computers as a tool for school learning, are more interested in ICT, and perceive themselves more competent and autonomous in ICT use. In 2009 PISA, self-confidence in ICT had the highest relationship (b_math = 5.94, b_science = 6.44, p < 0.001) followed by positive attitudes toward computers (b_math = 5.35, b_science = 5.25, p < 0.001).

In the 2012 cycle, “attitudes towards computers: limitations of the computer as a tool for school learning” had the largest ICT-related relationship (b_math = -10.30, b_science = -11.82, p < 0.001). The scale measured the degree to which students “think that using computers for learning is troublesome and using the internet resources as a learning tool is not useful and suitable”, and this variable appeared to be associated with lower math and science performance. Conversely, this result also somewhat suggested that those “who believe that computers and Internet are useful tools for school learning” have higher achievement scores (2012; b_math = 2.07, b_science = 4.26, p < 0.001).

In the 2015 and 2018 cycles, students’ perceived autonomy had the strongest association with academic performance (b_math(2015) = 9.43, b_math(2018) = 8.93, b_{science(2015)} = 11.90, b_{science(2018)} = 10.20, p < 0.001), reflecting the changing nature of the current educational settings in the way that students are more inclined to exert influence over their learning environments in order to increase their knowledge and abilities (Pellegrino, 24). Autonomy was followed by students’ interest in ICT (b_math(2015) = 2.82, b_math(2018) = 3.65, b_{science(2015)} = 3.72, b_{science(2018)} = 5.06, p < 0.001) and their perceived ICT competence (b_math(2015) = 2.30, b_math(2018) = 2.63, p < 0.01; and b_{science(2015)} = 2.71, b_{science(2018)} = 3.86, p < 0.001).

In terms of ICT infrastructure, the number of available computers per student in the school appeared to have no substantive association with math and science performance for any cycle. However, the proportion of available computers connected to the net appeared to have generally positive associations (see exception for 2015, Math) with math and science performance (b_math(2009) = 3.85, p < 0.001; b_math(2015) = − 1.22, p < 0.05; b_math(2018) = 2.96, p < 0.05; b_{science(2009)} = 5.39, b_{science(2018)} = 4.26, p < 0.001).

Incidentally, and as expected, at the school level, ESCS maintained the most substantive confounding association with student math and science performance for all cycles with coefficients for math between 63.67 to 74.61 (p < 0.001) and coefficients for science ranging between 66.80 to 71.98 (p < 0.001). Also incidentally, it is noted that a schools’ level of provision of extra-curricular activities appears to have an substantive, consistent, and positive associations with student math and science performance for all cycles. Finally, parenthetically, after accounting for the role of school socio-economic advantage and provision of extra-curricular activities, counterintuitively, school designation as a public institution appears to afford an advantage.

Discussion

This study aimed to explore the role of student engagement with ICT technologies and the role of school ICT infrastructure on students’ math and science abilities for the last four PISA cycles (2009, 2012, 2015, and 2018). The results of this study drew upon multiple ICT-related PISA variables to provide insights into the changing role of ICT infrastructure and behavior on students’ academic performances in mathematics and science. Although studies using the PISA data from different countries revealed different patterns of relationships between ICT related variables and students’ math and science performance (Odell, Galovan, & Cutumisu, 34), the current study provides an overall view, taking all the participating countries into account across all PISA cycles spanning the last decade.

Country- and school- level effects

At the country-level, GDP per capita of the countries involved in the ICT survey only had an association with math and science achievement scores in 2012. Although the country-level ICCs suggested substantial differences in science and math achievement in the current study, GDP could not consistently explain the achievement gap between countries. Another variable, such as the “national ICT development level” that was not included in this study, could have provided some explanatory power for the achievement gap between countries, as explored by Skryabin et al. (34). As the number of participating countries increase in international large-scale assessment studies, more extensive work could be undertaken in this area.

At the school levels, counterintuitively, the number of available computers per student appeared to have no substantive association with school-level math and science performance for any cycle. This result concurs with early PISA studies on the topic. For example, Fuch and Ludger (9) found that, after controlling for family background and general school infrastructure, the availability of computers at schools had no statistically significant association with student academic performance. The authors posit that the relationship between school access to computers and performance may be more U-shaped. Therefore, more specific research into possible non-linear relationship is certainly in order here. However, the proportion of available school computers connected to the internet did have an expected positive relationship for both math and science in 2009 and 2018. Therefore school connectivity may be important, though this is not conclusive. Certainly, further international research into the role of school internet speed and student accessibility to websites (not necessarily used for learning) beyond simple proportion of computers connected should be explored in the future so to provide more pertinent insight of the digital divide in schools internationally (for a discussion, see Valadez & Duran, 21).

Within-school effects of ICT use and availability

In this section the current findings associated with the student-level effects of ICT use both (1) outside of school lessons, and (2) in school are discussed in contrast with the research literature. For convenience, the discussion is provided in the order of fixed effects presented in the Tables 3 and 4.

In terms of within-school effects, there is a negative association between ICT availability at home and students’ math and science performance, as supported by previous studies (Hu et al., 21; Juhaňák et al., 9; Tan & Hew, 24). While some studies found a positive association between these two variables (Delen & Bulut, 9; Papanastasiou et al., 21; Srijamdee & Pholphirul, 44), others such as Juhaňák et al. (9) suggested no association. Also to note is that Bulut and Cutumisu (10) found positive relations for Turkish students but no relations for Finnish students. Considered broadly, the results here call into question the utility of unlimited availability of ICT materials at home and the possibility of distractive effects. It appears that unrestrained home access may have substantive detrimental relationship with adolescent academic learning.

ICT use outside of school for entertainment is associated with lower math and science performance in the current study, which is in line with the previous research findings (Bulut & Cutumisu, 10, for Finnish students; Petko et al., 34; Skryabin et al., 34, for math only; Juhaňák et al., 9; Luu & Freeman, 24; Rodrigues & Biagi, 21, high-intensity users; Kunina-Habenicht & Goldhammer, 24). Therefore, these findings in the current study support the idea that, the frequency of use of ICT for entertainment, though outside of school, can place students at a disadvantage academically when student performance is contrasted with counterparts inside schools.

ICT use outside of school for schoolwork activities is negatively associated with math and science achievement in the current study, corroborating the findings of the previous studies (Carrasco & Torrecilla, 9; Skryabin et al., 34; Rodrigues & Biagi, 21, medium and high users; Kunina-Habenicht & Goldhammer, 24; Hu et al., 21; Petko et al., 34, only for science; Juhaňák et al., 9, only for science). The findings here are somewhat troublesome given that the focus here is students’ frequency of computer use at home for school-related purposes. While counter-intuitive, it may be that use of such devices may involve a higher potential for distraction for the study period—the potential for distraction for which adolescent students may not manage well. However, we note that these effects are generally quite small (b_math(2012) = − 0.40, p < 0.01; b_sci(2009) = − 5.69, p < 0.001) so further research is needed on this topic.

ICT as a topic in social interaction is also negatively associated with student math and science performance, further confirming previous findings (Carrasco & Torrecilla, 9; Rodrigues & Biagi, 21; Skryabin et al., 34). This finding comes as no surprise given that the index reflects the level of ICT use for interpersonal communication.

Finally, in terms of ICT-use outside of school lessons, students subject-related ICT use outside of lessons, defined as the extent to which students use UCT for specific subject-related tasks was also negatively associated with academic performance. This pattern is also revealing and confronting as even student ICT use focused on school work appears to also have a detrimental association with academic performance.

At this juncture, we turn to the role of ICT use in school itself for the four PISA cycles.

Findings in this study also reveal a negative association between ICT availability at school and students’ math and science performance, as supported by research by Koğar (21). Therefore, overall, and for the age-group of interest, ICT availability at school, akin to that at home, may also have a prominent distracting effect. Therefore, consistent negative associations for home and school use for both math and science across all PISA cycles may reveal the need to manage and constrain adolescent engagement with ICT devices and content.

ICT use at school, both in general and subject-related use during lessons, was associated with lower math and science performance for all cycles, confirming the results of previous studies (Erdogdu & Erdogdu, 9; Hu et al., 21; Juhaňák et al., 9; Luu & Freeman, 24; Petko et al., 34; Skryabin et al., 34; Bulut & Cutumisu, 10; Kunina-Habenicht et al., 24). Given the results above pertaining to ICT use and availability at school, it is understandable that involvement in ICT tasks at school might also be disruptive to student learning and development. However, here, year-by-year confirmation that student subject-related use is also associated with poor academic performance is quite confronting. This suggests that the integration of ICT for classroom activities may be associated with more damage than good.

Odell, Cutumisu, & Gierl (21) concluded in their scoping review of the secondary analyses of the PISA data that moderate use of ICT, rather than high or no use of it, may be positively associated with students’ math and science performance. However, our research here points to the consistent finding that ICT availability both at home and at school and ICT use both inside and outside school may be distractive for most students, decreasing their achievement levels. Even if they make use of ICT at school for subject-related purposes, it might be distractive and reduce their academic performance in science and math, subjects requiring focus and concentration to improve (Hu et al., 21). One explanation for this may be provided by Kunina-Habenicht and Goldhammer (24) who argue that more frequent use of ICT at school can be linked with remedial purposes for lower-performing students. Rodrigues and Biagi’s (21) findings are supportive of this contention by pointing out that high performers in math and science are the ones who use ICT at lower levels inside and outside school while the low performers are the ones who use ICT from medium to high levels.

Within-school effects of attitudes toward ICT

The most significant finding of this study is related to the role of the more recently fielded attitudinal variables in 2015 and 2016. Students’ positive attitudes and beliefs toward ICT use have a substantive positive influence on both their math and science performance for all cycles. More specifically, self-confidence in ICT high-level tasks, positive attitudes toward computers, belief in the usefulness of computers and the Internet as a tool for school learning, interest in ICT, perceived ICT competence, and perceived autonomy in ICT use appear to have a positive influence on students’ math and science performance. Previous studies have also found that successful students in math and science have more positive attitudes toward computers (Petko et al., 34; Tourón et al., 34), are more confident in ICT use (Guzeller & Akin, 9; Luu & Freeman, 24), are more interested in ICT use (Christoph et al., 9; Hu et al., 21; Meng et al., 21; Koğar, 21; Kunina-Habenicht & Goldhammer, 24), and feel more competent (Hu et al., 21; Koğar, 21; Kunina-Habenicht & Goldhammer, 24; Luu & Freeman, 24; Papanastasiou et al., 21; Srijamdee & Pholphirul, 44) and autonomous in using ICT (Hu et al., 21; Juhaňák et al., 9; Kunina-Habenicht & Goldhammer, 24; Meng et al., 21).

The findings of this study corroborate the assumptions of the self-determination theory and the ICT engagement concept (Deci & Ryan, 9; Goldhammer et al., 9), suggesting that academically successful students have a higher content-specific inner motivation related to ICT (ICT interest), more positive beliefs about their ICT knowledge and skills (ICT competence), and a feeling of self-directedness and control in ICT-related activities (autonomy). Given these relationships, it may be that there exists a cluster of student attributes associated with positive beliefs and attitudes around learning in ICT and in general. More work could be done to explore this. It should also be noted that the current findings posit that student enjoyment of social interaction around ICT has a negative influence on students’ math and science performance, confirming the findings of the previous studies (Hu et al., 21; Juhaňák et al., 9; Kunina-Habenicht & Goldhammer, 24; Meng et al., 21). In addition, it may be that lower performing students use ICT more often for social interaction to solve their school-related problems, such as requesting help from others instead of searching for written information, as was proposed by Kunina-Habenicht and Goldhammer (24).

Incidental findings

While students’ math and science performance in public schools was found to be lower than that in private schools, after controlling for the role of ESCS and the provision of extracurricular activities, school designation as a public institution appears to offer an advantage (Tourón et al., 34). This was a surprising result as it appears to be counter-intuitive. However, Zhang and Liu (21) findings also confirm the same pattern after controlling for ESCS. This finding appears to extend previous research that found no statistically significant relationship between private schooling and student performance in Australia (Nghiem et al., 21). Evidence in the current study of a consistent and growing reverse relationship (i.e., public school advantage, ceteris paribus) for the past decade in PISA. On a speculative note, this pattern may be associated with the general, and perhaps inefficient, trend toward school privatization and socio-economic segregation (Lam et al., 21; Valenzuela et al., 24; Willms, 57). Finally, the provision of extra-curricular activities can be a critical complement to science and math performance that appears to consistently raise the learning bar and possibly ameliorate the role of socio-economic disadvantages (Willms, 57). Finally, our study adds to the growing body of literature on the role of gender for math and science performance. We note that boys tend to have a moderate advantage for math and more slight advantage for science, ceteris paribus.

Conclusion

The results of this study imply that the most substantial ICT-related predictor of students’ are an appropriate set of positive attitudes, competencies, and skills. In other words, the intensity or the quantity of ICT use itself may not make a difference, and the students may not realize the expected benefits if they do not use the ICT purposefully and consciously. These results are in line with the previous research findings suggesting that the quality of the ICT use is more predictive of students’ academic outcomes than the quantity (Lee & Wu, 24; Lei, 9; Petko et al., 34). Since ICT availability and ICT use have varying influences on students’ academic performance, educators and parents are recommended to be extra cautious in using ICT both inside and outside school. It can be helpful for educational leaders, teachers, and parents to invest more time in developing strategies for the students to effectively use educational technologies as a learning tool and to refrain from their distractive effects. The results also imply the importance of students’ positive attitudes and beliefs toward ICT and their interest in ICT for their math and science performance. Based on these results, teachers and parents are advised to nurture students’ positive attitudes and beliefs toward ICT to supplement learning and empower them to be self-competent and autonomous learners in order to improve their learning.

There are several limitations concerning the data used in this study, and they have implications for future studies. The cross-sectional nature of the PISA data set does not allow us to make direct causal inferences from the findings; instead, we intended to explore the associations between the selected variables. Other researchers can use experimental or longitudinal designs to better explore cause-and-effect relationships between those variables. The self-reported nature of the PISA data used in this study poses a methodological limitation that might provide an exaggerated or biased approximation of the ICT related attitudes and perceptions, and this might not give an accurate estimation of the ICT use. Other researchers can use different research designs and datasets that provide a more precise delineation of the ICT use and other ICT-related variables. Another limitation is that the complete set of items in the ICT questionnaire varied between different PISA cycles. It maybe that the items do not represent ICT-related behaviour in a comprehensive way in order to cover all aspects of the ICT related perceptions and attitudes or ICT use. In addition, all the results need to be interpreted in the context of the current research design (i.e., inclusion of specific country, school, and student-related variables). Therefore, researchers can use other data sets covering other ICT related variables such as teachers’ and parents’ perceptions and attitudes towards educational technologies, teacher support, or parental support in ICT use. Finally, future research that explores student accessibility to and attitude toward ICT during and after the recent schooling restrictions (associated with the pandemic) will also shed light on this field.