Assessment of student ICT competence according to mathematics, science, and reading literacy: evidence from PISA 2018

Abstract

To assess the integration of information and communication technologies (ICTs) in teaching and learning, a new construct for ICT competence was developed by the Programme for International Student Assessment (PISA) in 2022. The framework for ICT assessment involves four major dimensions; access to ICT, use of ICT, students’ ICT competencies, and subject-related ICT use. In this paper, we analysed this ICT assessment in terms of validity construction of test scores with data from PISA 2018 and investigated ICT competence in relation to the variables of mathematics, science, and reading literacy. The analyses were based on public data from PISA 2018 for Turkey (N = 6890). After removal of univariate and multivariate outliers from the Turkish sample, factor analysis, correlation analysis, and structural equation modelling were performed on 5438 data items. In the study, we confirmed the four-dimensional structure of ICT competence for the Turkish sample. In this study, the construct validity of the PISA 2018 application ICT test was verified using the Turkish sample. The ESCS variable was included as a control variable in the analyses due to its significant impact on the relevant variables. The analysis showed that the mathematical literacy scores of the PISA 2018 Turkish students were positively low affected by the availability of ICT, negatively low affected by the entertainment of ICT, and positively low affected by the interest and perceived competence in ICT, use of ICT for educational purposes at school and outside school, and the use of ICT related to the subject. Considering the ICT variables that affected the PISA 2018 students’ science literacy scores; the entertainment of ICT and the interest and perceived competence in ICT significantly low affected the students’ use of ICT on the subject. On the other hand, the availability of ICT negative significantly low affect their science literacy scores. While the entertainment of ICT negatively low affected the science literacy scores, interest and perceived competence in ICT and use of ICT for educational purposes at school and outside school positively low affected the students’ use of ICT on the subject. Finally, when we look at the ICT variables that low affected the students’ reading literacy scores; and use of ICT for educational purposes at school and outside school and the interest and perceived competence in ICT significantly positive low affected the use of ICT on the subject. On the other hand, the availability of ICT negative significantly low affect reading literacy scores. While the entertainment of ICT negatively low affected reading literacy scores; interest and perceived competence in ICT positively low affected the use of ICT on the subject.

Introduction

The sweeping changes seen in the application of different technologies has caused significant increases in the use of information and communication technologies (Homiakova et al., 2017). This outcome has been defined differently in various sources. Lennon et. al. (2003) defined ICT literacy as ‘the interest, attitude, and ability of individuals to appropriately use digital technology and communication tools to access, manage, integrate, and evaluate information, construct new knowledge, and communicate with others in order to participate effectively in society’ (p. 8). In another source, Gowsalya and Vaitheeswari (2017) defined ICT as various technological tools and resources used to manage, communicate, disseminate, and store information. Also, computers, laptops, mobile phones, and satellites are part of this technology. When the education fields in which ICT are used are considered, it can be said that they have significant roles in what and how students learn.

In Turkey, various tools are used to measure student success in the international arena, of which the Programme for International Student Assessment (PISA) is a key resource. The PISA is a largescale evaluative tool applied by the Organisation for Economic Co-operation and Development (OECD, 2016, p. 25). Its aim is to evaluate 15-year-old students’ knowledge in the areas of reading, science, and mathematics in order to assess to what extent students can apply their knowledge in real-life situations (Milli Eğitim Bakanlığı, 2013). In PISA’s optional ICT Familiarity Questionnaire, questions are included on ICT availability, ICT familiarity, and ICT usage. Accordingly, the purpose of the current study is to evaluate PISA’s ICT assessment in terms of the validity construction of test scores based on data from PISA 2018. Moreover, the study aims to investigate students’ ICT competence in relation to the variables of mathematics, science, and reading literacy scores.

Theoretical background

The PISA 2022 ICT framework

Today, ICT competence and digital literacy are recognised as important skills that students need to acquire for life in the digital age. As reported in OECD (2019a, p. 3), ICT is integrated into schools and learning in two major forms:

• Student engagement with ICT both in and outside of school can affect their cognitive processes and wellbeing, and eventually influence what they learn.

• Teachers use ICT for instruction, as well as for administrative and communication purposes, but primarily within their instructional practices and pedagogical approach.

An assessment framework offered by the OECD (2019a, p. 5) to evaluate ICT competence covers three key dimensions:

• Access to ICT encompasses availability, accessibility, and quality of ICT resources with a special focus on (connected) technologies that can support learning (e.g., digital learning resources, learning management systems, etc.).

• Use of ICT covers the intensity as well as the types and modalities of ICT usage by students within informal, and possibly unsupervised, environments for learning and leisure, and also supervised classroom tuition, notably through teachers’ pedagogical practices with ICT. This also includes teachers’ alternative ICT applications in support of their teaching practices.

• Students’ ICT competencies describe the core competency areas identified through existing assessment frameworks on ‘digital literacy’, as well as learners’ attitudes and dispositions towards ICT usage (for learning and for leisure). A self-efficacy measure is proposed to assess students’ ICT competencies.

As depicted in the PISA 2022 ICT conceptual framework, teachers’ and students’ practices with ICT are affected to some degree by the availability, accessibility, and quality of ICT resources. Moreover, the way in which students utilise ICTs can also be affected by the digital equipment they use. In addition, the quality of ICT resources, for example, equipment with poor technical capacity or slow/inadequate Internet connection, can affect the conduct of classes and activities where ICTs are used. In contrast, well-designed activities and educational software may be considered by learners to be interesting and accessible but may not be appropriate for the teacher’s instructional capacity or to the scope of the teaching curriculum.

When ICT use in the classroom is examined; students can learn traditional subjects such as mathematics, reading, or science with the use of ICT resources. Students’ cognitive performance can be negatively affected through their interaction with instructional strategies and their learning participation may be affected where ICT-assisted instruction is applied. The way in which teachers utilise technology and how it is reflected in their teaching also affects this process. Students’ ICT competencies and attitudes can therefore make it easier, or more difficult, for students to use ICT to learn other subjects. ICT-specific education can also increase students’ awareness of the potential effects of ICT usage on their wellbeing. ICT may be used not only in the school environment, but also outside of school with home-based activities, self-directed learning, and for leisure purposes. Despite a growing body of literature that has focused on the relationship between student ICT engagement and their education outcomes, there has been no consensus reached with regards to the contribution itself (OECD, 2019a, p. 3).

Student ICT competence according to mathematics, science, and reading literacy

Considering the availability of school ICT resources, a number of studies failed to reveal evidence of any relationship between the availability of computers at school and student reading performance (Angrist & Lavy, 2002; Goolsbee & Guryan, 2006; Lee & Wu, 2012; Rouse et al., 2004). Similarly, the relationship between the availability of ICT at home and reading performance is unclear, with some studies revealing evidence of a positive relationship whilst others reported a negative relationship. Lee and Wu (2012) stated that access to various ICT items at home had a negative correlation with PISA reading literacy. In addition, they also proved that a positive and indirect relationship exists between ICT resources at home and academic success through online reading participation. Lee and Wu (2012) found that access to various ICT elements at home had a positive correlation with reading literacy when students took advantage of online resources, or participated in online discussion forums, and read online materials in a meaningful way. In a study published by Gubbels et. al. (2020), it was examined how the availability of ICT resources, students’ use of those resources at school, outside of school for schoolwork, outside of school for leisure, and students’ attitudes toward ICT relate to students’ reading literacy performance according to data from Dutch students in PISA 2015. Controlling for the relationships of student gender and students’ economic, social, and cultural status, the results showed that students with moderate access to ICT resources, moderate use of ICT either at school or outside of school for schoolwork, and moderate interest in ICT resulted in the highest digitally assessed reading performance. Moreover, excessive access to ICT resources, the excessive use of ICTs, and excessive interest in ICT was shown to relate to lower digitally assessed reading performance. Studies have also revealed that when students have higher interest, autonomy, and competence in ICT, they also exhibit significantly higher levels of reading performance (Hu et al., 2018; Lee & Wu, 2012; Petko et al., 2017).

When the availability of ICT is considered in terms of mathematics literacy, it may be said that ICT availability has a negative correlation with mathematics, science, and reading literacy (Lee & Wu, 2012). Contrary to this statement, some studies have shown that a positive relation exists between ICT availability and students’ success according to data from PISA 2009 and 2012 (Delen & Bulut, 2011; Erdoğdu & Erdoğdu, 2015). Regarding the impact of ICT usage on mathematics literacy, a positive effect has been shown to exist (Papanastasiou & Ferdig, 2006). Likewise, Skryabin et. al. (2015) revealed that the development level of ICT and its individual application can affect fourth- and eighth-grade students’ literacy in science, mathematics, and reading. Also, according to data gathered from PISA 2000, Fuchs and Woessmann (2004) revealed that a positive relation exists between student success and their access to computers at home and at school. Also, according to PISA 2012, a positive relation was revealed between students’ ICT usage at home and mathematics literacy (Meggiolaro, 2018). Similarly, Petko et. al. (2017) also revealed a positive relation between students’ ICT usage at home for schoolwork and mathematics literacy. Contrary to these findings, Meggiolaro (2018) revealed that when ICT is used intensively at school, Italian students’ mathematics literacy was found to be negatively affected. Likewise, a negative relationship between students’ ICT usage at school and their mathematics scores was revealed based on Finnish and Turkish students’ 2012 PISA results (Bulut & Cutumisu, 2018). Additionally, research related to students’ self-efficacy and their ICT efficacy have shown that if students exhibit higher levels of ICT interest, autonomy, and competence, their mathematics, science, and reading performances are positively affected (Hu et al., 2018). This result also aligns to other studies in the literature (e.g., Petko et al., 2017).

In terms of the availability of ICT and its relationship with science, the literature has also revealed certain differences. According to Bulut and Cutumisu (2018), the availability of ICT at home and at school was found to have a positive correlation for Turkish students’ science scores. This result was also consistent with another study in which Li et. al. (2020) established an indirect relationship between ICT availability at home and science literacy performance. Considering ICT use and literacy in science, previous studies have pointed to certain differences in their findings. Petko et. al. (2017) reported finding a negative relationship between ICT use at home for homework and science scores for the majority of low-performing countries. Hu et. al. (2018) revealed that overall lower science scores were linked to ICT usage at home for schoolwork, whilst Luu and Freeman (2011) discussed ICT usage at home for homework as having a negative relationship with science scores for Australian students, other than their general usage of computers. Petko et. al. (2017) reported that a positive relationship was found between ICT usage at home for homework and science, but only for the top-performing countries. Rodrigues and Biagi (2017) examined student data from PISA 2015 for 25 European countries and found that students with low levels of ICT usage also tended to perform better in science. A null (i.e., neutral) relationship was revealed by Luu and Freeman (2011) having examined Canadian students’ ICT usage at home for homework (HOMESCH) and science. Bulut and Cutumisu (2018) also revealed that no link was found between students’ science scores and their ICT usage at home for schoolwork. Also, prior research related to ICT competence and science literacy has shown that ICT competence and autonomy are significantly and positively related to science literacy (Areepattamannil & Santos, 2019; Lee & Wu, 2012; Zhang & Liu, 2016).

The socio-economic differentiation in Turkey has an effect upon mathematics, science and reading literacy scores. While examining the effect of students’ ICT related variables on their literacy levels, taking SES under control explains the reasons for the effects more accurately. Overall, it may be said that various studies in the current literature have revealed the existence of a relationship between students’ ICT use, availability, and interest, and their mathematics, science, and reading literacy, although some differences in their findings are evident. Whether there exists a negative, positive, or even no correlation between ICT use, availability, and efficacy, and students’ mathematics, science, and reading scores, it is clear that the effect of ICT on student literacy cannot be ignored.

Research questions

In the current study, we pursue the following goals related to ICT competence in accordance with the PISA 2018 ICT Familiarity questionnaire:

• 1) Is construct validity of the AVB_ICT, USE_ICT, INT_ICT, and SUB_ICT variables in the PISA 2018 ICT test provided for the Turkish sample?

• 2) What is the effect of ICT variables on students’ mathematics, science, and reading literacy according to the PISA 2018 application when ESCS is moderated?

  • a) Does the availability of ICT, the entertainment use of ICT, use of ICT for educational purposes outside school, use of ICT for educational purposes at school, the interest and perceived competence in ICT, and subject-related ICT use affect the mathematical literacy scores of Turkish students participating in the PISA 2018 application?

  • b) Does the availability of ICT, the entertainment use of ICT, use of ICT for educational purposes outside school, use of ICT for educational purposes at school, the interest and perceived competence in ICT, and subject-related ICT use affect the science literacy scores of Turkish students participating in the PISA 2018 application?

  • c) Does the availability of ICT, the entertainment use of ICT, use of ICT for educational purposes outside school, use of ICT for educational purposes at school, the interest and perceived competence in ICT, and subject-related ICT use affect the reading skills scores of Turkish students participating in the PISA 2018 application?

Method

Sample

The analyses conducted were based on data from the PISA 2018 application in Turkey. The sample used in the current study was formed using two-stage stratified sampling and consisted of 6890 students receiving formal education at the seventh grade or above, aged between 15 years 3 months and 16 years 2 months at the time of the PISA 2018 implementation (OECD, 2019b, p. 29). Differences were found to exist in the number of blank data items based on the items of the ICT test, and this varied from 1.4 to 9.0%.

Instruments

The assessment instruments consist of 11 derived variables in total, measured through Likert-type items, namely the availability of ICT, usage of ICT, interest and perceived competence in ICT, and subject-related ICT use (OECD, 2019b, p. 26–27). Descriptive statistics for the 11 derived variables in the ICT test are presented in Table 1.

Table 1 Descriptive statistics for ICT scales, Turkey sample

Descriptive statistics of mean, standard deviation, and internal consistency coefficient (Cronbach’s alpha) values of the scale’s data are shown in Table 1. As can be seen, the Cronbach’s alpha values were between 0.87 and 0.93, and are therefore deemed to be sufficient in terms of the scale’s reliability (Cortina, 1993).

The scale used for the availability of ICT consists of simple questionnaire indices (OECD, 2019b, p. 26–27), with 3-point, Likert-type items to assess students’ access to and use of various technological equipment. Table 2 presents the items for the availability of ICT variable.

Table 2 Items for the assessment of availability of ICT

The scale for the usage of ICT consists of derived variables based on IRT scaling (OECD, 2019b, p. 26). Regarding students’ usage of ICT, frequency of use of digital devices for leisure activities outside of school, schoolwork outside of school, and activities at school are assessed through 5-point, Likert-type items (1 = never or hardly ever, 2 = once or twice a month, 3 = one or twice a week, 4 = almost every day, 5 = every day). Table 3 presents the items for the ICT use variable.

Table 3 Items for the assessment of usage of ICT

The scale for interest and perceived competence in ICT consists of derived variables based on IRT scaling (OECD, 2019b, p. 27). In terms of students’ interest and perceived competence in ICT, their interest, perceived competency, and perceived self-efficacy in ICT use, and ICT being a part of their social life are evaluated through 4-point, Likert-type items (1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree). Table 4 presents the items for the interest and perceived competence in ICT variable.

Table 4 Items for the assessment of interest and perceived competence in ICT

The scale for subject-related ICT use consists of derived variables based on IRT scaling (OECD, 2019b, p. 27). There are two questions with five response options regarding the use of technological tools outside and inside of the classroom. Table 5 presents the items for the subject-related ICT use variable.

Table 5 Items for the assessment of subject-related ICT use

In the PISA 2018 application, there were 245 questions related to reading literacy, 83 for mathematical literacy, and 115 questions for science literacy. Students participating in the PISA 2018 application answered the questions according to the individualised test design. According to the item response theory Rasch model, probable scores in mathematics, reading, and science literacy were calculated (OECD, 2019b, p. 2).

Statistical analyses

Prior to the analysis, univariate and multivariate outliers were extracted from the dataset, and normality and multicollinearity assumptions were met. Assumptions were made using IBM’s SPSS 25.0 program, with structural equation models and path analysis formed using R Gui 4.2.2 software’s ‘lavaan’ and ‘semPlot’ packages (Rosseel, 2012). The R codes of the analyses are as presented in appendix of this paper. The robust maximum likelihood (MLR) method was used for estimation in the data analysis. As PISA 2018 used a two-stage random sampling process, the probability of participating in the research differs. In order to account for these variations in all analyses, final student weight ‘W_FSTUWT’ was used for weighting. ‘W_FSTUWT’ can be found in the datasets for public use.

Root mean square error of approximation (RMSEA) and comparative fit index (CFI) values were taken into account for the evaluation of model fit. Values of less than 0.08 for RMSEA or greater than 0.80 for CFI were deemed to indicate an acceptable fit, whilst values of less than 0.05 for RMSEA or greater than 0.95 for CFI indicate a model with a good fit (Hu & Bentler, 1999).

In order to calculate the correlation between the reading, science, and mathematics literacy scores for PISA 2008 according to the variables of availability of ICT, usage of ICT, interest and perceived competence in ICT, and subject-related ICT use, path analyses were created for 10 plausible values and the results were then averaged. For each participant, the scores for science, mathematics, and reading literacy are given as 10 plausible values. The studies were performed using all ten probable values, and using the R imputation code, the model parameters that resulted for each value were pooled in accordance with Rubin’s combination criteria. The simple random sample assumptions for computing standard errors of estimates do not apply because of the stratified multistage sampling method utilised by PISA. As a result, the conventional variance formulas for parameter estimates are inappropriate. The modified balanced repeated replication (BRR) approach is used in PISA (Rutkowski et al., 2010). The standard error also accounts for imputation error for the plausible values of student performance.

Results

Evidence based on the internal structure of availability of ICT, usage of ICT, interest and perceived competence in ICT, and subject-related ICT use

In order to investigate the dimensional structure of the four areas that form the PISA 2018 ICT test, measurement models were constructed to evaluate the variables of each area. In examining the dimensional structure of the scales, a two-dimensional measurement model (see Fig. 1) was formed for the availability of ICT, a three-dimensional model (see Fig. 2) for the usage of ICT, a four-dimensional model for interest and perceived competence in ICT (see Fig. 3), and a two-dimensional model (see Fig. 4) for subject-related ICT use. The model fit indices calculated for each measurement model are presented in Table 6.

Fig. 1

Measurement model for Availability of ICT (Model fit: ${\upchi }^{2}$ = 3785.815, df = 179, CFI = 0.83, RMSEA = 0.06)

Fig. 2

Measurement model for Usage of ICT (Model fit: ${\upchi }^{2}$ = 7962.127, df = 505, CFI = 0.89, RMSEA = 0.06)

Fig. 3

Measurement model for interest and perceived competence in ICT (Model fit: ${\upchi }^{2}$ = 3575.659, df = 183, CFI = 0.93, RMSEA = 0.06)

Fig. 4

Measurement model for Subject-related ICT use (Model fit: ${\upchi }^{2}$ = 4108.187, df = 120, CFI = 0.92, RMSEA = 0.08)

Table 6 Model fit indices for CFA findings

When Table 6 and Fig. 1 are examined, it can be seen that a significant difference exists between the population covariance matrix and the sample covariance matrix (3785.815 $\chi^{2}$, p < 0.05), that the CFI value is above 0.80, and the RMSEA value (badness of fit) is less than 0.08.

When Table 6 and Fig. 2 are examined, a significant difference is seen to exist between the population covariance matrix and the sample covariance matrix (7962.127 $\chi^{2}$, p < 0.05), that the CFI value is above 0.80, and the RMSEA value (badness of fit) is less than 0.08.

When Table 6 and Fig. 3 are examined, a significant difference is seen to exist between the population covariance matrix and the sample covariance matrix (49,833.657 $\chi^{2}$, p < 0.05), that the CFI value is above 0.90, and the RMSEA value (badness of fit) is less than 0.08.

When Table 6 and Fig. 4 are examined, a significant difference is seen to exist between the population covariance matrix and the sample covariance matrix (4108.187 $\chi^{2}$, p < 0.05), that the CFI value is above 0.90, and the RMSEA value (badness of fit) is less than 0.08.

When the four models are examined, it can be seen that factor loads are below 0.50, and that they contain similar statements (e.g., items IC001Q01TA and IC001Q02TA consider access to a desktop or laptop computer in the availability of ICT model). These items may also be under-represented in the Turkish sample.

When the error variances of the four models are examined, values for the first model (availability of ICT) range from 0.53 to 0.96, from 0.35 to 0.86 for the second model (ICT use), from 0.29 to 0.68 for the third model (ICT interest), and from 0.30 to 0.91 for the fourth model (subject-related ICT use). The variables observed for all four models show significant load values at the 0.05 level in explaining the latent variables, whereas factor correlation for the four models does not exceed 0.65. While the interest and perceived competence in ICT model for the Turkish sample has a perfect fit, the other three models (availability of ICT, usage of ICT, and subject-related ICT use) show an acceptable fit.

Correlation between availability of ICT, USESCH, HOMESCH, ENTUSE, interest and perceived competence in ICT, and subject-related ICT use

The correlation matrix between the variables in the ICT test is presented in Table 7. The highest correlation between the variables is seen between entertainment use of ICT and interest and perceived competence in ICT (r = 0.47), whilst a positive low or moderate relationship exists between all dimensions.

Table 7 Correlation between variables in the ICT test and literacies

In addition, when Table 7 is examined, the relationships between ICT variables and literacy areas can be seen. Mathematical literacy shows a low-level positive correlation with availability of ICT, interest and perceived competence in ICT, and subject-related ICT variables. Moreover, there is a low-level positive correlation with use of ICT for educational purposes at school and use of ICT for educational purposes outside school, while a low-level negative correlation is observed with entertainment use of ICT. Similar results were found for both science literacy and reading literacy, with the ENTUSE variable showing a low-level negative correlation, and the other ICT variables showing a low-level positive relationship.

Effect between availability of ICT, use of ICT for educational purposes at school, use of ICT for educational purposes outside school, entertainment use of ICT, interest and perceived competence in ICT, subject-related ICT use, and performance in PISA literacy tests

The fit indices obtained for the structural model created between literacy and ICT variables are presented in Table 8.

Table 8 Fit indices for models

When the fit indices of the mathematics, science, and reading literacy models are examined, it can be seen that all three show an acceptable fit.

When Table 9 is examined, a negative and significant relationship can be seen in the model established between mathematical literacy and the ICT variables between availability of ICT and mathematical literacy (β = − 0.13, t = − 8.88, p = 0.00). A one-unit increase in the availability of ICT was shown to result in a 4.23-point decrease in mathematical literacy. A positive and significant relationship can be seen between mathematical literacy and the variable of interest and perceived competence in ICT (β = 0.10, t = 101.46, p = 0.00), and between subject-related ICT use and mathematical literacy (β = 0.23, t = 91.60, p = 0.00). Similarly, there is a positive correlation between the mathematics literacy and use of ICT for educational purposes at school variables (β = 0.10, t = 155.88, p = 0.00) and mathematics literacy and use of ICT for educational purposes outside school variables (β = 0.11, t = 111.37, p = 0.00) at low level. Unlikely, a significant and negative correlation is seen between entertainment use of ICT and mathematical literacy (β = − 0.14, t = − 14.39, p = 0.00), with a one-unit increase in the usage of ICT variable indicating a 6.46-point decrease in mathematical literacy scores. The structured equation model (SEM) established between mathematical literacy and ICT variables is presented in Fig. 5.

Table 9 Effect coefficients between variables in models

Fig. 5

Model 1: Mathematical Literacy

The effect of ICT variables on mathematical literacy according to low, medium, and high socio-economic and cultural level (ESCS) is presented in Fig. 6. Similar increases or decreases are observed in variables other than the sub-related ICT. However, while the scores of students from low socio-economic levels decrease in the use of sub-related ICT, the mathematical literacy scores of students from upper and medium socio-economic levels increase.

Fig. 6

Mathematics literacy scores according to ICT variables while ESCS moderate

In the second model, between science literacy and the ICT variables, a positive correlation is shown between interest and perceived competence in ICT and science literacy (β = 0.11, t = 117.55, p = 0.00), and between subject-related ICT and science literacy (β = 0.16, t = 84.74, p = 0.00). Similarly, a positive correlation between science literacy and use of ICT for educational purposes at school variables (β = 0.10, t = 179.12, p = 0.00) and science literacy and use of ICT for educational purposes outside school variables (β = 0.09, t = 110.31, p = 0.00) is available at low degree. On the contrary, a significant and negative correlation is seen between entertainment use of ICT and science literacy scores (β = − 0.08, t = − 0.90, p = 0.00). In other words, a one-unit increase in the entertainment use of ICT variable results in a 4.89-point decrease in science literacy scores. Similarly, a negative relationship (β = − 0.14, t = − 8.14, p = 0.00) established between availability of ICT and science literacy. The SEM model established between science literacy and the ICT variables is presented in Fig. 7.

Fig. 7

Model 2: science literacy

The effect of ICT variables on science literacy according to low, medium, and high socio-economic and cultural level (ESCS) is presented in Fig. 8. Similar increases or decreases are observed in variables other than the sub-related ICT. However, while the scores of students from low socio-economic levels decrease in the use of sub-related ICT, the science literacy scores of students from upper and medium socio-economic levels increase.

Fig. 8

Science literacy scores according to ICT variables while ESCS moderate

In the third model, between the ICT variables and reading literacy, a significant and positive correlation is shown between interest and perceived competence in ICT and reading literacy (β = 0.16, t = 135.50, p = 0.00), and between subject-related ICT and reading literacy (β = 0.16, t = 86.74, p = 0.00). Similarly, a positive correlation is available between reading literacy and use of ICT for educational purposes at school variables (β = 0.09, t = 203.20, p = 0.00) and reading literacy and use of ICT for educational purposes outside school variables (β = 0.21, t = 105.10, p = 0.00) at low degree. On the other hand, the availability of ICT variable was significantly negative affect reading literacy scores (β = − 0.15, t = − 10.12, p = 0.00). As entertainment of ICT increases, reading literacy scores are seen to decrease (β = − 0.13, t = − 14.63, p = 0.00). The SEM model established between science literacy and the ICT variables is presented in Fig. 9.

Fig. 9

Model 2: reading literacy

The effect of ICT variables on read literacy according to low, medium, and high socio-economic and cultural level (ESCS) is presented in Fig. 10. Similar increases or decreases are observed in variables other than the sub-related ICT. However, while the scores of students from low socio-economic levels increase in the use of sub-related ICT, the read literacy scores of students from upper and medium socio-economic levels increase.

Fig. 10

Read literacy scores according to ICT variables while ESCS moderate

Discussion

The first aim of this study was to verify the construct validity of the ICT test in terms of the 2018 Turkish application of PISA. The second aim was to determine whether or not the availability of ICT, the use of ICT, interest and perceived competence in ICT, and the subject-related use of ICT affected to a low degree the mathematics, science, and reading literacy scores of Turkish students participating in the PISA 2018 application. Factor analysis, correlation analysis, and SEM were performed on data from 5438 students from the Turkish sample (N = 6890) of the PSA 2018 application after having removed univariate and multivariate outliers.

The findings confirmed a four-dimensional structure of ICT competence for the Turkish sample according to structural equation modelling approach. Items from the PISA ICT test were found to be indicators of the students’ behavioural skills related to ICT, and the structure of the data was found to be significant for the Turkish sample. Similar findings have been reported in the literature from construct validity studies using ICT tests with samples from various countries (e.g., Kunina-Habenicht & Goldhammer, 2020).

In line with second research question, structural equation models were established under the moderation of ESCS with variables obtained from the mathematics, science, reading literacy and ICT survey. Accordingly, while mathematical, science and reading literacy have low level negative correlation with the availability of ICT and entertainment of ICT variables, it has positive correlation with use of ICT for educational purposes at school and outside school, interest and perceived competence in ICT, and subject-related use of ICT variables. Accordingly, a study emphasized that not only school but also home ICT use positively influenced 4th grade students’ performance in three areas (maths, science, reading) considering the results of TIMSS 2011 and PIRLS 2011 data (Skryabin et al., 2015). However, according to the study conducted by Petko et. al. (2017) revealed that school ICT use has significant negative influence on students’ mathematics, science and reading achievement. Moreover, Petko et. al. (2017) discovered a primarily negative correlation between students’ ICT leisure use at home and academic performance, but a positive correlation between students’ ICT academic use at home and achievement in mathematics, science, and reading. However, a study conducted by Gumus and Atalmis (2011) found that the use of computers for educational purposes showed a negative impact on Turkish students’ reading literacy, on the other hand the use of computers for entertainment purposes had a positive effect. Moreover, in the study of Hu et. al. (2018), it is concluded that ICT academic use outside school showed negative correlation with students’ reading and science achievement, but it did not have significant effect on mathematics achievement. This result can be arisen from the fact that students may not use ICT for educational purposes outside the school. When ESCS is under control similar interactions are observed. In terms of mathematics, science and reading literacy only the subject-related use of ICT variable has a negative effect on those with low socio-economic levels; however, it has a positive effect on those with high and medium socio-economic levels. Accordingly, families with high level of education are more willing to pay attention the cultural development and academic success of their children (Martins & Veiga, 2010). There are many studies showing the socioeconomic factors on students’ success (Ateş & Karadağ, 2017; Sun & Bradley, 2011; Türkan et al., 2015).

As a result of the analyses conducted in the current study, the availability of ICT, use of ICT for educational purposes at school and outside school, entertainment of ICT, interest and perceived competence in ICT, and subject-related use of ICT were found to significantly affect the mathematical literacy scores of Turkish students participating in the PISA 2018 application. While entertainment of ICT and availability of ICT negatively low affected mathematical literacy scores; interest and perceived competence in ICT, use of ICT for educational purposes at school and outside school positively low affected literacy score. However, as the entertainment of ICT increases, mathematical literacy scores were found to decrease.

The reason behind this negative correlation could relate to the quality of ICT use at home; that is, students may have ease of access to ICT at home or there may be a lack of monitoring their ICT usage at home, leading to addiction to computer games, for example, or other negative social or psychological outcomes (Grüsser et al., 2006). On the other hand, as subject-related use of ICT increases, students’ mathematical literacy scores were shown to increase. However, studies related to ICT use for academic purposes portray a degree of variance in terms of their results. The results of the current study were found to be consistent with other works in the literature (e.g., Petko et al., 2017; Skryabin et al., 2015); for example, Petko et. al. (2017) investigated the relation between ICT use, ICT related attitudes, and PISA 2012 test scores, revealing a positive relation between ICT use at home for schoolwork and mathematics literacy. However, their result contradicted other studies, such as Carrasco and Torrecilla (2012) who reported that use of ICT for academic purposes showed no correlation with mathematics literacy. Similarly in another study, Hu et. al. (2018) revealed that ICT availability at home had a negative association with student academic literacy. Also, studies related to student self-efficacy and their ICT efficacy have shown that where students exhibit higher levels of ICT interest, autonomy, and competence, there is a positive low affect seen in their mathematics, science, and reading performance (Hu et al., 2018); a result that also aligns with other studies (e.g., Petko et al., 2017). In terms of ICT availability, contrary to the results of the current study, some published research has revealed a positive correlation between ICT availability and mathematics literacy (e.g., Delen & Bulut, 2011; Erdoğdu & Erdoğdu, 2015).

Considering the ICT variables that were shown to significantly affect the science literacy scores of students participating in the PISA 2018 application in Turkey, entertainment of ICT, use of ICT for educational purposes at school and outside school and interest and perceived competence in ICT were shown to affect to a low degree the students’ subject-related use of ICT. On the other hand, availability of ICT was not found to significantly affect science literacy scores.

Similar to the current study’s results, ICT access at home was not found to have a significant relation with scientific literacy in data from PISA applications in both Canada and Australia. However, Bulut and Cutumisu (2018) revealed that a positive correlation between ICT availability and science literacy was found in data from Turkey, which was in contrast to that of Finland. Likewise, Fuchs and Woessmann’s (2004) study examined the relation of computer availability on student academic performance based on data from PISA 2000 and revealed that computer availability both at home and at school had a positive correlation with student literacy scores. However, some studies have shown that ICT availability at home was negatively associated with science literacy (Hu et al., 2018; Lee & Wu, 2012). According to research conducted by Luu and Freeman (2011), Canadian 2006 PISA results portrayed that the Internet connectivity of school computers had no significant impact on students’ scientific literacy, whilst a negative relationship was revealed for Australian students, meaning that use of ICT can have a negative effect on science literacy scores. It could be, as purported by Martin-Perpiñá et. al. (2019), that students may use ICT for their own purposes rather than in support of their learning. On the other hand, interest and perceived competence in ICT and subject-related use of ICT were shown in the current study to positively to low degree affect student science literacy. In other words, as interest and perceived competence in ICT and subject-related use of ICT increase, students’ science literacy scores also increase. Other research related to ICT competence and science literacy has also shown that ICT competence and autonomy have a significant and positive relation to science literacy. Areepattamannil and Santos (2019) revealed that students’ ICT competence and ICT autonomy were significantly and positively related to their science self-efficacy, thereby influencing their science literacy. This positive effect of ICT competence on science literacy was also reported in another research too (e.g., Lee & Wu, 2012; Zhang & Liu, 2016).

When we consider how ICT variables affected the students’ reading literacy scores in the 2018 PISA application, use of ICT and interest and perceived competence in ICT were found to significantly to low degree affect subject-related use of ICT. On the other hand, availability of ICT was not shown to significantly affect reading literacy scores, and Lee and Wu (2012) reported that ICT availability at school did not affect either engagement in online reading activities or reading performance. However, this result is contradictory to research published by Hu et. al. (2018), who found a negative correlation between ICT availability and students’ reading performances. Another study reported that ICT availability at home positively to a low degree affected student engagement in online reading activities; however, student reading performance was negatively influenced, leading to conclude that reading literacy can be negatively to a low degree affected by the use of ICT. Hu et. al.’s (2018) study also proved that a negative correlation exists between ICT use at home and reading literacy. This finding may have been due to the teachers’ instructional methods; that is, they did not change their teaching styles according to technology in use (Hew & Brush, 2007). On the other hand, reading literacy scores have been shown to increase as interest and perceived competence in ICT increases. In align with this result, the literature has also revealed that when students have higher levels of interest, autonomy, and competence in ICT, they exhibit significantly higher performance in reading (Hu et al., 2018; Lee & Wu, 2012; Petko et al., 2017).

The variable of ICT usage was found to significantly and negatively to a low degree affect mathematics, reading, and science literacy in students. In other words, as the 15-year-old Turkish students’ use of ICT as a leisure activity, their cognitive skills as seen in their mathematics, science, and reading literacy scores decrease. In addition, the increase in students' use of ICT for school tasks at home and their use of ICT at school increases mathematics, science, and reading literacy scores. When the literacy scores were examined in terms of availability of ICT, the presence of ICT at home or at school was found to significantly and positively to a low degree affect student mathematics literacy scores, but that no significant effect was seen on their reading and science literacy scores. Also, the variables of interest in ICT and subject-related use of ICT were shown to significantly and positively to a low degree affect students’ mathematics, reading, and science literacy scores.

In other words, in terms of 15-year-old Turkish students’ interest and perceived competence in ICT, perceived autonomy related to ICT use, and ICT use as a topic in social interaction; as their ICT literacy increased, an increase was seen in their mathematics, science, and reading skills. Similarly, as students’ subject-related use of ICT increased, it was observed that their mathematics, science, and reading literacy scores also showed an increase. The findings also showed that the use of ICT for non-curricular purposes, at home and at school in general, did not significantly affect the students’ literacy, while interest in ICT, self-efficacy, autonomy, and use of ICT in class or at home did significantly affect their literacy scores.

Conclusion

In this study, the construct validity of the PISA 2018 application ICT test was verified according to the Turkish sample. The analysis results showed that the mathematical literacy scores of Turkish students participating in the PISA 2018 application were negatively to a low degree affected by the availability of ICT and entertainment of ICT, positively to a low degree affected by their use of ICT for educational purposes at school and outside school and affected by their interest and perceived competence in ICT and their subject-related use of ICT.

Considering the ICT variables that were shown to affect the science literacy scores of Turkish students participating in the PISA 2018 application to some degree, however; they affect the entertainment of ICT, use of ICT for educational purposes at school and outside school and interest and perceived competence in ICT significantly to a low degree affected the students’ subject-related use of ICT. While the entertainment of ICT negatively to a low degree affected science literacy scores, interest and perceived competence in ICT, use of ICT for educational purposes at school and outside school positively to a low degree affected the students’ subject-related use of ICT. Finally, from examining the ICT variables that affected students’ reading literacy scores, it was seen that entertainment of ICT, use of ICT for educational purposes at school and outside school, the presence of ICT and interest and perceived efficacy in ICT significantly to a low degree affected the students subject-related use of ICT. While entertainment of ICT negatively to a low degree affected reading literacy scores, interest, and perceived competence in ICT positively to a low degree affected the students subject-related use of ICT.

As with any research, the current study has certain limitations. The dataset used in the study was limited to only the Turkish sample from PISA 2018. Therefore, further research could be conducted on students’ use of ICT based on datasets from other countries, including cross-country comparisons. In the current study, only the effect of ICT variables on literacy scores was examined, whereas, literacy scores could also be evaluated according to multiple hierarchical models with other variables from the student and teacher questionnaires, in addition to the ICT variables.