School-level predictors for the use of ICT in schools and students’ CIL in international comparison

Abstract

The increasing relevance of information and communication technologies (ICT) and society’s transition towards an information or knowledge society have led to the emergence of new challenges for schools and school systems. Thus, the need for students to develop new forms of skills like digital literacy or computer and information literacy (CIL) is constantly gaining in importance. In the IEA’s (International Association for the Evaluation of Educational Achievement) ICILS 2013 (International Computer and Information Literacy Study), the aforementioned competencies were investigated—along with CIL learning contexts and outcomes (such as school-level factors in different education systems)—for the first time for secondary schools by applying computer-based student tests. The research presented in this paper focuses on the school-level factors that support or hinder the use of ICT by teaching staff and students’ CIL, drawing in the process on information obtained through school and teacher questionnaires. A multilevel approach was chosen for this research, drawing on representative data from four of the countries which participated in ICILS 2013, namely Australia, Germany, Norway and the Czech Republic. The results show that the relevance of school-level determinants for the use of ICT by teaching staff in schools differs between education systems. Only in Germany, for example, does pedagogical IT support seem to be crucial for the use of ICT in teaching. In the Czech Republic, the self-efficacy of teaching staff plays a key role, whereas in Australia, the participation of teaching staff in professional development activities can be identified as relevant for students’ acquisition of CIL. The results also show a statistically significant correlation between the teachers’ use of ICT in schools and students’ CIL for Germany, yet indicate no significant effects for Australia, Norway and the Czech Republic. In addition to these and the more specific findings for the considered countries, the international comparison presented in this paper reveals both strengths and developmental potential for the selected education systems.

Background

Education systems around the world face new challenges from the rapid developments in technology and society’s transition towards an information or knowledge society (Anderson 2008; Eickelmann 2011; Voogt and Knezek 2008). Besides discussing new ways of learning and the potentials of ICT from a pedagogical point of view, schools and school systems have acknowledged that new skills and competences are needed to prepare students for life and work in the information age. Thus, the need for students to develop such new kinds of skills, i.e. digital literacy or computer and information literacy (CIL), to enable them to participate effectively in the digital age is constantly gaining in importance (European Commission 2014; Fraillon et al. 2013; Voogt et al. 2013). In this context, it seems to be increasingly important to look at the contexts in which students develop such skills and examine the factors which support or hinder their acquisition. In this regard, the school itself is particularly relevant (e.g. Davis et al. 2013; Eickelmann et al. 2016; Hatlevik et al. 2014; Petko et al. 2015, 2016; Tondeur et al. 2008).

With regard to the factors that contribute to the development of CIL in schools, the ICILS 2013 study (International Computer and Information Literacy Study, 2010–2014; Fraillon et al. 2014) conducted by the International Association for the Evaluation of Educational Achievement (IEA) provides first-time data both on students’ CIL as well as on school-level factors in different education systems. The study investigates the CIL of secondary school students (Grade 8) in 21 education systems using computer-based tests. In addition, it gathers representative student, teacher and school data related to the contexts in which students develop these competencies in all participating countries. By means of an in-depth analysis of ICILS 2013 data, this paper investigates the factors that support or hinder the development of students’ CIL at school level by comparing four education systems around the world (including the top-performing country Czech Republic) using student achievement data as well as data obtained from school and teacher questionnaires.

To understand the relevance of school factors for the acquisition of CIL, the contextual framework of ICILS 2013 (Fraillon et al. 2013) serves as theoretical background in our research. The ICILS framework provides a model which categorizes relevant factors that are in agreement with the multilevel structure inherent in the student CIL acquisition process. It differentiates between antecedents and processes, following the assumptions that antecedents influence processes and that processes are closely linked to the outcome, i.e. the level of CIL competence. It is assumed that both—antecedents and processes—need to be taken into account to explain variation in students’ CIL (see Fig. 1).

Fig. 1

ICILS 2013 theoretical framework (Eickelmann et al. 2014a, based on Fraillon et al. 2013)

As a secondary analysis of the ICILS 2013 data, this paper focuses on four school-level factors as part of both the antecedents and the processes to identify supporting and hindering factors: (1) the school’s ICT equipment, (2) the teaching staff’s professional development, (3) school goals, and (4) the teaching staff’s views/self-efficacy. All of these factors are relevant in the ICILS 2013 contextual framework (Fraillon et al. 2013) and have also been identified as relevant for ICT implementation in schools in other research (e.g. Eickelmann 2011; Kozma 2003; Law et al. 2008). Figure 2 shows the underlying research model behind this paper and the analyses it contains.

Fig. 2

Research model on school-level predictors for ICT use in schools and students’ CIL

To investigate the school-level predictors for the use of ICT by teaching staff in schools and the level of students’ CIL in an international comparison, our research looks at the following two questions:

• 1. What effects do school-level predictors (such as ICT equipment, teaching staff’s professional development, school goals, and teaching staff’s views/self-efficacy) have on the use of ICT by teaching staff in schools in different education systems?

• 2. What is the relation between the conditions identified as most relevant for the teaching staff’s use of computers at school and the average level of students’ CIL in the respective education systems?

Methods

Data sources

As already mentioned, the data for the secondary analyses are derived from IEA’s ICILS 2013, in which the computer and information literacy of Grade 8 students was examined for the first time in an international comparison using computer-based testing. In addition, information on teaching and learning with ICT was collected using questionnaires for students, teachers, school principals and ICT coordinators as well as a national context questionnaire (Jung and Carstens 2015). 21 education systems around the globe participated in ICILS 2013, whose research design defined two target populations: Grade 8 students and teachers teaching in Grade 8 (Jung and Carstens 2015). Within each of the selected schools, a random sample of 20 students and 15 teachers was chosen. The countries chosen for the international comparison in our paper were Germany, Australia, Norway and the Czech Republic. The education systems in Australia and Norway have a long tradition in implementing ICT in teaching and learning, while the Czech Republic is a top-performing country in the ICILS 2013 ranking (Fraillon et al. 2014). Germany, in contrast, has a highly developed education system but with a low pervasion of ICT use for educational purpose. An added value of the international comparison approach is that it allows us to learn from other countries and gain information that will help education systems to accept the challenge of developing for 21st century needs.

To identify those school-level factors which are essential to enhance students’ CIL, we used students’ (Grade 8) achievement data in CIL as well as background questionnaire data from our four ICILS 2013 participant countries to identify similarities between countries as well as country-specific hindering and supporting factors. More specifically, four data sources were taken into account (see, for example, Jung and Carstens 2015):

• Data from the computer-based student questionnaire. To control for relevant student background variables at student level in the analyses pertaining to research question 2, the students’ gender, immigration status and two family socio-economic status variables (home literacy and highest ISCED of parents) were taken into account. As the focus of the research presented in this paper lies on school-level predictors, the results at the individual level will be neither illustrated nor interpreted.

• Data from the student competence test data. Students’ achievement data was collected by means of an authentic computer-based CIL assessment administered to students in the eighth year of schooling (Fraillon et al. 2014) and has been integrated into the analyses as a latent construct of the five plausible values at both the individual as well as the school level. At the school level, it can be interpreted as the average level of students’ CIL in a school.

• Data from the school questionnaire, i.e. information provided by the school principals and ICT coordinators about ICT equipment, school goals and the professional development of teaching staff.

• Data from the teacher questionnaire providing information about the views, self-efficacy and age of teaching staff.

In the ICILS 2013 design, the teacher questionnaire was included in order to provide additional contextual information about the school as well as on general aspects of teaching with regard to CIL (Jung and Carstens 2015). The teacher data has therefore been aggregated at school level to provide information about the school environment (see section on “Methods” for information about the respective weighting). However, it should be noted that two of the four selected countries (Germany and Norway) did not meet the IEA’s high sampling requirements for the teacher sample, while all four showed a teacher participation rate of 75% or above (Australia: 86.5%; Czech Republic: 99.9%; Germany: 79.5%; Norway: 83.1%; cf. Bos et al. 2014, p. 331). However, to permit the comparison, these countries have nonetheless been included in our analyses. These data are more prone to bias, and the results should therefore be interpreted with caution. An analysis of the German teacher sample, for instance, showed no bias with regard to teachers’ gender and their school subjects (Eickelmann et al. 2014a).

Table 1 shows the school-level items and indicators taken from the aforementioned questionnaires that were used in our analyses.

Table 1 ICILS 2013 indicators used and coding

The positive views held by teachers on ICT is an internationally scaled index (“positive views on using ICT in teaching and learning, T_VWPOS, Jung and Carstens 2015), derived from 8 items. The scale has a Cronbach’s alpha of .83. The teachers’ self-efficacy in ICT is also an internationally scaled index (“ICT self-efficacy”, T_EFF, Jung and Carstens 2015) containing 14 items. This scale has a comparably good Cronbach’s alpha of .87. Both indices have a mean of 50 and a standard deviation of 10.

For the analyses pertaining to both our research questions, data is included from about 9500 students (student level) in around 550 schools (school level) in our four selected countries. The average cluster sizes range between 16 and 18 Grade 8 students (see Table 2).

Table 2 Analysis sample in the selected four education systems

Methods

To answer our first research question, i.e. the importance of different school-level predictors for the use of ICT by teaching staff in teaching, a linear regression was conducted at school level. In the case of our second research question, a multilevel structural equation model was carried out to analyze the relation between the conditions identified as most relevant for the use of computers by teaching staff as well as the relation between the latter and the students’ average level of CIL at a school. The students’ CIL was included in the model as a latent factor comprised of the five plausible values. At the student level, the model is controlled by the aforementioned student background variables (students’ gender, immigration status, family socio-economic status). Both models were carried out by using the statistical software Mplus (Version 7; Muthén and Muthén 2012).

Within these analyses, weighting variables are included to account for the complex structure of the ICILS 2013 data: As teacher data is aggregated to the school level, providing information about the teaching staff in a participating school, and is defined as characteristic of the respective school, the weighting variable at the school level is conducted by combining the school base weight with the school nonparticipation adjustment for the teacher survey (WGTFAC1 × WGTADJ1T, Meinck and Cortes 2015).

The full information maximum likelihood method (FIML) was likewise applied (e.g. Enders 2006). Thus, missing values were not imputed, while population parameters and standard errors were estimated based on the data available (e.g. Enders 2006). Additionally, a robust maximum likelihood estimator (MLR) was used to account for the complex data structure (Muthén 2004).

Results

Effects of school-level predictors on use of ICT in teaching (research question 1)

To answer our first research question, the effects of school-level variables on the use of computers by teaching staff in teaching were analyzed in a first step. Figure 3 shows the results of the applied regression model. The model fit is satisfactory (CFI = 1, TLI = 1, RMSEA = .00). The figure shows the standardized coefficients, which are highlighted as significant when they have a p value of < .5 or smaller.

Fig. 3

Analysis model of school-level predictors for the use of ICT by teaching staff in schools

Overall, it becomes obvious that firstly most of the supporting factors for the use of ICT in teaching can be identified among the teaching staff characteristics and secondly that there are a lot of country-specific results.

Starting with the relevance of the ICT equipment, it can be shown for Australia that a favorable ICT equipment situation plays an important role for the use of ICT by teaching staff: the highly significant negative effect of β = −.20 indicates that the fewer the number of students who have to share a computer at school, the more frequently teaching staff uses ICT for teaching and learning. In contrast, the student-computer-ratio does not play a significant role for the use of ICT in Germany, Norway or the Czech Republic. Furthermore, the lack of hardware is not a hindering condition for ICT use in class in any of the four education systems. Regarding technical support, an unexpected result can be shown for the Czech Republic, where the provision of technical support has a negative effect on the use of ICT by teaching staff (β = −.22). For teaching staff in Germany, the provision of pedagogical support appears to be significantly important for the use of ICT in teaching (β = .51).

Regarding the importance of professional development, the results reveal that for Australia and Norway the participation of many or all teachers in a school in courses on the use of ICT is significantly positively related to their use of ICT in teaching (β = .29 and β = .17 respectively). Cooperation in a community of practice for ICT in teaching does not appear to be a relevant predictor for the use of ICT by teaching staff in all four countries.

A look at school goals reveals that the high importance attributed to the use of ICT by the respective school in developing students’ understanding and skills can only be identified as a statistically significant predictor for the use of ICT by teaching staff in the Czech Republic (β = .14). This aspect does not appear to play a relevant role in any of the other three education systems included in our analyses.

As already mentioned above, teaching staff characteristics appear to be the most important supporting factors for the use of ICT in teaching. Indeed, the self-efficacy of the teaching staff in a school is the strongest predictor for three of the four education systems. In the Czech Republic in particular (β = .58), the level of confidence of teaching staff in their ability to use ICT, e.g. knowing how to change the settings on a computer, appears to be a supporting factor for the use of ICT in teaching. This effect can also be shown for Australia (β = .44), Germany (β = .41) and Norway (β = .32). In the latter, the positive views of the teaching staff in a school, e.g. the attitude that the use of ICT enables students to access better sources of information, are equally important for the use of ICT by teaching staff in class (β = .19). Regarding the approximate age of the teaching staff, the results are ambivalent: While a lower average age among teaching staff is related to a more frequent use of ICT in class in Norway (β = −.20), the result for the Czech Republic indicates the opposite (β = .31).

Relation between school-level factors, use of ICT in teaching and students’ CIL (research question 2)

To answer our second research question, the relevance for students’ average CIL in a school was additionally assessed in a multilevel structural equation model context. Figure 4 illustrates the results.

Fig. 4

Analysis model of school-level predictors for the use of ICT by teaching staff in schools and students’ CIL

A small positive effect of the use of ICT by teaching staff in class on students’ CIL could only be identified for Germany (β = .21). For Australia, Norway and the Czech Republic, this effect is not statistically significant.

A closer look at school-level predictors and their importance for students’ CIL shows that neither technical support nor school goals and teaching staff’s self-efficacy have significant effects on student achievement. In Germany and the Czech Republic, for instance, an unfavorable student-computer-ratio is related to a higher student CIL (β = .25 and β = .21 respectively). As far as the quality of the ICT equipment is concerned, a better quality of ICT hardware is positively related to student achievement in Germany (β = .20).

Moreover, the provision of pedagogical support is both highly significant and positively related to students’ average CIL in Germany (β = .44). Unlike the result for our first research question, cooperation among teaching staff concerning ICT in teaching has a highly significant effect on students’ CIL in Australia (β = .39). The participation of a majority of teachers in courses on the use of ICT can be identified as a relevant predictor for student achievement in Germany (β = .38) and Norway (β = .33). For the Czech Republic, a negative effect can be shown (β = .−27). This result can be read as follows: the fewer the number of teachers participating in professional ICT development activities, the better the students’ CIL. The results for the average age of teaching staff are again ambivalent: In Germany, having younger teachers is related to a higher level of students’ CIL (β = −.19), whereas in Norway the result is the opposite (β = .39).

The model explains 75% of the variance in the students’ CIL at the school level in Germany, 38% of the variance for Norway and 29% for Australia. The variance explanation in the Czech Republic is much lower at 17%.

Discussion and conclusions

The objective of this paper was to examine four education systems (Australia, Germany, Norway and the Czech Republic) with regard to the relevance of school-level factors for the use of ICT by teachers in teaching and learning as well as the effect of the latter on students’ CIL, as measured in IEA ICILS 2013. Based on the ICILS 2013 theoretical framework and results from previous research¸ four aspects appeared to be crucial and were thus taken into account for the analysis carried out in this paper: (1) ICT equipment, (2) professional development of teaching staff, (3) school goals, and (4) views/self-efficacy of teaching staff.

The results for our first research question reveal both similarities in the four education systems yet also some country-specific results. As far as the similarities are concerned, the self-efficacy of teaching staff with regard to ICT was identified as a very important supporting factor in all four education systems. This confirms earlier research which identifies teachers as a keystone species (Davis et al. 2013) when it comes to the integration of ICT in schools as well as the apparent importance of teachers’ own perceptions of their competences (Drossel et al. 2016) on both the individual and the collective level (i.e. among the entire teaching staff in a school). As a consequence, a common developmental strategy within a school might be to attribute importance to providing support for teachers, thereby raising their own assessments of their competencies. This support might help to make (secondary school) teachers feel more capable in using ICT in their teaching activities and in adapting their competences in the subject-specific use of ICT. Professional development activities are one way of strengthening this support factor and could be given stronger emphasis in the transition of education systems to 21st century needs.

Moving to the country-specific results, the participation of a large proportion of teaching staff in courses on the use of ICT was identified as a supporting factor for the use of ICT in teaching in Australia. Keeping in mind the comparably high participation rates of Australian teachers in personal development activities (Fraillon et al. 2014), this result underlines the need for ongoing development of technological and pedagogical applications of ICT in schools along with the need for accompanying teacher education and could serve as an example for other countries.

In the Czech Republic, the importance of school goals, or more precisely goals referring to the development of students’ competences in ICT, might possibly be explained by the fact that successful schools use these to establish a bottom-up counterpart to the nationwide ‘framework educational programme for basic education’ established by the Ministry of Education (MŠMT 2007), which includes a detailed account of how ICT should be integrated in each subject and which attributes the relevance of ICT for teaching and learning in a top–down way. Under this national framework, each school has to adapt and integrate more holistic strategies into their own program, adopting the national plans within the scope of the single schools’ conditions. In addition, the Czech School Inspectorate evaluates whether the individual school programs contain everything that needs to be included under the nationwide framework. It also conducts inspection visits to schools to ensure that the program has been enforced.

As far as Germany is concerned, the most relevant school-level predictor for the use of ICT by teaching staff seems to be the availability of corresponding pedagogical support in the classroom. This perhaps offers another indication that teachers in Germany are not sufficiently trained in using ICT (Eickelmann et al. 2016b). Indeed, at school level, this represents the greatest challenge to the development of the German education system. This result is complemented by the finding that the focus of the development of support systems for schools in Germany is still more technical than pedagogical. Furthermore, the responsibility for implementing technical ICT support lies at regional or local authority level, which leads to great variation in support systems across the country and, in some cases, can cause problems when teachers need immediate technical support in the classroom. Future developments in the German education system could therefore focus on providing better pedagogical support for schools and on better preparing teachers for the demands of the pedagogical integration of new technologies in teaching and learning (Eickelmann 2016).

Positive views on ICT on the part of teaching staff are one of the supporting factors for the use of ICT which could be identified at school level for Norway. This result could be interpreted as an indication that the Norwegian top-down strategy used to implement ICT in schools nationwide following the 2006 National Plan (Erstad and Quale 2009) might not have taken into consideration the perspective of all teachers and the need to convince teachers of the usefulness of the use of ICT in teaching and learning.

Regarding our second research question, a significant positive correlation between the use of ICT by teaching staff and the students’ level of CIL could only be identified for Germany. In comparison to previous results dealing with the students’ use of ICT in teaching and learning on an individual level, which showed a negative correlation with students’ CIL in Germany (Eickelmann et al. 2014b) and could be explained by the aforementioned compensatory approach, this result underlines the potential of the use of ICT by teaching staff in teaching and learning for the competencies of their students. In Germany’s case, it could be interesting for further research to conduct the analyses separately by school type as there is a large difference between the competence levels of students at Gymnasia (grammar schools) and other secondary schools.

A limitation of the research presented in this paper is the above-mentioned cross-sectional data base. Even though the research approach takes into account both a theoretical framework as well as previous empirical results and applies a hypothesis-testing method, no causality can be postulated. Therefore, alternative hypotheses and interpretations might apply. The results for Germany, for example, where a less favorable student–computer-ratio has an effect on students’ average CIL at one school could also be interpreted the other way round: schools with a higher average level of students’ CIL might not necessarily have great ICT equipment. This interpretation might be supported by acknowledging that computers in German schools are still often used as an instrument to compensate learning needs and, thus, follow the objective of promoting students with lower competencies rather than all students (for more on this topic in the case of primary schools see, for example, Drossel and Eickelmann 2014). Furthermore, the result that cooperation concerning ICT in communities of practice in teaching has a positive effect on students’ CIL might also be interpreted differently: in schools where students have high levels of CIL, there would seem to be a higher recognition of the need for teachers to cooperate in a more institutionalized and formal way. In the case of our second research question in particular, alternative hypotheses should be taken into consideration. The significant, positive correlation between the use of ICT by the teaching staff and the students’ level of CIL could also be interpreted as an indication that teachers in schools where students show high levels of CIL use ICT more often for teaching, possibly because they assume that their students use ICT in a more competency-oriented and thoughtful way.

Based on these findings, further research besides longitudinal studies should also draw on a qualitative approach to generate more detailed explanations regarding the relevance of the various factors. Comparative case studies might, for example, gain more detailed insights into supporting and hindering factors and thus serve to generate knowledge which provides valuable hints for the development of educational systems from an international and intercultural perspective. This seems to be crucial in appreciating the transition towards an information and knowledge society and the related challenges for education systems. Furthermore, more of the factors that have been identified in previous research as relevant for the use of ICT (e.g. school leadership) should also be included in further analyses.

Concerning the implications for school development, it can be concluded that in all four countries studied, the characteristics of the teaching staff are important for the frequency of computer use but not necessarily for CIL acquisition. This points to a need for increased skills-based learning with ICT and competence-orientated learning supporting the acquisition of CIL. For this, acknowledgement of the relevance of school-level factors is required.