Investigating the effect of COVID-19 disruption in education using REDS data

Abstract

The Covid-19 pandemic has triggered a rapid and unprecedented transformation of global educational systems. The research community has uncovered important impacts of the pandemic on education worldwide, including detrimental effects on student learning, a decline in the well-being of teachers and students, and the exacerbation of educational disparities. Within this context, the Response to Education Disruption Survey (REDS) database stands out as one of the first large-scale international surveys specifically designed to examine the disruption caused by Covid-19 in education. This database serves as an initial step towards providing a more comprehensive understanding of the worldwide effects of the pandemic on education. The relevance of this database has inspired the creation of this special issue, collecting six studies that analyse the consequences of Covid-19 using REDS data.

Introduction

In the early months of 2020, we witnessed an unprecedented transformation in global educational systems due to the rapid disruption caused by the Covid-19 pandemic. In April 2020, at the peak of the first wave of the pandemic, over 90% of students worldwide faced school closures in response to the crisis (UNESCO, 2020). Consequently, students, teachers, and school administrators had to quickly adapt to remote learning. In their initial response, several education systems implemented measures such as the introduction of online learning, home-schooling programmes, provision of free online resources, and the deployment of public TV and radio broadcasting channels for educational programming (UNESCO, UNICEF & World Bank, 2020). As a result, the school closure necessitated a switch to remote learning, with students interacting with peers and with teachers using digital devices at their homes, where they had to share resources with other family members. This period of remote learning has profound effects. By August 2020, students in 108 countries had, on average, missed approximately 10 weeks of in-person classes (UNESCO, UNICEF & the World Bank, 2020).

Over the past three years, the research community has offered valuable insights into the impact of the pandemic on educational systems and student learning. Recent meta-analyses have reported the presence of significant learning loss due to the Covid-19 disruption. These effects are quantified as an average reduction of approximately − 0.14 standard deviation according to Betthäuser et al. (2023), -0.17 standard deviation as reported by Patrinos et al. (2022), and − 0.19 standard deviation according to Di Pietro (2023). Nevertheless, substantial heterogeneity exists across countries, student groups, characteristics, subjects, and grades (Bertoletti et al., 2023a, 2023b; Betthäuser et al., 2023). Monitoring the situation after the first year of the pandemic, students seem to have made partial progress. While they still exhibit learning loss, there are signs of recovery, especially in some subjects (see Gambi & De Witte, 2021).

Additionally, school closures had significant consequences for the well-being of students, leading to feelings of isolation from their peers and a lack of emotional support (Copeland et al., 2021; Hoofman & Secord, 2021). The challenges associated with online instruction, such as the absence of in-person interaction with teachers and the struggle to maintain focus and attention, also contribute to a decline in students' motivation and engagement (Tan, 2020). Moreover, the literature highlights an increase in anxiety and distress of students, particularly among girls (Prowse et al., 2021). Consequently, during this period, families played a central role in supporting remote learning (Gaxiola Romero et al., 2022), with educators representing key figures in ensuring educational continuity (Choi et al., 2021).

It is also important to note that numerous studies have highlighted the potential exacerbation of educational disparities by the pandemic, disproportionately affecting disadvantaged students. Specifically, these students are likely to come from low-income families (Parolin et al., 2021; Betthäuser et al., 2023), as well as having special needs or disabilities (De Witte & François, 2023; Di Pietro et al., 2020). Evidently, these students appear to have faced challenges during the period of school closure and in the use of remote learning. For instance, they are more likely to experience limited access to technology, insufficient support from educators, and difficulties in maintaining engagement and motivation.

The studies available in the literature, such as the ones briefly described in this section, represent essential contributions in providing insights into the consequences of Covid-19 for the global education landscape. However, these are mainly single-country analyses that lack internationally comparable information. For this reason, they can offer only a partial and fragmented perspective of the broader consequences of the Covid-19 pandemic on education.

The Response to Education Disruption Survey (REDS) aims at filling this gap. It represents the first large-scale international survey specifically designed to track and study the disruption in education caused by Covid-19. The database provides systematic and comparable information that can be freely accessed. Therefore, REDS constitutes a valuable source of information for on this topic, which can, in turn, offer valuable insights to support the decisions made by governments and education systems.

The relevance of this data source has motivated this special issue "on investigating the effect of Covid-19 disruption in education using REDS data”. The special issue gathers six studies that employ REDS data to assess the impact of Covid-19 on students, teachers, and schools. A brief description of the REDS database and the studies contributing to the special issue is provided in the following sections.

What is REDS?

REDS was an international comparative educational study conducted at the initiative of UNESCO and the International Association for the Evaluation of Educational Achievement (IEA) between December 2020 and July 2021 in 11 countries: Burkina Faso, Denmark, Ethiopia, India, Kenya, Russian Federation, Rwanda, Slovenia, United Arab Emirates, Uruguay and Uzbekistan. It was developed to examine how education systems in different countries coped with the challenges posed by the Covid-19 pandemic and the resulting sudden shift to remote learning. The aim of the study was to capture the perspectives of students, teachers and schools on the learning process during this difficult period. However, the study was not limited to the experience of learning during the pandemic, but also took into account respondents' general wellbeing, both emotional and physical, their relationships with others, as well as anxieties and concerns about the impact of the pandemic on their lives. While students were perhaps the most vulnerable group, interviews with teachers and school principals helped in getting a better perspective on students' learning during the period of school closure. What is more, they are a source of information on the measures schools took to mitigate the adverse consequences of the disruption.

The study was conducted in an adverse environment. At the time of its implementation, measures to prevent the spread of Covid-19 were still in place in many participating countries. Of the countries that responded to the invitation to participate in the study, only some surveyed all three target groups. For example, India and Uruguay did not survey students, while Rwanda chose only to survey schools. Moreover, the data collected did not always meet the methodological requirements set by the IEA. For example, student survey data in Burkina Faso, Denmark, Ethiopia and Kenya are not fully representative of the target population. Despite these limitations, REDS is an important source of information on how education systems in different countries coped with the challenge of providing students with the opportunity to continue learning when their school buildings were closed. Due to rich information on students' family backgrounds, collected in the student survey, REDS makes it possible to identify those groups of students who may have been at a higher risk of facing adverse consequences of the disruption. Interviews with teachers and school principals focused on measures taken by the schools to assist their students in this difficult period, but also the extent to which they were generally prepared—in terms of suitable infrastructure, technology, and skills—to make the sudden shift to remote learning and teaching. A detailed overview of the conceptual framework behind REDS, its methodology and implementation can be found in Meinck et al. (2022).

The contributions of the special issue studies

Table 1 presents a summary of the papers included in the Special Issue. Each paper is characterised by the subset of the REDS data it relies on, the geographical coverage of the analysis, substantive research focus, and the method it employs.

Table 1 Summary of the papers included in the Special Issue

Rožman et al. (2023) examine students’ psychological distress during the pandemic. Drawing from recent research on well-being, including studies on well-being during Covid-19, they build a scale of students’ psychological distress. These items asked students about how concerned or worried they were about the effect the disruption might have on their learning and their future education or about how the pandemic affected their local area or the world in general. The authors use various multivariate methods, such as item response theory models and linear regression. After building the scale and verifying that it satisfies the criteria for group comparability, the authors regress the scale on student characteristics, such as gender, age, and an index of home resources. They also include variables corresponding to students’ perceptions of school belonging, school support, or stressful situations experienced during the lockdown. The effects of these variables are found to differ between countries. For example, the level of psychological distress was lower among female students in Slovenia, Denmark, the Russian Federation, and United Arab Emirates, but in other countries the relationship was insignificant. On the other hand, the sense of school belonging had a consistently positive effect on students’ distress. The authors propose two potential explanations for their findings.

The article by Kennedy et al. (2022) investigates inequality in remote learning quality (RLQ). Building on a conceptual framework developed by UNESCO, the authors identify (a) access to suitable technology, (b) effective teachers, and (c) engaged students as three major components of RLQ which could be captured using REDS data. For each component, they construct a scale using responses to questionnaire items in the student questionnaire. In the next step, they explore within-country inequalities in the components of RLQ by regressing the estimated scores on students’ background characteristics, i.e., socioeconomic status, gender, speaking the test language at home, and urbanicity. Students in African countries (e.g., Kenya, Ethiopia, and Burkina Faso) reported lower levels of access to suitable technology than their peers in other countries. Furthermore, the percentage of students participating in remote learning in these countries was also much lower. In addition, relative to other countries included in the analysis, Ethiopia and Kenya scored lower on the scales of effective teachers and engaged students. Along with the variation between countries, Kennedy et al. (2022) point out to inequalities within countries with respect to access and confidence in technology use between groups of students. In line with studies on digital divide, students with a lower socioeconomic background or those living rural areas were more likely to face greater challenges in participation in remote learning.

Diaz Lema et al. (2023) examine teacher well-being in the Russian Federation, the United Arab Emirates, and Uzbekistan. Their study explores how school digital strategies have helped mitigate the potential negative effects of the pandemic on teachers' well-being. This research represents one of the first large-scale evaluations of the influence of school digital tools and strategies on teachers' well-being during the Covid-19 pandemic. The authors employ a two-step methodology that incorporates a mixed linear model to assess the extent to which school-level factors account for variations in teachers' well-being. Additionally, they use regression trees to identify the most relevant school factors supporting teachers' well-being. A particular point of interest in this paper is the integration of data from two different REDS surveys: the School survey and the Teacher survey. The findings indicate that schools played a significant role in influencing teachers' perceived well-being during the Covid-19 disruption, with the school level accounting for over 7% of work environment well-being and 8% of individual well-being. Moreover, digital skills and the quality of infrastructure and equipment provided by schools are the most relevant factors associated with teachers' well-being. However, in countries with less advanced technical infrastructure, such as Uzbekistan, pedagogical techniques and professional development activities for remote teaching are perceived as more crucial than technical skills.

The perspective of the teachers is also taken into account by the work of Mirazchiyski and Mirazchiyski (2023). In this case, the authors integrate all three REDS surveys, using information from teachers, school principals and students. Their investigation focuses on the educational system in Slovenia, exploring teachers', school principals’ and students' perceptions of the learning outcomes during the school closure caused by the Covid-19 pandemic. The data analysis is based on descriptive and multivariate statistical techniques that are particularly suited to the REDS categorical data used in the paper. The results indicate a divergence in perceptions between students on one hand and school principals and teachers on the other. The former tend to view their learning experiences during the Covid-19 disruption more positively, while the latter generally exhibit a more pessimistic view regarding students' learning, progress, and overall development during this period. The study also provides insight on the grading process employed by teachers during the Covid-19 disruption. Interestingly, both students and teachers acknowledged that grades awarded during this period tended to be more favourable compared to what might be expected under normal classroom conditions for the same level of knowledge, suggesting therefore a grade inflation.

Shelton and Gezer (2023) and Bertoletti et al., (2023a, 2023b) have studied student outcomes during the Covid-19 pandemic, with a focus on specific groups of students. Shelton and Gezer (2023) have examined the educational experiences of students with disabilities during the Covid-19 disruptions in five different countries: Burkina Faso, Denmark, Ethiopia, Kenya, and the United Arab Emirates. The authors use data from a dedicated module in the REDS student questionnaire to study experiences of students with disabilities during the pandemic. This module holds particular significance as the experiences of students with disabilities during the pandemic have been relatively underexplored, especially when considering a cross-country perspective. The empirical analysis relies on both descriptive statistics and t-test analyses, which have been utilised to compare students with disabilities to students without disabilities. The empirical analyses reveal that, while students without disabilities reported better learning experiences, students with disabilities had higher ratings regarding their learning progress. Specifically, students with disabilities displayed higher ratings in terms of learning as effectively as they did before the Covid-19 school disruptions and making more progress compared to their pre-disruption performance. Nevertheless, the findings of the work underscore the importance of providing additional support for students with disabilities during Covid-19 disruptions.

Bertoletti et al., (2023a, 2023b) have investigated the gender disparities in students’ outcomes during the period of school closures. More precisely, the research examines how the Covid-19 disruption affected learning outcomes differently for girls and boys, while also identifying the key factors contributing to these gender disparities. The analysis uses data from REDS Student survey covering four countries: the Russian Federation, Slovenia, the United Arab Emirates, and Uzbekistan. The results obtained from ordered logit regressions on students' perceived improvements in learning and grades reveal that girls tended to assess changes in their academic progress less positively than boys. Specifically, girls were found to be 22% (for improvement in perceived learning) and 25% (for improvement in grades) less likely to respond affirmatively compared to boys. In the second part of the analysis, the authors employ a Karlson-Holm-Breen decomposition to assess the significance of six potential factors that may explain the gender disparities in the academic outcomes. The findings highlight that family climate, physical activity/fitness, and psychological distress are the primary drivers of the gender gap in perceived learning improvement during the pandemic. In the context of this paper, the use of REDS data has been essential to provide new insight to the literature, in which cross-country evidence on the gender inequalities associated with Covid-19 was still missing.

Conclusions

Papers collected in this special issue are different from one another in many ways. They focus on different target groups (e.g., teachers, students, or subgroups of students with special characteristics), they cover different geographical areas, they use different methods of data analysis (from the simple comparison of means using t-tests to advanced modelling involving item response theory and confirmatory factor analysis) and, most importantly, they differ in terms of their substantive focus: gender gaps in student performance, students’ psychological distress, inequality in remote learning quality, teacher well-being, experiences of students with special needs, perceptions of learning outcomes by students, teachers, and principals.

Therefore, the papers included in this special issue speak to the richness of the data and demonstrate few of the many ways in which REDS could be employed in research on educational disruptions during the Covid-19 pandemic. The diversity of information, coupled with the availability of comparable multi-country data, constitutes the main value of the REDS database. In this regard, the database serves as an initial step towards fostering a more comprehensive understanding of the global effects of the pandemic on education, thereby providing valuable insights to inform decisions by governments and education systems.