Young videogamers and their approach to science inquiry

Abstract

Background

Written instructions seldom need to be read when playing videogames. Instead, gaming often involves early information foraging and expansive exploration behaviors. We use data from the Programme for International Student Assessment (PISA) to explore whether students who regularly play videogames (daily gamers) adopt behaviors that are typical of gaming while they complete a computer-based assessment of science and if such behaviors matter for performance in procedural science knowledge.

Methods

We use item-level data from PISA 2018 from countries that administered the optional ICT questionnaire. Using a sample item and the full set of interactive science items, we develop regression models to estimate differences between daily gamers and other students in information harvesting, time to first action, and procedural science knowledge. We report average associations across countries, country-specific estimates, as well as differences between boys and girls. We report raw differences as well as differences adjusting for background characteristics. To account for the multilevel nature of the data and for the two-stage sampling design, we use replicate weights supplied with the PISA database to estimate robust standard errors using balanced repeated replication (BRR).

Results

In 2018, 33% 15-year-olds reported playing videogames every day or almost every day. Among boys, that proportion was 49%. Daily gamers do not differ from other students in science content knowledge and in reading fluency. Nevertheless, daily gamers spend marginally less time reading instructions and display more active exploration behaviors in the assessment on items that include simulation tools.

Conclusions

Science teachers and assessment developers may find inspiration in games to develop scenarios in which students can practice effective strategies for information harvesting. The extensive exploration of a problem space in order to obtain data in support of future decisions often corresponds to a positive behaviour with multiple advantages in authentic problem situations. By contrast, fast transitions into action may, in the particular situation of an assessment, be an inadequate response. Assessment developers can ensure that instructions are carefully read and understood by test takers and teachers and can guide their students to read instructions adequately.

Introduction

Policy makers in OECD countries and beyond lament a misalignment between labour market needs for individuals with solid skills in science and mathematics and the number of individuals who possess such skills (OECD, 2021). Science-related employment is expected to grow worldwide (Langdon et al., 2011), but interest in science among youth appears to be declining (Galton, 2010; Vedder-Weiss & Fortus, 2011). Furthermore, participation in science remains highly unequal, with women, socio-economically disadvantaged youngsters, and youngsters with a minority or immigration background being especially unlikely to pursue careers in science or develop a level of understanding of scientific phenomena that is necessary to be active participants in societies that are deeply reliant on scientific facts and discoveries (Archer et al., 2015; Smith, 2011).

Previous research has examined the role of out-of-school activities such as playing with science kits, watching science TV, discussing science in everyday conversations, and going to science museums as precursors of students’ science capital (Archer et al., 2012). Science capital reflects an individual’s science-related qualifications, understanding and knowledge about science and acquaintance with role models who operate in science-related jobs (Archer et al., 2015). However, parents with little science capital themselves are unlikely to promote and engage in such activities with their children.

In this work, we show that students’ approach to a “science problem” is associated with their familiarity with videogames, and should this association be causal, suggest that educators may exploit video-game practice as a non-conventional form of building science capital, accessible to a wider set of students compared to traditional forms of science-capital accumulation (concerted cultivation). In particular, because videogaming behavior is not strongly related to parental levels of education or to parental occupation, educators working with children from more disadvantaged backgrounds may build on students’ experience with videogaming and develop both competence and interest in scientific forms of inquiry.

When gamers approach a videogame for the first time they are rarely confronted with instructions; rather, they are expected to understand the rules by playing. Therefore, videogames allow individuals, irrespective of their attitudes towards formal education and learning, to practice problem solving and to exercise a scientific mode of inquiry (Steinkuehler & Duncan, 2008). Gamers engage in scientific reasoning during gaming sessions and while they discuss videogames with other gamers (Steinkuehler & Duncan, 2008).

Previous work examined the effect of videogaming on self-reported general problem-solving skills (Adachi & Willoughby, 2013) and the effects of being a proficient, rather than a novel player, on the set of problem-solving strategies individuals adopted in the gaming situation (VanDeventer & White, 2002). In this work we use data from the 2018 edition of the Programme for International Student Assessment (PISA) to provide evidence from large, representative samples of adolescents, on differences in the problem-solving behaviors adopted when solving science problems between teenagers who play videogames daily and teenagers who play videogames less frequently (if at all). We make several contributions. First, we document how pervasive videogaming is worldwide. Second, we identify if daily gamers approach scientific problem-solving differently from other students and if daily gamers have a different level of science achievement compared to other students. Third, we examine gender differences in gaming and if the set of behavioral tendencies in the approach to science problems that are associated with daily gaming differ across genders. Fourth, to examine how associations differ depending on use, we illustrate differences in associations between 15-year-olds who play games daily, those who have an intermediate level of use and those who never or hardly ever play videogames.

Our target situation involves eight science tasks included in an academic test of science, which are characterized by their “interactive nature”, i.e. by the fact that the task environment dynamically responds to the test-takers’ actions, e.g. by revealing new data that were previously unavailable. These tasks are meant to simulate the process of doing science – of designing experiments, interpreting results, and making predictions informed by data and prior knowledge. The interactive nature of the tests also incorporates many of the features that are typical of videogames. We find that students who play games daily on average start exploring the problem more rapidly and seek more information from the system than what would be strictly necessary to reach a solution, although differences are not pronounced. Other things being equal, we do not find differences between daily gamers and other students in the likelihood of success on interactive science tasks, measured by their probability of solving the tasks correctly. We find that boys are considerably more likely to play videogames than girls, start exploring the problem space faster than girls, on average, and engage in greater information harvesting than girls. Finally, we find a dose-response relationship between the frequency of gaming and the behavioral tendencies observed in the science test: daily gamers exhibit stronger differences in behaviors compared to students who never or almost never play videogames than students who play videogames but not daily.

Theory and study aims

The literature has examined at length the effects of gaming on the academic achievement and mental well-being of teenagers (Drummond & Sauer, 2020; Gentile, 2009; Gnambs et al., 2020; McDool et al., 2020; Przybylski, 2014; Przybylski & Weinstein, 2017; Weis & Cerankosky, 2010). Such literature can be divided in two groups: studies that examine the indirect displacement effects of videogaming and studies that examine the direct effects of videogaming.

Proponents of the displacement hypothesis predict that videogaming will have negative effects on achievement because time spent videogaming is time not spent on activities that are strongly and positively associated with academic achievement (Weis & Cerankosky, 2010). Results in this literature are inherently relative, because they depend on the alternative uses of time and their expected achievement benefits. While self-study and doing homework may be strongly associated with academic achievement, it is possible that teenagers who play videogames would not be doing such activities if they were prevented from playing videogames (Drummond & Sauer, 2020; Hartanto et al., 2018).

The literature on the direct effects of gaming is more diverse. On the one side, researchers have examined the negative effects of gaming on attention difficulties (Ferguson, 2015; Gentile et al., 2011), violence and aggressiveness (Burkhardt & Lenhard, 2022; Drummond et al., 2020), and psychosocial health (Bányai, Griffiths, Demetrovics, & Király, 2019; Przybylski, 2014; Przybylski & Weinstein, 2017). On the other side, researchers have identified also positive effects of gaming on the acquisition of skills. For example, videogaming has been shown to promote the development of visual spatial skills (Brilliant, Nouchi, & Kawashima, 2019; Dale et al., 2020; De Lisi & Wolford, 2010; Spence & Feng, 2010) which are important in themselves and are pre-cursors for the development of skills in mathematics (Xie et al., 2020). In particular, the literature indicates that non-problematic users of videogames have improved top-down visual attention control, processing speed, response speed, ability to track multiple objects simultaneously, and an improved ability to switch tasks (Cain, Prinzmetal, Shimamura, & Landau, 2014; Dye & Bavelier, 2010; Nuyens et al., 2019; Pohl et al., 2014).

Although this last line of research shows that gamers outperform non gamers on some tasks, there is much debate regarding whether the training benefits of videogames are task-specific or task-general. Lack of transferability of skills across tasks and lack of applicability of skills developed through gaming to educational settings would render the learning gains of gaming inconsequential for education.

The learning-to-learn theory proposes that playing videogames could lead to the development of transferable skills that are important in educational settings such as improved attentional control, pattern recognition, and resource allocation (Bavelier, Green, Pouget, & Schrater, 2012; Feng & Spence, 2018; Green & Bavelier, 2012; Weinstein & Lejoyeux, 2015). Work has also identified an association between videogaming and self-reported persistence while engaging in problem solving (Adachi & Willoughby, 2013) and between videogaming and performance-based measures of persistence (Ventura, Shute, & Zhao, 2013).

By contrast, the common demands theory maintains that any post-training benefits arising from gaming will be task-specific, and that performance improvements will only be observed in tasks that share very similar cognitive demands to those involved in the training task (i.e. the game) (Azizi, Abel, & Stainer, 2018; Oei & Patterson, 2014, Oei & Patterson, 2015; Sala et al., 2018).

Empirical research on the learning effects of videogaming indicates that different games require players to practice different sets of skills, although most games encourage, to a larger or smaller degree, inductive discovery as an effective gaming strategy. Inductive discovery describes the cognitive process of formulating hypotheses about rules governing a situation, identifying patterns and implementing strategies in response to stimuli received. Players practice inductive discovery when they use input received during a gaming session to develop an understanding of how the game works. Inductive discovery means that gamers typically discover gaming strategies through a process of trial and error: by playing multiple sessions and through a wide-ranging exploration of the gaming environment. Because gamers can play a potentially infinite number of rounds, when they encounter a new game, they typically over-explore the problem space, especially in the earlier rounds of the game. Making mistakes and exploring the game in its entirety in fact allows players to simulate alternative scenarios and test hypotheses about the effectiveness of different gaming strategies (Greenfield et al., 1994). These features apply both to games in which the player needs to adapt his or her behavior to the behavior of others (in multiplayer games) and to situations that evolve based on algorithms built in the game (like in solo card games or single-player computer games).

In line with the learning to learn theory, we hypothesize that gamers will approach problem situations in which inductive discovery is a possible strategy similarly to the way in which they would approach a game, even when such problem situations arise outside of a videogame. In particular, we expect that in a computer-based assessment of science which includes simulation tools, children who play videogames regularly will be less likely to devote time to read instructions and will interact with the computer situation to a higher degree than what would be strictly necessary to solve the problem at hand. We also explore whether behavioral tendencies of gamers and other students will differ depending on the characteristics of students. Behavioral and neural evidence in fact indicates that gaming can have both positive and negative effects on attention, memory and executive control depending on the characteristics of gamers (Vedechkina & Borgonovi, 2021): individual characteristics and prior gaming experience explain some of the observed variability in skill acquisition, performance improvement, and skill transfer rates observed as a result of gaming (Spence et al., 2009).

Materials and methods

The programme for international student assessment

PISA is an international large-scale assessment that has been administered to samples of 15-year-old students every three years since 2000 and, since 2015, is administered on computers. Computer delivery allows to trace how students interact with the test questions and identify indicators that describe problem-solving strategies; it also allows test developers to create tasks that evolve dynamically, in reaction to test takers’ actions. Such interactive tasks lend themselves for example to assessing the ability of students to conduct scientific inquiry in a virtual laboratory. PISA involves large representative samples of students from countries that vary widely in cultural, linguistic and social background, pedagogical approaches used in schools and share of teenagers who regularly play videogames.

Participants

Our data come from the 2018 edition of PISA. All cases used in our analyses were extracted from the public-use files for the PISA 2018 computer-based test, which can be downloaded from: http://www.oecd.org/pisa/data/. PISA participants were selected from the population of 15-year-old students in participating countries according to a two-stage random sampling procedure, so that weighted samples are representative of students who are enrolled in grade 7 or above and are between 15 years and 3 months and 16 years and 2 months at the time of the assessment administration (generally referred to as 15-year-olds in this work). In the first stage, a stratified sample of schools was drawn. In the second stage, students were selected at random in each sampled school. Finally, on the day of the test, students are assigned to one of many distinct, but partially overlapping test forms. We focus on students assigned to a test form containing interactive science items (see section “instruments”, below): this includes about 26% of the PISA 2018 sample, when the major focus of the assessment was reading, and only a reduced sample within each school was assigned to test forms including science items.

While more than 70 national samples exist for 2018, our study is based on the subset of countries that administered the optional Information and Communication Technology (ICT) questionnaire to students. The ICT questionnaire is a 10–15 min questionnaire designed to identify technology availability and use among 15-year-olds. In total, 377,635 students were included in the PISA sample for these countries in 2018. Furthermore, for regression analyses which relate videogame practice with test-taking behaviors, our sample is restricted to the subset of students who were assigned one of the test forms containing interactive science questions (see Table 1 for final sample sizes and descriptive statistics). Since PISA assigns students to test forms at random, this subset is representative of the wider population of 15-year-old students (excluding a small percentage of students with special education needs, who were either excluded from PISA samples because no adaptation was available for them, or assigned to a shorter test and questionnaire, not containing the items and questions used in our analysis). We excluded students from the samples used for our analysis if information was missing on one or more variables used in the analysis (listwise deletion). In particular, interactive science tasks were administered to 118,389 students across the 50 countries/economies included in regression analyses. Variables used in these analyses (see full description of models and variables below) are fully non-missing for 96,410 students: listwise deletion thus results in the loss of about 19% of the sample. The corresponding numbers for analyses based on a single sample task are 49,765 (original sample size) and 42,675 (sample size after listwise deletion), corresponding to a loss of 14%.

Table 1 Descriptive statistics (international average)

Fig. 1

Trends in the percentage of 15-year-old students who play videogames daily between 2015 and 2018, by country. Note: Countries/economies are ranked in descending order of the 2018%. Only 52 countries and economies with data about videogaming frequency in PISA 2018 are shown. PISA 2015 data are not available for Brunei, United States, Albania, Serbia, Panama, Georgia, Morocco and Kazakhstan. Source: PISA 2015 and 2018 databases

Instruments

We focus on three units (groups of items built around a common set of stimuli available to students) which were included in the PISA science test in 2018 (OECD, 2019). These three units reflect the affordances of computer-based tests for the assessment of science. A common feature of these units is their “interactive” nature: among the stimuli provided to students is a simple simulation device, which students can use by manipulating inputs and running multiple simulations. In most items included in these units, students must interact with the simulation tool to generate data required to successfully answer the assessment task (we exclude from our analysis the few tasks, within these units, where the simulation tool is not available; these non-interactive tasks were presented either at the beginning or at the end of some of the units).

Although all items used in this study must remain confidential, because they continue to be used in operational PISA tests, an illustrative unit for this type of test task was released by the OECD and can serve to illustrate the main features of these units. Unit RUNNING IN HOT WEATHER can be found at http://www.oecd.org/pisa/PISA2015Questions/platform/index.html?user=&domain=SCI&unit=S623-RunningInHotWeather&lang=eng-ZZZ.

In the simplest items in these units (exemplified by Question 1 of unit RUNNING IN HOT WEATHER), students are guided in their exploration and must follow instructions; typically, a single run of the simulation tool (with the adequate settings) is sufficient to answer the question. In more difficult items (such as Question 2 in the same sample unit), students must figure out by themselves which simulations to run, and must run multiple simulations to get the right answer (i.e. they must design and carry out their own scientific inquiry). In some items (see Questions 3, 4 and 5 in the sample unit), students also had to type an answer in an open-entry field.

All students were familiarized with the simulation tool in the orientation section to the science test, before the proper test began, and through a “dummy item” at the beginning of these units (called introduction); in this dummy item, they had to run one simulation before they could proceed to the proper question items. This served to confirm that all students had located the controls for running simulations (data from the introductory item are not used).

Procedure

On the day of the test, students who were selected to take part in the PISA study sat in a dedicated room fitted with computers under the supervision of an invigilator. Participants were first administered a timed two-hour test and then a questionnaire designed to take around 30 min for completion. Participants were typically selected from different classes and grades.

Students first familiarized themselves with the PISA computer platform. They were told that the test would last for two hours, with a break after the first hour of testing, and that the test would be followed by a questionnaire. They were also given an opportunity to practice all response formats and to explore the (simple) navigation tools embedded in the test platform before starting the test. Students who were assigned to the science section of the test were also introduced to the simulation tool, and could practice running simulations before starting the test. After the two-hour test, students were asked to complete a questionnaire (whose total duration never exceeded one hour).

Students’ response data (e.g. the selected option, in a multiple-choice question), a limited set of “generic” process data including time-on-task, time-to-first-action, the overall number of actions, and a number of task-specific pre-programmed features (e.g. the number of simulation runs in interactive items) were captured by the computer platform.

Variable description

Outcome variables

We use two indicators to identify students’ problem-solving strategies: time to first action and number of simulation trials. The time-to-first-action indicator represents the time span between the moment a test question first appears on the screen (start) and the moment in which students take the first action that involves a (meaningful) interaction with the computer platform (action_x). This lag can be taken as a proxy of how much time students spend reading instructions before they interact with the problem situation or with answer fields.

The number-of-trials indicator is specific to interactive items that include a simulation tool; it represents the number of simulation runs performed by the student and can be considered a proxy of the amount of information harvesting.

Since the time to first action depends on the length of the prompt and the number of simulation runs depends on features of individual items, in order to compare the indicators across different items, we use norm-referenced scores (percentile scores). For each indicator we use percentile values based on the distribution of the underlying indicator in each country, to net out differences across countries in reading load due to, for example, language characteristics. These analyses allow to compare the behavior of different groups of students (i.e. students who engage in daily videogaming and other students) within countries but not across countries. The percentile transformation forces an approximately uniform distribution on the timing and actions data, while maintaining any mass points that exist in the underlying variable (equal values in the underlying variable are mapped the same, middle, percentile value); it also reduces the influence of any outlier on the analysis.

We also develop a measure of procedural science performance – a percent-correct score based on the same interactive items used for the behavioral analysis – to examine whether differences between daily gamers and other students in how they solve problems are also mirrored in similar differences in whether they solve these problems successfully. To the extent that inductive discovery is an appropriate procedure for the scientific problems presented, we expect a similar relationship with this performance measure as observed on behavioral indicators.

Key independent variable

Our key independent variable is a dichotomous indicator of whether students play videogames daily (value 1) or not (value 0). In the ICT familiarity questionnaire students are asked to report how often they use digital devices outside of school to play one-player games, collaborative online games and online games via social networks. Students could report playing each of such games ‘never or hardly ever’; ‘once or twice a month’; ‘once or twice a week’; ‘almost every day’; and ‘every day’. For the main set of analyses, we consider differences between students who play videogames daily and who do not play daily. We refer to students who report playing any of the three types of games daily or playing at least two of the three games almost every day as ‘daily gamers’(value 1) and those who do not as ‘other students’ (value 0). The binary classification of participants into ‘daily gamers’ and ‘other students’ is in line with our interest in students who play videogames as part of an everyday routine. The information reported by students in the questionnaire also allows to explore other margins in the distribution of students’ exposure to videogames. To identify the existence of a dose-response relationship between “gaming frequency” and the outcome variables described above, we define a three-level categorical variable which distinguishes “other students” between those who never or hardly ever play videogames and those who play videogames but are not daily gamers according to our definition. In models developed to test dose-response relationships students who never or hardly ever play videogames are the reference category and we report estimates on differences in outcomes associated with being intermediate users of videogames and with being daily gamers.

Control variables

Students’ sex was reported in the student tracking form completed by school administrators as well as by students in the questionnaire; we use an indicator variable for “boys” in our analyses (boy = 1, girl = 0). Students’ reading fluency was introduced to control for how fast students read; the measure is available only in 2018, when reading was the major focus of the assessment. The measure was derived using the total time students took to read (and understand) 22 sentences (reading fluency items). Since almost all students correctly identified the meaningless sentences among the 22, accuracy was not considered. For each student we assign a within country percentile distribution of total completion time with the fastest student being assigned a value of 100 and the slowest student being assigned a value of 0.

We introduce a percent-correct score on traditional science items, measuring students’ knowledge of science facts and theories (content knowledge), to control for science knowledge. The variable is used to confirm that the difference in procedural science achievement between daily gamers and other students is not confounded by differences in more traditional science knowledge.

We introduce a measure of total time on task to control for students’ overall response behavior. In line with the definition of the outcome variables (see above), we use the raw indicator of total time on task when analyzing a single sample task, and percentile values based on the distribution of time-on-task in each country and for each task when analyzing all eight simulation-based science tasks jointly.

Finally, we control for students’ socio-economic condition through the PISA index of economic, social and cultural status (ESCS), an aggregate indicator reflecting students’ household resources, parental educational attainment and occupational status (Avvisati, 2020) and for students’ experience with computers through the age at which students reported having first used a digital device. Students could report never having used a digital device, or having used a digital device for the first time when they were 3 years old or younger, when they were between the age of 3 and 6, when they were between the age of 7 and 9, when they were between the age of 10 and 12 or when they were 13 years old or older.

Analysis

We first report descriptive evidence on the prevalence of videogaming, recent trends in the engagement with videogames among 15-year-old students as well as descriptive statistics of participants in our sample and differences in key characteristics between daily gamers and other students. Next, we illustrate, using a sample item, differences between daily gamers and other students in how much information harvesting the two groups engaged in and the amount of time elapsed between being presented the item and the moment individuals started engaging with the item. We then develop regression analyses aimed at identifying the association between videogaming and time to first action and information harvesting across different items. In Table 2 we report average associations across countries, based on separate country-specific regressions. Average coefficients are obtained as an equally-weighted average of country-level coefficients (each country contributes equally irrespective of size of the sample or size of the underlying target population); standard errors for these averages are obtained under the assumption of independent sampling errors across countries. In Fig. 2 we present country-specific results for the two parameters of interest - difference in time to first action and number of simulation trials.

Table 2 Gaming-related differences in behavior and success in simulation-based science items (8 items)

Fig. 2

Gaming-related differences in problem-solving behavior, by country. Note: Each bar corresponds to the difference between daily gamers and other students, after adjusting for possible confounding variables (model 2), estimated on 50 national samples (countries/economies that administered the complete ICT familiarity questionnaire in PISA 2018). Statistically significant differences (p < .05) are marked in a darker tone. Source: PISA 2018 database

In addition to reporting the raw differences in the behaviors adopted by daily gamers and other students and in achievement on procedural science items (model 1), we develop models that allow to estimate these differences accounting for background characteristics (model 2) and for differences in total time on task (model 3).

We develop analyses to identify if the association between videogaming and behaviors and between gaming and achievement differs by gender. We do so by introducing interaction terms between the dichotomous videogaming variable and whether the respondent is a boy or a girl (model 3). Finally, we identify if the association between videogaming and behaviors and between gaming and achievement differs when we do not consider differences related to regular daily use of videogames but differences related to having at least some familiarity with playing videogames. We do so by developing models in which we compare behaviors and achievement of students who never or hardly ever play videogames with (1) the behaviors and achievement of students who have intermediate levels of videogaming use and (2) with the behaviors and achievement of who engage in videogaming daily.

Results

Descriptive evidence

Figure 1 indicates that gaming is a widespread activity among teenagers worldwide: the percentage of students who reported playing videogames daily increased markedly between 2015 and 2018 in most education systems with available data. In some, including Chile, Germany, Hong Kong, Japan, Korea, Macao (China), Singapore, Spain, Chinese Taipei and Thailand, it increased by 10% points or more. In all countries with available data in 2018 at least one in five 15-year-olds played daily in 2018.

Table 1 illustrates descriptive statistics for all variables used in the analysis. Daily gamers and other students do not differ, on average across countries, in terms of reading fluency, science content knowledge and socio-economic status. By contrast, daily gamers appear to have a larger number of years spent using digital devices, spend less total time on interactive tasks than other students, have lower reading achievement, and are more likely to be boys. In fact, the lower reading achievement of daily gamers reflects the gender distribution of gamers and the fact that boys are more likely to lag behind in reading than girls (Buchmann, DiPrete, & McDaniel, 2008; DiPrete & Buchmann, 2013).

Videogaming, behavioral tendencies and procedural science knowledge

Single item

We first illustrate our findings with a single item from the PISA 2018 science assessment. This item (CS615Q07TA) is similar to Question 1 in RUNNING IN HOT WEATHER: it features a simulation with multiple input variables (controlled by the test taker) and multiple output variables, whose values are shown, after each simulation run, in a table (together with the corresponding input values). Just like in Question 1 in RUNNING IN HOT WEATHER, a single simulation run, using values provided to test takers in the instructions, is sufficient to generate the data required to answer the question correctly. However, the number of simulations that test takers can run is not limited; students can freely explore the environment.

Table 1 indicates differences in behavioral tendencies between daily gamers and other students in the sample task. Daily gamers spent an average of 19.3 s before taking their first action, while other students spent an average of 22.4 s, a difference of 3 s. In addition, daily gamers logged an average of 3.5 simulation runs on the sample item, while other students logged on average of 0.4 fewer runs (3.2). This illustrative item appears to be a difficult item (few students respond correctly) and daily gamers appear to perform marginally worse: 22% of daily gamers responded correctly to the sample item, while 24% of other students did.

When we control for background characteristics in model 2 of Table 5 we observe that the difference in time elapsed to the first action and the number of actions performed between daily gamers and other students is reduced by half: other things being equal the difference in time on task associated with daily gaming is reduced to 1.6 s (from 3 s) and the difference in the number of simulation runs is reduced to 0.19 runs (from 0.37 runs). Results do not change when we further control for the total time students spent on the task in model 3. In the sample item daily gamers appear to underperform compared to other students, a difference that remains statistically significant but is quantitatively small (2.0% points when not controlling and 2.3% points when controlling for background characteristics and 2.2% points when further controlling for time on task).

General patterns

The pattern observed on this single item reflects a more general pattern, whereby daily gamers tend to log a marginally greater number of actions on interactive, simulation-based items, and to start interacting with the item earlier than other students. Table 1 reveals that when we examine all target items daily gamers are, on average, at the 48.6 percentile of the time needed to take the first action while other students are at the 51.8 percentile, a difference of over three percentiles. Across all eight interactive, simulation-based items, the average correct response rate was similar among daily gamers and other students: it was 45% among daily gamers and 44.2% among other students.

Because differences reported in Table 1 could be due to compositional differences in 15-year-old daily gamers and other students, in Table 2 we report results while controlling for background differences and differences in the total amount of time students spent on interactive tasks. Results indicate that differences in the background characteristics of daily gamers and other students explain around half of the observed differences in time to first action and in the number of actions taken by daily gamers and other students. Other things being equal, on average daily gamers are 1.4 percentiles below other students in the distribution of time to first action and 0.6 percentiles above other students in the distribution of information harvesting (number of simulations run) with similar background characteristics (results presented in model 2 of Table 2). These differences remain stable when differences between daily gamers and other students in the total time spent on interactive tasks are further controlled for in models 3 of Table 2. The difference between observed differences and differences estimates after accounting for compositional differences are mostly due to the fact that boys are more likely to be gamers but also to adopt behaviors such as a rapid transition into action and to over-explore the problem space. Other things being equal, on average boys are five percentiles below girls in the distribution of time to first action and 1.3 percentiles above girls in the distribution of information harvesting (number of simulation runs performed). Students who have greater reading fluency have a faster transition into action and display greater information harvesting and so do students with greater experience using digital devices. Students with greater science content knowledge have slower transitions into action and display greater information harvesting.

Table 2 also reveals that after accounting for compositional differences in teenage students who play videogames daily and those who do not, no differences in overall performance in procedural science knowledge could be identified: daily gamers display similar levels of achievement on these interactive, simulation-based items as other students. By contrast, we find that boys, students with greater reading fluency, students with higher content knowledge in science, students with a larger number of years spent using digital devices and students who spend more overall time on interactive science tasks have higher levels of achievement in procedural science tasks than other students with similar characteristics.

Results presented in Table 2 illustrate average findings across the 50 national samples that took part in the PISA 2018 study and administered the optional ICT questionnaire. As such, they reveal aggregate patterns across a large number of independent samples, each representing a population with different levels of prevalence of daily gamers and other students, average levels of achievement, cultural preferences and potential preference for different test-taking and problem-solving behaviors. Figure 2 illustrates country-specific results on the association between daily gaming and time to first action and between daily gaming and information harvesting after accounting for background characteristics. Results reveal a high degree of consistency in the direction of associations, although the null of no association can be rejected only in a subset of countries at the 5% level because, due to small sample size, associations are imprecisely estimated at the individual country level.

Moderating effects: gender differences in the association between videogaming, behavioral tendencies and achievement

We explore gender differences in estimated associations in Table 3. Results reveal that, other things being equal the gender gap in how fast test takers move from seeing an item to taking their first action is smaller among daily gamers than among other students. Among other students, boys are 4.5 percentiles below girls in the distribution of time to first action, on average. However, while male daily gamers are roughly similar to other male students (0.4 percentiles difference), female daily gamers are on average about 2 percentiles below other female students. Similarly, daily gaming is associated with smaller gender gaps in information harvesting: among other students, boys are about 2.4 percentiles above girls in the distribution of simulation trials; a difference that is essentially the same (2.5 percentiles) among daily gamers. Interestingly, while among daily gamers there are no gender differences in achievement in procedural science knowledge, gender differences emerge among daily gamers. Female daily gamers underperform compared to other female students (by around 1.2% points) and compared to male daily gamers, while male daily gamers outperform other male students (by around 1.1% points) and female daily gamers.

Table 3 Gaming-related differences in behavior and success in simulation-based science items, by gender

Differences in associations related to frequency of engagement with gaming

Analyses presented in previous sections identify differences between students who play videogames daily and other students. In Table 4 we present more detailed estimates that allow to identify problem-solving behaviors and achievement in interactive science tasks depending on whether students never or hardly ever play videogames; play videogames but not daily; or play videogames daily. Results presented in models 1 of Table 4 suggest that students who never of hardly ever play videogames engage in the least number of simulation trials and have the slowest transitions into action when they are presented with an interactive science problem. On average students who engage in intermediate levels of videogaming – i.e. they play videogames but not daily – are 1.4 percentiles above non gamers in the distribution of simulation trials and 2.8 percentiles below non gamers in the distribution of time to first action. Students who play videogames daily are 2.4 percentiles above non gamers in the distribution of simulation trials and 5.2 percentiles below non gamers in the distribution of time to first action. Results presented in Table 4 also indicate that students who play videogames are almost 1 percentile above intermediate gamers in the distribution of simulation trials and 2.4 percentiles below intermediate gamers in the distribution of time to first action. Importantly, all these differences (between non gamers and non-daily gamers, but also between non-daily gamers and daily gamers) are statistically significant. Controlling for student background characteristics in models 2 and further controlling for total time spent on interactive tasks reduces the magnitude of the differences but not their statistical significance. For example, results presented in models 3 indicate that intermediate gamers are 1 percentile above non gamers and daily gamers are 1.8 percentiles above non gamers in the distribution of simulation trials and that intermediate gamers are 1.5 percentiles below non gamers and daily gamers are 2.2 percentiles below non gamers in the distribution of time to first action. Table 4 further indicates that after controlling for student background characteristics and the overall time students spent on science tasks, students who never or hardly ever play videogames had lower levels of achievement in interactive science tasks than students who play videogames at least sometimes. Differences are statistically significant at the 5% level. In contrast, no differences in achievement can be identified between daily gamers and students who play games with less-than-daily frequency.

Table 4 Differences in behavior and success in simulation-based science items (8 items) related to the intensity of exposure to videogames

Discussion

Overall, the results in Tables 5, 2 and 4 support our hypothesis that daily gamers adopt slightly different problem-solving strategies when compared to other students who have similar background characteristics, although differences are not large. In particular, we find that daily gamers have marginally faster transition times between being exposed to a problem and starting to engage with it. This is not simply a reflection of the fact that daily gamers spend less time on problems in general: even when controlling for differences in the total amount of time spent on a task, daily gamers have faster transition into action. At the same time, daily gamers engage in marginally greater information harvesting than other students. This latter behavior may be of particular interest to science educators since it denotes a form of engagement with the problem and a willingness to explore a scientific situation from different perspectives. Finally, our results suggest that, while daily gamers approach problems slightly differently from other students, on average their approaches are equally effective (no difference is observed in achievement).

Table 5 Gaming-related differences in behavior and success in simulation-based science items (sample task)

Estimated associations are small according to conventional levels (Cohen, 1988), although these were reliably estimated in large-representative samples of students taking part in a low-stakes timed assessment with tasks carefully designed by international experts to measure science achievement. All assessment tasks were selected according to strict technical standards, were designed to be as clear as possible for test takers with only essential elements being presented in the initial stimuli, and did not afford unlimited opportunities for exploration of the problem space. It is possible that the behavioral tendencies that we observe in these ‘controlled’ settings may be larger in the presence of assessment tasks that allow for greater variability in engagement with the problem set. Moreover, even effects that are considered to be very small when explaining single events, can have potentially large effects for individual outcomes when these small effects cumulate over time (Funder & Ozer, 2019). This is the case in education settings in which important differences in outcomes, such as attendance in prestigious university courses, are, among other things, the result of the sum of effects arising from small differences in the likelihood of receiving top marks in each grade and thus attending more prestigious and advanced programs over the school years.

Gender specific analyses reveal differences in the association between daily gaming and problem-solving behaviors and achievement. Among non-daily gamers boys have, other things being equal, faster transitions into action and more extensive exploration. Gender differences in time to first action are only half as large among daily gamers. By contrast, the gender gap in engagement in simulation trials is similar among daily gamers and other students. Interestingly, among daily gamers boys achieve at a higher level than girls whereas we do not identify a gender gap in procedural science knowledge among other students.

Results indicate that, other background characteristics being the same, 15-year-old students who play videogames daily engage in a more extensive exploration of the problem space and move faster into action than 15-year-old students who play videogames but not daily. In turn, these students engage in a more extensive exploration of the problem space and move faster into action than 15-year-old students who never or hardly ever play videogames. No differences in achievement in interactive science tasks were identified between daily gamers and students who have intermediate levels of use, whereas both groups outperform students who never or hardly ever play videogames, although the difference is quantitatively very small and statistically significant only at the 5% level.

The extensive exploration of a problem space in order to obtain data in support of future decisions often corresponds to a positive behaviour with multiple advantages in authentic problem situations, especially in the context of science, where formulating hypotheses through observation is key. A key component of science literacy is the understanding that scientific knowledge is complex, tentative and evolving and that scientific hypotheses should be formulated on the basis of theory and observation and validated through evidence. Alongside knowledge about science, the willingness to practice scientific experimentation is key if children are to pursue science in their studies and careers but equally if they are to be able to adequately evaluate the quality of information they encounter in everyday life. Active learning approaches that encourage learners to explore a system (even if this means making mistakes or taking longer to reach a solution) have been shown to be superior to learning based on following instructions and avoiding making mistakes, especially in novel situations (Bell & Kozlowski, 2008). Exploration and experimentation during the learning process activate individuals’ metacognition, i.e. their capacity to plan, monitor and revise behaviour given emerging stimuli (Bell & Kozlowski, 2008) and, by so doing, enhance learning and transfer (Keith & Frese, 2005). Social, technological, and economic transformations reduce the need for individuals to memorise facts while yielding increasing returns to those who are able to explore problem spaces in innovative ways (OECD, 2013). Technological innovations are reshaping the skills that are needed to participate successfully in the labour market so that there is now a markedly higher share of nonroutine tasks, i.e. tasks for which the capacity to practice inductive reasoning is beneficial (Autor, Levy, & Murnane, 2003; Ikenaga & Kambayashi, 2010; Spitz-Oener, 2006). Science teachers as well as assessment developers may find inspiration in games (and videogames) to develop scenarios in which students can practice effective strategies for information harvesting.

Fast transitions into action may, in the particular situation of an assessment, be an inadequate response. Our study focuses on relationships observed in the teenage years, when many teenagers and their families make important educational, training and labour market decisions, decisions that are often determined by the opportunities they have because of their achievements in tests and assessments. In the teenage years the executive function of inhibitory control is still developing and many teenagers experience, as a result of these neurological changes, increased impulsivity, difficulty in evaluating long-term benefits vis a vis short term costs (Sapolsky, 2017). These effects may affect all but may be especially marked in some. In particular, daily gamers may be especially susceptible to impulsivity and restlessness and, as such, may fail to put an adequate amount of time reading instructions when completing assignment or doing other work for school.

To the extent that our finding on the faster transition into action among daily gamers applies to all tests, rather than narrowly to science tests administered in low-stakes settings, it could inform the design and administration of tests and assessments. Even if results were to reflect behavioural tendencies of individuals who are likely to become regular videogamers rather than causal effects of gaming, they suggest that some students spend too little time understanding what is required of them in the assessment situation. If results were causal, since very frequent videogaming is increasingly prevalent, a growing number of teenagers can be expected to engage in behaviours that lead them to spend too little time on familiarising themselves with the requirements of the tests. This is especially relevant since in recent years, tests and assessments administered in school have become more diverse, in order to exploit the affordances of computer-based assessments, and students cannot rely on their experience of past tests to understand what is required to solve a problem. Modern, computer-based tests include tools such as simulations, scenarios, and games that replicate the diverse and rich contexts of performance in real life (Quellmalz and Pellegrino, 2009).

Assessment developers can ensure that instructions are carefully read and understood by test takers, particularly when test questions or what is required of test takers are significantly different from what is generally expected of them in tests or from what was expected of them in previous questions in the same test, i.e. if they deviate from the usual status quo. Similarly, teachers and other education professionals can provide additional input and support to ensure that instructions are adequately understood by all, together with feedback on how many students fail their tests because of lack of understanding of what is required rather than ability to solve the test.

Limitations and future directions

Our study suffers from a number of limitations which could be addressed in future research. First, the evidence we present is descriptive and does not establish a causal link; multiple explanations are possible for the associations found. Future work should attempt to complement our observational evidence with experimental or quasi-experimental evidence. Second, questions contained in PISA 2018 on videogaming do not allow to construct a precise indicator of how much time students spent playing videogames. In the absence of precise information on use, we focused our study on examining associations between daily videogaming and the behavioral tendencies of students in the PISA science test. We complemented these analyses with analyses that identify differences in behavioral tendencies and achievement between students who never or hardly ever play videogames, students who play videogames but not daily, and students who play videogames daily. These results suggest a dose-response association: results indicate that daily gamers engage in the largest number of simulation trials and have the fastest transition into action, students who never or hardly ever play videogames engage in the smallest number of simulation trials and have the lowest transition into action and intermediate gamers being in the middle. Future research could estimate more precisely the association between videogaming and behavioral tendencies, with a focus on establishing if associations are linear or, in fact, depend on the intensity of use. Third, PISA data do not contain any information on the type of games different individuals play, and it is therefore impossible within our study to establish if behavioral differences observed, especially differences between boys and girls, reflect differences in the types of games that they typically played or if they reflect other dimensions across which boys and girls differ. Future research could consider if associations depend on the type of videogames children play. Fourth, our results reflect behavior observed in the context of the administration of the PISA test in 2018. While a wide range of national contexts are covered in this study, general conclusions that refer to age-groups, countries, or periods that were not observed must remain cautious.

Conclusions

In schools, inquiry-oriented pedagogical approaches have been embedded in day-to-day activities to promote learners’ active and personal construction of knowledge (Lee & Songer, 2003; Shymansky et al., 1990). However, the evidence over the effects of such approaches on science literacy, interest in, and engagement with science is mixed (McConney et al., 2014) and teachers often indicate that they lack adequate resources to incorporate inquiry-oriented approaches (Hofstein & Lunetta, 2004). Virtual experiments have been proposed as alternatives to physical manipulation and research indicates that such experiments can be just as effective as real-life experimentation in promoting conceptual understanding of science (Zacharia & Olympiou, 2011), but have been found to be less effective in promoting motivation (Corter et al., 2011).

In this work we built upon prior work identifying a strong association between videogaming and self-reported problem-solving skills (Adachi & Willoughby, 2013) to identify if daily gamers differ from other students in how they approach scientific problems, and if an activity that appeals to a broad array of children from diverse backgrounds could be used to promote active engagement with science problems and thus to build science capital. We relied on an interactive assessment administered to representative samples of 15-year-old students in 50 education systems worldwide to assess which behavioral tendencies daily gamers display and if these translate into higher or lower achievement in science.

Already before the COVID-19 pandemic, videogaming was popular worldwide. During the COVID-19 pandemic, existing trends accelerated and in 2020 videogaming was one of the fastest growing forms of entertainment (Witkowski, 2021). Much of the debate in the popular press and the academic literature on videogames has focused on the effects of gaming on physical health, mental health and academic achievement. Despite popular claims on the negative consequences of videogaming for children’s cognitive development and their well-being, the research literature indicates that videogames can effectively develop several cognitive skills, such as executive control as well as visual and attentional skills (Basak et al., 2008; Green & Bavelier, 2012). Our research suggests that young people who play videogames daily have slightly different behavioral tendencies when they approach solving interactive science tasks in an assessment setting but have similar levels of science achievement as other students. This information could help educators promote greater engagement with science and build science capital among young people.