- Research
- Open access
- Published:
Influences of internet access on civic knowledge measurement in Taiwan
Large-scale Assessments in Education volume 12, Article number: 20 (2024)
Abstract
Internet-related issues have influenced how civic knowledge is educated and measured. The International Civic and Citizenship Education Study (ICCS) is a well-known large-scale assessment concerning how civic knowledge is educated and measured globally. Regardless of the emerging roles of internet access and usage, the influences of internet access on civic knowledge have yet to be investigated in ICCS research. Hence, this study aims to study whether the multiple-choice items in the civic knowledge test of ICCS 2016 are affected by internet access, what causes the effect, and to what degree Internet access influences the measurement. Results indicated that the ICCS civic knowledge test included six differential item functioning (DIF) items on internet access, of which five favored families with internet access, and one favored families without internet access. In addition, the primary source of DIF items was highly related to socioeconomic status. Overall, the students in families where the internet was available possessed higher civic knowledge than the students who were not. In conclusion, this study provides evidence of item inequivalence and identifies suspicious sources. These results can be used as a basis to investigate related research on digital learning, online teaching, and social media engagement in civic literacy.
Introduction
Civic education serves as the bedrock of effective democratic citizenship, laying the groundwork for informed and active participation in society (Muleya, 2017). Helvoort (2018) highlights the significance of civic knowledge within the broader framework of civic literacy. It encompasses knowledge about the rights and responsibilities of citizens, the structure of government, and the history of civic struggles and achievements, and equips individuals with the requisite understanding for meaningful participation in democratic processes. Accordingly, civic knowledge plays a fundamental role in fostering civic engagement and participatory democracy and resonates deeply within both the educational sphere and broader society (McCabe et al., 2017).
The International Civic and Citizenship Education Study (ICCS), conducted by the International Association for the Evaluation of Educational Achievement (IEA), assesses the civic knowledge of Grade 8 students on a large scale. The last iteration of this program was conducted in 2016, which involved 94,000 students from 24 countries (Schulz et al., 2018). Considering the recent conclusion of this program (in 2022), investigation of the factors affecting civic knowledge based on existing data becomes crucial.
Research has shown that students’ gender and school location influence students’ civic knowledge (Cicognani et al., 2012; Metzger & Ferris, 2013). In addition, internet access can also be essential for learning civic knowledge. Past research has highlighted the vital role of the Internet in helping students improve their civic knowledge by searching for information or news on the Internet (Gottfried et al., 2017; Partheymüller & Faas, 2015). Given the increased prevalence of online teaching trends that require internet access (Engin & Donanci, 2015) and online learning tools (Fu & Hwang, 2018), students without internet access at home may face challenges in civic knowledge acquisition through online sources. More importantly, the availability of online learning methods can dominate students’ attitudes toward civic education (Virgiawan & Sundawa, 2022). Considering the fact that equal access to the internet remains a challenge even within developed countries, internet access should be considered a critical variable when measuring civic knowledge. The previous literature has focused on the correlation between internet accessibility and the fields of science, mathematics, and reading (Courtney et al., 2022; Gubbels et al., 2020; Lorah, 2018). The presented study aims to fill the research gap by delving into the impact of internet access on students’ civic knowledge in Taiwan using data from ICCS 2016.
The investigation of the influence of background variables (internet access in this study) embraces a common approach of comparing the mean difference between group test scores. Notably, such comparisons are meaningful only if the assessment tool is fair (i.e., unbiased) to examinees in different groups. Thus, testing the measurement equivalence is essential. The ICCS 2016 surveyed internet access availability for students at home (i.e., yes or no) and measured their civic knowledge through eight booklets. For each booklet, Taiwanese students with internet access at home exhibit typically higher mean scores in civic knowledge items in comparison to those without internet access. Such comparisons are however not exhaustive and have at least two major limitations. First, we need to discover whether the test items were equivalent for the two groups. Second, although the eight booklets were linked with common items, the (raw) test scores were not comparable across booklets because they were not calibrated on the same scale.
To investigate the equivalence of items, a differential item functioning (DIF) assessment has been conducted in the test. In large-scale assessments such as the Programme for International Student Assessment (PISA), the Trends in International Mathematics and Science Study (TIMSS), or the ICCS, DIF assessment has become a standard procedure (e.g., Schulz et al., 2011; Schulz et al., 2018; Martin et al., 2020). Recently, each country has employed standard techniques to avoid DIF among genders and ethnicities (Cuellar et al., 2021). Hence, although DIF items can be easily identified using existing methods, in-depth exploration of the underlying causes of the occurrence of DIF becomes critical. In other words, a post hoc explanation of what causes the item(s) to display DIF is more valuable (Zumbo, 2007). However, only gender was assessed for DIF in the ICCS 2016 technical report without further investigating the root cause of the gender DIF displayed by the item. Therefore, an additional analysis (i.e., DIF source investigation) should be conducted to address the limitations mentioned above in the methodology when conducting a DIF-related study.
Accordingly, the current study aims to determine the existence of DIF among students with and without internet access. Moreover, multiple analyses will be executed to provide further evidence regarding the influence of internet access on civic knowledge measurement.
Literature review
Internet access and students’ civic knowledge
Civic knowledge learning involves the study of all kinds of information related to society (Greenberg, 1970). Students need to have a solid foundation of civic knowledge and comprehension to allow them to construct well-informed ideas and perspectives, understand the institutions and operations of government, and engage in civic and political life actively (Lo & Adams, 2018). They can create this foundation by reading significant civic publications, gaining an understanding of ideas that stimulate in-depth thoughts about social concerns, and developing the ability to establish connections between significant historical events and their contemporary relevance (Zorwick & Wade, 2016). Students can improve their knowledge of relevant topics throughout the curriculum by using civic engagement as an effective educational technique for addressing real-world situations (Kelley & Knowles, 2016).
The phenomenal growth of the Internet and telecommunications has made global information resources readily accessible in homes and schools (Diamond, 2020). The internet has heralded the rise of a new form of knowledge production and distribution and a virtually unlimited wealth of information resources with simultaneous availability worldwide (Gruchel et al., 2022). This phenomenon has significantly changed students’ knowledge acquisition patterns globally (Diamond, 2020), particularly empowering them to access all kinds of information and learning materials regardless of the limitations of space and time (Chen & Tsai, 2021). The Internet has been confirmed as one of the primary sources for students to gain knowledge (Diacopoulos & Crompton, 2020). Previous studies have examined the relationship between online news and individuals’ political knowledge acquisition (Gottfried et al., 2017; Partheymüller & Faas, 2015) as well as the differential effects of different media formats on political information acquisition (Dimitrova et al., 2014; Weaver & Drew, 1993). It has also been demonstrated that using social networking services (SNS) to meet new people positively correlates with bonding and bridging social capital in the virtual world, which fosters online civic involvement (Zhong, 2014). Moreover, in a globalized era, the study also indicated that multicultural exposure, multicultural interaction, and social media usage are positively correlated with global civic engagement (Tarman & Kilinc, 2022). These can be easily achieved when students have better internet access.
The internet and media are more than a source of civic knowledge learning; they also serve as a stimulus that propels students' interest in learning civic education (Pang, 2018). Moreover, the Internet is also closely related to online teaching and learning tools that have recently been widely employed and are readily accessible on mobile phones and tablets (Engin & Donanci, 2015; Fu & Hwang, 2018). McGrew et al. (2019) observed that American students’ media literacy and civic online reasoning levels were low. They further emphasized that more comprehensive research-based approaches to digital education were necessary to equip students with the IT abilities required for their active participation in civic life. In other words, the internet gap or students’ lack of IT ability or media literacy could impede the development of their civic knowledge.
In contrast, Virgiawan and Sundawa (2022) analyzed the obstacles to studying civic education and rationalized the relevance of building students' character in online learning environments. The results showed that innovation in online learning methods and models, coupled with teachers’ commitment to establishing discipline in learning civic education, can overcome the challenges that could hinder students' sense of responsibility and democratic attitude development in civic education. Specifically, students with no internet access at home will be excluded from all these civic knowledge acquisition and online learning opportunities (Kewajiban et al., 2018). The internet gap threatens students’ learning outcomes, which could be further intensified during situations like the COVID-19 pandemic or other similar scenarios (Nischal, 2022).
Students’ family backgrounds and their internet access, utility, and civic knowledge
Students’ socioeconomic background, internet access, and civic knowledge
Students’ family backgrounds can influence digital divides, internet access, and civic education. Graves et al. (2021) indicated that rural students face more trouble accessing the technology and connectivity needed for remote learning than their urban peers. Inadequate internet infrastructure in rural areas hinders remote learning. Therefore, initiatives for policy improvement and technology advancement must consider regional differences to achieve educational equity. Furthermore, Metzger et al. (2020) examined the differences in civic beliefs and efficacy among youth living in rural or city areas and from higher or lower socioeconomic households. Youth from more educated households viewed social movement engagement as more important and morally worthy, whereas rural youth and youth from less-educated households had lower political efficacy and rated standard political activity as less important and obligatory. These findings highlight the importance of community and socioeconomic context for the civic development of the youth.
The influence of students’ socioeconomic backgrounds has emerged in several other ways. Zhong (2014) found that the direct effect of using SNS to meet new social ties on civic engagement was partially mediated by social capital, while the effect of using SNS to connect with friends on civic participation was completely mediated by social capital. Since family capital, including social capital, is considered an important factor for the educational achievements of students from different social backgrounds (Liu & Gao, 2011), it is argued that students’ socioeconomic background can also affect the relationship between SNS use and civic engagement, as mentioned above.
Parenting, internet utility, and civic knowledge
The influence of students’ socioeconomic background can be traced back to their internet usage patterns determined by parenting styles. As summarized previously, internet access provides more opportunities for students to retrieve additional learning materials and information, but the kind of content students browse can be another concern. Internet use is significant in children’s everyday lives and can be distinguished into different uses (e.g., entertainment-related, informative, or school-related) (Gruchel et al., 2022). Several studies have supported the idea that internet utilization is most prevalent among younger and more educated individuals (Hoffman et al., 2000). Alshahrani et al. (2017) concluded that the use of the internet profoundly impacted students’ academic self-confidence, self-reliance, and student–lecturer connectedness. This is because using the internet on their mobile phones allows students to search and access academic information instantly (Mamudu & Oyewo, 2015).
Moreover, students believe that the use of the internet enables them to perform research ahead of time and tackle multiple home assignments along with widening the scope of reading and learning, promoting self-learning, encouraging and fostering peer learning, and enhancing student examination preparation (Apuke & Iyendo, 2018). They also believe that the internet helps them to discover new knowledge related to their courses, individually or as a group. They feel that the availability of online video tutorials on platforms like YouTube broadens their academic research and helps them to be conversant with certain practices and technical aspects of their various courses. These results support Hamid et al.'s (2015) findings, which established that internet utilization helps Australian and Malaysian students share materials for a group assignment, which leads to their improved self-directed learning. In other words, if students use the Internet for civic knowledge-related learning, they can broaden the scope of content they receive from school teachers and increase their active, spontaneous learning, which can further enhance their learning outcomes in civic education.
When reviewing the literature on Internet inquiries, the family holds particular importance as an informal place of learning. Parents' motivational beliefs are associated with parental instruction in children's practical and school-or-learning-related internet use (Gruchel et al., 2022). Specifically, parents’ perceived parental instruction predicts entertainment-related internet use negatively and informative or school-related use positively. Moreover, parents can guide their children's internet use for educational-related purposes. However, Zhong (2014) indicated that using SNS to meet new people was positively correlated with bonding and bridging social capital in the virtual world, which was positively related to online civic involvement but not significantly related to offline civic behaviors. In other words, when students have internet access at home and they browse civic knowledge content under their parents’ guidance, their civic literacy can be potentially enhanced further by engaging in real-life civic activities and learning materials.
The role of parents in promoting students’ civic engagement and learning is also evident in patterns of their discussions on civic events with their children. In their study, Chan and Mak (2020) concluded that adolescents’ political engagement and participation were influenced by parental participation, and parents with higher socioeconomic status tended to have higher political engagement. Oosterhoff and Metzger (2016) further examined the associations between mother–adolescent communications and teens’ own judgments about civic duty and behavior. They argued that mothers’ involvement in community service and messages concerning respect and helping others were positively associated with teens’ community service behavior.
In summary, students from a higher socioeconomic family tend to have better access to the internet at home, which enables them to browse all online resources and experience more multicultural exposure, multicultural interaction, and social media usage that enhances their local and global civic engagement. Moreover, parents could be more aware of guiding students toward more productive use of the internet for school- or learning-related tasks and spend more time discussing and participating in civic activities with them. All these factors may equip students with better civic literacy and engagement.
Differential item functioning and its implementation
DIF assessment methods can be roughly classified into parametric and nonparametric approaches. In parametric methods such as the logistic regression method (LR; Swaminathan & Rogers, 1990), the multiple indicators and multiple causes (MIMIC) model (Shih & Wang, 2009; Woods, 2009), and the likelihood-ratio test method (LRT; Thissen et al., 1988), the parameter associated with DIF is modeled for direct estimation of the size and significance of DIF for every studied item. However, such methods are relatively complex for users to operate. In contrast, nonparametric methods, such as the Mantel–Haenszel (MH) method (Holland & Thayer, 1988) and the odds ratio (OR) method (Jin et al., 2018), do not have a statistical model for observed responses; hence, users can easily calculate the statistic to judge whether the studied item exhibits DIF or not. In particular, the OR method can accommodate sparse data matrices in large-scale tests (e.g., ICCS) and is user-friendly (Jin et al., 2018). Hence, this study implemented the OR method to assess DIF.
Investigating the influence of internet access on civic knowledge measurement
To summarize, the preceding sections have highlighted past research regarding internet access, civic knowledge, and the methodology for investigating the background variables. Our examination of the existing literature reveals several key trends in the field. Building upon these insights, this study seeks to address the following research questions:
-
a)
Were the items measuring civic knowledge in the ICCS 2016 fair to students regardless of whether they have internet access at home or not?
-
b)
What factor(s) makes the items unfair for students who do not have the internet at home?
-
c)
To what extent did students who used the internet at home have a higher level of civic knowledge than those who did not?
Note that in this paper, "fair (unfair)" refers to the equal (unequal) probability of students from households with and without internet access answering specific civic knowledge items correctly, assuming their civic knowledge abilities are equal.
Material and methods
Instrument
The civic knowledge test for ICCS 2016 comprised 79 multiple-choice items and 9 constructed-response items. All items were assigned to eight blocks, and eight booklets were created with three blocks each. Since the descriptions of construct-response items are not available publicly, only the multiple-choice items with complete descriptions of each option in the codebook were emphasized in this study.
Students’ background variables were collected through international student questionnaires. Notably, students were asked to provide a binary response (yes or no) to IS3G13 (Do you have an internet connection at home?). Thus, the binary response was used as the basis variable for grouping students to assess the DIF items.
To provide a meaningful explanation for the possible DIF items, several background variables were selected from both student and school questionnaires (see Table 1). These variables focused on devices, students’ socioeconomic status, and their interactions with parents. Most background variables were rated directly by participants on a four-point Likert scale, with the sole exception of the SES index (S_NISB), which is an integrated variable. The SES index was derived from the highest occupational status of parents, the highest educational level of parents, and the number of books at home, and the algorithm is presented in the technical report (Schulz et al., 2018) as follows:
-
1.
Recode each indicator and conduct an imputation with a regression method.
-
2.
Convert three indicators into z-standardized variables.
-
3.
Conduct a principal component analysis of these indicator variables.
-
4.
The SES index consists of factor scores for the first principal component.
Sample
A total of 3953 students in Taiwan participated in the ICCS 2016 study. After excluding 15 cases with missing or invalid responses to the grouping variable, responses of 3605 students who answered ‘yes’ and 333 students who answered ‘no’ to IS3G13 were retained to investigate the effect of "has/does not have the internet at home" on civic knowledge measurement. Notably, the disparity ratio between groups is not uncommon in the education field. For example, a test item may be interpreted differently by aboriginal (minor group) and non-aboriginal (major group) groups (Guhn et al., 2007; Marotta et al., 2015).
Procedure
To reach the study goals, we conducted multiple statistical analyses sequentially, including DIF assessment, latent regression analysis in the Rasch model (Wilson et al., 2008), logistic regression, and correlation.
DIF Assessment
The OR method (Jin et al., 2018) was implemented to detect DIF items. As mentioned earlier, the eight booklets of the civic knowledge test were linked with common items, suggesting that the combined data matrix was sparse and had several missing responses. Such data properties make common DIF assessment methods cumbersome and less effective. The OR method is especially suitable for a huge dataset with missing responses (Jin et al., 2018).
In the OR method, the ratio of the odds of success for two groups (e.g., with vs. without internet access) is computed for each item as follows:
where \(\widehat{\lambda }_{i}\) is the statistic of item i for DIF assessment; \({n}_{R1i}\) and \({n}_{R0i}\) are the numbers of students in the reference group who respond correctly and incorrectly to item i; \({n}_{F1i}\) and \({n}_{F0i}\) are the numbers of students in the focal group who answer correctly and incorrectly to item i, respectively. Furthermore, a string of \(\widehat{\lambda }\) values is assumed to follow a normal distribution with a mean of \(\uplambda\) asymptotically. When data follow the Rasch model without DIF items, \(\widehat{\lambda }_{i}\) is an unbiased estimate of \(\uplambda\) (the mean ability difference between the reference and focal groups, usually called impact in DIF literature), with a standard deviation of \(\left( {\widehat{\lambda }_{i} } \right)\):
To have a reference to assess DIF items, the median of \(\widehat{\uplambda }\) values (\(\widetilde{\widehat{\uplambda }}\)) can be used as a robust estimator (Jin et al., 2018). Thus, item i will be deemed DIF when \({\widehat{\uplambda }}_{i}\) is far away from \(\widetilde{\widehat{\uplambda }}\). Specifically, the item is identified as DIF when the distance between the \({\widehat{\uplambda }}_{i}\) and \(\widetilde{\widehat{\uplambda }}\) is significantly different from 0, and the absolute value of the distance (effect size) is larger than 0.3 (Schulz et al., 2018). In this study, the reference group denoted those students who had internet at home, and the focal group denoted students who lacked internet. When \({\widehat{\uplambda }}_{i}\) is largely higher than \(\widetilde{\widehat{\uplambda }}\), item i favors the reference group. Likewise, when \({\widehat{\uplambda }}_{i}\) is lower than \(\widetilde{\widehat{\uplambda }}\), item i favors the focal group. The scale-purification procedure (Navas-Ara & Gomez-Benito, 2002; Jin et al., 2018) was also implemented to control the Type I error rate.
Considering that a large group mean difference can bias the estimate of \(\widetilde{\widehat{\uplambda }}\) (DeMars, 2021), we also employed the LR and LRT methods as well as the Rasch DIF modeling approachFootnote 1 (Wang, 2008) in R (version 4.3.2) to triangulate the findings. Because students were assigned to different booklets, raw scores cannot be used to present students’ abilities. Thus, to perform the LR analysis, the person estimates from the Rasch analysis (assuming all items are DIF-free) were entered as the matching variable using the difLogistic function in the difR package (Magis et al., 2010). Also, the difLRT function in the difR was used to conduct the LRT analysis. The Rasch DIF modeling approach was realized using the tam.mml.mfr function in the TAM package (Robitzsch et al., 2022).
Investigating the possible sources of DIF
To avoid collinearity, a two-stage strategy was adopted to gain a deeper understanding of why students performed differently on the DIF items. The background variable might be the possible source of the DIF item when the background variable is predictable for the DIF item and related to the grouping variable. In the first stage, the test score of a suspected DIF item was regressed by the selected background variables (Table 1). In the second stage, the relationship between IS3G13 and significant background variables was examined. As S_NISB is a continuous variable, the point-serial correlation between IS3G13 and S_NISB was calculated. However, for the other (categorical) background variables, a chi-square test of independence was performed. The target background variable(s) are likely to be the source(s) of DIF when a correlation exists.
Latent regression analysis in the rasch model
The comparison of test scores between groups is meaningful only when all the test items are DIF-free. Thus, after removing the suspected DIF items, we applied the Rasch model to the DIF-free items and performed a latent regression analysis (with the binary response to IS3G13 as the independent variable) using the R package TAM (Robitzsch et al., 2022), so that the group difference (in logit) can be directly estimated in one go.
Results
DIF Assessment
Unfortunately, the LRT method failed to converge within 72 h, possibly due to the large number of missing values in the data matrix. Hence, only the DIF detection results of the OR, LR, and Rasch modeling methods were compared.
The \(\widehat{\uplambda }\)s for the 79 multiple-choice items were between –0.42 and 0.79 (Mean = 0.24, Median = 0.19, SD = 0.27). Figure 1 further shows a negatively skewed distribution of \(\widehat{\uplambda }\)s, implying that using the median to express central tendency is more appropriate than the mean. Therefore, \(\widetilde{\widehat{\uplambda }}\) (= 0.19) was used as the reference in the initial stage of the OR method. Six items were initially screened out as DIF, and the results did not change after scale purification. The \(\widetilde{\widehat{\uplambda }}\) for the remaining 73 DIF-free items was 0.17.
As listed in Table 2, five items had a \(\widehat{\uplambda }\) value significantly higher than 0.17, whereas one item had a \(\widehat{\uplambda }\) value significantly lower than 0.17. Moreover, the DIF size for the six items exceeded 0.3 logits uniformly. To sum up, six items were suspected of displaying DIF in the civic knowledge test.
Five out of the six DIF items favoring students accessing the internet at home are related to social responsibility, government action, environment, and election. The only item (CI3CAM1) favoring students who could not access the internet at home was related to “art as a political expression.”
Common DIF items were also identified in other analyses. In the LR analysis,Footnote 2 three of them (CI2ECM1, item CI3CAM1, and CI3CRM2) were found to exhibit DIF. In the Rasch DIF modeling approach,Footnote 3 all six items were flagged. Therefore, these six DIF items were carried forward to the next analysis.
Investigating the possible source of DIF
A series of logistic regressions were completed to investigate how the background variables influenced the probability of answering the DIF item correctly, in which the binary scores on DIF items were regressed by one background variable at a time. As shown in Table 3, some DIF items’ scores could be explained by selected background variables. Scores on CI2CCM1 were associated with S_NISB. Scores on CI2ECM1 and CI312MI were associated with IS3G14E, and scores on CI3CRM2 were associated with S_NISB and IS3G12B. These six background variables could not explain the other two DIF items.
The correlation results for background variables are shown in Table 4. The highest correlation was 0.27 (p < 0.001), which occurred between "Number of tablets or e-readers (IS3G12B)" and "SES (N_SIB)." Logically, families with high SES are more likely to afford tablet devices and e-readers than those with lower SES. Hence, the potential sources of DIF items focused on SES.
In conclusion, differences in SES might cause DIF in CI2CCM1 and CI3CRM2, and the frequency of “talking about what is happening in other countries with parents” might be the source of DIF for CI2ECM1 and CI312M1. Detailed elaborations of the four DIF items are provided in the discussion section.
Rasch analysis and latent regression on internet access
For the remaining 73 DIF-free items, the difficulties ranged from − 4.01 to 0.46 (M = − 1.82, SD = 1.01), indicating that the items were relatively easier for the students. The item’s unweighted mean square (i.e., outfit) statistic was between 0.60 and 1.27 (M = 0.95, SD = 0.18), while the weighted mean square statistic (i.e., infit) was between 0.84 and 1.20 (M = 0.99, SD = 0.08). Based on the reasonable range for outfit and infit (from 0.6 to 1.4) suggested by Bond et al. (2021), all items fell within an acceptable fit. In the latent regression model, the estimate of the group difference in civic knowledge was 0.22Footnote 4 logits (p < 0.001), indicating that the students who accessed the internet at home outperformed their counterparts in the civic knowledge test.
Discussion and conclusions
Discussion
This study complemented ICCS research on DIF assessment involving internet access to civic knowledge and identified six DIF items for the Taiwan sample. Specifically, five of the six items that signify social issues, such as corporate citizenship, asking for advice, government action, environment, and elections, exhibited DIFs and favored students in internet-using families. The other DIF item related to art and political expression favored students in non-internet-using families. These findings confirm that internet access can have both positive and negative impacts on civic knowledge. As an international, large-scale assessment, the ICCS may necessitate additional scrutiny regarding potential cultural and social biases inherent in the DIF items. Given the ICCS's global scope in assessing students' civic literacy, disregarding these DIF items could result in inaccurate conclusions regarding student performance, leading to either overestimation or underestimation.
On the other hand, schools can harness this positive potential by integrating online platforms, interactive simulations, and multimedia resources to enhance students’ learning experiences and deepen their understanding of civic concepts (Apuke & Iyendo, 2018). Furthermore, the government should take proactive measures to ensure equitable access to high-quality educational resources, including the internet. This could involve initiatives to bridge the digital divide by providing internet access and devices to underserved communities (Graves et al., 2021). By addressing disparities in access, policymakers can help ensure that all students have the opportunity to benefit from online resources and participate fully in formal and informal civic learning inside and outside the school. Therefore, parents should be attentive to signs of Internet Addiction Disorder in their children, ensuring that excessive internet usage does not impede their engagement with the real world and meaningful experiences.
Contrastingly, the only DIF item related to art and political expression favored students in non-internet-using families. Political expression encompasses a broad spectrum of activities, including participating in elections, attending protests and rallies, writing letters to elected officials, or engaging in discussions about public policy and governance. In this case, students from families with internet access might spend more of their leisure time on online activities, potentially preventing them from participating in person in the aforementioned activities, thereby losing the opportunity to immerse themselves in political expression through artwork. In other words, internet access can serve as an alternative source to retrieve civic education content and a vehicle for civic knowledge learning (Alshahrani et al., 2017). However, it can also facilitate entertainment-related use (Gruchel et al., 2022), which becomes a negative factor hindering student learning. In this case, parents’ guidance to their children on how to use the internet would be a key area for improvement. For example, parents should encourage their children to manage their time to allow different types of activities, including visits to galleries and museums. Additionally, parents might also direct their children to browse art and political expression online, which can also benefit their learning in this regard. Moreover, the government can support initiatives promoting digital literacy and responsible online behavior through public awareness campaigns, teacher training programs, and curriculum development endeavors. By investing in digital literacy education, policymakers can help equip citizens with the skills they need to navigate the digital world effectively and contribute to civic life positively.
To trace some potential factors for the six DIF items, this study also found that some students’ background variables and internet access could concurrently explain the DIF items, including students’ family socioeconomic status (DIF items “affection by damage to the environment” and “corporate citizenship”), the number of tablet devices or e-readers in students’ homes (DIF item “corporate citizenship”), and frequency of talking with parents about the events occurring in other countries (DIF items “reason to ask for advice” and “democratic government action”). This reinforces the argument that family background can influence students’ learning in many direct or indirect ways.
It has been argued that students’ family socioeconomic status is related to their internet access at home (Graves et al., 2021) and civic engagement (Metzger et al., 2020). With higher SES, students are born with better social capital and all kinds of resources, including internet access and digital technologies (Graves et al., 2021), which ensures their accessibility to more diversified vehicles and sources of knowledge acquisition and learning and thus improves their civic knowledge and educational quantity and quality. This is furthered by their parents’ parenting style (Chan & Mak, 2020). A previous study has shown that parents from the upper class engage more in civic and political conversations with their kids (Oosterhoff & Metzger, 2016). Therefore, the digital divide and the inequality among students are the governments’ and schools’ concerns. As suggested by the literature reviewed in the current study and its results, SES might influence internet access and lead to students obtaining a higher pass rate on the DIF items “affection by damage to the environment” and “corporate citizenship.”
Unfortunately, this study failed to determine potential factors for the DIF item related to “informed voters,” although it has been difficult in the past to trace the sources of DIF for all DIF items (e.g., Allalouf et al., 1999; Ercikan et al., 2004). In short, this study identifies six items as DIF and finds possible DIF sources of the five DIF items (83%).
The difference in civic knowledge between the two groups was investigated after the DIF assessment. The remaining 73 DIF-free items were used in the Rasch analysis with latent regression. We found that Taiwanese students who could access the internet at home showed higher civic knowledge than those who could not, echoing McGrew et al.'s (2019) argument that the internet gap could impede the development of students’ civic literacy.
On the other hand, as an internet-based platform facilitating interaction among people, social media is a collective term for websites focusing on communication, interaction, content sharing, and collaboration that might potentially influence the impact of internet access on civic knowledge. The relationship between social media and civic engagement is apparent (Christensen et al., 2021; Schulz & Friedman, 2019). As reflected in the literature review above, the relationship between civic engagement and civic knowledge is exciting (McCabe et al., 2017). As students rarely use other people’s electronic devices to access social media, internet access might affect their civic knowledge through the use of social media to engage in political or social issues. Due to the complexity of social media issues, which is not the focus of the present study, the effect of social media will be discussed in further study.
Conclusions and future study
This study investigated the influence of internet access on civic knowledge measurement using the ICCS’ global survey data. The findings of this study can be a starting point for further investigation of these items by ICSS to ensure better reliability and validity of the test. However, this study also poses certain limitations. First, only civic knowledge is investigated in this study, rendering the need for examining the skills, attitudes toward, and engagement with internet access in the future. Second, the generalizability of the current findings to other international settings may be limited due to the study data being restricted to Taiwan. Following this, the effect of social media is highly related to internet access, and it is valuable to be investigated in future studies. Its generalizability should be furthered by future research in other countries’ data. Finally, the “informed voters” DIF item will be reviewed and discussed with experts or linked to other background variables to find evidence on the source of inequivalence in civic knowledge measurement in further research.
Availability of data and materials
The datasets generated and/or analysed during the current study are available in the International Civic and Citizenship Education Study, 2016, repository, https://www.icpsr.umich.edu/web/civicleads/studies/37147.
Notes
The mean of item difficulties is assumed to be identical for each group. Thus, the sum of item-by-group interactions is constrained to zero.
Five items were flagged: CI2ECM1, CI2PCM2, CI308M1, CI3CAM1, and CI3CRM2.
Twenty-two items were flagged.
The estimate was very close to \(\widetilde{\widehat{\uplambda }}\) (= 0.17) in the OR method.
References
Allalouf, A., Hambleton, R. K., & Sireci, S. G. (1999). Identifying the causes of DIF in translated verbal items. Journal of Educational Measurement, 36(3), 185–198. https://doi.org/10.1111/j.1745-3984.1999.tb00553.x
Alshahrani, S., Ahmed, E., & Ward, R. (2017). The influence of online resources on the student–lecturer relationship in higher education: A comparison study. Journal of Computer Education, 4(2), 87–106. https://doi.org/10.1007/s40692-017-0083-8
Apuke, O. D., & Iyendo, T. O. (2018). University students’ usage of the internet resources for research and learning: Forms of access and perceptions of utility. Heliyon, 4(12), e01052. https://doi.org/10.1016/j.heliyon.2018.e01052
Bond, T. G., Yan, Z., & Heene, M. (2021). Applying the Rasch model: Fundamental measurement in the human sciences (4th ed.). Routledge.
Chan, R. C., & Mak, W. W. (2020). Empowerment for civic engagement and well-being in emerging adulthood: Evidence from cross-regional and cross-lagged analyses. Social Science and Medicine, 244, 112703. https://doi.org/10.1016/j.socscimed.2019.112703
Chen, C.-H., & Tsai, C.-C. (2021). In-service teachers’ conceptions of mobile technology-integrated instruction: Tendency towards student-centered learning. Computers and Education, 170, 104224. https://doi.org/10.1016/j.compedu.2021.104224
Christensen, I. R., Biseth, H., & Huang, L. (2021). Developing digital citizenship and civic engagement through social media use. In Nordic schools In Northern Lights on Civic and Citizenship Education (pp. 65–92). Cham: Springer. https://doi.org/10.1007/978-3-030-66788-7_4
Cicognani, E., Zani, B., Fournier, B., Gavray, C., & Born, M. (2012). Gender differences in youths’ political engagement and participation. The role of parents and of adolescents’ social and civic participation. Journal of Adolescence, 35(3), 561–576. https://doi.org/10.1016/j.adolescence.2011.10.002
Courtney, M., Karakus, M., Ersozlu, Z., & Nurumov, K. (2022). The influence of ict use and related attitudes on students’ math and science performance: Multilevel analyses of the last decade’s pisa surveys. Large-Scale Assessments in Education, 10(1), 1–26. https://doi.org/10.1186/s40536-022-00128-6
Cuellar, E., Partchev, I., Zwitser, R., & Bechger, T. (2021). Making sense out of measurement non-invariance: How to explore differences among educational systems in international large-scale assessments. Educational Assessment, Evaluation and Accountability, 33(1), 9–25. https://doi.org/10.1007/s11092-021-09355-x
DeMars, C. E. (2021). A note on the odds ratio DIF index. Applied Psychological Measurement, 45(1), 71–73. https://doi.org/10.1177/0146621620947185
Diacopoulos, M. M., & Crompton, H. (2020). A systematic review of mobile learning in social studies. Computers and Education, 154, 103911. https://doi.org/10.1016/j.compedu.2020.103911
Diamond, L. (2020). Ill winds: Saving democracy from Russian rage, Chinese ambition, and American complacency. Penguin.
Dimitrova, D. V., Shehata, A., Strömbäck, J., & Nord, L. W. (2014). The effects of digital media on political knowledge and participation in election campaigns: Evidence from panel data. Communication Research, 41(1), 95–118. https://doi.org/10.1177/0093650211426004
Engin, M., & Donanci, S. (2015). Dialogic teaching and iPads in the EAP classroom. Computers and Education, 88, 268–279. https://doi.org/10.1016/j.compedu.2015.06.005
Ercikan, K., Gierl, M. J., McCreith, T., Puhan, G., & Koh, K. (2004). Comparability of bilingual versions of assessments: Sources of incomparability of English and French versions of Canada’s national achievement tests. Applied Measurement in Education, 17(3), 301–321. https://doi.org/10.1207/s15324818ame1703_4
Fu, Q.-K., & Hwang, G.-J. (2018). Trends in mobile technology-supported collaborative learning: A systematic review of journal publications from 2007 to 2016. Computers and Education, 119, 129–143. https://doi.org/10.1016/j.compedu.2018.01.004
Gottfried, J. A., Hardy, B. W., Holbert, R. L., Winneg, K. M., & Jamieson, K. H. (2017). The changing nature of political debate consumption: Social media, multitasking, and knowledge acquisition. Political Communication, 34(2), 172–199. https://doi.org/10.1080/10584609.2016.1154120
Graves, J. M., Abshire, D. A., Amiri, S., & Mackelprang, J. L. (2021). Disparities in technology and broadband internet access across rurality: Implications for health and education. Family and Community Health, 44(4), 257–265. https://doi.org/10.1097/FCH.0000000000000306
Greenberg, E. S. (1970). Political socialization. Atherton Press.
Gruchel, N., Kurock, R., Bonanati, S., & Buhl, S. (2022). Parental involvement and Children’s internet uses - Relationship with parental role construction, self-efficacy, internet skills, and parental instruction. Computers and Education, 182, 104481. https://doi.org/10.1016/j.compedu.2022.104481
Gubbels, J., Swart, N. M., & Groen, M. A. (2020). Everything in moderation: ICT and reading performance of Dutch 15-year-olds. Large-Scale Assessments in Education, 8, 1–17. https://doi.org/10.1186/s40536-020-0079-0
Guhn, M., Gadermann, A., & Zumbo, B. D. (2007). Does the EDI measure school readiness in the same way across different groups of children? Early Education and Development, 18(3), 453–472. https://doi.org/10.1080/10409280701610838
Hamid, S., Waycott, J., Kurnia, S., & Chang, S. (2015). Understanding students’ perceptions of the benefits of online social networking use for teaching and learning. Internet and Higher Education, 26, 1–9. https://doi.org/10.1016/j.iheduc.2015.02.004
Helvoort, J. V. (2018). Four spaces of civic literacy education: A literature review. In European Conference on Information Literacy (pp. 94–102). Springer Cham.
Hoffman, D. L., Novak, T. P., & Schlosser, A. (2000). The evolution of the digital divide: How gaps in Internet access may impact electronic commerce. Journal of Computer Mediated Communication, 5(3), 1–55. https://doi.org/10.1111/j.1083-6101.2000.tb00341.x
Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129–145). Lawrence Erlbaum Associates Inc.
Jin, K.-Y., Chen, H.-F., & Wang, W.-C. (2018). Using odds ratios to detect differential item functioning. Applied Psychological Measurement, 42(8), 613–629. https://doi.org/10.1177/0146621618762738
Kelley, T. R., & Knowles, J. G. (2016). A conceptual framework for integrated STEM education. International Journal of STEM Education, 3(1), 1–11. https://doi.org/10.1186/s40594-016-0046-z
Kewajiban, T. H., & MustajiBachri, B. S. (2018). Challenges and solutions of web-based learning on mobile devices. In K. A. Persichitte, A. Suparman, & M. Spector (Eds.), Educational technology to improve quality and access on a global scale (pp. 287–296). Springer Cham.
Liu, T., & Gao, F. (2011). Industrial process identification and control design: Step-test and relay-experiment-based methods. Springer. https://doi.org/10.1007/978-0-85729-977-2
Lo, J. C., & Adams, C. I. (2018). Civic literacy through literacy instruction: Using structured academic controversy in a government classroom. Citizenship Teaching and Learning, 13(1), 83–104. https://doi.org/10.1386/ctl.13.1.83_1
Lorah, J. (2018). Effect size measures for multilevel models: Definition, interpretation, and TIMSS example. Large-Scale Assessments in Education, 6(1), 1–11. https://doi.org/10.1186/s40536-018-0061-2
Magis, D., Béland, S., Tuerlinckx, F., & De Boeck, P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/BRM.42.3.847
Mamudu, P. A., & Oyewo, A. O. (2015). Use of mobile phones for academic purposes by law students of Igbinedion University, Okada, Nigeria. International Journal of Library Science, 4(4), 65–72. https://doi.org/10.5923/j.library.20150404.01
Marotta, L., Tramonte, L., & Willms, J. D. (2015). Equivalence of testing instruments in Canada: Studying item bias in a cross-cultural assessment for preschoolers. Canadian Journal of Education, 38(3), 1–23.
Martin, M. O., von Davier, M., & Mullis, I. V. (2020). Methods and Procedures: TIMSS 2019 Technical Report. https://timssandpirls.bc.edu/timss2019/methods/pdf/TIMSS-2019-MP-Technical-Report.pdf
McCabe, H. A., Hylton, M. E., Kooreman, H. E., Sarmiento Mellinger, M., & Day, A. (2017). Civic literacy and social work education: Results from a multi-site study. Journal of Policy Practice, 16(1), 81–94. https://doi.org/10.1080/15588742.2015.1137852
McGrew, S., Smith, M., Breakstone, J., Ortega, T., & Wineburg, S. (2019). Improving university students’ web savvy: An intervention study. British Journal of Educational Psychology, 89(3), 485–500. https://doi.org/10.1111/bjep.12279
Metzger, A., Alvis, L., & Oosterhoff, B. (2020). Adolescent views of civic responsibility and civic efficacy: Differences by rurality and socioeconomic status. Journal of Applied Developmental Psychology, 70, 101183. https://doi.org/10.1016/j.appdev.2020.101183
Metzger, A., & Ferris, K. (2013). Adolescents’ domain-specific judgments about different forms of civic involvement: Variations by age and gender. Journal of Adolescence, 36(3), 529–538. https://doi.org/10.1016/j.adolescence.2013.03.003
Muleya, G. (2017). The conceptual challenges in the conceptualization of civic education. Journal of Lexicography and Terminology, 1(1), 59–81.
Navas-Ara, M. J., & Gómez-Benito, J. (2002). Effects of ability scale purification on the identification of dif. European Journal of Psychological Assessment, 18(1), 9–15. https://doi.org/10.1027/1015-5759.18.1.9
Nischal, S. (2022). The gravity of Covid-19 pandemic impact upon education sector in India. In L. K. Sharma (Ed.), Impact and Challenges of COVID-19 on Health, Livelihoods, Environment and Education (pp. 43–52). Nitya Publication.
Oosterhoff, B., & Metzger, A. (2016). Mother–adolescent civic messages: Associations with adolescent civic behavior and civic judgments. Journal of Applied Developmental Psychology, 43, 62–70. https://doi.org/10.1016/j.appdev.2016.01.001
Pang, H. (2018). Can microblogs motivate involvement in civic and political life? Examining uses, gratifications and social outcomes among Chinese youth. Online Information Review, 42(5), 663–680. https://doi.org/10.1108/OIR-04-2017-0136
Partheymüller, J., & Faas, T. (2015). The impact of online versus offline campaign information on citizens’ knowledge, attitudes and political behaviour: Comparing the German Federal Elections of 2005 and 2009. German Politics, 24(4), 507–524. https://doi.org/10.1080/09644008.2015.1021789
Robitzsch, A., Kiefer, T., & Wu, M. (2022). Package ‘TAM’ (Version 4.1–4). https://cran.r-project.org/web/packages/TAM/index.html
Schulz, W., Ainley, J., & Fraillon, J. (2011). ICCS 2009 Technical report. IEA. https://www.iea.nl/sites/default/files/2019-04/ICCS_2009_Technical_Report.pdf
Schulz, W., Carstens, R., Losito, B., & Fraillon, J. (Eds.) (2018). ICCS 2016 Technical Report. IEA. https://www.iea.nl/sites/default/files/2019-07/ICCS%202016_Technical%20Report_FINAL.pdf
Schulz, W., & Friedman, T. (2019). Young people’s use of social media and internet for civic engagement in 21 countries. 7th IEA International Research Conference. Copenhagen. https://iccs.acer.org/files/IRC2017_ICCS16_UseOfSocialMedia_SchulzFriedman.pdf
Shih, C.-L., & Wang, W.-C. (2009). Differential item functioning detection using the multiple indicators, multiple causes method with a pure short anchor. Applied Psychological Measurement, 33(3), 184–199. https://doi.org/10.1177/0146621608321758
Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic-regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
Tarman, B., & Kilinc, E. (2022). Predicting high school students’ global civic engagement: A multiple regression analysis. The Journal of Social Studies Research. https://doi.org/10.1016/j.jssr.2022.02.001
Thissen, D., Steinberg, L., & Wainer, H. (1988). Use of item response theory in the study of group differences in trace lines. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 147–169). Erlbaum.
Virgiawan, I., & Sundawa, D. (2022). Building civic responsibility and democratic attitudes of students in learning Pancasila and civic education. In Annual Civic Education Conference (ACEC 2021) (pp. 170–174). Atlantis Press.
Wang, W.-C. (2008). Assessment of differential item functioning. Journal of Applied Measurement, 9(4), 387–408.
Weaver, D., & Drew, D. (1993). Voter learning in the 1990 off-year election: Did the media matter? Journalism and Mass Communication Quarterly, 70(2), 356–368. https://doi.org/10.1177/107769909307000211
Wilson, M., De Boeck, P., & Carstensen, C. H. (2008). Explanatory item response models: A brief introduction. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 83–110). Hogrefe & Huber Publishers.
Woods, C. M. (2009). Evaluation of MIMIC-model methods for DIF testing with comparison to two-group analysis. Multivariate Behavioral Research, 44(1), 1–27. https://doi.org/10.1080/00273170802620121
Zhong, Z. J. (2014). Civic engagement among educated Chinese youth: The role of SNS (Social Networking Services), bonding and bridging social capital. Computers and Education, 75(1), 263–273. https://doi.org/10.1016/j.compedu.2014.03.005
Zorwick, L. W., & Wade, J. M. (2016). Enhancing civic education through the use of assigned advocacy, argumentation, and debate across the curriculum. Communication Education, 65(4), 434–444. https://doi.org/10.1080/03634523.2016.1203005
Zumbo, B. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832
Acknowledgements
Not applicable.
Funding
This study was supported by the Nation Science and Technology Council [MOST108-2410-H-202-001].
Author information
Authors and Affiliations
Contributions
Chi-Chen Chen undertook all data analysis and interpretation, serving as the principal contributor to manuscript composition. Chia-Wei Tang provided invaluable perspectives on civic knowledge and made substantial contributions in elucidating the sources of differential item functioning (DIF). Kuan-Yu Jin orchestrated the research framework and data analysis methods, offering pivotal insights in the interpretation of statistical methods and analysis results. All authors read and approved the final manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Chen, CC., Tang, CW. & Jin, KY. Influences of internet access on civic knowledge measurement in Taiwan. Large-scale Assess Educ 12, 20 (2024). https://doi.org/10.1186/s40536-024-00209-8
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s40536-024-00209-8