A statistical analysis of the characteristics of the intended curriculum for Japanese primary science and its relationship to the attained curriculum

Abstract

This study statistically investigates the characteristics of the intended curriculum for Japanese primary science, focusing on the learning content. The study used the TIMSS 2011 Grade 4 Curriculum Questionnaire data as a major source for the learning content prescribed at the national level. Confirmatory factor analysis was used to determine the extent to which a topic area was covered, as compared to the average among the 59 TIMSS 2011 participating countries. The study revealed that the topic areas of “Human Health” and “Changes in Environments,” both in the life science domain, showed statistically less coverage in the Japanese primary science curriculum when compared to the international average. Furthermore, in discussion, the study relates the characteristics found in the intended curriculum to those in the attained curriculum, examining the percent correct statistics for relevant items from the science assessment. Based on these findings, the study proposes two recommendations for revision of the Japanese primary science curriculum.

Background

Japan’s national curricula, the “Courses of Study,” have been revised roughly once every 10 years by the Ministry of Education, Culture, Sports, Science and Technology (MEXT) since first being implemented in 1947. The current Course of Study for Elementary Schools was announced in March 2008, and came fully into force in April 2011 with the objectives of revision including keeping up with societal change over the years and meeting the needs of learners on an international level. The Course of Study for Elementary Schools sets out the overall goals, objectives, and content for each subject and learning area.

Textbooks in Japan must be authorized by MEXT, and their drafts are always checked against the content described in the Courses of Study. Since these textbooks are extensively used by primary school teachers throughout Japan, this system functions to maintain the levels of learning content expected for both teaching and learning at schools. Thus, selecting the appropriate learning content at the right grade level is clearly of great importance when it comes to revising the national curricula, particularly in the case of Japan’s Course of Study for Elementary Schools. Multiple perspectives should be considered when judging the appropriateness of learning content for a particular grade; one such perspective that can be particularly useful is international comparison of learning content.

One of the models used to frame curriculum studies is the well-known curriculum model of the TIMSS, which comprises three aspects: the intended, the implemented, and the attained curriculum (Mullis et al. 2009). In this study, Japan’s primary science curriculum is regarded as the model’s intended curriculum since it is part of the national curriculum. The primary science textbooks, used extensively by the teachers, are regarded as the implemented curriculum whereas the pupils’ performance or achievement on standardized tests is regarded as the achieved curriculum. Using this curriculum model and the available data, the learning content described in the national (intended) curriculum can be compared to an international standard. It should be noted that while the context of an individual country’s particular learning content can be dealt with in terms of both the implemented and attained curriculum, this study will discuss only the attained curriculum.

The TIMSS 2011 Curriculum Questionnaire provides important information necessary for an international comparison of primary science curricula at the intended curriculum level. It offers up-to-date information on which topic areas (units of learning content) are taught in science curricula by grade in all the TIMSS 2011 participating countries. The TIMSS 2003 Curriculum Questionnaire, which surveyed National Research Coordinators about the grades in which particular science topic areas were expected to be taught, included 53 topic areas and reported primarily on general international trends of change in science curricula (Martin et al. 2004). Using data from past cycles and the TIMSS 2003 international data, Saruta (2007) summarized trends in Japan’s science curriculum since the 1960s, and pointed out several characteristics of the Japanese national science curriculum on a topic-area basis. The results of this study contributed to development and revision of Japan’s current science curriculum. Saruta’s study counted the number of countries in which a topic was taught based on both topic area and grade, and used this as a basis for comparison. Although this produced some important information, a stronger means of international comparison is required. This study, with the help of statistical analysis, attempts to determine the extent to which topic areas are covered taking into account international perspectives.

Purpose

This study investigates the characteristics of the Japanese primary science curriculum through an international comparison of learning content. It is expected to contribute to the upcoming revision of the national curriculum. Life science was chosen as the focus domain. The study addresses three research questions: (1) Which learning content areas in life science are less thoroughly covered by Japan’s primary-level science curriculum when compared to other countries’ primary-level science curricula? (2) How do Japanese pupils at the primary level perform in these less-covered learning content areas? (3) What recommendations can be made for future revision of the Japanese primary science curriculum, considering the content covered by the national curriculum and the performance of the pupils?

Methods

Data sources

For internationally comparable information about learning content at the intended curriculum level, this study used the results of the TIMSS 2011 Grade 4 Curriculum Questionnaire (SC408A), which provides information on 59 participating countries and regions. The questionnaire surveyed participants about the grades (from preprimary through upper secondary school) in which certain topics in the life science domain are intended to be taught. Results involving preprimary (PP) and G7 to G12 levels were excluded since this study is concerned with revision of the primary-level education curriculum. Table 1 shows the six topics and their descriptions as they appear in SC408A.

Table 1 Topic areas described in the Curriculum Questionnaire

For internationally comparable information about less-thoroughly covered learning content in Japan’s primary science curriculum at the attained curriculum level, this study used the percent correct statistics for items from the TIMSS 2011 science assessment almanacs data, along with the item information and TCMA (Test Curriculum Matching Analysis) data (Martin et al. 2012).

Statistical model used in this study

To examine which learning content areas in life sciences are less thoroughly covered by Japan’s primary science curriculum in comparison to other countries’ science curricula, we analyzed the data on the content domain of life science at the primary school level using an additive model based on confirmatory factor analysis. Specifically, as in an analysis of the multitrait-multimethod matrix (Campbell and Fiske 1959), we use the CT-CM (correlated trait-correlated method) model (e.g., Widaman 1985) with categorical (binary) indicators, treating topics as trait factors and grades as method factors (see Fig. 1).

Fig. 1

An illustration of CT-CM model. (in case of assuming four factors on both facets) with categorical indicators

To investigate which of the six topics in life science are (more or) less covered in a country’s science curriculum, we need to measure the degree to which the country covers each topic, after controlling for the effect of the curriculum coverage of life science in each grade. As Fig. 1 shows, the CT-CM model assumes that each item measures one of the factors for traits (topics in this study) and one of the factors for methods (grades in this study) independently. From this point, applying this model to the TIMSS Curriculum Questionnaire data is thought to be appropriate in light of the purpose of this study.

However, the CT-CM model is known to have identification issues and to result in Heywood cases (Kenny and Kashy 1992; Marsh 1989).¹ To deal with these issues and the small sample size of the TIMSS Curriculum Questionnaire data (N = 59), we used a Bayesian estimation. However, a Bayesian estimation of the categorical factor model with binary indicators and with a small sample size is known to show prior assumption dependence (Asparouhov and Muthén 2010b). Therefore, we conducted simulation studies to vary the conditions of prior distributions before real-data analysis. For the analyses of this study, we used Mplus version 7.3 (Muthén and Muthén 1998–2012).

Simulation studies

We assume that 60 countries were asked whether six topics were covered in six of the country’s respective grade levels. Therefore, there are 36 observed dichotomous variables per country.

In the CT-CM model, we set all factor loadings to 1, and all factor correlations to 0.6 as true values. We set the true values of threshold parameters as follows: 1.5 for grade 1, 0.9 for grade 2, 0.3 for grade 3, −0.3 for grade 4, −0.9 for grade 5, and −1.5 for grade 6 on all topics. The means and variances of the factors are fixed at 0 and 1, respectively.

We set four conditions on prior distribution of factor loadings and thresholds. Specifically, while Mplus default priors in the categorical factor model are N(0,5), we additionally set N(0,1), N(0,2.25), and N(0,4) conditions, because the simulation study of the categorical factor model for small sample sizes suggests that small variance priors such as N(0,1) may be more preferable (Asparouhov and Muthén 2010b). The prior distribution of the factor correlation matrix on topics was IW(I,p + 1) (p was the number of factors, six in this study) and the prior distribution of grade was also IW(I,p + 1), the Mplus default priors in the categorical factor analysis.

The number of MCMC (Markov chain Monte Carlo) chains for each parameter was two, and the minimum number of iterations for each chain was 100,000. By inputting THIN = 2 in Mplus, we used every second iteration and discarded the first half of each chain (Muthén and Muthén 1998–2012). The convergence was judged using a potential scale reduction (PSR) (Gelman and Rubin 1992) of less than 1.1. Two hundred simulations were run under each of the four conditions. We used medians as point estimates of the parameters.

Real-data analysis

We used the four conditions shown in the previous section. The number of MCMC chains for each parameter was five, and the minimum number of iterations for each chain was 100,000. By inputting THIN = 5 in Mplus, we used every fifth iteration and discarded the first half of each chain. The convergence criterion was the same as in the simulation study.

Results

Results of the simulation studies

All 800 runs were successful computations according to the output files. We evaluated the results in terms of parameter bias and 95 % coverage. The latter is the proportion of replications for which the 95 % confidence interval contains the population parameter value (Muthén and Muthén 1998–2012). Table 2 shows the minimum, average, and maximum bias, and Table 3 shows the 95 % coverage for each kind of parameter (factor loadings, factor correlations, and thresholds).

Table 2 Parameter bias in the simulation studies

Table 3 95 % coverage in the simulation studies

Underestimation was found on factor loadings in the N(0,1) condition, and overestimation was found on factor loadings and thresholds in the N(0,4) and N(0,5) conditions. Underestimation on factor correlations was found in all conditions, to a higher degree as prior variance was bigger. Conversely, the coverages were close to 95 % (over 90 %) in almost all conditions, except for some factor loadings in the N(0,5) condition for which coverages below 90 % were found.

These results suggest that the point estimate is inappropriate even when bigger or smaller variances are used for prior distributions of factor loadings and thresholds. Instead, interval estimation may be more preferable and appropriate, as its results seem robust over a range of prior distributions. Therefore, in the real-data analysis in the next section, we will show the results from all four conditions in the simulation study, but will primarily report the results of the credible interval estimation.

Results of the real-data analysis

After 100,000 iterations, PSRs were less than 1.01 for all conditions and the posterior predictive p value was 0.348 for the N(0,1) condition, 0.374 for the N(0,2.25) condition, 0.381 for the N(0,4) condition, and 0.382 for the N(0,5) condition. Additionally, R-squared values did not exceed 1 for any of the 36 indicators or factor correlations in all conditions. From these results, we judged that all model estimations terminated normally and that goodness-of-fit was high to some extent, particularly for the latter three conditions.² The 95 % credible intervals of each parameter for each condition are shown in Table 4. Results showed that none of the factor loadings for the topic factors had negative intervals, and that more than half of these had positive intervals in all conditions.

Table 4 Credible intervals of each parameter

We obtained each country’s posterior distributions of the factors for each topic area and for each grade, drawing 10,000 plausible values per factor and per country. Asparouhov and Muthén (2010a, p. 2) said, “…the individual level latent variable is of interest. One such example is the case when we want to construct a factor scores estimate for a latent variable and a standard error for that factor score estimate. To be able to do this it is necessary to use many imputed values.” They also “conclude that for small sample size the Bayes factor score estimates and standard errors are more reliable than those obtained by the ML estimator” (p. 5). Hence, we drew each country’s imputed plausible values for latent variables and computed their 95 % intervals.

To confirm the accuracy of plausible values in each condition, we show the minimum, average, and maximum means and variances of 12 factors in Table 5. These results tell us that the estimates of the variances in the N(0,1) condition were obviously overestimated. Variance estimates in this condition were thus less accurate than in the other conditions.

Table 5 Means and variances of plausible values for the factors

Japan’s 95 % intervals for the plausible values of the factors are shown in Table 6. The result showed negative intervals in all four conditions for Japan’s factors on the topics of “Human Health” and “Changes in Environments.” This means that the coverages of the Japanese primary science curriculum on these two topics are below the international averages, after controlling for the effect of the curriculum coverage of life science in each grade. In the N(0,1) condition, all topic factors except for “Life Cycles and Reproduction in Plants and Animals” also showed negative intervals for Japan. However, this condition does not seem to be relatively well-fitted and it does not provide accurate estimates for factor variances. Therefore, we interpret this to mean that the topic areas of “Changes in Environments” and “Human Health” were relatively less thoroughly covered in the Japanese primary science curriculum compared to the average of the curricula of the 59 TIMSS 2011 participating countries.

Table 6 95 % intervals of Japan’s plausible values for each factor

From the viewpoint of grades, the results showed the intervals including zero for Japan’s factors on grades in all the four conditions. Therefore, we interpret this to mean that coverage of the curriculum in Japanese primary science at each grade level does not differ much compared to the average of coverage in the respective grades of the 59 TIMSS 2011 participating countries.

Discussion

The results of the statistical analysis revealed that the topic areas of “Human Health” and “Changes in Environments” were relatively less covered in the Japanese primary science curriculum. In discussion, the study related the characteristics found in the learning content in the intended curriculum to the corresponding learning content in the attained curriculum. To compare the achievement of Japanese primary pupils in the less thoroughly covered learning content, we examined the percent correct statistics for each of the released TIMSS 2011 science items. The framework for these science items is described in the TIMSS 2011 Science Framework as follows, showing the content domain of life science and an overview of the topic areas to be covered:

“Life science includes understandings of the characteristics and life processes of living things, the relationships between them, and their interaction with the environment. The topic areas for life science are as follows:

• Characteristics and life processes of living things

• Life cycles, reproduction, and heredity

• Interaction with the environment

• Ecosystems

• Human health.”

(Source: TIMSS 2011 Science Framework, p. 53)

Commonalities between the learning content described in the Curriculum Questionnaire and the TIMSS 2011 Assessment Framework

It is necessary to determine whether the Curriculum Questionnaire and the TIMSS 2011 Assessment Framework deal with the same learning contents. Accordingly, we considered the commonalities between learning content, matching the topic areas described in the Curriculum Questionnaire with the corresponding topic areas described in the TIMSS 2011 Assessment Framework. The Curriculum Questionnaire’s topic areas of “Human Health” and “Changes in Environments” corresponded to the “Human Health” and “Ecosystem” topic areas described by the TIMSS 2011 Assessment Framework. It should be noted that the topic area “Changes in Environments” described in the Curriculum Questionnaire did not correspond to the TIMSS 2011 Assessment Framework’s “Interaction with the Environment” topic, which concerns the association of physical features and behavior patterns in plants and animals with the environment in which they live, and the ways in which plants and animals adapt themselves to particular environments (Mullis et al. 2009).

The topic area of “Human Health” described in the Curriculum Questionnaire includes such learning content as transmission/prevention of communicable diseases, signs of health/illness, diet, and exercise, as shown in Table 1. The topic area of “Human Health” described in the TIMSS 2011 Assessment Framework consists of two topic areas (objectives): “(1) Recognize ways that common communicable diseases (e.g., colds, influenza) are transmitted; identify signs of health or illness and some methods of preventing and treating illness” (Mullis et al. 2009, p. 56); and “(2) Describe ways of staying healthy including eating a balanced diet and regular exercise; identify common food sources (e.g., fruits, vegetables, grains)” (Mullis et al. 2009, p. 56). Keywords shared by the “Human Health” learning content of the Curriculum Questionnaire and the TIMSS 2011 Assessment Framework include “transmission/transmitted,” “prevention/preventing,” “communicable diseases,” “signs of health or illness,” and “diet and exercise.” Based on these commonalities, the learning content of the “Human Health” topic areas described by the Curriculum Questionnaire and the TIMSS 2011 Assessment Framework are considered to be equivalent.

The topic area of “Changes in Environments” described in the Curriculum Questionnaire includes such learning content as effects of human activity, pollution and its prevention, as shown in Table 1. The learning content of the “Ecosystems” topic area described in the TIMSS 2011 Assessment Framework comprises three objectives, only a third of which is considered directly correlated to “Changes in Environments”: “(3) Explain ways in which human behavior can have positive or negative effects on the environment; provide general descriptions and examples of the effects of pollution on humans, plants, animals, and their environments, and of ways of preventing or reducing pollution” (Mullis et al. 2009, p. 56). Key words and phrases shared by the learning content of the Curriculum Questionnaire’s “Changes in Environments” topic area and described in the TIMSS 2011 Assessment Framework’s “Ecosystems” topic area include “effects of human activity/human behavior can have positive or negative effects,” and “pollution and its prevention/preventing.” Based on these commonalities, the learning content of the “Changes in Environment” topic area described in the Curriculum Questionnaire and the “Ecosystems” topic area described in the TIMSS 2011 Assessment Framework are considered similar to a certain extent.

Students’ performance at the attained curriculum level in relatively less covered topics

As discussed above, we can relate learning content in the “Human Health” and “Changes in Environments” topic areas described by the Curriculum Questionnaire to student performance by way of the percent correct statistics for items under the TIMSS 2011 Assessment Framework. According to TIMSS 2011, the “Human Health” topic area includes 11 items, while the “Ecosystems” topic area includes 14 items.

Table 7 summarizes the percent correct statistics for Japanese fourth grade students for the 11 items of the “Human Health” topic, along with international averages and some item information. With the help of the TCMA data, information was added to Table 7 concerning whether or not the item was covered in the national curriculum. In addition, the items in Table 7 were divided into two groups based on information about which objective of the TIMSS 2011 Assessment Framework each item belongs to.

Table 7 Percent correct statistics for the items of the topic area of “human health”

As shown in Table 7, none of the 11 “Human Health” items were covered in the Japanese primary science curriculum. This is consistent with this study’s findings at the intended curriculum level. Furthermore, Table 7 shows that Japanese fourth graders performed higher than international averages on all 11 items. It can be interpreted that Japanese fourth graders are able to perform higher than the international averages in this topic area. These achieved curriculum results, along with the results from the intended curriculum level, show that in spite of the learning content for “Human Health” receiving less coverage in the national science curriculum, Japanese fourth graders perform better on this learning content than the international averages. It should be noted that the subject of physical education covers some learning content relevant to the topic of “Human Health”.

According to the Course of Study (MEXT 2008), one of the objectives in physical education for grade 4 and grade 5 is “To enable pupils to understand a healthy life and physical growth and development, and to develop the qualities and abilities for leading healthy and safe everyday lives” (p. 94); additionally, one of its learning content areas is “Health” (p. 96). This could help to explain Japanese fourth graders’ performance on the “Human Health” learning content. Another contributing factor may be the commonly held school activities to raise awareness of health, usually initiated by Japanese primary school nurses. One recommendation for curriculum revision drawn from these results is that learning content relevant to the topic area of “Human Health” need not be preferentially strengthened in the Japanese primary science curriculum.

Table 8 summarizes the percent correct statistics for Japanese fourth grade students for the 14 items of the “Ecosystems” topic, along with the international averages and some item information. Like Tables 7 and 8 includes information about whether or not the item was covered in the national curriculum. The items in Table 8 were divided into three groups based on which of the TIMSS 2011 Assessment Framework’s objectives they belong to. Five items were related to the third objective, whose learning content aligned with the Curriculum Questionnaire’s “Changes in Environments” topic. The five items concerning the third objective are shown in italics for easy reference.

Table 8 Percent correct statistics for the items of the topic area of “ecosystem”

As shown in Table 8, 13 of the 14 “Ecosystems” topic area items were not covered in the Japanese primary science curriculum. None of the five items shown in italics comprising the third objective were covered in the Japanese primary science curriculum. This is again consistent with the study’s findings at the intended curriculum level. Although Japanese fourth graders performed higher than the international averages on nine of the 14 total “Ecosystems” items, this ratio falls when looking only at the five third-objective items, on only two of which Japanese fourth graders performed higher than the international averages. Although this result should be treated with caution due to the limited number of items being discussed here, it can be reasonably interpreted that Japanese fourth graders do not perform higher than the international averages on learning content concerning “Changes in Environments.” These attained curriculum results, along with the results from the intended curriculum level, show that with respect to learning content concerning the “Changes in Environments” topic area, which receives less coverage in the national science curriculum, Japanese fourth graders do not always perform above the international averages. We hypothesize that this could be the result of the lesser coverage given to this learning content in the national curriculum. This hypothesis becomes more persuasive when considering the general trend of Japanese fourth graders’ performance, which is often higher than the international averages in other topics.

A recommendation for curriculum revision drawn from these results is that learning content concerning the topic area of “Changes in Environments” could be strengthened in the Japanese primary science curriculum.

Conclusions

Statistical analysis provided important information at the intended curriculum level, namely that the topic areas of “Changes in Environments” and “Human Health” as described in the Curriculum Questionnaire were relatively less covered by the Japanese primary science curriculum in comparison to the average of the 59 TIMSS 2011 participating countries. By integrating information obtained at the attained curriculum level, this study revealed that despite less coverage of learning content on the topic of “Human Health” in the national science curriculum, Japanese fourth graders perform above than the international averages in this area. On the other hand, Japanese fourth graders do not consistently perform above international averages in learning content concerning the topic area of “Changes in Environments.” The study hypothesizes that this may be the result of the lesser coverage of this learning content in the national curriculum.

This study proposes two recommendations for revision of the Japanese primary science curriculum. First, it is not necessary to preferentially strengthen learning content concerning the topic area of “Human Health.” Second, it may be worth strengthening learning content concerning the topic area of “Changes in Environments,” where it is possible to do so.

Although this study adapted TIMSS’s curriculum model, the implemented curriculum level was not dealt with. The next expected step is a discussion of learning content from the perspective of the implemented curriculum, for instance by examining the textbooks of the major publishing companies.