The PIAAC longitudinal study in Germany: rationale and design

Abstract

In Germany, the respondents who had participated in the 2012 survey of the Programme for the International Assessment of Adult Competencies (PIAAC) were re-approached for the panel study PIAAC-L. PIAAC-L aims at investigating the longitudinal effects of skill outcomes over the life course and the development of the key skills assessed in PIAAC. Moreover, additional and alternative background information was collected and analyzed within PIAAC-L. PIAAC-L consists of three follow-up waves to the initial PIAAC 2012 survey. The present paper describes the rationale for PIAAC-L and the benefits of conducting a longitudinal PIAAC follow-up study in Germany. In addition, we outline the general design of PIAAC-L and the specific design of the three waves of data collection. Finally, we address the analytic potential of PIAAC-L data set and its availability to the scientific community.

The Organisation for Economic Co-operation and Development (OECD) initiated the Programme for the International Assessment of Adult Competencies (PIAAC) in 2008 with the aim of assessing basic skills of the adult population in an internationally comparable way. The skills assessed—literacy, numeracy, and problem solving in technology-rich environments—are considered to be essential for successful participation in modern society and to be a foundation for developing numerous other, more specific, skills or competences (OECD 2013a).¹ PIAAC provides information about the extent to which the adult population in the participating countries differs in terms of the basic skills assessed. Moreover, it examines factors associated with the acquisition, retention, and maintenance of these skills, and sheds light on their effects on social and, in particular, economic participation.

The PIAAC strategy entails repeating a cross-sectional survey design at regular intervals. The data collection for the first round of PIAAC was conducted in parallel in 24 countries—including Germany—in 2011/2012. This first round, the results of which were published in 2013, marked the starting point of this multi-cycle program. Further cycles are planned at 10-year intervals and will allow us to monitor and analyze how key skills are changing in our adult populations.

In each country participating in PIAAC, a sample of at least 5000 adults (aged 16–65 years) was randomly selected from the target population in order to achieve unbiased and representative pictures of all countries. In Germany, a two-stage register-based sampling approach was implemented. Compared to other large-scale surveys conducted in Germany in recent years, fieldwork for PIAAC in Germany was quite successful, achieving a response rate of 55%. Approximately 5400 interviews were carried out as computer-assisted personal interviews (background questionnaire) followed by a self-administered skills assessment (in the presence of the interviewer), which was usually computer-based, but with an optional paper-based version. On average, the administration of the background questionnaire and the cognitive assessment lasted 1 h and 40 min in Germany (see Zabal et al. 2014). In order to obtain permission to re-contact German PIAAC respondents, at the end of the interview they were asked whether they would be tentatively willing to participate in a follow-up study.

In summary, the resulting data in each country participating in PIAAC provide estimates of the literacy, numeracy and problem-solving (in technology rich environments) skills of the adult population and offer the possibility of comparing and contrasting these skills with sociodemographic and other background variables as well as with skills use in the workplace and in daily life. The international data from this first round of PIAAC are available as public use files via the OECD website.² In addition, more comprehensive data sets are available as national scientific use files for several countries, including Germany.

The motivation and context of the German longitudinal follow-up

The PIAAC data provide detailed information about the level and distribution of the basic skills in the adult population in the participating countries. However, due to the fact that limited time was available for the PIAAC interview and thus also for the administration of the background questionnaire, only a somewhat restricted amount of background information could be collected. Thus, while the PIAAC background questionnaire included the key predictors and social and economic outcomes of the basic skills measured in the assessment, the number of background variables was by necessity limited. These limitations constrain the extent to which the PIAAC data allows to investigate key predictors and outcomes of the PIAAC skills and how extensively research and policy questions can be addressed. Hence, central survey questions, for example (a) how competences are acquired, (b) how skill use helps individuals to maintain and further develop skills, (c) how skills shape labor market outcomes and job mobility over the life course, and (d) whether adults are prepared for the challenges of modern knowledge societies (see OECD 2013a) can be answered only partially on the basis of these data.

In addition to the need to further enrich the background information for the PIAAC data, the investigation of skill maintenance and development versus skill loss can be thoroughly investigated only by using longitudinal data. Only a longitudinal design can disentangle whether self-selection into certain professions might be a crucial determining factor in the correlation between earnings and competences.

The cross-sectional design used in PIAAC allows an initial exploration of the associations between background variables and the cognitive skills assessed. However, finding, for example, an association between skills and participation in continuing education might indicate that participation in continuing education helps to maintain cognitive skills. Another interpretation could be that such an association indicates that higher skilled people have a greater tendency to participate in continuing education (without this necessarily having any further effect on their cognitive skill level). To investigate which of the two hypotheses is more appropriate, a longitudinal design is needed. Only a prospective panel design can provide insights into the effects of basic skills for example on (a) participation and opportunities or risks in the labor market, e.g. income or unemployment, (b) participation in continuing education as outlined above, and (c) causes for skill maintenance and development.

As mentioned above, fieldwork in PIAAC Germany was quite successful as evidenced by the response rate of 55%. Although Germany ranks in the lower midfield compared to other countries participating in PIAAC (OECD 2013b), in comparison to other large-scale face-to-face surveys in Germany conducted in recent years the achieved response rate of 55% is extraordinarily high. For example, the German General Social Survey (ALLBUS) obtained response rates between 34 and 38% in the last years (Wasmer et al. 2012, 2014). Similarly in the last rounds of the European Social Survey (ESS) Germany realized response rates of 33% on average (European Social Survey 2012, 2014). Hence, as the methodological standards for, and the costs of, PIAAC sampling and fielding were very high, and the German PIAAC sample is of extremely high quality compared to other German samples, it would have been a waste of a high-potential sample not to make use of it for further investigations. This is even more the case due to the fact that 98% of PIAAC respondents could be re-contacted and this was considered to be an excellent starting point for a follow-up study. Finally, re-using the PIAAC sample also incurred substantially lower costs.

Given the need to enhance the background information within PIAAC and the interest in obtaining longitudinal follow-up data on the PIAAC participants, a PIAAC longitudinal study (PIAAC-L) was initiated in Germany. Funded by the German Federal Ministry of Education and Research, PIAAC-L has been implemented as a joint research project on the part of the following three key institutions in the field of social sciences in Germany who have substantial experience in one or more areas central to PIAAC-L, namely the assessment of adult skills, the collection of background information, and conducting panel surveys:

GESIS—Leibniz Institute for the Social Sciences has first-hand expertise in PIAAC, as it was commissioned by the Federal Ministry of Education and Research to act as National Project Manager for PIAAC Germany. In addition, GESIS conducts several large-scale national and, in particular, international surveys on a regular basis.

The German Institute for Economic Research (DIW Berlin) has coordinated the key longitudinal household survey of income and living conditions in Germany, the German Socio-Economic Panel (SOEP), since 1984.

The Leibniz Institute for Educational Trajectories (LIfBi) is the coordinator of the National Educational Panel Study (NEPS), a project that collects longitudinal data on educational processes from early childhood to adulthood in Germany.

In order to combine expertise from these three central longitudinal and cross-sectional surveys, a consortium was constituted under the aegis of GESIS. All three institutes are experienced in research data infrastructures and are members of the Leibniz Association.³

Some of the other countries that participated in the first round of PIAAC have also followed up on their PIAAC respondents. In Canada, a subset of the sample persons for the Canadian longitudinal social survey Longitudinal and International Study of Adults (LISA) also participated in PIAAC and these respondents are being re-interviewed biennially as part of the longitudinal household survey. There was also a Polish follow-up study on PIAAC (postPIAAC), which collected longitudinal and additional background information on the PIAAC respondents in one additional wave of data collection (2014/2015) and also implemented some basic cognitive skills tests as well as a basic ICT skills test. Finally, a follow-up of PIAAC in Italy (also in 2014/2015) gathered longitudinal information on Italian PIAAC respondents and focused on obtaining more data on their non-cognitive skills. However, of all the PIAAC panel surveys, only the German PIAAC-L has carried out a second assessment of literacy and numeracy using PIAAC instruments (see description of Wave 2).

The design of the German PIAAC longitudinal study

PIAAC-L was designed to follow up the German PIAAC 2012 respondents that could be re-contacted for a prospective PIAAC-related survey (n = 5225 individuals referred to in PIAAC-L as “anchor persons”)⁴ over three additional waves of data collection, combining research questions and measurement instruments from PIAAC, NEPS, and SOEP, and extending the focus by also including adult members of the anchor persons’ households. Each of the three waves had a particular focus (cf. Table 1).

Table 1 PIAAC-L design and sample size for each of the three follow-up waves

The first follow-up wave was conducted in 2014, and thus approximately two years after the PIAAC 2012 assessment. It focused on collecting additional background information. All respondents (anchor persons and additional household members aged 18 years and older) were administered a comprehensive CAPI questionnaire (average duration: approximately 45 min) mainly consisting of items and questions regularly collected in the SOEP (see Wagner et al. 2007). It collected, among other individual characteristics, additional information on education and work history, employment situation, income, family background, place of origin, health status, religious affiliation, and time use (an overview of the concepts assessed in each of the three waves is provided in “Appendix”). To obtain more context information on the household, a separate household questionnaire (duration: 15 min on average)—similar to that applied in the SOEP—was administered. This questionnaire collected information on household income, living situation, living conditions and costs, as well as varied information on household members—especially children.

The second follow-up wave, which was conducted in 2015, concentrated on the assessment of basic skills and the repetition of several questionnaire items from PIAAC (full interview duration: 100 min on average). Therefore, the central skills assessed in PIAAC, namely literacy and numeracy, were re-assessed for all anchor persons.⁵ In addition, as one of the central methodological aims of PIAAC-L was to compare the different approaches to skills assessment and their operationalization in PIAAC and NEPS, the corresponding competence tests from NEPS for reading (comparable to literacy in PIAAC) and mathematics (comparable to numeracy in PIAAC) were also administered in the 2015 wave. Hence, both the PIAAC and the NEPS assessment instruments were administered in parallel using a randomized design with eight test conditions for the anchor persons. This design is illustrated in Table 2. As can be seen from the table, each anchor person completed either (a) the PIAAC numeracy and literacy instruments, or (b) the NEPS math and reading instruments, or (c) either the PIAAC literacy instrument and the NEPS reading instrument or the PIAAC numeracy instrument and the NEPS math instrument.

Table 2 Design, sample sizes and mean durations for competence measurement in PIAAC-L Wave 2

Besides the anchor persons, their spouses or partners living in the same household were interviewed and tested in the 2015 wave. However, spouses and partners were administered only the NEPS instruments for reading and mathematics, with just two different assessment conditions (in the second condition, the instruments were administered in reverse order). The resulting data thus enable similarities and differences in competences within couples to be compared.

Both skills assessments were administered under the same conditions as in PIAAC or NEPS, respectively. While the PIAAC assessment was computer-based by default, with an optional paper-based version for a small number of respondents who lacked computer experience, the NEPS assessment was paper-based only.⁶ Respondents taking the PIAAC assessment had unlimited time to solve the tasks, whereas respondents taking the NEPS assessment were timed by the interviewer and were allowed up to 28 min per domain. The administration of both timed and untimed tasks within the cognitive assessment is an innovative component of PIAAC-L.

In addition to the cognitive instruments, all respondents were administered a comparatively short background questionnaire which focused on central questions from the PIAAC questionnaire, for example on education, work status, and occupation. Some alternative measures to those implemented in PIAAC were also included, for example a module on skills use at work, which was conceptually similar to that administered in PIAAC but with a different operationalization (taken from the NEPS). In addition, efforts were made to collect a wider range of information relevant to the cognitive assessment, for example more information on computer experience and health. Given the design of the second wave of data collection, the questionnaire also included a module administered to the spouse or partner of the anchor person. A set of plausible values for the competence data will provide the basis for comparing PIAAC test performance over time, for comparing the actual test instruments from PIAAC and NEPS, and for analyzing the competence data within couples.

The third follow-up wave, which was conducted in 2016, again included all members of the anchor person’s household who were aged 18 years or over. In this wave, detailed background information—especially updates of previous information—was collected. The average duration of the person questionnaire and of the household questionnaire was 45 and 10 min, respectively. In addition, alternative operationalizations of particular constructs initially assessed in PIAAC were tested in this wave. They included an extended and re-worked module for measuring continuing education and revised questions to assess skill mismatch (Perry 2016; see also McGowan and Andrews 2015). These modules, and a number of other new items, were developed by researchers from the PIAAC Leibniz Network, a collaborative project of eight German Leibniz institutes; the PIAAC Leibniz Network aims at further analyzing, advancing, and disseminating PIAAC, and thus also PIAAC-L.

In addition to the background information, basic general cognitive skills of all respondents (i.e., anchor persons and all other participating adult household members) were assessed. For this purpose, short general cognitive tasks used in the SOEP were administered. The two ultra-short cognitive performance tasks allow for a reliable assessment of general intellectual ability and distinguish between two components of intellectual functioning: cognitive mechanics and pragmatics (see Richter et al. 2013).

To facilitate analyses of the competence data and the repeated measures, plausible values, which are the state of the art in analysis techniques for large-scale assessment data, will be provided, adopting the procedures from PIAAC as far as possible. Plausible values capture latent estimates for relations between competence measures and context data (OECD 2013b). With the additional data assessed in PIAAC-L, additional sets of plausible values will be needed for analyses. As a first step, the additional context data will be taken into account (first set of plausible values). As a second step, the repeated measurement of PIAAC literacy and numeracy will be included in an extended scaling and analysis model to enable longitudinal analyses (second set of plausible values). Furthermore, the analysis of the dyadic competence and context data will also require another extended item response model (third set of set of plausible values). For all three additional models presented, the complexity and the amount of context data in PIAAC-L constitutes a methodological challenge that will have to be addressed, as the standard techniques for generating plausible values might not work. Thus, alternative approaches in drawing plausible values by reducing the number of context variables either statistically or by theoretical selection and combinations thereof will be evaluated.

To account for selectivity, in each PIAAC-L wave weighting factors (nonresponse and cross-sectional) for each participating anchor person are computed and provided to the data users (see Bartsch and Poschmann 2016, for documentation on weighting in wave 1).

Analytic potential

The extended and longitudinal data based on the German PIAAC 2012 sample provide researchers with a vast range of opportunities for addressing different research questions regarding skill development and sustainability—in particular skill development in adulthood as a function of central social and demographic circumstances, the effects of basic skills on labor market participation, careers, and unemployment, and, finally, the effects of basic skills on educational and employment mobility over the life course. The extended background information collected in the three waves of PIAAC-L also enables the investigation of the relationship between basic skills assessed in PIAAC and various constructs, such as central non-cognitive skills (see Rammstedt et al. 2016a, 2017), life satisfaction (Danner et al. 2016), and health.

In addition, the PIAAC-L data also enable methodological research questions to be addressed. For example, the PIAAC-L design makes it possible to compare the competence measures used in PIAAC, NEPS, and the SOEP. PIAAC-L was also set up to investigate alternative operationalizations of central constructs assessed in the PIAAC background questionnaire, such as skill mismatch and participation in continuing education. In addition, the longitudinal design allows panel attrition to be investigated, especially with regard to the anchor persons’ competence levels. Preliminary results provide first indications of possible nonresponse bias within PIAAC-L with regard to this key outcome variable (Rammstedt et al. 2016b).

Finally, the extension of the design to include the entire household and to measure the competences of the anchor person’s spouse or partner offers additional insights, for example into skill formation (Cunha et al. 2010) in the course of professional careers and educational trajectories and into assortative mating.

The above list of potential research questions is by far not exhaustive nor does it predetermine any prioritizing of the different endeavors.

Data availability

Data from the three waves of the PIAAC-L study as well as accompanying documentation—including the questionnaires (in German), codebooks (in English), and technical reports (English)—are, or will be, available to researchers in the form of scientific use files through the PIAAC Research Data Center and the Data Archive at GESIS.⁷ Both German and international researchers interested in the data can submit a short application. Users will be required to sign a data user contract and will then obtain access to the data free of charge.⁸ Data from the first PIAAC-L wave were published at the end of the first quarter of 2016;⁹ data from the second wave was published in an aggregated form with the first wave by the end of 2016,¹⁰ and the third wave will be available by the end of 2017. For anchor persons, it will be possible to match data from all three PIAAC-L waves to data from the German PIAAC scientific use file.¹¹ In addition, it is planned to link data of consenting anchor persons from PIAAC and PIAAC-L to microdata from the Federal Employment Agency.¹² Access to these data will be made available through the Secure Data Center of the Statistics Department of the Federal Employment Agency.

From 2017 onwards, it is intended to include the remaining PIAAC-L participants and all other household members in the SOEP as a new refresher sample. In this course, information on the respondents will be updated according to the SOEP standardly assessed household and person information. In 2017, it is planned that PIAAC-L respondents will provide data on their current wealth situation. The future follow-ups will also be made available to the research community through the SOEP Research Data Center.

Conclusions

In sum, the resulting data from the German PIAAC longitudinal study allow a wide range of content-related and methodological research questions in the field of competence development and maintenance to be addressed, which would not be possible on the basis of PIAAC data only. Results of these analyses provide insights into the quality and predictive power of the concepts assessed in PIAAC. These results furnish crucial information for the revision of the instruments for upcoming PIAAC cycles and also provide empirical evidence on alternative measures and their quality. Finally, and very importantly, the design of PIAAC-L also provides empirical evidence on additional concepts that could be included in upcoming cycles of PIAAC. For example, the Big Five personality domains—the most highly used and most extensively researched personality approach (see for example John et al. 2008; McCrae and Costa 1999)—were assessed within PIAAC-L.¹³ In a recent analysis of the PIAAC-L data we were able to show that personality in terms of the Big Five and other concepts of non-cognitive skills prove to be highly predictive of numerous outcome variables measured in PIAAC (see Rammstedt et al. 2017).

From a national point of view, PIAAC-L has merged for the first time three distinct and very heterogeneous large-scale surveys conducted by three research data infrastructure institutes within the Leibniz Association in Germany. A direct comparison of measures such as the competence tests used in the different studies is thus possible using the PIAAC-L data. From a more general perspective, PIAAC-L can be regarded as a milestone on the road to stronger collaboration between, and perhaps a partial integration of, different surveys studying overlapping concepts. For data users, this can, in the medium term, improve concept harmonization and operationalization and thus yield higher data quality and new insights into causal mechanisms of skill formation and the outcomes of cognitive skills over the life course.

Appendix

Central constructs assessed in each of the three PIAAC-L waves:

• Wave 1

  • Household questionnaire (based on the SOEP core questionnaire)

    • Living situation and costs

    • Income, benefits, and wealth

    • Children and other household members

  • Person questionnaire (based on the SOEP core questionnaires):

    • Family situation, family background, childhood

    • Biographic calendar: education and employment history

    • Work status and work situation

    • Income and benefits

    • Personality and individual preferences

    • Health and time use

• Wave 2

  • Questionnaire (based on PIAAC and other questionnaires):

    • Education, work status, work history, occupation etc. (based on PIAAC)

    • Skills use at work (based on the NEPS questionnaires), and a self-assessment of literacy/numeracy

    • Extended information on computer experience and use

    • Languages, including mother tongue and foreign languages

    • Module administered to spouse or partner of the anchor person

    • Parental information

    • Health, satisfaction, critical life events, leisure activities

  • Cognitive assessment instruments

    • PIAAC Literacy (per default computer-based, optionally paper-based)

    • PIAAC Numeracy (per default computer-based, optionally paper-based)

    • NEPS Reading (including Reading Speed in certain assessment conditions; paper-based only)

    • NEPS Math (paper-based only)

• Wave 3

  • Household Questionnaire:

    • As in Wave 1

  • Person questionnaire (similar to Wave 1 but somewhat abbreviated):

  • SOEP-based (longitudinal repeat of a subset of questions from Wave 1):

    • Education

    • Work status and work situation

    • Personality

    • Critical changes in family since last data collection

• New items:

  • Extended and improved module on continuing education

  • Improved questions for skill mismatch

  • Cognitive assessment:

    • SOEP basic cognitive skills scale