The PIAAC longitudinal study in Germany: rationale and design
© The Author(s) 2017
Received: 24 August 2016
Accepted: 18 January 2017
Published: 31 January 2017
In Germany, the respondents who had participated in the 2012 survey of the Programme for the International Assessment of Adult Competencies (PIAAC) were re-approached for the panel study PIAAC-L. PIAAC-L aims at investigating the longitudinal effects of skill outcomes over the life course and the development of the key skills assessed in PIAAC. Moreover, additional and alternative background information was collected and analyzed within PIAAC-L. PIAAC-L consists of three follow-up waves to the initial PIAAC 2012 survey. The present paper describes the rationale for PIAAC-L and the benefits of conducting a longitudinal PIAAC follow-up study in Germany. In addition, we outline the general design of PIAAC-L and the specific design of the three waves of data collection. Finally, we address the analytic potential of PIAAC-L data set and its availability to the scientific community.
KeywordsPIAAC PIAAC-L Large-scale assessment Basic skills Longitudinal
The Organisation for Economic Co-operation and Development (OECD) initiated the Programme for the International Assessment of Adult Competencies (PIAAC) in 2008 with the aim of assessing basic skills of the adult population in an internationally comparable way. The skills assessed—literacy, numeracy, and problem solving in technology-rich environments—are considered to be essential for successful participation in modern society and to be a foundation for developing numerous other, more specific, skills or competences (OECD 2013a).1 PIAAC provides information about the extent to which the adult population in the participating countries differs in terms of the basic skills assessed. Moreover, it examines factors associated with the acquisition, retention, and maintenance of these skills, and sheds light on their effects on social and, in particular, economic participation.
The PIAAC strategy entails repeating a cross-sectional survey design at regular intervals. The data collection for the first round of PIAAC was conducted in parallel in 24 countries—including Germany—in 2011/2012. This first round, the results of which were published in 2013, marked the starting point of this multi-cycle program. Further cycles are planned at 10-year intervals and will allow us to monitor and analyze how key skills are changing in our adult populations.
In each country participating in PIAAC, a sample of at least 5000 adults (aged 16–65 years) was randomly selected from the target population in order to achieve unbiased and representative pictures of all countries. In Germany, a two-stage register-based sampling approach was implemented. Compared to other large-scale surveys conducted in Germany in recent years, fieldwork for PIAAC in Germany was quite successful, achieving a response rate of 55%. Approximately 5400 interviews were carried out as computer-assisted personal interviews (background questionnaire) followed by a self-administered skills assessment (in the presence of the interviewer), which was usually computer-based, but with an optional paper-based version. On average, the administration of the background questionnaire and the cognitive assessment lasted 1 h and 40 min in Germany (see Zabal et al. 2014). In order to obtain permission to re-contact German PIAAC respondents, at the end of the interview they were asked whether they would be tentatively willing to participate in a follow-up study.
In summary, the resulting data in each country participating in PIAAC provide estimates of the literacy, numeracy and problem-solving (in technology rich environments) skills of the adult population and offer the possibility of comparing and contrasting these skills with sociodemographic and other background variables as well as with skills use in the workplace and in daily life. The international data from this first round of PIAAC are available as public use files via the OECD website.2 In addition, more comprehensive data sets are available as national scientific use files for several countries, including Germany.
The motivation and context of the German longitudinal follow-up
The PIAAC data provide detailed information about the level and distribution of the basic skills in the adult population in the participating countries. However, due to the fact that limited time was available for the PIAAC interview and thus also for the administration of the background questionnaire, only a somewhat restricted amount of background information could be collected. Thus, while the PIAAC background questionnaire included the key predictors and social and economic outcomes of the basic skills measured in the assessment, the number of background variables was by necessity limited. These limitations constrain the extent to which the PIAAC data allows to investigate key predictors and outcomes of the PIAAC skills and how extensively research and policy questions can be addressed. Hence, central survey questions, for example (a) how competences are acquired, (b) how skill use helps individuals to maintain and further develop skills, (c) how skills shape labor market outcomes and job mobility over the life course, and (d) whether adults are prepared for the challenges of modern knowledge societies (see OECD 2013a) can be answered only partially on the basis of these data.
In addition to the need to further enrich the background information for the PIAAC data, the investigation of skill maintenance and development versus skill loss can be thoroughly investigated only by using longitudinal data. Only a longitudinal design can disentangle whether self-selection into certain professions might be a crucial determining factor in the correlation between earnings and competences.
The cross-sectional design used in PIAAC allows an initial exploration of the associations between background variables and the cognitive skills assessed. However, finding, for example, an association between skills and participation in continuing education might indicate that participation in continuing education helps to maintain cognitive skills. Another interpretation could be that such an association indicates that higher skilled people have a greater tendency to participate in continuing education (without this necessarily having any further effect on their cognitive skill level). To investigate which of the two hypotheses is more appropriate, a longitudinal design is needed. Only a prospective panel design can provide insights into the effects of basic skills for example on (a) participation and opportunities or risks in the labor market, e.g. income or unemployment, (b) participation in continuing education as outlined above, and (c) causes for skill maintenance and development.
As mentioned above, fieldwork in PIAAC Germany was quite successful as evidenced by the response rate of 55%. Although Germany ranks in the lower midfield compared to other countries participating in PIAAC (OECD 2013b), in comparison to other large-scale face-to-face surveys in Germany conducted in recent years the achieved response rate of 55% is extraordinarily high. For example, the German General Social Survey (ALLBUS) obtained response rates between 34 and 38% in the last years (Wasmer et al. 2012, 2014). Similarly in the last rounds of the European Social Survey (ESS) Germany realized response rates of 33% on average (European Social Survey 2012, 2014). Hence, as the methodological standards for, and the costs of, PIAAC sampling and fielding were very high, and the German PIAAC sample is of extremely high quality compared to other German samples, it would have been a waste of a high-potential sample not to make use of it for further investigations. This is even more the case due to the fact that 98% of PIAAC respondents could be re-contacted and this was considered to be an excellent starting point for a follow-up study. Finally, re-using the PIAAC sample also incurred substantially lower costs.
Given the need to enhance the background information within PIAAC and the interest in obtaining longitudinal follow-up data on the PIAAC participants, a PIAAC longitudinal study (PIAAC-L) was initiated in Germany. Funded by the German Federal Ministry of Education and Research, PIAAC-L has been implemented as a joint research project on the part of the following three key institutions in the field of social sciences in Germany who have substantial experience in one or more areas central to PIAAC-L, namely the assessment of adult skills, the collection of background information, and conducting panel surveys:
GESIS—Leibniz Institute for the Social Sciences has first-hand expertise in PIAAC, as it was commissioned by the Federal Ministry of Education and Research to act as National Project Manager for PIAAC Germany. In addition, GESIS conducts several large-scale national and, in particular, international surveys on a regular basis.
The German Institute for Economic Research (DIW Berlin) has coordinated the key longitudinal household survey of income and living conditions in Germany, the German Socio-Economic Panel (SOEP), since 1984.
The Leibniz Institute for Educational Trajectories (LIfBi) is the coordinator of the National Educational Panel Study (NEPS), a project that collects longitudinal data on educational processes from early childhood to adulthood in Germany.
In order to combine expertise from these three central longitudinal and cross-sectional surveys, a consortium was constituted under the aegis of GESIS. All three institutes are experienced in research data infrastructures and are members of the Leibniz Association.3
Some of the other countries that participated in the first round of PIAAC have also followed up on their PIAAC respondents. In Canada, a subset of the sample persons for the Canadian longitudinal social survey Longitudinal and International Study of Adults (LISA) also participated in PIAAC and these respondents are being re-interviewed biennially as part of the longitudinal household survey. There was also a Polish follow-up study on PIAAC (postPIAAC), which collected longitudinal and additional background information on the PIAAC respondents in one additional wave of data collection (2014/2015) and also implemented some basic cognitive skills tests as well as a basic ICT skills test. Finally, a follow-up of PIAAC in Italy (also in 2014/2015) gathered longitudinal information on Italian PIAAC respondents and focused on obtaining more data on their non-cognitive skills. However, of all the PIAAC panel surveys, only the German PIAAC-L has carried out a second assessment of literacy and numeracy using PIAAC instruments (see description of Wave 2).
The design of the German PIAAC longitudinal study
PIAAC-L design and sample size for each of the three follow-up waves
RR (in %)
Extensive background information
Additional house-hold members
Extensive background information
Household composition, living conditions
Direct assessment of key competences (PIAAC numeracy and literacy; NEPS math and reading), PIAAC background variables and additional information
Spouse or partner
Direct assessment of key competences (NEPS math and reading), PIAAC background variables and additional information
Basic cognitive competences (SOEP), extensive background information, alternative question operationalizations
Additional house-hold members
Basic cognitive competences (SOEP), sociodemographic characteristics, alternative question operationalizations
Household composition, living conditions
The first follow-up wave was conducted in 2014, and thus approximately two years after the PIAAC 2012 assessment. It focused on collecting additional background information. All respondents (anchor persons and additional household members aged 18 years and older) were administered a comprehensive CAPI questionnaire (average duration: approximately 45 min) mainly consisting of items and questions regularly collected in the SOEP (see Wagner et al. 2007). It collected, among other individual characteristics, additional information on education and work history, employment situation, income, family background, place of origin, health status, religious affiliation, and time use (an overview of the concepts assessed in each of the three waves is provided in “Appendix”). To obtain more context information on the household, a separate household questionnaire (duration: 15 min on average)—similar to that applied in the SOEP—was administered. This questionnaire collected information on household income, living situation, living conditions and costs, as well as varied information on household members—especially children.
Design, sample sizes and mean durations for competence measurement in PIAAC-L Wave 2
Mean duration (minutes)
Besides the anchor persons, their spouses or partners living in the same household were interviewed and tested in the 2015 wave. However, spouses and partners were administered only the NEPS instruments for reading and mathematics, with just two different assessment conditions (in the second condition, the instruments were administered in reverse order). The resulting data thus enable similarities and differences in competences within couples to be compared.
Both skills assessments were administered under the same conditions as in PIAAC or NEPS, respectively. While the PIAAC assessment was computer-based by default, with an optional paper-based version for a small number of respondents who lacked computer experience, the NEPS assessment was paper-based only.6 Respondents taking the PIAAC assessment had unlimited time to solve the tasks, whereas respondents taking the NEPS assessment were timed by the interviewer and were allowed up to 28 min per domain. The administration of both timed and untimed tasks within the cognitive assessment is an innovative component of PIAAC-L.
In addition to the cognitive instruments, all respondents were administered a comparatively short background questionnaire which focused on central questions from the PIAAC questionnaire, for example on education, work status, and occupation. Some alternative measures to those implemented in PIAAC were also included, for example a module on skills use at work, which was conceptually similar to that administered in PIAAC but with a different operationalization (taken from the NEPS). In addition, efforts were made to collect a wider range of information relevant to the cognitive assessment, for example more information on computer experience and health. Given the design of the second wave of data collection, the questionnaire also included a module administered to the spouse or partner of the anchor person. A set of plausible values for the competence data will provide the basis for comparing PIAAC test performance over time, for comparing the actual test instruments from PIAAC and NEPS, and for analyzing the competence data within couples.
The third follow-up wave, which was conducted in 2016, again included all members of the anchor person’s household who were aged 18 years or over. In this wave, detailed background information—especially updates of previous information—was collected. The average duration of the person questionnaire and of the household questionnaire was 45 and 10 min, respectively. In addition, alternative operationalizations of particular constructs initially assessed in PIAAC were tested in this wave. They included an extended and re-worked module for measuring continuing education and revised questions to assess skill mismatch (Perry 2016; see also McGowan and Andrews 2015). These modules, and a number of other new items, were developed by researchers from the PIAAC Leibniz Network, a collaborative project of eight German Leibniz institutes; the PIAAC Leibniz Network aims at further analyzing, advancing, and disseminating PIAAC, and thus also PIAAC-L.
In addition to the background information, basic general cognitive skills of all respondents (i.e., anchor persons and all other participating adult household members) were assessed. For this purpose, short general cognitive tasks used in the SOEP were administered. The two ultra-short cognitive performance tasks allow for a reliable assessment of general intellectual ability and distinguish between two components of intellectual functioning: cognitive mechanics and pragmatics (see Richter et al. 2013).
To facilitate analyses of the competence data and the repeated measures, plausible values, which are the state of the art in analysis techniques for large-scale assessment data, will be provided, adopting the procedures from PIAAC as far as possible. Plausible values capture latent estimates for relations between competence measures and context data (OECD 2013b). With the additional data assessed in PIAAC-L, additional sets of plausible values will be needed for analyses. As a first step, the additional context data will be taken into account (first set of plausible values). As a second step, the repeated measurement of PIAAC literacy and numeracy will be included in an extended scaling and analysis model to enable longitudinal analyses (second set of plausible values). Furthermore, the analysis of the dyadic competence and context data will also require another extended item response model (third set of set of plausible values). For all three additional models presented, the complexity and the amount of context data in PIAAC-L constitutes a methodological challenge that will have to be addressed, as the standard techniques for generating plausible values might not work. Thus, alternative approaches in drawing plausible values by reducing the number of context variables either statistically or by theoretical selection and combinations thereof will be evaluated.
To account for selectivity, in each PIAAC-L wave weighting factors (nonresponse and cross-sectional) for each participating anchor person are computed and provided to the data users (see Bartsch and Poschmann 2016, for documentation on weighting in wave 1).
The extended and longitudinal data based on the German PIAAC 2012 sample provide researchers with a vast range of opportunities for addressing different research questions regarding skill development and sustainability—in particular skill development in adulthood as a function of central social and demographic circumstances, the effects of basic skills on labor market participation, careers, and unemployment, and, finally, the effects of basic skills on educational and employment mobility over the life course. The extended background information collected in the three waves of PIAAC-L also enables the investigation of the relationship between basic skills assessed in PIAAC and various constructs, such as central non-cognitive skills (see Rammstedt et al. 2016a, 2017), life satisfaction (Danner et al. 2016), and health.
In addition, the PIAAC-L data also enable methodological research questions to be addressed. For example, the PIAAC-L design makes it possible to compare the competence measures used in PIAAC, NEPS, and the SOEP. PIAAC-L was also set up to investigate alternative operationalizations of central constructs assessed in the PIAAC background questionnaire, such as skill mismatch and participation in continuing education. In addition, the longitudinal design allows panel attrition to be investigated, especially with regard to the anchor persons’ competence levels. Preliminary results provide first indications of possible nonresponse bias within PIAAC-L with regard to this key outcome variable (Rammstedt et al. 2016b).
Finally, the extension of the design to include the entire household and to measure the competences of the anchor person’s spouse or partner offers additional insights, for example into skill formation (Cunha et al. 2010) in the course of professional careers and educational trajectories and into assortative mating.
The above list of potential research questions is by far not exhaustive nor does it predetermine any prioritizing of the different endeavors.
Data from the three waves of the PIAAC-L study as well as accompanying documentation—including the questionnaires (in German), codebooks (in English), and technical reports (English)—are, or will be, available to researchers in the form of scientific use files through the PIAAC Research Data Center and the Data Archive at GESIS.7 Both German and international researchers interested in the data can submit a short application. Users will be required to sign a data user contract and will then obtain access to the data free of charge.8 Data from the first PIAAC-L wave were published at the end of the first quarter of 2016;9 data from the second wave was published in an aggregated form with the first wave by the end of 2016,10 and the third wave will be available by the end of 2017. For anchor persons, it will be possible to match data from all three PIAAC-L waves to data from the German PIAAC scientific use file.11 In addition, it is planned to link data of consenting anchor persons from PIAAC and PIAAC-L to microdata from the Federal Employment Agency.12 Access to these data will be made available through the Secure Data Center of the Statistics Department of the Federal Employment Agency.
From 2017 onwards, it is intended to include the remaining PIAAC-L participants and all other household members in the SOEP as a new refresher sample. In this course, information on the respondents will be updated according to the SOEP standardly assessed household and person information. In 2017, it is planned that PIAAC-L respondents will provide data on their current wealth situation. The future follow-ups will also be made available to the research community through the SOEP Research Data Center.
In sum, the resulting data from the German PIAAC longitudinal study allow a wide range of content-related and methodological research questions in the field of competence development and maintenance to be addressed, which would not be possible on the basis of PIAAC data only. Results of these analyses provide insights into the quality and predictive power of the concepts assessed in PIAAC. These results furnish crucial information for the revision of the instruments for upcoming PIAAC cycles and also provide empirical evidence on alternative measures and their quality. Finally, and very importantly, the design of PIAAC-L also provides empirical evidence on additional concepts that could be included in upcoming cycles of PIAAC. For example, the Big Five personality domains—the most highly used and most extensively researched personality approach (see for example John et al. 2008; McCrae and Costa 1999)—were assessed within PIAAC-L.13 In a recent analysis of the PIAAC-L data we were able to show that personality in terms of the Big Five and other concepts of non-cognitive skills prove to be highly predictive of numerous outcome variables measured in PIAAC (see Rammstedt et al. 2017).
From a national point of view, PIAAC-L has merged for the first time three distinct and very heterogeneous large-scale surveys conducted by three research data infrastructure institutes within the Leibniz Association in Germany. A direct comparison of measures such as the competence tests used in the different studies is thus possible using the PIAAC-L data. From a more general perspective, PIAAC-L can be regarded as a milestone on the road to stronger collaboration between, and perhaps a partial integration of, different surveys studying overlapping concepts. For data users, this can, in the medium term, improve concept harmonization and operationalization and thus yield higher data quality and new insights into causal mechanisms of skill formation and the outcomes of cognitive skills over the life course.
The terms skills and competences are used interchangeably in this article.
As outlined below, a household design was adopted in PIAAC-L. In order to differentiate between respondents initially recruited within PIAAC and the additional participating household members, the original PIAAC respondents are referred to as “anchor persons”.
PIAAC-L does not include the domain problem solving in technology-rich environments in its cognitive assessment.
In PIAAC, computer-based and paper-based administration could also occur within a single interview: respondents with low literacy and numeracy skills in the computer-based assessment subsequently worked on paper-based Reading Components tasks (OECD 2013b). Mixing modes in the assessment in this way thus proved to be feasible and unproblematic.
There is only a small handling fee of 20€ for 5 data sets.
GESIS—Leibniz Institute for the Social Sciences, German Socio-Economic Panel (SOEP) at DIW Berlin & LIfBi—Leibniz Institute for Educational Trajectories (2016a).
GESIS—Leibniz Institute for the Social Sciences, German Socio-Economic Panel (SOEP) at DIW Berlin & LIfBi—Leibniz Institute for Educational Trajectories (2016b).
Data stored at the Federal Employment Agency include detailed information on income and occupational biography on an individual basis. In order to allow to link individual data from PIAAC and PIAAC-L with the Federal Employment Agency data, anchor persons had to give their informed consent.
BR had the initial idea for the manuscript. BR, AZ and SM wrote and revised major parts of the manuscript. SM computed the data for the tables, AZ contributed the information in the appendix. CC added the section on the scaling and the computation of plausible values. JS contributed to the section on analytical potential of the longitudinal data. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Bartsch, S. & Poschmann, K. (2016). Weighting for PIAAC-L 2014. GESIS Papers 2016|xx: Köln.
- Block, J. (1995). A contrarian view of the five-factor approach to personality description. Psychological Bulletin, 117, 187–215.View ArticleGoogle Scholar
- Cunha, F., Heckman, J., & Schennach, S. (2010). Estimating the technology of cognitive and noncognitive skill formation. Econometrica, Econometric Society, 78, 883–931.View ArticleGoogle Scholar
- Danner, D., Luhmann, M., & Rammstedt, B. (2016). Competencies as predictors of life satisfaction. (Manuscript in preparation).
- European Social Survey. (2012). ESS6—2012 Fieldwork Summary and Deviations. Retrieved from http://www.europeansocialsurvey.org/data/deviations_6.html. October 16, 2016.
- European Social Survey. (2014). ESS7—2014 Fieldwork Summary and Deviations. Retrieved from http://www.europeansocialsurvey.org/data/deviations_7.html. October 16, 2016.
- GESIS—Leibniz Institute for the Social Sciences, German Socio-Economic Panel (SOEP) at DIW Berlin & LIfBi—Leibniz Institute for Educational Trajectories (2016a). PIAAC-Longitudinal (PIAAC-L), Germany. GESIS Data Archive, Cologne. ZA5989 Data file Version 1.1.0. doi:10.4232/1.12576.
- GESIS—Leibniz Institute for the Social Sciences, German Socio-Economic Panel (SOEP) at DIW Berlin & LIfBi—Leibniz Institute for Educational Trajectories. (2016b). PIAAC-Longitudinal (PIAAC-L), Germany. GESIS Data Archive, Cologne. ZA5989 Data file Version 2.0.0. doi:10.4232/1.12707.
- John, O. P., Naumann, L. P., & Soto, C. J. (2008). Paradigm shift to the integrative Big-Five trait taxonomy: History, measurement, and conceptual issues. In O. P. John, R. W. Robins, & L. A. Pervin (Eds.), Handbook of personality: Theory and research (3rd ed., pp. 114–158). New York, NY: Guilford Press.Google Scholar
- McCrae, R. R., & Costa, P. T. (1999). A five-factor theory of personality. In L. A. Pervin & O. P. John (Eds.), Handbook of personality (pp. 139–153). New York: Guilford.Google Scholar
- McGowan, M. A., & Andrews, D. (2015). Skill mismatch and public policy in OECD countries. OECD Working Paper, ECO/WKP(2015)28.
- OECD. (2013a). OECD skills outlook 2013: First results from the survey of adult skills. doi:10.1787/9789264204256-en.
- OECD. (2013b). Technical report of the survey of adult skills (PIAAC). Retrieved from OECD website: https://www.oecd.org/skills/piaac/_Technical%20Report_17OCT13.pdf.
- Perry, A. (2016). Developing a subjective skill mismatch measure for PIAAC and other surveys. (Manuscript in preparation).
- Rammstedt, B., Danner, D., & Lechner, C. (2017). The association between personality and life outcomes—results from the PIAAC longitudinal study in Germany. Large-scale Assessment in Education.
- Rammstedt, B., Danner, D., & Martin, S. (2016a). The association between personality and cognitive ability: Going beyond simple effects. Journal of Research in Personality, 62, 39–44. doi:10.1016/j.jrp2016.03.005.View ArticleGoogle Scholar
- Rammstedt, B., Lechner, C., & Martin, S. (2016b). Factors for selectivity in the transition from a cross-sectional assessment survey to a panel survey: Results from PIAAC 2012 and PIAAC-L. (Manuscript in preparation).
- Rammstedt, B., Martin, S., Zabal, A., Konradt, I., Maehler, D., Perry, A., Massing, N., Ackermann-Piek, D., & Helmschrott, S. (2016c). Programme for the International Assessment of Adult Competencies (PIAAC), Germany—Reduzierte Version. GESIS Datenarchiv, Köln. ZA5845 Datenfile Version 2.2.0. doi:10.4232/1.12660.
- Richter, D., Metzing, M., Weinhardt, M., & Schupp, J. (2013). SOEP scales manual. SOEP Survey Papers 138: Series C. Berlin: DIW/SOEP.
- Wagner, G. G., Frick, J., & Schupp, J. (2007). The German Socio-Economic Panel Study (SOEP)—Scope, evolution and enhancements. Schmollers Jahrbuch, 127(1), 139–169.Google Scholar
- Wasmer, M., Blohm, M., Walter, J., Scholz, E., & Jutz, R. (2014). Konzeption und Durchführung der “Allgemeinen Bevölkerungsumfrage der Sozialwissenschaften“(ALLBUS) 2012. GESIS Technical Report 2014/22.
- Wasmer, M., Scholz, E., Blohm, M., Walter, J., & Jutz, R. (2012). Konzeption und Durchführung der “Allgemeinen Bevölkerungsumfrage der Sozialwissenschaften” (ALLBUS) 2010. GESIS Technical Report 2012/12.
- Zabal, A., Martin, S., Massing, N., Ackermann, D., Helmschrott, S., Barkow, I., et al. (2014). PIAAC Germany 2012: Technical Report. Münster: Waxmann.Google Scholar