Coverage of the migrant population in large-scale assessment surveys. Experiences from PIAAC in Germany

Abstract

Background

European countries, and especially Germany, are currently very much affected by human migration flows, with the result that the task of integration has become a challenge. Only very little empirical evidence on topics such as labor market participation and processes of social integration of migrant subpopulations is available to date from large-scale population surveys. The present paper provides an overview of the representation of the migrant population in the German Programme for the International Assessment of Adult Competencies (PIAAC) sample and evaluates reasons for the under-coverage of this population.

Methods

We examine outcome rates and reasons for nonresponse among the migrant population based on sampling frame data, and we also examine para data from the interviewers’ contact protocols to evaluate time patterns for the successful contacting of migrants.

Results and Conclusions

This is the first time that results of this kind have been presented for a large-scale assessment in educational research. These results are also discussed in the context of future PIAAC cycles. Overall, they confirm the expectations in the literature that factors such as language problems result in lower contact and response rates among migrants.

Introduction

European countries, and especially Germany, are currently very much affected by human migration flows, in particular refugee flows. At present, the major drivers of migration are economic factors and armed conflicts. Although the refugees are seeking first and foremost temporary refuge, they may also wish to make a new home for themselves. European countries have long been confronted with the task of integrating migrants.¹ However, because of the somewhat unexpected extent of the current flows, the integration task is a challenge for most countries. Researchers have already developed several assumptions and models of how the integration process takes place. However, the empirical underpinning of these assumptions is based mostly on a small number of observed cases. This is due to several reasons, for example problems in reaching the migrant population and gaining their cooperation. Reaching more members of this population allows for more reliable verification of assumptions and more accurate derivation of action plans. Therefore, we need to improve the share of migrants in large-scale population surveys and thus enable policymakers to answer the integration questions that Europe is dealing with today. This paper aims to contribute to increasing the coverage of the migrant population in surveys (e.g., social surveys and large scale assessments) by investigating the reachability of adult migrants using level-of-effort paradata and by exploring reasons for migrant nonresponse.

Background

As Font and Méndez (2013) pointed out, there are, on the one hand, surveys that are specifically designed to measure and capture the realities of migrants. On the other hand, there are surveys that are designed to cover the general resident population in a given country, which includes individuals with a migration background. Several large-scale assessments, such as the Programme for the International Assessment of Adult Competencies (PIAAC), are examples of the latter approach. PIAAC is an international survey conducted under the auspices of the OECD that assesses basic skills of adults aged 16–65 years in the areas of literacy, numeracy, and problem solving in technology-rich environments in the official language or languages of the respective participating countries. Information on social and language background, time in country, education, and labor force status, among other topics, was collected in a personal interview. As proficiency in the language of the host country is one central factor for the successful social and economic integration of migrants, the PIAAC data are an excellent source with which these questions can be further explored (OECD 2013a). For integration-related research, however, it is crucial that the core migrant population be suitably covered. For migrants, the basic skills assessed in PIAAC, such as literacy and numeracy, can serve as indicators of the extent to which they have achieved key prerequisites for social participation or structural integration in the host country. Literacy, for instance, is assessed through tasks such as reading and understanding text passages of varying length and difficulty, for example a medical package insert, a short newspaper article, or an online job advertisement (OECD 2013a). The tasks used to measure literacy are related to everyday life situations and are comparable for individuals from different countries as well as from various migrant subgroups within these countries (Zabal et al. 2014). However, if we take a look at the skills of migrants in the PIAAC countries overall (e.g., OECD 2013a; Maehler et al. 2014), we can observe a literacy gap between natives and migrants: The proportion of adults classified as individuals with a low literacy level (i.e., level I and below) is, on average, twice as high in the subpopulation with a migration background (operationalized here as first-generation migrants). Considering literacy as the target variable, and assuming that persons with low literacy skills are less likely to participate in surveys, the results might even be further tilted against migrants’ literacy. In this context, it is important to look at effects that are potentially induced by nonresponse. In the case of a registry-based sample design (e.g., PIAAC Germany; Zabal et al. 2014), sample units—in this case, individuals—are selected from a population register. These registers might be affected to some extent by incomplete or out-of-date information. If, for example, migrants were selected into the gross sample on the basis of the current information available in the population register, but they have since moved abroad without de-registering with their local registration office (Martin et al. 2015; Salentin 2014), it would not have been possible to contact them and they would have become nonrespondents. If migrants with low literacy skills happened to be more likely to have moved abroad than those with high literacy levels—for example, because, being low-skilled, they had fewer job opportunities—this could lead to an overestimation of the literacy level in the resident migrant subpopulation.

In addition, a comparison with the majority population (i.e., natives) can provide information as to whether a given nonresponse behavior is specific to persons with a migration background. Thus, the question that arises is whether, and why, the response rate of the migrant population differs from that of the majority population. There could be several reasons why nonresponse occurs, for example incomplete address information, refusal to participate, inability to participate because of absence, or even inability to communicate because of language barriers. Information about structural differences in response rates could contribute to improving future large-scale assessments. PIAAC offers a unique opportunity to pursue this question on the basis of a high-quality, large-scale survey in educational research, thus covering a broad range of the adult population in the participating countries.

However, the comparability of the country-specific sample designs and selection procedures in PIAAC is limited. First, there is variation in sampling designs and sampling frames across countries (Mohadjer et al. 2013). On the one hand, there is a distinction between countries using household samples (e.g., Canada, England, or the United States) and countries using registry-based samples (e.g., Austria, Spain, or Sweden). On the other hand, within the group of countries who use registry-based samples, the sampling frames are sometimes decentralized registers (e.g., in Germany) and sometimes centralized registers (e.g., in Sweden). Second, the differences in the sampling frame information across PIAAC countries have an impact on both the identification and the classification of migrants. Countries with household samples do not usually have information on migration background (such as citizenship and country of birth) in their gross sample. By contrast, registry countries have access to some migration-related data. However, this information is not harmonized across countries (e.g., country of birth in Sweden, citizenship in Austria and Germany). Beyond that, the coverage of the migrant subpopulation across countries is subject to other moderating factors, such as different fieldwork strategies. These measures include, for instance, the translation of invitation letters (e.g., Germany; Zabal et al. 2014, p. 70), the use of bilingual interviewers (e.g., United States; Hogan et al. 2016, p. 5–9) or translators to administer the questionnaire (e.g., Austria; Statistik Austria 2013, p. 31), the translation of questionnaires into the languages of selected migrant groups (e.g., Austria, United States), or even the exclusion of the migrant population altogether (e.g., Japan; OECD 2016, p. 52). Finally, the largest obstacle is the restricted access to the data required for nonresponse analyses, such as gross sample and outcome information (e.g., disposition codes). Thus, our analyses will focus on German PIAAC data only, as Germany is currently one of the EU countries most affected by the flow of refugees and is in need of more information in order to make decisions about future integration measures.

Sampling frame and the coverage of the migrant population in Germany

There are several possible approaches to sampling an adult migrant population, such as name-based selection from telephone directories, use of person-centered networks (e.g., snowballing), or selection from population registers on the basis of distinguishing personal characteristics. In his overview for Germany, Salentin (2014) compared the advantages and disadvantages of sampling frames and concluded that a combination of a registry-based sample design and a name-based procedure (onomastics), or name-based sampling in a population register, would be the most appropriate approach to achieve a representative sample of the migrant population. At present, the implementation of a registry-based sample design in Germany for such a representative sample of the migrant population is subject to potential restrictions (Salentin 2014): first, registry-based samples can be drawn only if the survey is of public interest; second, not all existing information can be used as a selection criterion. For sample selection in large-scale surveys of public interest in Germany, information obtainable from population registers is limited to specific variables, namely age, gender, and citizenship. Although place of birth is generally recorded, the current registration legislation does not permit the distribution of these data.² However, citizenship as the sole criterion leads to an underestimation of individuals with a migration background and to a potential distortion of the social structure (Salentin 2014). For example, naturalized migrants (i.e., migrants who have acquired the citizenship of the host country) who were born abroad are classified as natives. In addition, in the case of Germany, it is not possible to identify for sampling purposes one large migrant sub-population, namely the ethnic German resettlers (Aussiedler).

The German 2012 PIAAC sample is a registry-based sample and is representative of Germany’s adult population aged between 16 and 65 years. The core population includes all individuals who were resident in Germany at the time of data collection and who were not living in institutions, such as prisons, nursing homes, etc. (Zabal et al. 2014). The German population registers hold information on all individuals who are permanently resident in Germany (mandatory registration) and on individuals who enter Germany (legally) and expect to stay in the country for at least three months. Hence, to be part of the German PIAAC core population, it was crucial that the usual place of residence (principal residence) was Germany, while citizenship, legal status, or first language were not critical in this case (Mohadjer et al. 2013; OECD 2010). In Germany, the realized net sample comprised 5465 respondents (Zabal et al. 2014, p. 9). With regard to the available sampling frame criterion citizenship, for example, 395 (unweighted) were non-Germans.

As mentioned earlier, a registry-based random sample has limitations that are relevant for the selection of the migrant population. For example, persons who have recently moved and have not yet registered at their new place of residence cannot be covered by the register. This is particularly relevant for migrants in Germany, who are most likely to be unfamiliar with the practice of registration (Salentin 2014). In certain circumstances, moves from one municipality to another are reported only with considerable delay, or only if proof of current residence is required for other purposes. In addition, some migrants fail to deregister with their local registration office when they permanently return to their country of origin. In consequence, in some cases, the addresses of selected individuals are no longer current (Martin et al. 2015). Moreover, migrants may be residing in the host country illegally. According to an estimate provided by the Hamburgisches Weltwirtschaftsinstitut (2010),³ the proportion of illegal immigrants in the target population in Germany at the time of sample selection in PIAAC Germany was approximately .5%. However, this figure is subject to change due the current refugee movements.

Furthermore, an additional particularity of the PIAAC study should be mentioned. Following international standards, the procedure for weighting the German PIAAC data consisted of several steps (Martin et al. 2013; Zabal et al. 2014). While age and gender had to be included in the set of variables for the final weighting step, there was no guideline regarding the inclusion of further variables, such as citizenship or country of birth (OECD 2010). Weighting adjustment for nonresponse, however, included the variables citizenship (German vs. non-German), age, and municipality size. In the final weighting step (benchmarking to external data), the German PIAAC data were adjusted to data from the 2010 Microcensus⁴ for age, gender, region, and education level (Zabal et al. 2014). Citizenship was not considered in this step in order to ensure both the inclusion of the most essential variables and the minimization of the number of weighting cells. Even though citizenship was used at some point during the weighting process, (similarly to other countries), it was not benchmarked to population totals in the final step, so that an almost perfect alignment could not be achieved. As a result, the weighted proportion of non-German individuals in the PIAAC sample was 9.4%, compared to 10.7% in the 2010 Microcensus data (Zabal et al. 2014, p. 88).

In summary, the target population in PIAAC was the non-institutionalized population aged 16–65 years residing in the country at the time of data collection. In Germany, this core population was successfully covered, and the appropriate sampling frame was used (Zabal et al. 2014). However, it cannot be assumed that the target migrant subpopulation was fully covered and sampled without survey errors (e.g., because migrant-specific information was not used for stratification in the sampling procedure). This is also likely to be the case in other countries participating in PIAAC. Hence, a registry-based sampling frame can be a limiting factor for answering specific migration-related questions and for subsequent analyses. For integration-related questions, the extracted survey data are quite appropriate when migrant is operationalized through foreign citizenship (Maehler et al. 2015; Salentin 2014). As noted earlier, the German population registers provide only selected information, such as age, gender, and citizenship. To operationalize migrant on the basis of place of birth in order to analyze different generations of migrants, this information must be requested directly during the survey. Subsequently, the sample population can then be compared, for example, with Microcensus data to retrospectively check the representativeness of the sample, and weighting procedures can be used post hoc to adjust for deviations.

In the next section, we will summarize the relevant literature on nonresponse, which is related to outcome rates such as contact and cooperation rates, and the reasons for nonresponse among the migrant population, particularly in Germany.

Nonresponse among the migrant population: previous findings and theoretical framework

One of the main aspects addressed in this paper is nonresponse error. In the reference literature, it is specifically classified as a “unit nonresponse error” (Groves and Couper 1998), which occurs when a sampled unit—such as an individual from the migrant subpopulation—refuses to participate in a face-to-face survey or when an eligible sample member cannot be contacted (Biemer 2010). As Groves (2006) pointed out, undercoverage problems resulting from the sampling frame and from nonresponse lead to underrepresentation of population (sub-)groups. Consequently, parts of the core population are not adequately represented.

The consequences of the registry-based sampling frame for the coverage of the migrant population in PIAAC Germany have been discussed above. Nonresponse, on the other hand, is related to non-contact or to non-cooperation once contacted. In the context of contact rates, several authors (e.g., Baykara-Krumme 2010; Blohm and Diehl 2001; Koch 1997; Feskens et al. 2006) have reported a low accessibility of migrants (operationalized by the criterion citizenship) in Germany because of higher mobility (e.g., longer visits to the country of origin) or due to specific work schedules (e.g., shift work) or self-employment. It transpires that the probability of making contact with the sample persons, in particular migrants, is related to the time spent at home. Feskens et al. (2006), for example, discovered that in several European countries the non-contact rates were higher for migrants than for non-migrants and that these substantially lower contact rates still held true when socio-economic status, urbanization, and several other demographic variables were controlled for. In Germany, it appears that especially older migrants and male migrants are more difficult to reach (Feskens et al. 2006): Older migrants, and Turkish migrants in particular, often visit their country of origin for a longer period of time. Based on nonresponse analyses using German ALLBUS data (1996), Blohm and Diehl (2001) reported that incorrect addresses were also a reason for non-contact among the migrant population. However, in more complex analyses based on ALLBUS data from the year 2000, migration status (citizenship) had no effect on the probability of contact (e.g., Blohm et al. 2007).

Throughout the literature, there is no consistent evidence that non-cooperation rates are higher for individuals with a migration background (Font and Méndez 2013). While Blohm and Diehl (2001) found evidence of lower refusal rates for migrants, Deding et al. (2013) showed that non-cooperation rates were higher for the migrant groups observed in their survey. Their investigation of Iranian, Turkish, and Pakistani migrants in Denmark revealed, that, for migrants, indirect refusal (i.e., other persons refuse contact with the person in question) occurred more often in the case of women from patriarchal cultures, for example. Feskens et al. (2006) compared outcome rates from surveys in six different countries and found that cooperation rates were higher for migrants (in the authors’ terminology ethnic minorities) than natives. However, they assumed that these results could have been masked by language problems, as migrants may have had problems communicating a refusal and were instead coded as not able to participate. Nonparticipation due to inability was found to be always higher for migrant populations. This finding is supported by Baykara-Krumme (2010), who observed a high non-cooperation rate among migrants in Germany, due mainly to language-related issues.

The literature about the probability of cooperation in surveys with migrants is based on the social isolation hypothesis (e.g., Font and Méndez 2013; Helmschrott and Martin 2014). According to this hypothesis, socially isolated individuals are out of touch with mainstream society and behave in line with subgroup norms, or rather reject the norms of the majority. It is assumed that socially isolated individuals will be less likely to accede to a survey request than non-isolated individuals (Groves and Couper 1998). This might be the case for individuals who have immigrated to a new country and are not (yet) integrated into the host society. Thus, it is also associated with the length of stay in a country and with the question of whether migrants have acquired the citizenship of that country (by naturalization). These factors may have an effect on the survey cooperation rate as they are related to different dimensions of integration (cultural, economic, social, and emotional) in the host country (e.g., Esser 2008; Maehler 2012).

As mentioned above, the non-cooperation rate among migrants in Germany is strongly related to language issues (Baykara-Krumme 2010; Feskens et al. 2006). The implementation of surveys in the German language leads to systematic nonresponse among migrants, and especially among those migrants with a shorter length of stay in the country (e.g., Salentin 2014). Hence, to ensure a high response rate in PIAAC, strict standards were established by the international PIAAC Consortium (e.g., Rammstedt et al. 2014). These requirements were not only essential for sampling the core population, but were, in part, also suggested in the literature on the surveying of migrant populations (Font and Méndez 2013). Therefore, we will address related steps aimed at enhancing survey response in PIAAC.

Enhancing survey response in PIAAC

Fieldwork procedures can have an effect on the response rate of migrants (Feskens et al. 2006; Font and Méndez 2013). Font and Méndez (2013) recommended tailoring fieldwork procedures to the considerably different survey response behaviors of migrants and non-migrants. Méndez et al. (2013), for example, proposed strategies such as the alignment of interviewing times to better suit the schedules of migrants or the provision of special training to interviewers in order to enable them to adapt to different types of non-national respondents. It is assumed that these activities could contribute to achieving higher response rates among migrants and better coverage of that population.

In Germany—as in many other countries—a constant decline in response rates has been observed in large-scale face-to-face surveys over the last decades (European Social Survey 2012, 2013; Wasmer et al. 2012). Thus, in PIAAC Germany, great efforts were made to increase participation of the core population, thereby yielding a strikingly high overall response rate of 55% (Zabal et al. 2014). The international PIAAC Consortium defined a detailed set of high quality standards, such as the requirement to achieve a high response rate (OECD 2010). However, no general recommendations were made to put specific effort into reaching and including migrants in the sample.

Some of the measures taken, such as introductory material (e.g., advance letter, brochure) or the use of incentives (Martin et al. 2014; Zabal et al. 2014), addressed the respondent directly. Other measures, such as an intensive five-day training workshop prior to fieldwork or thorough quality control and monitoring throughout fieldwork, were aimed at improving interviewer performance. Special attention was given to contacting target persons and gaining their cooperation. Interviewers were instructed to make at least four in-person contact attempts before closing a case. With a view to increasing the contact rate, interviewers were instructed to contact target persons on different days of the week and at different times of the day.

Fieldwork was organized into main working phases and several re-issue phases. Approximately one-third of the sample were considered for re-issuing and were followed up in one of the re-issue phases. The re-issued cases were mainly soft refusals,⁵ non-contacts, or sampled persons who had moved to another municipality or who had an invalid address. For the latter group, procedures were employed to trace the new addresses of these individuals. In some cases, interviewer reassignments were made. In order to address more of the population with non-German citizenship, special documents were developed for the re-issue phase: (a) an endorsement letter from the German Federal Ministry of Education and Research and (b) advance letters and FAQ documents in the languages of the major migrant groups in Germany (among others, Turkish, Polish, and Russian).⁶

Hypotheses about nonresponse among the migrant subpopulation in PIAAC

In this paper, three main research questions will be addressed using data from PIAAC Germany. In our first, three-part, research question, we will investigate whether migrants and non-migrants differ in terms of response rates. When doing so, we will focus, first, on differences in the outcome rates of migrants and non-migrants and, second, on differences in the outcome rates of migrants by gender and age group. After examining the overall outcomes (contact, able to be interviewed, cooperation, and participation), we will focus in the third part of our first research question on one specific outcome, namely contact (see the aforementioned findings by, e.g., Baykara-Krumme 2010; Feskens et al. 2006), and compare, in particular, the contact rates among migrants and non-migrants by age group and gender. As explained above, the only key indicator in the PIAAC gross sample that can be used to operationalize migration background is citizenship. Thus, for our analyses migrants are defined as non-German citizens and non-migrants as holders of German citizenship. Consequently, our second research question explores reasons for possible differences in the response rates. And finally, to increase participation of migrants in future surveys, it is important to know when the target sub-groups are reachable and whether they differ from non-migrants in this regard. This is the subject of our third research question.

As discussed in the literature, response rates of persons with a migration background are lower than those of natives. We will investigate, first, whether migrants and non-migrants differ in their response behavior, and we will test the following hypothesis:

(1.1) The overall outcome rates for migrants are lower than for non-migrants.

Focusing only on migrants’ outcome rates, it is assumed that, as proposed in the literature (e.g., Feskens et al. 2006), older migrants and male migrants were more difficult to reach. Thus, transferring these findings to PIAAC Germany, we will test the following hypotheses:

(1.2) The outcome rates for older migrants are lower than for younger migrants.

(1.3) The outcome rates for migrant males are lower than for migrant females.

After investigating the overall outcome rates of migrants, we will compare the contact rates of migrants and non-migrants. As proposed in the literature, we assume that the contact rates of migrants are lower than those of non-migrants. We will investigate whether this assumption is valid for different age groups. Due to the high mobility and specific work schedules of migrants (Feskens et al. 2006), and their higher rate of self-employment, the third part of our first research question asks whether the proportion of non-contacted males was higher among migrants than among non-migrants. Thus, the following hypotheses will be tested:

(1.4) The contact rates for migrants are lower than for non-migrants across age groups.

(1.5) The contact rate for migrant males is lower than for non-migrant males.

Our second research question focuses on the main reasons for nonresponse among migrants in PIAAC Germany. Helmschrott and Martin (2014) investigated the potential for nonresponse bias in the PIAAC Germany data and found that being a migrant correlated with nonresponse: Migrants were indeed significantly less likely to participate than non-migrants. Thus, we ask: How do migrants and non-migrants differ in their response behavior and what are the reasons for nonresponse? In accordance with the literature (i.a., Blohm and Diehl 2001), it is expected that nonresponse among migrants is due mainly to (1) language problems, (2) refusals (direct, or indirect through other persons), and (3) address-related reasons (e.g., invalid address or the person has moved). As the survey language is usually the official language of the country in question, it could be expected that language problems are the major cause of nonresponse. Consequently, we are interested in testing whether refusals, language problems, and address-related reasons are the main causes of nonresponse among migrants, and whether migrants differ from their non-migrant counterparts because of address-related and literacy-related reasons. The following hypothesis will be tested:

(2) A low response rate of migrants compared to non-migrants is related mainly to refusals, language problems, and address-related reasons.

Our third research question asks: When are migrants reachable? According to a study of households in the Netherlands conducted by Stoop (2004), the chances of contacting the total population were higher in the evening. Similarly, Blohm et al. (2007) stated that interviewers working primarily in the afternoon were more successful in contacting target persons from the German population compared to other times of the day, and that the contact rate was lower on the weekend. To help achieve a higher response rate of migrants and a better coverage of the migrant population, Méndez, Ferreras, and Cuesta (2013) proposed strategies such as the alignment of interviewing times to better suit the needs of the foreign population. In order to meet these requirements in PIAAC Germany, interviewers were—among other measures—instructed to establish contact with the sampled persons on different days of the week and at different times of the day (Zabal et al. 2014). The PIAAC data offer a vast dataset with which to explore contact rates by time. Thus, we will explore whether the contact rate is related to the indicators for contact time (namely, the time of day, the day of the week, and the period of the year). Based on the few findings in the literature, we will investigate if the contact rates of migrants and non-migrants are correlated with contact times, in particular we investigate the following hypothesis:

(3.1) The probability to contact both migrants and non-migrants is higher during the evening than during other times of the day.

(3.2) The probability to contact both migrants and non-migrants is higher during the week than during the weekend.

(3.3) The probability to contact both migrants and non-migrants during holidays is lower than during the other periods of the year.

Methods and data

For our analyses, we need data that are available for both respondents and nonrespondents. Thus, we use frame information (such as first citizenship, age, and gender) as well as auxiliary variables and paradata such as disposition codes or contact data from interviewers collected in Germany (Rammstedt et al. 2014) during the PIAAC fieldwork phase (August 2011–March 2012). PIAAC is designed to provide representative measures of cognitive skills of adults aged 16–65 years. In PIAAC, a sampled person is defined as a completed case if the person completed an adequate proportion of the background questionnaire and at least some basic part of the cognitive assessment, or if he or she was classified as a literacy-related nonrespondent for whom age and gender were collected (OECD 2010, 2013b). Literacy-related reasons for nonresponse include language problems, reading and writing difficulties, and learning or mental disabilities. According to the OECD (2013a), these respondents tended to have lower proficiency levels. In the German PIAAC net sample, approximately 1.6% were literacy-related nonrespondents (.8% non-migrants). As stated above, we have to use the information about citizenship as a proxy for the migration status. Hence, in what follows, persons who are not holders of German citizenship are defined as migrants and persons who hold German citizenship are defined as non-migrants.

The unweighted gross sample consists of N = 10,240 cases. Based on the frame information first citizenship, n = 931 target persons are classified as migrants and n = 9049 as non-migrants.⁷ 51.7% of the migrants are males, compared to 50.4% of the non-migrants. The average age of migrants (38 years) is slightly lower than that of non-migrants (41 years). Regarding citizenship, the largest proportion of the migrants hold a Turkish passport (22.1%), followed by Italian (7.8%), Polish (7.3%), Greek (5.2%), former Yugoslavian (4.3%), Russian (3.4%), and Croatian (3.1%) passport holders.

To answer the first (three-part) research question, and to test the hypotheses derived from it (1.1 to 1.5) regarding the outcome rates in general and the contact rates in particular, we used PIAAC disposition codes and computed outcome rates according to AAPOR standards (The American Association for Public Opinion Research 2016)⁸:

• Contact (following AAPOR CON1: I + P + R + O/I + P + R + O + NC + U)

• Able to be interviewed (I + P + R/I + P + R + O)

• Cooperation (following AAPOR COOP4: I + P/I + P + R

• Participation (following AAPOR RR2: I + P/I + P + R + O + NC + U

To investigate the reasons for non-participation (our second research question), we focused on the disposition codes used in PIAAC Germany, as these differentiate several literacy- and address-related reasons for nonresponse. We used the final distribution of disposition codes for the unweighted German gross sample, separated by citizenship.⁹ For comparison purposes, a differentiation similar to that in the technical report for the overall population (Zabal et al. 2014, p. 76) was chosen.

To analyze contact rates by time as outlined in our third research question, we used the paradata, that is, the data provided by interviewers in their contact protocols (in PIAAC, these protocols are called case folders). The PIAAC case folder is a document that is available for each sampled person and is used by the interviewer to record all contact activities (such as date, number and time of contact attempts, and the result of each contact or contact attempt). The majority of the sampled individuals were successfully contacted in one of the first two contact attempts. For example, among migrants the first contact attempt was successful in 37.1% of the cases and among Germans in 36.2% of the cases.

Hence, to test the third hypothesis (that the contact rates of migrants and of non-migrants are correlated with contact time), we used three time indicators from the PIAAC paradata for the first contact (attempt),¹⁰ namely the time of day, the day of the week, and the period of the year. The time of the day was categorized into three ranges: before lunch (12 am), after lunch (12 pm to 5 pm) and in the evening (after 5 pm). The days of the week were grouped into four periods: (1) Monday/Tuesday, (2) Wednesday, (3) Thursday/Friday, and (4) Saturday/Sunday. And finally, the period of the year was categorized into school holidays (no/yes) and religious holidays (no/yes). To address school holidays, we used the information about school holidays in the respective German federal states during the assessment time (summer, winter, and autumn holidays). To address religious holidays, we categorized Easter and Christmas time in the respective years of assessment. And finally, we also controlled for gender and age. To predict the contact probability, we performed separate regression analyses for migrants and non-migrants. The dichotomous variable contact (yes/no) was used as an independent variable.

Results

Do migrants and non-migrants differ in their response behavior?

To provide an overview on the response rate of migrants in PIAAC Germany, and to verify the assumptions in literature, we coded survey outcomes in accordance with the AAPOR standards into four groups: (a) contact, (b) able to be interviewed,¹¹ (c) cooperation, and (d) participation. Table 1 provides a descriptive overview categorizing migrants and non-migrants by gender and age group.

Table 1 Outcome rates by migration status in PIAAC Germany (categorized in accordance with AAPOR standards)

First, it can be noted that, for all four fields, migrants’ outcome rates are lower than the outcome rates of non-migrants (hypothesis 1.1). There is a difference of 11.7 percentage points [χ² (1) = 138.915, p < .001] for the contact rate, 10.4 percentage points [χ² (1) = 209.023, p < .001] for the ability to be interviewed, 4.5 percentage points [χ² (1) = 4.869, p < .05] for the cooperation rate, and 15.2 percentage points [χ² (1) = 74.712, p < .001] for the response rate.

Looking at migrants only, it can be observed that some outcome rates vary significantly across age groups (hypothesis 1.2). While the contact rates are fairly closely distributed around 80% across age groups [χ² (4) = 7.980, p < .10], there is a steady decline in the cooperation rate [χ² (4) = 40.928, p < .001] and the participation rate [χ² (4) = 46.951, p < .001] from the youngest to the oldest age group.

The evaluation of the outcome rates by gender (hypothesis 1.3) show that male migrants are significantly more difficult to contact than female migrants [χ² (1) = 5.339, p < .021]. However, contrary to the expectation, a significant difference between genders could not be confirmed with regard to migrants’ cooperation and participation rates.

The next question of interest is whether there are differences in contact rates between migrants and non-migrants across age groups (hypothesis 1.4). While, for the non-migrant population, the highest contact rate (95.9%) can be observed for the oldest age group (55–65 years), migrants in the middle working-age group (35–44 years) show the highest contact rate (86.1%). For both migrants and non-migrants, the contact rate for individuals between the ages of 25 and 34 is the lowest compared to other age groups. Comparing contact rates of migrants and non-migrants by age shows that, for each inspected age group, the contact rate for migrants is significantly lower than for non-migrants (16–24: χ² (1) = 15.933, p < .001; 25–34: χ² (1) = 21.415, p < .001; 35–44: χ² (1) = 9.862, p < .01; 45–54: χ² (1) = 43.035, p < .001; 55–65: χ² (1) = 66.687, p < .001).

Table 1 shows that male migrants have the lowest contact rate overall (77.6%) compared to female migrants (83.7%), non-migrant males (91.2%), and non-migrant females (93.5%). In accordance with the hypothesis (1.5), the probability of being contacted was significantly lower for migrant males than for non-migrant males [χ² (1) = 83.268, p < .001].

How do migrants and non-migrants differ in their response behavior and what are the reasons for nonresponse?

Table 2 shows the final distribution of disposition codes for the unweighted non-migrant gross sample, separated by migration status (operationalized by citizenship). In general, the main reason for the nonresponse of migrants (28.8%) and non-migrants (34.4%) was refusal by the sample person. Refusals were more common among non-migrants than migrants (z = 1.95, p = .05). It can also be seen that an address-related issue, for example an invalid address, was apparently more often the reason for nonresponse among migrants than among non-migrants (z = 2.36, p = .02). In line with the hypothesis, migrants differed from non-migrants with regard to nonresponse due to literacy-related reasons, such as language problems or reading or writing difficulties (.6% vs. 7.7%; z = 2.16, p = .02). Furthermore, Table 2 shows that non-contact with the household or the sample person seems to be more likely in the case of migrants (7.6%) than non-migrants (5.3%), even though this difference does not reach significance. Migrants also appear to be more often unreachable because they have left Germany (3.5% vs. .5%, n.s.).

Table 2 Disposition codes for PIAAC Germany by migration status

In summary, migrants and non-migrants differed in their response behavior, particularly with regard to address-related and literacy-related reasons. Thus, in line with the hypothesis (2), a low response rate of migrants was related mainly to refusals, language problems, and address-related reasons.

If we look at reasons for nonresponse between genders, Table 3 shows a similar picture for migrants and non-migrants: There are no significant differences. However, among migrants, it appears that literacy-related reasons were more often a cause of non-participation on the part of females than males (10.2 vs. 5.4%, n.s.). By contrast, male migrants appear to have been harder to reach due to address-related issues than female migrants (12.1 vs. 7.3%, n.s.).

Table 3 Disposition codes for PIAAC Germany by migration status and gender

Looking closer at response behavior by age, Table 4 shows the results for hypothesis 1.2 that across all age groups the rate of completed interviews is higher for non-migrants than for migrants (z ranges between 2.03 and 4.08, ps = .00). This difference is particularly visible for the older age groups of migrants.

Table 4 Disposition codes for PIAAC Germany by migration status and age

Across all age groups, nonresponse due to address-related issues was slightly more pronounced for migrants than for non-migrants. In addition, among migrants, nonresponse due to literacy-related reasons slightly increases across age groups and was higher for older migrants than for non-migrants in the comparable age group (12.3 vs. .9%, n.s.).

When are migrants reachable?

In the next step, the contact rate for the first attempt was predicted using the time indicators day of the week, time of the day, and time of the year (school and religious holidays) as independent variables, and controlling for gender and age (see Table 5).

Table 5 Logistic regression to predict contact on first attempt, by migrants and non-migrants

As can be seen in Table 5, the results show that, among migrants, controlling for gender and age, the time of the day, and the period of the year had a significant impact on the probability of being contacted. The probability of contacting a migrant individual was higher in the evening (in line with hypothesis 3.1) compared to before lunch time (p = .05). However, contrary to the assumption, for migrants the probability of contact is not related to the time within the week (hypothesis 3.2). The criterion time of contact within the year (hypothesis 3.3) indicates that migrants were not equally reachable during the school holidays (summer, winter, and autumn). During school holidays, for instance, the probability of contacting people with a migration background was lower (p = .001) than outside the school holidays. By contrast, the contact rate of migrants is not correlated with periods of the year that are religious holidays.

For the non-migrant subpopulation, the time of the day and the day of the week had a significant effect on the probability of being contacted. The evening was (in line with hypothesis 3.1) the best time to reach non-migrants in comparison to times before or after lunch (ps = .001). In addition, the probability of being contacted was (contrary to hypothesis 3.2) significantly higher on Saturdays and Sundays, compared to the other periods of the week, Monday/Tuesday, Wednesday, or Thursday/Friday. In contrast to the migrant subpopulation, the period of the year (school holidays) does (as assumed in hypothesis 3.3) not seem to have had an influence on the probability of being contacted.

Discussion

Global migration has considerably increased in the last decade and has been overwhelming for many European countries. Within a short time, the number of refugees and asylum seekers has risen considerably in many countries. In the coming months and years, these people will become part of the societies of the European countries concerned. Large-scale social surveys, such as future cycles of PIAAC, will be confronted with this fact and will have to assess whether and how to cover this group of people in the survey design. Using data from Germany, a country that is strongly affected by migration flows, as a case example, the present article addresses substantial questions related to the nonresponse behavior of migrants.

With the aim of examining the migrant subpopulation, the first objective of the present contribution was to find out more about response behavior differences between the migrant subpopulation and the majority population and differences in the outcome rates within the migrant subpopulation by age and gender. As the results confirm, the overall outcome rates were lower for migrants than for non-migrants. Looking only at the migrant subpopulation, it was revealed, as expected, that outcome rates—in particular the cooperation rate and the response rate—were lower for older than for younger migrants. With regard to outcome rates by gender, the only effect was found for the contact rates. In line with previous findings, male migrants were more difficult to contact than female migrants. By further investigating contact rate differences between migrants and non-migrants, it could be observed that contact rates were lower for migrants across all age groups. It should be emphasized that the highest contact rate among migrants was observed in the middle working-age group (35–44 years), whereas for the non-migrant subpopulation the highest contact rate was found for the oldest age group (55–65 years). In addition, the data revealed that male migrants were the most difficult to contact. Overall, the results confirm the expectations in the reference literature. It is noteworthy, however, that migrants in the middle working-age group (35–44 years) were the most accessible group. We assume that this fact may be related to their specific employment schedules.

Addressing the second main research question, reasons for differences in the response behavior of migrants and non-migrants were further investigated. The PIAAC results are in line with the literature. Migrants and non-migrants differed in their response behavior, particularly due to address- and literacy-related reasons. It could be observed that especially literacy-related reasons (including language problems), refusals, and address-related reasons (unavailability during the field period) had an impact on the response behavior of migrants.

Finally, we looked for the best time to contact the migrant and non-migrant subpopulations. We did not find any evidence relating to the migrant population in the previous literature. As our results revealed, contact attempts for both groups were most successful in the evening, compared to before and after lunch. The day of the week (e.g., Monday/Tuesday vs. Saturday/Sunday) of the contact attempt does not appear to affect the reachability of migrants. However, compared to other periods of the week, the weekend was the most appropriate time to contact the non-migrant subpopulation. In line with the literature, school holiday time led to a lower contact rate, but only for the migrant subpopulation. As the present analyses focused on the first contact attempt only, future research should investigate whether interviewers varied contact day or contact time for subsequent contact attempts and whether this had an effect on the contact rate of the more reluctant migrant subpopulation.

Limitations when surveying migrants and suggestions for future research

Addressing limitations of large-scale surveys in general, it must be noted that, on the one hand, they aim to cover the whole resident population, but, on the other hand, they exclude de facto quite a large proportion of the population if survey instruments are administered in only one language version. The core resident population is restricted to those who have sufficient proficiency in the respective country’s official language. Not only new migrants, whose social inclusion process would be interesting to investigate, but also long-resident migrants are more likely to be excluded because they tend to refuse to participate in the survey.

The results from PIAAC Germany indicate that, to some extent, the migrant subpopulation was not covered by the survey. This was due most probably to language barriers. Assuming that this problem also exists in several other OECD countries, it could be discussed whether measures could be taken to better address the migrant subpopulation in future cycles of PIAAC. These measures could include recruiting bilingual interviewers (Méndez et al. 2013) or translating questionnaires into several languages within countries. In some countries, such as Austria or the United States, for example, the PIAAC background questionnaires were translated for a substantial percentage of the migrant population. However, it must be considered that implementing such measures to account for diverse cultural backgrounds can involve some constraints, such as a significant increase in costs.

Besides the above-mentioned potential measures to better address the migrant population, the selection of a separate migrant sample could be considered (e.g., as an add-on study). With the increasing number of refugees and related illegal immigrants not only in Germany but also across Europe, it appears to be even more advisable to consider this approach. A separate migrant sample would allow researchers to focus on specific analyses with regard to labor market and social integration processes. To find out more about integration processes and acculturation, it would, for example, be advisable for future research to compare migrants from the same country of origin (e.g., Syria, Turkey, Eritrea, Ghana) across different PIAAC countries. However, if the survey focus were on refugees only, PIAAC countries vary considerably regarding the accessibility of refugees for interviewing (e.g., because they live in refugee housing facilities). This is due mainly to differences in the legal regulation systems (from arrival to application for asylum) and the speed with which migrants are structurally and socially integrated. Furthermore, comparisons across countries will be limited because (a) the countries of origin of the current refugees vary across the different host countries, and (b) the proportion of refugees per country may vary substantially (see, e.g., EUROSTAT 2016). In Germany, for instance, some refugees who migrated to Germany in the last few years or months might get a residence permit and thereby become part of the resident population (and thus the target population) and might be selected and interviewed in a future cycle of PIAAC. For others without a residence permit, or for illegal immigrants, for example, the use of population registers as a sampling frame may lead to exclusion and thus render it necessary to consider alternative sampling approaches. These are precisely the challenges in survey research that researchers and survey institutes in Germany are dealing with at present (see, e.g., the feasibility study conducted by the Expert Council of German Foundations on Integration and Migration; Schiefer 2016). Implementing a cluster sample of local registration offices might be an option. However, these data do not contain information about the residency status, for example (one of several possible selection criteria). In addition, the data are not equally distributed with regard to the countries of origin, or the address quality is poor (e.g., strong fluctuation; Schiefer 2016).

Returning to limitations regarding the registry-based sampling frame that were illustrated exemplarily for Germany in this article, and to the definition of the term migrant, a major challenge appears to persist. In the case of Germany, a large percentage of individuals are classified as non-migrants (according to their citizenship) for the purposes of analyzing the contact rate, but, according to their birthplace, they have a migration background. The information about the place of birth would allow for more accurate nonresponse analyses and thus the planning of precise measures to increase outcome rates, in particular the contact rate.

The aim in large-scale surveys such as PIAAC could be to obtain more reliable data (e.g., large N) for the migrant population that can be compared with the general population, and to survey a sufficient proportion of different migrant groups (at least the largest ones). Non-contacts or refusals are not the only main obstacle to achieving high response rates of migrants. As Font and Méndez (2013) pointed out, a further obstacle is the fact that some migrant groups are difficult to locate. For random samples of population registers, the authors suggested obtaining samples that are much larger than the final sample one wants to achieve. Due to different patterns of response rates between migrants and non-migrants, Morales and Ros (2013) recommended designing fieldwork procedures tailored to the considerably different survey response behaviors of these different groups. Méndez et al. (2013), for example, endorsed strategies such as providing training to interviewers to enable them to adapt to different types of non-national respondents, which could help to achieve a higher response rate by migrants and a better coverage of the population.

In conclusion, for the next PIAAC cycle, it would be important to put more effort into reaching and including the migrant subpopulation in the sample. In addition to gaining representativeness in terms of their proportion in general, a more accurate coverage of migrants should also be pursued, for example with regard to length of residency, skills in the host country’s language, country of origin, and other sociodemographic characteristics. It can be assumed that the composition of the migrant population in a country is diverse and that migrants differ with regard to their profiles and nonresponse behavior. For example, recently arrived migrants, in particular, presumably have lower skills in the language of the host country and are thus more likely to achieve comparatively low results in literacy and to be more prone to nonresponse. Addressing these circumstances, thought should first be given to whether the definition of the PIAAC target population should, for example, deliberately include or exclude recently arrived migrants. If a re-definition of the target population is not a suitable option, it would be interesting to further pursue measures to overcome language barriers as reasons for non-participation. Results presented in this paper show language to be a barrier to contact and participation of migrants in PIAAC, so that translation of the background questionnaire, for the largest migrant groups at least, seems reasonable. Last but not least, it is important to evaluate other PIAAC countries’ experiences in surveying migrants and to derive therefrom specific recommendations for better coverage of the migrant subpopulations. In a further step, this would enable the formulation of specific guidelines for addressing migrants in the sample (e.g., specific contact strategies, use of bilingual interviewers, etc.) and thus facilitate the international comparability of the outcomes for migrant subpopulations.