Threat-Inducing Violent Events Exacerbate Social Desirability Bias in Survey Responses

A key challenge in survey research is social desirability bias: respondents feel pressured to report acceptable attitudes and behaviors. Building on established findings, we argue that threat-inducing violent events are a heretofore unaccounted for driver of social desirability bias. We probe this argument by investigating whether fatal terror attacks lead respondents to overreport past electoral participation, a well-known and measurable result of social desirability bias. Using a cross-national analysis and natural and survey experiments, we show that fatal terror attacks generate turnout overreporting. This highlights that threat-inducing violent events induce social desirability, that researchers need to account for the timing of survey fieldwork vis-à-vis such events, and that some of the previously reported post-violent conflict increases in political participation may be more apparent than real.     

How to estimate the policy preferences of party supporters: Disaggregating data from voting advice applications versus modeling survey responses

Valid and reliable estimates of the policy preferences of political parties' supporters are essential for the study of political representation. However, such estimates are not directly available from standard surveys of public opinion, which are typically representative by design only at the national level and rarely ask questions about public support for specific policies. In this article, we explore the possibility to use data from voting advice applications (VAA) to estimate the policy preferences of party supporters. To do that, first, we identify 10 questions on preferences towards issues of public policy that were asked around the same time and with similar wording in traditional surveys of public opinion and in VAAs fielded in Germany and in the Netherlands. Then we compare the VAA data disaggregated by political affiliation of the respondents to the survey data adjusted via multilevel regression modeling with poststratification (MRP). We find strong positive correlations between the estimates derived from both methods, especially after weighting the VAA data. Yet, point estimates are not always very close, and the match is sensitive to the treatment of neutral and ‘don't know’ answers. Overall, our results bode well for the validity of using VAA data in empirical research on political representation.     

Asked and Answered: Knowledge Levels When We Will Not Take “Don't Know” for an Answer

A pivotal claim in research on citizen competence is that the typical citizen knows very little about politics. Public opinion surveys provide a considerable body of evidence in support of this position. However, survey protocols with respect to factual questions about politics violate established norms in psychometric research on educational testing in that don't know answers are encouraged rather than discouraged. Because encouraging don't know responses potentially confounds efforts to identify substantive understanding, this practice may lead to the systematic understatement of political knowledge. We explore this possibility with data drawn from three split-ballot tests: one conducted as part of a survey in the Tallahassee, Florida, metropolitan area, one conducted as part of the 1998 NES Pilot, and one conducted as part of the 2000 NES. Results reveal that the mean level of political knowledge increases by approximately 15% when knowledge questions are asked in accordance with accepted practices in educational testing.     

The Advent of Internet Surveys for Political Research: A Comparison of Telephone and Internet Samples

The Internet offers a number of advantages as a survey mode:low marginal cost per completed response, capabilities for providingrespondents with large quantities of information, speed, andelimination of interviewer bias. Those seeking these advantagesconfront the problem of representativeness both in terms ofcoverage of the population and capabilities for drawing randomsamples. Two major strategies have been pursued commerciallyto develop the Internet as a survey mode. One strategy, usedby Harris Interactive, involves assembling a large panel ofwilling respondents who can be sampled. Another strategy, usedby Knowledge Networks, involves using random digit dialing (RDD)telephone methods to recruit households to a panel of Web-TVenabled respondents. Do these panels adequately deal with theproblem of representativeness to be useful in political scienceresearch? The authors address this question with results fromparallel surveys on global climate change and the Kyoto Protocoladministered by telephone to a national probability sample andby Internet to samples of the Harris Interactive and KnowledgeNetworks panels. Knowledge and opinion questions generally showstatistically significant but substantively modest differenceacross the modes. With inclusion of standard demographic controls,typical relational models of interest to political scientistsproduce similar estimates of parameters across modes. It thusappears that, with appropriate weighting, samples from thesepanels are sufficiently representative of the U.S. populationto be reasonable alternatives in many applications to samplesgathered through RDD telephone surveys.     

Money, Time, and Political Knowledge: Distinguishing Quick Recall and Political Learning Skills

Surveys provide widely cited measures of political knowledge. Do seemingly arbitrary features of survey interviews affect their validity? Our answer comes from experiments embedded in a representative survey of over 1200 Americans. A control group was asked political knowledge questions in a typical survey context. Treatment groups received the questions in altered contexts. One group received a monetary incentive for answering the questions correctly. Another was given extra time. The treatments increase the number of correct answers by 11–24%. Our findings imply that conventional knowledge measures confound respondents' recall of political facts with variation in their motivation to exert effort during survey interviews. Our work also suggests that existing measures fail to capture relevant political search skills and, hence, provide unreliable assessments of what many citizens know when they make political decisions. As a result, existing knowledge measures likely underestimate people's capacities for informed decision making.     

The Persistent Connection Between Language-of-Interview and Latino Political Opinion

Since the advent of public opinion polling, scholars have extensively documented the relationship between survey response and interviewer characteristics, including race, ethnicity, and gender. This paper extends this literature to the domain of language-of-interview, with a focus on Latino political opinion. We ascertain whether, and to what degree, Latinos’ reported political attitudes vary by the language they interview in. Using several political surveys, including the 1989–1990 Latino National Political Survey and the 2006 Latino National Survey, we unearth two key patterns. First, language-of-interview produces substantively important differences of opinion between English and Spanish interviewees. This pattern is not isolated to attitudes that directly or indirectly involve Latinos (e.g., immigration policy, language policy). Indeed, it emerges even in the reporting of political facts. Second, the association between Latino opinion and language-of-interview persists even after statistically controlling for, among other things, individual differences in education, national origin, citizenship status, and generational status. Together, these results suggest that a fuller understanding of the contours of Latino public opinion can benefit by acknowledging the influence of language-of-interview.     

The Effect of Interviewer Gender on the Survey Response

We examined the existence of gender-of-interviewer effects in two local-area surveys in which male and female interviewers were randomly assigned to interview male and female respondents. Small but consistent gender-of-interviewer effects arose on questions related to the women's movement, women's issues, and gender equality, demonstrating that, as expected, respondents were more likely to provide feminist answers to female interviewers. Gender-of-interviewer effects were somewhat more pronounced and consistent on controversial political topics: the women's movement (feminists and political activism) and their policy agenda. There was mixed evidence on whether respondents were equally susceptible to gender-of-interviewer effects. In one of the surveys, gender-of-interviewer effects were more pronounced among less well-educated and younger respondents than among respondents who were better educated or older. This effect was not replicated in the second survey.     

Artificial Inflation or Deflation? Assessing the Item Count Technique in Comparative Surveys

While the popularity of using the item count technique (ICT) or list experiment to obtain estimates of attitudes and behaviors subject to social desirability bias has increased in recent years among political scientists, many of the empirical properties of the technique remain untested. In this paper, we explore whether estimates are biased due to the different list lengths provided to control and treatment groups rather than due to the substance of the treatment items. By using face-to-face survey data from national probability samples of households in Uruguay and Honduras, we assess how effective the ICT is in the context of face-to-face surveys—where social desirability bias should be strongest—and in developing contexts—where literacy rates raise questions about the capability of respondents to engage in cognitively taxing process required by ICT. We find little evidence that the ICT overestimates the incidence of behaviors and instead find that the ICT provides extremely conservative estimates of high incidence behaviors. Thus, the ICT may be more useful for detecting low prevalence attitudes and behaviors and may overstate social desirability bias when the technique is used for higher frequency socially desirable attitudes and behaviors. However, we do not find strong evidence of variance in deflationary effects across common demographic subgroups, suggesting that multivariate estimates using the ICT may not be biased.     

Partisan Gaps in Political Information and Information‐Seeking Behavior: Motivated Reasoning or Cheerleading?

Do partisan disagreements over politically relevant facts, and preferences for the information sources from which to obtain them, represent genuine differences of opinion or insincere cheerleading? The answer to this question is crucial for understanding the scope of partisan polarization. We test between these alternatives with experiments that offer incentives for correct survey responses and allow respondents to search for information before answering each question. We find that partisan cheerleading inflates divides in factual information, but only modestly. Incentives have no impact on partisan divides in information search; these divides are no different from those that occur outside the survey context when we examine web‐browsing data from the same respondents. Overall, our findings support the motivated reasoning interpretation of misinformation; partisans seek out information with congenial slant and sincerely adopt inaccurate beliefs that cast their party in a favorable light.     

Reconsidering the Measurement of Political Knowledge

Political knowledge has emerged as one of the central variablesin political behavior research, with numerous scholars devotingconsiderable effort to explaining variance in citizens' levelsof knowledge and to understanding the consequences of this variancefor representation. Although such substantive matters continueto receive exhaustive study, questions of measurement also warrantattention. I demonstrate that conventional measures of politicalknowledge—constructed by summing a respondent's correctanswers on a battery of factual items—are of uncertainvalidity. Rather than collapsing incorrect and "don't know"responses into a single absence-of-knowledge category, I introduceestimation procedures that allow these effects to vary. Grouped-datamultinomial logistic regression results demonstrate that incorrectanswers and don't knows perform dissimilarly, a finding thatsuggests deficiencies in the construct validity of conventionalknowledge measures. The likely cause of the problem is tracedto two sources: knowledge may not be discrete, meaning thata simple count of correct answers provides an imprecise measure;and, as demonstrated by the wealth of research conducted inthe field of educational testing and psychology since the 1930s,measurement procedures used in political science potentiallyresult in "knowledge" scales contaminated by systematic personalityeffects.     

Political learning during the 2000 U.S. presidential election: The impact of the campaign on the gender gap in political knowledge

We examine how political campaigns influence individuals’ levels of correct, incorrect, and don’t know responses and the gender gap in political knowledge during the 2000 American presidential campaign. Using data from the 2000 National Annenberg Election Survey (NAES), we demonstrate that as the campaign progresses the electorate provides more correct answers and fewer incorrect answers. Moreover, the political campaign significantly reduces (and possibly eliminates) the direct effect of sex on political knowledge. While the political campaign decreases the number of incorrect answers provided by both men and women, the number of correct answers provided by women increases. Our findings highlight the importance of the political campaign in determining relative levels of political knowledge for men and women.     

Political Competition and Ethnic Identification in Africa

This article draws on data from over 35,000 respondents in 22 public opinion surveys in 10 countries and finds strong evidence that ethnic identities in Africa are strengthened by exposure to political competition. In particular, for every month closer their country is to a competitive presidential election, survey respondents are 1.8 percentage points more likely to identify in ethnic terms. Using an innovative multinomial logit empirical methodology, we find that these shifts are accompanied by a corresponding reduction in the salience of occupational and class identities. Our findings lend support to situational theories of social identification and are consistent with the view that ethnic identities matter in Africa for instrumental reasons: because they are useful in the competition for political power.     

Vote Buying and Social Desirability Bias: Experimental Evidence from Nicaragua

Qualitative studies of vote buying find the practice to be common in many Latin American countries, but quantitative studies using surveys find little evidence of vote buying. Social desirability bias can account for this discrepancy. We employ a survey‐based list experiment to minimize the problem. After the 2008 Nicaraguan municipal elections, we asked about vote‐buying behavior by campaigns using a list experiment and the questions traditionally used by studies of vote buying on a nationally representative survey. Our list experiment estimated that 24% of registered voters in Nicaragua were offered a gift or service in exchange for votes, whereas only 2% reported the behavior when asked directly. This detected social desirability bias is nonrandom and analysis based on traditional obtrusive measures of vote buying is unreliable. We also provide systematic evidence that shows the importance of monitoring strategies by parties in determining who is targeted for vote buying.     

Quadratic voting in the wild: real people,real votes

Since their introduction in 1932, Likert and other continuous, independent rating scales have become the de facto toolset for survey research. Scholars have raised significant reliability and validity problems with these types of scales, and alternative methods for capturing perceptions and preferences have gained traction within specific domains. In this paper, we evaluate a new, broadly applicable approach to opinion measurement based on quadratic voting (QV), a method in which respondents express preferences by ‘buying’ votes for options using a fixed budget from which they pay quadratic prices for votes. Comparable QV-based and Likert-based survey instruments designed by Collective Decision Engines LLC were evaluated experimentally by assigning potential respondents randomly to one or the other method. Using a host of metrics, including respondent engagement and process-based metrics, we provide some initial evidence that the QV-based instrument provides a clearer measure of the preferences of the most intensely motivated respondents than the Likert-based instrument does. We consider the implications for survey satisficing, a key threat to the continued value of survey research, and discuss the mechanisms by which QV differentiates itself from Likert-based scales, thus establishing QV as a promising alternative survey tool for political and commercial research. We also explore key design issues within QV-based surveys to extend these promising results.     

Attitude generalization and the measurement of trust in American leadership

For nearly a decade informed observers of American public opinion have vigorously debated the “crisis of confidence” question: Does the ongoing tide of political distrust ported cataclysmic change in the body politic or is the public harmlessly letting off some steam? This paper approaches the debate from a new perspective by reflecting on the manner in which researchers have sought to measure public confidence in leaders and political institutions. It is argued herein that researchers' dependence on global measures of political trust has lead to overestimation of the extent to which public confidence has eroded. Evidence is presented which suggests that survey respondents engage in a process of attitude generalization which causes them to express distrust of leaders generally when only certain specific leaders are actually seen as culpable. The methodological and theoretical significance of this sort of generalization is discussed.     

Comparing and Combining List and Endorsement Experiments: Evidence from Afghanistan

List and endorsement experiments are becoming increasingly popular among social scientists as indirect survey techniques for sensitive questions. When studying issues such as racial prejudice and support for militant groups, these survey methodologies may improve the validity of measurements by reducing nonresponse and social desirability biases. We develop a statistical test and multivariate regression models for comparing and combining the results from list and endorsement experiments. We demonstrate that when carefully designed and analyzed, the two survey experiments can produce substantively similar empirical findings. Such agreement is shown to be possible even when these experiments are applied to one of the most challenging research environments: contemporary Afghanistan. We find that both experiments uncover similar patterns of support for the International Security Assistance Force (ISAF) among Pashtun respondents. Our findings suggest that multiple measurement strategies can enhance the credibility of empirical conclusions. Open‐source software is available for implementing the proposed methods.     

The wisdom of crowds design for sensitive survey questions

Survey research on sensitive questions is challenging because respondents often answer untruthfully or completely refuse to answer. Existing indirect questioning techniques address the problem of social desirability bias at the expense of decreasing estimates' efficiency. We suggest the Wisdom of Crowds survey design that does not pose a tradeoff between anonymity and efficiency as an alternative. We outline the conditions necessary for the technique to work and test them empirically. Moreover, we compare the Wisdom of Crowd estimate of a right-wing populist party's vote share to alternative indirect questioning techniques' estimates as well as to the official election result in the 2017 German federal election. Provided its conditions are met, the Wisdom of Crowds design performs best in terms of both bias and efficiency. We conclude that the Wisdom of Crowds design is an important addition to social scientists' survey methodology toolbox.     

Revisiting the Gender Gap in Political Knowledge

Gender-based differences in political knowledge are pervasive in the United States and abroad. Previous research on the source of these differences has focused on resource differentials or instrumentation, with scholars arguing either that the gender gap is real and intractable, or that it is an artifact of the way the concept is measured. Our study differs from past work by showing that (1) male–female differences in political knowledge persist even when knowledge is measured with recommended practices, but that (2) knowledge gaps can be ameliorated. Across laboratory, survey, and natural experiments, we document how exposure to information diminishes gender-based differences in political knowledge. The provision of facts reduces—and often eliminates—the gender gap in political knowledge on questions covering a range of topics.     

Social Desirability Bias in Measuring Public Service Motivation

ABSTRACT

Public service motivation (PSM) is usually measured using self-report data, which suggests that PSM measures can be influenced by social desirability bias. This study investigates whether respondents tend to report an inflated view of their attitudes and orientations during surveys on PSM. Experimental survey research (list experiment) is conducted to analyze the magnitude of social desirability bias in PSM measurements and to examine the relationship between socio-demographic factors and social desirability bias in Korea. The results show evidence of social desirability responding bias, although the pattern of bias varies across socio-demographic subgroups. Respondents in their forties or fifties, conservatives, Protestants, and those without a religious affiliation show more bias than other subgroups. This finding implies that correlational analysis in PSM research may be prone to the moderating effect of social desirability bias. Research that does not recognize and compensate for this bias may produce unwarranted theoretical or practical conclusions.