Bias in OMB's economic forecasts and budget proposals

The Office of Management and Budget (OMB) annually submits the President's budget for the U.S. government to the Congress. The economic forecasts and revenue and outlay proposals contained in the budget have been criticized as biased, especially during the 1980s. Tests for bias in one year ahead proposals for the 1963–89 period show no bias in economic forecasts and revenue estimates, but substantial bias in outlay proposals. Most outlay proposals are consistently less than actual outcomes, accounting for underprediction of the Federal deficit. OMB outlay proposals appear to be influenced by politics. Republican administrations show more significant proposal biases, with defense proposals higher and domestic outlay proposals lower than outcomes. The Office of Management and Budget consistently understates the deficits by resorting to the most optimistic economic assumptions it can credibly — and now sometimes even incredibly — employ. — Senator James SasserWall Street Journal, 25 January 1990     

All the Best Polls Agree with Me: Bias in Evaluations of Political Polling

Do Americans consider polling results an objective source of information? Experts tend to evaluate the credibility of polls based on the survey methods used, vendor track record, and data transparency, but it is unclear if the public does the same. In two different experimental studies—one focusing on candidate evaluations in the 2016 U.S. election and one on a policy issue—we find a significant factor in respondent assessments of polling credibility to be the poll results themselves. Respondents viewed polls as more credible when majority opinion matched their opinion. Moreover, we find evidence of attitude polarization after viewing polling results, suggesting motivated reasoning in the evaluations of political polls. These findings indicate that evaluations of polls are biased by motivated reasoning and suggest that such biases could constrain the possible impact of polls on political decision making.

     

The Distortion of Related Beliefs

When forming beliefs about themselves, politics, and how the world works more generally, people often face a tension between conclusions they inherently wish to reach and those which are plausible. And the likelihood of beliefs about one variable (e.g., the performance of a favored politician) depends on beliefs about other, related variables (e.g., the quality and bias of newspapers reporting on the politician). I propose a formal approach to combine these two forces, creating a tractable way to study the distortion of related beliefs. The approach unifies several central ideas from psychology (e.g., motivated reasoning, attribution) that have been applied heavily to political science. Concrete applications shed light on why successful individuals sometimes attribute their performance to luck (“imposter syndrome”), why those from advantaged groups believe they in fact face high levels of discrimination (the “persecution complex”), and why partisans disagree about the accuracy and bias of news sources.     

Why do analysts use structured analytic techniques? An in-depth study of an American intelligence agency

This article presents findings from the first publicly available survey generalizable to an intelligence agency to explore why analysts use structured analytic techniques (SATs). Mandated by the Intelligence Reform and Terrorism Prevention Act (2004), SATs are simple methodologies supposed to make analysis more transparent and, hopefully, valid. Despite the US government’s investment in training thousands of analysts, there is no solid evidence on how often or why analysts actually use SATs. A survey of 80 analysts and nine follow-up interviews at the State Department’s Bureau of Intelligence and Research reveals a simple, but important, truth: implementing the techniques requires training and compelling evidence they will improve analysis. Other factors, most notably the amount of time pressure an analyst experiences, were not significantly related with the use of the techniques despite anecdotal accounts and conjecture from the literature. Future research should examine other intelligence agencies to cross-validate these findings. If these findings hold in other cases, intelligence agencies should focus on reforming and incorporating evidence into the training process.     

General courts,specialized courts,and the complementarity effect

Among the major decisions any legal system must make is deciding whether to establish general courts with broad jurisdiction, or specialized courts with limited jurisdiction. Under one influential argument—advanced by both judges and legal theorists—general courts foster coherence within the legal system. This Article identifies a distinct effect of establishing general courts: the “complementarity effect.” In the case of complementarity, general courts strategically apply different principles in different fields, such that litigants losing in one sphere (e.g., public law) are compensated in another (e.g., private law). We support this conjecture by analyzing three case studies.     

Lessons in disguise: multivariate predictive mistakes in?collective choice models

Many political processes can be characterized as repeated collective decisions, in which individual choices rendered by the same actors are combined to produce salient outcomes (e.g., voting in legislatures). The inherent interdependence among the choices within repeated collective decisions is often missed in statistical analysis. I introduce the joint prediction error (JPE)—a?grouping of individual decisions that is poorly predicted by a model. JPEs capture the intersecting information missed by conventional diagnostics. I?demonstrate the use of JPEs on data from two published articles—one on U.S. Supreme Court voting and another on international defense alliances.     

No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments

Manipulation checks are often advisable in experimental studies, yet they rarely appear in practice. This lack of usage may stem from fears of distorting treatment effects and uncertainty regarding which type to use (e.g., instructional manipulation checks [IMCs] or assessments of whether stimuli alter a latent independent variable of interest). Here, we first categorize the main variants and argue that factual manipulation checks (FMCs)—that is, objective questions about key elements of the experiment—can identify individual‐level attentiveness to experimental information and, as a consequence, better enable researchers to diagnose experimental findings. We then find, through four replication studies, little evidence that FMC placement affects treatment effects, and that placing FMCs immediately post‐outcome does not attenuate FMC passage rates. Additionally, FMC and IMC passage rates are only weakly related, suggesting that each technique identifies different sets of attentive subjects. Thus, unlike other methods, FMCs can confirm attentiveness to experimental protocols.     

The Condition of Community Streets and Parks: A Comparison of Resident and Nonresident Evaluations

Practitioners and scholars are concerned that citizen surveys about community services are heavily influenced by respondents' opinions on other issues and by their sociodemographic backgrounds. We search for these biases by examining the extent to which citizen assessments of streets and parks in Iowa communities match the assessments of a nonresident. The citizens' ratings correlate significantly with the nonresident's ratings, indicating that citizen evaluations are not entirely the product of other influences. However, further analysis reveals some bias. In particular, streets are rated higher in wealthy towns, towns high in political efficacy, and towns where residents rate government services good overall. Parks are rated higher in towns where people come together to solve problems and in towns where people rate government services good overall. Even with these biases, our research indicates that citizen evaluations convey reasonably accurate information about the condition of community streets and parks.     

The Product and Difference Fallacies for Indirect Effects

Political scientists often cite the importance of mechanism‐specific causal knowledge, both for its intrinsic scientific value and as a necessity for informed policy. This article explains why two common inferential heuristics for mechanism‐specific (i.e., indirect) effects can provide misleading answers, such as sign reversals and false null results, even when linear regressions provide unbiased estimates of constituent effects. Additionally, this article demonstrates that the inferential difficulties associated with indirect effects can be ameliorated with the use of stratification, interaction terms, and the restriction of inference to subpopulations (e.g., the indirect effect on the treated). However, indirect effects are inherently not identifiable—even when randomized experiments are possible. The methodological discussion is illustrated using a study on the indirect effect of Islamic religious tradition on democracy scores (due to the subordination of women).     

Using Bayesian Aldrich‐McKelvey Scaling to Study Citizens' Ideological Preferences and Perceptions

Aldrich‐McKelvey scaling is a powerful method that corrects for differential‐item functioning (DIF) in estimating the positions of political stimuli (e.g., parties and candidates) and survey respondents along a latent policy dimension from issue scale data. DIF arises when respondents interpret issue scales (e.g., the standard liberal‐conservative scale) differently and distort their placements of the stimuli and themselves. We develop a Bayesian implementation of the classical maximum likelihood Aldrich‐McKelvey scaling method that overcomes some important shortcomings in the classical procedure. We then apply this method to study citizens' ideological preferences and perceptions using data from the 2004–2012 American National Election Studies and the 2010 Cooperative Congressional Election Study. Our findings indicate that DIF biases self‐placements on the liberal‐conservative scale in a way that understates the extent of polarization in the contemporary American electorate and that citizens have remarkably accurate perceptions of the ideological positions of senators and Senate candidates.     

What's in a word?

In this paper, we argue that citizens' views about the personality characteristics of their political leaders are subject to systematic bias, motivated by the desire to maintain existing impressions. We report two studies designed to explore how such trait biases, involving judgments of real politicians, are manifested. Drawing on work in social psychology (John, Hampson, and Goldberg, 1991), we hypothesize, and demonstrate experimentally, that traitbreadth plays a critical role in the judgment process. Specifically, people tend to select broad positive traits (e.g., kind) and narrower negative traits (gullible) to describe politicians that they like; conversely, people tend to select broad negative traits (unintelligent) and narrower positive traits (soft-hearted) to describe politicians that they dislike. This trait breadth attribution bias was much stronger among those who had a positive, rather than negative, impression of the politician, a result we discuss in the context of related evidence of affective asymmetries in political judgment (e.g., Kinder, 1986; Marcus and Mackuen, 1993). More generally, these studies illustrate the potential associated with integrating cognitive, affective, and motivational factors in the service of fully understanding political judgment and choice.     

Can Leviathan governments mitigate the tragedy of the commons?

This paper explores the conjecture whether the Leviathan motive of politicians — to tax for the purpose of raising revenues rather than for benevolent, Pigovian motives — helps to overcome the inefficiency of international pollution spillovers such as in the cases of acid rain and global warming. It turns out that this conjecture is true in a static context that captures flow externalities, e.g., acid rain, as long as environmental damages are not too high. In contrast, Leviathan motives aggravate the already existing inefficiency in the case of stock externalities (e.g., global warming) despite probably high taxes at the beginning.     

Rethinking the training of intelligence analysts

Abstract

Despite intense scrutiny and promised fixes resulting from intelligence ‘transformation’ efforts, erroneous analytic assessments persist and continue to dominate news coverage of the US intelligence community. Existing analytic training teaches analysts about common cognitive biases and then aims to correct them with structured analytic techniques. On its face, this approach is eminently reasonable; on close inspection, incomplete and imbalanced. Current training is anchored in a mid-twentieth century understanding of psychology that focuses on checking over-confidence and rigidity but ignores the problems of under-confidence and excessive volatility. Moreover it has never been validated against objective benchmarks of good judgment. We propose a new approach: (a) adopting scientifically validated content and regularly testing training to avoid institutionalizing new dogmas; (b) incentivizing analysts to view training guidelines as means to the end of improved accuracy, not an end in itself.     

An Effective Approach to the Repeated Cross‐Sectional Design

Repeated cross‐sectional (RCS) designs are distinguishable from true panels and pooled cross‐sectional time series (PCSTS) since cross‐sectional units (e.g., individual survey respondents) appear but once in the data. This poses two serious challenges. First, as with PCSTS, autocorrelation threatens inferences. However, common solutions like differencing and using a lagged dependent variable are not possible with RCS since lags for i cannot be used. Second, although RCS designs contain information that allows both aggregate‐ and individual‐level analyses, available methods—from pooled ordinary least squares to PCSTS to time series—force researchers to choose one level of analysis. The PCSTS tool kit does not provide an appropriate solution, and we offer one here: double filtering with ARFIMA methods to account for autocorrelation in longer RCS followed by the use of multilevel modeling to estimate both aggregate‐ and individual‐level parameters simultaneously. We use Monte Carlo experiments and three applied examples to explore the advantages of our framework.     

Motivated Skepticism in the Evaluation of Political Beliefs

We propose a model of motivated skepticism that helps explain when and why citizens are biased-information processors. Two experimental studies explore how citizens evaluate arguments about affirmative action and gun control, finding strong evidence of a prior attitude effect such that attitudinally congruent arguments are evaluated as stronger than attitudinally incongruent arguments. When reading pro and con arguments, participants (Ps) counterargue the contrary arguments and uncritically accept supporting arguments, evidence of a disconfirmation bias . We also find a confirmation bias —the seeking out of confirmatory evidence—when Ps are free to self-select the source of the arguments they read. Both the confirmation and disconfirmation biases lead to attitude polarization —the strengthening of t₂ over t₁ attitudes—especially among those with the strongest priors and highest levels of political sophistication. We conclude with a discussion of the normative implications of these findings for rational behavior in a democracy.     

Irresolvable cultural conflicts and conservation/development arguments: Analysis of Korea’s Saemangeum project

This paper examines, through argument analysis & grid-group model, how cultural bias, as the frame of reference of an advocacy coalition (AC), brings about irresolvable conflicts and produces divided arguments between coalitions for development and conservation in Koreas Saemangeum project. Based on different cultural biases, two ACs – the advocacy coalition for development (ACD) and the advocacy coalition for conservation (ACC) – interpreted the same facts differently in line with their cultural orientations and ways of life. The dynamic argument patterns reflected each coalitions cultural bias, which restricted the frame of reference of actors in each AC. After reviewing argument analysis as an analytic tool, we introduce cultural theoryin which ways of life, consisting of cultural biases and social relations, amplify the irresolvable conflicts between two ACs. Second, to show the culturally constructed nature of the conflicts, we analyze the contrasting arguments between the ACD (dominated by a hierarchy bias) and ACC (led by egalitarianism) in the Saemangeum project. Third, we discuss the implications of Mary Douglas and Aaron Wildavskys cultural theoryto advocacy coalition framework (ACF) as follows: 1) constraining effects on inter-coalitional learning by cultural biases, 2) coexistence of different solidarities under a coalition, and 3) asymmetric relationships between parties in a coalition.     

Googling the requester: Identity‐questing and discrimination in public service provision

Innovating upon previous field experiments and theories of identity‐based discrimination, we test whether public officials are using searches (“identity‐questing”) to profile citizens and acting on latent biases. Pairs of “institutional” and “noninstitutional” requesters send lower and moderate burden freedom of information (FOI) requests—providing no identity cues apart from undistinctive names, e‐mails, and ID numbers—to nearly 700 of Brazil's largest municipalities. Results show institutional requesters receive one‐fifth more responses than noninstitutional comparators. For moderate versus lower burden requests, noninstitutional requesters are 11% less likely to receive a compliant response than their institutional comparators. The only plausible explanation for these results is identity‐questing, a phenomenon that has far‐reaching policy implications. Most of the world's FOI laws, for example, contain vague ID obligations, which translate incoherently from laws to regulation and practice. Results enjoin public service providers to protect the identities of citizens by default or upon request.     

Food or Thought? Assessing Internal and External Factors Affecting Evaluations of Instructor Effectiveness

Within the broader literature concerned with potential bias in student measures of instructor effectiveness, two broad types of bias have been shown to operate in a course: internal and external. Missing is an assessment of the relative influence of each bias type in the classroom. Do internal or external types of bias matter more or less to student measures of instructor effectiveness? This question is of interest to those who recognize that many forms of bias operate in the classroom simultaneously and to great effect on course evaluations. Through the combination of quasi-experimental and survey methods, we attempt to do something not done before — simultaneously measure and compare the effects of one internal and one external type of bias to which we refer, following convention in the literature, as an internal factor and an external factor. Our findings indicate that, while both factors studied are important, perception of instructor partisanship (internal factor) is more influential on student course evaluations than our external factor (experimental treatment — literally a chocolate treat). The upshot is that American government instructors should be especially careful stating partisan positions in class.     

Knowing your sources: Partisan media and voters’ perceptions of the economy

This paper explains how media systems influence the extent to which partisanship colors voters’ perceptions of the economy (i.e., the strength of the partisan screen). It builds upon research on individual-level biases in economic perceptions, seeking to extend existing work by considering how the availability of partisan media for a given party affect such biases. The implication of this is that the greater the availability of media sources favorable to a party, the stronger the partisan screen for its partisans. This follows from several mechanisms including selective acceptance of messages, selective exposure to partisan sources, and incidental exposure to partisan sources. Each of these suggests that differences in the availability of partisan media across parties leads to corresponding differences in the extent of partisan bias for partisans of these parties. I test this Hypothesis in 14 European countries over four time-periods using data built from expert surveys on media characteristics.     

The grapes of rent: A history of renting in a country of owners

Abstract

In a recent study of neighborhood development, Goetz and Sidney (1994) found an “ideology of property” separating the interests of homeowners from the interests of lower‐income tenants. According to this ideology, owners are better citizens than renters, and therefore public policy should benefit owners at the expense of renters. In spite of continuing research that shows this allegation to be false, a widespread bias against renters persists. Why is this so?

A deliberate bias favoring property owners and harming renters has been prominent in American public policy from colonial times to the present, although its exact form has varied over time—property requirements for suffrage, land redistribution schemes promising ownership but delivering tenancy and poverty, and tax policies that privilege ownership and punish tenancy. Public policy that stigmatizes renters represents a bias as pernicious as other biases of gender, race, religion, and nationality.