Towards an evidence-based approach to communicating uncertainty in intelligence analysis

Intelligence products have a degree of uncertainty associated with them. This is typically expressed using linguistic probabilities (e.g., ‘likely’), and some organizations have adopted standardized lexicons for communicating uncertainty. This paper empirically shows that intelligence analysts use a wide heterogeneity of language to communicate uncertainty. This does not include all of the phrases in standardized lexicons used by the intelligence community. In addition, analysts may use some phrases differently to that advocated. Miscommunication of uncertainty can have deleterious effects on decision-making, and so standardization of uncertainty communication should be evidence-based. This paper discusses ways in which such evidence can be generated.     

Why do analysts use structured analytic techniques? An in-depth study of an American intelligence agency

This article presents findings from the first publicly available survey generalizable to an intelligence agency to explore why analysts use structured analytic techniques (SATs). Mandated by the Intelligence Reform and Terrorism Prevention Act (2004), SATs are simple methodologies supposed to make analysis more transparent and, hopefully, valid. Despite the US government’s investment in training thousands of analysts, there is no solid evidence on how often or why analysts actually use SATs. A survey of 80 analysts and nine follow-up interviews at the State Department’s Bureau of Intelligence and Research reveals a simple, but important, truth: implementing the techniques requires training and compelling evidence they will improve analysis. Other factors, most notably the amount of time pressure an analyst experiences, were not significantly related with the use of the techniques despite anecdotal accounts and conjecture from the literature. Future research should examine other intelligence agencies to cross-validate these findings. If these findings hold in other cases, intelligence agencies should focus on reforming and incorporating evidence into the training process.     

Favorite INTs: how they develop,why they matter

Collection of important information is a critical part of the intelligence business. Less recognized and studied is the differential use of types of intelligence information based on personal and organizational preferences for, and biases against, specific intelligence collection disciplines, or ‘INTs’. This article presents a framework for assessing the implications of ‘favorite INTs’ for policy-making, policy implementation, and intelligence analysis. The record shows that favorite INTs negatively influence analysts and the use of intelligence by senior political leaders and military commanders. Practitioners can improve intelligence support and scholars can better understand how intelligence influences decision-making by appreciating how and why favorite INTs develop and influence decision-makers and analysts.     

Managers of Analysts: The Other Half of Intelligence Analysis

The literature on intelligence analysis has a glaring hole: assessment of the management of analysts and their work. This gap is striking because managers influence all aspects of analysis and share responsibility with analysts for analytic successes and failures. While many managers ably recruit, train, mentor, and manage their analysts, they also are responsible for pathologies that negatively affect analysis and are often incorrectly attributed to analysts. Accurate understanding of the roles and influence of managers should change scholars’ views of the processes of analysis and better balance attention on analysts and managers.     

Intelligence Analysis

American intelligence analysts have been severely criticized for failing to anticipate the 9/11 Al Qaeda attacks and for the ill-advised invasion of Iraq. The resulting Intelligence Community reorganization, intended to repair these presumed deficiencies, reflects a misunderstanding of what intelligence analysts can do and where responsibility for political and military decisions lie. In fact, the Intelligence Community is far more diversified in its tasks than is generally realized. Where analysis is done and for whom makes a large difference in its effectiveness. Moreover, changing technology is altering how it is done, where, and by whom. The border between what is ‘strategic’ or ‘national’ intelligence and ‘tactical’ intelligence is much diminished. At the same time, intelligence analysts all too often fail to incorporate the growing amounts of open source information, as well as analytic concepts and theories available from academic and scholarly literature.     

Improving information evaluation for intelligence production

National security decision-making is informed by intelligence assessments, which in turn depend on sound information evaluation. We critically examine information evaluation methods, arguing that they mask rather than effectively guide subjectivity in intelligence assessment. Drawing on the guidance metaphor, we propose that rigid ‘all-purpose’ information evaluation methods be replaced by flexible ‘context-sensitive’ guidelines aimed at improving the soundness, precision, accuracy and clarity of irreducibly subjective judgments. Specific guidelines, supported by empirical evidence, include use of numeric probability estimates to quantify the judged likelihood of information accuracy, promoting collector-analyst collaboration and periodic revaluation of information as new information is acquired.     

Intelligence and knowledge development: what are the questions intelligence analysts ask?

Question-asking is central to intelligence. Despite recognition of the importance of questions, in the absence of empirical research, much of what we understand about question-asking within the intelligence process remains speculative or limited to personal experience and memory. To develop an evidence-based approach to improving intelligence, this paper argues that empirical research into the questions that intelligence analysts ask is required. Drawing on insights from educational research, an initial set of research questions is proposed to understand the questions that intelligence analysts actually ask.     

Reflections on CIA Analysis: Is it Finished?

Despite the CIA's improved analytical tradecraft and increased resources, the future of its analytic mission remains in doubt. Post-9/11 improvements have been coupled with a continued focus on current intelligence priorities that minimize attention to the development of strategic research and deeper knowledge. Simply increasing the number of analysts has not produced deeper expertise. The CIA's traditional recruitment and training methods, as well as its rewards and promotion system, encourage analysts to avoid concentrating on any single area of regional or functional expertise in favor of moving around the agency to build a successful career. A continued reliance on risk-avoidance security practices also restricts analysts' contact with non-government and foreign experts who often have needed political and cultural knowledge of intelligence topics. To rectify these inadequacies, the CIA's analytic directorate needs to develop incentives for analysts who wish to develop more strategic analysis and remove the security barriers to closer collaboration with experts outside the US government. Developing cross-agency analytic collaboration would also maximize expertise and would benefit from intelligence community-wide training programs similar to what the US military does at its senior service colleges.     

The future of the intelligence analysis task

Abstract

Information technology will have a significant impact on the intelligence analysis workflow, skills, and organization in the next couple of decades. In future, instead of ingesting information themselves, analysts will use a range of information tools to add value to data. Future analysts will need less knowledge of subject matter, and more general reasoning skills. The future task will involve more creativity, and less focus on detail, than today.     

Big Data,intelligence, and analyst privacy: investigating information dissemination at an NSA-funded research lab

The Laboratory for Analytic Sciences (LAS) at North Carolina State University, funded by the National Security Agency, is a collaborative, long-term research enterprise focused on improving intelligence analysis using Big Data. In its work, LAS has recently begun dealing with the trade-off between the collection, storage, and use of large unclassified data-sets and analyst privacy. We discuss particular privacy challenges at LAS, analyze privacy principles in the life cycle of LAS unclassified data-sets, what intelligence analysts themselves think about these privacy concerns, and recommend possible best practices potentially applicable to LAS, as well as future Big Data laboratories and research centers that collaborate with intelligence communities.     

Intelligence Studies Centers: Making Scholarship on Intelligence Analysis Useful

Abstract

Improving intelligence analysis requires bridging the gap between scholarship and practice. Intelligence studies as an academic discipline is not very theoretical compared to the more established disciplines of political science and international relations. In terms of conceptual depth, levels of abstraction and theoretical development, even the theoretical portions of the academic intelligence studies literature could be described as policy relevant and potentially useful for practitioners, including intelligence analysts. Yet despite this orientation to the practitioner, there is still a substantial gap between scholars and practitioners, thus replicating within a more applied context the conventional theory/practice divide that exists in other fields. Those fields do, however, possess a variety of ideas and recommendations that could be used to bring scholarship on intelligence analysis closer to practice. If implemented, these ideas might help actualize the benefits of scholarship that are as yet still unrealized potential.     

Intelligence in war: how important is it? How do we know?

The role of intelligence in determining outcomes of battles and wars has varied dramatically across conflicts and time, contributing to wide differences in judgments about the importance of intelligence in war. Some analysts minimize the importance of intelligence while others argue it is essential. Opposing perspectives often do not engage each other’s points and generalize from small samples of conflicts. The result is a disjointed discussion of the roles and importance of intelligence in armed conflicts. This article’s aim is to partially rectify this situation by deriving an analytic framework that links intelligence to military operations and strategic outcomes.     

Budgetary Decision Making by Executive and Legislative Budget Analysts: The Impact of Political Cues and Analytical Information

We explore the extent to which attributes of individual analysts and institutional factors lead to budgetary decisions based on political cues and/or analytical information among executive and legislative budget analysts. We surveyed executive and legislative budget analysts in 13 western states to ascertain the factors that influence their budgetary decisions. The findings build on evidence from previous studies that confirms the complexity of factors affecting budget analysts' decisions, and they provide empirical support to analysts' use of a combination of information labeled "budget rationality" by Thurmaier and Willoughby. Also, we provide more conclusive evidence that both executive and legislative analysts follow similar decision-making patterns.     

The 1962 Cuban intelligence estimate: A methodological perspective

The performance of American intelligence in the 1962 Cuban crisis was measured inadequately. An account of its failures suggests that the roots of poor performance, and of SNIE 85–3–62 in particular, were in a flawed methodological approach. Problems in four aspects of the intelligence methodology are identified: in the use of comparative logic; in the use of evidence for theory validation; in the formulation of research questions; and in the development and review of assumptions. Finally, the methodological discussion yields lessons concerning the analysis of intelligence performance and makes suggestions for improvement of the intelligence assessment process.     

Intelligence Analysis Theory: Explaining and Predicting Analytic Responsibilities

Theoretical work involves explanation and prediction, but thus far there has been little scholarly work explaining and predicting the role of intelligence analysts in support of foreign and national security policies. Without a theory of intelligence, it becomes difficult to decide what the appropriate substantive analytical responsibilities of the intelligence community should be. Accordingly, a theory of foreign intelligence analysis is necessary. This paper presents a theoretical framework developed during the immediate post-Cold War timeframe to explain why there was such a wide variety of perspectives regarding the future need for intelligence, embeds these ideas within the existing intelligence theory literature, applies this framework more generally in a way that can be used to explain variations in the substantive coverage of intelligence analysis in the past and predict possible variations in the future, and then tests the theory's ability to explain the analytical focus of domestic intelligence organizations.     

Why strategic intelligence analysis has limited influence on American foreign policy

Why does strategic intelligence analysis have limited influence on American foreign policy? Intelligence analysis is frequently disregarded, this paper contends, because it is a duplicated step in the decision-making process and supplements but does not supplant policy assessment. Many intelligence analyses will confirm policy assessments and be redundant or – if the assessments are different – policy-makers will choose their own interpretations over those of intelligence analysts. The findings of this paper provide scholars with important insights into the limits of intelligence analysis in the foreign policy process as well as recommendations for increasing its positive impact on policy.     

Experimental Intelligence

This article explores the use and application of experimental methodologies in intelligence analysis. Previous experimental work has much to tell us about how people process information in making judgments. Calibration, and how appropriate levels of confidence might be improved with systematic feedback, is addressed. Training around uncertainty biases is covered. Extant experimental work suggests mechanisms and tests which might be used to better screen potential analysts for personality characteristics that might make them more suited for some tasks over others. The paradigm of experimental manipulation serves as a useful template for exploring alternative conceptualizations of uncertain environments.     

Britain and the Missile Gap: British Estimates on the Soviet Ballistic Missile Threat, 1957–61

Between 1957 and 1961, American National Intelligence Estimates overestimated the Soviets’ capabilities to produce and deploy intercontinental ballistic missiles, creating the ‘missile gap’ controversy. This article examines the contemporaneous estimates of British intelligence on the Soviet ballistic missile program, which were based upon very similar, if not the same, raw intelligence. It demonstrates that British estimates of the Soviet ICBM program were more accurate. However, this success did not continue in the analysis of the medium- and intermediate-range ballistic missile (M/IRBM) threat, which was relatively poor for most of the period. It concludes that the reasons for this lie in the different assumptions held by intelligence analysts on both sides of the Atlantic, and a degree of conservatism in both intelligence establishments.     

Mind games: cognitive bias,US intelligence and the 1968 Soviet invasion of Czechoslovakia

This article examines the impact of cognitive bias on the analytic output of the United States intelligence community during the Prague Spring. Utilising a range of primary sources, including declassified documents, oral history and contemporary accounts, this article argues that as a result of heuristic biases, analysts formed the mindset that the Soviet Union would not invade Czechoslovakia, and did not alter that assumption in the face of increasing evidence to the contrary. Consequently analysts possessed a distorted understanding of both Soviet intentions and the prevailing political environment and did not accurately convey the likelihood of military action to consumers.     

Analytic Outreach for Intelligence: Insights from a Workshop on Emerging Biotechnology Threats

This article describes a new effort to engage in analytic outreach between academic scholars and intelligence analysts on the issue of emerging biotechnology threats to US national security. The context of this outreach was a September 2012 meeting in London to explore possibilities for enhanced analytic outreach in relation to emerging biotechnology threats, supported by the UK Genomics Policy and Research Forum. This meeting consisted of a mix of current and former intelligence practitioners and policy officials, and social science and scientific experts, from both the UK and the US. As will be described below, this unique pairing of experts and subjects revealed new insights into how to improve intelligence assessments on biotechnology and bioweapons threats. It also revealed continuing challenges in reforming assessments within existing intelligence work routines.