Measuring the performance of research and technology programs: A balanced scorecard approach

Research programs, like other government programs, are now being requested to demonstrate relevance and value added to national social and economic needs. Complexity, unpredictability and other factors make it difficult to define specific performance measures for R&D programs. This paper describes the performance measurement efforts of one technology development program within the U.S. Department of Energy and proposes a strategy for applying this balanced scorecard approach to a fundamental research organization. Simple logical models of the inputs, activities, outcomes and long term results of R&D programs are proposed. A critical few measures of performance that answer questions from multiple audiences are then chosen across this performance spectrum. This paper describes work performed by Sandia National Laboratories Energy Policy and Planning Department, Albuquerque, New Mexico 87185, with the support of the U.S. Department of Energy under contract DE-AC0494AL85000. The continued support and enthusiams of Darrell Beschen in the DOE Office of Energy Efficiency and Renewable Energy and Iran Thomas in DOE Office of Energy Research, Basic Energy Sciences, is appreciated. The authors also acknowledge the contributions of John Reed and the training in the logic chart and performance spectrum provided by Ron Corbeil and Steve Montague. The opinions expressed are those of the authors and do not represent the opinions of the U.S. Department of Energy.     

Stalking acknowledgement and reporting among college women experiencing intrusive behaviors: Implications for the emergence of a “classic stalking case”

Research confirms the risk of victimization faced by women who attend the nation's universities. While stalking began to receive heightened attention in the 1990s, relative to other types of victimization, research regarding college women's stalking experiences is in its infancy. In particular, relatively little is known about victim acknowledgement of stalking and the process whereby victims decide to report to police. Drawing upon data from 1,010 female university students, this study provided important advances in understanding stalking-related victimization for college women, highlighting the influences that fear, offender relationship, types of behaviors involved, and other victimization experiences have on victim acknowledgement and police reporting. Findings offered important advancements in defining a “classic stalking case.” This study had implications for university and public policymakers, educators, and police officers who could benefit from improved understanding of women's perceptions of stalking and their formal help-seeking behavior in response to stalking-related behavior.     

The Private Interests of Public Officials: Financial Regulation in the US Congress

Legislators' private financial holdings affect policy decisions. Due to financial self‐interest, we theorize that legislators whose personal investment portfolios include equities from firms affected by proposed policies vote for legislation that benefits those firms. We also theorize that legislators with greater personal exposure to equity investments support policies that benefit equities markets generally. We create a novel data set of legislators' personal stock investments and examine major congressional actions since the 1990s on financial deregulation and market intervention. US House members who own stocks in firms who benefit from financial deregulation vote for deregulation. House members with greater exposure to financial and automotive stocks support the financial and auto bailouts, respectively. General exposure to equities markets is also associated with support for key legislation boosting markets. The normative implications are significant, as legislators' private interests influence decisions in the public sphere.     

How to Cautiously Uncover the “Black Box” of Machine Learning Models for Legislative Scholars

Machine learning models, especially ensemble and tree-based approaches, offer great promise to legislative scholars. However, they are heavily underutilized outside of narrow applications to text and networks. We believe this is because they are difficult to interpret: while the models are extremely flexible, they have been criticized as “black box” techniques due to their difficulty in visualizing the effect of predictors on the outcome of interest. In order to make these models more useful for legislative scholars, we introduce a framework integrating machine learning models with traditional parametric approaches. We then review three interpretative plotting strategies that scholars can use to bring a substantive interpretation to their machine learning models. For each, we explain the plotting strategy, when to use it, and how to interpret it. We then put these plots in action by revisiting two recent articles from Legislative Studies Quarterly.     

When Heads Roll: Assessing the Effectiveness of Leadership Decapitation

Leadership targeting has become a key feature of current counterterrorism policies. Both academics and policy makers have argued that the removal of leaders is an effective strategy in combating terrorism. However, leadership decapitation is not always successful, and existing empirical work is insufficient to account for this variability. As a result, this project answers three primary questions: (1) Under what conditions does leadership decapitation result in the dissolution of a terrorist organization?; (2) Does leadership decapitation increase the likelihood of organizational collapse beyond the baseline rate of collapse for groups over time?; and (3) In cases where decapitation does not result in group collapse, to what extent does it result in organizational degradation and hinder a group's ability to carry about terrorist attacks? I develop a dataset of 298 incidents of leadership targeting from 1945–2004 in order to determine whether and when decapitation is effective. First, I identify the conditions under which decapitation has been successful in bringing about organizational decline. The data show that a group's age, size, and type are critical in identifying when decapitation will cause the cessation of terrorist activity. As an organization grows in size and age, it is much more likely to withstand the removal of its leadership. Organizational type is also significant in understanding the susceptibility of an organization to decapitation. Ideological organizations are most likely to experience a cessation of activity following the removal of leader, while religious organizations are highly resistant to leadership decapitation. Second, I determine whether decapitation is an effective counterterrorism strategy that results in organizational collapse. The data show that decapitation does not increase the likelihood of organizational collapse beyond a baseline rate of collapse for groups over time. Organizations that have not had their leaders removed are more likely to fall apart than those that have undergone a loss of leadership. The marginal utility of decapitation is negative for many groups, particularly for larger, older, religious, and separatist organizations. Finally, I look at the extent to which decapitation results in organizational degradation and hinders a group's ability to carry about terrorist attacks. Case studies illustrate whether decapitation has an effect on the operational capacity of an organization by identifying whether the removal of key leaders changes the number and lethality of attacks. If certain organizations are more resilient than others, it is important to know when decapitation should be effective and when it could lead to counterproductive outcomes. Overall, these findings illustrate the need to develop a new model for evaluating the efficacy of leadership decapitation and for developing effective counterterrorism policies.