Statistical inference and meta-analysis

Statistical inference is a common feature of meta-analysis. Statistical inference depends on a formal model that accurately characterizes certain key features of how the studies to be summarized were generated. The implications of this requirement are discussed and questions are raised about the credibility of confidence intervals and hypothesis tests routinely reported.

Unjustified inferences about meta-analysis

Berk (Statistical Inference and Meta-Analysis, 2007) asserts that the results of inferential statistics make scientific sense only if the data to which they are applied were actually generated through random sampling from a defined real population. Because meta-analysis data are not generated in that manner, he claims that the statistical conclusions of meta-analysis are fictional and suggests that conventional research review procedures be used instead. This rejoinder argues that Berk’s position on statistical inference represents a narrow literalism that he fails to justify and that does not reflect the way inferential statistics are used or generally understood in contemporary practice. Consequently, his critique has little significance for meta-analysis or any of the other widespread forms of social science research that apply inferential statistics in similar spirit. Berk’s advocacy of conventional literature reviews omits any explanation of how they would avoid the well-documented deficiencies of that approach or be conducted in a manner that offers any advantage over meta-analysis.

Participation in and frequency of delinquent behavior: A test for structural differences

In this paper we address whether there are distinctive differences in the processes determining participation in offending vs frequency of offending. We develop a number of tests to examine not only whether the correlates of participation and frequency are similar but also whether the same underlying statistical model is consistent with the data on both these dimensions of a criminal career. The tests are applied using data from the first two waves of the National Youth Survey. While the findings are not without ambiguities, they do not support the idea that distinguishing among the dimensions of a criminal career is a fundamental requirement for conducting sound research on the causes of crime and delinquency.     

Working with Misspecified Regression Models

Objectives

Conventional statistical modeling in criminology assumes proper model specification. Very strong and unrebutted criticisms have existed for decades. Some respond that although the criticisms are correct, there is for observational data no alternative. In this paper, we provide an alternative.

Methods

We draw on work in econometrics and statistics from several decades ago, updated with the most recent thinking to provide a way to properly work with misspecified models.

Results

We show how asymptotically, unbiased regression estimates can be obtained along with valid standard errors. Conventional statistical inference can follow.

Conclusions

If one is prepared to work with explicit approximations of a “true” model, defensible analyses can be obtained. The alternative is working with models about which all of the usual criticisms hold.

     

Using Booking Data to Model Drug User Arrest Rates: A Preliminary to Estimating the Prevalence of Chronic Drug Use

Public policy is often concerned with the size and characteristics of special populations that are difficult to reach in household surveys. Chronic drug users, who often live outside conventional households, provide the illustration motivating this paper. An alternative to household surveys is to question chronic drug users where they congregate—jails, treatment programs, and shelters, for example. Using such opportunistic data for prevalence estimation raises difficult problems for statistical inference: Study subjects who arrive at the collection points cannot be deemed a random sample of the general population. However, if we could estimate the rates at which chronic drug users arrive at the collection points, then we could use those estimates to weight the sample to represent the population. This paper presents a modified Poisson mixture model used to estimate the stochastic process that accounts for how chronic drug users get arrested. It uses that model to estimate arrest rates for 38 counties using up to sixteen quarters of data from the Arrestee Drug Abuse Monitoring survey.

Analyzing block randomized studies: the example of the Jersey City drug market analysis experiment

Objectives

While block randomized designs have become more common in place-based policing studies, there has been relatively little discussion of the assumptions employed and their implications for statistical analysis. Our paper seeks to illustrate these assumptions, and controversy regarding statistical approaches, in the context of one of the first block randomized studies in criminal justice—the Jersey City Drug Market Analysis Project (DMAP).

Methods

Using DMAP data, we show that there are multiple approaches that can be used in analyzing block randomized designs, and that those approaches will yield differing estimates of statistical significance. We develop outcomes using both models with and without interaction, and utilizing both Type I and Type III sums-of-squares approaches. We also examine the impacts of using randomization inference, an approach for estimating p values not based on approximations using normal distribution theory, to adjust for possible small N biases in estimating standard errors.

Results

The assumptions used for identifying the analytic approach produce a comparatively wide range of p values for the main DMAP program impacts on hot spots. Nonetheless, the overall conclusions drawn from our re-analysis remain consistent with the original analyses, albeit with more caution. Results were similar to the original analyses under different specifications supporting the identification of diffusion of benefits effects to nearby areas.

Conclusions

The major contribution of our article is to clarify statistical modeling in unbalanced block randomized studies. The introduction of blocking adds complexity to the models that are estimated, and care must be taken when including interaction effects in models, whether they are ANOVA models or regression models. Researchers need to recognize this complexity and provide transparent and alternative estimates of study outcomes.

     

Statistical techniques for juror and jury research

Juror and jury research is a thriving area of investigation in legal psychology. The basic ANOVA and regression, well‐known by psychologists, are inappropriate for analysing many types of data from this area of research. This paper describes statistical techniques suitable for some of the main questions asked by jury researchers. First, we discuss how to examine manipulations that may affect levels of reasonable doubt and how to measure reasonable doubt using the coefficients estimated from a logistic regression. Second, we compare models designed for analysing the data like those which often arise in research where jurors first make categorical judgments (e.g., negligent or not, guilty or not) and then dependent on their response may make another judgment (e.g., award, punishment). We concentrate on zero‐inflated and hurdle models. Third, we examine how to take into account that jurors are part of a jury using multilevel modelling. We illustrate each of the techniques using software that can be downloaded for free from the Internet (the package R) and provide a web page that gives further details for running these analyses.     

The internationalization of science and its influence on academic entrepreneurship

We examine whether scientists employed in foreign countries and foreign-educated native researchers are more “entrepreneurial” than their “domestic” counterparts. We conjecture that foreign-born and foreign-educated scientists possess broader scientific skills and social capital, which increases their likelihood that they will start their own companies. To test this hypothesis we analyze comprehensive data from researchers at the Max Planck Society in Germany. Our findings provide strong support for the conjecture that academic entrepreneurship can be stimulated by facilitating the mobility of scientists across countries.     

University-industry linkages in nanotechnology and biotechnology: evidence on collaborative patterns for new methods of inventing

The nanotechnology and biotechnology “revolutions” are so-called because their enabling technological breakthroughs were not simply inventions, but discoveries of entirely new methods of inventing. We hypothesize that university participants in either or both of these areas will exhibit greater collaboration with industry than researchers in other areas. We explore this hypothesis for 454 faculty members who conducted research that was patented during the period 1994–1999. Because our data include patents, publications, and funding at the individual level, we are able to examine the industry interaction of faculty who participated in the nanotechnology and biotechnology revolutions, as well as the interaction of faculty contributing to other areas of patentable science. We examine a variety of linkages, including sponsored research, consulting, publication with firm employees and measures of the potential for cross-campus collaboration. The results are striking in that they show significant differences in collaborative behavior across patent types and across the major program areas biological sciences, physical sciences and engineering. The results are consistent with a greater degree of tacit knowledge within the new methods of inventing. We also find significant differences in the embryonic nature and importance of patents across areas.     

Officer work orientations, perceptions of readiness and anticipated effectiveness of an agency-wide community policing effort within a county Sheriff’s office

Across occupations of every form, including those in criminal justice, administrators, managers, and supervisors frequently introduce changes to the structure and/or processes of the work environment. These changes may be seen as necessary to enhance worker productivity, but may be viewed by some workers as an unwanted disruption to their routines. Should employees interpret changes in such a manner, they can, and often do, negate these reforms. Employee work orientations (i.e., philosophical approaches toward work) undoubtedly play a role in such an interchange because they influence both how employees behave on the job and how they perceive their job. The move toward community-oriented policing constitutes a major paradigm shift to the practice of law enforcement. This structural and procedural change to the routines of policing could be perceived as threatening to those law enforcement officers whose work orientations are inconsistent with the philosophical foundation of community-based policing. The presence of a sufficient number of such officers could doom this movement to failure. This study uses survey data collected from a sample of sheriff’s deputies involved in an agency-wide community policing effort. In this study we examine the effects of deputies’ work orientations on their perceptions of the agency’s readiness for and the anticipated effectiveness of this initiative. The findings suggest interesting differences in the efforts of a traditional crime control orientation versus a more progressive community service orientation toward policing.     

Digital humanities and the history of working women: a cascade

In this introduction to the special issue on ‘Women's work in changing labour markets’, we argue that a combination of digital advances, notably the digitization of individual- and contextual-level data, the creation of internationally comparable occupation-based classifications, and the development of statistical models allowing for contextually informed analysis, has brought us to the brink of new developments in the field of women's work. Census and vital registration data contain more information on occupations of women than previously thought, and when used in combination with other digitized sources they allow one to assess the possible under-registration of women's work, as illustrated by some of the contributions to this special issue. Other contributions show how standardizing occupation-based classifications allows for temporal and regional comparisons of women's work and makes it feasible to study how community or regional characteristics influence that work. None of these developments – large-scale digitization of individual-level data, standardization of occupational titles and measures of stratification, and contextually informed analyses – is completely new; in some cases they are actually rooted in a venerable research tradition. However, in combination they might well constitute a cascade in the history of working women.     

Informal transfers,men, women and children: Family economy and informal social security in early 20th century Finnish households

Due to their role in discussions on community solidarity and social security in rich and poor countries alike, informal transfers between households have gradually become established as a research topic in economic and social history. Qualitative research has tended to emphasize the female-dominated nature of informal assistance. While research on intra-household resource allocation has demonstrated the potential for discerning gendered outcomes with household level data, quantitative research on informal assistance tends to ascribe a singular, “family” logic to transfers. Using an early 20th century Finnish household budget survey, this article analyses the differences in the statistical determinants of the reception of informal transfers in cash and in kind in the context of gendered household economy. Record linkage and statistical inference are utilized to reveal the sources of the different types of transfers and show how they were related to the position and welfare of men, women and children within households. The transfers in cash were actually controlled by men in the male breadwinner families of the data, exhibited elements of informal insurance, and were linked to trade union membership. The transfers in kind appeared to be the realm of particularly those women who were mainly engaged in unpaid work in households. At the same time, however, they were linked to outbound reciprocity in cash, indicating implicit dependence on male earnings.     

事实认定的概率分析

根据概率分析所涉及证据数量的不同,事实认定概率分析可分为单一证据维度和证据组合维度。在单一证据维度中,司法鉴定领域会不断涌现像DNA证据那样兼具实证统计数据和高度科学确认度的统计概率证据。在证据组合维度中,事实认定不可能通过数学推理实现,其原因主要有：事实认定过程复杂,数学推理难以模拟;数学推理并非司法证明思维的＂母语＂,而是一门需要翻译的＂外语＂;数学推理会通过＂量＂上的运算模糊、混淆乃至掩盖了＂质＂上的差别。     

Researchers’ Industry Experience and Productivity in University–Industry Research Centers: A “Scientific and Technical Human Capital” Explanation

We examine the impact of researchers’ previous industry experience on the research outputs and outcomes of university faculty affiliated with NSF and DOE research centers. Using a dataset combining curriculum vita and surveys, our results indicate significant differences between the researchers who have previous industry experience and those who do not. Using a simple model of research productivity, we found that academic researchers who had prior industry exposure produce fewer total career publications, but they support more students. Most important, and perhaps surprising, we could not establish any difference between the two groups’ publication activity when focusing on a five-year cross-section (years 1996–2000) rather than total career publications. We found statistical evidence that previous industry experience raised the annual publication productivity of junior faculty members and women researchers in our sample of research center personnel. We believe the unique blend of research center affiliation, academic post, and past industry experience gives an individual who embodies or possesses all three characteristics a diverse source of scientific and technical human capital and particular advantages over those who have no industry experience (though the “academic-only” set also has particular advantages in cumulative publishing productivity).     

American Terrorism and Extremist Crime Data Sources and Selectivity Bias: An Investigation Focusing on Homicide Events Committed by Far-Right Extremists

This paper examines the reliability of the methods used to capture homicide events committed by far-right extremists in a number of open source terrorism data sources. Although the number of research studies that use open source data to examine terrorism has grown dramatically in the last 10 years, there has yet to be a study that examines issues related to selectivity bias. After reviewing limitations of existing terrorism studies and the major sources of data on terrorism and violent extremist criminal activity, we compare the estimates of these homicide events from 10 sources used to create the United States Extremist Crime Database (ECDB). We document incidents that sources either incorrectly exclude or include based upon their inclusion criteria. We use a “catchment-re-catchment” analysis and find that the inclusion of additional sources result in decreasing numbers of target events not identified in previous sources and a steadily increasing number of events that were identified in any of the previous data sources. This finding indicates that collectively the sources are approaching capturing the universe of eligible events. Next, we assess the effects of procedural differences on these estimates. We find considerable variation in the number of events captured by sources. Sources include some events that are contrary to their inclusion criteria and exclude others that meet their criteria. Importantly, though, the attributes of victim, suspect, and incident characteristics are generally similar across data source. This finding supports the notion that scholars using open-source data are using data that is representative of the larger universe they are interested in. The implications for terrorism and open source research are discussed.     

Power Dynamics in an Experimental Game

We introduce a new experimental method for studying power. Drawing from multiple theoretical perspectives, we conceptualize power as relational and structural, as well as comprised of different forms through which basic human needs can be met. Thus, the method we introduce examines how, when faced with a particular need, people use multiple forms of power concurrently and within a “field of influence,” namely, the other players in a game. This enabled us to examine how one form of power is transformed into another and how power is transferred from one player to another through interaction, as well as to measure power as behavior, as the exercise of choice, as potential, and as outcomes. Two experiments using egalitarian start conditions and a survivable ecology demonstrated that participants used power to gain more power, creating inequality. Being the target of force made some players unable to “survive” in the local ecology. Theoretical and methodological issues in the study of power are discussed and the application of our game method to the study of power in other fields is considered.     

Using multiple choice questions to examine the content of the qualifying law degree accurately and reliably: the experience of the Qualified Lawyers Transfer Scheme

The Qualified Lawyers Transfer Scheme (QLTS) provides a route for lawyers from other jurisdictions and barristers from England and Wales to qualify as solicitors in England and Wales. One of the three tests in the QLTS, the Multiple Choice Test (MCT), uses multiple choice questions to examine the syllabus of the qualifying law degree together with some pervasive subjects from the Legal Practice Course. This paper examines the MCT in detail. Particular attention is paid to the format and structure of the questions. We describe the detailed editing which each question goes through before it is used, the statistical analysis and review which take place after an exam, and the rationale for these processes. An explanation is given of what reliability and accuracy mean and how they are measured statistically. There is also an explanation of the “Angoff method” by which pass marks are set. Finally, the paper reports on the first four sittings of the MCT and their statistical results including their reliability and accuracy. Use of the multiple choice test, though well established for examining applied knowledge in medicine and in law in other jurisdictions, is a radical departure for the assessment of law in England and Wales. The experience of QLTS has shown that a carefully constructed multiple choice test of a suitable length can assess the qualifying law degree content both reliably and accurately.     

Reconciling questions about dichotomizing variables in criminal justice research

Purpose

Despite accumulating evidence against the practice of artificial dichotomization, its continued use among criminal justice researchers indicates that there are still unresolved questions about its appropriateness. Farrington and Loeber (2000) provided a discussion of how these issues impact research on delinquency, and many researchers have cited their article as a justification for dichotomization within the field of criminal justice. In the current study, we examine the reasons why researchers have cited Farrington and Loeber as a mechanism for answering some unresolved questions about whether and when dichotomization may be justified.

Methods

We used a forward citation search in PsycInfo to locate all articles citing Farrington and Loeber (2000) in support of dichotomization.

Results

This search identified 126 articles which provided a total of 191 reasons supporting dichotomization. We explore these reasons, discussing whether they are consistent with evidence from simulation-based analyses and whether they are supported by existing statistical and methodological theory.

Conclusions

Despite the large number of reasons for dichotomization provided by authors, we found very few that had empirical or theoretical support.     

A simulation model of pretrial felony case processing: A queuing system analysis

Case processing tends to be examined with data analysis or evaluation designs. Both limit our understanding of how case processing as a whole operates and how its parts relate to each other. This article suggests queue simulation modeling as a method for dealing with these issues. We report here the initial development and analysis of a queuing model of arraignment to trial assignment. Conceptualizing on the basis of court functions and empirical findings, rather than institutional structures, we conceive a five-stage pretrial process. Using case-level, rather than system-level data, we construct a single-server, multiphase queuing model and use the model to simulate the behavior of a pretrial case processing system. Simulations show the strong impact of the final phase (trial assignment) on the entire system and that most of this impact is delay rather than service. The system is then analyzed using a factorial design that systematically alters model parameters thought to be important determinants of performance. Simulations are run for each possibility in the design, and analysis of variance is used to examine results. Analysis confirms prior results concerning final phase impact and points specifically to the import of phase capacity and exit rate. The utility of modeling is considered by suggesting some policy implications of the results for judicial staffing and behavior.     

Commentators on the Cārvākasūtra: A Critical Survey

In spite of the fact that the mūla-text of the Cārvākasūtra is lost, we have some 30 fragments of the commentaries written by no fewer than four commentators, namely, Kambalāśvatara, Purandara, Aviddhakarṇa, and Udbhaṭa. The existence of other commentators too has been suggested, of whom only one name is mentioned: Bhāvivikta. Unfortunately no extract from his work is quoted anywhere. The position of the Cārvākas was nearer the Buddhists (who admitted both perception and inference) than any other philosophical system. But in order to brand the Cārvākas as pramāṇaikavādins they were made to appear as one with Bhartṛhari. Even though the commentators of the Cārvākasūtra had some differences among themselves concerning the interpretation of some aphorisms, they seem to have been unanimous in regard to the number of pramāṇas to be admitted. It was perception and inference based on perception. Only in this sense they were pramāṇaikavādins. Unlike other systems of philosophy, the Cārvāka/Lokāyata did not accord equal value to perception and inference. Inference, they said, must be grounded on perception first, so it was of secondary kind (gauṇa). From the available evidence it is clear that the commentators were unanimous in one point, namely, primacy of perception which includes admittance of such laukika inference as is preceded and hence can be tested by repeated observations. In this respect both Aviddkarṇa and Udbhaṭa were in agreement with Purandara. Bhaṭṭodbhaṭa or Udbhaṭabhaṭṭa was known as a commentator who differed from the traditional Cārvākas and broke new grounds in explaining some of the aphorisms. His commentary is creative in its own way but at the same time unreliable in reconstructing the original Cārvāka position. Udbhaṭa seems to have digressed from the original, monist materialist position by taking a dualist position concerning the body-consciousness relation. Moreover, he seems to verge on the idealist side in his explication of an aphorism. In this sense he was a reformist or revisionist. Aviddhakarṇa, like Udbhaṭa, attempted to interpret the Cārvāka aphorisms from the Nyāya-Vaiśeṣika point of view, perhaps without being converted to the Cārvāka. Since it is not possible at the present state of our knowledge to determine whether they were Cārvākas converted to Nyāya or Naiyāyikas converted to Lokāyata, the suggestion that they simply adopted the Cārvāka position while writing their commentaries without being converted to the Cārvāka, may be taken as a third alternative. In spite of the meagre material available, it is evident that (1) not unlike the other systems, there is a lack of uniformity in the commentary tradition of the Cārvākasūtra, (2) not all commentators were committed monistic materialists; at least one, namely, Udbhaṭa, was a dualist, and (3) in course of time Nyāya-Vaiśeṣika terminology, such as gamya, gamaka, etc., quite foreign to the traditional Cārvāka, has been introduced into the Cārvāka system.