A new analysis of Chicago’s ShotSpotter argues that dismissing “statistically insignificant” results can mislead policy, and assesses whether benefits likely exceed costs.
When researchers evaluate a public program, a familiar rule of thumb often governs how results are read: if an estimated impact is not “statistically significant,” it is treated as inconclusive and therefore unhelpful for policy. A new analysis of Chicago’s use of ShotSpotter—an acoustic sensor network designed to detect gunfire and speed police response—argues that this convention can mislead decision-makers, especially when the outcomes are rare and the stakes are high.
The authors’ core point is straightforward. Policy choices typically hinge not on whether an effect can be cleanly distinguished from zero in a hypothesis test, but on whether the benefits are likely to outweigh the costs. In many real-world settings, studies are “underpowered,” meaning they do not include enough observations to reliably detect effects of the size needed to make a program worthwhile. In those cases, focusing narrowly on statistical significance can cause governments to discard evidence that still meaningfully informs a cost-benefit decision.
To make the case, the researchers examine ShotSpotter’s impact in Chicago, where the system is intended to reduce the time it takes for police to learn about shootings and reach the scene—an operational change that could plausibly improve victims’ chances of survival. Using a geographic regression discontinuity (RD) design, they compare areas just inside ShotSpotter coverage boundaries with areas just outside, aiming to isolate the effect of the technology from other neighborhood differences.
Their central estimate suggests ShotSpotter reduced the shooting fatality rate by 2.5 percentage points—an effect they translate into roughly 49 fewer deaths per year. That estimate, however, does not meet conventional thresholds for statistical significance. Under a significance-testing mindset, the temptation would be to treat the findings as offering little guidance.
But the authors argue that such a conclusion ignores what the estimate implies economically. Even with statistical uncertainty, an effect of that magnitude would generate benefits far larger than the cost of the service. The practical obstacle is that, because shooting deaths are relatively infrequent compared with the number of shootings and because year-to-year variation is large, pinning down the effect with high precision is extraordinarily difficult. The paper calculates that detecting an effect of the estimated size using a conventional hypothesis-testing approach would require about 87 years of Chicago shooting data—an obviously unrealistic standard for decision-making.
Instead of asking whether the effect is different from zero, the study adopts a Bayesian framework that treats the life-saving effect as uncertain and estimates a probability distribution over possible impacts. The decision-relevant output is then the likelihood that benefits exceed costs, rather than a binary “significant/not significant” label.
Across assumptions, the analysis finds positive expected net benefits. The posterior probability that ShotSpotter’s benefits outweigh its costs ranges from 70% under a flat prior combined with a 13.6 million value of a statistical life.
The project received support from Arnold Ventures (Grant 24-12760) and the University of Chicago Crime Lab’s general operating funds, and was approved by the University of Chicago Institutional Review Board (IRB25-1416). The authors report no financial or other conflicts of interest. They also note that the work was conducted while Terence Chau was at the University of Chicago Crime Lab and is unrelated to his position at Amazon.
Why it matters
Cities routinely make high-stakes spending decisions based on evidence that is noisy, limited, and slow to accumulate—especially in public safety. This study argues that treating “not statistically significant” as “no useful information” can lead policymakers to overlook interventions that are likely to save lives and pass a cost-benefit test. The next step is broader adoption of decision-focused reporting—probabilities of net benefit, not just p-values—when governments evaluate expensive, hard-to-measure programs.