Public programs and regulations often focus on a narrow slice of the public—people who meet specific eligibility rules or who are directly affected by a new restriction. That targeting can make policies more efficient, but it also creates a persistent problem for researchers: demonstrating what the policy actually did. Even when economists and other social scientists apply widely used quasi-experimental approaches—methods designed to estimate cause-and-effect without a randomized trial—the evidence can be thin if the affected group is small and hard to observe in commonly available datasets.
In a new study, economists Kelly M. Jones and Mayra Pineda-Torres examine this measurement challenge head-on. Their focus is not on proposing a new identification strategy, but on a practical issue that can quietly undermine many policy evaluations: statistical power, or the likelihood that a study can reliably detect an effect of a given size. When a policy touches only a minority population, researchers often rely on large, population-level data sources that were not built to closely track that subgroup. As a result, even a real and meaningful impact can be difficult to distinguish from noise.
To make the problem concrete, the authors use the example of abortion bans and their potential effects on economic well-being. The question of whether abortion restrictions change people’s economic circumstances is a causal one, and it is typically investigated with quasi-experimental designs that compare outcomes across places and time as laws change. But because the directly affected population is a relatively small fraction of the overall population captured in broad surveys or administrative records, estimates can lack the precision needed to detect plausible policy effects.
Jones and Pineda-Torres use power calculations to quantify just how difficult this can be. Power calculations are commonly used before clinical trials and experiments, but they can also be applied to observational policy studies. The idea is to compute, in advance, whether the available data—given the size of the impacted group, expected variability in outcomes, and the study design—are sufficient to identify effects that policymakers and the public would consider substantively important.
The researchers’ contribution is twofold: they lay out why targeted policies can be especially hard to evaluate with “big” population datasets, and they offer a roadmap for doing ex-ante power checks. That roadmap is meant to help analysts answer a blunt but essential planning question: with the data at hand, can this study realistically detect reasonably meaningful effect sizes for the group the policy is intended to affect? If not, the authors suggest, researchers can adjust expectations, refine designs, or pursue alternative data strategies rather than discovering limitations only after results come back imprecise or inconclusive.
The work was supported by the Society of Family Planning through grant SFP19-HEO7. The authors note that the conclusions are their own and do not necessarily reflect the views of the Society of Family Planning, the National Bureau of Economic Research, or other affiliated institutions. They also acknowledge research assistance from Simran Patel and Hayden Baillie and report no competing interests.
Why it matters
Many high-stakes policies—from health rules to social benefits—target small or vulnerable groups, yet evaluations often rely on broad datasets that may be ill-suited to detect impacts on those populations. By emphasizing power calculations before analysis begins, the study pushes researchers and funders to confront whether a proposed evaluation can credibly answer the questions it raises. What happens next will depend on whether agencies and scholars invest in better subgroup measurement, larger samples, or new data sources that match the policy’s true footprint.