New Study Warns Survey Rating Scales Can Distort Well-Being and Economic Research Findings

Survey-based research in economics and the social sciences often relies on respondents to translate their experiences into numbers—say, choosing a point on a 0–10 life satisfaction scale. But those numbers can be misleading if different people interpret and use the same scale in systematically different ways. A “7” may signal very different levels of well-being depending on whether a respondent is generally generous or conservative with high ratings. A new paper by Daniel J. Benjamin, Kristen B. Cooper, Ori Heffetz, Miles S. Kimball, and Jiannan Zhou tackles this long-standing measurement problem by laying out a more general framework for correcting what researchers call “scale-use heterogeneity.”

The authors formalize and extend the theory behind a family of fixes that use “calibration questions,” including survey vignettes. The basic idea is to ask additional questions designed to reveal how each person uses the response scale. Those calibration responses can then be used to place different respondents’ answers onto a common yardstick—aiming to ensure that comparisons across people reflect true differences in the underlying trait, not quirks of scale interpretation.

Using a large dataset of roughly 10,000 respondents, alongside evidence from prior studies, the researchers report a striking empirical regularity: the relationship between one person’s use of the subjective scale and another’s is, to a good approximation, linear. That near-linearity matters because it allows a practical correction method rather than a complicated, highly parameterized model. Building on this finding, the paper develops a simple estimator that researchers can plug into regression analyses to adjust coefficients for differences in scale use.

The team then puts the approach through a reality check using subjective ratings of height and weight—quantities with objective, external benchmarks. If scale use varies across individuals, two people of the same height might choose different ratings on a “subjective height” scale. By applying their correction, the authors assess whether the adjusted subjective reports line up more closely with the underlying objective measures, providing a validation exercise for the method.

Finally, the paper illustrates the stakes with an application to life satisfaction regressions. Here, the authors find that correcting for heterogeneous scale use does not overturn the main qualitative conclusions researchers typically draw—for example, which factors are associated with higher or lower life satisfaction. However, the correction can materially change the relative magnitudes of estimated effects. In one example, the adjustment increases the implied well-being cost of unemployment relative to income by a factor of three, highlighting how uncorrected scale differences can distort comparisons that are central to policy debates.

The authors note that this study replaces an earlier working paper, “Adjusting for Scale-Use Heterogeneity in Self-Reported Well-Being” (NBER Working Paper 31728, dated September 25, 2023), while analyzing different data. The research draws on the Understanding America Study (UAS), run by the Center for Economic and Social Research at the University of Southern California, and used high-performance computing resources at the University of Colorado Boulder, including the Summit supercomputer and Alpine. The project received support from NIH/NIA grants (R01-AG065364 and R01-AG051903), obtained institutional review board approval, and the authors report no material financial interests related to the work.

Why it matters

Governments, employers, and researchers increasingly use subjective survey measures—especially well-being metrics—to evaluate policies and social conditions. If people use response scales differently, estimates comparing groups or weighing tradeoffs (such as the burden of unemployment versus income changes) can be skewed even when the underlying data appear robust. The paper’s evidence for an approximately linear mapping offers a simpler, more usable path to correction—and a reminder that “small” measurement issues can shift the policy-relevant numbers.