Alternatives to Probabilistic Psychological Measurement

Philosophical framing

Psychological measurement, at its core, grapples with the fundamental question of whether psychological attributes possess a quantitative structure amenable to numerical representation. This isn't merely a technical issue; it delves into the very nature of psychological reality. The prevailing probabilistic psychometrics, with its reliance on latent variables and statistical inference, implicitly assumes such a quantitative structure, often without rigorous empirical justification. This lecture will explore traditions that challenge this assumption, seeking to understand psychological phenomena through lenses that do not presuppose commensurability with standard numerical scales, drawing from philosophical critiques of measurement itself.

Introduction

The landscape of psychological measurement has long been dominated by probabilistic approaches, primarily rooted in Classical Test Theory (CTT) and Item Response Theory (IRT). These frameworks conceptualize psychological attributes as latent variables, measurable with some degree of error, and rely heavily on statistical inference to quantify these constructs [McNeish et al., 2020]. However, this probabilistic paradigm has faced persistent, fundamental critiques, questioning its very foundations and its suitability for capturing the complexity of psychological phenomena.

One of the most potent criticisms centers on the assumption that psychological attributes are inherently quantitative. Michell argues that psychometrics often assigns numbers to attributes "without ever considering whether those attributes can sustain the operations represented within the empirical numeric relation system so imposed" [Barrett, 2003, с. 75]. This "measurement imperative," stemming from a Pythagorean philosophy that equates science with quantification, has led to a "pathology of science" where numerical manipulation precedes empirical validation of quantitative structure [Barrett, 2003]. This lecture aims to survey the intellectual alternatives that have emerged from such critiques, exploring frameworks that seek to measure psychological phenomena without relying on probability theory, or at least, without its foundational assumptions.

Critique and limitations

One significant limitation of current psychometric practices, particularly those relying on sum scores, is the implicit assumption of a quantitative structure that may not empirically exist for many psychological attributes. As Michell and Barrett forcefully argue, psychometrics often assigns numbers without first verifying if the underlying attribute can sustain the operations implied by those numbers. If a psychological construct, say "anxiety," does not possess a true interval or ratio scale property, then treating sum scores as such for statistical analyses (e.g., calculating means, standard deviations, or using parametric tests) can lead to misleading conclusions about the magnitude and differences between individuals. Without this foundational empirical validation, any subsequent statistical inference, however sophisticated, operates on potentially flawed premises, making it difficult to interpret what the numbers truly represent beyond mere ordinal rankings.

Another critical weakness lies in the often-unexamined assumptions embedded within statistical models, which can severely compromise the generalizability and replicability of findings. Greenland et al. highlight that "every method of statistical inference depends on a complex web of assumptions," many of which are "unrealistic or at best unjustified" in practice. For instance, the assumption of random sampling or independence, crucial for many probabilistic models, is frequently violated in psychological research, especially with convenience samples. If these foundational assumptions are not met, the statistical tests and confidence intervals derived from them become unreliable, leading to inflated Type I error rates or inaccurate effect size estimates. This means that even if a study reports statistically significant results, these findings might not generalize to other populations or contexts, and attempts to replicate them could fail not due to a lack of a true effect, but due to the invalidity of the underlying statistical model's assumptions in the new context.

Finally, the widespread reliance on traditional probabilistic models often overlooks the dynamic, contextual, and emergent nature of many psychological phenomena. Jana Uher critiques the application of propensity probabilities to individuals, arguing that "the individual’s empirical frequencies cannot converge (i.e., stabilise) in the long run as may be true for physical systems." Psychological attributes, such as personality, are assumed to develop and change over the lifespan, meaning that probabilities themselves are not static. This transience and context-dependency challenge models that seek to measure fixed, latent traits. If psychological states are better understood as emergent properties of complex, interacting systems, as suggested by network approaches [Borsboom et al., 2021], then static probabilistic models that treat attributes as stable, independent entities will inevitably misrepresent the phenomenon. The difficulty in addressing this limitation lies in developing mathematical frameworks that can adequately capture such inherent dynamism and context-sensitivity without sacrificing the rigor required for scientific inquiry.