Subjective Judgments and Bias ?>

Subjective Judgments and Bias

Project Title: Model-based Methods for Debiasing Individual Probability Assessments: Theory and Experiments


When decision and risk analysts obtain subjective probabilities from experts, the objective is to obtain probabilities that are both consistent and free from bias. The former is relatively easy to achieve; with little coaxing, experts provide consistent (i.e., fully additive) probabilities. Obtaining unbiased probabilities, though, is more problematic. Much research has shown that subjective probabilities are subject to a wide variety of biases. Moreover, the impact of biases such as overconfidence or partition dependence can be substantial. The objective of this research is to develop a method to correct for such biases. We specifically examine the effect of three biases, overconfidence, partition dependence, and carryover bias, an understudied order effect in which responses to early stimuli in an elicitation session affect later responses.


The objective of this research is to develop mathematical models with which an analyst can determine the degree of bias for an individual and thereby estimate unbiased probabilities for that person. The research focuses on three biases: overconfidence, the tendency to be too sure that a particular event will occur; partition dependence, in which judged probabilities depend on the particular events for which probabilities are judged; and carryover, an ordering effect in which an individual’s stated probabilities may be affected by previous judgments. The bias measurement and debiasing methods are developed and tested in experimental settings using a large group of participants; the experimental results will show the extent of the biases under various circumstances, and the effectiveness of the method for removing the bias. This research will enable analysts to provide more accurate probabilities when engaged in public- and private-sector risk analyses. Potential applications may include policy areas such as climate change and terrorism risk, areas in which data scarcity, coupled with high stakes, make the use of expert judgments essential.


Funders: This project is funded by the National Science Foundation (NSF).