International Choice Modelling Conference, International Choice Modelling Conference 2015

Font Size: 
Subjective versus Objective Outcome Uncertainty in Discrete Choice Experiments: Implications for Modeling and Questionnaire Design
Christos Makriyannis, Robert John Johnston

Last modified: 18 May 2015


Discrete choice experiments (DCEs) in environmental economics often include choice attributes that are subject to outcome uncertainty, defined as uncertainty regarding the actual outcomes that will occur.  This type of uncertainty can affect respondents’ choices whether or not it is mentioned in DCE questionnaires. Causes of outcome uncertainty can include (1) uncertainty in scientific models and predictions, and (2) uncertainty in the efficacy of policy interventions.  Although the DCE literature often describes this as outcome uncertainty, most cases can be simplified to the more narrow case of outcome risk, in which the probabilities of alternative outcomes are known. 

Despite the ubiquity of outcome risk in DCE scenarios (whether acknowledged or not), the presentation and modeling of this risk remains relatively simplistic.  The majority of DCE questionnaires provide no quantitative information on outcome risk, often with an unstated assumption that presented attribute levels reflect an expected value.  Such treatments typically overlook the fact that respondents’ observed choices in such cases may be conditional on their own subjective perceptions of outcome risk that differ from objective reality, or from what the survey designer intends.  These interpretations can have important implications for the interpretation and validity of model results, particularly if choices are not motivated by linear expected utility theory.  In addition, when risk is quantified in DCEs, it is generally presented as a single probability of policy success or of a single event taking place (e.g., an algae bloom or disease).  These approaches typically assume all-or-nothing outcomes, in which a policy will either be effective (in which case attributes take on presented levels), or ineffective (in which case effects are presumed to be zero).  In reality, very few policies have such all-or-nothing outcomes, leading to questions regarding whether and under what conditions resulting preference and willingness to pay (WTP) estimates are relevant to policy evaluation. 

While some areas of the DCE literature have given greater attention to such topics, the majority of the literature either overlooks outcome risk or models situations that do not closely approximate uncertainties present in actual policy contexts.  Moreover, most extant analyses focus primarily on implications for econometric modeling rather than the underlying treatment of risk in questionnaires. For example, to the knowledge of the authors, there is no DCE in the environmental economics literature that both (1) evaluates the impacts of subjective versus subjective risks, and (2) considers cases other than simple all-or-nothing probabilities.  Such issues are particularly relevant for DCEs applied to topics such as adaptation to coastal flood risk, where respondents’ understanding of risk is central to preference modeling, and in which outcomes cannot be accurately presented in all-or-nothing terms (e.g., floods can occur at different levels of severity, with different effects and probabilities). 

This paper evaluates the extent to which common simplifications in the modeling and presentation of outcome risk can influence preference and WTP estimates.  Unlike prior treatments of outcome risk in the DCE literature, we focus on both the presentation of risk in questionnaires and modeling of resulting response data. The case study addresses preferences and implicit prices related to coastal flood adaptation, including the protection of homes, infrastructure, and natural systems such as beaches and marshes.  Specific attention is given to tradeoffs between engineered versus natural adaptation. The questionnaire, Adapting to Coastal Storms and Flooding, was developed over two years in a collaborative process involving economists and natural scientists, including 13 focus groups. The resulting DCE presents a more realistic, continuous distribution of uncertain outcomes (e.g., homes flooded under storms of different severities) and tests for risk-related scenario adjustment by systematically comparing models estimated using subjective versus objective risks.  This is accomplished using data from three otherwise identical DCE questionnaires.  The first version presents objective probabilities of storms of different severities, with storm damage related to severity. The second version is identical, but elicits parallel subjective probabilities.  The third version is a simplified questionnaire similar to existing DCEs in the literature; this version presents objective outcome risk related only to a single all-or-nothing event (i.e., a storm of a specific severity). 

Attributes and levels were identical for all questionnaires, and were grounded in coastal flooding forecasts developed for the Coastal Resilience platform (, with additional input from focus groups and pretests.  The experimental design minimized D-error for a choice model covariance matrix, assuming both main effects and selected two-way interactions.   The survey was implemented by mail over a sample of 1,728 randomly-selected residents of Old Saybrook, Connecticut (USA).  Out of 1,489 deliverable surveys, 489 were delivered for a response rate of 32.8%.  Models are tested using alternative mixed logit and generalized mixed logit specifications, with various assumptions regarding such factors as scale, fixed versus random coefficients and the distribution and correlation of random coefficients.  We also estimate models using structural utility specifications that allow for different types of linear and non-linear risk preferences.  Results for each DCE version are generally robust across model specifications, showing similar rates of substitution between coastal adaptation attributes.

Results demonstrate clearly that the treatment of outcome risk can have significant and potentially unrecognized implications for the economic and policy conclusions drawn from DCEs.  This relevance extends beyond econometric modeling issues to include core aspects of questionnaire design.  Results suggest that responses to subjective risk treatments differ systematically from responses to objective risk treatments, leading to different conclusions regarding preferences for hard versus soft adaptation.  That is, when respondents are not provided with probabilities of specific outcomes in DCEs, results appear to be influenced by subjective probabilities that differ substantially from objective reality.  These differences can have profound impacts on model results. Results also show that presenting objective probabilities only as all-or-nothing rather than as a more realistic probability distributions influences preference model estimates and policy conclusions.  Taken together, our results suggest the importance of additional research into the way that outcome risk is presented in DCE questionnaires and modeled in the resulting data.








Conference registration is required in order to view papers.