International Choice Modelling Conference, International Choice Modelling Conference 2015

Font Size: 
Response Error in HB Choice Models
Keith Chrzan

Last modified: 11 May 2015


By now it’s well known that response consistency affects the scale of MNL model coefficients (Ben Akiva and Lerman 1985, Swait and Louviere 1993).  As responses in a choice experiment have more heterogeneity, MNL coefficients tend to be smaller, all else being equal.  Heterogeneous responses may come from between-respondent differences in preferences (resulting from taste variation or situational effects) or from within-respondent inconsistency (due to context effects and response error).  Consider this a macro-level effect of response consistency:  whatever its source, across a large number of choice sets inconsistent responses tend to cancel out to create a non-directional “white noise” that reduces the magnitude of an MNL model’s coefficients.


At a micro level, however, response inconsistency has a second effect.  Imagine a small model, one based on a dozen choice sets.  In this smaller data set response inconsistencies tend not to cancel out but to result in biased coefficients. 


Smaller data sets are common in commercial choice-based conjoint applications, where analysts frequently use hierarchical Bayesian (HB) MNL to produce their models’ utilities.  In HB-MNL an individual respondent’s choices strongly influence her utilities.  As noted above, in a large sample of responses, inconsistent answers tend to cancel out to produce white noise, but in a small sample of responses we can also get bias.  Call this micro-level bias-inducing effect Error Masquerading as Heterogeneity, or EMH.  So some of the differences we see among respondents’ HB utilities owe to differences in preferences, some to the macro effect of scale and some to EMH.


This presentation seeks to decompose the variance in utilities from HB choice models into (1) the macro effect of scale, (2) EMH and (3) preference heterogeneity.  To accomplish this we graft realistic differences in response error onto utilities taken from respondents in several commercial marketing research studies.  Basing our artificial respondents on actual respondents gives them realistic utilities and patterns of heterogeneity.  We also use realistic levels of response error, namely ones that produce the level of test-retest reliability we often see in well-designed commercial studies.  This enables experimental manipulation of both preference differences and differences in the magnitude of the scale factor (which in turn generates differences in response error).



We find that EMH explains much more of the variance in HB utility estimates than does the macro effect of scale.  Moreover, EMH accounts for about as much variance in HB utilities as does true preference heterogeneity, something analysts need to keep in mind when interpreting HB model results.  In the absence of experimental control, EMH will be indistinguishable from preference heterogeneity so it can lead to over-interpretation of between-respondent utility differences.   The presentation concludes with a discussion of implications for practitioners, as well as some incidental findings about the usefulness of covariates in HB models.   

Conference registration is required in order to view papers.