International Choice Modelling Conference, International Choice Modelling Conference 2017

Font Size: 
On The Robustness Of Efficient Experimental Designs
Sander Van Cranenburgh, John Rose, Caspar Chorus

Last modified: 28 March 2017


On The Robustness Of Efficient Experimental Designs


Sander van Cranenburgha, John M. Roseb, Caspar G. Chorusa

aTransport and Logistics Group, Delft University of Technology

bUniversity of Technology Sydney


Stated Choice (SC) experiments are widely used to acquire understanding on choice behaviour. Nowadays, SC experiments are increasingly being based on so-called “efficient designs”. Efficient designs aim to generate stated choice tasks that maximize the collected information in the data, yielding more reliable parameter estimates with an equal or lower number of observations than traditional orthogonal designs. While earlier research efforts on efficient experimental design mainly focussed on extending the design theory to encompass more advanced models of choice, such as Nested Logit and (Panel) Mixed Logit models, recent efforts are shifting towards understanding the robustness of  the modelling results towards the efficient experimental design. These studies focus on exploring to what extent a particular experimental design loses efficiency when the data generating process does not match the model on which the design is based. This literature has predominantly focussed on misspecification in terms of parameter priors, interaction effects and the way in which (correlations between) error terms are modelled  (Ferrini and Scarpa 2007; Yu et al. 2008; Bliemer and Rose 2011; Ojeda-Cabral et al. 2016)


However, despite compelling evidence that decision-makers use a wide range of decision rules when making choices (Hess et al. 2012), and despite the rapidly growing interest in the choice modelling community into alternative decision rules (Leong and Hensher 2012; Guevara and Fukushi 2016), robustness issues concerning potential misspecification of the presumed decision rule has attracted only very limited attention within the literature. In fact, to the authors’ knowledge, research on experimental designs has exclusively been based on the (often implicit) assumption that decision-makers make choices based on a (linear-additive) Random Utility Maximization (RUM) rule. As a consequence, it is currently unclear what is the influence of different assumptions regarding the decision rules on the statistical efficiency of the design, and on the performance of different choice models under different design assumptions. For instance, do non-RUM designs differ much from RUM-designs?; does a misspecification of the decision rule result in perhaps highly suboptimal designs?; do RUM designs favour RUM models in terms of model fit?


This paper aims to fill these knowledge gaps. To do so, we construct efficient designs based on a non-RUM model – in casu: a Random Regret Minimization (RRM) model – and assess the effects of the design decision rule[1] analytically as well as empirically. We use an RRM model for our analyses because RRM models are among the more popular non-RUM models. Specifically, we use the P-RRM model (Van Cranenburgh et al. 2015) as this model has very convenient mathematical properties for constructing efficient designs. First, we investigate the effect of decision rule misspecification on the statistical efficiency of the SC design. Specifically, we consider two cases: (1) the case in which the experimental designs are optimized for linear-additive RUM while the true Data Generating Process (DGP) is P-RRM, and (2) the case in which the experimental efficient designs are optimized for P-RRM while the true DGP is linear-additive RUM. After that, we use empirical data (collected specifically for this study) to investigate the influence of the design decision rule on the modelling results.



The methodological contributions of this paper to the experimental design literature are twofold. Firstly, we show that for the P-RRM model (Van Cranenburgh et al. 2015) efficient designs can relatively easily be constructed. Because the P-RRM model has a piecewise linear form, the Asymptotic Variance Covariance matrix – which is needed to construct efficient designs – can be determined analytically. Thereby, we complement the choice modeller’s toolbox for designing efficient experimental designs. Secondly, we extend the experimental design literature by developing new insights on the effects of misspecification of the assumed design decision rule on the statistical efficiency. Finally, the substantive contribution of this paper is that we develop new empirical insights on the robustness of modelling results with respect to the assumed design decision rule.

Key findings

  • Conventional RUM efficient designs can be statistically highly inefficient in case RRM is the better representation of the actual choice behaviour


  • Inferences based on empirical SC data on what decision rule (in casu: RUM or RRM) best explains the observed choices are found to be highly sensitive to the particular design that is being used. Model fit differences can be substantial and highly significant.


  • To the extent that a design is more efficient for one particular decision rule (RUM or RRM), the choice modeller is more likely to conclude – based on comparison of the final Log-Likelihoods – that that particular decision rule is the better representation of the actual observed choice behaviour.


Bliemer, M. C. J. & Rose, J. M. (2011). Experimental design influences on stated choice outputs: An empirical study in air travel choice. Transportation Research Part A: Policy and Practice, 45(1), 63-79.

Ferrini, S. & Scarpa, R. (2007). Designs with a priori information for nonmarket valuation with choice experiments: A Monte Carlo study. Journal of Environmental Economics and Management, 53(3), 342-363.

Guevara, C. A. & Fukushi, M. (2016). Modeling the decoy effect with context-RUM Models: Diagrammatic analysis and empirical evidence from route choice SP and mode choice RP case studies. Transportation Research Part B: Methodological, 93, Part A, 318-337.

Hess, S., Stathopoulos, A. & Daly, A. (2012). Allowing for heterogeneous decision rules in discrete choice models: an approach and four case studies. Transportation, 39(3), 565-591.

Leong, W. & Hensher, D. A. (2012). Embedding Decision Heuristics in Discrete Choice Models: A Review. Transport Reviews, 32(3), 313-331.

Ojeda-Cabral, M., Hess, S. & Batley, R. (2016). Understanding valuation of travel time changes: are preferences different under different stated choice design settings? Transportation, 1-21.

Van Cranenburgh, S., Guevara, C. A. & Chorus, C. G. (2015). New insights on random regret minimization models. Transportation Research Part A: Policy and Practice, 74(0), 91-109.

Yu, J., Goos, P. & Vandebroek, M. (2008). Model-robust design of conjoint choice experiments. Communications in Statistics—Simulation and Computation®, 37(8), 1603-1621.

[1] We use the term ‘design decision rule’ to refer to the decision rule of the model under consideration when optimizing the experimental design.

Conference registration is required in order to view papers.