International Choice Modelling Conference, International Choice Modelling Conference 2015

Font Size: 
A new method for preference elicitation: a comparison of trio-wise and best-worst scaling
Seda Erdem, Danny Campbell

Last modified: 18 May 2015


While people can usually comfortably rank a small list of items, as the list of items that are to be ranked increases, the ranking task, obviously, becomes more cognitively challenging and, importantly, susceptible to a range of anomalous behaviours. The best-worst scaling technique avoids this by breaking tasks into more manageable sizes, thereby reducing—if not eliminating—difficulty in ranking the full list of items in terms of their importance (or preferability). Not surprisingly, in recent years we have witnessed an increase in the number of best-worst scaling applications. The technique involves respondents choosing two items in a subset of a large list in terms of an underlying scale of importance (e.g., best and worst, or most and least important, items).


As an alternative to best-worst scaling, we propose a new preference elicitation technique, which we call trio-wise.  Instead of asking respondents their most and least preferred items, we ask respondents to click the point in a triangle that best describes their preference for three different items. Respondents are free to click any point within the triangle.  The Euclidean distances between the point clicked by a respondent and the three corners can be translated into a ranking of the three items.  However, the added benefit of the trio-wise preference elicitation method compared to best-worst scaling is that not only do we obtain a respondent’s rank, but we also get some insight into the strength of their preferences.


In this paper we compare the best-worst scaling approach to our trio-wise preference elicitation method. Specifically, we explore the extent to which preferences are consistent across methods and whether there isany variability of choice through identification of different scale parameters for each preference elicitation method. The modelling approach builds on the standard multinomial logit (MNL) and random parameters logit (RPL) models, but we accommodate the fact that the patterns may be different across respondents. To achieve this we use a latent class framework to probabilistically ascertain the share of respondents whose preferences and error variance are the same as well as those where they differ. For this, we propose a scale adjusted latent class model that is aimed at uncovering both types of differences (preferences and error variance) simultaneously. The beneficial feature of this is that we can disentangle the elicitation method effect upon both the preference parameters as well as the scale parameter.


To demonstrate our approach we use a dataset that was collected via an online survey that had the aim of establishing preferences for different characteristics of public involvement activities in health care system. The extent and nature of public involvement in health care varies considerably. Ranging from (in R&D) identifying research topics, shaping research questions, giving feedback on research materials, helping in the running of studies, and disseminating findings, or participation in clinical decision-making, or priority-setting in health care systems. The means by which such involvement can be achieved also varies: from interviews, focus groups, forums and structured meetings through to postal and online questionnaire methods. Whilst methods of involvement are well established, less well known is (1) how public involvement participants would like to be involved in decisions; and (2) how they rank features of involvement activities (e.g., frequency of involvement, where it happens, financial and time cost; priority area of involvement – e.g., local, national) in terms of their importance to themselves as citizens. Therefore, in this case study we provide a framework for meaningfully incorporating public opinion into the design and application of involvement activities in health care services that are acceptable to participants.


The characteristics of public involvement activities included in the study were based on the focus group discussions, literature review and discussions with a number of health professionals in the UK. As a result, we had eight attributes that describe public involvement activities in health care: activity format, its impact, focus (i.e. local or national), who leads it, where it happens, how often it happens, how much time it requires and how much it costs.  Our study included two treatments: 1) a standard, case-1, best-worst scaling survey that included nine best-worst tasks, each having three items, 2) our new trio-wise preference elicitation technique that included nine preference tasks, each having three items. Both treatments were based on the same main-effect orthogonal experimental design.  



Results from our models suggest that both methods retrieve consistent preferences and that the error variance does not differ substantially between the two methods. However,we show that the strength of their preferences captured through ourtrio-wise preference elicitation method, leads us to prefer it over the standard best-worst scaling technique. Moreover, the feedback we received from the participants also showed that respondents found trio-wise “intuitive” and “an easy way to convey how they felt rather than ticking boxes”.

Conference registration is required in order to view papers.