Appendix B: Holdout Choice Tasks in Conjoint Studies

We think it is wise to include holdout choice tasks in conjoint interviews, even though they may not appear to be needed for the main purpose of the study. They almost always turn out to be useful, for these reasons:

•They provide a proximal indication of validity, measured by the utilities' ability to predict choices not used in their estimation.

•They provide a check on the scaling of the utilities. If the most popular concepts are over-predicted, then the scale parameter should be reduced. If the predictions are too flat, then the scale parameter should be increased.

•They permit identification and removal of inconsistent respondents.

•They can be used for testing specific product configurations under consideration. Much value can be added by direct measurement of these concepts.

It's hard to design good holdout concepts without some prior idea of respondent preferences. There's no point in asking people to choose among concepts where one dominates in the sense that everyone agrees which is best. And, similarly, it's good to avoid presenting concepts that are equally attractive, since equal shares of preference would be predicted by a completely random simulator. If you present triples of concepts, it's probably best if their shares of choices are somewhere in the neighborhood of 50/30/20.

When conducting CBC studies, if you plan to do segmentation with latent class analysis, it's wise to consider the kinds of groups you expect to get, and to design products in holdout choice sets so that one alternative will be much more preferred by each group.

It isn't necessary to have many holdout sets to check the basic face validity of your utilities or their scaling. However, if you want to use those choices to identify and eliminate inconsistent respondents, you need several choice sets. For ACA studies, holdout concepts can be included in the Web-based interview using general question types.  For CBC, they can be included by adding "Fixed" choice tasks to the design.

We've shown an example of a holdout choice task below:

If you were shopping for a credit card and these were your only options,

which would you choose?

It is probably not very useful to include a "None" option in holdout choice tasks, particularly when these are paired with traditional conjoint exercises which don't have a "None" option. The exception to that rule would be if modeling "None" and using it in the market simulator was of interest.  

Finally, if you do have several choice sets, it's useful to repeat at least one of them so you can obtain a measure of the reliability of the holdout choices. Suppose your conjoint utilities are able to predict only 50% of the respondents' holdout choices. Lacking data about reliability, you might conclude that the conjoint exercise had been a failure. But if you were to learn that repeat holdout tasks had reliability of only 50%, you might conclude that the conjoint utilities were doing about as well as they possibly could, and that the problem lies in the reliability of the holdout judgments themselves.