Holdout Choice Tasks
One of the reasons that Choice-Based Conjoint has gained in popularity over the years at the expense of traditional full-profile conjoint is that the resulting CBC utilities are scaled based on the choice data (rather than taking its scaling from the arbitrary choice of the ratings scale in CVA). When applied in market simulations, these choice-developed utilities lead to shares of preference for competitive products that are directly linked to probabilities of choice. In contrast, CVA utilities are estimated from rating scales or rank-order data. These scales do not automatically lead to market simulations with appropriate choice probability scaling.
CVA users can achieve approximate choice-probability scaling for utilities by adding holdout choice tasks within their CVA surveys (as Select, Grid, or Free Format question types). By tuning the scale factor (exponent) within Share of Preference or Randomized First Choice simulations to best fit holdout choice probabilities, the simulator may be tuned to resemble choice-based scaling. The tutorial in this online help describes the use of a holdout choice task. Additional information on holdout choice tasks can also be obtained from a technical paper entitled "Including Holdout Choice Tasks in Conjoint Studies" at www.sawtoothsoftware.com in the Technical Papers library.
Holdout choice tasks present a set of product profiles (usually three or four) described in full profile using the same attributes and levels used in the CVA questionnaire. Respondents are asked to choose the one alternative from the set that they most prefer. The analyst observes the probabilities of choice across the competing alternatives by simply tabulating the selections. Then, these same product profiles are specified as a scenario in the market simulator. Using a Share of Preference or Randomized First Choice model, the analyst uses the CVA utilities to predict the probabilities of selection for the same alternatives in the holdout task. If the conjoint data are working well and are scaled properly, the predictions should closely match the observed choice probabilities. The analyst tunes the exponent (scale factor) in the market simulator so that the predicted choice probabilities most closely fit the observed probabilities of choice from the questionnaire.
Unless you have additional choice information by which to rescale your utilities for use in market simulations, the scaling of shares of preference may not be appropriate when using CVA. We strongly encourage you to use holdout choice tasks in every CVA study, and that you tune your market simulations using the exponent to fit these holdout choices.
CVA-Looking Holdout Tasks
Some researchers may wish to include ratings-based holdout questions within their CVA studies that look exactly like the other CVA questions. To do this, you can export your CVA design to a .csv file, and then modify the design file to insert a few additional holdout tasks. Then, modify the design settings to specify a new number of total questions (including the additional tasks you've added). Then, import the new design.
When you conduct analysis, make sure not to include the holdout tasks within utility estimation. Within the Settings dialog, use the Tasks to Include filter to exclude the holdout tasks from utility estimation.