Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Designing Holdout questions in CVA


I plan on building 3 holdout choice tasks on top of my CVA scenarios.

I have three questions.

First, in terms of a better design of holdout sets, would ratings-based holdout questions identical to other CVA scenarios be better than holdout choice tasks in grid type with three columns? Or is it purely a preference of a researcher?

Second, upon reading the technical papers and other postings about holdout sets, it stated that "Attribute level not repeated within a task".

I have two attributes, both only have two levels.

In building three columns/choices within a holdout set, is it better to include two predicted to be preferred levels and one less preferred level?

Third, if I understood correct, the good holdout sets may be built with some kind of idea of respondents' preferences.

Does this imply that the relevant academic literature along with the news articles can be the reference point in building the holdout tasks? Or, possibly using the simulation results by collecting 50 samples (similar to the example provided in the technical paper) and combine the ranked products possible with 50/30/20% shares?

Thanks for your answers in advance.
asked Jun 18 by anonymous

1 Answer

0 votes
I would prefer choices as the holdouts rather than ratings.  

For the second question, yes, it will be OK to have one of the two levels repeated, so that you have 2 alternatives with level 1 and 1 with level 2, for example.

For the third question, just try not to make questions that have entirely simple answers - for example, having one alternative that has the best level of all attributes so that the choice is a no-brainer for respondents.  Easy choices are too easy to predict and won't be discriminating.  

I don't understand your last sentence.
answered Jun 18 by Keith Chrzan Platinum Sawtooth Software, Inc. (95,775 points)
You totally answered all of my questions. Thank you so much. Take care and be well.