Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

How to create and estimate holdout tasks for model valuation?


I create an incentive ACBC for asking the Willingness to pay for product features with summed prices.
I want to create Holdout Tasks, with two or three fixed tasks of a CBC? Where to put them best in the survey? Between Screening and Choice Task Section?
And how to deal with them?

Thank you in advance!
asked Nov 12, 2020 by Monska

1 Answer

0 votes
We have two good white papers that discuss the use of holdouts in conjoint analysis (see links below).  A big hint I can give you is don't include a None concept in the CBC holdout for validating ACBC results!  

And, of course, you should think about whether you even need holdouts in the first place.  To be useful for validation or fine-tuning conjoint models, you would typically need 5 or more holdout choice tasks.  2 or three is usually not enough for that purpose.


answered Nov 12, 2020 by Bryan Orme Platinum Sawtooth Software, Inc. (184,140 points)
Thank you!
So would you recommend to put Holdout Tasks at the end?
Just need them for quality criteria.
I never heard about holdout tasks before. I understand the meaning, but it is not clear to me how to estimate them and how to interpret them afterwards..
I'd recommend them after the ACBC section.  Holdout choice tasks are not estimated.  We simply count what percent of the respondents picked each alternative in the holdout task.  Then, we use our model built from the separate conjoint experiment to predict what percent of respondents we would expect should have chosen each alternative in the holdout task.  We compare the predicted probabilities to the actual probabilities.  The "miss" is usually calculated as mean absolute error (MAE) or mean squared error (MSE).
Thank you.
So I created 5 Holdout Tasks with 3 Options each.
I can see online in the market simulation how much people chose each option in %.
Now I don't know which utilities I should use to estimate the actual probabilities.
Can you help me, how to get the value?
The MAE is then the predicted - the actual % , correct?
You should use the Raw ACBC HB utilities, using either the logit (share of preference equation) or Randomized First Choice (RFC).  MAE is predicted minus actual, averaged across the concepts.
When I use an CBC and want to create one holdout task, would you include the none option?
As mentioned in posts above, I don't like the idea of asking a single holdout task.  It's usually not enough to serve the purposes of holdout tasks.

If you are using CBC utilities to predict the holdout CBC task.  Then, the holdout task should look exactly like the CBC tasks used to estimate the utilities.  If that means including a None concept, then you should do it.