Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Testing MaxDiff HB model


I read that to test the HB model it is advisable to use "holdout" tasks. I have one holdout task in my survey that all respondents have answered and was not included in the estimation of the model since it is programmed using the free format. Now I want to use it to test the "hit rate" and "Likelihood" of my model but I do not know how to do that.
Can someone guide me?

asked Jan 6, 2020 by AMYN Bronze (2,980 points)

1 Answer

+2 votes

Using holdouts is much more common when folks run HB models - I rarely see them used with MaxDiff.  

To test the hit rate, use your HB utilities to predict, for each respondent, which item has the highest utility and which the lowest.  For those respondents for whom, say, the predicted best item was selected as best int eh holdout task, that's a hit.  Dividing the number of hits by the number of respondents gives you the hit rate.  Of course you could the same for the worst choices and you could report the two hit rates separately or average them for a total.  

The reason folks use them more often for CBC models is that prediction and simulations are much more important in CBC than they are with MaxDiff.  If you have enough holdout tasks (or better yet an entire holdout sample) then you can use the holdouts to test alternative model formulations (does my model with a linear price function predict better than a model with a categorical price function? does a model with an interaction included improve one without? does a model with monotonic constraints on some ordered attributes predict better than an unconstrained model? etc.)
answered Jan 6, 2020 by Keith Chrzan Platinum Sawtooth Software, Inc. (115,150 points)