How to check if the holdout choice task was utility balanced after the HB estimation?

Question

Hello Sawtooth Community,

I had one fixed choice task in my design with four alternatives, which was excluded from the HB estimation and used as a hold out to calculate the hit rate. However, I would like to know, if I can check whether this choice task is utility balanced. Since I have individual utility values for each respondent, I do not quite understand how to do it. Should I average the individual utilities?
Thank you in advance for explanation!

Best regards
Anastasia

Answer 1

Anastasia,

I'm not sure why you want to do this, but you can check the degree of utility balance two ways.

the first way is sort of indirect, but I think it tells you what you want to know:

If you've built a simulator, you could just simulate shares  for the holdout question.  For example, let's say your holdout had 3 alternatives.  To the extent the choice set was utility balanced, your shares should be 1/3, 1/3, 1/3 .  To the extent your shares deviate from an equal 33.333% for each alternative, your utilities were imbalanced.

I suppose you could also sum the utilities for each alternative in the holdout choice set and see how even they are, but because different respondents answer with different amounts of response error, I think this direct measure will be a lot less intuitive than the share-based measure above.

Comments

Thank you Keith for your answer. The share-based measure seems reasonable and I will try it.
The reason I would like to check the utility balance of the fixed task is that in Orme, 2014 "Getting started with conjoint analysis" it says: "...a respondent’s holdout choices usually can be predicted with a hit rate of roughly 75 to 85 percent. These choice tasks typically include between three and five different product concepts...". The hit rate in my study is about 50%, which is of course greater than a chance rate with four alternatives + None. However, I thought, that my hit rate might be much smaller since the fixed task is quite utility balanced and therefore might be difficult to predict.

Best regards,
Anastasia