Mistake creating Hold-Out Tasks - What do I do now?

Question

Hello,

The following survey was conducted for my university thesis.
I did a CBC consisting of 10 random tasks and 2 fixed tasks.
There were 3 stimuli with no none option per task. In addition, there were 5 characteristics, each with a positive and negative value.

I did not create the Hold-out Tasks according to the recommended 50/30/20 rule, but more or less arbitrarily. Very stupid, I know. I didn't know better...

In the first fixed task, I presented in each stimuli three characteristics in positive expression and two in negative expression.

When I now threw out the utility values via Sawtooth and then reconstructed my hold-out tasks in Excel to determine the highest utility values and based on this my hit rate, I noticed that often 2 of the 3 possible alternatives have the same highest part worth.

Therefore, I cannot determine whether it was a hit because, for example, although the respondent chose stimuli 1, both stimuli 1 and 2 have the (same) highest part worth.

What should I do now or is there an error somewhere in the calculation? Can this still be saved?

I would be incredibly grateful for help!

Answer 1

Hi, Anna,

It seems a little peculiar that you that often have EXACTLY equal utilities.

I'm not sure exactly the purpose for which you collected holdout choices, but I assume you're perhaps doing some sort of validation?  Assuming so and assuming that the utilities really are often exactly equal, I could imagine going one of two ways:
(1) I think you'd be safe to count it as a hit if the choice was of either of the highest utility options.
(2) I think you might count it as a 1 if the (single) highest utility alternative was chosen, 0.5 if one of the two tied highest utility alternatives was chosen and as 0 otherwise.  

I might be more inclined to go with (1) above if you're making comparisons across different treatment cells and you want to test which performed better on holdout prediction,  and with (2) above if you just want to report the hit rate as if that validated your model.

Comments

Hi Keith,

thanks for your ideas!
I have a question: Did I even do it the right way?
I went to Sawtooth Lightstudio -> Analysis -> did a Hierarchical Bayes (HB) -> took the Utilities (Zero-Centered Differences) to Excel -> Then built my Hold-Out Tasks together in Excel and compared the highest value(s) with the real answers of the participants.
Was this the right way to do calculate the Hit-Rate?

Thank you very much!

---

Yes, if you summed the utilities for the levels that comprised each product into a total utility for your holdouts that's what I'd have done.

---

Thank you Keith!
Do you know how the MAE (Mean absolute error) can be calculated in my case since I do have a lot of equal values?

Sorry if I ask too much, but these are the last things that I need for my thesis. :)

---

MAE is easy enough, but you need to simulate the shares from your holdouts (unless your sample size is very large, say many hundreds, you'll probably want to use the logit choice rule rather than just the maximum utility rule you've used so far on your holdouts - write me at keith@sawtoothsoftware if you need help with this part).

Once you've simulated the holdout shares you'll have actual and predicted (from your logit simulation) shares for each holdout question.  Say you have 3 concepts in each holdout question.  And say the respondents  actually chose the following proportions in the first holdout question:  50%, 30%, 20%.  Now say your simulation predicts shares of 45%, 40% and 15%, respectively.  If you subtract simulated from actual shares, then you have
50-45=5   
30-40=-10
20-15=5.
MAE is just the average of the absolute deviations, or (5+10+5)/3 = 6.67.  Of course you'd take that mean across all 6 holdout shares from your two holdout questions.