Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

hitrate and MAE - which simulation method

Dear all,

1) Which simulation method of hold out tasks do I need to use when the goal is calculating a hit rate as well as MAE?

I used RFC. For calculating the hit rate it should yield the same results as using first Choice simulation, since in my calculation I kind of follow the same idea: I look at the individual responses and assume that a respondent chooses the option with the highest percentage.

I would like to look at predictive validity and calculate MAE. Therefore, I compared the predicted choices (based on RFC simulation) for three hold out tasks with the actual choices of respondents of these hold out tasks. I am not sure if the RFC method is correct, since the shares of preferences for the options yields different results (RFC vs First Choice). Which simulation method should I use?

2) I have 3 hold out tasks. (6 attributes, 2-4 levels, 4 concepts, 11 random tasks, n=1000)
I can calculate three hit rates, one for each hold out task.
I also calculated a hit rate across the three hold out tasks. I listed three colums for the three hold out tasks with the results (hit or no hit) for all respondents. Then, when there is no hit at all, I compute 0. When there is one out of three I compute 33 .33 %, when there is 2 hits out of three I compute 66.66 %. When a respondent had three hits across the three hold outs I compute 100 %. Then I average the value and report this "overall hit rate".
Is this approach correct?
asked Dec 13, 2020 by Ida27 (295 points)

1 Answer

0 votes
For Raw Hit Rate, the first choice method is used.  If you have 3 holdout tasks, then the possibilities are indeed 0/3, 1/3, 2/3, or 3/3 hits per person.

For computing MAE, you can use either the Share of Preference or the Randomized First Choice method.  You use RFC is you think there is a need to correct for product similarity.
answered Dec 13, 2020 by Bryan Orme Platinum Sawtooth Software, Inc. (184,340 points)
...