Dear all,
1) Which simulation method of hold out tasks do I need to use when the goal is calculating a hit rate as well as MAE?
I used RFC. For calculating the hit rate it should yield the same results as using first Choice simulation, since in my calculation I kind of follow the same idea: I look at the individual responses and assume that a respondent chooses the option with the highest percentage.
I would like to look at predictive validity and calculate MAE. Therefore, I compared the predicted choices (based on RFC simulation) for three hold out tasks with the actual choices of respondents of these hold out tasks. I am not sure if the RFC method is correct, since the shares of preferences for the options yields different results (RFC vs First Choice). Which simulation method should I use?
2) I have 3 hold out tasks. (6 attributes, 2-4 levels, 4 concepts, 11 random tasks, n=1000)
I can calculate three hit rates, one for each hold out task.
I also calculated a hit rate across the three hold out tasks. I listed three colums for the three hold out tasks with the results (hit or no hit) for all respondents. Then, when there is no hit at all, I compute 0. When there is one out of three I compute 33 .33 %, when there is 2 hits out of three I compute 66.66 %. When a respondent had three hits across the three hold outs I compute 100 %. Then I average the value and report this "overall hit rate".
Is this approach correct?
