Holdout Task Market Simulation SMRT

Question

Dear support team,

I conducted a CBC/HB and decided to include a random task as a holdout task (instead of a fixed one). I  heard this was the recommended way. Now I have issues running a simulation (in order to calculate my hit rate). I understand how to include a fixed holdout tasks, yet I don't understand how I can do the same with a random one.  

Thanks in advance!

Regards,
Dora

Answer 1

Hi, Dora,

I would NOT usually recommend adding an extra "random" task for use as a holdout, especially if running simulations in SMRT.  In SMRT you would need to run the simulations one at a time for each random holdout in order to calculate your hit rate, so if you have 457 respondents, you'd run SMRT 457 times!

What you can do to test your hit rate would be to take the simulation out and run it in Excel.  Then just as you have a row of utilities for each respondent you could have a row (or more) per respondent describing your holdout choice sets and you could make your respondent-specific simulations for your hit rate.  

So it can be done, but it's a bit more work than having a finite number of fixed holdouts in SMRT.  

Please not that we don't recommend having just one or two fixed holdouts.  It's wise to split your sample into quarters or eights, say, and give each cell two or one different holdout questions.  This is a nice, time saving  compromise between having unique holdouts for each respondent and having only one for the entire sample.