Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Is there anyway to calculate "hit rate" in SSI or SMRT?


I have read many technical papers of Sawtooth and many mention hit rate of the prediction. I wonder whether there is any function or command built in the program of do I need to do it by hand for each respondent and average it? Thank you very much.

PS. I used CBC/HB in my research.
asked May 22, 2014 by anonymous

1 Answer

0 votes
Assuming you read your CBC/HB results (the .hbu file into our market simulator SMRT)...no automatic way in SMRT...but you can get SMRT to help you do the tricky part.

To compute raw hit rate (1 if hit; 0 if miss), set up your holdout as a simulation scenario within SMRT.  While editing the scenario, check the box (bottom left of dialog) to export individual results to file (studyname.shr will be written out to your study folder).  Run the simulation with the First Choice simulation rule (not default Randomized First Choice).

Now, you have a studyname.shr file that contains respondent number followed by which product in the scenario the respondent is predicted to pick.

Export the raw responses to the holdout choice tasks using File + Data Management area + Export Data area.

Open the studyname.shr file in Excel (parse it as space-delimited when prompted, since it is a text-only file, space-delimited).

Open the raw responses to the holdout choice tasks in Excel as well.  Make sure the respondent IDs match up (they should, assuming you used the same filters in the market simulator as you did in SSI Web for exporting.

Check (using Excel's IF instruction logic) to see for each respondent if the actually chosen holdout concept matches the predicted chosen holdout concept from SMRT.  If so, return as a "1" for the respondent; if not return as a "0".  Average across the 1s and 0s to get the raw hit percentage.

Please note that every time you re-run SMRT, it appends new individual-level results at the bottom of the .shr file...so you need to always look at the bottom-most rows (or delete the .shr file to start it anew).
answered May 22, 2014 by Bryan Orme Platinum Sawtooth Software, Inc. (198,315 points)