Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Examining Hit Rates in Summed Price investigation

Dear Sawtooth Software community,
browsing though the forum, I recognized I'm not the first to implement holdout tasks in ACBC investigations.
However, I could not find support regarding how to estimate hit rates (based on these holdout tasks), when using the <b>summed price</b> approach.

In contrast to CBC, where HB estimations will provide path-worth utilities (pwu) for all price levels so I could compared utilities for holdout concepts with actual choices from the holdout task, this is different using summed price.

I've determined a base price of 180$ with 10% variations upwards and downwards. Consequently, I'll receive zero-centered pwu for 162$ and 296$.  In the free format holdout tasks, concepts are priced with 208$, 215$ and 238$.

I've tried to compute hit rates by breaking down the interval between the absolute price difference and the related absolute zero-centered pwu difference. As 229,39 marks the center of the intervall (296-162=(134/2+162)), this should be the price for which the zero-centered pwu become 0.

For 208$ (smaller then price interval center, assuming positive puw) I used the rule of three to derive pwu:
In Excel -> =(($AG$1-$AD$1)/AD2)*($AG$1-$AE$1)
I did the same for 215$. For 238$ (higher than interval center), I've modified the last bracket to ((296+229,39)-238)*(-1), resulting "successfully" in negative pwu, but still a less negative amount than the upper benchmark of 296$.

As I have not tried anything like this before, I'm very uncertain whether this approach really works. As mentioned before, I'd assume that some must have had these difficulties before and might come up with a functional solution.

Thanks in advance!
asked Feb 25, 2020 by Ben

1 Answer

0 votes
It's often easier just to let our market simulator do the predictions for the holdout choice task.  If you specify the holdout choice task as a scenario in the market simulator, use your HB run to do the prediction, and use the First Choice simulation method...and if you request that the output include individual-level predictions, then the simulator will predict for each respondent which concept from the holdout scenario we would expect him/her to most likely pick.  You can then compare that to what the respondent actually picked and compute a hit rate across respondents.  

Rather than use the simulator to do the work for you, you can take the data into a separate analysis tool (such as Excel) and do the work on your own.  You may use each respondent's zero-centered diffs (normalized) utilities to predict his/her choice in the holdout question.  If you used summed pricing, then the default in the software is to use a linear price term in the HB analysis.  The software looks at your base price, level component prices, and random price shock to calculate the low and high prices respondents could have seen.  Then, it uses the linear price term to multiply by the extreme two prices to compute for you the expected utility that would be associated with the low and the high price.  Because this model assumed linearity across the price range, you can linearly interpolate anywhere within that range to compute the utility for price.  Of course, the market simulator software does this linear interpolation for you.

Of course, if you do the analysis in Excel on your own, I'd take a few minutes to double-check your work against the predictions and hit rate that could be worked out by using the automatic market simulator tool built into our software.  Hope that helps!
answered Feb 25, 2020 by Bryan Orme Platinum Sawtooth Software, Inc. (187,915 points)
edited Feb 25, 2020 by Bryan Orme