Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Compute RLH for HB

Hi there,

I would like to compute the RLH (and later also the hit rates and percent certainty) of observed responses for the hold-out choice tasks of a CBC.

Based on the forum posts and tutorials indicated below, I have thus calculated:
a) Utility of concept 'a' for the respondent 'i':
U_ai = U_aif1 + .. + U_aif5, where U_aif1 through U_aif5 is the raw utility of respondent 'i' for the features f1 to f5 of the product concept 'a'.
As described in the link below, the utility of each concept is thus simply the sum of the respondent's part worth utilities for each of the respective features.
(See: https://www.sawtoothsoftware.com/help/lighthouse-studio/manual/estimating_utilities_with_logi.html)

b) Likelihood of observing chosing concept 'a' in each choice task 'x' of respondent 'i' as:
L_xi= P_ai=e^U_ai/(e^U_ai+e^U_bi+...+e^U_di), where P_ai is the probability of picking concept 'a' from the set of concepts shown in choice task 'x' to the respondent 'i', U_ai through U_di are the utilities of concepts ai through di for the respondent 'i', as described in point a) above
(see: https://sawtoothsoftware.com/forum/16929/how-is-rlh-for-maxdiff-calculated?show=16929#q16929)

c) Individual RLH of each respondent i as:
RLHi = (L_x1i * ... * L_x6i ) ^(1/6), where L_x1i through L_x6 i are the likelihoods of the observed responses in the hold-out choice-tasks x1 to x6 using the formula under point b) above.
(see "predict the proportion of respondents who would choose each concept" in:

d) RLH over all respondents as:
RLH = (RLH1 * ... * RLHn) ^(1/n), where RLH1 through RLHn are the RLH of the respondents number i=1 to i=n from point c) above.

To make sure, that these formulas are correct, I first calibrated the raw utilities using all choice tasks, and compared the results with the individual RLH (for point c) and the overall RLH (for point d) that are reported by Sawtooth. However, the results were very different.

Can you spot, whether there is an error in the formulas above, or is there any other diagnostic output I could use (e.g. for steps a) and b), or a tutorial describing the steps more in detail?

Thank you in advance for your reply and happy new year ;-)!
asked Dec 30, 2019 by some1 (175 points)

1 Answer

+1 vote
Best answer
Sawtooth Software's programs compute the RLH for each draw and then take the arithmetic mean of the RLH values across draws for a respondent.  Thus, if you are trying to reproduce the RLH for a respondent and are working from the Point Estimate (the average of the draws), you will get a different answer.

Next, Sawtooth Software's programs take the arithmetic mean across respondents to summarize the RLH for the entire solution.
answered Dec 30, 2019 by Bryan Orme Platinum Sawtooth Software, Inc. (188,115 points)
selected Dec 30, 2019 by some1
Ok, many thanks!  

So in step d) I should use the arithmetic, instead of geometric mean.

In step c), to calculate the RLH of a respondent in the same way as sawtooth, I need to first calculate the RLH for each draw, and then compute the arithmetic mean of the RLH for the different draws.

However, wouldn't it make more sense to describe the quality of the fit by using the RLH resulting from the average part worths across different draws? (i.e. the individual part worths, which are reported by sawtooth at the end of the HB analysis as an estimator of the true part worths)
Yes, it would seem to make sense to use the point estimate (average of the draws) to characterize the fit of the respondent to her choices.  It was the choice of the methodologist/programmer back in the late 1990s here at Sawtooth Software to compute RLH based on each draw and then to average the RLH values across the draws to summarize the RLH for a respondent.  The RLH will be lower this way than if calculating the RLH once based on the average of the draws utilities.
Compute Log Likelihood for LC