Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Reporting RLH and Percent Certainty


as there seem to be differences in the RLH and Percent Certainty values of the HB estimation monitor and the respective values computed from the reports, I am wondering which ones to include in my thesis as goodness of fit measures.

Can I just use the average values given on the monitor after the computation has been completed or should i rather calculate them from the reports?

Best regards,
asked Jul 17, 2017 by Colin (305 points)

2 Answers

+1 vote
The on-screen reporting of RLH and Percent Certainty use an exponential smoothed average, where the most recent iterations have more weight than earlier results.  It's useful to have the exponential moving average to plot trend and "see" where the HB results are moving to.

For final summary of fit statistics, I recommend computing RLH and Percent Certainty from the saved draws (after discarding the first, say, 5K or 10K iterations as not yet converged).
answered Jul 17, 2017 by Bryan Orme Platinum Sawtooth Software, Inc. (184,140 points)
thanks Bryan! does this mean I have to check the "Save draws" option before HB estimation and use the draws excel file? I tried this and chose 10 as a skip factor but somehow the file contains all 10,000 draws per respondent.
I think what went wrong for you is that you also need to change the "Number of draws to be used for each respondent" from 10000 to 1000.  Continue to use skip factor of 10.

But, the more I think about it, the RLH that is stored in the standard "point estimates" file is the average of the draws for each respondent.  So, since you are just trying to summarize the average RLH across the draws across the respondents, you might as well just use the RLH found in the point estimates for each respondent and average across those.  Should yield the same answer.
thanks for the answer! do you mean the individual RLH values in the exported utilities file? Should i take the normal arithmetic mean or the geometric mean of these to get the overall RLH?
Arithmetic mean.
Thanks! I want to follow the procedure outlined in your post https://sawtoothsoftware.com/forum/4035/conversions-rlh-into-percent-certainty-and-log-likelihood.
However, I have a dual response none option and 14 random tasks. You said this would be more nasty, should I take 14 or 28 as the number of tasks? Best regards!
It's really nasty, because you have to calculate for each respondent how many choice tasks the would have had formatted for them in the data file.  That depends on how many times they answered "yes" versus "no" to the dual-response None.  When they answer "yes I would buy" then the task is formatted as a single task.  When they answer "no would not buy" then the task is formatted as two separate tasks.
And, to add to the nastiness, the number of concepts differs between concepts formatted to include the None (when the person says they really would buy it) versus those not including the None (when the person says they really would not buy it).  So, the likelihood of data becomes a mixture of tasks that have differing numbers of concepts within each.  This means that comparing the percent certainty across respondents will not be exactly apples-to-apples (though for all practical purposes, it should still work out quite well).
Okay, thanks Bryan I think I can manage to do that with excel! Are there any research papers about this that I can cite?
I am facing the same questions currently and did the following observations with data from a real survey:

1.) Utility report file contains the respondent-level RLH values --> Computed their average which is 0.670
2.) Log file: Computed average RLH based on last 20k out of 40k iterations: RLH = 0.663
3.) same as 2.) just with all 40k iterations: RLH = 0.662
4.) RLH reported by the HB estimation monitor: 0.662

So when it comes to reporting the average RLH value for all respondents, I'd be inclined to take approach number 1, however I do not understand why there is a deviation between approach 1 and 2/3 (the average RLHs in the log file)?
Which RLH values to report?
+1 vote
I might add that, depending on the field you're in, the percent certainty (more commonly known as McFadden's Rho-squared) might be the preferred statistic to report in your dissertation - I think it's the more widely used of the two statistics for academic reporting.
answered Jul 17, 2017 by Keith Chrzan Platinum Sawtooth Software, Inc. (102,700 points)
thanks Keith, I would like to report both RLH and Percent Certainty. Which file should I use to compute the PC?
The saved draws file, as Bryan recommends.