Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Null RLH for ACBC

This topic has been discussed in length in this forum and the conclusion is that there is no easy so solution to find null RLH for ACBC due to the different amount of tasks on respondent level.

Currently I take the effort to actually do exactly that in order to be able to interpet the average RLH - and to do so it needs null RLH as a reference. I'm writing a script (sort of) that calculates null RLH values for every respondent based on data captured in the .CHO file. The idea is then to average those null RLH values (arithmetic) so that I can compare the "average null RLH of the model" with the actual average RLH of the model.

Does that make sense?

Also, if my calculation of the "average null RLH of the model" is correct, the figure should be very similar to the average RLH from a study with let's say n = 200 robotic respondents, correct? At least that's what I'd expect from a rational point of view.
related to an answer for: Evaluation RLH for ACBC studies
asked Apr 20, 2021 by danny Bronze (1,310 points)

1 Answer

0 votes
HB tries to fit data, even if the data are random.  So, the RLH from HB estimation from random responders should be much higher than the Null RLH you would be computing.
answered Apr 20, 2021 by Bryan Orme Platinum Sawtooth Software, Inc. (203,065 points)
I see. Thinking about this: The calculated null root likelihood is the "chance level", so initially I thought it might make more sense to remove respondents who are below this level (e.g. in a simple CBC triple choice tournament the bad respondents would be those whose RLHs are < 0.33).
Now, there is the other approach with the 95th percentile cut-off which will throw away more respondents. Those respondents are between the real chance level and the calculated 95th percentile of robotic respondents. It's kind of hard to judge which of these methods, in my particular case, makes more sense ...