CBC Model fit

Question

Dear all,

for the logit model you typically evaluate the Chi-Square statistic (2*(LL1-LL0) for significance.

Do you typically do this step also for a HB model? Or only RLH and Pct. Certainty?

Thanks!

Answer 1

Log-likelihood tests (for Sawtooth Software tools) are associated with Aggregate logit and Latent class logit estimation.  With HB, we tend to look at RLH and Pct. Certainty (also known as McFadden's Pseudo Rsq).

With RLH and Pct. Certainty, be careful, because striving for the highest fit possible under HB isn't necessarily the goal that leads to better predictions of held out (new) observations.

For example, one can easily increase RLH by re-running the HB model with a higher prior variance setting.  But, this can lead to overfitting at the individual level.