Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Are larger values of Avg Variance and parameter RMS better?

The CBC/HB manual is pretty clear that higher values of pct. Cert and RLH are better, but it doesn't clearly state that larger values of average variance and parameter RMS are better.  I'm looking at two different runs (with different constraints) where pct Cert and RLH are higher in one run but average var and parameter RMS are lower for that run (i.e., the order is reversed between the two runs).
asked Dec 8, 2020 by dgbakken (200 points)

1 Answer

0 votes
Well, with Avg Variance, it depends on what you think the true variance is for the parameters across the population.  If you felt that the variance across the population should be small and yet HB is giving you too large of Average Variance, then you'd think that HB was not fitting the data properly.

Parameter RMS describes the average magnitude of the parameters.  With the logit model, the higher the magnitude of the parameters, the higher the certainty of the predictions (closer to 0s or 1s)  and the lower the response error.  But, high RMS could also mean overfitting in some cases, which wouldn't be good.

So, I suppose I'm giving you the old "it depends" answer.  It isn't clear that higher or lower variance and magnitude of the parameters is a good thing for any particular CBC or MaxDiff dataset...unless you know something about the true parameters (which you usually do not).  

You may recall that Walt Williams and I wrote a paper in which we investigated 50 or so CBC and MaxDiff datasets and looked at whether the prior variance and prior DF settings in our HB software seemed to be appropriate to fit the data well...but not overfit.  We then adjusted the prior variance for CBC/HB software to be 1.0 instead of 2.0, because it seemed like our previous default settings had been a bit too aggressive and typically led to some overfitting (which means too high of Parameter RMS and too high of Avg Variance).
answered Dec 8, 2020 by Bryan Orme Platinum Sawtooth Software, Inc. (184,140 points)