Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

0 votes

Well, with Avg Variance, it depends on what you think the true variance is for the parameters across the population. If you felt that the variance across the population should be small and yet HB is giving you too large of Average Variance, then you'd think that HB was not fitting the data properly.

Parameter RMS describes the average magnitude of the parameters. With the logit model, the higher the magnitude of the parameters, the higher the certainty of the predictions (closer to 0s or 1s) and the lower the response error. But, high RMS could also mean overfitting in some cases, which wouldn't be good.

So, I suppose I'm giving you the old "it depends" answer. It isn't clear that higher or lower variance and magnitude of the parameters is a good thing for any particular CBC or MaxDiff dataset...unless you know something about the true parameters (which you usually do not).

You may recall that Walt Williams and I wrote a paper in which we investigated 50 or so CBC and MaxDiff datasets and looked at whether the prior variance and prior DF settings in our HB software seemed to be appropriate to fit the data well...but not overfit. We then adjusted the prior variance for CBC/HB software to be 1.0 instead of 2.0, because it seemed like our previous default settings had been a bit too aggressive and typically led to some overfitting (which means too high of Parameter RMS and too high of Avg Variance).

Parameter RMS describes the average magnitude of the parameters. With the logit model, the higher the magnitude of the parameters, the higher the certainty of the predictions (closer to 0s or 1s) and the lower the response error. But, high RMS could also mean overfitting in some cases, which wouldn't be good.

So, I suppose I'm giving you the old "it depends" answer. It isn't clear that higher or lower variance and magnitude of the parameters is a good thing for any particular CBC or MaxDiff dataset...unless you know something about the true parameters (which you usually do not).

You may recall that Walt Williams and I wrote a paper in which we investigated 50 or so CBC and MaxDiff datasets and looked at whether the prior variance and prior DF settings in our HB software seemed to be appropriate to fit the data well...but not overfit. We then adjusted the prior variance for CBC/HB software to be 1.0 instead of 2.0, because it seemed like our previous default settings had been a bit too aggressive and typically led to some overfitting (which means too high of Parameter RMS and too high of Avg Variance).

...