Confidence Intervall in HB Utility Report

Question

Hello,

I would like to test for significance between attribute levels of an attribute. I have three attribute levels. Is it possible to simply look at the confidence interval in the HB Utility Report? Would it be sufficient to say that attribute level x is not significantly different from attribute level y because the confidence intervals overlap?
Or is it necessary to do a t-test? I have read in the forum that this is also suitable. With three attribute levels, however, I would have to do an ANOVA in order not to increase the type 1 error? But I don't know how to implement this in SPSS, because I don't have directly a dependent and independent variable.

I would appreciate a short feedback very much!

Answer 1

Looking at 95% confidence intervals to see if they overlap (for levels within the same attribute) is the quick-and-dirty shortcut that is a decent approximation for a more precise t-test.  The repeated-measures t-test using HB utilities (preferably normalized utilities, such as zero-centered diffs) is a better option from a Frequentist standpoint.

However, if you want to be true to the Bayesian nature of the estimated utilities, you'd want to do a true Bayesian test for differences in utilities.  This involves looking at the draws of population means (the draws of alpha).

These are described in an article we posted on our website: https://www.sawtoothsoftware.com/169-support/technical-papers/cbc-related-papers/2033-statistical-testing

Comments

Thank you very much for the information!

Would it therefore be a legitimate procedure to first check the confidence intervals and then, in the case of an overlap for the respective attribute levels, to perform a t test for more precise results of the significance?

I'm a little confused because both the confidence intervals and the t-test test significance, right? Is the t test just more precise?

---

I don't think so.  If you just want a quick and dirty test (that isn't as sensitive or precise), then you'll take the lazier approach of looking for non-overlapping confidence intervals.  If you want more precise tests that will tend to find a greater number of truly significant differences, then you'd do one of the more proper tests.

---

Oh okay, thank you very much.
I would do the t-test if I only had two attribute levels. But I have three attribute levels each and it would look a bit random if I only test two certain attribute levels each. With more than two attribute levels, the error of the first type would increase. Therefore, I would still tend to the confidence intervals.

---

Sorry, I have another question about this.
I have done two CBC in a survey. Both CBCs had the same attributes and attribute levels, only the product was different, so the attribute levels for price were adjusted (with the same distances).
Now I want to check if the importance of an attribute is significantly different between the two CBCs. Does this make sense in principle? To do this, I would again simply look at the confidence intervals.
Or could you even tell me which t-test I could use for this in SPSS? I would be very grateful!

---

Looking at whether there is overlap in the confidence intervals isn't the strongest/most reliable test.  Other between-groups tests for continuous variables such as the F-test would be more appropriate.  This could be provided by SPSS or other statistical programs.

---

Okay, thank you. So in this case a matched t test would be suitable because it is one group (whole sample), right?

---

Oh, that's fortunate for you if the same respondents did both conjoint surveys.  That means you can perform the more powerful matched samples (repeated measures) t-test.