Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Significance test for attribute levels

I would like to test the attribute levels for their significance in an HB. I know, there have been a few questions about this. But I am irritated that the primary advice is to do a t-test. If I have more than two attribute levels, then I understand that I should do an ANOVA or MANOVA. Nevertheless, I do not know what the fixed factor in SPSS should be if I only have the utility values of the three levels of an attribute.

Best Regards,
asked Nov 10, 2020 by bugsbunny (300 points)

1 Answer

0 votes
Best answer

It depends on what kind of significance test you want to run.  

If you want to run a test to see if coefficients are different form zero, sorry, that really is a t-test and that's how all logit programs I know of indicate the significance of individual parameters.  It's not all that useful of a t-test because (a) the attributes in a conjoint experiment are almost always significantly different from zero and (b) because the coding used in utility estimation can itself cause attributes to have zero (or close to zero) utility - in the former case, if you use dummy coding, the reference level will necessarily be zero; in the latter case, if you use effects coding, the standard in our software, and if you have >2 levels in an attribute, very often one of them will be close to the center of the scale, and to zero:  a non-significant result here doesn't mean "this attribute level has no effect," it means "because of the way the data are coded, this attribute level ended up near zero."  

On the other hand, if you want to test whether your levels are different from one group of respondents to the next, then you could, if you wanted do an ANOVA (not a MANOVA) where your fixed effect is sub-group membership (assuming you have >2 subgroups - otherwise, it's still just a t-test).  Of course there are better tests to run here (including the scale-adjusted logit version of the traditional Chow test) but if you like ANOVA, you can do that, too.
answered Nov 10, 2020 by Keith Chrzan Platinum Sawtooth Software, Inc. (102,700 points)
selected Nov 13, 2020 by bugsbunny
Hello Keith,

thank you for your detailed answer! I can fully understand the first part of your answer. But I have another problem. I want to do a group comparison between different segments. Therefore I have selected the respective segment in the utility report. Unfortunately I can only see the aggregated utility value and the corresponding partial utility values for Total, the segment and Other. As I understand it, I would now have to perform an ANOVA or MANOVA to check the significance of the results. However, I lack the standard deviations or individual utility values for this purpose.
If you have run HB analysis, you have the respondent level utilities, right?   So for any subgroup you can identify, you can, in Excel, get their means, standard deviations, etc. for running t-tests.  You can also submit the respondent level utilities to whatever stats program you like and run ANOVA.
Thank you very much, this has already helped me. But I still have an inconsistency, which is the reason why I cannot make any progress with my evaluation. You already mentioned that I have to do an ANOVA and not a MANOVA. But I do have separate utility values for each segment, so in my opinion there are several dependent variables that I want to check for their significance. To be absolutely clear: I want to check if a construct outside the CBC changes the part worth of an attribute. Using the construct I have, as you mentioned, formed three segments. Now I want to check if the differences between the segments are significant. For this purpose I would have performed a MANOVA with the utility values as dependent variable and the respective segment as fixed factor.
I am really sorry if the question is trivial, but unfortunately I have not yet understood this point.
MANOVA is NOT a test for multiple variables, though people often use it that way.  The specific hypothesis that MANOVA is testing is about linear combinations of variables.  Now, some software packages include the univariate F statistics as part of the MANOVA output, and if so you can use those.  But not all do it that way, so you may find that you are running multiple one-way ANOVAs (which is fine - if you want to account for multiple comparisons error, which is, again, NOT what MANOVA does, then I recommend something like the Benjamini-Hochberg routine.
Thank you very much, I was actually not aware of that.

I have one final question: In order to test the significance and effect strength of a construct outside the CBC (trust) on the part worths of an attribute (online trader), I performed a median split for trust. With this, I want to check whether a high level of trust has an effect on the part worth. For this purpose, I used an ANOVA with the trust (fixed factor) on the part worth of a specific online trader (dependent variable) and the test effect strength (partial eta quadrat) output. Does this represent a legitimate way?
Yes, I think it does.  Of course since trust and partworths are both metric, you could run a correlation, too, right?
Many thanks for your support. You have helped me a lot!