Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Significance test for attribute levels

I would like to test the attribute levels for their significance in an HB. I know, there have been a few questions about this. But I am irritated that the primary advice is to do a t-test. If I have more than two attribute levels, then I understand that I should do an ANOVA or MANOVA. Nevertheless, I do not know what the fixed factor in SPSS should be if I only have the utility values of the three levels of an attribute.

Best Regards,
asked Nov 10, 2020 by bugsbunny (300 points)

1 Answer

0 votes
Best answer

It depends on what kind of significance test you want to run.  

If you want to run a test to see if coefficients are different form zero, sorry, that really is a t-test and that's how all logit programs I know of indicate the significance of individual parameters.  It's not all that useful of a t-test because (a) the attributes in a conjoint experiment are almost always significantly different from zero and (b) because the coding used in utility estimation can itself cause attributes to have zero (or close to zero) utility - in the former case, if you use dummy coding, the reference level will necessarily be zero; in the latter case, if you use effects coding, the standard in our software, and if you have >2 levels in an attribute, very often one of them will be close to the center of the scale, and to zero:  a non-significant result here doesn't mean "this attribute level has no effect," it means "because of the way the data are coded, this attribute level ended up near zero."  

On the other hand, if you want to test whether your levels are different from one group of respondents to the next, then you could, if you wanted do an ANOVA (not a MANOVA) where your fixed effect is sub-group membership (assuming you have >2 subgroups - otherwise, it's still just a t-test).  Of course there are better tests to run here (including the scale-adjusted logit version of the traditional Chow test) but if you like ANOVA, you can do that, too.
answered Nov 10, 2020 by Keith Chrzan Platinum Sawtooth Software, Inc. (103,325 points)
selected Nov 13, 2020 by bugsbunny
I don't see the need for a paired t - if you're comparing subgroups, then it's an independent groups t-test, right?
Yes that would be for subgroup comparison (e.g. gender). But if I simply want to test whether attribute levels A and B are statistically different - those are not subgroups.
Correct, that's where paired t would come in.  Paired t isn't ONLY for measuring the same entity at two points in time, it's for any time there's a dependency between the observations (e.g. they're two measures from the same respondent (as in this case) they're from husband-wife pairs of respondents, etc.
Great, thank you for the peace of mind Keith. And in terms of attributes: If I want to report that relative importances are all different from each other, what is the best approach here? ANOVA?
ANOVA would test whether ANY were significantly different from one another, not that ALL are significantly different than one another.  If your hypothesis is that all pairs are significantly different from one another, then I think you're talking about a series of t-tests (with appropriate multiple comparison procedures, of course).