Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Significance test for attribute levels

I would like to test the attribute levels for their significance in an HB. I know, there have been a few questions about this. But I am irritated that the primary advice is to do a t-test. If I have more than two attribute levels, then I understand that I should do an ANOVA or MANOVA. Nevertheless, I do not know what the fixed factor in SPSS should be if I only have the utility values of the three levels of an attribute.

Best Regards,
asked Nov 10, 2020 by bugsbunny (300 points)

1 Answer

0 votes
Best answer

It depends on what kind of significance test you want to run.  

If you want to run a test to see if coefficients are different form zero, sorry, that really is a t-test and that's how all logit programs I know of indicate the significance of individual parameters.  It's not all that useful of a t-test because (a) the attributes in a conjoint experiment are almost always significantly different from zero and (b) because the coding used in utility estimation can itself cause attributes to have zero (or close to zero) utility - in the former case, if you use dummy coding, the reference level will necessarily be zero; in the latter case, if you use effects coding, the standard in our software, and if you have >2 levels in an attribute, very often one of them will be close to the center of the scale, and to zero:  a non-significant result here doesn't mean "this attribute level has no effect," it means "because of the way the data are coded, this attribute level ended up near zero."  

On the other hand, if you want to test whether your levels are different from one group of respondents to the next, then you could, if you wanted do an ANOVA (not a MANOVA) where your fixed effect is sub-group membership (assuming you have >2 subgroups - otherwise, it's still just a t-test).  Of course there are better tests to run here (including the scale-adjusted logit version of the traditional Chow test) but if you like ANOVA, you can do that, too.
answered Nov 10, 2020 by Keith Chrzan Platinum Sawtooth Software, Inc. (103,325 points)
selected Nov 13, 2020 by bugsbunny
Hello Keith,

thank you for your detailed answer! I can fully understand the first part of your answer. But I have another problem. I want to do a group comparison between different segments. Therefore I have selected the respective segment in the utility report. Unfortunately I can only see the aggregated utility value and the corresponding partial utility values for Total, the segment and Other. As I understand it, I would now have to perform an ANOVA or MANOVA to check the significance of the results. However, I lack the standard deviations or individual utility values for this purpose.
If you have run HB analysis, you have the respondent level utilities, right?   So for any subgroup you can identify, you can, in Excel, get their means, standard deviations, etc. for running t-tests.  You can also submit the respondent level utilities to whatever stats program you like and run ANOVA.
Thank you very much, this has already helped me. But I still have an inconsistency, which is the reason why I cannot make any progress with my evaluation. You already mentioned that I have to do an ANOVA and not a MANOVA. But I do have separate utility values for each segment, so in my opinion there are several dependent variables that I want to check for their significance. To be absolutely clear: I want to check if a construct outside the CBC changes the part worth of an attribute. Using the construct I have, as you mentioned, formed three segments. Now I want to check if the differences between the segments are significant. For this purpose I would have performed a MANOVA with the utility values as dependent variable and the respective segment as fixed factor.
I am really sorry if the question is trivial, but unfortunately I have not yet understood this point.
MANOVA is NOT a test for multiple variables, though people often use it that way.  The specific hypothesis that MANOVA is testing is about linear combinations of variables.  Now, some software packages include the univariate F statistics as part of the MANOVA output, and if so you can use those.  But not all do it that way, so you may find that you are running multiple one-way ANOVAs (which is fine - if you want to account for multiple comparisons error, which is, again, NOT what MANOVA does, then I recommend something like the Benjamini-Hochberg routine.
Thank you very much, I was actually not aware of that.

I have one final question: In order to test the significance and effect strength of a construct outside the CBC (trust) on the part worths of an attribute (online trader), I performed a median split for trust. With this, I want to check whether a high level of trust has an effect on the part worth. For this purpose, I used an ANOVA with the trust (fixed factor) on the part worth of a specific online trader (dependent variable) and the test effect strength (partial eta quadrat) output. Does this represent a legitimate way?
Yes, I think it does.  Of course since trust and partworths are both metric, you could run a correlation, too, right?
Many thanks for your support. You have helped me a lot!
If examining whether the utilities of 2 levels within an attribute are significantly different from each other, one would take the t-test (paired one). However the assumption of a t-test is normal distribution. So if the difference of utilities between level 1 and 2 is not normally distributed, t-test would not work and I guess Wilcoxon Signed-Ranks would be the test to use?
Yes, if you think the utilities are not normally distributed it would be more conservative to use a nonparametric stat test.
Understand. I'm just still having trouble understand why a paired t-test is the right thing (let's assume we have a normal distribution). A paired t-test usually compares one factor/variable that was measured at 2 different points in time and aims to assess wheter or not something has changed significantly. So I could test whether part-worths of one attribute changed. But is it correct to include the part-worths of 2 attribute levels and compare them with each other?
I don't see the need for a paired t - if you're comparing subgroups, then it's an independent groups t-test, right?
Yes that would be for subgroup comparison (e.g. gender). But if I simply want to test whether attribute levels A and B are statistically different - those are not subgroups.
Correct, that's where paired t would come in.  Paired t isn't ONLY for measuring the same entity at two points in time, it's for any time there's a dependency between the observations (e.g. they're two measures from the same respondent (as in this case) they're from husband-wife pairs of respondents, etc.
Great, thank you for the peace of mind Keith. And in terms of attributes: If I want to report that relative importances are all different from each other, what is the best approach here? ANOVA?
ANOVA would test whether ANY were significantly different from one another, not that ALL are significantly different than one another.  If your hypothesis is that all pairs are significantly different from one another, then I think you're talking about a series of t-tests (with appropriate multiple comparison procedures, of course).