Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

matched sample t-tests for multiple levels

Hi everyone,

I just wanted to make sure whether I understand it correctly. The Statistical Testing chapter of the book says that matched sample t-tests are recommended for comparing levels within attributes.

So for example, if I have the attribute price and three levels (low, medium, high), than I would do the following t-tests:
1. low - medium
2. low - high
3. medium - high

Would that be correct?

Also, it only makes sense to compare levels of the same attribute, doesn't it?

Thank you!

Kind regards
asked Aug 3 by maxive94 Bronze (1,510 points)

1 Answer

0 votes
Best answer

It DOES make sense to test for differences between the levels of an attribute, if the attribute has a significant effect overall.  Using t-tests you're not accounting for experimentwise error (i.e. you have 3 tests but a t-test at 95% confidence assumes only a single test).  I'd run repeated-measures ANOVA as an omnibus test and then I'd run a multiple comparison test in the case of a significant ANOVA to find where the significant difference is.
answered Aug 3 by Keith Chrzan Platinum Sawtooth Software, Inc. (116,150 points)
selected Aug 7 by maxive94
Thank you very much. So whenever I compare more than 2 groups with one another, I should use F-tests because of the experimentwise error? I was just confused because the chapter of the book says to use matched sample t-tests to compare levels of attributes.

How can I find out whether the attributes have a significant effect?
Matched t-tests are fine for most applied purposes, but for something you want to be able to defend to academics, you would want to account for experimentwise error.  

An attribute have a significant effect if any of its levels are significantly different from zero.  Because of the effects coding conjoint experiments typically use for categorical variables, a lot of 3+ level attributes will have a level that's close to - and therefore not significantly different from - zero.