Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Significance testing

Hello,

I’ve done a CBC experiment and want to conduct now some tests and see if results are significant.
I read a lot about this topic in the forum and the book chapter “Statistical testing” and wanted to make sure that I understood it right (I decided to go for frequentist tests):



1. Significance of attribute levels / significance of attribute importances

Option 1 – Calculation of t-ratio/Hypothesis testing (estimate unequal 0)
Test: Calculate t-ratio (estimate divided by SE) – if absolute value >1.96, p < .05
Data: Average utilities/Average importances

Option 2 – Confidence Intervals
Test: Look if confidence intervals do not contain 0
Data: Average utilities/Average importances

Rather than spend time testing whether levels/attributes have a significant effect in conjoint analysis (e.g., effect coding can be misleading, attributes mostly significant from 0), you should concentrate on differences



2.1 Significant difference between attribute levels (attribute with 2 levels)

Test: Dependent t-test  (if assumption of normality of differences violated: Wilcoxon Signed-Rank Tests)

Data: Individual level point estimates (zero-centered diffs)



2.2 Significant difference between attribute levels (attribute with 3 levels)

Test: One-way repeated measures ANOVA with dependent t-tests as post-hoc tests - Bonferroni correction to be applied (if assumptions of normality and sphericity violated: Friedman’s ANOVA with Wilcoxon Signed-Rank Tests as post-hoc tests)

Data: Individual level point estimates (zero-centered diffs)



3. Significant different between attribute importances (more than 2 attributes)

Test: One-way repeated measures ANOVA with dependent t-tests as post-hoc tests - Bonferroni correction to be applied (if assumptions of normality and sphericity violated: Friedman’s ANOVA with Wilcoxon Signed-Rank Tests as post-hoc tests)

Data: Individual level importance scores
asked Aug 27, 2021 by anonymous

Your solution to the original question

Please only use this to answer the original question. Otherwise please use comments.
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
Anti-spam verification:

To avoid this verification in future, please log in or register.
...