Hello,
I’ve done a CBC experiment and want to conduct now some tests and see if results are significant.
I read a lot about this topic in the forum and the book chapter “Statistical testing” and wanted to make sure that I understood it right (I decided to go for frequentist tests):
1. Significance of attribute levels / significance of attribute importances
Option 1 – Calculation of t-ratio/Hypothesis testing (estimate unequal 0)
Test: Calculate t-ratio (estimate divided by SE) – if absolute value >1.96, p < .05
Data: Average utilities/Average importances
Option 2 – Confidence Intervals
Test: Look if confidence intervals do not contain 0
Data: Average utilities/Average importances
Rather than spend time testing whether levels/attributes have a significant effect in conjoint analysis (e.g., effect coding can be misleading, attributes mostly significant from 0), you should concentrate on differences
2.1 Significant difference between attribute levels (attribute with 2 levels)
Test: Dependent t-test (if assumption of normality of differences violated: Wilcoxon Signed-Rank Tests)
Data: Individual level point estimates (zero-centered diffs)
2.2 Significant difference between attribute levels (attribute with 3 levels)
Test: One-way repeated measures ANOVA with dependent t-tests as post-hoc tests - Bonferroni correction to be applied (if assumptions of normality and sphericity violated: Friedman’s ANOVA with Wilcoxon Signed-Rank Tests as post-hoc tests)
Data: Individual level point estimates (zero-centered diffs)
3. Significant different between attribute importances (more than 2 attributes)
Test: One-way repeated measures ANOVA with dependent t-tests as post-hoc tests - Bonferroni correction to be applied (if assumptions of normality and sphericity violated: Friedman’s ANOVA with Wilcoxon Signed-Rank Tests as post-hoc tests)
Data: Individual level importance scores
