Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Confidence Interval Importances > Alpha draws > Importances > 2.5 & 97.5% percentile?

Dear forum,

in order to report the confidence interval of the importances of attributes in a HB estimation I did the following steps. Is this a legitimate approach, because it is not explicitly described like this in the literature.

1. Take the saved alpha draws file
2. Calculate the attribute importances in each row (non-Burn-In alpha draws)
3. Calculate mean importances based on the alpha draws of the sample
4. Calculate 2.5% percentile of the importances for the lower bound of the 95% Confidence Interval
5. Calcualte 97.5% percentile of the importances for the upper bound of the 95% Confidence Interval

Thanks in advance for your response!

asked Jul 4, 2019 by Frank

1 Answer

0 votes
This is probably not a good approach, unless all your attributes have preferences with a priori and logical order, because importances generally should be calculated at the individual level to avoid the complexity of issues that arise for this due to respondent heterogeneity.

For example, imagine an attribute like brand with just 2 levels: brand a and brand b.  Imagine that 50% of the sample loves brand a and 50% of the sample loves brand b to an equal extent.  In each alpha draw, the utilities for brand a and brand b will be right around 0 and 0 (due to the typical zero-centering in the utility estimation procedure).  The importance calculation using average utilities for the sample will make it look like brand has nearly a zero importance, even though in reality respondents may have very strong opinions about their brand preferences.

On the other hand, if all your attributes are ordered attributes (speed, performance, price, etc.) where essentially all respondents would agree regarding which level was worst and which was best, then the approach like you describe would be reasonable.

All this said, I would probably stop focusing on Importance scores for reporting conjoint analysis results.  They have relative meaning, based on the levels the researcher selected to be in the study.   If you conduct a new study and shift the attribute ranges in terms of attribute levels, then you will obtain different attribute importances even though respondent preferences may remain constant.
answered Jul 5, 2019 by Bryan Orme Platinum Sawtooth Software, Inc. (195,915 points)
Confidence Interval for Importances