Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

quoteable source for the significance-levels > 5% (0.05) in conjoint-analysis needed

Dear Forum

a very popular level of significance in my department (social sciences) is  5% (0.05).
However one of my professors mentioned that Bryan K. Orme pointed out "somewhere" that in conjoint-analysis one could probably also regard siginificance levels of 10% (0.1) or even 20% (0.2).

Right now I am writing my bachelor-thesis and I'd like to if there is a quoteable source for the sign.-levels of 10 and 20% in conjoint-analysis?

Thank you very much in advance! :)
With kind Regards,

asked Jun 19, 2013 by strix (140 points)
retagged Apr 21, 2014 by Walter Williams

1 Answer

0 votes
Dear Sascha,

I might just be forgetful, but I cannot recall a specific thing we wrote in that book ("Getting Started with Conjoint Analysis") regarding this.  But, I do recall writing something related to this in our "Menu-Based Choice documentation" (https://www.sawtoothsoftware.com/download/mbcbooklet.pdf).

First, let's consider standard conjoint analysis.  I can certainly imagine a situation in which a person conducted a conjoint analysis and might be interested in making a decision even though it fell short of 95% confidence.

For example, imagine you conducted a conjoint study to figure out whether a new product you want to bring to market should use level 1 or level 2 of a given attribute.  Let's say also that for some reason sample was either extremely expensive or extremely difficult to obtain, such that you had limited sample size.  

Now, imagine you conduct a statistical test based on market simulations involving your product with either level 1 or level 2 and find that your product with level 1 is preferred to your product when it has level 2 with a confidence level of 90%.  The magic question is whether you would fail to recommend going with level 1 because it fell short of the 95% confidence mark commonly cited in social sciences literature?  

If your boss asked you to make a recommendation, would you say "I don't know whether we should make our new product with level 1 or level 2, because there isn't a statistically significant difference at the 95% confidence level"?  

Or, would you recognize that going with the conjoint analysis results (at 90% confidence) is probably better than a 50/50 coin toss regarding which feature to recommend be included in your new product?

Now, regarding Menu-Based Choice and what we said there regarding including terms in the model (such as cross-effects) that represent less than 95% confidence in being different from zero.  Some researchers might not include an explanatory variable in a model unless it had 95% confidence.  However, with MBC we are trying to build models that make good predictions.  (The emphasis is not necessarily in  making inferences with a low probability of being wrong about the population regarding whether certain predictors are positive or negative).  My opinion is that the benefits of including a term in the MBC model that has a 80% or 90% confidence (of being different from zero) is probably helpful (rather than omitting the term) for obtaining good predictions of market behavior.

See pages 35, 74, 78, and 113 of the MBC documentation for our specific statements regarding use of less than 95% confidence for entering new terms in the MBC conjoint models.
answered Jul 1, 2013 by Bryan Orme Platinum Sawtooth Software, Inc. (175,715 points)