Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Statistical significance test on MaxDiff Simulator Share of Preference Results


While using the online MaxDiff analyzer Simulator, we see the attribute preference % based on Share of Preference method. My question is - Is there a way to say that the % for Attribute A is SIGNIFICANTLY more / less than of Attribute B? In other words, how much of a difference do we need to say that things are significantly different from one another?

Does it even make sense to ask the 'significantly different' question for HB model outputs? If no, then why not?

If yes, then does it depend on the sample size, number of concepts being tested or anything else? Can we do this significance analysis anywhere in the online simulator? If not, how can we do that offline (the formula)? I am relatively new in this, and would really appreciate some hand-holding while I figure this out please.

Thank you for your time in advance,

Kind regards
asked Jan 17 by kshitijkumarsingh (300 points)
edited Jan 17 by kshitijkumarsingh

1 Answer

0 votes
Best answer
Kshitij, to reply to your questions in order:

The output in the online MaxDiff Analyzer shows 95% confidence intervals, both in small numbers (in parentheses) when you look at the results in table mode, or graphically, if you request it, if you request it in the bar charts.  A quick and dirty test is that If two items' 95% confidence intervals don't overlap, then they're significantly different.  You could also be more technical, inter from the confidence interval what the standard errors are, then use a formula for a dependent t-test, and while that's a little more accurate, it's also a good bit more work.    

It does make sense to wonder if items' scores are significantly different.

Of course the sample size does matter (that's taken into account already in the 95% confidence intervals, and you'll need to take it into account if you use the t-test formula).  

You can compare confidence intervals in Analyzer, or run tests yourself outside of it.  Formulas for dependent t-tests are available in any introductory stat book.
answered Jan 17 by Keith Chrzan Platinum Sawtooth Software, Inc. (111,275 points)
selected Jan 18 by kshitijkumarsingh
Thank you so much Keith, this was helpful! And pardon my stupidity. :)