Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Can I show 4 questions per set in MaxDiff when I have 6 elements in total

I want to conduct a MaxDiff study.
We want to test 6 elements.
We have decided that an anchored MaxDiff is the best way to go.
We are using the indirect method for the anchor.
We want to show each item 4 times, so we decided on 6 sets of 4 attributes.

When generating the design, I get a warning that you should not show more than half of the total list. But I rather not go down to 3 items per set, since then we would have to go up to 8 sets to be able to show each item 4 times.

Is it absolutely necessary to go down to 3 items per set, or is it okay to have 4 items per set, even though we are only testing 6 items in total?
asked May 19, 2020 by Marten (230 points)

1 Answer

0 votes
A number of years back we did a simulation study to test regarding the optimal number of items to show per set.  (see: https://www.sawtoothsoftware.com/support/technical-papers/maxdiff-best-worst-scaling/accuracy-of-hb-estimation-in-maxdiff-experiments-2005)

We found that showing more than half the items per set as exist in the MaxDiff experiment list hurts statistical efficiency.  That's why it would be better if you have just six items in your study to show just 3 items per set.

If you show 4 items per set, you will lose statistical efficiency in each question and waste respondent effort.  

At the extreme, you see why you wouldn't want to show all six of your items in each set.  The respondent would just choose the global best and worst out of the six total each time, giving you no information about the relative ranking of the middle items.  (Assuming respondents didn't answer with error.)

Similarly, showing five of your six items in each set would lead to very little information about the relative position of the middling items, no matter how many such questions you asked a willing respondent.

Showing four out of six items doesn't yet let you identify with good precision the relative utilities of the six items.  

Showing three items at a time would be optimal for a six-item study.

It isn't absolutely necessary for most projects to show each item 4x to each respondent.  So much depends on how much time you think you have in the questionnaire to spend on the MaxDiff questions.  You have to think through the necessity to have very precise individual-level estimates versus being able to compensate by sample size to cancel out the error at the individual level.  Most researchers I know are happy to target showing each item 3x per respondent to accomplish the typical aims of the typical MaxDiff research project.

In some circumstances, especially when the item list is large, we relax the goal of trying to show each item 3x per respondent and show each item considerably fewer times.  This is called "sparse MaxDiff" and leads to a focus on pooled (aggregate) score estimation via latent class or logit rather than the estimation of individual-level scores via HB.
answered May 19, 2020 by Bryan Orme Platinum Sawtooth Software, Inc. (200,340 points)