Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

More design versions than respondents: good or bad?

Quick question:
If I plan for a sample of 500 respondents, is there any merit to only generate 500 versions (vs 900 for example)?

Phrased differently: Does the balanced overlap algorithm "optimize" the balance across the number versions I generate? And if I only field a subset thereof, am I missing out on some of the parts that contribute to the "full potential" of  the larger design?

We are wondering if showing a 100 version design to 200 respondents somehow better or worse than showing 200 versions of a 800 version design?
asked Mar 16 by alex.wendland Bronze (2,355 points)

1 Answer

0 votes

I think it's neither.  Each version (or block) is a good design and more of them is better, with strongly diminishing returns after 10-30 versions.  So to some minuscule extent it's maybe a little better to have 500 versions rather than 100 versions, but there's no harm if you have 900 for 500 respondents.  

I change the default setting of 300 versions pretty infrequently.
answered Mar 17 by Keith Chrzan Platinum Sawtooth Software, Inc. (93,025 points)
Hi Keith
Thanks for the response.
I do recall the research from the conferences about the diminishing returns.
Was this done for varying number of respondents?
My thinking was more whether the ratio between number of versions and number of respondents makes a difference.
From what you write it doesn't matter at all if parts of a design are not shown. Each sufficiently large subset of versions of a design in theory is as good as the next subset or the full version design.