Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Recommended design to test main effects for sample size 60

Hi Sawtooth Community,

I'm designing a CBC Experiment with 4 attributes, each with a different number of levels (5, 3, 2, and 3 respectively). This will be a paper and pencil survey and I'm anticipating about 60 respondents.  I'm considering having 4 concepts per task, along with a "none" option. I'm interested primarily in main effects, but I don't want to discount interactions either. I have several questions that perhaps there is no easy answer for.

1. Is 8 an appropriate number of unique versions?
2. Should I use a balanced overlap, complete enumeration or entirely random design?
3. Would the design I use change the optimal number of respondents?
4. In regards to Question 3 I've read this post https://www.sawtoothsoftware.com/forum/8399/sample-size-cbc?show=8399#q8399. I was wondering if there is another way internally determine what is an optimal sample size.
5. Is having 12 tasks sufficient, given an anticipated 60 respondents.

I'm testing design efficiency at the moment, but I hope that by finding answers to these questions I can save some time from avoiding a trial and error way of finding the best design.

Thanks very much!
asked Oct 30, 2015 by Jessie Q. C.

1 Answer

0 votes

Replies to your questions:

1.  8 versions might well work but you would want to test it.  Use the Advanced Test option after pressing the Test Design button if you're using version 8 of our software.  This allows you to test your design with 60 respondents to make sure it will be estimable.  It will identify as well your ability to capture interactions.

2.  Balanced overlap is our default design and it should work well for you in this case, as you want to be able to look at interactions.  You might try random but only if you had a lot of prohibitions (combinations that you want not to occur in your concepts).   Given your sample size such prohibitions will likely cause a lot of trouble, so avoid them if you can.

3.  Optimal sample size, all else being equal, is a census.  The design you choose doesn't change that.  As you trade-off the cost of sample and its ability to buy you precision you'll find that some designs are better than other but balanced overlap is a design strategy that balances well the competing aspects of design quality.  Complete Enumeration or Shortcut might give you designs that have more statistical efficiency, but statistical efficiency isn't the only important thing (at least when you're interviewing humans instead of robots).

4.  More sample is always better.  From a quality standpoint 100 respondents will always be better than 60, 200 respondents will always be better than 100 and so on.  Of course more sample costs more than less sample and some people's budgets are bigger than others.  The link you referenced suggests using the Test Design feature to see how large are your stand errors and whether you can live with those given your budget and I can't give you better advice than that.  

5.  Here I have to repeat the answer from (1) above.  The software gives you the tools to make these decisions but with just 60 respondents you will find that increasing the number of questions/respondent from 12 to 15 say would give you a bump in precision similar to moving from 60 respondents to 75.

I hope this helps.
answered Oct 30, 2015 by Keith Chrzan Platinum Sawtooth Software, Inc. (95,675 points)