Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

CBC design selection


I wanted to know how much leeway one has as far as letting the standard errors (of main effect for example) grow beyond the recommended 0.05? Is a 0.057 acceptable or not advisable at all?

For example I have a situation in which the attribute with the most levels has std errors per the test deign report going upto 0.057 that's with the maximum sample, maximum tasks per respondent and maximum (reasonably maximum) concepts per task I can accommodate. Increasing the concepts does not seem to help significantly and considering the overkill it would involve I don't think that'd be advisable. The alternate  option I can see is dropping 1-2 levels in that attribute with most levels and then do an interpolation of utilities. I achieve std errors within 0.05 doing so. I understand given the levels become different I cannot use the design strengths to compare.

Can anyone advise what would be the best way forward in this case?

related to an answer for: CBC Design Efficiency - how to compare
asked Jun 1, 2018 by gaurav

1 Answer

0 votes
Best answer
The 0.05 is a rule-of-thumb based on experience with many past studies.  While they're a little hard to imagine, I suppose there could be some studies where the stakes might be so high that 0.05 would not be precise enough and where you would want much tighter standard errors (maybe for a study involving life-and-death decisions).  

For many other studies (probably most) going a little above  the 0.05 recommendation isn't likely to be fatal:  statistics is a land of hills, not cliffs, so it's not like your study leads to eternal perfect sunny days if the standard errors are 0.05 or less while it's a crash-and-burn disaster of bankrupted companies and devastated analyst careers if it goes a little over.  

In any case, the 0.05 rule-of-thumb was established for aggregate MNL analysis and most of us are running HB analysis these days, so the rule is an approximation to begin with (it's a good approximation and we think you should pay attention to it, but it's not written in stone and it's a rule we can bend from time to time).
answered Jun 1, 2018 by Keith Chrzan Platinum Sawtooth Software, Inc. (105,750 points)
selected Jun 1, 2018 by Nathan Bryce