Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Test Design in ACBC

We wish to test the validity of our ACBC design.  Please let us know if we shall rely on 'level counts' or 'std error' or both.

Do we follow same rule of 0.05 in ACBC as well?
asked Aug 28, 2019 by Saurabh Aggarwal Gold (34,700 points)

1 Answer

0 votes
0.05 is a good rule of thumb for considering the overall precision of the design across the sample, assuming aggregate-level (pooled) estimation using aggregate logit.  In that sense, the 0.05 standard that we often teach for standard error threshold under aggregate logit helps you assess sample size requirements (minimums) as well as taking into account the quality of the design (avoidance of prohibitions, etc.).  Even though we teach 0.05, I prefer 0.03 personally as the target threshold.

But, if your goal is to achieve strong individual-level estimation under ACBC or CBC, the aggregate test design via pooled logit doesn't speak to that.  We then rely on rules of thumb for obtaining enough information for each respondent to obtain strong individual-level estimation.  In ACBC, that rule of thumb is for each (non-BYO) level to appear about 3 times per respondent.

So, the best approach is to consider the overall precision of the design (look for standard errors under aggregate logit and random responding robots of 0.05 or less) and also pay attention to rules of thumb regarding getting enough information at the individual level to stabilize individual-level HB estimates.
answered Aug 28, 2019 by Bryan Orme Platinum Sawtooth Software, Inc. (177,015 points)