Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

CBC Design Testing

I have a couple of questions on the CBC design testing feature in SSI Web.

I'm trying to design a CBC exercise with alternative-specific attributes. What should I look out for in the "test design" diagnostics? I get a message that the design is deficient,  but all the standard errors are less than 0.05.

More generally, what model does the "test design" module estimate? It says OLS efficiency is reported. Are the results from a linear regression of shares/counts v/s product features, or is it a logit type model?
asked Oct 17, 2013 by Varun Pattabhiraman

1 Answer

+1 vote
There are two "Test Design" approaches in CBC software: the "Quick Test" (which is automatically reported when you generate an experimental design) and the "Advanced Test" (which you need to invoke later).  If you are looking at a report that tells you it is using OLS, then you are using the "Quick Test".

The "Quick Test" uses OLS (not the actual statistical model you'll end up using to estimate utilities in the end).  It zero-centers the design matrix within task and performs a quick OLS analysis.  The main virtue of this test is it will quickly report if the design is deficient.

We recommend users also do the "Advanced Test", which you run by clicking Test Design from the Design tab.  Then, you check the Advanced Test box.  That will generate dummy respondent data with the sample size and None% you request.  It runs aggregate logit, reports the standard errors for the utilities and also reports relative D-efficiency.  As a basic rule of thumb, we recommend trying to strive for a questionnaire and sample size so that the largest standard error across any of the attribute levels is somewhere around 0.05 for main effects (preferably 0.03)...or 0.10 for interaction effects (preferably 0.06).  Of course, this assumes ideal conditions where sample is not terribly expensive to recruit and the population of interest is large.
answered Oct 17, 2013 by Bryan Orme Platinum Sawtooth Software, Inc. (190,165 points)
Great, this is helpful. What behavioral/model assumptions does Sawtooth make while simulating respondent choices in the "Advanced Test" tab? Does it assume some logit model coefficients/utilities to generate choices and then re-estimate the model given the observed/generated choices? Or are the choices determined using completely random utilities for the respondents (i.e. no systematic components)?