Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

CBC Design Efficiency - how to compare

Hello

Let's assume I want to compare several CBC designs (same attributes, same levels).
A. Tasks: 5; Concepts per Task: 4; Respondents 500; D-Eff: 200
B. Tasks: 6; Concepts per Task: 4; Respondents 500; D-Eff: 220
C. Tasks: 5; Concepts per Task: 4; Respondents 700; D-Eff: 400

From what I gathered form the documentation / forum I believe only design A and C can be compared directly since they have the same number of tasks and concepts per task. However, when testing the design the message "(The ratio of strengths of design for two designs reflects the D-Efficiency of one design relative to the other.)"  would suggest that I can also compare design A and B and conclude that adding one task means increasing the efficiency by 10%.

So I guess my question is - Can I compare design strengths regardless of tasks / concepts per task?

Best Regards,
Michal
asked May 22, 2013 by Michal Dudek Bronze (525 points)
retagged May 22, 2013 by Michal Dudek

1 Answer

+1 vote
As long as the attribute and level lists are the same, then you could use relative strength of design (relative D-efficiency) to compare situations involving designs differing in terms of sample size, #tasks, and #concepts.  It just depends on the type of comparisons you are wanting to make.

For example, either doubling the sample size or doubling the tasks should lead to a doubling of relative D-efficiency...since we are doubling the information.  

But, doubling the number of concepts shown per task does not double the relative D-efficiency.

Typically, researchers hold the sample size, #tasks, and #concepts per task constant so they can observe the differences in D-efficiency due to other issues (such as prohibitions).

And, remember, gains in D-efficiency don't necessarily equate to improvements in the final utilities!  D-efficiency assumes people answer like logit-based robots.  Real humans often don't answer according to logit (additive rule, with exponentiated utilities proportional to choice likelihoods).  Plus, real humans often tire out and have limitations to information processing (e.g. real humans don't necessarily have full resilliance if doubling the tasks or doubling the concepts shown per task).
answered May 22, 2013 by Bryan Orme Platinum Sawtooth Software, Inc. (179,015 points)
Thank you for your answer, Bryan.

On a related note, can the Aggregate Std Err's be compared between different designs? I'm thinking to something similar we can do to see the effect prohibitions have on attribute levels.
CBC design selection
...