Regarding the # recommended tasks in a CVA

Question

It is indicated in the CVA technical paper that the recommended number of tasks is 3 x (total # levels - # attributes + 1).

Just to make sure I understand this right, say we have 3 attributes with 3 levels each. This means that there are 27 total possible combinations. If I go by the formula above, I get: 3 x (9 - 3 + 1) = 24 tasks, which is almost all the possible combinations. If we have 3 attributes, with respectively 2, 2, 3 levels, there would be a possible 12 combinations, but the formula would give us: 3 x (7 - 3 + 1) = 15 tasks.

Am I misinterpreting the formula?

Answer 1

No, I think you're reading it correctly, but the formula is intended as a "rule of thumb" for when the number of questions isn't obvious.  Faced with a 3x3x3 experiment most researchers would go with 9 questions, the standard fractional factorial plan, IF they were willing to assume away interactions.  If they wanted to estimate interactions, larger plans (e.g. an 18 run plan) might do for selected interactions and 27 for any possible interaction.  If you're using HB estimation and allowing several versions of the questionnaire then you might go back to the 9 questions/respondent and estimate your model with interactions.  

With 3x2x2 I think most folks would just use the full factorial of 12 questions.

Comments

Rich Johnson, founder of Sawtooth Software, came up with the rule of thumb to have 3 times the number of questions (in CVA) as parameters to be estimated.  He wanted to suggest that multiplier because it always distressed him that researchers would field CVA plans with very few (or no!) degrees of freedom.  So, it was essentially a desire of his that if we had to go on record with the software to make a suggestion, that the suggestion encouraged researchers to ask well more than the minimum.  (He of course recognized that sometimes the 3x num_parameters guideline wouldn't be practical, and he readily conceded to 2x.)

But, researchers now have HB estimation, which can get away with fewer observations and still obtain equally good results as OLS (where OLS is using more observations).  Most CVA users (applying HB) would be feeling pretty good with 1.5x as many questions as parameters to estimate (Total_levels - Total_attributes +1).

As Keith points out, one needs to think about whether main effects are enough (as CVA assumes) or interactions are needed.  There are power tricks with CVA to estimate interaction effects.  It usually involves collapsing two or more attributes into a single attribute, using multiple versions (blocks) of the design (as CVA supports) and HB estimation.