Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Comparing CVA chip allocation to CBC single profile w/ none and chip allocation

CVA, single profile with chip allocation
CBC, single profile with None and chip allocation

These two options seem approximately equivalent, if not exactly equivalent.  Chip allocation with CVA does not require you to allocate chips to a none option like CBC does in these setups, but the mathematical implication is equivalent – you could essentially run this CVA design and add a none field post hoc and just fill it in with the remaining chips out of 100 – then run through CBC analysis.  Is this true, that they are equivalent or is there some pro or con in terms of one being more robust way to generate utilities – because the datasets, are essentially the same, and reflects the same input thought process from respondents.

This then leads to the natural follow-up.  If I am testing a design in CVA (again, with the setups I described above), CVA only provides me a test of the efficiencies which is interesting and somewhat informative, but it doesn’t take into account sample size or really give me a net target number to work towards.  CBC, on the other hand, gives me reliabilities so I can see with my target of <.05, how good our design is and allow me to have more confidence in what tweaks are more beneficial.  So my thought is to test the design with CBC but then implement it in CVA and run the analysis in CVA.  I don’t want respondents to have to fill in a chip allocation for the profile and then unnecessarily fill in the sum to 100 in the none option (for CBC) – its an unnecessary additional step for them.  

Does anything here sound off-base in my thinking of these CVA vs CBC designs?
asked Feb 6 by stevetlg.com (480 points)

1 Answer

+2 votes
Yes, good observation.  But, HB-OLS estimation is different from HB-Logit estimation.  Of course, you can ask HB-OLS to do a logit transform of the dependent variable.  And, that would get HB-OLS results perhaps closer to the HB-Logit results.

Of course, with a bit of data processing, you could collect the results in CVA, but export and prepare the data (with a bit of data processing) as if it were chip allocation vs. the None, then submit that modified file to our CBC/HB system for HB-Logit estimation.
answered Feb 6 by Bryan Orme Platinum Sawtooth Software, Inc. (177,015 points)
Great, that definitely helps.  If the goal is to maximize reliability/accuracy of the utilities in a situation where there are probably on the lower side of acceptable numbers of exercises (because single profile is such a sparse design), would you choose to run the analysis in CVA or move over to CBC/HB?
If by CVA you mean CVA-HB estimation with a logit transform of the dependent variable Ithis is a setting in the CVA analysis interface), then I would think that HB-Reg and CBC/HB should produce somewhat similar results.  Though, because of the tighter connection between the CBC/HB formulation of the problem and its estimation, it produces utility scores that are closer linked to choice probabilities (when those scores are exponentiated).