Dear Sawtooth Team,
I have conducted two CBC studies with different respondents. The two CBC studies were the same in terms of attributes and attributes levels and only differed regarding the described starting situation/context. Before comparing these two groups I wanted to check, whether there are any interaction effects I need to incorporate in the HB model(s).
For this reason I used the automated interaction search tool.
For treatment group one, the interaction search tool proposes an interaction between attributes A and B.
For treatment group two, the tool proposes an interaction between attributes A and B as well as A and C.
As the automated interaction search tool makes use of logit estimation, I wanted to check, whether I also need to incorporate these interactions, when using (individual-level) HB.
As described in the Sawtooth Software Manual the best approach for doing this (albeit time consuming) is running separate HB models for each interaction effect.
That´s what I did. so I ran three additional HB estimations and checked respectively, whether they can outperform the basic HB model with only main effects in terms of holdout prediction hit rates.
Is this (simply comparing holdout prediction rates) a reasonable approach to decide, whether or not to incorporate interaction effects in HB models? Or are there any other parameters, which I could use to make a decision? Do I need to run an additional estimation for treatment group two, where both proposed interactions (A and B as well as A and C) are incorporated simultaneously?
I only found one interaction (treatment group one, attributes A and B) to slightly increase holdout prediction rate. In absolute terms this model was able to predict one more respondent´s choice correctly (per holdout task) compared to the basic HB model with only main effects. This caused the Hit-Rate to increase by 0.76%. Given this rather small increase I inferred to not incorporate this interaction in the HB model.
Is this a reasonable conclusion?
Also, in the manual you write that you „recently undertook a substantial investigation of about two-dozen CBC datasets, looking for interaction effects that could significantly lift holdout prediction hit rates [were your found] only one or two CBC datasets that seemed to benefit much from inclusion of interaction effects“. Are you referring to the paper `What Are the Optimal HB Priors Settings for CBC and MaxDiff Studies?´
Thanks in advance!
Anja
