Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Different results using effects vs. dummy coding in CBC / HB

Hello there,

I am using CBC / HB version 5.5.4 and I have a question regarding differences in resulting relative importances when running the estimation with effects vs. dummy coding. My burn-in phase consists of 200k iterations and I use 1800 draws for each respondent with a skip factor of 50 (I cannot increase this number since the software crashes otherwise. My data set is very large, I include up to 50 interaction terms and one dummy-coded covariate. I also tried increasing the number of draws by running it in batch mode, so the graph is turned off).

Previously, I estimated many different models (main effects only, with a few interaction terms, with all interaction terms, with/without covariates etc.) always using dummy coding. The resulting relative importances for my five attributes were very similar across all models (I calculated importances on individual level first and averaged them second). Now, I estimated a model using effects coding and the relative importances changed by up to 5%-points. If I calculate importance for the interaction terms too, I notice that all interaction terms lose importance and the main attributes gain importance (up to 9%-points). What do you think could be the reason for that? I also wonder which results are more reliable.

Thank you very much in advance and kind regards.
asked Sep 5, 2018 by Leo (310 points)

1 Answer

0 votes
Dummy coding does not allow you to retain the same interpretation of the part-worth utilities for main effects after you have also specified interactions.  Meaning, if you use interactions with dummy coding, the utility that was previously being partialed out and attributed to the main effects may no longer be partialed out (captured) and may spill over and be attributed to interaction effects.

As another example, if I estimate main effects using effects-coding and compare those main-effect utilities to another model where I added an additional interaction effect, the main-effect utilities between the two models will look very similar in terms of their relative patterns.  However, if I do the same thing for dummy-coding and compare the main effects before and after adding an interaction effect, the main-effect parameters may look VERY different...because some of their main effect signal has been captured instead in the interaction terms.  This is because in the design matrix, the coding between main effects and interaction terms is not orthogonal for dummy coding.

That's why so many researchers prefer to use effects coding when estimating interaction effects.
answered Sep 5, 2018 by Bryan Orme Platinum Sawtooth Software, Inc. (176,515 points)
That was a comprehensive and well understandable answer. Thank you very much!!