I have been trying to make sense of some CBC data I found in a recent study. I initially ran the counts analysis, and found differences based on the answer to a yes/no question in the survey with some significance (p<0.01 in counts).
I then ran this as a categorical covariate, and I do get differences in the alpha file, showing that the alpha draws for one answer are over 9,500 positive (after removing the first 10,000 draws), indicating some kind of significance. However, when I separate the utility scores of these panelists based on the yes/no groups (Individual zero-centered difference utilities), I do not get any difference, and in fact, the opposite seems true (I get a negative utility for this group, where the counts and covariate seem to indicate positive).
Running these two groups as independent HB analyses gives me utilities that mirror the results from the counts - participants answer the CBC differently when they answer "yes" vs. "no".
Is it statistically acceptable to then treat these two groups as independent and compare two sets of HB results? All documentation I have found indicates that co-variates are the way to go, and I do see this on an alpha level, but can't seem to get the utilities or importance to reflect these differences, and I've only ever seen utilities and importance reported in previous work in my field.
I realize that the alpha file is the population-level statistic, and the individual utilities are individual-level data, but am confused as to how to proceed when these two aspects do not match up, and what is the best practice to report in this instance.
I apologize if I am simply misunderstanding how to use the software or analysis tools!
Any advice or resources are appreciated.