Comparing CBC (HB) results of two treatment groups

Question

I conducted a study including two CBC exercises. Participants were randomly assigned to either perform CBC exercise 1 or 2, so I have two treatment groups with different respondents.

The CBC exercises were the same in terms of attributes and attribute levels and only differed in terms of the described CBC starting scenario. (Background info: My CBC exercise(s) included 5 attributes, 2 of which have 3 levels, and the remaining 3 attributes have 2 levels respectively. There are no interaction effects I need to include in HB estimation)

Now I want to examine, how the starting scenario affects the results of the CBC and thus I want to compare the results of the two treatment groups. I use HB estimation. (I ruled out alternative explanations for group differences such as differences in demographics)

Generally, participants of both treatment groups seem to agree on which attribute levels they prefer. So I was thinking about comparing attribute importances. However, as the attribute importances are relative and add up to 100% within each treatment group, comparing attribute importances across treatment groups does not allow me to draw inferences such as attribute A is more important to treatment group 1 than to treatment group 2 (in absolute terms). To draw such conclusions I would have to compare the absolute range of partworth-utilities for an attribute across the treatment groups, right?

What is the best way to do this or generally to compare treatment groups?

I was thinking about plotting the part-worth utilities of both treatment groups in one graph so as one can see, which treatment group has a "steeper" curve/ stronger preference of one attribute level over the other, compared to the other treatment group.

Which parth-worth utilities do I use here? The ones of the separate HB analysis?

Thank you very much for your help! :)

Answer 1

Anja,

There are a few ways to do this.  There is a formal statistical test for differences between groups.  Because utilities in logit models can differ because your two groups of respondents have different preferences differ or because two groups of respondents might answer with different amounts of response error, the Swait-Louviere test (see the Swait and Louviere paper in the Journal of Marketing Research Vol. 30, No. 3, Aug., 1993) decomposes the between-groups test into a test of differences in response error and differences in preference.  

The downside of the Swait-Louviere test is that it uses aggregate logit (not HB) and it requires an iterative procedure that can be cumbersome the first time you try it.  

Shortly I'll reply with how you might handle this within HB, but I've a client meeting I need to attend first.

Comments

Thank you Keith!

I´m looking forward to your answer.

---

OK another way to do this is by subjecting the respondent-level HB utilities to t-tests (I'd use the zero-centered diffs, not the raw scores, as they have tried to correct for differences in response consistency across respondents).  Of course if you have 7 parameters to test (2 from each of your 3 level variables, one each from your 2 level variables) then you will need to correct for the possibility of multiple comparison error, and to avoid that I'd adjust my 7 t-tests to account and correct for false discovery.  For this I recommend the Benjamini-Hochberg procedure, which you can Google or you can find explained in our LinkedIn discussion group here:  https://www.linkedin.com/groups/1715557/1715557-5895943758326820864

---

You can also perform some more properly Bayesian tests on your HB utilities.  These and all the tests discussed so far you can find in the Statistical Testing chapter of our Becoming an Expert In Conjoint Analysis book.

---

Thank you Keith for your extensive response. I´ve read about the Bayesian test before and think it is most suitable. Unfortunately I do not have access to the book as it does not ship to Germany.

More importantly, I have trouble in preparing my CSV file such that I can analyze both CBC´s together. I have designed my study using skip logic that is two CBC´s are included in the study. Treatment group 1 skipped the second CBC (-->empty values in CBC 2) and treatment group 2 skipped the first CBC (--> empty values in CBC1). In the Analysis Manager I can either choose to analyze CBC 1 or CBC 2.

Consequently, I just tried to merge the information of both CBC exercises, so I can analyze them jointly.  First I tried to simply copy the information from CBC 2 to CBC 1. However, when I tried to analyze CBC1 it did not work and the following error message was shown to me "Incorrect exercise ID in the data file". I think the problem is that the different CBC´s are referring to different design ID´s.

Then I found out that in the Export Jobs I can select CBC Export and create a CSV file that only contains the information for the selected CBC exercise. I did this for both treatment groups and created a merged CSV file. I also added the Treatment Group Variable, which I want to use as covariate. As my CBC exercise(s) consists of 15 tasks with 4 concepts (3 plus none option) each, every respondent has 60 lines (in excelCSV) describing how he performed on the CBC exercise. This information can not be put in the Data Management. Only the 15th task and the last concept (concept 4 = No Choice) are imported. Obviously,  there is thus no information available, which could be used to estimate part-worth utilities using treatment group as covariate.

How do I need to transform my CSV file so that it works?

Thank you very much for helping me out!

---

Anja, the book is available as a PDF download here:  http://www.sawtoothsoftware.com/81-products/conjoint-analysis-software/1852-new-advanced-conjoint-analysis-book

Please contact Support@sawtoothsoftware.com about concatenating your data files - they're probably better than me at explaining the steps to do it.

---

Thank you Keith. I have bought the book and I was also able to merge both data files with the help of the support team. I also conducted the Bayesian testing with treatment group as covariate. Now I only have three questions left (regarding the alpha file) to make sure I´m doing it right (most urgent question is question 3.):

1.    First, how do I know, which treatment group was used as reference group? I created a dummy variabe for treatment group were 0 represents treatment group two and 1 represents treatment group 1. According to the paper „Application of Covariates“ the reference group/intercept should be the last level when using a categorial variable. So I assume that in my case treatment group 1 is the reference group (represented by the intercept) right?

2.    To see whether for example treatment group 2 derives a higher value from a specific attribute level than treatment group 1 (reference group), I do simply count the number of times (after ignoring the first 10K iterations) that the covariate weight for this attribute level is >0. If out of the last 10K iterations 9.500 of the covariate weights are >0 then I am 95% sure that treatment group 2 prefers the respective attribute level stronger than treatment group 1. Right?

3.    Most importantly, I have a question regarding the scaling of the resulting part-worth utilities in the alpha file.
Before I ran the Bayesian testing I conducted two separate HB analysis for each treatment group respectively. The resulting average part-worth utilities (zero centered diffs) were on a large scale ranging from
59,05044
11,81122
-70,86167
within an attribute. (for TG1, reference group)
However, the resulting part-worth utilities in the alpha file are on a much smaller scale. (Intercepts (TG1) of 2,0280299674    0,4865802057    -2,5146101731 for the same attribute levels).
Where does this come from?
Why does the scale level differ?

Thank you for being that helpful and patient. It´s the first time ever I´m conducting a CBC so I´m simply lacking experience.

---

Often Sawtooth Software programs report averages of utilities across people for you on a scale called "zero-centered diffs".  Zero-centered diffs puts each person on the same scale, where for each person the range of utilities for attributes has an average of 100.  

However, the original "raw" utility estimates flowing from HB are on logit scaling, typically running from about -3 to +3 for categorical attributes.  Each person's scaling for raw utilities depends on how much response error is involved, so the magnitude of the utilities can differ quite a bit from person to person.  That's why we like to put respondents on the same scale (such as zero-centered diffs) prior to taking averages across people.

But, the market simulator actually uses the raw utilities when converting respondent's utilities into their "votes" (shares of preference).

The way our software documentation reads,  a covariate (such as a demographic) is originally coded as integers starting with 1.  So, gender might be 1=male, 2=female.   When applying a covariate in HB estimation for gender, the software internally changes this to dummy-coding, and you obtain two vectors of utilities: an intercept vector of utilities and the alternative state(s) vector(s) of utilities.  The final level of your covariate (in the case of a categorical covariate) is your reference level (the intercept).  So, the intercept utilities correspond to the  "female" population in my example.  The vector corresponding to Male is the utility adjustment required for the male population, after considering the intercept utility.  Therefore, to obtain the utility for the male respondents for any attribute level, you need to add the intercept utility (the female utility) plus the male utility.

So, if 95% of the draws (after convergence) for the Male utility value within the Male vector of utility (for a given attribute level) are >0, that means you are 95% confident that the male group prefers this level more than females do, relative to the other levels within that same attribute.