Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

HB estimation, choice simulator & test for significance

Hey there,

I know there are a lot of forum posts regarding this topic, but however, they are not very clear for my understanding. This is why I again would like to bring this topic up and I am really looking forward in easy understanding information to further procedure.

So I know that there are some frequentist and some bayes ways to calculate quality criteria for HB estimation (as written in your Book: Becoming an Expert in Conjoint Analysis). But my understanding is lacking at the very beginning of this:

1) Which data do I use for those tests? Is it the part-worth utilities on individual level (so simply the result of the HB estimation in Sawtooth), or are other outputs needed (which need to be ticked in the miscellaneous options)? So when I want to perform a t-test with HB estimations, how do I proceed?

2) also do I perform those tests (e.g. the t-test) in an extern software like R, SPSS or Excel or is this done within Sawtooth LHS?

3) Lets say i am interested in comparing my HB estimation results to different groups (e.g. male/female) and I want to know a) if there are differences, b) if the differences are significant between those groups and c) where the differences are.  How do you recommend to proceed?

4) My last questions regards the choice simulator: It is possible to perform a simulation (RFC) of different groups of respondents (like male/female). The result shows me the different Shares of preferences of those groups. But how do I test if those differences are significant? And furthermore lets say I not only simulate 2 scenarios and compare them but a matrix of 100 at the same time. Is there a possibility to provide significance information?

It is very clear how to perform those tests within the aggregate Logit estimations, but I am really lacking here. Thanks a lot for your help and again I apologize for bringing this frequently discussed topic up again.

All the best,
asked Feb 15, 2019 by bs77 Bronze (790 points)

1 Answer

+1 vote
Answers to your questions:

1) Depending on the test you are trying to do, sometimes you'll be using the individual-level point estimates of part-worths (the average of 1000s of draws per respondent, leading to a single vector of utilities per respondent), sometimes you'll be using the individual-level part-worth draws (100s or 1000s of vectors of utilities per respondent), and sometimes you'll be using the sample-level part-worth draws (the "alpha" estimates, 100s or 1000s of vectors of utilities representing the average sample preferences).  The point estimates are automatically output by the software without requesting anything special.  If you want the individual-level draws (within the Lighthouse Studio HB estimation dialog), you'll be checking the "Save Random Draws" box on the Estimation Settings dialog.  If you want the alpha draws, you'll go to the Advanced Output Options menu and click "Successive estimates of alpha (CSV).

The key thing is you need to decide if you'll be using Frequentist tests (in that case use the point estimates) or Bayesian tests (in that case you'll be using either the individual-level draws or the sample-level alpha draws...depending on the test you select that is described in Chapter 12 of the book you mention).

2)  I usually do the tests by pasting the data into Excel...or directly opening the .CSV file or .XLS file in Excel.

3) For comparing different groups on part-worth utilities, decide if you want a Frequentist test or a Bayesian test.

Frequentist test:  Make sure to use the point estimates (one set of utilities per respondent) and that you use the normalized utilities (zero-centered diffs) which are on a normalized and much bigger scale than the raw HB utilities.  Then, take the normalized individual-level utilities plus the grouping variable into a program like SPSS, SAS, or R.  Ask that software to compute an F-test, for significance between groups.  Watch out for the fact that conducting so many independent F-tests can lead to false discovery of significant effects.  Refer to section 12.3 of the book for remedies.

Bayesian test: You'll need to estimate the HB model with covariates in place, representing the groups you'll be comparing.  You do that by clicking "Use Covariates" on the Advanced dialog in Lighthouse Studio HB estimation for CBC.  Also, you'll need to request the alpha file output, as described above.  Then, follow the instructions in the manual or follow the instructions in https://www.sawtoothsoftware.com/download/techpap/HBCovariates.pdf

to count for how many of the "used" draws (those draws after convergence is assumed) have (say) males thinking an attribute level has a higher score than the females.  If (say) 99% of the draws after convergence show males having a higher utility than females on a given attribute level, then you are 99% confident that males have a higher utility (relative to the other levels within the same attribute) than the females.

4.  With RFC and market simulations, you'll need to do a Frequentist t-test.  Note that for each share of preference for the sample, we provide a standard error.  Follow the same formula as expressed in section 12.2.d in Chapter 12 of "Becoming an Expert in Conjoint Analysis".  That's the difference in share of preference between the two groups divided by the pooled standard errors of the shares of preference for the two groups.
answered Feb 15, 2019 by Bryan Orme Platinum Sawtooth Software, Inc. (176,815 points)
Thanks a lot for that really helpful answer, Bryan!
Calculate significance in Choice Simulator