Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Running MBC: If the MNL model better predicts holdout tasks than the HB model, should I stick to the MNL model?

I'm running a MBC survey in which I'm only observing pricing sensitivity to a bunch of features. I've run the analysis using both MNL and HB, and the MNL model is doing a better job predicting both of my holdout tasks than the HB model.

If I understand the IIA problem correctly, it's an issue that arises in a context where we remove or add an option into the simulator. However in this case, my MBC simulator is only predicting preferences on prices I specify. If my understanding is correct, does this mean I would be better off using the MNL model over the HB model?
asked Apr 24, 2014 by Phil

1 Answer

+1 vote
This is a superb question, and my answer will be based on the fact you are using Menu-Based Choice (MBC) rather than CBC.  I'm assuming you showed respondents a menu, where the items were always available within a certain context and prices were showing.  

We have found with such MBC studies that aggregate logit tends to predict "on-the-margin" proportions of choice (summary across the sample) at essentially the same quality as HB.  (For CBC, we've consistently found HB to outperform CBC, but issues such as IIA are much more problematic for CBC than for the type of MBC study you're describing where the items are always present within a single context that all respondents viewed.)

Aggregate logit for your MBC study would seem to work well, if the following are true:

1) You don't need to build a combinatorial prediction simulator, which predicts for the client the likelihood of different combinations of items being chosen (given prices specified in the simulator).  (In other words, you're only interested in the marginal choice likelihoods for the sample of individual items on the menu).

2) You don't need to do a bunch of predictions by different sub-segments of the population for the client.  If the client wants a single simulator that allows them to slice and dice the data by many different market segments, then doing aggregate logit means you'll need to build a new logit model (with its simulator) for each and every slice of the data that the client wants to see. With HB, you'd be able to post hoc slice and dice by segments within a single simulator, using a single HB run.
answered Apr 24, 2014 by Bryan Orme Platinum Sawtooth Software, Inc. (179,015 points)