Placement of Houldout Tasks in ACBC

Question

Hi everyone,

I'd like to take the opportunity and first say thank you to the Sawtooth Team and the community of this forum for providing so many great insights that helped me a lot so far. Unfortunately, for the question on hand, I was not able to find a clear answer in the previous posts or the technical papers.

I am conducting a ACBC analysis that just went through pretesting. So far, the results are looking quite okay. The study is designed closely to the optimal ACBC design flow, including a calibration section. What I did not account for, are holdout tasks. As I understand, those holdout tasks can only be placed before or after the ACBC section and they have to be created via a fixed CBC task.

What I am wondering now is, where to put these holdout tasks. If I put them before the ACBC section, respondents are not familiar with the concept of the survey or the different attributes and levels. I believe it would be very difficult for them to make a decision. If I put them after the ACBC task (and probably instead of the calibration section), it will be very confusing for respondents to see choice sets with levels they marked as unacceptable. Given that, I imagine either positioning could have a negative impact on the hit rate of the holdout tasks.

So, in summary, I am interested in the following questions:
1) Are holdout tasks a must have for ACBC?
2) Is there any other way to test predictive validity in ACBC without holdout tasks?
3) What is the recommended placement of holdout tasks in ACBC design flow?

Thank you very much!

Best regards,
Thomas

Answer 1

Good questions.  First, the software allows you to add questions anywhere within an ACBC survey.  So, if you created some fixed CBC holdout tasks using a CBC exercise, or a Free Format question, you could insert them within an ACBC exercise.

Next, when I've done this, I've put the fixed CBC-looking holdouts after the ACBC questions.  And, I've told the respondents that we (or a panel of experts) created the holdout questions prior to interviewing them, so they may see some levels that they previously said were unacceptable.  And, that this was expected and OK.

Next, holdout tasks are not always needed.  It just depends on your reason to do them.  Most practitioners don't use CBC-looking holdouts alongside their ACBC questionnaires.

If you are planning on using holdouts to try to compare different model formulations or for some other (typically) academic reason, then you'll need lots of them (typically 5 or more, otherwise you have too few holdout observations to discriminate between good and not-so-good models).  

If you could afford it, the best approach is to interview an entire different group of respondents (drawn to match the same characteristics of your ACBC respondents) who only receive the holdout questions and each completes about a dozen CBC-looking tasks.  You create a market simulator using the ACBC respondents and (after tuning the Exponent down to something like 0.2 to 0.4) you predict the shares of choice for the CBC-looking holdouts.

Also, calibration section in ACBC is not necessary, unless you want to rescale the None parameter to be something specifically tied to a researcher-specified point along the 5-point purchase intent scale.

Comments

Hi Bryan,

thank you very much for the clarification. Since I am conducting the ACBC with an academic background and would like to compare two different samples, I'll include the holdout tasks in my study.

I have a couple of follow up questions regarding the design of those holdouts:
1) Should they look exactly the same as the prior tasks? For example, for ACBC with 3 concepts to choose in the tournament section, should the holdouts have 3 concepts without a none option, as well? Or should/can they look differently, e.g. 2 concepts and a none option?

2) When you say 5 holdouts or more, are you talking about 5 different concept cards or 5 different tasks (leading to something like 15 different concept cards given 3 concepts per task)? For the analysis with the market simulator, I'll probably be interested in something like 5 specific concepts, so any more would mainly serve for predictive validity testing.

3) Is there a recommended way for the pricing of holdout concepts? For some holdouts, one would possibly use (close to) real world prices. But if I'd have to create lots of holdouts, should I just go with the same approach as in the ACBC part (which is a base price + prices for the different levels, that are randomized)?

Thanks again!

Best regards,
Thomas

---

Yes, avoid the None alternative.  The None alternative coming out of ACBC is asked quite differently from the None alternative in a standard CBC.  You will avoid lots of headaches if you avoid the None in the holdout validation.

It isn't necessary to limit yourself to CBC-looking holdout tasks with 3 concepts per set.  ACBC has mixed multiple exercises (BYO, consideration phase, and tournament) which each had different contexts.  Just remember that when predicting these CBC-looking holdouts (when predicting the shares of choice given to these holdouts for the group of respondents) that you should tune the exponent (the scale factor) of the ACBC utilities to obtain best fit to the holdouts (and often the Exponent tuning needs to be cranked down to around 0.2 to 0.4.)  That is just to adjust for the different response error rates between ACBC information and holdout information.  ACBC tends to obtain higher precision and lower error than CBC-looking holdout tasks.  So, the predicted shares of preference from the ACBC simulator will typically be too much accentuated relative to the shares of preference observed in the CBC-looking holdouts.  Also note that if you are using the holdouts for individual-level hit rate calculations, tuning the exponent will not change raw hit rates (since the exponent adjustment will not change which concept per task is predicted to be the best one).

When I say 5 holdouts or more, I'm referring to 5 holdout tasks, where each task has 3 to 5 concepts.  So, 15 to 25 separate "cards" are being shown.  And, I would make them realistic to the world, where the prices are quite related to the features provided by the concepts.  I also would use lots of level overlap (the same levels often repeating across concepts within the same tasks) which is common to real-world arrays of products offered in the marketplace.

---

Thank you very much, Bryan!