Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Holdout cards within ACBC

Hello everyone,

in order to analyze my study´s predictive validity and retest reliability, I need to add holdout cards into my ACBC. The manual Lighthouse 9.2 only specifies the approach for CBC and the ACBC Design settings do not offer the section "Fixed Tasks".

Should I add a CBC section to my ACBC survey and choose fixed tasks only (Random Tasks: 0 | Fixed Tasks: 6 )?

If I would like to include the fixed choice tasks (holdouts) into the utility estimations of the ACBC after having tested the validity and reliability, are the CBC choices compatible with the ACBC estimation?

Thanks very much in advance for your response.

asked Feb 26, 2017 by Christian

1 Answer

0 votes
I like the approach of adding a CBC exercise (only to include holdout tasks) together with an ACBC.  Very nice.

You cannot automatically add the CBC tasks to the ACBC tasks for utility estimation.  A power user with excellent data processing skills can export the .CHO file for ACBC utility estimation (a text-only file) and also export the .CHO file for the CBC holdout tasks (also text-only file).  Then (with excellent data processing skills) you can merge the two files so that you add the CBC tasks to the ACBC tasks per person.  WARNING: each respondent's record is of different length for ACBC, so it really can be nasty data processing to merge and combine the two files properly.
answered Feb 27, 2017 by Bryan Orme Platinum Sawtooth Software, Inc. (179,015 points)
Great, thanks for the advice!

I just implemented a couple of holdout tasks as mentioned above (CBC holdouts after an ACBC). Now I am wondering if it is possbile to import the same attribute and level lists I used for the ACBC into the CBC holdout task design? I don´t see the "list" field on CBC Exercise Settings --> Attributes.

Furthermore I wonder how the predictive validity is calculated, since there seems to be no link between the ACBC utility estimates and CBC holdout tasks (if I enter the attributes and levels manualy into the holout design screen). How would the prediction of the experimentee´s choice work, if I combine an ACBC with CBC holdout tasks?

Best regards,

CBC software does not use lists in Lighthouse Studio.  You'll need to use CBC's attribute editor and copy-and-paste the attribute levels manually into your CBC exercise.

You would need to compute the predictive validity manually, since the software cannot do it automatically.

Just two holdout tasks will probably not be enough to judge predictive validity very well or to give reliable guidance regarding modeling the ACBC data one way or another.  Just too few validation data points.  To do a thorough job of this, you should include probably at least 5 holdout CBC tasks.

Holdout validity can be computed in many different ways.  There is individual-level predictive validity (typically measured in terms of hit rates) and share prediction accuracy (a group-level measure).

In either case, you end up building a market simulator using the ACBC utilities.  Then, you specify the CBC holdout scenarios as separate market simulation scenarios (you have to re-specify the attribute levels in the market simulator interface, making sure they are exactly as you fielded them in the CBC holdout exercise).

For individual-level hit rate validation, you can tell our simulator to use the first choice rule (give 100% of the vote to the best product alternative in each holdout choice task), and you ask our simulator to report results to you at the individual level.  This gives you a row per respondent and tells you which product alternative each respondent would be most likely to pick.  You then compare those predictions with the actual holdout choices respondents made.  A hit is counted as 1 and a miss is a 0 (when predicted choice and actual choice agree or disagree).  Average across the hits and misses to obtain the %hit.

For group-level pooled share prediction accuracy, tabulate the actual choices for the holdout CBC tasks.  Those become probabilities summing to 100% across the concepts for each holdout task.  Then, use the market simulator acting on the ACBC utilities to predict shares of preference summing to 100% for those same scenarios.  Compare the predicted shares to the actual shares.  Typical to compare the results in terms of Mean Absolute Deviation (MAD), MSE (Mean Squared Error), or RMSE (Root Mean Squared Error).
Graying out Levels in ACBC holdouts
Source for "No Holdout Tasks in ACBC"?