Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Measure validity without holdout tasks


is there a option to measure validity without holdout tasks?
Unfortunately I did non integrate fixed tasks in my Survey.

Thank you very much,
asked May 28, 2019 by bea007 (255 points)

1 Answer

+1 vote
If you're using CBC or MaxDiff, there is a way to calculate in-sample holdout predictability using the CBC/HB Model Explorer.  This is a free tool available for those licensed with our CBC software.  

What the Model Explorer does is to jack-knife sample over the experimentally-designed choice tasks that were included in your experiment.  For example, imagine you had 12 total choice tasks in your experiment that were designed using our "Balanced Overlap" CBC design approach.  The Model Explorer would randomly hold out one or two of those choice tasks from each respondent's record, then estimate the utilities using the remaining tasks.  The utilities would be used to predict the responses to the held out choice tasks.  It repeats this many times using different randomly selected holdout tasks for each replication.  The overall predictive hit rate for the sample is reported.

To download, read about, and use the CBC/HB Model Explorer, visit: https://www.sawtoothsoftware.com/support/downloads/tools-scripts

Warning: this tool is best left to run overnight on your machine, as it often takes multiple hours to finish and it "takes over" your machine, rather than running silently in the background.
answered May 28, 2019 by Bryan Orme Platinum Sawtooth Software, Inc. (179,015 points)
Thank you very much for your help.
I tried the Model Explorer, but unfortunately i always get this error:

"An error occurred reading the utility file.  The file may be corrupted or have non-numeric values (excluding the header row).
More information: On or near line 1: One or more fields cannot be converted to numbers."

Do you know what the problem is?

Thanks in Advance
Sorry, I haven't encountered that before.  Please write walt@sawtoothsoftware.com and see if he has an idea what is the issue.  He's the programmer for  that tool.  Best, Bryan.