Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Number of hold-out task in CBC


How many hold-out task should be included in CBC? What range of hit rate represent a robust measurement? Why the more alternative per task reduce the hit rate?(mentioned in glossary of website)
How much is acceptable range of Mean Absolute Error or Mean Squared Error in assessing simulator validity?
Thanks for your support
asked Jun 21, 2017 by Robin59 Bronze (545 points)
retagged Jun 21, 2017 by Walter Williams

1 Answer

0 votes

Please check our LinkedIn discussion group where we posted this article about the number of holdout questions (https://www.linkedin.com/groups/1715557/1715557-5977114043452841985)

To answer your other questions, more alternatives in the task reduces hit rate because, just as, for example, it's easier to predict a coin toss (50/50 chance correct by chance) than the roll of a 6-sided die (where the likelihood of a correct guess by chance alone is 1 in 6).  

There are no set standards for MAE or MSE in absolute - they are comparative measures that we can use to judge if one model is performing better than another.
answered Jul 2, 2017 by Keith Chrzan Platinum Sawtooth Software, Inc. (116,275 points)