Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Validity without holdout tasks for CVA

Dear Sawtooth professionals,

For my survey, the total number of parameters to be estimated is 9 (=13 – 5 + 1).

Based on the min # of tasks ranged between 1.5 and 2 times the parameters to estimate, I built 15 CVA scenarios.

I already soft-launched the survey using the customer panel.  

Is there a way to measure the validity without holdout tasks?? Or, can I holdout one task among 15 and measure the validity similar to CBC tasks as shown below?


If there is a way without increasing the total number of scenarios, please let me know.

Or, if there isn't a way, should I include 1 or 2 additional holdout sets for the actual launch of the survey?

Thank you for the answers.
asked Jun 12 by anonymous

1 Answer

0 votes
Unfortunately, we've not programmed a function for CVA as we have for our choice experiments.  But, depending on the scale you're using in your CVA, we can often reformat CVA data so that it can be analyzed as a choice model (and then you could use the CBC Model Explorer).

If not, including a couple of fixed holdouts will allow you to give your client the "feel good" information that you can predict holdouts with a high degree of accuracy.  This isn't really a surprise, and it almost always happens (unless there's a major error somewhere in your analysis).  

But because, barring a disaster, the correlations of predicted and holdout performance are almost invariably high, adding a holdout or two has very little diagnostic value.   Adding enough holdouts to be of diagnostic value is something I do when conducting methodological research, but it is something I rarely do in commercial research, because my clients don't want to pay the extra costs required to do it.
answered Jun 13 by Keith Chrzan Platinum Sawtooth Software, Inc. (95,775 points)