Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Testing model validity with identical fixed tasks


I have conducted a CBC and analyzed the results using Logit and LCA. I have already determined the significance of several attributes (using the -2 Log-Likelihood test) and of all my levels (using t-ratios).

Now I was also told that I could check the validity of the model using fixed tasks. Therefore, I have included 2 fixed (holdout) tasks to see if respondents filled in the CBC 'seriously'. Those tasks have the exact same three concepts (and a none option), with identical levels.

My question is; Is it okay for me to just count (in SPSS or Excel) how many respondents chose the same concept in both fixed tasks and divide this amount by the total amount of respondents to get to a percentage and call this a 'hit' rate? Or does some sort of test exist that I can use to determine the validity of the model in a %, using the fixed tasks?

Also, if the simple counting method is correct, what is a desired % of respondents that chose the same concept in both fixed tasks?

Thank you in advance!

asked Oct 10, 2018 by Floor (310 points)

1 Answer

+1 vote
Yes, just counting the percent who answer the same way in both holdouts is the way to go.  In a perfect world you would have 100% hits, right?  But of course we don't live in a perfect world.  All else being equal, higher is better and lower is worse, but we usually see about 70% hit rate for triples.

I mentioned that "all else being equal" because think about it for a minute:  if all your respondents straightlined and always picked the 3rd concept, for example, then your hit rate would be 100% and you would have terrible quality data.  So straightliners may will inflate your hit rate unless you changed the order of the concepts in one of your holdouts (of course, you should probably identify and eliminate straightliners anyway).
answered Oct 11, 2018 by Keith Chrzan Platinum Sawtooth Software, Inc. (104,650 points)
Thank you for your help!