Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

What's the easiest way to identify Speeders and "Bad" Responders in an ACBC and/or MaxDiff, before data-processing?

I have an ACBC task for one study, and a MaxDiff for another.

Two data quality indicators might be the following:

1) Speeder: Terminate a respondent if they have a very low total survey duration.
2) "Bad" responder: Terminate a respondent if they keep clicking the same option position repeatedly for x number of times in a set of choice task

How and what sort of code would I use to identify such respondents *during* their time in the study, so that they can be identified to not count towards my sample quota?

(Sorry, I'm still a beginner with Perl!)
asked Apr 27 by yorkmr (450 points)

1 Answer

0 votes
Perhaps the two most helpful metrics would be the page times and the fit statistics.  Fit statistics are calculated after the data is collected and you run your HB model on the data, so not a great option for ACBC.  There is a way to run the fit statistic in real time for a MaxDiff, though it's probably not as good as when you run HB with other respondents.  Here's a paper that goes into more detail on trying to identify random respondents: https://www.sawtoothsoftware.com/168-support/technical-papers/general-conjoint-analysis/2228-consistency-cutoffs-to-identify-bad-respondents-in-cbc-acbc-and-maxdiff

You can calculate page times in the middle of the survey using the PageTime(StartPage, EndPage) function.  For example, you could put skip logic in that terminated a respondent who went faster than you would like.

Be careful with positions though.  With a MaxDiff, for example, the software will try to show respondents all things equally in each row of the question, so someone could like the first item as best many times in a row.

I'm not sure if you need to get into any complicated Perl.  For example, you could add a quota with two cells, one that used the PageTime() function above to put people into a fast versus normal cell (set the cell limits to 999 so nobody actually triggers over quota behavior) or something like that?
answered Apr 27 by Brian McEwan Gold Sawtooth Software, Inc. (47,900 points)