reliability check

Question

Hi everyone,

is there any paper about testing the reliability of respondents?

I have included two identical fixed tasks in my CBC study to check whether respondents answer consistently (my supervisor refers to that as a reliability check).

I have been trying to find a paper about it. Unfortunately, I couldn't find any. The only thing I found that is close to what I did is a "test-retest", which is similar but differs in terms of the time horizon.
"Test-retest reliability measures the consistency of results when you repeat the same test on the same sample at a different point in time." This is kind of what I did with my two fixed questions, but not at a different point in time.

Kind regards,
Max

Answer 1

Dominant options and repeated choice tasks are often used as internal validity tests.
You might find this review useful

The Sensitivity and Specificity of Repeated and Dominant Choice Tasks in Discrete Choice Experiments
Marcel F Jonker et al. Value Health. 2022.

Dan

Answer 2

Holdout choice tasks are usually used to test the external validity of your model. Having said that, you need to carefully consider how to build your alternatives in the tasks.

Maybe this paper is also intersting for you: https://sawtoothsoftware.com/resources/technical-papers/including-holdout-choice-tasks-in-conjoint-studies

And I can also recommend this book:
Mariel, Petr; Hoyos, David; Meyerhoff, Jürgen; Czajkowski, Mikolaj; Dekker, Thijs; Glenk, Klaus et al. (2021): Environmental Valuation with Discrete Choice Experiments. Cham: Springer International Publishing (SpringerBriefs in Economics).

On p. 120, you can find the following regarding holdout tasks: "An alternative strategy is to randomly drop a certain percentage of the observations, and estimate the model without the dropped observations (hold-out sample). The estimated parameters are then used to predict the choices of the excluded observations. The number of correct predictions can be used as an indication of how well the model performs outside of the sample. This procedure is less computationally intensive than the leave-one-out test, because each model is estimated only once. The hold-out sample approach is therefore more adequate for larger samples and computationally intensive models."