Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

CVA reliability

I forgot to repeat the same CVA scenario twice in order to do test retest reliability of my CVA scenarios.
Is there any other way to measure reliabilty for CVA or suggestion?

Out of a total of 500 samples, I may be able to export-import by adding duplicated CVA scenario and get about 50 to 100 samples.

Would this amount (10 - 20%) be okay to test the reliability of CVA scenarios?

Since the point is to see the consistency of the measure?

Any thoughts would be appreciated.
asked Jul 14 by anonymous
Could you clarify what you mean by export-import and adding scenarios?  Are you talking about collecting additional respondents?
Dear Brian, I was talking about repeating same CVA scenario twice by export-import-ing original scenarios. Yes, I still have about 10-20% more respondents to collect. I was wondering whether it is okay to only have a little percentage of total sample size are tested for reliability (test-retest) of CVA. As far as I know, test-retest reliability should be done for the total sample. Thank you for your response.
Unfortunately you're going to run into some issues if you change your experimental design in the middle of data collection.  When you push the Generate Design button in the software, a unique Design ID is created and is saved in the respondent data records.  If you were to modify your design right now by exporting it, changing it, and re-importing it, the software would think (rightly so) that there is a different set of questions than what is currently in the software and it would assign a new design ID.  You would then get errors when you are ready to run your analysis because all your existing data didn't see the same design as your current day.

I'm not sure about the best way to handle this to make it easy.  If it was very important to you to have this test in place, you might have to add a second CVA exercise to the questionnaire to handle the extra questions, and then potentially export everything out to Excel to combine data if you wanted to use the duplicate questions for utility estimation.  Honestly I would probably just add the second CVA exercise for the test/re-test reliability and probably not worry about including them in the modeling to prevent a lot of extra work.

Keep in mind you will also get a fit statistic as part of the utility model and survey page times, so if you do not have this test/re-test metric it's not like you have nothing to help you look whether or not someone was just clicking through the survey.
Dear Brian, I thank you for your wisdom and advice. I will surely keep in mind that a fit statistic in the utility model may be used to see the reliability of the CVA scenarios. Your comments may have saved my hard work and possible publication in my field. I truly thank you for your time.

Your solution to the original question

Please only use this to answer the original question. Otherwise please use comments.
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
Anti-spam verification:

To avoid this verification in future, please log in or register.