Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Randomize survey version presentation in equal porportions

Hello all,

I have two different versions of the same survey (similar, but with different questions). I want to randomly present them to the user that clicks on the provided URL. The only constraint is that I need to assure that both versions are presented the same number of times (for instance, in 300 respondents, version 1 must be presented randomly to 150 users, the same in what concerns to version 2), so "pure" randomization is not an option.

I thought of making a script in Perl (or using JavaScript) to randomly store "1" or "2" in a array of size 300. When one of the numbers had been stored 150 times, the rest of the array would be filled with the other number. This would assure the conditions I'm looking for.

But then I thought that maybe this could be possible inside SSI Web, maybe using one block for each survey version in conjunction with lists, or something. Is this possible? If yes, how?

Thank you very much,
João Fernandes
asked Nov 15, 2012 by João Fernandes Bronze (2,040 points)
retagged Nov 15, 2012 by Walter Williams
Another method is to insert a "1" or "2" as a merged sample field and control the counts via quotas.

2 Answers

+1 vote
Best answer
Place a quota in the survey with two cells where the logic for both cells is "Always qualify."  Then under the advanced tab, select the option to check for cell membership randomly (randomize all cells).  This basically randomly assigns people to one of two groups, and you can set the cell limit at 150 for each.

Once you get the assignment, you can use it anywhere you want, i.e. if quota=2, skip them past the first block of questions to the second, use it to add to a list to control block randomization, etc.
answered Nov 16, 2012 by Brian McEwan Platinum Sawtooth Software, Inc. (50,625 points)
selected Jan 2, 2013 by João Fernandes
People reading this thread, the conclusion is here:https://www.sawtoothsoftware.com/forum/2626/presenting-different-code-in-each-maxdiff-question

Again, thanks, Brian!
Brian, I've set up the quotas and it's working, I'm just missing the randomization part. The "Randomize All" option on the "Settings" tab doesn't seem to take effect. Have you any kind of advice on this?

Thank you!
Make sure you run through the survey a handful of times.  Randomization is based off of the respondent number, but it's entirely likely that if you had two quota cells, the first few respondents could be randomly assigned to the same cell.  If you run through it 5+ times and keep getting the same cell assignment, then something is probably wrong.  Also, if you run the survey and then reset it, you would be respondent 1 each time and would get the same quota allocation.
Thanks for the tips. I think it was like you pointed out, because yesterday I've made a small test survey, with 2 quota cells, each one with 2 as cell limit. After running it fully and reset it some times, the cells were always presented "in order" (1 1 2 2). Today, I've increased the cell limit to 10, and thinks changed. I've also noticed that the order is always the same, even after resetting the survey. Can you confirm me this? Taking into account what you mentioned ("Randomization is based off of the respondent number, (...)") that would make some sense (in that case, it would be more like a pseudo-randomization, wouldn't it?). Thanks again!
Yes, since randomization is based off the respondent number, if you took your survey and respondent 1 rolled a 2, then reset the survey and took it again as respondent 1, you would get a 2 again.  This allows us to recreate what respondents saw if we ever have to test or debug something.
+1 vote
This should be straight forward with randomized blocks and constructed lists.
Put each version of the survey in a block then make the constructed list based on their respondent number (odds take one version, evens take another).  Or you could use any other variable to do this.
answered Nov 15, 2012 by Mike Lodder Gold (23,395 points)
Thanks for the quick reply, Mike. The only problem with that approach is that it doesn't assure that the survey versions are randomly assigned to the respondents. However, If I randomly assign them, though, I can't assure that half of the respondents has one version, and the other one has another.
I also have a maxdiff survey with 8 versions and need to ensure they are answered equal number of times. How do I put each version of the survey in blocks in Sawtooth? (I'm using Sawtooth version 6.6.18)
If you're still interested, I'm just missing the randomizing part of it. I'm not using blocks, though, only skip logic.