Block randomization and quota control

Question

Hi all!

I'm doing a survey with 2 blocks of 17 (16 random choice tasks and 1 fixed choice task to test response quality) CBC choice tasks. I would like respondents to be randomly assigned to either block 1 (Q1 - Q17) or block 2(Q18-Q34) (Not both). Additionally, I would like respondents to be assigned 50/50.

I found an answer on a related topic on this forum (March 2015), on which I based my actions. This is what I did:

In the Write questionnaire menu, I clicked the randomize button and creatded two blocks using the anchor questions:
block1 (Q1-Q17)
block2 (Q18-Q34)

Then I added a new parent list called Blocklist, in which I defined coded 1+2 as follows:

1.  1
2.  2

Then I created a constructed list called BlockconList :

ADD(BlockList)
Randomize()

Then I selected the "BlockedconList" from the "Use constructed list" dropdown menu within the randomize option.

To provide quotas, I added a quota question called QTConcept after Q34 (on the same page):

Quota1:
Value=1
Cell name = Concept1
Cell limit = 1
Qualification Logic= ListValue(BlockConList,1)=1

Quota2
Value=2
Cell Name=Concept2
Cell Limit= 2
Qualification Logic=ListValue(BlockConList,1)=2

I intentionnaly lowered the cell limits to test if the quota's worked. Unfortunately, concept 2 (so block 2) was shown more than 2 times while concept1 (block 1) was not shown to respondents.

Anyone that knows how to solve this problem?

Many thanks in advance!

Kind regards,

Luuk

Answer 1

Your constructed list randomization is done per-respondent, not taking into account how the randomization worked out for any other respondents.  So with only two respondents, there's a considerable chance that they're not going to necessarily go into two different buckets.  Statistically speaking, the distribution should grow closer and closer to a proper 50:50 as the number of respondents go up, but we can do better than that.

In your quota question, set both cells to always qualify and then set the quota question to "Least Fill."  Now respondents will be placed into whichever cell has the fewest completes, only using randomization to settle ties.

Comments

Thank you for your answer! However, this doesn't seem to work. Every time, concept 2 is shown instead of concept 1, even when the quota for concept 2 is already full. So if I complete the survey three times, concept 2 is shown 3 times, and the quotas are displayed as follows:
concept 1: 0/1 (0%)
concept 2: 2/2 (100%)

While other quotas like men are: 3/10 (30%)..

Am I missing something here? The randomization settings should still be the same, right? The "use constructed list" > "BlockconList".

I also don't get why I should tick the 'always qualify' box. Then there is no connection left with BlockconList anymore, so it would just be a random quota (i.e. Sawtooth cannot know that these quotas refer to the 2 blocks, right?)

Hope you can point me in the right direction. Many thanks in advance!

---

I wouldn't have thought to involve a constructed list at all.  Once the respondents have been placed into one of the two quota cells, you can just use skips to move respondents to whichever sections you want them to see.

If you are going to test this behavior with two test respondents, you will need to make sure that the first respondent completes the survey (without being marked as disqualified) before starting the second respondent.  Otherwise, the least fill quota may or may not place the second respondent into the same bucket.

Answer 2

Hi Luuk,

I can think of two ways around this. The first is simply to use Lighthouse Studio's "Least Filled Quota" option, where respondents who qualify for more than one bucket are assigned based on which one is "Least Filled" at the time. The algorithm we use guarantees that each bucket gets filled nearly evenly.

The second is to assign the quota based on their internal respondent number. This would split them exactly evenly when they came into the survey, but might not lead to an exact split when you're looking at completes due to random dropout/screenout variation. To do this, you'd set up the quota qualification logic IF to say they qualified for Group A if "respnum() Mod 2 = 0" and Group B if  "respnum() Mod 2 = 1". (The Mod function takes the respnum and divides it by the number specified (in this case, 2). If the remainder is odd (rem=1), then they're Group B, if it's even (rem=0), then they're Group A.)

Comments

Hi Aaron, thanks for your answer. Your first solution was the same answer I got from Zachary. However, that didnt work:

very time, concept 2 is shown instead of concept 1, even when the quota for concept 2 is already full. So if I complete the survey three times, concept 2 is shown 3 times, and the quotas are displayed as follows:
concept 1: 0/1 (0%)
concept 2: 2/2 (100%)

While other quotas like men are: 3/10 (30%)..

Am I missing something here? The randomization settings should still be the same, right? The "use constructed list" > "BlockconList".

I have a question about your second solution: How does Sawtooth know that group A and B refer to block 1 and 2? Cause if I dont specify this, it would just be a random quota, correct?

Hope you can help me out. Many thanks in advance!

---

Hi Luuk,
Have you sorted this problem? I am trying to do the same thing and would appreciate if you could share your solution.

---

I´m also trying to solve the same problem. Can anybody help me sorting this problem?


I have 4 blocks defined.  I would like respondents to be randomly assigned to 2 of 4 blocks.
1. For that I´ve constructed as described a constructed list "Parentlist" with values 1,2,3 and 4.
2. Then I arranged a blocklist with the following formula:
ADD(ParentList)
Randomize()
ListMax(2)
3. I defined the settings for the randomized blocks and used the constructed list "Blocklist".
4. Then I defined the pre scip logic with the following logic: Not(ListValue(Blocklist,1)=1)