Even distribution of multiple items in a constructed list

Question

We are working on a study with 88 list items, where we are randomly assigning 6 of these items to each respondent.  This is currently done with a constructed list, with a random assignment for each respondent.  We are finding, however, that there is not an even distribution among the 88 items.  We do expect this to even out a little bit as we collect more responses, but there are very specific targets for each list item (i.e minimum of 30 responses per item) and with this many items we are hoping to find a more precise solution than the current setup.  

We have considered using least-fill quotas to address this issue, but are struggling to find a working solution.  Since each quota's cells are mutually exclusive, we cannot assign 6 list items in a single quota.  We have also considered using 6 quotas, assigning 1 item per quota - but are not sure how we would ensure equal distribution of list items given an item may be overrepresented in one quota, but underrepresented in another.  

Any ideas on how we can find a more balanced assignment of list items on this project?

Comments

This post does something similar but only for selecting 2 items rather than 6:

https://legacy.sawtoothsoftware.com/forum/22960/least-fill-list

If you think this option would work for you, I could help expand it to support an arbitrary number of items.

---

Hi Zachary.  I think this solution would work for us, thanks for providing!  If you can help us adapt the code for 6 items, that would be greatly appreciated!

Answer 1

Terrific.  It's going to take a minute to setup because of all the quota cells involved, but I think I've got a solution for you.  It will involve creating six quota questions, each with eighty-eight cells and set to check for membership sequentially.  They should share a simple base name (e.g., "QuotaQ1," "QuotaQ2," ...) and be separated by page breaks.  The cell limits and skip to question don't really matter.

The logic for each quota cell should look like this:

Begin Unverified Perl
# Params
my $quotaBaseName = 'QuotaQ';
my $totalQuotas = 6;
my $totalCells = 88;

my $thisQuota = 1;
my $thisCell = 1;

# Run
my %previousQuota = ();
for (my $i = 1; $i < $thisQuota; $i++) {
    $previousQuota{GETVALUE($quotaBaseName . $i)} = 1;
}

my @bestCells = ();
my $bestCount = 99999;
for (my $cell = $thisCell; $cell <= $totalCells; $cell++) {
    if ($previousQuota{$cell}) {
        next;
    }
    
    my $count = 0;
    for (my $i = 1; $i <= $totalQuotas; $i++) {
        $count += QUOTACELLCOMPLETES($quotaBaseName . $i, $cell);
    }
    
    if ($count < $bestCount) {
        @bestCells = ($cell);
        $bestCount = $count;
    }
    elsif ($count == $bestCount) {
        push(@bestCells, $cell);
    }
}

return $bestCells[rand(@bestCells)] == $thisCell;
End Unverified

Lines 3-5 should be set to the base question name, the number of quota questions, and the number of cells per question, respectively; these three values should be constant between all your quotas / cells.

Lines 7 and 8 should reflect the quota and cell of the quota cell that the code is being placed into.  So the fifth cell of the second quota question will have this:

my $thisQuota = 2;
my $thisCell = 5;

The values of those quota questions should represent your six selected items in random order.  Like I said in the above link, you can use the values directly with Sawtooth Script, add them to a constructed list's instructions, etc.

Comments

Hi Zachary.  Thanks for posting this solution last week!  We tried to implement, but are unfortunately running into an issue.  When selecting next in the question immediately before the quota section, a loading screen appears but then it jumps back to the same question right before the quotas.  Do you have any ideas how to resolve this?  If easier, you can email directly if you'd like to see the programming file (j.deutsch@skimgroup.com)

---

The max respondents per cell isn't being hit, right?  If it were hit, respondents would end up being sent to whatever the "skip to" question is.  As long as those max respondents are large, that shouldn't ever happen with this code.

If you can share a copy of your .ssi demonstrating the problem to support@sawtoothsoftware.com, I'd be happy to take a look.

---

Hi Zachary.  Cell limits are set to 9999, so that shouldn't be a problem.  I will send over an email.

---

It looks like the code is actually running as expected, but the code simply isn't inefficient enough to tackle six questions and eighty-eight cells before Lighthouse Studio times out.  Sorry about that.  I believe I can offer a solution that will be faster.  It will require the quota cell logic scripts to change again, but this time in a simpler manner so you should be able to easily export and import your quota cells to do this quickly

I started by creating six pass-in fields, with a similar naming strategy to the six quota questions.  Then I placed this script into the first cell of the first quota question:

Begin Unverified Perl
# Params
my $quotaBaseName = 'QuotaQ';
my $passInBaseName = 'PassInQuota';
my $totalQuotas = 6;
my $totalCells = 88;

# Run
my @map = ();
for (my $cell = 1; $cell <= $totalCells; $cell++) {
    my $completes = 0;
    for (my $quota = 1; $quota <= $totalQuotas; $quota++) {
        $completes += QUOTACELLCOMPLETES($quotaBaseName . $quota, $cell);
    }
    if ($map[$completes]) {
        push(@{$map[$completes]}, $cell);
    }
    else {
        $map[$completes] = [$cell];
    }
}

my $add = 1;
for (my $m = 0; $m < scalar(@map) && $add <= $totalQuotas; $m++) {
    my @cells = @{$map[$m]};
    for (my $i = scalar(@cells) - 1; $i > 0; $i--) {
        my $j = int(rand($i + 1));
        @cells[$i, $j] = @cells[$j, $i];
    }
    for (my $c = 0; $c < scalar(@cells) && $add <= $totalQuotas; $c++) {
        SETVALUE($passInBaseName . $add, $cells[$c]);
        $add++;
    }
}

return GETVALUE($passInBaseName . 1) == 1;
End Unverified

Lines 3-6 have to be updated like before, now including the base name given to the pass-in fields.

For ALL other quota cells, the logic just needs to be a simple check against those pass-in fields.  For example, the fifth cell of the second quota question will have this logic:

PassInQuota2 = 5

Does that make sense?  This solution should be much faster than my earlier code.