Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Workflow for Study Whose Design Generation Takes a Day?

Q1: Is there a best practices method for working with a study whose design takes a full day (in my case, 16hrs) to generate?

Q2: For the study below, is 16hrs on a single machine a typical amount of time? Why is this taking so long?

I have a CBC study:

* Tasks: 20 random tasks
* Concepts: 4 concepts per task + an outside option
* Attributes: 10 attributes per concept
* Levels: About 6 levels per attribute (well-balanced across attributes)
* Versions: 30 survey versions
* Random task generation method: Balanced Overlap
* Everything is standard/default

It takes 16hrs to generate the design for this study! (on a single Intel i7, 2 cores, 16GB RAM machine using SSI Web 8.4.8). Am I doing something wrong? Does this surprise anyone?

My workflow approach: I basically worked on the study, where I reduced the number of tasks to 3, used "shortcut" for random task generation method, and cut the number of survey versions down to 1. In this way I was able to generate designs quickly and iterate. Now that I'm comfortable with what I've created and am ready to host my survey, I'd like to generate the full design, as above. But this takes 16hrs. I'm worried that if I've made a mistake, then I have go back and take another 16hrs to run this.
asked Nov 9, 2017 by ertingen Bronze (615 points)

1 Answer

+1 vote
The Shortcut design approach (rather than Balanced Overlap) should work in seconds, as you've seen.  And, you typically only lose 3 to 8% of efficiency relative to the Complete Enumeration or Balanced Overlap approach.

But, before jumping right into Shortcut method, you should think about whether you want some level overlap (repeated levels within the same attribute).  If that's the case, then you'll need to think about a different approach: like Balanced Overlap.

With 10 attributes, it doesn't surprise me that it takes this long.  Like Complete Enumeration, balanced overlap slows down considerably with the increase in attributes.  It has to look at all full factorial possible product combinations in filling out each concept of the design.  If you calculate the total possible number of concepts that could be generated by your design, you see what Balanced Overlap is trying to search over to select each concept in your design!

30 versions is fine as a simplification.  300 versions is the default, but typically is overkill.
answered Nov 9, 2017 by Bryan Orme Platinum Sawtooth Software, Inc. (175,315 points)
I did check and Balanced Overlap's runtime basically scales linearly in the number of survey versions (e.g., it took 16hrs to run for 30 survey versions and would take about 180hrs to run for 300 survey versions). So I guess 300 survey versions with Balanced Overlap is not an option. I wonder if we can say anything about the performance of Random with, say, 1000 versions vs Balanced Overlap with 30 versions. I'm guessing that with 1000 versions, Random performs very closely to Balanced Overlap AND doesn't take long