Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

the covariance matrix has a determinant of zero


We will conduct a CBC study with two attributes, model and price.
Model has 27 (yes 27!) levels, and price has 7.
We already advised the client that this is too much, but it is what it is.
The last project we did for the same client has 25 levels for model and 5 for price, we did it with 18 tasks showing 7 concepts each.
Now Im testing the design for this one, and with the same number of tasks the "strenght" of the design is sbout 60% of that of the previous project.
I went and increased the number of tasks to 22 and got this message:

"The strength of this design could not be calculated (the covariance matrix has a determinant of zero)"

I also tried with 27 and 20 tasks, all provide me the same message.

1. What does this message mean?
2. By any chance is this an indication that we definitely should not go over 18 tasks?

Thank you,

asked May 11 by vgalimbertti (235 points)

1 Answer

0 votes
Are you allowing the designer to generate multiple versions (blocks) for this design?  Or, are you trying to do it with just 1 version?  With multiple versions, you should be able to find an estimable and efficient design.  A good rule of thumb is across the versions of the design to have at least 80 unique choice tasks.  But, this is just a rough guideline and using the test design capability will show you how things are turning out for a given number of tasks, versions of the tasks (blocks), and number of respondents.
answered May 11 by Bryan Orme Platinum Sawtooth Software, Inc. (198,715 points)

Im using 28 questionnaire versions
complete enumeration
300 respondents

I tried increasing to 350 respondent and got still the same message.
I tried using more versions (45), and still the same message.
Up to 19 random tasks it gets a "design strenght", which is 51.
But 20 or more tasks it wont calculate it.
Do you have a bunch of attribute prohibitions that are hindering the design efficiency?  How many concepts are you showing per task?
No prohibitions.
7 concepts per task.
I'm not seeing your troubles.  I just tried a CBC design within our Lighthouse Studio software with the following specifications:

Attribute 1: 27 levels
Attribute 2: 7 levels
Number of Tasks per version: 12
Number of concepts per task: 7, plus a traditional None concept
Number of versions: 28
Design method (either complete enumeration or balanced overlap...I tried both)

For the design test, I told the software to generate 300 respondents answering randomly.  I get no errors about defficient designs or covariance matrix with a determinant of zero.

Perhaps you can call our tech support consultants at +1 801 477 4700 to discuss?  I'm just not seeing your troubles...so maybe I'm not replicating what you're doing correctly.
Hi, thank you for trying to replicate the results Im getting.
I think I found something out.
I deleted the CBC exercise and created another one.
Tried running the test with 22 tasks and it worked. But then I realized that the strength provided was much higher than what I was getting.
Then I turned on two-way interaction effects in the test (which is something important for this study), and now again, I got the same error message.
Ah, that explains your problem.  You're trying to test efficiency for interaction effects!

Very unlikely that you need 2-way interaction effects or that you could even hope that they would be estimable with reasonable sample sizes given a 27-level attribute times a 7-level attribute.  Interaction effects are not very often done for CBC nowadays now that HB estimation (individual-level) estimation is used.  

If you really did need an interaction effect between 27 models x 7 prices, you would likely need to economize the model by fitting the 7 price levels with a single price slope, leading to (27-1)(1) interaction terms rather than (27-1)(7-1) terms!  But, then you are assuming linearity for the price attribute, which may not be something you'd be willing to do.

If you really were going to estimate interaction effects between a 27-level attribute and a 7-level attribute with a decent degree of precision, you'd need more versions of the questionnaire and HUGE sample sizes...well over perhaps 2000 or 3000 respondents.
BTW, there are lots of fancy workarounds that advanced CBC modelers do if they want to have their cake and eat it too: potential interaction effects for 27 SKUS (or in your case, models).  

I mentioned one, which is to estimate price as a linear terms rather than as a standard effects-coded (dummy-coded) model.

Other fancier things that involve extra coding and specialized model building include grouping SKUs (Models in your case) into a handful of tiers and estimating price-specific slopes for that reduced number of tiers.

I just wanted full disclosure related to your issues, as these are common concerns that advanced CBC analysts have been able to tackle with some extra effort.

Thank you for the explanation.
So, since HB with individual level estimation is basically all I ever use, I will not include 2-way interactions in my design tests in the future.

Thank you again!