Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Number of levels appearance for HB estimation (rule-of-thumb)

I noted a rule-of-thumb in my notes which said that each level should appear 6 times or more for HB estimation.

This translated mathematically into:

 (#concept * #task) /# levels in largest attribute >=6

Unfortunately I cannot find this rule-of-thumb anywhere anymore and therefore I cannot add any citation to my paper.

Does anybody knows this rule of thumb, or an equivalent rule of thumb giving a minimum number of tasks needed in HB? And if so, does anybody have a paper to advise me?
asked Aug 29, 2019 by Jean Mansuy

1 Answer

0 votes

At Sawtooth Software, we offer rules of thumb to researchers to help them think about reasonable setups for their CBC questionnaires.  However, each study is different.  How many attributes the respondent must trade off, the length of the text involved with the levels, respondent heterogeneity, how many concepts are shown per task, the subject matter and engagement of the respondents.  These all matter to accurate HB estimation for CBC studies.

Of course, with HB estimation, we're concerned that we have adequate information to estimate each attribute level.  So, rules of thumb based on aggregate logit (pooled logit) don't seem to address your question.

John Howell, formerly of Sawtooth Software, provided the rule of thumb for CBC that each attribute level should appear about 6 times per respondent.  It seems like a useful rule of thumb, but I haven't seen it formally tested.  As far as my memory serves, I think the only place we've written this 6x rule of thumb is in our 2- and 3-day CBC training workshops.  That's a PowerPoint deck.

In terms of empirical evidence, Hoogerbrugge and van der Wagt did a meta study of 35 CBC studies, examining the hit rate to holdout concepts as the tasks increase.  That paper is entitled, “How Many Choice Tasks Should We Ask?” Hoogerbrugge and Van Der Wagt, 2006 Sawtooth Software Conference.  You may download it from:

However, their study did not examine how many times each level for complex attributes appeared.
answered Aug 29, 2019 by Bryan Orme Platinum Sawtooth Software, Inc. (175,415 points)
Dear Bryan,

I thank you very much for your very helpful answer.
Based on your answer I think I will focus my argumentation on the total number of questions.

Best regards,