Analysis with Tiny Sample Sizes

Generally, sample sizes in the multiple hundreds are used for conjoint-analysis surveys.  But, there are instances in which the market consists of just a few customers, and the researcher wants to gain insights regarding these few people.  We have heard of entire conjoint studies designed for the purposes of understanding the tradeoffs of a single customer.

One of the advantages of ACBC is the ability to stabilize part-worth utility estimates with smaller sample sizes than typically are used with traditional CBC.  With ACBC, more information is collected at the individual-level, and respondents seem willing to spend even greater time in the adaptive CBC survey than the non-adaptive CBC.  Respondents have reported equal or greater engagement/interest/satisfaction after taking ACBC interviews than with much shorter CBC interviews.

Our research suggests that purely individual-level analysis (via monotone regression) is feasible for ACBC, though when hundreds of respondents are available in the dataset, HB estimation consistently provides more accurate results.  (Purely individual-level analysis means that no information is "borrowed" from other respondents in the sample, so results could be computed even for n=1.)  Despite the general superiority of HB analysis over monotone regression with ACBC data, we've wondered at what sample size one might start to favor monotone regression over HB.  Is a sample size of n=30 sufficient for HB, such that the population estimates of means and covariances can be leveraged to produce more accurate part-worth utilities than monotone regression?  What about n=20?

Certainly, answers to these questions hinge upon how many attributes and levels are in the study, and whether the questionnaire has been made relatively short or long (as the researcher has flexibility in defining how many total concepts are generated in the near-neighbor pool for the Screening section).  Elsewhere in this documentation, we have suggested that you review the designs generated for a few test respondents to ensure that each (non-BYO selected) level taken forward to the ACBC questionnaire appears at least two or three times if using HB, and potentially 4 or more times if purely individual-level estimation is required due to very small sample sizes.

We re-analyzed an existing dataset (the "laptop" dataset that is described in our published ACBC papers) to investigate the use of HB with even tiny sample sizes.  The dataset contained 286 respondents.  In addition to the ACBC interview, each respondent completed four holdout choice tasks that were formatted like CBC tasks.  We can use those to compute hit rates (how well the part-worth utilities for each person can predict which product concept the respondent would choose in the holdout sets).

Using purely individual-level analysis (monotone regression), we obtained a 52.8% hit rate.  Pooling all respondents, we achieved 56.5% hit rate under HB.  This confirms other findings, where HB has proven more successful than monotone regression when analyzing hundreds of respondents in ACBC.  Default settings were used for both HB and monotone regression, with the exception that price was constrained negative.

Next, we split the sample into random partitions, estimating HB separately within each subset of respondents.  We re-computed the total hit rate (across all 286 respondents) under each condition.  For example, 286 respondents can be divided equally into two groups of 143.  But, dividing the sample into four groups results in four subsamples of 72, 72, 71, and 71 respondents.  In the table below, when the number of respondents in the sub-sample didn't divide evenly into 286, we report the largest sample size represented (for example, 72, when sub-samples of 71 and 72 were used).

As expected, the hit rate for HB estimation declines as less information is available to "borrow" from respondents within the same dataset to estimate part-worth utilities.  Even with sample sizes as small as n=9, HB's hit rate is 53.5% compared to 52.8% for monotone regression.  So, it appears that HB can be effective with ACBC data for even very tiny sample sizes as low as n=9.

We should caution that the absolute scale of the parameters increases as smaller sample sizes are used with HB.  The Parameter Root Mean Squared (typical size of the point estimates of the part-worth utilities) was 2.13 when all 286 respondents were included in a single HB run, but was 2.95 when the respondents were estimated in sub-samples of n=9.  This leads us to reemphasize that one should tune the market simulator via the Exponent to achieve proper scaling of simulated shares, as the scaling resulting from an HB estimation of ACBC data is affected by sample size and often needs tuning down (below 1.0) to best predict market behavior.