In the previous section, we described the method that ACBC uses to account for how many times each non-BYO level has been used in the design and how many times each attribute has been varied from its BYO-specified level. We described that the algorithm increases the likelihood that a level or attribute will be chosen if a deficit in the "counts array" is noted. We also said that the process tends to lead to good, but not perfect balance and target frequencies. With ACBC, we are not approaching design generation in a traditional sense (we are using an adaptive procedure). Strict attribute/level balance is contrary to our aims and to the goal of reducing uncertainty in the utility estimates.
There are a few reasons why the target level frequencies or number of times each attribute has been varied from its BYO-specified level may not perfectly match the goals of the algorithm:
1. If prohibitions are in place (including the setting to avoid dominated concepts), it may be impossible to achieve target frequencies.
2. When respondents specify certain unacceptable or must-have rules, certain levels are forced in/out of the design for replacement cards, which by necessity leads to imbalance.
3. Even under the "Attribute Balance" design generation option, attributes with very few levels tend to be chosen less often (for modification from their BYO levels). That is because there are fewer levels to choose from to generate new, unique concepts. If duplicate concepts are generated, the algorithm chooses new attributes to try. This outcome is probably favorable, because better overall level balance occurs if we spend more effort investigating (relative to BYO choices) attributes with more levels than attributes with fewer levels.
4. When subsets of attributes are chosen to be modified from their BYO-specified levels and levels are chosen within the attribute that is being varied, we scan to see which one of those chosen attributes or levels has the highest representation in the current respondent's design. We then discard that choice of attribute or level in favor of the attribute or level that has the least representation in this respondent's design. This process nudges us in the direction of achieving targeted frequencies. But because only one attribute or level per draw may be discarded in favor of one under-represented attribute or level, it may not be able to "fill the holes" fast enough to achieve the desired attribute/level frequencies. But, if we were to allow the algorithm to catch up as quickly as needed in favor of perfectly matching target frequencies, patterns of attribute/level inclusion could emerge to the detriment of randomization and design efficiency.
5. If for some reason ACBC is unable to generate enough concepts following its design algorithm to satisfy the demands of the questionnaire, the attribute/level frequency targets are ignored and new concepts are generated following a randomization strategy. Note: if the experimental designer cannot find a design within the allotted time that satisfies the requested goals and constraints, it may need to relax the goals and constraints to produce a valid questionnaire for the respondent. If this occurs, a message is stored in the design log table of the database. You can download this from the Admin Module, by clicking Survey Administration | Advanced | Design Log Table. This downloads a .csv file that you can view in a program such as Excel, showing per respondent messages related to the designer process.