Orthogonality of experiment

Question

Dear Experts,

Would you please let me know if the following understanding is correct in context of CBC:
Orthogonality means each pair of levels appears equally across all pairs of attributes within the design. Therefore, orthogonality is not only relevant for alternative design. It is also relevant for choice sets design.

Moreover, according to definition of orthogonality a full factorial design is perfectly orthogonal that can capture interaction also. Fractional factorial design can also be orthogonal and at the same time measure some interactions. Therefore, orthogonality in general does not mean that interaction are ignored. In fact OMEP is a specific kind of orthogonal design that measure only main effect not interaction. Correct?

If this is correct, how do you interpret definition of orthogonality in Orme's book glossary
"A statistical term that, when applied to conjoint analysis experimental
designs, refers to experiments in which the attributes are uncorrelated
across product concepts. In more technical terms the column in the design matrix have zero correlation between levels of different attributes"

According to this definition full factorial design that inherently includes interaction terms should not be orthogonal. I think above definition should be related to orthogonal main effect plan (OMEP) that is an specific kind of orthogonal design. Correct?
Many Thanks for your support

Answer 1

MY RESPONSES IN CAPS DIRECTLY FOLLOWING YOUR COMMENTS/QUESTIONS.  --BRYAN

Orthogonality means each pair of levels appears equally across all pairs of attributes within the design.

YES, THAT IS TRUE ONLY AS LONG AS EACH ATTRIBUTE HAS THE SAME NUMBER OF LEVELS.  WHEN ATTRIBUTES HAVE DIFFERENT NUMBERS OF LEVELS, ORTHOGONAL DESIGNS OFTEN HAVE TO OVERREPRESENT SOME LEVELS WITHIN AN ATTRIBUTE COMPARED TO OTHER LEVELS WITHIN THE SAME ATTRIBUTE.  THUS, EACH PAIRING OF LEVELS WOULD NOT NECESSARILY BE REPRESENTED AN EQUAL NUMBER OF TIMES.  RATHER, ORTHOGONAL DESIGNS WOULD SHOW PAIRINGS OF LEVELS PROPORTIONALLY OFTEN.

Therefore, orthogonality is not only relevant for alternative design. It is also relevant for choice sets design.  YES, ORTHOGONALITY IS RELEVANT TO TRADITIONAL(CARD-SORT) CONJOINT DESIGNS AS WELL AS DISCRETE CHOICE EXPERIMENTS (ALSO KNOWN AS CBC).

Moreover, according to definition of orthogonality a full factorial design is perfectly orthogonal that can capture interaction also. TRUE, IT’S ORTHOGONAL.  TRUE, FULL FACTORIAL CAN MEASURE ALL INTERACTION EFFECTS.

Fractional factorial design can also be orthogonal and at the same time measure some interactions. TRUE.  A SPECIFIC SUBSET OF FRACTIONAL FACTORIAL DESIGNS CAN MEASURE CERTAIN INTERACTION EFFECTS.

Therefore, orthogonality in general does not mean that interaction are ignored.   NOR DOES IT MEAN THAT THEY WILL BE CAPTURED.  ORTHOGONALITY AND THE ABILITY OF A DESIGN TO MEASURE INTERACTIONS ARE SEPARATE CONCEPTS. OFTEN, ORTHOGONAL FRACTIONAL FACTORIAL DESIGNS CANNOT MEASURE ALL POTENTIAL INTERACTIONS.  

In fact OMEP is a specific kind of orthogonal design that measure only main effect not interaction. Correct?  YES.

If this is correct, how do you interpret definition of orthogonality in Orme's book glossary "A statistical term that, when applied to conjoint analysis experimental designs, refers to experiments in which the attributes are uncorrelated across product concepts. In more technical terms the columns in the design matrix have zero correlation between levels of different attributes"

According to this definition full factorial design that inherently includes interaction terms should not be orthogonal. I’M SORRY, I’M CONFUSED WITH YOUR LOGIC.  A FULL FACTORIAL DESIGN INDEED IS ORTHOGONAL IN TERMS OF ALL POTENTIAL MAIN EFFECTS AND ALL POTENTIAL INTERACTION TERMS THAT THE RESEARCHER WOULD WANT TO INCLUDE IN THE MODEL.

I think above definition should be related to orthogonal main effect plan (OMEP) that is an specific kind of orthogonal design. Correct?

NO, THE DEFINITION IN THE ORME BOOK SHOULD ALSO APPLY TO PLANS WHERE MAIN EFFECTS ARE ORTHOGONAL AND INTERACTION EFFECTS ARE ALSO ORTHOGONAL – MEANING THAT THE CORRELATIONS IN THE DESIGN MATRIX BETWEEN DIFFERENT ATTRIBUTES BOTH FOR MAIN EFFECTS AND INTERACTION EFFECTS ARE ORTHOGONAL.  --BRYAN

Comments

Hi Bryan,
All are clear now except definition of orthogonality in book.The reason should be my very limited knowledge in statistic. When you say "Orthogonality  refers to experiments in which the attributes are uncorrelated across product concepts" what does uncorrelated mean? I thought it means there is no interaction.
Moreover, statistical design of CBC has two steps;1.alternative design 2.choice sets design. Assume that I have a fractional factorial design (for alternative design stage). How can I know that which interactions can be measured by this? Does the method I am going to choose for design of choice sets (second stage)e.g random or BIBD also impact ability to measure interactions? After performing HB analysis how can I know that these interaction are really existed in decision rule of respondent or nor? How are the magnitude of interactions  expressed in the result of analysis?

---

Ah, I see your confusion.  Orthogonality of the experimental design means that the patterns of attribute levels SHOWN to respondents are balanced and without correlation between attributes.  For example, if you have 2 attributes each with 3 levels, there is a 9-cell contingency table reporting how many times each pair of levels between those two attributes was shown.  Those counts in the 9 cells should be equal if the design is orthogonal and the number of levels per attribute is balanced in the design.

The other issue of concern has to deal with respondent PREFERENCES for pairings of those two attributes.  That's what we refer to as an interaction effect.  If the preferences for the pairings of the two attribute levels represented by those 9 cells in the contingency table can be near-perfectly predicted by cross-multiplying the main effects (the preferences "on the margins"), then we don't have a significant interaction effect.

In the old days, when people used to obtain their fractional factorial arrays for a series of cards (product concepts) from design catalogs, each design was described in the catalog in terms of which effects it could capture.  Some designs were OMEP (main effects only); other designs could support main effects and just one or a few of the interaction terms.  These designs would report which interaction effects were aliased, meaning confounded and not estimable.

With our CBC software, we take a more pragmatic approach for design generation and design testing.  Assuming you use the defaults in the software: 300 versions (blocks) of the design, and multiple tasks per person, where each task shows multiple concepts, and no prohibitions between attributes...then our default design strategy (balanced overlap) should support all main effects and first-order interaction effects (meaning, the interactions between attributes taken 2-at-a-time).  The user can test this by using the Test Design facility in our software.  The Test Design facility generates random-responding robotic respondents of the sample size that the user requests.  We run an aggregate logit analysis and report the standard errors.  In our experience, standard errors of .05 or less for main effects are good in practice.  Standard errors of .1 or less for interaction effects are also good.  So, those are practical targets.

In the Test Design report you can look at the specific interaction effects you are interested in to see if you are obtaining standard errors (precision) of .1 or less.

If you are constructing your own experimental designs not using our software, then indeed the way that you assemble the different profiles (concepts) into sets of concepts matters to how well interaction effects can be estimated.  The concept of "minimal overlap" comes into play.  Minimal overlap (repeating the same levels as few times as possible within the same choice set) maximizes design efficiency for main effects, but it sacrifices design efficiency with respect to interaction effects.  Therefore, a good design strategy is to bring the profiles together into sets such that it results in a modest degree of level overlap, which sacrifices only a tiny bit of efficiency of main effects for more substantial gains in the precision of interaction effects.

After using HB, there are multiple ways to look at the strength of interaction terms.  The classical way is to look at the trail of alpha draws (the population estimates at each iteration in the alpha file that is reported) after convergence is assumed (typically throw away the first 5K or 10K iterations).  For interaction terms that used zero-centered design matrix coding (effects coding, which our software does by default), you are going to want to look for certain interaction terms where 95% or more of the alpha draws fall on one side of zero or the other.

A more practical test that focuses more on how much better prediction one gets in HB from using interaction effects vs. not can be produced by our "CBC/HB Model Explorer" software.  This is a free add-on tool for those licensed to use our HB software (if you have a CBC license, you have access to this add-on tool).  The Model Explorer does multiple replicate HB runs with jack-knife holdout task validation, reporting the average hit rate over the replications.

---

Thanks Bryan,
How can I check orthogonality in the context of choice sets design?
​What does orthogonality in the context of choice sets design mean? I know that perfect orthogonality means each attribute level appears an equal number of times in combination with all other attribute levels in the whole choice sets​. My question is, whether this combination is within a product concept or across product concepts in a choice set? For example assume I want to count number of times limo appears with colour levels. I start with below example (that shows one choice set of three products)  Should I count only 1 for limo and red (within one product concept) or should I count 3 for limo, red, yellow and green (across product concepts in a choice set)?

Profile1 Limo, red, 100HP
Profile12 Sedan, yellow, 120HP
Profile3 Combi, Green, 300HP

Regards, Reza