Does ASD setup lead to tiny non-sign parameters?

Question

I generated an experiment where some concepts are fixed and not described by attributes of interest for the NEW concept - let's say NEW is a completely new mode of transportation for which there is no overlap in attributes with the other concepts.
I have 9 random screens where each shows between 1 and 3 NEW concepts. The older concepts are each shown only ~10% of the times and 80% of all concepts are NEW.
The NEW concept is described by 10 attributes. So these attributes are conditional on attribute "concept" being level "NEW".

In total across all screens 725 respondents chose:
old concept 1 8%
old concept 2 14%
NEW concept 73%
none  5%

Estimation now yields the concerning finding that the majority of effects for the attributes are very small and non-significant. The only exception is the primary "old/new concept" attribute.

Could someone from Sawtooth please have a look at my setup and let me know what you think?

Answer 1

Alex,

Alt-spec CBC designs should lead to significant parameters.  I'm assuming you're using aggregate logit to test the significance of parameters?  Perhaps using the t-test for each parameter?  As these are typically effects-coded, the parameters are zero-centered within attributes and thus the t-test considers the null hypothesis that the utility equal to zero.

How large are the aggregate logit standard errors for the principal attribute and the alternative-specific attributes for your n=725 choice experiment?  The rule of thumb is that the standard errors for levels of the principal attribute should be about 0.05 or less.  The standard errors for levels of the alt-spec attributes should be about 0.10 or less.  Did you design achieve at least this degree of precision?

Comments

Hi Bryan

Happy New Year and thanks for the quick response.
Please find below the pre-field counts and logit test. These looked good to me. At the very bottom I have copied the parameters from the logit run on the respondent data.
We just never saw an experiment where almost none of the attributes have an impact against what intuition, reasoning and prior qualitative research suggested.
It seems like data is collected correctly as the effects of the first (primary) attribute and price (5 levels, second to last) are meaningful and sensible.

An idea I'm trying to validate is that the choice is, in fact, "two-stage" and nested modelling/estimation should be applied.
To investigate I'm trying to transform the data into MBC format. There I would first limit the tasks to all these where only 3 NEW concepts were compared (no old concepts on the screen, attribute 1 always level 3) to see if the attributes have effect when the presumed 1st stage choice is not modeled.
Then with a full set of tasks I could imagine modeling the choice between old/NEW/no concept and then model the 2nd stage.
However, I'm not sure how to do the latter as the NEW concept may show in different positions, 1, 2 or 3 per screen.

I'm grateful for additional ideas and comments!
I'm also happy to share the survey data and use some of the consulting hours included in the subscription.


#########################
CBC Design: Preliminary Counting Test
Copyright Sawtooth Software
1/5/2020 4:54:25 PM
Using an imported design.
Based on 100 version(s).
Includes 900 total choice tasks (9 per version
Each choice task includes 3 concepts and 10 at

Legacy (OLS) Efficiency Test
----------------------------------------------
Att/Lev   Freq.   Actual    Ideal      Effic.
 1    1       315 (this level has been deleted
 1    2       330   0.0891   0.0720   0.6543  
 1    3      2055   0.1360   0.0720   0.2807  

 2    1       507 (this level has been deleted
 2    2       524   0.0707   0.0638   0.8155  
 2    3       501   0.0712   0.0638   0.8028  
 2    4       523   0.0704   0.0638   0.8225  

 3    1       413 (this level has been deleted
 3    2       405   0.0805   0.0724   0.8095  
 3    3       415   0.0798   0.0724   0.8241  
 3    4       398   0.0812   0.0724   0.7944  
 3    5       424   0.0797   0.0724   0.8262  

 4    1       521 (this level has been deleted
 4    2       517   0.0704   0.0638   0.8200  
 4    3       492   0.0713   0.0638   0.7993  
 4    4       525   0.0697   0.0638   0.8378  

 5    1       692 (this level has been deleted
 5    2       684   0.0578   0.0528   0.8357  
 5    3       679   0.0582   0.0528   0.8236  

 6    1       503 (this level has been deleted
 6    2       532   0.0698   0.0634   0.8252  
 6    3       523   0.0711   0.0634   0.7958  
 6    4       497   0.0712   0.0634   0.7932  

 7    1      1029 (this level has been deleted
 7    2      1026   0.0490   0.0444   0.8197  

 8    1       687 (this level has been deleted
 8    2       668   0.0584   0.0530   0.8260  
 8    3       700   0.0578   0.0530   0.8423  

 9    1       430 (this level has been deleted
 9    2       405   0.0803   0.0727   0.8205  
 9    3       405   0.0804   0.0727   0.8172  
 9    4       405   0.0788   0.0727   0.8504  
 9    5       410   0.0789   0.0727   0.8491  

10    1      1034 (this level has been deleted
10    2      1021   0.0485   0.0439   0.8176  


################################
Logit Efficiency Test Using Simulated Data
-------------------------------------------------------------
Main Effects: 1 2 3 4 5 6 7 8 9 10
Build includes 600 respondents.

Total number of choices in each response category:
Category   Number  Percent
-----------------------------------------------------
       1    1354   25.07%
       2    1327   24.57%
       3    1364   25.26%
       4    1355   25.09%

There are 5400 expanded tasks in total, or an average of 9.0 tasks per respondent.


Iter    1  Log-likelihood = -7471.83510  Chi Sq = 28.30890  RLH = 0.25066
Iter    2  Log-likelihood = -7471.27175  Chi Sq = 29.43561  RLH = 0.25068
Iter    3  Log-likelihood = -7471.24881  Chi Sq = 29.48148  RLH = 0.25068
Iter    4  Log-likelihood = -7471.24789  Chi Sq = 29.48332  RLH = 0.25068
Iter    5  Log-likelihood = -7471.24785  Chi Sq = 29.48340  RLH = 0.25068
Iter    6  Log-likelihood = -7471.24785  Chi Sq = 29.48340  RLH = 0.25068
*Converged
          Std Err    Attri
  1       0.03825    1 1
  2       0.03772    1 2
  3       0.02803    1 3

  4       0.03493    2 1
  5       0.03373    2 2
  6       0.03478    2 3
  7       0.03361    2 4

  8       0.04031    3 1
  9       0.04016    3 2
 10       0.03908    3 3
 11       0.04095    3 4
 12       0.03984    3 5

 13       0.03401    4 1
 14       0.03398    4 2
 15       0.03489    4 3
 16       0.03425    4 4

 17       0.02683    5 1
 18       0.02673    5 2
 19       0.02721    5 3

 20       0.03471    6 1
 21       0.03330    6 2
 22       0.03410    6 3
 23       0.03471    6 4

 24       0.01950    7 1
 25       0.01950    7 2

 26       0.02703    8 1
 27       0.02713    8 2
 28       0.02674    8 3

 29       0.03914    9 1
 30       0.04056    9 2
 31       0.04055    9 3
 32       0.03987    9 4
 33       0.04016    9 5

 34       0.01931    10  
 35       0.01931    10  

 36       0.03620    NONE


##############################
Number of Respondents    746               
                   
Iteration    Chi-Square    Fit Statistic (RLH)           
1    2025.4653    0.2907           
2    2292.40442    0.29654           
3    2311.96947    0.29697           
4    2312.1778    0.29698           
5    2312.17783    0.29698           
6    2312.17783    0.29698           
*Converged after 0.78 seconds.                   
                   
Log-likelihood for this model    -8151.49142               
Log-likelihood for null model    -9307.58034               
Difference    1156.08892               
                   
Percent Certainty    12.42094               
Akaike Info Criterion    16354.98285               
Consistent Akaike Info Criterion    16558.09355               
Bayesian Information Criterion    16532.09355               
Adjusted Bayesian Info Criterion    16449.4719               
Chi-Square    2312.17783               
Relative Chi-Square    88.92992               
                   
                   
Att/Lev     Effect        Std Error        t Ratio
 1    1     -0.32779        0.03529        -9.28909
 1    2     0.06384        0.03236        1.97246
 1    3     0.26395        0.02457        10.74472
                   
 2    1     0.049        0.02738        1.78928
 2    2     -0.00193        0.02747        -0.0704
 2    3     -0.04271        0.028        -1.52519
 2    4     -0.00436        0.02718        -0.16027
                   
 3    1     -0.0151        0.03225        -0.46825
 3    2     -0.0254        0.03246        -0.78236
 3    3     -0.05578        0.03245        -1.71894
 3    4     -0.00492        0.03304        -0.14896
 3    5     0.1012        0.03175        3.18777
                   
 4    1     -0.02025        0.02736        -0.74013
 4    2     0.01739        0.02752        0.63184
 4    3     0.04871        0.02808        1.73445
 4    4     -0.04585        0.02729        -1.67989
                   
 5    1     -0.00879        0.02133        -0.41222
 5    2     0.02058        0.02117        0.97256
 5    3     -0.01179        0.02154        -0.54744
                   
 6    1     -0.01054        0.02758        -0.38221
 6    2     -0.0632        0.02727        -2.31735
 6    3     0.03501        0.02727        1.2839
 6    4     0.03874        0.02769        1.39877
                   
 7    1     -0.02252        0.01562        -1.44136
 7    2     0.02252        0.01562        1.44136
                   
 8    1     -0.04273        0.02142        -1.99544
 8    2     0.00991        0.02153        0.46042
 8    3     0.03282        0.0212        1.54828
                   
 9    1     0.11342        0.03103        3.65469
 9    2     0.0146        0.03279        0.44514
 9    3     0.02115        0.03289        0.64301
 9    4     -0.04398        0.03254        -1.35133
 9    5     -0.10519        0.03286        -3.20123
                   
10    1     -0.00227        0.01541        -0.14753
10    2     0.00227        0.01541        0.14753
                   
NONE   -1.80911           0.06225        -29.06272

---

Hmmm, indeed it's strange that attributes 2, 4, 5, 7, and 10 aren't significant from the aggregate logit report at the 95% confidence level.

But, is it possible that these attributes are not ordered attributes and different respondents think different levels are best (and that their preferences cancel out in aggregate?).  That could be the explanation.

---

This is possible and we are currently investigating via LC and by comparing to self-reported relevance of attributes and circumstances to see if utilities match.
Can I compare the RLH of a aggr logit (~0.3) and a HB (~0.6) run?
Are there any other indicators / metrics that could support the heterogeneity hypothesis?

Interestingly, when I filter down to use only tasks that had 3 NEW concepts and estimate based on these (i.e., skip the presumed 1st stage choice for attribute 1) I get these estimates (via MBC so level 1 utility is 0)

         Effect          Std Err            t Ratio
2    2    0.29752    0.05954    4.9973
2    3    0.38499    0.05324    7.23162
2    4    0.3559              0.0573    6.21099

3    2    0.29425    0.06203    4.74384
3    3    0.23107    0.06739    3.42897
3    4    0.3339            0.07023    4.75448
3    5    0.33632    0.06512    5.16435

4    2    0.24258    0.05989    4.05016
4    3    0.29247    0.05974    4.89572
4    4    0.19837    0.05728    3.4632

5    2    0.14984    0.04889    3.06461
5    3    0.09453    0.0484           1.95334

6    2    0.25018    0.05862    4.26813
6    3    0.27386    0.05706    4.79976
6    4    0.29523    0.05917    4.98997

7    2    0.13317      0.0416    3.20115

8    2    0.17918    0.05039    3.55583
8    3    0.21797    0.04759    4.57978

9    2    0.28249    0.06246    4.52294
9    3    0.11356    0.06815    1.66632
9    4    0.11478    0.06352    1.80696
9    5    0.12149    0.06627    1.83336

10    2    0.14974    0.03916    3.82324