Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

ACBC scale factors w/ Otter's method

Recently we auto-populated a part of the BYO section based on prior knowledge from the survey questions and common sense.
The HB estimation using the BYO task resulted in the following scale factors:

BYO: 37582.014 (!!)
Screener: 0.826
Tournament: 0.18

I'm not sure what to make of this as I do not understand the underlying math of the method. I suspect that this could indicate a problem of "non-identification" for level effects (with no observations) where auto-populated BYO preferences are uniform across the entire sample in the section. At the very least it isn't surprising that homogeneous BYO-data would generate extreme scale.

The following validation with hitrates etc didn't indicate anything was broken and we opted to keep "Otter's" factors active.

Are there any other interpretations? For example: Does the scale factor indicate that BYO section barely influenced the overall results or the opposite?

Are there other things to consider when auto-population of BYO?
Do homogeneous BYO preferences generally lead to larger scale factors?
Are there known pitfalls where such homogeneity in BYO-responses can "break" the results or unduly increase/reduce the role of BYO responses in ACBC estimation?

Thanks for your thoughts!
asked Aug 18, 2020 by alex.wendland Bronze (2,545 points)

Your solution to the original question

Please only use this to answer the original question. Otherwise please use comments.
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
Anti-spam verification:

To avoid this verification in future, please log in or register.