Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Interpretation Mean Average Error (MAE)


I cannot find a suitable paper how to interpret the MAE of the prediction of my holdout tasks.

For the first holdout task the MAE is 2.79 and for the second on it is 5.99.

However, is this a good or a bad result?

Thanks for your support!
asked Jan 17 by Stefanie

1 Answer

0 votes
The larger the sample size you have, the smaller the MAE should be.  In my experience, when I'm using about n=800, I often can get MAE across a series of holdout tasks to be about 3 or a bit lower on average.  

Only 2 holdout choice tasks is probably enough to gain a cursory view of whether the model seems to be predicting OK.  But, we've found that to have strong confirmation and to be able to use holdouts to  distinguish between alternative predictive models (such as a main-effects model vs. the same model with additional interaction effects), one would typically need about 5 to 7 holdout choice tasks to have enough statistical information.

Sawtooth Software by default "suggests" 2 holdout choice tasks, mainly to suggest to the user the idea of using holdout validation; but two isn't enough to do a thorough job for strong validation for the purposes of academic-type reporting or journal articles.
answered Jan 17 by Bryan Orme Platinum Sawtooth Software, Inc. (177,115 points)
edited Jan 17 by Bryan Orme
Oh, and I should have noted that the number of concepts per CBC task has a huge effect on MAE.  The larger the number of concepts per task, the smaller the MAE becomes.  So, in my comments above I was assuming you were using about 3 to 5 concepts per choice task!
Thank you for you response. That helps a lot.

Is this also explained in any of your technical papers that I can cite?