Model Performance Metrics

What are good values for accuracy and error?

There is inherent subjectivity in the application of the rating system and everyone will have their own unique corner cases which make modeling difficult. However, based on experiments of a user's drift over a 2-month period, the upper bound for accuracy is around 65% while the lower bounds for mean squared and absolute error are around .35. Thus, a model achieving at least 60% accuracy and MAEs (Mean Absolute Error) and MSEs (Mean Squared Error) of less than .5 is doing quite well. We want to maximize the accuracy score while minimizing the error scores. For the case of the Random Forest Classifier model, the MSE will always be greater than the MAE unless there are no predictions that are more than 1 rating point off, which is another indication the model is doing quite well. So we want the MAE and MSE values to not differ too much.