Determining if the difference between two error values is significant

Question

I'm evaluating a number of different algorithms whose job is to predict the probability of an event occurring.

I am testing the algorithms on large-ish datasets. I measure their effectiveness using "Root Mean Squared Error", which is the square root of the ((sum of the errors) squared). The error is the difference between the predicted probability (a floating point value between 0 and 1) and the actual outcome (either 0.0 or 1.0).

So I know the RMSE, and also the number of samples that the algorithm was tested on.

The problem is that sometimes the RMSE values are quite close to each-other, and I need a way to determine whether the difference between them is just chance, or if it represents an actual difference in performance.

Ideally, for a given pair of RMSE values, I'd like to know what the probability is that one is really better than the other, so that I can use this probability as a threshold of significance.

Answer 1

The MSE is an average and hence the central limit theorem applies. So testing whether two MSEs are the same is the same as testing whether two means are equal. A difficulty compared to a standard test comparing two means is that your samples are correlated -- both come from the same events. But a difference in MSE is the same as a mean of differenced squared errors (means are linear). This suggests calculating a one-sample t-test as follows:

1. For each x compute a error e for procedure 1 and 2.
2. Compute differences of squared errors (e2^2-e1^2).
3. Compute the mean of the differences.
4. Compute the standard deviation of the differences.
5. Compute a t-statistic as mean/(sd/sqrt(n)).
6. Compare your t-statistic to a critical value or compute a p-value. For instance, reject equality at 5% confidence level if |t|>1.96.

The RMSE is a monotonic transformation of MSE so this test shouldn't give substantively different results. But be careful not to assume that MRSE is RMSE.

A bigger concern should be overfitting. Make sure to compute all your MSE statistics using data that you did not use to estimate your model.

Answer 2

You are entering into a vast and contentious area of not only computation but philosophy. Significance tests and model selection are subjects of intense disagreement between the Bayesians and the Frequentists. Triston's comment about splitting the data-set into training and verification sets would not please a Bayesian.

May I suggest that RMSE is not an appropriate score for probabilities. If the samples are independent, the proper score is the sum of the logarithms of the probabilities assigned to the actual outcomes. (If they are not independent, you have a mess on your hands.) What I am describing is scoring a "plug-in" model. Proper Bayesian modeling requires integrating over the model parameters, which is computationally extremely difficult. A Bayesian way to regulate a plug-in model is to add a penalty to the score for unlikely (large) model parameters. That's been called "weight decay."

I got started on my path of discovery reading Neural Networks for Pattern Recognition by Christopher Bishop. I used it and and Practical Optimization by Gill, et al to write software that has worked very well for me.

Answer 3

I am responding here to questions in the comments. The subject is far too big to handle in comments.

Cliff Notes version.

The types of scores we are talking about measure probabilities. (Whether that is appropriate for what you are doing is another question.) If you assume that the samples are independent, you get the "total" probability by simply multiplying all the probabilities together. But that usually results in absurdly small numbers, so equivalently, you add the logarithms of the probabilities. Bigger is better. Zero is perfect.

The ubiquitous -squared error, -x^2, where x is the model's error, comes from the (frequently unjustified) assumption that the training data comprise observations (measurements) corrupted with "Gaussian noise." If you look in Wikipedia or something at the definition of a Gaussian (aka normal) distribution, you'll find that it contains the term e^(-x^2). Take the natural logarithm of that, and voila!, -x^2. But your models do not produce most-likely "pre-noise" values for measurements. They produce probabilities directly. So the thing to do is simply to add the logarithms of the probabilities assigned to the observed events. Those observations are assumed to be noise-free. If the training data says it happened, it happened.

Your original question remains unanswered. How to tell if two models differ "significantly"? That is a vague and difficult question. It is the subject of much debate and even emotion and rancor. It's also not really the question you want answered. What you want to know is which model gives you the best expected profit, all things considered, including how much each software package costs, etc.

I'll have to break this off soon. This is not the place for a course on modeling and probability, and I am not really qualified as the professor.