An equation for calculating accuracy of precipitaion forcasts?

Question

I've used absolute error for determining the accuracy of temperatures, but with precipitation, which is measured in probablity, I need a different equation. Help?

Answer 1

You've opened a can of worms with no clear answer. Statisticians have been arguing about these issues for a very long time.

Even in the simpler case, such as your example of temperatures, there is significant disagreement. People have proposed using the absolute error, as you do, but others have argued that big errors are more important than small errors so they prefer using the square of the error.

(It also happens to be the case the mathematics of using the square turns out to be simpler, and this was very important in the days before computers. Now it is less so.)

An important factor is what you are using the calculation for.

Another factor is whether the uncertainty is in the calculation or in the real world. (In the case of the weather, we assume that it will either rain or not so the uncertainty is in our calculation. However, if we are predicting the chances of a truly random experiment, we end up with probabilities even if our calculations are perfect.)

So, there is no right answer. But here is one very simple possibility:

Suppose the forecast is for X% chance of rain and it rains, then the lower X is the bigger the error. So if X is between 0 and 100, the error will be ((100 - X)^2)/10000. If it doesn't rain, let the error be (X^2)/1000. (If your predictions are probabilities in the range [0,1], use (1-X)^2 and X^2 instead.

This gives the following nice properties:

1. The error is always between 0 and 1.

2. The error is zero when the forecast is with complete confidence (0% or 100%) and is right.

3. The error is highest when the forecast is with complete confidence and wrong.

But it has also has the property that if N is 50%, the error will always be .25, even if the prediction is right.

(To see that this can be a problem, consider

1. the situation in a desert where it almost never rains. A prediction of 50% chance of rain may mean something.

2. Suppose the forecast is for 50% chance 10 days in a row and it actually rains 5 of the 10 days. Do you want to think of the forecast as being more accurate, less accurate, or the same as if it rained every day? as if it didn't rain at all?)

So this is clearly not perfect, but it is simple. I hope it is good enough.