Recently I set myself the goal of loosing about five pounds of weight. In order to measure my progress one thing I’m doing is tracking the food I eat. I don’t do this everyday, but perhaps will do it for a couple of days every few weeks.

One observation I’ve made is that after a good day, where I eat nutrious meals, plenty of vegies and don’t overeat, I tend to follow it up immediately with a worse day. Similarly, I’ve noted that if I have a bad day, where I might indulge in one too many sugary sweets or have a few too many grain based meals, I will often follow it up with a day that is above average.

My first reaction to this data was to fall back on the willpower hypothesis. Bad days follow good days because willpower is impossible to sustain for large periods of time. Similarly good days follow bad days because the bad days allow the possibility to replenish will power.

Although this explanation sounds logical and at face value is extremely appealing, I now think that there might be an easier explanation, a statistical phenomenon called regression to the mean.

Let me quickly explain what that is. Regression toward the mean is the phenomenon that if a measurement is extreme on one trial then on the next trial it will tend to be closer to the average. One way to think of this is by visualizing the normal distribution (also known as the bellshape curve). If on one trial you get a result that is on the extreme edge of the curve, then it is likely that your next measurement will be closer to the center of the distribution simply because the ‘bell’ is fattest in the middle.

I think that we often try and explain our inability to sustain good habits by reverting to the willpower hypothesis. But the fact is that if we’ve been studying extremely effectively for a long period of time, or have been exercising well, or having been following our diets closely, then eventually we are bound to experience a dip in our performance. The true explanation for this isn’t a lack of power, it’s simply that we’ve been operating at the extremes of our abilities and are therefore statistically likely to experience a drop in performance.

One psychologist who has explored regression to the mean closely in his work is the author of the best selling book, Thinking, Fast and Slow, Daniel Khaneman. In one 1974 paper published in Science, Khaneman and his collaborator Amos Tversky show how failure to appreciate regression to the mean can lead to erroneous conclusions:

“The failure to recognize the import of regression can have pernicious consequences, as illustrated by the following observation. In a discussion of flight training, experienced instructors noted that praise for an exceptionally smooth landing is typically followed by a poorer landing on the next try, while harsh criticism after a rough landing is usually followed by an improvement on the next try. The instructors concluded that verbal rewards are detrimental to learning, while verbal punishments are beneficial, contrary to accepted psychological doctrine.

This conclusion is unwarranted because of the presence of regression toward the mean. As in other cases of repeated examination, an improvement will usually follow a poor performance and a deterioration will usually follow an outstanding performance, even if the instructor does not respond to the trainee’s achievement on the first attempt. Because the instructors had praised their trainees after good landings and admonished them after poor ones, they reached the erroneous and potentially harmful conclusion that punishment is more effective than reward.”

I like this example because it shows how simple statistical facts can often provide the simplest explanations for our data.

By thinking about regression to the mean I think that we can plan better for the eventual dip that follows any good period of performance. Instead of interpreting our off days as representing a lack of willpower, I think it’s more freeing just to accept them as a result of statistics. Just because we can’t sustain performance at our max for long periods of time does not mean that we can’t improve our max level of performance in the long run.

Think of performance like economic growth. On any one day we should expect the possibility of either a good performance or a bad one. This is the result mainly of random probability and can’t be predicted. If, however, we continue to improve productivity in general and focus on developing our strengths then we can expect that in the long run we will experience sustained growth.