Here is a generic criticism that has been, is, and will be levied at forecasting models like Fivethirtyeight and Votamatic:
The forecasting models fail to account for x, y, and z. But x, y, and z are fundamentally important! Therefore, we should not use the forecasting models.
Fallacious! I think we can all agree that, for various reasons, being able to predict the outcome of elections is important. We cannot just stop forecasting tomorrow. Given that, the question is a matter of what methodology we use to predict outcomes.
In that light, the above criticism fails to highlight the real question. Rather than asking “are forecasting models perfect?” we should be asking “are forecasting models better than the alternative?” In other words, we should treat what we currently have (talking heads on Fox News, CNN, and MSNBC) as the null hypothesis and the forecasting models as the alternative hypothesis. And that being the case, the forecasting models beat the hell out of political punditry.
Yet, the full criticism we often hear is this:
The forecasting models fail to account for x, y, and z. But x, y, and z are fundamentally important! Therefore, we should not use the forecasting models and instead keep pretending my inane rants actually have meaning.
Of course, the political pundit’s inane rants have absolutely no meaning. The pundit is quick to criticize what he does not like but then gives himself a free pass. However, not only does his punditry fail to account for x, y, and z, it is also completely made up horse manure, often fabricated for the sake of ratings. (Or page views…coughunskewedpollscough…)
Now, we should not take forecasting models completely off the hook. They have problems, and their creators are the first to admit that. But, as with anything else in life, we need to ask ourselves whether this devil is better than the other devil. And personally, I’d rather have Nate Silver’s pitchfork pointed at me than Joe Scarborough’s.