I once knew a teenager whose mother questioned whether she was getting the most out of her education. Given that Lanna’s recent activities had had included an unscheduled road trip with a band called Blissful Discharge, you might be inclined to agree, but anyway, after Mom had enumerated some skills that might have helped L lead a more fulfilling life, Lanna said “But I can make toast!”
Probably neither of them realized that the ability operate a toaster would qualify Lanna for a career as a political pundit, and would actually make her one of the better prognosticators on The McLaughlin Group. As Nate Silver demonstrates in The Signal and the Noise, the predictions offered very week by the guests and host of that show are correct less than half the time, meaning that their accuracy could be improved by a person using a toaster (basing the prediction on which slice of bread gets thrown higher when ejected). John McLaughlin, in fact, has much in common with a toaster, since both generate more heat than light, but most toasters have better manners.
Silver (or more likely his long-suffering research assistant) sorted through years’ worth of predictions to reach his conclusion, but as a particularly fun example he cites the episode where, on the eve of the 2008 election, only one of the appliances, er, panelists managed to predict that Obama would defeat McCain. Now, I don’t really think that Eleanor Clift and Fred Barnes are dumber than toasters*; the big problem here is not so much stupidity as dishonesty. The panelists are not really trying to say things that are true, only things that are provocative and favor their side.
And there is no penalty: no-one ever calls them on their dumbass predictions, and the viewers don’t seem to mind a weekly dose of cynical garbage. It’s not just the one show, of course. The aptly-named Dick Morris has achieved decades-long fame and success without, to my knowledge, ever made a statement based on reality. So why do these things piss me off? Can’t I just change the channel? I can, and do—but the culture of pull-it-out-of-your-ass punditry debases honest analysis in the same that real science is debased when Philip Morris sponsors a study showing that cigarettes are good for you or the Family Research Council funds a study on the sociological evils of gay people. The effect is to erode people’s respect for genuine inquiry, which dog knows is pretty limited to begin with.
You could see this in the way that people reacted to Silver’s work at fivethirtyeight.com; many of the objections started from the assumption that he had devised his method in order to support his prejudices, and the idea that he was mainly trying to get it right seemed more than people could grasp.
Of course, inaccuracy is not always the result of dishonesty. I don’t think anyone really thought McCain was about to win in 2008—those who said so were probably fibbing—but last year the shock in the Romney camp seemed genuine. They found it so implausible that, after four years, Americans wer still buying Obama’s snake oil that they got in the habit of ‘unskewing’ any poll data that they didn’t like. Eventually, they even trained their own pollsters to produce data that confirmed their preconceptions, and everyone danced in a happy circle until one sad November night. Silver’s book came out too early to use this example, but it’s a very common pattern, and he discusses in detail how the ratings agencies, such as S&P, managed to persuade themselves that mortgage-backed securities were safer than US treasuries. You start with a cash cow (rating all the new derivatives was a very lucrative business), then you make sure to include only data from the current real-estate boom, ignoring other historical data and evidence from other countries where bubbles had burst. Then you feed it all into a fancy computer model, with the paramaters suitably tweaked, and presto, you have AAA-rated securities without having any real idea how risky they are.
A AAA rating from S&P means that they predict the security has a 0.12% chance of defaulting. Of the Collateralized Debt Obligations that they rated AAA, 28% defaulted. So they were only wrong by a factor of 20,000%. Hi ho.
Some other interesting stuff from The Signal and the:
It helps to understand the underlying dynamics of what you’re trying to model. Weather is terribly complicated and even chaotic (it was one of the main sources of chaos theory), but we do basically understand the equations that govern it, and as our observations and our computers improve, so do our forecasts. In economics, by contrast, some forecasters are still just throwing a bunch of variables into the computer and seeing what correlations pop out, and despite faster and faster computers, our projections of GDP are still just as crappy as they were 40 years ago. Kinda depressing.
But getting back to weather, it turns out to be another domain where many people are content to be deceived. The forecasts from the National Weather Service are honest, but that’s not what you get from your local TV station. They habitually tart up their forecasts, for example regularly turning a 5% chance of rain into 20%. If you say it’s going to rain and it doesn’t, nobody complains, while if you predict sun and get rain, you catch hell. (I’m assuming these are not forecasts for farm country.)
Another persistent theme in Silver’s book is that people hate uncertainty and are irresistibly tempted to downplay it. A study of predictions by experts in politics, economics, and international relations found that when they said there was “no chance” that X would happen, X happened about 15% of the time. If you follow sports, you’ve probably heard tipsters claiming to offer bets that are “locks”—and you’ve probably wondered who who falls for such a pitch. But when the lock is, say, not having a big earthquake near your nuclear reactor, there seem to be lots of takers.
*Keep in mind that I am talking about simple old-fashioned toasters. I’m sure there are programmable toaster-ovens nowadays that could give Pat Buchanan a run for his money.