Friday, December 21, 2012

Why you should reconsider that "expert's" opinion

December 2012 Book Review

The Signal and the Noise
Why so many predictions fail – but some don’t
I went in with high expectations and bought into the marketing machine (primarily an interview I saw on Jon Stewart’s The Daily Show).  As a result I think I set the bar a little high and felt the book fell a little short.  In hindsight, I think Silver wrote the book with the idea that different examples would appeal to different people.  I was all in on the weather, earthquake and economic prediction chapters, but felt lost in the chapters about chess, poker and the financial crisis.  I love how Silver describes the idea of some analysts/forecasters “overfitting” data to produce a detailed outlook, when in reality this kind of analysis ends up being more harmful than helpful.  I can’t tell you how many business leaders and analysts I've encountered doing this kind of analysis.  I also love how he calls out “Economists” in this book for what they are not…reliable in their predictions.  Every chapter may not hit a home run, but at 450+ pages, I suspect Silver will touch on at least two or three fascinating areas specific to your interests.
Here are a few of the better passages I came across:
“The experts in his survey – regardless of their occupation, experience, or subfield – had done barely any better than random chance, and they had done worse than even rudimentary statistical methods at predicting future political events.”
“The overfit model scores those extra points in essence by cheating – by fitting noise rather than signal.  It actually does a much worse job of explaining the real world.”
“Getting feedback about how well our predictions have done is one way – perhaps the essential way – to improve them.”
“If you give a computer bad data, or devise a foolish set of instructions for it to analyze, it won’t spin straw into gold.  Meanwhile, computers are not very good at tasks that require creativity and imagination, like devising strategies or developing theories about the way the world works.”
The Fightin' Analyst Book Rating:
out of five stars