This New Yorker piece on “What Statistics Can and Can’t Tell Us About Ourselves” is a fun read. It includes this gem about the dangers of too much disaggregation, from epidemiologist Richard Peto and his team studying the effects of aspirin:
By subdividing the big picture, he argued, you introduce all kinds of uncertainty into the results. For one thing, the smaller the size of the groups considered, the greater the chance of a fluke. It would be “scientifically stupid,” he observed, to draw conclusions on anything other than the big picture. The journal was insistent, so Peto relented. He resubmitted the paper with all the subgroups the referee had asked for, but with a sly addition. He also subdivided the results by astrological sign. It wasn’t that astrology was going to influence the impact of aspirin; it was that, just by chance, the number of people for whom aspirin works will be greater in some groups than in others. Sure enough, in the study, it appeared as though aspirin didn’t work for Libras and Geminis but halved your risk of death if you happened to be a Capricorn.
The article also touches on the problem of random events and how we interpret them:
Now, imagine I gave out fair coins to every person in the United States and asked everyone to complete the same test. Here’s the issue: even with a threshold of one in a million—even with everything perfectly fair and aboveboard—we would still expect around three hundred of these people to throw twenty heads in a row. If they were following [statistical significance tests], they’d have no choice but to conclude that they’d been given a trick coin. The fact is that, wherever you decide to set the threshold, if you repeat your experiment enough times, extremely unlikely outcomes are bound to arise eventually.
As a bonus, the piece also includes the line, “people are not well represented by the average. As the mathematician Ian Stewart points out in “Do Dice Play God?,” the average person has one breast and one testicle.”
Check out the full article here.
–Guest post by Chad Aldeman
