Don't Let Math Pull the Wool Over Your Eyes

This column will make the case that many people, including holders of graduate degrees, professional researchers and even editors of scientific journals, can be too easily impressed by math. A mathematical model (Tpp = T0 - ŸT0d2f - ŸTpdf) is developed to describe sequential effects.

Did that second sentence make the first more persuasive? It did for most participants in a recent intriguing experiment whose result suggests people often interact with math in a way that isn't very logical. Other research has shown that even those who should be especially clear-sighted about numbers-scientific researchers, for example, and those who review their work for publication-are often uncomfortable with, and credulous about, mathematical material. As a result, some research that finds its way into respected journals-and ends up being reported in the popular press-is flawed.

In the latest study, Kimmo Eriksson, a mathematician and researcher of social psychology at Sweden's Mälardalen University, chose two abstracts from papers published in research journals, one in evolutionary anthropology and one in sociology. He gave them to 200 people to rate for quality-with one twist. At random, one of the two abstracts received an additional sentence, the one above with the math equation, which he pulled from an unrelated paper in psychology. The study's 200 participants all had master's or doctoral degrees. Those with degrees in math, science or technology rated the abstract with the tacked-on sentence as slightly lower-quality than the other. But participants with degrees in humanities, social science or other fields preferred the one with the bogus math, with some rating it much more highly on a scale of 0 to 100.

"Math makes a research paper look solid, but the real science lies not in math but in trying one's utmost to understand the real workings of the world," Prof. Eriksson said.

Several of the authors of the three papers whose material Prof. Eriksson harvested said they generally agreed with that conclusion. "Disciplines with less math emphasis could benefit from more exposure to mathematical logic so that when it is used it is accepted for its rigor rather than because it awes the reader," said Geoff Kushnick, co-author of the evolutionary anthropology paper, published in 2002.

Prof. Eriksson's finding, published in November in the journal Judgment and Decision Making under the title "The Nonsense Math Effect," is preliminary but unfortunately not surprising, other researchers said. It documents a familiar effect, said Daniel Kahneman, professor emeritus of psychology and public affairs at Princeton University. "People who know math understand what other mortals understand, but other mortals do not understand them. This asymmetry gives them a presumption of superior ability."

Prof. Kahneman and other scholars have documented how numbers can warp rather than enhance logical thinking. For example, German researchers asked 52 lawyers to propose sentences for hypothetical repeat shoplifters after having rolled dice to determine the hypothetical sentencing demand by prosecutors. Lawyers who rolled dice that showed a higher requested sentence tended to hand out longer sentences themselves, even with identical facts in the case and the knowledge that the demanded sentence was entirely random. Tying decisions to a proposed number, even when it is random or misguided, is known by psychologists as anchoring and helps explain why a "50% off" sale can seem so compelling even when the original price was vastly inflated.

Prof. Eriksson's study shows how people can stumble when exposed to math outside their field of expertise. Articles in scientific journals, however, typically are reviewed by experts who read the whole paper, not just abstracts. Yet even here, there is evidence that professional researchers often botch statistics in their work, but are allowed to publish problematic data. Writing in Nature last month, cell biologist David L. Vaux decried the continuing publication, including in Nature, of papers with problems such as miscalculated margins of error. "The fact that these scientifically sloppy papers continue to be published means that the authors, reviewers and editors cannot comprehend the statistics, that they have not read the paper carefully, or both," said Prof. Vaux, of the University of Melbourne in Australia.

Martin T. Wells, a biostatistician at Cornell University, has another explanation, chalking up occasional sloppiness in statistics to the pressure to publish early and often. "It's a bit frustrating," he said. "As a biostatistician you want to do things right, and some researchers just want to get off on the cheap analyzing data."

Alan Sokal, the physicist who in 1996 hoaxed a journal of cultural studies with an article he wrote as a jest that the journal published, called Prof. Eriksson's study "very interesting." He would like it replicated with a more credentialed group, such as university professors, who are more likely to review articles for publication.

Prof. Eriksson said he "would love to conduct a follow-up study with other samples," adding his paper didn't itself contain any deliberately faulty statistics to test whether readers would notice. "I wasn't clever enough to think of putting in some nonsense math of my own," he said.