The Wandering Skeptic
Tuesday, January 10, 2006
My Review of Freakonomics
The scientific fidelity of social science is a topic of heated contention in academics. Steven Levitt and Stephen Dubner have successfully brought this debate to the mainstream in the form of their joint book, Freakonomics. But do they make a strong case for validating statistical analyses of an infinitely complex human society?
As any statistician will tell you, one of the major pitfalls of their field is the confusion of correlation and causation. Just because X and Y have similar trends does not necessarily mean that X caused Y or that Y caused X. Numerous times throughout the book, Levitt and Dubner chastise various experts, pundits, and conventional wisdoms for failing to observe this basic tenet. Yet so tempting is this trap that the authors fall right in along with their targets.
Take, for example, the chapter on parenting. A full six paragraphs are devoted to warning about correlation versus causation, the caution of which is thrown immediately to the wind with a set of highly dubious stabs at the causes of various correlations regarding parenting. The data in question comes from Levitt's regression analysis of numerous factors which conventional wisdom believes may play some role in the academic outcome of children. So, for example, correlations were found between a child's test scores and the number of books the parents have in their house, but not how often the parents read to the child. So far, so good. The authors then conclude from similar datapoints that it is the nature of the parents' lives that influence a child's scores, not what the parents do. Granted, it has a certain logical appeal, but it amounts to no more than an educated guess. What's wrong with that? you may ask.
The problems with this example illustrate some of the major difficulties associated with social science. What you may notice about the correlations is that - by necessity - they lack a certain level of detail. What *kind* of books to the parents have? What kind do they read to their child? How often does a child actually pick up one of numerous books? These are questions for which there are few or no practical solutions. The reasons are manifold, including: the number of data points may never be enough (consider how many categories you may have to break predominating book types into: comic books, encyclopedias, TV trivia, etc.); you never know which test subject is lying, exaggerating, or remembering incorrectly; and you can never be sure that test scores are the right thing to measure.
This last difficulty is made more extreme when you consider the following quote from Freakonomics: "Sorry. Culture cramming may be a foundational belief of obsessive parenting, but the ECLS data show no correlation between museum visits and test scores." There should be little surprise at the lack of correlation: there are very few things that a museum offers that would help on the SATs or state exams. But that doesn't mean that museum visits have no positive impact on the intelligence of a child. The authors make the mistake of equating test scores to intelligence. It may very well be true that a child that goes to museums will score no better on entrance exams than a child that doesn't, but it may affect which hobbies they take up, their job performance, and various other important aspects of life that have little or nothing to do with measurable intelligence.
Similar errors in thinking occur throughout the book. In the bagel-seller example, statistics are carelessly and bizarrely used to justify a stance on morality. Because only 13% of people failed to pay for bagels when left out with a payment box, the authors conclude that the majority - in fact, 87% - of people have an innate honesty. I was floored by this kind of uncritical thinking. People may have paid out of fear of getting caught or out of guilt, but not necessarily out of honesty. But more so than that, honesty in one small area of life does not an honest man make. If Dubner and Levitt wanted to conclude simply that statistics is useful for understanding human motivation, that would be fine. But to make sweeping generalizations about whether humans are born innately good or innately bad on a single study is simply irresponsible.
The only positive thing to say about Freakonomics is that it makes you think. But any controversial book can do that. Though there are some fairly solid examples in the book such as regards the real estate agents, the sumo wrestlers, and the cheating teachers, overall the book is uncritical of its own thinking. It would be fine if Levitt and Dubner acknowledged that there may be other interpretations at least as good as their own, but they choose instead to pontificate their own views, in flagrant violation of their professed objectivism. And oddly enough, I happen to agree with most of their views, just not with how they reached them. Levitt is clearly a brilliant man, and I hope he continues to churn out interesting statistical correlations on unusual subjects... but he and Dubner ought to leave the interpretations to others. ¶ 5:20 PM
The scientific fidelity of social science is a topic of heated contention in academics. Steven Levitt and Stephen Dubner have successfully brought this debate to the mainstream in the form of their joint book, Freakonomics. But do they make a strong case for validating statistical analyses of an infinitely complex human society?
As any statistician will tell you, one of the major pitfalls of their field is the confusion of correlation and causation. Just because X and Y have similar trends does not necessarily mean that X caused Y or that Y caused X. Numerous times throughout the book, Levitt and Dubner chastise various experts, pundits, and conventional wisdoms for failing to observe this basic tenet. Yet so tempting is this trap that the authors fall right in along with their targets.
Take, for example, the chapter on parenting. A full six paragraphs are devoted to warning about correlation versus causation, the caution of which is thrown immediately to the wind with a set of highly dubious stabs at the causes of various correlations regarding parenting. The data in question comes from Levitt's regression analysis of numerous factors which conventional wisdom believes may play some role in the academic outcome of children. So, for example, correlations were found between a child's test scores and the number of books the parents have in their house, but not how often the parents read to the child. So far, so good. The authors then conclude from similar datapoints that it is the nature of the parents' lives that influence a child's scores, not what the parents do. Granted, it has a certain logical appeal, but it amounts to no more than an educated guess. What's wrong with that? you may ask.
The problems with this example illustrate some of the major difficulties associated with social science. What you may notice about the correlations is that - by necessity - they lack a certain level of detail. What *kind* of books to the parents have? What kind do they read to their child? How often does a child actually pick up one of numerous books? These are questions for which there are few or no practical solutions. The reasons are manifold, including: the number of data points may never be enough (consider how many categories you may have to break predominating book types into: comic books, encyclopedias, TV trivia, etc.); you never know which test subject is lying, exaggerating, or remembering incorrectly; and you can never be sure that test scores are the right thing to measure.
This last difficulty is made more extreme when you consider the following quote from Freakonomics: "Sorry. Culture cramming may be a foundational belief of obsessive parenting, but the ECLS data show no correlation between museum visits and test scores." There should be little surprise at the lack of correlation: there are very few things that a museum offers that would help on the SATs or state exams. But that doesn't mean that museum visits have no positive impact on the intelligence of a child. The authors make the mistake of equating test scores to intelligence. It may very well be true that a child that goes to museums will score no better on entrance exams than a child that doesn't, but it may affect which hobbies they take up, their job performance, and various other important aspects of life that have little or nothing to do with measurable intelligence.
Similar errors in thinking occur throughout the book. In the bagel-seller example, statistics are carelessly and bizarrely used to justify a stance on morality. Because only 13% of people failed to pay for bagels when left out with a payment box, the authors conclude that the majority - in fact, 87% - of people have an innate honesty. I was floored by this kind of uncritical thinking. People may have paid out of fear of getting caught or out of guilt, but not necessarily out of honesty. But more so than that, honesty in one small area of life does not an honest man make. If Dubner and Levitt wanted to conclude simply that statistics is useful for understanding human motivation, that would be fine. But to make sweeping generalizations about whether humans are born innately good or innately bad on a single study is simply irresponsible.
The only positive thing to say about Freakonomics is that it makes you think. But any controversial book can do that. Though there are some fairly solid examples in the book such as regards the real estate agents, the sumo wrestlers, and the cheating teachers, overall the book is uncritical of its own thinking. It would be fine if Levitt and Dubner acknowledged that there may be other interpretations at least as good as their own, but they choose instead to pontificate their own views, in flagrant violation of their professed objectivism. And oddly enough, I happen to agree with most of their views, just not with how they reached them. Levitt is clearly a brilliant man, and I hope he continues to churn out interesting statistical correlations on unusual subjects... but he and Dubner ought to leave the interpretations to others. ¶ 5:20 PM
