Think Like a Freak - Steven D. Levitt, Stephen J. Dubner (2014)
Chapter 2. The Three Hardest Words in the English Language
Imagine you are asked to listen to a simple story and then answer a few questions about it. Here’s the story:
A little girl named Mary goes to the beach with her mother and brother. They drive there in a red car. At the beach they swim, eat some ice cream, play in the sand, and have sandwiches for lunch.
Now the questions:
1. What color was the car?
2. Did they have fish and chips for lunch?
3. Did they listen to music in the car?
4. Did they drink lemonade with lunch?
All right, how’d you do? Let’s compare your answers to those of a bunch of British schoolchildren, aged five to nine, who were given this quiz by academic researchers. Nearly all the children got the first two questions right (“red” and “no”). But the children did much worse with questions 3 and 4. Why? Those questions were unanswerable—there simply wasn’t enough information given in the story. And yet a whopping 76 percent of the children answered these questions either yes or no.
Kids who try to bluff their way through a simple quiz like this are right on track for careers in business and politics, where almost no one ever admits to not knowing anything. It has long been said that the three hardest words to say in the English language are I love you. We heartily disagree! For most people, it is much harder to say I don’t know. That’s a shame, for until you can admit what you don’t yet know, it’s virtually impossible to learn what you need to.
Before we get into the reasons for all this fakery—and the costs, and the solutions—let’s clarify what we mean when we talk about what we “know.”
There are of course different levels and categories of knowledge. At the top of this hierarchy are what might be called “known facts,” things that can be scientifically verified. (As Daniel Patrick Moynihan was famous for saying: “Everyone’s entitled to their own opinion but not to their own facts.”) If you insist that the chemical composition of water is HO2 instead of H2O, you will eventually be proved wrong.
Then there are “beliefs,” things we hold to be true but which may not be easily verified. On such topics, there is more room for disagreement. For instance: Does the devil really exist?
This question was asked in a global survey. Among the countries included, here are the top five for devil belief, ranked by share of believers:
1. Malta (84.5%)
2. Northern Ireland (75.6%)
3. United States (69.1%)
4. Ireland (55.3%)
5. Canada (42.9%)
And here are the five countries with the fewest devil believers:
1. Latvia (9.1%)
2. Bulgaria (9.6%)
3. Denmark (10.4%)
4. Sweden (12.0%)
5. Czech Republic (12.8%)
How can there be such a deep split on such a simple question? Either the Latvians or the Maltese plainly don’t know what they think they know.
Okay, so maybe the devil’s existence is too otherworldly a topic to consider at all factual. Let’s look at a different kind of question, one that falls somewhere between belief and fact:
According to news reports, groups of Arabs carried out the attacks against the USA on September 11. Do you believe this to be true or not?
To most of us, the very question is absurd: of course it is true! But when asked in predominantly Muslim countries, the question got a different answer. Only 20 percent of Indonesians believed that Arabs carried out the 9/11 attacks, along with 11 percent of Kuwaitis and 4 percent of Pakistanis. (When asked who was responsible, respondents typically blamed the Israeli or U.S. government or “non-Muslim terrorists.”)
All right, so what we “know” can plainly be sculpted by political or religious views. The world is also thick with “entrepreneurs of error,” as the economist Edward Glaeser calls them, political and religious and business leaders who “supply beliefs when it will increase their own financial or political returns.”
On its own, this is problem enough. But the stakes get higher when we routinely pretend to know more than we do.
Think about some of the hard issues that politicians and business leaders face every day. What’s the best way to stop mass shootings? Are the benefits of fracking worth the environmental costs? What happens if we allow that Middle Eastern dictator who hates us to stay in power?
Questions like these can’t be answered merely by assembling a cluster of facts; they require judgment, intuition, and a guess as to how things will ultimately play out. Furthermore, these are multidimensional cause-and-effect questions, which means their outcomes are both distant and nuanced. With complex issues, it can be ridiculously hard to pin a particular cause on a given effect. Did the assault-weapon ban really cut crime—or was it one of ten other factors? Did the economy stall because tax rates were too high—or were the real villains all those Chinese exports and a spike in oil prices?
In other words, it can be hard to ever really “know” what caused or solved a given problem—and that’s for events that have already happened. Just think how much harder it is to predict what will work in the future. “Prediction,” as Niels Bohr liked to say, “is very difficult, especially if it’s about the future.”
And yet we constantly hear from experts—not just politicians and business leaders but also sports pundits, stock-market gurus, and of course meteorologists—who tell us they have a pretty good idea of how the future will unspool. Do they really know what they’re talking about or are they, like the British schoolkids, just bluffing?
In recent years, scholars have begun to systematically track the predictions of various experts. One of the most impressive studies was conducted by Philip Tetlock, a psychology professor at the University of Pennsylvania. His focus is politics. Tetlock enlisted nearly 300 experts—government officials, political-science scholars, national-security experts, and economists—to make thousands of predictions that he charted over the course of twenty years. For instance: in Democracy X—let’s say it’s Brazil—will the current majority party retain, lose, or strengthen its status after the next election? Or, for Undemocratic Country Y—Syria, perhaps—will the basic character of the political regime change in the next five years? In the next ten years? If so, in what direction?
The results of Tetlock’s study were sobering. These most expert of experts—96 percent of them had postgraduate training—“thought they knew more than they knew,” he says. How accurate were their predictions? They weren’t much better than “dart-throwing chimps,” as Tetlock often joked.
“Oh, the monkey-with-a-dartboard comparison, that comes back to haunt me all the time,” he says. “But with respect to how they did relative to, say, a baseline group of Berkeley undergraduates making predictions, they did somewhat better than that. Did they do better than an extrapolation algorithm? No, they did not.”
Tetlock’s “extrapolation algorithm” is simply a computer programmed to predict “no change in current situation.” Which, if you think about it, is a computer’s way of saying “I don’t know.”
A similar study by a firm called CXO Advisory Group covered more than 6,000 predictions by stock-market experts over several years. It found an overall accuracy rate of 47.4 percent. Again, the dart-throwing chimp likely would have done just as well—and, when you consider investment fees, at a fraction of the cost.
When asked to name the attributes of someone who is particularly bad at predicting, Tetlock needed just one word. “Dogmatism,” he says. That is, an unshakable belief they know something to be true even when they don’t. Tetlock and other scholars who have tracked prominent pundits find that they tend to be “massively overconfident,” in Tetlock’s words, even when their predictions prove stone-cold wrong. That is a lethal combination—cocky plus wrong—especially when a more prudent option exists: simply admit that the future is far less knowable than you think.
Unfortunately, this rarely happens. Smart people love to make smart-sounding predictions, no matter how wrong they may turn out to be. This phenomenon was beautifully captured in a 1998 article for Red Herring magazine called “Why Most Economists’ Predictions Are Wrong.” It was written by Paul Krugman, himself an economist, who went on to win the Nobel Prize.* Krugman points out that too many economists’ predictions fail because they overestimate the impact of future technologies, and then he makes a few predictions of his own. Here’s one: “The growth of the Internet will slow drastically, as the flaw in ‘Metcalfe’s law’—which states that the number of potential connections in a network is proportional to the square of the number of participants—becomes apparent: most people have nothing to say to each other! By 2005 or so, it will become clear that the Internet’s impact on the economy has been no greater than the fax machine’s.”
As of this writing, the market capitalization of Google, Amazon, and Facebook alone is more than $700 billion, which is more than the GDP of all but eighteen countries. If you throw in Apple, which isn’t an Internet company but couldn’t exist without it, the market cap is $1.2 trillion. That could buy a lot of fax machines.
Maybe we need more economists like Thomas Sargent. He too won a Nobel, for his work measuring macroeconomic cause and effect. Sargent has likely forgotten more about inflation and interest rates than the rest of us will ever know. When Ally Bank wanted to make a TV commercial a few years ago touting a certificate of deposit with a “raise your rate” feature, Sargent was cast in the lead.
The setting is an auditorium whose stage evokes a university club: ornate chandeliers, orderly bookshelves, walls hung with portraits of distinguished gentlemen. Sargent, seated regally in a leather club chair, awaits his introduction. A moderator begins:
MODERATOR: Tonight, our guest: Thomas Sargent, Nobel laureate in economics and one of the most-cited economists in the world. Professor Sargent, can you tell me what CD rates will be in two years?
And that’s it. As the Ally announcer points out, “If he can’t, no one can”—thus the need for an adjustable-rate CD. The ad is a work of comic genius. Why? Because Sargent, in giving the only correct answer to a virtually unanswerable question, shows how absurd it is that so many of us routinely fail to do the same.
It isn’t only that we know less than we pretend about the outside world; we don’t even know ourselves all that well. Most people are terrible at the seemingly simple task of assessing their own talents. As two psychologists recently put it in an academic journal: “Despite spending more time with themselves than with any other person, people often have surprisingly poor insight into their skills and abilities.” A classic example: when asked to rate their driving skills, roughly 80 percent of respondents rated themselves better than the average driver.
But let’s say you are excellent at a given thing, a true master of your domain, like Thomas Sargent. Does this mean you are also more likely to excel in a different domain?
A sizable body of research says the answer is no. The takeaway here is simple but powerful: just because you’re great at something doesn’t mean you’re good at everything. Unfortunately, this fact is routinely ignored by those who engage in—take a deep breath—ultracrepidarianism, or “the habit of giving opinions and advice on matters outside of one’s knowledge or competence.”
Making grandiose assumptions about your abilities and failing to acknowledge what you don’t know can lead, unsurprisingly, to disaster. When schoolchildren fake their answers about a trip to the seashore, there are no consequences; their reluctance to say “I don’t know” imposes no real costs on anyone. But in the real world, the societal costs of faking it can be huge.
Consider the Iraq War. It was executed primarily on U.S. claims that Saddam Hussein had weapons of mass destruction and was in league with al Qaeda. To be sure, there was more to it than that—politics, oil, and perhaps revenge—but it was the al Qaeda and weapons claims that sealed the deal. Eight years, $800 billion, and nearly 4,500 American deaths later—along with at least 100,000 Iraqi fatalities—it was tempting to consider what might have happened had the purveyors of those claims admitted that they did not in fact “know” them to be true.
Just as a warm and moist environment is conducive to the spread of deadly bacteria, the worlds of politics and business especially—with their long time frames, complex outcomes, and murky cause and effect—are conducive to the spread of half-cocked guesses posing as fact. And here’s why: the people making these wild guesses can usually get away with it! By the time things have played out and everyone has realized they didn’t know what they were talking about, the bluffers are long gone.
If the consequences of pretending to know can be so damaging, why do people keep doing it?
That’s easy: in most cases, the cost of saying “I don’t know” is higher than the cost of being wrong—at least for the individual.
Think back to the soccer player who was about to take a life-changing penalty kick. Aiming toward the center has a better chance of success, but aiming toward a corner is less risky to his own reputation. So that’s where he shoots. Every time we pretend to know something, we are doing the same: protecting our own reputation rather than promoting the collective good. None of us want to look stupid, or at least overmatched, by admitting we don’t know an answer. The incentives to fake it are simply too strong.
Incentives can also explain why so many people are willing to predict the future. A huge payoff awaits anyone who makes a big and bold prediction that happens to come true. If you say the stock market will triple within twelve months and it actually does, you will be celebrated for years (and paid well for future predictions). What happens if the market crashes instead? No worries. Your prediction will already be forgotten. Since almost no one has a strong incentive to keep track of everyone else’s bad predictions, it costs almost nothing to pretend you know what will happen in the future.
In 2011, an elderly Christian radio preacher named Harold Camping made headlines around the world by predicting that the Rapture would occur on Saturday, May 21 of that year. The world would end, he warned, and seven billion people—everyone but the hard-core believers—would die.
One of us has a young son who saw these headlines and got scared. His father reassured him that Camping’s prediction was baseless, but the boy was distraught. In the nights leading up to May 21, he cried himself to sleep; it was a miserable experience for all. And then Saturday dawned bright and clear, the world still in one piece. The boy, with the false bravado of a ten-year-old, declared he’d never been scared at all.
“Even so,” his father said, “what do you think should happen to Harold Camping?”
“Oh, that’s easy,” the boy said. “They should take him outside and shoot him.”
This punishment may seem extreme, but the sentiment is understandable. When bad predictions are unpunished, what incentive is there to stop making them? One solution was recently proposed in Romania. That country boasts a robust population of “witches,” women who tell fortunes for a living. Lawmakers decided that witches should be regulated, taxed, and—most important—made to pay a fine or even go to prison if the fortunes they told didn’t prove accurate. The witches were understandably upset. One of them responded as she knew best: by threatening to cast a spell on the politicians with cat feces and a dog corpse.
There is one more explanation for why so many of us think we know more than we do. It has to do with something we all carry with us everywhere we go, even though we may not consciously think about it: a moral compass.
Each of us develops a moral compass (some stronger than others, to be sure) as we make our way through the world. This is for the most part a wonderful thing. Who wants to live in a world where people run around with no regard for the difference between right and wrong?
But when it comes to solving problems, one of the best ways to start is by putting away your moral compass.
When you are consumed with the rightness or wrongness of a given issue—whether it’s fracking or gun control or genetically engineered food—it’s easy to lose track of what the issue actually is. A moral compass can convince you that all the answers are obvious (even when they’re not); that there is a bright line between right and wrong (when often there isn’t); and, worst, that you are certain you already know everything you need to know about a subject so you stop trying to learn more.
In centuries past, sailors who relied on a ship’s compass found it occasionally gave erratic readings that threw them off course. Why? The increasing use of metal on ships—iron nails and hardware, the sailors’ tools and even their buckles and buttons—messed with the compass’s magnetic read. Over time, sailors went to great lengths to keep metal from interfering with the compass. With such an evasive measure in mind, we are not suggesting you toss your moral compass in the trash—not at all—but only that you temporarily set it aside, to prevent it from clouding your vision.
Consider a problem like suicide. It is so morally fraught that we rarely discuss it in public; it is as if we’ve thrown a black drape over the entire topic.
This doesn’t seem to be working out very well. There are about 38,000 suicides a year in the United States, more than twice the number of homicides. Suicide is one of the top ten causes of death for nearly every age group. Because talking about suicide carries such a strong moral taboo, these facts are little known.
As of this writing, the U.S. homicide rate is lower than it’s been in fifty years. The rate of traffic fatalities is at a historic low, having fallen by two-thirds since the 1970s. The overall suicide rate, meanwhile, has barely budged—and worse yet, suicide among 15- to 24-year-olds has tripled over the past several decades.
One might think, therefore, that by studying the preponderance of cases, society has learned everything possible about what leads people to commit suicide.
David Lester, a psychology professor at Richard Stockton College in New Jersey, has likely thought about suicide longer, harder, and from more angles than any other human. In more than twenty-five-hundred academic publications, he has explored the relationship between suicide and, among other things, alcohol, anger, antidepressants, astrological signs, biochemistry, blood type, body type, depression, drug abuse, gun control, happiness, holidays, Internet use, IQ, mental illness, migraines, the moon, music, national-anthem lyrics, personality type, sexuality, smoking, spirituality, TV watching, and wide-open spaces.
Has all this study led Lester to some grand unified theory of suicide? Hardly. So far he has one compelling notion. It’s what might be called the “no one left to blame” theory of suicide. While one might expect that suicide is highest among people whose lives are the hardest, research by Lester and others suggests the opposite: suicide is more common among people with a higher quality of life.
“If you’re unhappy and you have something to blame your unhappiness on—if it’s the government, or the economy, or something—then that kind of immunizes you against committing suicide,” he says. “It’s when you have noexternal cause to blame for your unhappiness that suicide becomes more likely. I’ve used this idea to explain why African-Americans have lower suicide rates, why blind people whose sight is restored often become suicidal, and why adolescent suicide rates often rise as their quality of life gets better.”
That said, Lester admits that what he and other experts know about suicide is dwarfed by what is unknown. We don’t know much, for instance, about the percentage of people who seek or get help before contemplating suicide. We don’t know much about the “suicidal impulse”—how much time elapses between a person’s decision and action. We don’t even know what share of suicide victims are mentally ill. There is so much disagreement on this issue, Lester says, that estimates range from 5 percent to 94 percent.
“I’m expected to know the answers to questions such as why people kill themselves,” Lester says. “And myself and my friends, we often—when we’re relaxing—admit that we really don’t have a good idea why people kill themselves.”
If someone like David Lester, one of the world’s leading authorities in his field, is willing to admit how much he has to learn, shouldn’t it be easier for all of us to do the same? All right, then: on to the learning.
The key to learning is feedback. It is nearly impossible to learn anything without it.
Imagine you’re the first human in history who’s trying to make bread—but you’re not allowed to actually bake it and see how the recipe turns out. Sure, you can adjust the ingredients and other variables all you want. But if you never get to bake and eat the finished product, how will you know what works and what doesn’t? Should the ratio of flour to water be 3:1 or 2:1? What happens if you add salt or oil or yeast—or maybe animal dung? Should the dough be left to sit before baking—and if so, for how long, and under what conditions? How long will it need to bake? Covered or uncovered? How hot should the fire be?
Even with good feedback, it can take a while to learn. (Just imagine how bad some of that early bread was!) But without it, you don’t stand a chance; you’ll go on making the same mistakes forever.
Thankfully, our ancestors did figure out how to bake bread, and since then we’ve learned to do all sorts of things: build a house, drive a car, write computer code, even figure out the kind of economic and social policies that voters like. Voting may be one of the sloppiest feedback loops around, but it is feedback nonetheless.
In a simple scenario, it’s easy to gather feedback. When you’re learning to drive a car, it’s pretty obvious what happens when you take a sharp mountain curve at 80 miles an hour. (Hello, ravine!) But the more complex a problem is, the harder it is to capture good feedback. You can gather a lot of facts, and that may be helpful, but in order to reliably measure cause and effect you need to get beneath the facts. You may have to purposefully go out and create feedback through an experiment.
Not long ago, we met with some executives from a large multinational retailer. They were spending hundreds of millions of dollars a year on U.S. advertising—primarily TV commercials and print circulars in Sunday newspapers—but they weren’t sure how effective it was. So far, they had come to one concrete conclusion: TV ads were about four times more effective, dollar for dollar, than print ads.
We asked how they knew this. They whipped out some beautiful, full-color PowerPoint charts that tracked the relationship between TV ads and product sales. Sure enough, there was a mighty sales spike every time their TV ads ran. Valuable feedback, right? Umm … let’s make sure.
How often, we asked, did those ads air? The executives explained that because TV ads are so much more expensive than print ads, they were concentrated on just three days: Black Friday, Christmas, and Father’s Day. In other words, the company spent millions of dollars to entice people to go shopping at precisely the same time that millions of people were about to go shopping anyway.
So how could they know the TV ads caused the sales spike? They couldn’t! The causal relationship might just as easily move in the opposite direction, with the expected sales spike causing the company to buy TV ads. It’s possible the company would have sold just as much merchandise without spending a single dollar on TV commercials. The feedback in this case was practically worthless.
Now we asked about the print ads. How often did they run? One executive told us, with obvious pride, that the company had bought newspaper inserts every single Sunday for the past twenty years in 250 markets across the United States.
So how could they tell whether these ads were effective? They couldn’t. With no variation whatsoever, it was impossible to know.
What if, we said, the company ran an experiment to find out? In science, the randomized control trial has been the gold standard of learning for hundreds of years—but why should scientists have all the fun? We described an experiment the company might run. They could select 40 major markets across the country and randomly divide them into two groups. In the first group, the company would keep buying newspaper ads every Sunday. In the second group, they’d go totally dark—not a single ad. After three months, it would be easy to compare merchandise sales in the two groups to see how much the print ads mattered.
“Are you crazy?” one marketing executive said. “We can’t possibly go dark in 20 markets. Our CEO would kill us.”
“Yeah,” said someone else, “it’d be like that kid in Pittsburgh.”
What kid in Pittsburgh?
They told us about a summer intern who was supposed to call in the Sunday ad buys for the Pittsburgh newspapers. For whatever reason, he botched his assignment and failed to make the calls. So for the entire summer, the company ran no newspaper ads in a large chunk of Pittsburgh. “Yeah,” one executive said, “we almost got fired for that one.”
So what happened, we asked, to the company’s Pittsburgh sales that summer?
They looked at us, then at each other—and sheepishly admitted it never occurred to them to check the data. When they went back and ran the numbers, they found something shocking: the ad blackout hadn’t affected Pittsburgh sales at all!
Now that, we said, is valuable feedback. The company may well be wasting hundreds of millions of dollars on advertising. How could the executives know for sure? That 40-market experiment would go a long way toward answering the question. And so, we asked them, are you ready to try it now?
“Are you crazy?” the marketing executive said again. “We’ll get fired if we do that!”
To this day, on every single Sunday in every single market, this company still buys newspaper advertising—even though the only real piece of feedback they ever got is that the ads don’t work.
The experiment we proposed, while heretical to this company’s executives, was nothing if not simple. It would have neatly allowed them to gather the feedback they needed. There is no guarantee they would have been happy with the result—maybe they’d need to spend more ad money, or maybe the ads were only successful in certain markets—but at least they would have gained a few clues as to what works and what doesn’t. The miracle of a good experiment is that in one simple cut, you can eliminate all the complexity that makes it so hard to determine cause and effect.
But experimentation of this sort is regrettably rare in the corporate and nonprofit worlds, government, and elsewhere. Why?
One reason is tradition. In our experience, many institutions are used to making decisions based on some murky blend of gut instinct, moral compass, and whatever the previous decision maker did.
A second reason is lack of expertise: while it isn’t hard to run a simple experiment, most people have never been taught to do so and may therefore be intimidated.
But there is a third, grimmer explanation for this general reluctance toward experimentation: it requires someone to say “I don’t know.” Why mess with an experiment when you think you already know the answer? Rather than waste time, you can just rush off and bankroll the project or pass the law without having to worry about silly details like whether or not it’ll work.
If, however, you’re willing to think like a Freak and admit what you don’t know, you will see there is practically no limit to the power of a good randomized experiment.
Granted, not every scenario lends itself to experimentation, especially when it comes to social issues. In most places—in most democracies, at least—you can’t just randomly select portions of the population and command them to, say, have 10 children instead of 2 or 3; or eat nothing but lentils for 20 years; or start going to church every day. That’s why it pays to be on the lookout for a “natural experiment,” a shock to the system that produces the sort of feedback you’d get if you could randomly command people to change their behavior.
A lot of the scenarios we’ve written about in our earlier books have exploited natural experiments. In trying to measure the knock-on effects of sending millions of people to prison, we took advantage of civil-rights lawsuits that forced overcrowded prisons in some states to set free thousands of inmates—something that no governor or mayor would voluntarily do. In analyzing the relationship between abortion and crime, we capitalized on the fact that the legalization of abortion was staggered in time across different states; this allowed us to better isolate its effects than if it had been legalized everywhere at once.
Alas, natural experiments as substantial as these are not common. One alternative is to set up a laboratory experiment. Social scientists around the world have been doing this in droves recently. They recruit legions of undergrads to act out various scenarios in the hopes of learning about everything from altruism to greed to criminality. Lab experiments can be incredibly useful in exploring behaviors that aren’t so easy to capture in the real world. The results are often fascinating—but not necessarily that informative.
Why not? Most of them simply don’t bear enough resemblance to the real-world scenarios they are trying to mimic. They are the academic equivalent of a marketing focus group—a small number of handpicked volunteers in an artificial environment who dutifully carry out the tasks requested by the person in charge. Lab experiments are invaluable in the hard sciences, in part because neutrinos and monads don’t change their behavior when they are being watched; but humans do.
A better way to get good feedback is to run a field experiment—that is, rather than trying to mimic the real world in a lab, take the lab mind-set into the real world. You’re still running an experiment but the subjects don’t necessarily know it, which means the feedback you’ll glean is pure.
With a field experiment, you can randomize to your heart’s content, include more people than you could ever fit in a lab, and watch those people responding to real-world incentives rather than the encouragements of a professor hovering over them. When done well, field experiments can radically improve how problems get solved.
Already this is happening. In Chapter 6, you’ll read about a clever field experiment that got homeowners in California to use less electricity, and another that helped a charity raise millions of dollars to help turn around the lives of poor children. In Chapter 9, we’ll tell you about the most audacious experiment we’ve ever run, in which we recruited people facing hard life decisions—whether to join the military or quit a job or end a romantic relationship—and, with the flip of a coin, randomly made the decision for them.
As useful as experiments can be, there is one more reason a Freak might want to try them: it’s fun! Once you embrace the spirit of experimentation, the world becomes a sandbox in which to try out new ideas, ask new questions, and challenge the prevailing orthodoxies.
You may have been struck, for example, by the fact that some wines are so much more expensive than others. Do expensive wines really taste better? Some years back, one of us tried an experiment to find out.
The setting was the Society of Fellows, a Harvard outpost where postdoctoral students carry out research and, once a week, sit with their esteemed elder Fellows for a formal dinner. Wine was a big part of these dinners, and the Society boasted a formidable cellar. It wasn’t unusual for a bottle to cost $100. Our young Fellow wondered if this expense was justified. Several elder Fellows, who happened to be wine connoisseurs, assured him it was: an expensive bottle, they told him, was generally far superior to a cheaper version.
The young Fellow decided to run a blind tasting to see how true this was. He asked the Society’s wine steward to pull two good vintages from the cellar. Then he went to a liquor store and bought the cheapest available bottle made from the same grape. It cost $8. He poured the three wines into four decanters, with one of the cellar wines repeated. Here was the layout:
When it came time to taste the wines, the elder Fellows couldn’t have been more cooperative. They swirled, they sniffed, they sipped; they filled out marking cards, noting their assessment of each wine. They were not told that one of the wines cost about one-tenth the price of the others.
The results? On average, the four decanters received nearly identical ratings—that is, the cheap wine tasted just as good as the expensive ones. But that wasn’t even the most surprising finding. The young Fellow also compared how each drinker rated each wine in comparison to the other wines. Can you guess which two decanters they judged as most different from each other? Decanters 1 and 4—which had been poured from the exact same bottle!
These findings were not greeted with universal good cheer. One of the elder connoisseur-Fellows loudly announced that he had a head cold, which presumably gummed up his palate, and stormed from the room.
Okay, so maybe this experiment wasn’t very sporting—or scientific. Wouldn’t it be nice to see the results of a more robust experiment along these lines?
Robin Goldstein, a food-and-wine critic who has studied neuroscience, law, and French cuisine, decided to run such an experiment. Over several months, he organized 17 blind tastings across the United States that included more than 500 people, ranging from wine beginners to sommeliers and vintners.
Goldstein used 523 different wines, from $1.65 to $150 per bottle. The tastings were double-blind, meaning that neither the drinker nor the person serving the wine knew its identity or price. After each wine, a drinker would answer this question: “Overall, how do you find the wine?” The answers were “bad” (1 point), “okay” (2 points), “good” (3 points), and “great” (4 points).
The average rating for all wines, across all tasters, was 2.2, or just above “okay.” So did the more expensive wines rack up more points? In a word: no. Goldstein found that on average, the people in his experiment “enjoy more expensive wines slightly less” than cheaper ones. He was careful to note that the experts in his sample—about 12 percent of the participants had some kind of wine training—did not prefer the cheaper wines, but nor was it clear that they preferred the expensive ones.
When you buy a bottle of wine, do you sometimes base your decision on how pretty the label is? According to Robin Goldstein’s results, this doesn’t seem like a bad strategy: at least it’s easy to tell labels apart, unlike the stuff in the bottle.
Goldstein, already bound for heretic status in the wine industry, had one more experiment to try. If more expensive wines don’t taste better than cheap ones, he wondered, what about wine critics’ ratings and awards—how legitimate are they? The best-known player in this arena is Wine Spectator magazine, which reviews thousands of wines and bestows its Award of Excellence to restaurants that serve “a well-chosen selection of quality producers, along with a thematic match to the menu in both price and style.” Only a few thousand restaurants worldwide hold this distinction.
Goldstein wondered if the award is as meaningful as it seems. He created a fictional restaurant, in Milan, with a fake website and a fake menu, “a fun amalgamation of somewhat bumbling nouvelle-Italian recipes,” he explained. He called it Osteria L’Intrepido, or “Fearless Restaurant,” after his own Fearless Critic restaurant guides. “There were two questions being tested here,” he says. “One was, do you have to have a good wine list to win a Wine Spectator Award of Excellence? And the second was, do you have to exist to win a Wine Spectator Award of Excellence?”
Goldstein took great care in creating L’Intrepido’s fictional wine list, but not in the direction you might imagine. For the reserve list—typically a restaurant’s best, most expensive wines—he chose wines that were particularly bad. The list included 15 wines that Wine Spectator itself had reviewed, using its 100-point scale. On this scale, anything above 90 is at least “outstanding”; above 80 is at least “good.” If a wine gets 75-79 points, Wine Spectator calls it “mediocre.” Anything below 74 is “not recommended.”
So how had the magazine rated the 15 wines Goldstein chose for his reserve list? Their average Wine Spectator rating was a paltry 71. One vintage, according to Wine Spectator, “smells barnyardy and tastes decayed.” Another had “just too much paint thinner and nail varnish character.” A 1995 Cabernet Sauvignon “I Fossaretti,” which scored a lowly 58 points, got this review from Wine Spectator: “Something wrong here … tasted metallic and odd.” On Goldstein’s reserve list, this bottle was priced at 120 euros; the average cost of the 15 bottles was about 180 euros.
How could Goldstein possibly expect that a fake restaurant whose most expensive wines had gotten terrible Wine Spectator reviews would win a Wine Spectator Award of Excellence?
“My hypothesis,” he says, “was that the $250 fee was really the functional part of the application.”
So he sent off the check, the application, and his wine list. Not long after, the answering machine at his fake restaurant in Milan received a real call from Wine Spectator in New York. He had won an Award of Excellence! The magazine also asked “if you might have an interest in publicizing your award with an ad in the upcoming issue.” This led Goldstein to conclude that “the entire awards program was really just an advertising scheme.”
Does that mean, we asked him, that the two of us—who don’t know a thing about running a restaurant—could someday hope to win a Wine Spectator Award of Excellence?
“Yeah,” he said, “if your wines are bad enough.”
Maybe, you are thinking, it is obvious that “awards” like this are to some degree just a marketing stunt. Maybe it was also obvious to you that more expensive wines don’t necessarily taste better or that a lot of advertising money is wasted.
But a lot of obvious ideas are only obvious after the fact—after someone has taken the time and effort to investigate them, to prove them right (or wrong). The impulse to investigate can only be set free if you stop pretending to know answers that you don’t. Because the incentives to pretend are so strong, this may require some bravery on your part.
Remember those British schoolchildren who made up answers about Mary’s trip to the seashore? The researchers who ran that experiment did a follow-up study, called “Helping Children Correctly Say ‘I Don’t Know’ to Unanswerable Questions.” Once again, the children were asked a series of questions; but in this case, they were explicitly told to say “I don’t know” if a question was unanswerable. The happy news is that the children were wildly successful at saying “I don’t know” when appropriate, while still getting the other questions right.
Let us all take encouragement from the kids’ progress. The next time you run into a question that you can only pretend to answer, go ahead and say “I don’t know”—and then follow up, certainly, with “but maybe I can find out.” And work as hard as you can to do that. You may be surprised by how receptive people are to your confession, especially when you come through with the real answer a day or a week later.
But even if this goes poorly—if your boss sneers at your ignorance or you can’t figure out the answer no matter how hard you try—there is another, more strategic benefit to occasionally saying “I don’t know.” Let’s say you’ve already done that on a few occasions. The next time you’re in a real jam, facing an important question that you just can’t answer, go ahead and make up something—and everyone will believe you, because you’re the guy who all those other times was crazy enough to admit you didn’t know the answer.
After all, just because you’re at the office is no reason to stop thinking.