The Laws of Truths and Half-Truths - The Drunkard's Walk: How Randomness Rules Our Lives - Leonard Mlodinow

The Drunkard's Walk: How Randomness Rules Our Lives - Leonard Mlodinow (2008)

Chapter 2. The Laws of Truths and Half-Truths

LOOKING TO THE SKY on a clear, moonless night, the human eye can detect thousands of twinkling sources of light. Nestled among those haphazardly scattered stars are patterns. A lion here, a dipper there. The ability to detect patterns can be both a strength and a weakness. Isaac Newton pondered the patterns of falling objects and created a law of universal gravitation. Others have noted a spike in their athletic performance when they are wearing dirty socks and thenceforth have refused to wear clean ones. Among all the patterns of nature, how do we distinguish the meaningful ones? Drawing that distinction is an inherently practical enterprise. And so it might not astonish you to learn that, unlike geometry, which arose as a set of axioms, proofs, and theorems created by a culture of ponderous philosophers, the theory of randomness sprang from minds focused on spells and gambling, figures we might sooner imagine with dice or a potion in hand than a book or a scroll.

The theory of randomness is fundamentally a codification of common sense. But it is also a field of subtlety, a field in which great experts have been famously wrong and expert gamblers infamously correct. What it takes to understand randomness and overcome our misconceptions is both experience and a lot of careful thinking. And so we begin our tour with some of the basic laws of probability and the challenges involved in uncovering, understanding, and applying them. One of the classic explorations of people’s intuition about those laws was an experiment conducted by the pair who did so much to elucidate our misconceptions, Daniel Kahneman and Amos Tversky.1 Feel free to take part—and learn something about your own probabilistic intuition.

Imagine a woman named Linda, thirty-one years old, single, outspoken, and very bright. In college she majored in philosophy. While a student she was deeply concerned with discrimination and social justice and participated in antinuclear demonstrations. Tversky and Kahneman presented this description to a group of eighty-eight subjects and asked them to rank the following statements on a scale of 1 to 8 according to their probability, with 1 representing the most probable and 8 the least. Here are the results, in order from most to least probable:

Statement

Average Probability Rank

Linda is active in the feminist movement.

2.1

Linda is a psychiatric social worker.

3.1

Linda works in a bookstore and takes yoga classes.

3.3

Linda is a bank teller and is active in the feminist movement.

4.1

Linda is a teacher in an elementary school.

5.2

Linda is a member of the League of Women Voters.

5.4

Linda is a bank teller.

6.2

Linda is an insurance salesperson.

6.4

At first glance there may appear to be nothing unusual in these results: the description was in fact designed to be representative of an active feminist and unrepresentative of a bank teller or an insurance salesperson. But now let’s focus on just three of the possibilities and their average ranks, listed below in order from most to least probable. This is the order in which 85 percent of the respondents ranked the three possibilities:

Statement

Average Probability Rank

Linda is active in the feminist movement.

2.1

Linda is a bank teller and is active in the feminist movement.

4.1

Linda is a bank teller.

6.2

If nothing about this looks strange, then Kahneman and Tversky have fooled you, for if the chance that Linda is a bank teller and is active in the feminist movement were greater than the chance that Linda is a bank teller, there would be a violation of our first law of probability, which is one of the most basic of all: The probability that two events will both occur can never be greater than the probability that each will occur individually. Why not? Simple arithmetic: the chances that event A will occur = the chances that events A and B will occur + the chance that event A will occur and event B will not occur.

Kahneman and Tversky were not surprised by the result because they had given their subjects a large number of possibilities, and the connections among the three scenarios could easily have gotten lost in the shuffle. And so they presented the description of Linda to another group, but this time they presented only these possibilities:

Linda is active in the feminist movement.

Linda is a bank teller and is active in the feminist movement.

Linda is a bank teller.

To their surprise, 87 percent of the subjects in this trial also ranked the probability that Linda is a bank teller and is active in the feminist movement higher than the probability that Linda is a bank teller. And so the researchers pushed further: they explicitly asked a group of thirty-six fairly sophisticated graduate students to consider their answers in light of our first law of probability. Even after the prompting, two of the subjects clung to the illogical response.

The interesting thing that Kahneman and Tversky noticed about this stubborn misperception is that people will not make the same mistake if you ask questions that are unrelated to what they know about Linda. For example, suppose Kahneman and Tversky had asked which of these statements seems most probable:

Linda owns an International House of Pancakes franchise.

Linda had a sex-change operation and is now known as Larry.

Linda had a sex-change operation, is now known as Larry, and owns an International House of Pancakes franchise.

In this case few people would choose the last option as more likely than either of the other two.

Kahneman and Tversky concluded that because the detail “Linda is active in the feminist movement” rang true based on the initial description of her character, when they added that detail to the bank-teller speculation, it increased the scenario’s credibility. But a lot could have happened between Linda’s hippie days and her fourth decade on the planet. She might have undergone a conversion to a fundamentalist religious cult, married a skinhead and had a swastika tattooed on her left buttock, or become too busy with other aspects of her life to remain politically active. In each of these cases and many others she would probably not be active in the feminist movement. So adding that detail lowered the chances that the scenario was accurate even though it appeared to raise the chances of its accuracy.

If the details we are given fit our mental picture of something, then the more details in a scenario, the more real it seems and hence the more probable we consider it to be—even though any act of adding less-than-certain details to a conjecture makes the conjecture less probable. This inconsistency between the logic of probability and people’s assessments of uncertain events interested Kahneman and Tversky because it can lead to unfair or mistaken assessments in real-life situations. Which is more likely: that a defendant, after discovering the body, left the scene of the crime or that a defendant, after discovering the body, left the scene of the crime because he feared being accused of the grisly murder? Is it more probable that the president will increase federal aid to education or that he or she will increase federal aid to education with funding freed by cutting other aid to the states? Is it more likely that your company will increase sales next year or that it will increase sales next year because the overall economy has had a banner year? In each case, even though the latter is less probable than the former, it may sound more likely. Or as Kahneman and Tversky put it, “A good story is often less probable than a less satisfactory…[explanation].”

Kahneman and Tversky found that even highly trained doctors make this error.2 They presented a group of internists with a serious medical problem: a pulmonary embolism (a blood clot in the lung). If you have that ailment, you might display one or more of a set of symptoms. Some of those symptoms, such as partial paralysis, are uncommon; others, such as shortness of breath, are probable. Which is more likely: that the victim of an embolism will experience only partial paralysis or that the victim will experience both partial paralysis and shortness of breath? Kahneman and Tversky found that 91 percent of the doctors believed a clot was less likely to cause just a rare symptom than it was to cause a combination of the rare symptom and a common one. (In the doctors’ defense, patients don’t walk into their offices and say things like “I have a blood clot in my lungs. Guess my symptoms.”)

Years later one of Kahneman’s students and another researcher found that attorneys fall prey to the same bias in their judgments.3 Whether involved in a criminal case or a civil case, clients typically depend on their lawyers to assess what may occur if their case goes to trial. What are the chances of acquittal or of a settlement or a monetary judgment in various amounts? Although attorneys might not phrase their opinions in terms of numerical probabilities, they offer advice based on their personal forecast of the relative likelihood of the possible outcomes. Here, too, the researchers found that lawyers assign higher probabilities to contingencies that are described in greater detail. For example, at the time of the civil lawsuit brought by Paula Jones against then president Bill Clinton, 200 practicing lawyers were asked to predict the probability that the trial would not run its full course. For some of the subjects that possibility was broken down into specific causes for the trial’s early end, such as settlement, withdrawal of the charges, or dismissal by the judge. In comparing the two groups—lawyers who had simply been asked to predict whether the trial would run its full course and lawyers who had been presented with ways in which the trial might reach a premature conclusion—the researchers found that the lawyers who had been presented with causes of a premature conclusion were much more likely than the other lawyers to predict that the trial would reach an early end.

The ability to evaluate meaningful connections among different phenomena in our environment may be so important that it is worth seeing a few mirages. If a starving caveman sees an indistinct greenish blur on a distant rock, it is more costly to dismiss it as uninteresting when it is in reality a plump, tasty lizard than it is to race over and pounce on what turns out to be a few stray leaves. And so, that theory goes, we might have evolved to avoid the former mistake at the cost of sometimes making the latter.

IN THE STORY of mathematics the ancient Greeks stand out as the inventors of the manner in which modern mathematics is carried out: through axioms, proofs, theorems, more proofs, more theorems, and so on. In the 1930s, however, the Czech American mathematician Kurt Gödel—a friend of Einstein’s—showed this approach to be somewhat deficient: most of mathematics, he demonstrated, must be inconsistent or else must contain truths that cannot be proved. Still, the march of mathematics has continued unabated in the Greek style, the style of Euclid. The Greeks, geniuses in geometry, created a small set of axioms, statements to be accepted without proof, and proceeded from there to prove many beautiful theorems detailing the properties of lines, planes, triangles, and other geometric forms. From this knowledge they discerned, for example, that the earth is a sphere and even calculated its radius. One must wonder why a civilization that could produce a theorem such as proposition 29 of book 1 of Euclid’s Elements—“a straight line falling on two parallel straight lines makes the alternate angles equal to one another, the exterior angle equal to the interior and opposite angle, and the interior angles on the same side equal to two right angles”—did not create a theory showing that if you throw two dice, it would be unwise to bet your Corvette on their both coming up a 6.

Actually, not only didn’t the Greeks have Corvettes, but they also didn’t have dice. They did have gambling addictions, however. They also had plenty of animal carcasses, and so what they tossed were astragali, made from heel bones. An astragalus has six sides, but only four are stable enough to allow the bone to come to rest on them. Modern scholars note that because of the bone’s construction, the chances of its landing on each of the four sides are not equal: they are about 10 percent for two of the sides and 40 percent for the other two. A common game involved tossing four astragali. The outcome considered best was a rare one, but not the rarest: the case in which all four astragali came up different. This was called a Venus throw. The Venus throw has a probability of about 384 out of 10,000, but the Greeks, lacking a theory of randomness, didn’t know that.

The Greeks also employed astragali when making inquiries of their oracles. From their oracles, questioners could receive answers that were said to be the words of the gods. Many important choices made by prominent Greeks were based on the advice of oracles, as evidenced by the accounts of the historian Herodotus, and writers like Homer, Aeschylus, and Sophocles. But despite the importance of astragali tosses in both gambling and religion, the Greeks made no effort to understand the regularities of astragali throws.

Why didn’t the Greeks develop a theory of probability? One answer is that many Greeks believed that the future unfolded according to the will of the gods. If the result of an astragalus toss meant “marry the stocky Spartan girl who pinned you in that wrestling match behind the school barracks,” a Greek boy wouldn’t view the toss as the lucky (or unlucky) result of a random process; he would view it as the gods’ will. Given such a view, an understanding of randomness would have been irrelevant. Thus a mathematical prediction of randomness would have seemed impossible. Another answer may lie in the very philosophy that made the Greeks such great mathematicians: they insisted on absolute truth, proved by logic and axioms, and frowned on uncertain pronouncements. In Plato’s Phaedo, for example, Simmias tells Socrates that “arguments from probabilities are impostors” and anticipates the work of Kahneman and Tversky by pointing out that “unless great caution is observed in the use of them they are apt to be deceptive—in geometry, and in other things too.”4 And in Theaetetus, Socrates says that any mathematician “who argued from probabilities and likelihoods in geometry would not be worth an ace.”5 But even Greeks who believed that probabilists were worth an ace might have had difficulty working out a consistent theory in those days before extensive record keeping because people have notoriously poor memories when it comes to estimating the frequency—and hence the probability—of past occurrences.

Which is greater: the number of six-letter English words having n as their fifth letter or the number of six-letter English words ending in ing? Most people choose the group of words ending in ing.6 Why? Because words ending in ing are easier to think of than generic six-letter words having n as their fifth letter. But you don’t have to survey the Oxford English Dictionary—or even know how to count—to prove that guess wrong: the group of six-letter words having n as their fifth letter words includes all six-letter words ending in ing. Psychologists call that type of mistake the availability bias because in reconstructing the past, we give unwarranted importance to memories that are most vivid and hence most available for retrieval.

The nasty thing about the availability bias is that it insidiously distorts our view of the world by distorting our perception of past events and our environment. For example, people tend to overestimate the fraction of homeless people who are mentally ill because when they encounter a homeless person who is not behaving oddly, they don’t take notice and tell all their friends about that unremarkable homeless person they ran into. But when they encounter a homeless person stomping down the street and waving his arms at an imaginary companion while singing “When the Saints Go Marching In,” they do tend to remember the incident.7 How probable is it that of the five lines at the grocery-store checkout you will choose the one that takes the longest? Unless you’ve been cursed by a practitioner of the black arts, the answer is around 1 in 5. So why, when you look back, do you get the feeling you have a supernatural knack for choosing the longest line? Because you have more important things to focus on when things go right, but it makes an impression when the lady in front of you with a single item in her cart decides to argue about why her chicken is priced at $1.50 a pound when she is certain the sign at the meat counter said $1.49.

One stark illustration of the effect the availability bias can have on our judgment and decision making came from a mock jury trial.8 In the study the jury was given equal doses of exonerating and incriminating evidence regarding the charge that a driver was drunk when he ran into a garbage truck. The catch is that one group of jurors was given the exonerating evidence in a “pallid” version: “The owner of the garbage truck stated under cross-examination that his garbage truck was difficult to see at night because it was gray in color.” The other group was given a more “vivid” form of the same evidence: “The owner of the garbage truck stated under cross-examination that his garbage truck was difficult to see at night because it was gray in color. The owner remarked his trucks are gray ‘because it hides the dirt. What do you want, I should paint ’em pink?’” The incriminating evidence was also presented in two ways, this time in a vivid form to the first group and in a pallid version to the second. When the jurors were asked to produce guilt/innocence ratings, the side with the more vivid presentation of the evidence always prevailed, and the effect was enhanced when there was a forty-eight-hour delay before rendering the verdict (presumably because the recall gap was even greater).

By distorting our view of the past, the availability bias complicates any attempt to make sense of it. That was true for the ancient Greeks just as it is true for us. But there was one other major obstacle to an early theory of randomness, a very practical one: although basic probability requires only knowledge of arithmetic, the Greeks did not know arithmetic, at least not in a form that is easy to work with. In Athens in the fifth century B.C., for instance, at the height of Greek civilization, a person who wanted to write down a number used a kind of alphabetic code.9 The first nine of the twenty-four letters in the Greek alphabet stood for the numbers we call 1 through 9. The next nine letters stood for the numbers we call 10, 20, 30, and so on. And the last six letters plus three additional symbols stood for the first nine hundreds (100, 200, and so on, to 900). If you think you have trouble with arithmetic now, imagine trying to subtract image from image! To make matters worse, the order in which the ones, tens, and hundreds were written didn’t really matter: sometimes the hundreds were written first, sometimes last, and sometimes all order was ignored. Finally, the Greeks had no zero.

The concept of zero came to Greece when Alexander invaded the Babylonian Empire in 331 B.C. Even then, although the Alexandrians began to use the zero to denote the absence of a number, it wasn’t employed as a number in its own right. In modern mathematics the number 0 has two key properties: in addition it is the number that, when added to any other number, leaves the other number unchanged, and in multiplication it is the number that, when multiplied by any other number, is itself unchanged. This concept wasn’t introduced until the ninth century, by the Indian mathematician Mahāvīra.

Even after the development of a usable number system it would be many more centuries before people came to recognize addition, subtraction, multiplication, and division as the fundamental arithmetic operations—and slowly realized that convenient symbols would make their manipulation far easier. And so it wasn’t until the sixteenth century that the Western world was truly poised to develop a theory of probability. Still, despite the handicap of an awkward system of calculation, it was the civilization that conquered the Greeks—the Romans—who made the first progress in understanding randomness.

image

THE ROMANS generally scorned mathematics, at least the mathematics of the Greeks. In the words of the Roman statesman Cicero, who lived from 106 to 43 B.C., “The Greeks held the geometer in the highest honor; accordingly, nothing made more brilliant progress among them than mathematics. But we have established as the limit of this art its usefulness in measuring and counting.”10 Indeed, whereas one might imagine a Greek textbook focused on the proof of congruences among abstract triangles, a typical Roman text focused on such issues as how to determine the width of a river when the enemy is occupying the other bank.11 With such mathematical priorities, it is not surprising that while the Greeks produced mathematical luminaries like Archimedes, Diophantus, Euclid, Eudoxus, Pythagoras, and Thales; the Romans did not produce even one mathematician.12 In Roman culture it was comfort and war, not truth and beauty, that occupied center stage. And yet precisely because they focused on the practical, the Romans saw value in understanding probability. So while finding little value in abstract geometry, Cicero wrote that “probability is the very guide of life.”13

Cicero was perhaps the greatest ancient champion of probability. He employed it to argue against the common interpretation of gambling success as due to divine intervention, writing that the “man who plays often will at some time or other make a Venus cast: now and then indeed he will make it twice and even thrice in succession. Are we going to be so feeble-minded then as to affirm that such a thing happened by the personal intervention of Venus rather than by pure luck?”14 Cicero believed that an event could be anticipated and predicted even though its occurrence would be a result of blind chance. He even used a statistical argument to ridicule the belief in astrology. Annoyed that although outlawed in Rome, astrology was nevertheless alive and well, Cicero noted that at Cannae in 216 B.C., Hannibal, leading about 50,000 Carthaginian and allied troops, crushed the much larger Roman army, slaughtering more than 60,000 of its 80,000 soldiers. “Did all the Romans who fell at Cannae have the same horoscope?” Cicero asked. “Yet all had one and the same end.”15 Cicero might have been encouraged to know that a couple of thousand years later in the journal Nature a scientific study of the validity of astrological predictions agreed with his conclusion.16 The New York Post, on the other hand, advises today that as a Sagittarius, I must look at criticisms objectively and make whatever changes seem necessary.

In the end, Cicero’s principal legacy in the field of randomness is the term he used, probabilis, which is the origin of the term we employ today. But it is one part of the Roman code of law, the Digest, compiled by Emperor Justinian in the sixth century, that is the first document in which probability appears as an everyday term of art.17 To appreciate the Roman applications of mathematical thinking to legal theory, one must understand the context: Roman law in the Dark Ages was based on the practice of the Germanic tribes. It wasn’t pretty. Take, for example, the rules of testimony. The veracity of, say, a husband denying an affair with his wife’s toga maker would be determined not by hubby’s ability to withstand a grilling by prickly opposing counsel but by whether he’d stick to his story even after being pricked—by a red-hot iron. (Bring back that custom and you’ll see a lot more divorce cases settled out of court.) And if the defendant says the chariot never tried to stop but the expert witness says the hoof prints show that the brakes were applied, Germanic doctrine offered a simple prescription: “Let one man be chosen from each group to fight it out with shields and spears. Whoever loses is a perjurer and must lose his right hand.”18

In replacing, or at least supplementing, the practice of trial by battle, the Romans sought in mathematical precision a cure for the deficiencies of their old, arbitrary system. Seen in this context, the Roman idea of justice employed advanced intellectual concepts. Recognizing that evidence and testimony often conflicted and that the best way to resolve such conflicts was to quantify the inevitable uncertainty, the Romans created the concept of half proof, which applied in cases in which there was no compelling reason to believe or disbelieve evidence or testimony. In some cases the Roman doctrine of evidence included even finer degrees of proof, as in the church decree that “a bishop should not be condemned except with seventy-two witnesses…a cardinal priest should not be condemned except with forty-four witnesses, a cardinal deacon of the city of Rome without thirty-six witnesses, a subdeacon, acolyte, exorcist, lector, or doorkeeper except with seven witnesses.”19 To be convicted under those rules, you’d have to have not only committed the crime but also sold tickets. Still, the recognition that the probability of truth in testimony can vary and that rules for combining such probabilities are necessary was a start. And so it was in the unlikely venue of ancient Rome that a systematic set of rules based on probability first arose.

Unfortunately it is hard to achieve quantitative dexterity when you’re juggling VIIIs and XIVs. In the end, though Roman law had a certain legal rationality and coherence, it fell short of mathematical validity. In Roman law, for example, two half proofs constituted a complete proof. That might sound reasonable to a mind unaccustomed to quantitative thought, but with today’s familiarity with fractions it invites the question, if two half proofs equal a complete certainty, what do three half proofs make? According to the correct manner of compounding probabilities, not only do two half proofs yield less than a whole certainty, but no finite number of partial proofs will ever add up to a certainty because to compound probabilities, you don’t add them; you multiply.

That brings us to our next law, the rule for compounding probabilities: If two possible events, A and B, are independent, then the probability that both A and B will occur is equal to the product of their individual probabilities.Suppose a married person has on average roughly a 1 in 50 chance of getting divorced each year. On the other hand, a police officer has about a 1 in 5,000 chance each year of being killed on the job. What are the chances that a married police officer will be divorced and killed in the same year? According to the above principle, if those events were independent, the chances would be roughly 1/50 × 1/5,000, which equals 1/250,000. Of course the events are not independent; they are linked: once you die, darn it, you can no longer get divorced. And so the chance of that much bad luck is actually a little less than 1 in 250,000.

Why multiply rather than add? Suppose you make a pack of trading cards out of the pictures of those 100 guys you’ve met so far through your Internet dating service, those men who in their Web site photos often look like Tom Cruise but in person more often resemble Danny DeVito. Suppose also that on the back of each card you list certain data about the men, such as honest (yes or no) and attractive (yes or no). Finally, suppose that 1 in 10 of the prospective soul mates rates a yes in each case. How many in your pack of 100 will pass the test on both counts? Let’s take honest as the first trait (we could equally well have taken attractive). Since 1 in 10 cards lists a yes under honest, 10 of the 100 cards will qualify. Of those 10, how many are attractive? Again, 1 in 10, so now you are left with 1 card. The first 1 in 10 cuts the possibilities down by 1/10, and so does the next 1 in 10, making the result 1 in 100. That’s why you multiply. And if you have more requirements than just honest and attractive, you have to keep multiplying, so…well, good luck.

Before we move on, it is worth paying attention to an important detail: the clause that reads if two possible events, A and B, are independent. Suppose an airline has 1 seat left on a flight and 2 passengers have yet to show up. Suppose that from experience the airline knows there is a 2 in 3 chance a passenger who books a seat will arrive to claim it. Employing the multiplication rule, the gate attendant can conclude there is a 2/3 × 2/3 or about a 44 percent chance she will have to deal with an unhappy customer. The chance that neither customer will show and the plane will have to fly with an empty seat, on the other hand, is 1/3 × 1/3, or only about 11 percent. But that assumes the passengers are independent. If, say, they are traveling together, then the above analysis is wrong. The chances that both will show up are 2 in 3, the same as the chances that one will show up. It is important to remember that you get the compound probability from the simple ones by multiplying only if the events are in no way contingent on each other.

The rule we just applied could be applied to the Roman rule of half proofs: the chances of two independent half proofs’ being wrong are 1 in 4, so two half proofs constitute three-fourths of a proof, not a whole proof. The Romans added where they should have multiplied.

There are situations in which probabilities should be added, and that is our next law. It arises when we want to know the chances of either one event or another occurring, as opposed to the earlier situation, in which we wanted to know the chance of one event and another event both happening. The law is this: If an event can have a number of different and distinct possible outcomes, A, B, C, and so on, then the probability that either A or B will occur is equal to the sum of the individual probabilities of A and B, and the sum of the probabilities of all the possible outcomes (A, B, C, and so on) is 1 (that is, 100 percent). When you want to know the chances that two independent events, A and B, will both occur, you multiply; if you want to know the chances that either of two mutually exclusive events, A or B, will occur, you add. Back to our airline: when should the gate attendant add the probabilities instead of multiplying them? Suppose she wants to know the chances that either both passengers or neither passenger will show up. In this case she should add the individual probabilities, which according to what we calculated above, would come to 55 percent.

These three laws, simple as they are, form much of the basis of probability theory. Properly applied, they can give us much insight into the workings of nature and the everyday world. We employ them in our everyday decision making all the time. But like the Roman lawmakers, we don’t always use them correctly.

IT IS EASY TO LOOK BACK, shake our heads, and write books with titles like The Rotten Romans (Scholastic, 1994). But lest we become unjustifiably self-congratulatory, I shall end this chapter with a look at some ways in which the basic laws I’ve discussed may be applied to our own legal system. As it turns out, that’s enough to sober up anyone drunk on feelings of cultural superiority.

The good news is that we don’t have half proofs today. But we do have a kind of 999,000/1,000,000 proof. For instance, it is not uncommon for experts in DNA analysis to testify at a criminal trial that a DNA sample taken from a crime scene matches that taken from a suspect. How certain are such matches? When DNA evidence was first introduced, a number of experts testified that false positives are impossible in DNA testing. Today DNA experts regularly testify that the odds of a random person’s matching the crime sample are less than 1 in 1 million or 1 in 1 billion. With those odds one could hardly blame a juror for thinking, throw away the key. But there is another statistic that is often not presented to the jury, one having to do with the fact that labs make errors, for instance, in collecting or handling a sample, by accidentally mixing or swapping samples, or by misinterpreting or incorrectly reporting results. Each of these errors is rare but not nearly as rare as a random match. The Philadelphia City Crime Laboratory, for instance, admitted that it had swapped the reference sample of the defendant and the victim in a rape case, and a testing firm called Cellmark Diagnostics admitted a similar error.20 Unfortunately, the power of statistics relating to DNA presented in court is such that in Oklahoma a court sentenced a man named Timothy Durham to more than 3,100 years in prison even though eleven witnesses had placed him in another state at the time of the crime. It turned out that in the initial analysis the lab had failed to completely separate the DNA of the rapist and that of the victim in the fluid they tested, and the combination of the victim’s and the rapist’s DNA produced a positive result when compared with Durham’s. A later retest turned up the error, and Durham was released after spending nearly four years in prison.21

Estimates of the error rate due to human causes vary, but many experts put it at around 1 percent. However, since the error rate of many labs has never been measured, courts often do not allow testimony on this overall statistic. Even if courts did allow testimony regarding false positives, how would jurors assess it? Most jurors assume that given the two types of error—the 1 in 1 billion accidental match and the 1 in 100 lab-error match—the overall error rate must be somewhere in between, say 1 in 500 million, which is still for most jurors beyond a reasonable doubt. But employing the laws of probability, we find a much different answer.

The way to think of it is this: Since both errors are very unlikely, we can ignore the possibility that there is both an accidental match and a lab error. Therefore, we seek the probability that one error or the other occurred. That is given by our sum rule: it is the probability of a lab error (1 in 100) + the probability of an accidental match (1 in 1 billion). Since the latter is 10 million times smaller than the former, to a very good approximation the chance of both errors is the same as the chance of the more probable error—that is, the chances are 1 in 100. Given both possible causes, therefore, we should ignore the fancy expert testimony about the odds of accidental matches and focus instead on the much higher laboratory error rate—the very data courts often do not allow attorneys to present! And so the oft-repeated claims of DNA infallibility are exaggerated.

This is not an isolated issue. The use of mathematics in the modern legal system suffers from problems no less serious than those that arose in Rome so many centuries ago. One of the most famous cases illustrating the use and misuse of probability in law is People v. Collins, heard in 1968 by the California Supreme Court.22 Here are the facts of the case as presented in the court decision:

On June 18, 1964, about 11:30 a.m. Mrs. Juanita Brooks, who had been shopping, was walking home along an alley in the San Pedro area of the city of Los Angeles. She was pulling behind her a wicker basket carryall containing groceries and had her purse on top of the packages. She was using a cane. As she stooped down to pick up an empty carton, she was suddenly pushed to the ground by a person whom she neither saw nor heard approach. She was stunned by the fall and felt some pain. She managed to look up and saw a young woman running from the scene. According to Mrs. Brooks the latter appeared to weigh about 145 pounds, was wearing “something dark,” and had hair “between a dark blond and a light blond,” but lighter than the color of defendant Janet Collins’ hair as it appeared at the trial. Immediately after the incident, Mrs. Brooks discovered that her purse, containing between $35 and $40, was missing.

About the same time as the robbery, John Bass, who lived on the street at the end of the alley, was in front of his house watering his lawn. His attention was attracted by “a lot of crying and screaming” coming from the alley. As he looked in that direction, he saw a woman run out of the alley and enter a yellow automobile parked across the street from him. He was unable to give the make of the car. The car started off immediately and pulled wide around another parked vehicle so that in the narrow street it passed within six feet of Bass. The latter then saw that it was being driven by a male Negro, wearing a mustache and beard…. Other witnesses variously described the car as yellow, as yellow with an off-white top, and yellow with an egg-shell white top. The car was also described as being medium to large in size.

A few days after the incident a Los Angeles police officer spotted a yellow Lincoln with an off-white top in front of the defendants’ home and spoke with them, explaining that he was investigating a robbery. He noted that the suspects fit the description of the man and woman who had committed the crime, except that the man did not have a beard, though he admitted that he sometimes wore one. Later that day the Los Angeles police arrested the two suspects, Malcolm Ricardo Collins, and his wife, Janet.

The evidence against the couple was scant, and the case rested heavily on the identification by the victim and the witness, John Bass. Unfortunately for the prosecution, neither proved to be a star on the witness stand. The victim could not identify Janet as the perpetrator and hadn’t seen the driver at all. John Bass had not seen the perpetrator and said at the police lineup that he could not positively identify Malcolm Collins as the driver. And so, it seemed, the case was falling apart.

Enter the star witness, described in the California Supreme Court opinion only as “an instructor of mathematics at a state college.” This witness testified that the fact that the defendants were “a Caucasian woman with a blond ponytail…[and] a Negro with a beard and mustache” who drove a partly yellow automobile was enough to convict the couple. To illustrate its point, the prosecution presented this table, quoted here verbatim from the supreme court decision:

Characteristic

Individual Probability

Partly yellow automobile

1/10

Man with mustache

1/4

Negro man with beard

1/10

Girl with ponytail

1/10

Girl with blond hair

1/3

Interracial couple in car

1/1,000

The math instructor called by the prosecution said that the product rule applies to this data. By multiplying all the probabilities, one concludes that the chances of a couple fitting all these distinctive characteristics are 1 in 12 million. Accordingly, he said, one could infer that the chances that the couple was innocent were 1 in 12 million. The prosecutor then pointed out that these individual probabilities were estimates and invited the jurors to supply their own guesses and then do the math. He himself, he said, believed they were conservative estimates, and the probability he came up with employing the factors he assigned was more like 1 in 1 billion. The jury bought it and convicted the couple.

What is wrong with this picture? For one thing, as we’ve seen, in order to find a compound probability by multiplying the component probabilities, the categories have to be independent, and in this case they clearly aren’t. For example, the table quotes the chance of observing a “Negro man with beard” as 1 in 10 and a “man with mustache” as 1 in 4. But most men with a beard also have a mustache, so if you observe a “Negro man with beard,” the chances are no longer 1 in 4 that the man you observe has a mustache—they are much higher. That issue can be remedied if you eliminate the category “Negro man with beard.” Then the product of the probabilities falls to about 1 in 1 million.

There is another error in the analysis: the relevant probability is not the one stated above—the probability that a couple selected at random will match the suspects’ description. Rather, the relevant probability is the chance that a couple matching all these characteristics is the guilty couple. The former might be 1 in 1 million. But as for the latter, the population of the area adjoining the one where the crime was committed was several million, so you might reasonably expect there to be 2 or 3 couples in the area who matched the description. In that case the probability that a couple who matched the description was guilty, based on this evidence alone (which is pretty much all the prosecution had), is only 1 in 2 or 3. Hardly beyond a reasonable doubt. For these reasons the supreme court overturned Collins’s conviction.

The use of probability and statistics in modern courtrooms is still a controversial subject. In the Collins case the California Supreme Court derided what it called “trial by mathematics,” but it left the door open to more “proper applications of mathematical techniques.” In the ensuing years, courts rarely considered mathematical arguments, but even when attorneys and judges don’t quote explicit probabilities or mathematical theorems, they do often employ this sort of reasoning, as do jurors when they weigh the evidence. Moreover, statistical arguments are becoming increasingly important because of the necessity of assessing DNA evidence. Unfortunately, with this increased importance has not come increased understanding on the part of attorneys, judges, or juries. As explained by Thomas Lyon, who teaches probability and the law at the University of Southern California, “Few students take a probability in law course, and few attorneys feel it has a place.”23 In law as in other realms, the understanding of randomness can reveal hidden layers of truth, but only to those who possess the tools to uncover them. In the next chapter we shall consider the story of the first man to study those tools systematically.