Total Recall: How the E-Memory Revolution Will Change Everything - C. Gordon Bell, Jim Gemmell (2009)

Part I

Chapter 3. THE MEETING OF E-MEMORY AND BIO-MEMORY

I was invited to a birthday party for computer graphics pioneer Ivan Sutherland. Would I be willing to say a few words?

Great, I thought. I can get up and tell them I run into Ivan about every five years, and we enjoy our conversations.

Honestly, that was all that came to me. Then I entered Ivan’s name into the search window in MyLifeBits, and to my surprise and relief, I immediately recalled emotionally evocative and intellectually intriguing details I had completely forgotten. Around 1963, when Ivan had been a first lieutenant stationed at the National Security Administration, he and I designed a display computer. A few years later, he had been instrumental in my becoming a professor at Carnegie Mellon University. Much more recently, Ivan had started opening his talks with “Gordon Bell and I have a friendly debate over whether I’ve wasted my life working on asynchronous logic.” (Come to think of it, I hope he’ll take my thousand-dollar bet that asynchronous logic will be widely recognized as a waste of time by 2020).

My biological memory had reduced my relationship with Ivan down to the humdrum, but my e-memory stepped in to restore the significance of our history, making it possible for me to compose a fitting toast for his birthday.

We all want better recall. The market for memory enhancement books, elixirs, computer programs, devices, and games is gigantic.

As people get older, they start to get paranoid about small memory lapses. When a forty-year-old misplaces his car keys, he feels annoyed. When a sixty-five-year-old loses the keys, he starts Googling about Alzheimer’s disease. In his search he might read about another condition known as mild cognitive impairment, which afflicts as much as 5 percent of the population past the age of seventy. It’s very real, and very scary.

The fear of oblivion before death is big enough to drive a $4.2 billion industry in medicinal herbs and supplements for memory enhancement. Health-food emporium shelves are stocked with herbs, micronutrients, antioxidants, tonics, supplements, and potions to boost your brainpower. Labels on bottles of coenzyme Q10, ginseng, ginkgo biloba, rosemary, and salvia promise to keep your mind nimble.

In 2007, the U.S. market for brain-fitness programs and “neuro-software” was $225 million. Nintendo sells a product called Brain Age that claims it can help you “[get] the most out of your prefrontal cortex!” The software program MindFit combines cognitive assessment of more than a dozen different skills with a personalized training regimen based on that assessment. Dr. Michael Merzenich, a neuroscientist at the University of California in San Francisco, developed a set of computer tools called the Brain Fitness Program intended to increase processing speeds in aging brains. And for about ten dollars a month, you can subscribe to Web sites like Lumosity.com and Happy-Neuron.com to tap into a variety of cognitive training exercises.

You can also buy books on how to exercise your brain with games, puzzles, and memory tricks. You can learn about the biology and behavior of memory from books such as In Search of Memory: The Emergence of a New Science of Mind by Nobel Laureate Eric Kandel and The Seven Sins of Memory: How the Mind Forgets and Remembers by Harvard University psychologist Daniel Schacter.

For all I know, ginkgo biloba and the Brain Fitness Program will indeed improve your bio-memory. But the world of Total Recall promises something broader: a revolution.

THE INESCAPABLE FRAILTY OF BIO-MEMORY

The memories in our brains are stored as patterns of connections between neurons, or nerve cells. Computers store information in a series of microscopic switches turned on or off. Brains and computers both store information, manipulate it, and use it to decide between courses of action. For these reasons we say that both systems have “memory,” but this similarity only holds up in the first approximation. Scratch the surface and you find vast differences between biological and digital memory.

To the owner of a human brain, memory feels like a single resource. It turns out this feeling is an illusion. Scientists who study biological memory describe three distinct systems:

✵ Procedural memory, sometimes known as muscle memory, is for physical skills such as riding a bike, ballet dancing, typing.

✵ Semantic memory encodes meanings, definitions, and concepts—facts that you know that aren’t rooted in time or place, such as “A cat has four legs” or “The capital of Japan is Tokyo.”

✵ Episodic memory, sometimes known as autobiographical memory, encodes experiences from your past. This is what allows you to know about and reexperience the things that have happened in your life, such as the time you sprained your ankle at the playground and your father bought you an ice-cream sundae to make you feel better, or the shower you took an hour ago.

Nothing is coming soon that can help us with our procedural memory. But our semantic and episodic bio-memories can and will be extended by our e-memories.

We all know that biological memory is fallible, but it’s still extremely unnerving to learn just how true this is. As neuroscientists have shown, episodic memories feel a lot more fleshed out and precise than they really are. Unlike computers, brains aren’t all that great at faithfully storing masses of detail. What brains are best at storing are patterns, meanings, and gestalts. The act of remembering an event from your past is less like playing back a mental videotape in your mind’s home theater system than it is like telling a story based on a few relevant facts.

In an age of Total Recall, anything, even everything, is easily recorded accurately into your e-memory. Your brain can’t do this. When it lays down a new memory of an experience, what it actually encodes is a sparse constellation of authentic details and salient junctures. When your brain retrieves the memory later, it uses that constellation as a scaffold for reconstructing the original experience. As the memory plays out in your mind you may have the strong impression that it’s a high-fidelity record, but only a few of its contents are truly accurate. The rest of it is like a bunch of props, backdrops, casting extras, and stock footage.

When a friend tells you a five-minute-long humorous story, the memory you come away with isn’t the exact sequence of words he uttered. When you repeat the “same” story to your friends at work on Monday, what you actually do is reconstruct it in your own way according to the same pattern and meaning. You follow the overall map provided by those key junctures you memorized, but you freely embellish and fill in any gaps to make the story flow smoothly between them. You might repeat verbatim a few key bits of the original telling, but most of the word choices are yours. Generally, all I can remember of a joke is the great punch line that made me laugh, and I have to reinvent the rest in order to share it.

And it gets even stranger. Sometimes a feature that was confabulated during one act of remembering gets reremembered during the next act. In the process, the confabulation can become a permanent element of the bio-memory. Here’s how the eminent neuroscientist Joseph LeDoux sees it:

Like many scientists in the field of memory, I used to think that a memory is something stored in the brain and then accessed when used. Then, in 2000, a researcher in my lab [convinced us] that our usual way of thinking was wrong. In a nutshell … each time a memory is used, it has to be restored as a new memory in order to be accessible later. The old memory is either not there or is inaccessible. In short, your memory about something is only as good as your last memory about it. This is why people who witness crimes testify about what they read in the paper rather than what they witnessed.

False memories can have tragic consequences. In the late 1980s and early 1990s, thousands of families were ripped apart when adult children claimed that they had recovered long-repressed memories of sexual molestation when they were little. It turned out many of these “memories” had been coaxed into being by gullible, credulous therapists who hadn’t realized what they were doing.

Most of our memories are not grossly altered as our brain repeatedly remembers them, but all of us harbor at least some memories that have been radically revised, and all of our memories are susceptible to gradual mutation and drift.

That is about to change.

E-MEMORY TRANSFORMS BIO-MEMORY

Biological memory is subjective, patchy, emotion-tinged, ego-filtered, impressionistic, and mutable. Digital memory is objective, dispassionate, prosaic, and unforgivingly accurate.

In our brains, memory, attention, and emotion conspire to warp, compress, and edit time and life experience in many ways. A video camera, the eye of an e-memory, in contrast, never blinks or winces, never drifts into a daydream or does a double take. A camera will record an hour of pedestrian crosswalk traffic with the same fidelity as it will witness an hour of bloody genocide.

E-memory will be the fact checker for those meanings, definitions, and concepts in our semantic memories. You probably already use Google or Wikipedia to look things like this up, when you can. But not everything you know is easy to find on the Web, or may not even be there. It will be there in your e-memory. And it will be easier to find, because you will be searching just your own memories, not the whole Web. I often find it is faster to use MyLifeBits to track down obscure facts I know I’ve been exposed to before but can’t recall directly, simply because I’ll often remember when or where or from whom I heard the thing I’m trying to recall.

Everyone knows the anxiety and frustration of not being able to remember someone’s name. With MyLifeBits I often track down a name using clues I do remember. I recently wanted to find the name of a fellow who nearly contributed to the Computer History Museum in 1983. I recalled the company he worked for. I thought he came to a lecture at the museum that same year. I wondered if his name was on the list of attendees, a copy of which I had kept in a box for years and then scanned… . Yes! In general, if I know that my e-memory has it, I will usually find it within a minute or two.

E-memory will become vital to our episodic memory. As you live your life, your personal devices will capture whatever you decide to record. Bio-memories fade, vanish, merge, and mutate with time, but your digital memories are unchanging. And e-memories will contain an unprecedented level of detail. With my bio-memory, I struggle to recall exactly when I was in San Francisco last year. With my GPS logged, I can recall the exact time of my walking down each individual street in San Francisco.

Total Recall will change how we think about our lives. It will also change how we feel about our lives.

Consider just one photo that popped up on my screensaver while I worked on this book. I glanced at it and was transported back to my fourth birthday in 1938. Mother told me I could invite anyone I wanted to my party, and so I did. We were an eclectic bunch, all eighteen of us ranging in age from two to fourteen. I’m in the middle front with a large sheet cake on my lap. It’s obvious that I’ve got more important things on my mind than getting my picture taken, like sticking my pudgy fingers into the creamy frosting, which although white, hid a middle that was pure devil’s food.

The faces in the picture spark recollections, like the really cute three-year-old girl from across the street, sitting in the front row with me. When my sister was born two years later, I picked the name Sharon, after that little girl, my first sweetheart, Sharon Lee. We were framed by my older teenage cousins, one with his hands in his pockets, looking ever so cool, while the other was praying for the photograph to be over. Glancing at each face, I was struck by one in particular. His name was Joe Bill, the minister’s son. He died a year later at the age of ten and it bothers me as much today as it did then.

This single picture from 1938 initiates an avalanche of memories, each connected to the other, via associations established in my brain decades ago. They elicit feelings of pleasure and sadness. Each strand tugs on a dozen others, all of them connected into the vast web of memories that make me uniquely me.

I recall a research demonstration at Microsoft headquarters of a huge grid of LCD monitors, three high and six wide, all filled with a time line displaying photos from my life. I stood transfixed for several minutes in front of this biographical vista, soaking in the perspectives and the details. Seeing so much of my life, all at once, was profoundly moving.

E-memories will not be trapped back in cabinets and shoe boxes. They will be on our end tables and walls. They will follow us on our travels. They will keep us company, showing us friendly faces, letting us hear cherished voices. E-memory will be an intimate extension of bio-memory. And change it into something new.

RECALLING WHAT MATTERS

I don’t know about you, but sometimes I’m absentminded. I forget where I put things. Sometimes, coming out of the airport terminal, I have a momentary flash of panic. Where did I park the car? Was it level one or two? I hate getting home from the grocery store, looking up at that burned-out lightbulb, and realizing that I forgot—again!—to buy a replacement.

Once I left my notebook computer containing most of my e-memory on the security table at San Francisco International Airport. I dashed back, my heart racing dangerously, wondering if someone had walked off with a digital copy of my life. Thankfully, it was still there. Then I forgot the computer again at the Dulles Airport security, and didn’t realize my mistake until I had boarded the plane and it was too late to go back. I managed to have it over nighted to me for $150, and all I could think was that I would gladly have paid many times that amount to ensure no one else had my data. More than a half million of my fellow Americans also left their computers at checkpoints in 2008.

A busy person may be plagued by absentmindedness, simply because he has a lot on his mind. You forgot to bring home the milk, because by the time you got to the grocery store you were thinking about the items needed for your pets. You forgot your lunch meeting, because you had just gotten off the phone with a colleague and were engrossed with new ideas for your next project.

Reminders must be made at the appropriate time; it is no good having a shopping list that is back at your office while you are in the grocery store. A reminder to make a phone call while you are wasting time in rush-hour traffic would be priceless. Likewise, you must be able to create reminders anytime, anyplace, or they may be lost. If you think of something that needs doing while driving, it will not suffice to have to wait a half hour to get home and write it down. Probably by then you will have thought of three other things that need doing and forgotten at least one.

This is why e-memory will be in the cloud, accessible anywhere, anytime. We want to be able to type or speak notes and to-do items whenever they occur to us. The to-do items will be associated with a time, place, or mode of activity. For example, the task “Buy milk ” is associated with the grocery store (a place). The task “E-mail Catherine” is associated with using your computer (an activity). The task “Pick up Suzy at 4:00” is associated with a time. If you are struck by the thought that a cell phone is capable of knowing time and place, and can input text, voice, and pictures, then you realize how close we are to the reality of this vision. All we need is a little more software that can understand such things as milk being available at grocery stores.

In addition to giving you all the right reminders, it will not be too long before your e-memories will fill in your other absentminded gaps. Your increasingly location-aware cell phone will remind you where you parked your car. You will track where you have left things like your glasses, either by noting where your devices last detected their RFID tag, or by taking pictures of them. When your mind is absent, your e-memory will always be there.

Having too much on my mind doesn’t just make me absentminded; it can make me feel mentally cluttered, impeding my productivity. David Allen’s popular book and seminar series Getting Things Done stands on the central premise that we are hindered by mental clutter:

First of all, if it’s on your mind, your mind isn’t clear. Anything you consider unfinished in any way must be captured in a trusted system outside your mind… .

Unlike your bio-memory, your e-memory will never be overwhelmed. Total Recall software can make sure you are protected from clutter. For instance, if I were to show you all my 150,000 recorded Web pages, you would see that nearly half are duplicates, or are near duplicates that differ by just a tiny amount from some other page. A natural reaction would be to delete all the near duplicates to eliminate clutter. However, e-memory never needs to suffer from clutter; only a poor recall interface looks cluttered. Good recall software could simply group all the near duplicates together and show a single representative in response to your queries. Suppose I have repeatedly gone to the same Web page and the only thing that changed on it was the advertisements. Most of the time, I just want to see one page, hiding away the clutter of all the extra copies with inconsequential differences. But on that one day that I want to recall an ad for 50 percent off a new GPS, suddenly the differences are very consequential—and I’ve still got them.

Total Recall software will hide away all the clutter as if it had been discarded, but whenever you actually want it, it will be at your fingertips. You will have all the advantages of complete retention, with none of the downside of clutter.

And e-memories are not depressed by dealing mostly in the mundane. One thing you’ll notice when perusing a lifelog is the sheer banality of 99 percent of life. Television producers periodically demonstrate this when they have a camera crew live with a family for a long time or Reality TV records a group surviving on an island twenty-four/seven. You will very quickly come to appreciate just how mind-numbingly dull, trite, predictable, tedious, and prosaic most of our life moments really are. Life as it appears in objective playback is tedium to the dullth power. But that’s not a problem for an e-memory. You know that you’ll never want or need to look back at virtually all of it ever again. You’ll also know that nothing important will be missed—just as Cathal Gurrin has that special moment of first meeting his girlfriend.

The team that Cathal works with at Dublin City University has attacked the banality of lifelogging by creating software that looks for novelty. It works something like this: Suppose your GPS says you were at the same place as usual this morning for breakfast, and the images look very similar to those taken most other mornings. Then it was probably the mundane, same old breakfast at home as usual. On the other hand, if you were at an unusual place for breakfast, or more faces were around the table than usual, that is more interesting. Cathal and his colleagues take thousands of SenseCam images, and boil them down to a presentation that highlights the unusual. The mundane is still there, but tucked away so that it doesn’t clutter up what is interesting.

Total Recall software will get better and better at spotting the interesting moments for you as we acquire more data such as your pulse, the pitch and volume of your voice, or even the brain waves you are emitting. Researchers at the University of Tokyo have already demonstrated a baseball cap that records both video and alpha waves from your brain. Based on the alpha waves, they can do a pretty good job of guessing what segments of the video were interesting.

For an e-memory, there’s no drawback to capturing the long stretches of banality that comprise most of life, there are only potential benefits. Go ahead and fill it with et ceteras, so ons, and ad nauseams. The more you record, the better.

Your mind can be freed from mundane memorization. Let your e-memory remember each detail, and show you the average, the maximum, the chart, the patterns, or the unusual. Then when you decide that one particular area bears further investigation, you can recall all the gory details—perhaps the actual data points, or some additional photos. Knowing that your e-memory has the task of perfectly remembering allows you to concentrate on more interesting things.

E-memories not only relieve you of memorizing what you don’t care much about, they can also help your bio-memory remember what counts.

My friend Sunil Vemuri is the CTO of reQall, a really fascinating memory-aid product. I use reQall to create reminders and notes for myself. I call up their phone number and say, “Add.” There’s a beep and then I make some comment, perhaps, “Book the flights for my vacation.” I can retrieve this reminder by calling in later and saying, “Recall,” instead of “Add.” I also get an e-mail, with a recording of what I said, and even a transcription of what I said. Besides using the phone and e-mail, I can use other interfaces like a Web browser and instant messenger. Being able to create and retrieve memos in all these ways is very powerful. I even use the phone to create dear-diary entries: I can be driving along, place a call on my cell phone, and tell a little story that ends up transcribed in my e-mail and also with my own voice recorded.

All that was exciting enough to get me to join reQall’s board, but Sunil expanded my vision of what is possible when he visited Jim Gemmell a little while ago. They sat down in Gemmell’s office and were chatting about all kinds of Total Recall ideas (when Sunil was still a graduate student at MIT he got involved in the CARPE research community, so he’s been a coconspirator with us for years). In the course of the conversation, Sunil made reference to Gemmell’s family, and even what school Gemmell’s oldest son attends. Gemmell had only mentioned this once, and that was more than a year ago, so he was astonished at Sunil’s memory. But Sunil gave up his secret later.

“Do you know how I remembered about your family and your son’s school?” he asked.

Gemmell shook his head and Sunil went on. “After our last meeting, I called reQall and spoke some notes about our meeting, including those facts about your family. Now, I also have reQall programmed to play random facts to me every so often. Since our last meeting, I’ve heard those facts about your family a couple of times, so now I remember them.”

This kind of memory refresh is the driving factor behind another product called SuperMemo. Instead of just randomly remind ing you, SuperMemo considers the typical pattern of memory loss. Cognitive scientists have measured how memories typically fade, and can plot the odds of your forgetting something after first hearing it, after one reminder, after two reminders, and so on. Supermemo intervenes when the odds of your losing the memory reach a certain level, say 15 percent. For instance, two days after hearing something, you might be projected to have a 15 percent chance of forgetting the fact, so you are reminded. Eight days later, you again are projected to have more than 15 percent odds of forgetting, so you are reminded again, and so on, with the time between reminders growing longer and longer.

RETAINING PAINFUL MEMORIES

When I give talks about MyLifeBits, someone in the audience usually says something like, “But isn’t forgetting a good thing sometimes? Isn’t the idea of recording our lives in excruciating detail actually a rather bad idea? Don’t we need to forget?”

Everyone has experienced embarrassing moments he’d rather forget. You’re the quarterback who ran the wrong way down the football field. You called your lover by the name of an ex-flame. One of my most embarrassing memories was shown on the Business Channel in 1983. I was one of a dozen company founders sitting at a table, explaining to the press our plan of merging with another company to turn our fortunes around. There was only one hitch with this ill-fated plan: We had nothing to sell. I have nothing but scorn for product announcements without an actual product, and here I was in the thick of it. Every time I think of it I feel an echo of the original horror I felt clamoring around in my gut—which is why I try not to think about it very much.

But you can easily avoid replaying such memories. And who knows? Maybe someday when you’re old and gray and retired and you look back across your life with the expanded perspective that often comes in the winter of life, you might actually be able to look back at your old fumbles, gaffes, faux pas, and humiliations and gain closure on them, forgive yourself for them, even laugh at the ultimately petty little anxieties that used to seem so serious. In a perverse way I would love to have a copy of the Business Channel tape just to see whether I was crawling under the table just as I wish to remember the moment. I know I don’t mind watching a 1972 video of me making a shortsighted prediction about where computing was headed.

But what about the really, truly bad memories? Not the harmless embarrassments that still make you blush, but the ones that are so compromising or potentially harmful—to your reputation, to your loved ones, or to your own sanity—that you just can’t abide the thought of keeping them?

What about a woman who’s been abused by her husband? When she finally escapes him and gets help putting a new life together, what could be more unwelcome than digital records of the horror that had been her life? The last thing she wants to do is relive the insults, the threats, the cat-and-mouse mind games, the screams, the beatings, and the bruises. It would only be natural for her to want to delete every last bit of it.

What about a young man who makes some bad decisions in high school? He makes the wrong friends, starts experimenting with drugs, and ends up in a stoned stupor in the backseat of a stolen car. He is arrested, and the experience with the juvenile justice system scares him straight. The state expunges his juvenile record and he goes on to become a law-abiding citizen with a family and a profession. In a world with no records, he could easily leave the past behind. In a world where most things are recorded and saved, would he have the same chance?

Nevertheless, I still advocate keeping everything, even the worst of it. They are your e-memories; you control the keys to them. Rather than erase them, you can seal them up. You can put a lock on those events you’d like to forget and never open them up again. What you really want to prevent in these cases is unwanted recall, not retention.

Imagine the abused woman has audio and video recordings of the abuse she endured. She has escaped, gotten treatment, and is living in a new city without fear. Her recordings can easily be locked so that they’ll never come up in regular interactions with her e-memory. But she may want keep them for legal proceedings. Or she may want to share them with future therapists.

Imagine the young man who was arrested, now grown older and involved in a community effort to block a new commercial development. His opponents start to circulate stories that he was a hard-core criminal in his youth, with gang connections lasting to this day. He finds it in his interest to show his youth record to defend himself against this slander.

Our impulse to hit the delete key may not be the right move to lock away the past. Daniel Schacter advises that “confronting, disclosing, and integrating those experiences we would most like to forget is the most effective counter to [unwanted recall].”

LOST BUT NOT FORGOTTEN

I have personal experience with unwanted recall using MyLifeBits. On Sunday January 28, 2007, my manager and dear friend Jim Gray took his forty-foot yacht, Tenacious, on a solo sailing trip out to the Farallon Islands near San Francisco. He went to scatter his mother’s ashes in the wild seascape around the rugged islands.

But Jim never returned. Despite clear weather and no signs of distress from his well-equipped yacht, Jim mysteriously vanished. A massive three-week search did not produce a single clue as to what had happened. As The New York Times reported, “A veritable Who’s Who of computer scientists from Google, Amazon, Microsoft, NASA, and universities across the country spent sleepless nights writing ad hoc software, creating a blog, and reconfiguring satellite images so that dozens of volunteers could pore over them, searching for a speck of red hull and white deck among a sea of gray pixels.”

For several months after Jim’s disappearance, I was deeply disturbed every time a picture of him came up on my screensaver. I avoided going into the office where we had worked. I was overcome with emotion. It was too painful for me to see his smiling face. Some people in my situation would have deleted those pictures, hoping it would bring relief or catharsis. But with time, my frame of mind changed. The same pictures now bring back happy memories and nourish my spirit. I’m glad I still have so many images of my old friend.

I used some of the pictures when I was asked to speak at Jim’s memorial service. I knew my emotions would get the better of me on the day, so I planned to get myself out of an actual speaking role. I used Microsoft’s Movie Maker to load my collected snapshots of Jim. The software allowed me to drag and drop in special effects, such as fading from one photograph to the next. I wrote a script that I voiced-over the pictures, and concluded with Pink Floyd’s “Wish You Were Here.”

It took more than a week to create. When it was shown at the service, tears welled up in eyes all over the auditorium, including mine. My e-memory, working alongside my bio-memory, told the story of Jim and me, how our paths had crossed, and how rich my life became because of him.

There are many ways to create stories. My mother spent several months typing the story of her family, the Gordons (I’m named after her). It’s about twenty-five pages long and it’s chock ful of stories about her family and about growing up as wonderful twentieth-century inventions like the automobile, electricity, and telephony became a part of her life—things we would have never known without those twenty-five pages. There’s only one thing wrong with it. I should have had her record it in her own voice.

That’s the kind of omission that will soon be a thing of the past.