Total Recall: How the E-Memory Revolution Will Change Everything - C. Gordon Bell, Jim Gemmell (2009)

ANNOTATED REFERENCES AND RESOURCES

This section is for people familiar with computer technology, or any reader with enough curiosity to dig deeper into the ideas behind Total Recall. Those eager to get started with Total Recall will find some useful pointers as well. In addition to citations for material in the book chapters, there are references and discussions of related topics that haven’t been covered yet. Here you can find references to printed publications, Web sites, people, products, conferences, and research labs.

The section is arranged by chapter, and the order of material follows the order of the chapter as much as possible.

1. THE VISION

Ray Ozzie is quoted from personal correspondence with the Authors. Other references on cloud computing:

Hayes, B. 2008. “Cloud Computing.” ACM, Communications of the ACM 51, Issue 7 ( July).

Gruman, Galen, and Eric Knorr. 2008. “What Cloud Computing Really Means. InfoWorld (April 7).

Martin, Richard, and J. Nicholas Hoover. 2008. “Guide to Cloud Computing.” Information Week (June 21).

Amazon Elastic Compute Cloud (Amazon EC2). http://aws.amazon.com/ec2Microsoft Azure Services Platform. http://www.microsoft.com/azure

Science fiction that grapples with e-memories:

Sawyer, Robert J. 2003. Hominids. New York: Macmillan.

Halperin, James. 2000. The Truth Machine. New York: Ballantine Books.

Naim, Oscar. 2004. The Final Cut. Lions Gate Entertainment.

Westbrook, Robert. 2004. The Final Cut. New York: Penguin.

Another related sci-fi work is The Observers, where alien microrobots have been recording Earth’s history in incredible detail. People in the book grapple with being watched and recorded. The aliens are able to copy all of the information related to a person to create a virtual person, raising the issue of digital immortality.

Williamson, S. Gill. 2006. The Observers. Lincoln, Neb.: iUniverse.

Don Norman suggested the Teddy life recorder. His other books on design are also well worth reading.

Norman, Donald A. 1992. Turn Signals Are the Facial Expression of Automobiles. Reading, Mass.: Addison-Wesley.

The Millennials, aka Generation Y:

Howe, Neil, and William Strauss. 2000. Millennials Rising: The Next Great Generation. New York: Random House.

Laurent, Anne. 2008. “Millennials: They’re Here. They’re Wired. Get Used to Them.” Tech Insider (March 24) http://techinsider.nextgov.com

Safer, Morley. 2007. “The ‘Millennials’ Are Coming.” CBS 60 Minutes (May 25). http://www.cbsnews.com/stories/2007/11/08/60minutes/main3475200.shtml

Olsen, Stefanie. 2005. “The ‘Millennials’ Usher in a New Era.” CNET News.com. (November 18).

Sweeney, Richard. 2006. “Millennial Behaviors and Demographics.” http://library2.njit.edu/staff-folders/sweeney/Mi l len n ia ls/A rticle-Millennial-Behaviors. doc

Strauss and Howe have an interesting generational theory that could play into the issue of why the Millennials seem to have a different attitude to privacy and technology.

Strauss, William, and Neil Howe. 1997. The Fourth Turning. New York: Broadway Books.

———. 1991. Generations: The History of America’s Future, 1584 to 2069. New York: William Morrow and Company.

Abe Crystal’s Ph.D. study found that “although all the students I observed were generally comfortable with technology, there was a large variance in technology-related expertise and knowledge.” Gibbons and Foster (2007) were “surprised to find that students are on average no more proficient with computer technology than are librarians and faculty members. Some students demonstrated broad knowledge of computers and facility in using them, but others were awkward and clumsy.”

Crystal, Abe. 2008. Design research for Personal Information Management systems to support undergraduate students, doctoral dissertation, University of North Carolina at Chapel Hill.

The International Technology Roadmap for Semiconductors is the single most important projection about the future of semiconductors. It encompasses all memory, including nonvolatile Flash memory, processors, and radios. The 2007 Roadmap projects continued biennial doubling of semiconductor density until 2016. One implication is that semiconductor memories will replace disks for portable computers.

International Technology Roadmap for Semiconductors Web site. http://www.itrs.net

Disk Storage Roadmaps are available from individual vendors and market intelligence firms like IDC that show nearly annual doubling of disk densities.

Arai, Masayuki. 2009. “Optical Disks Used for Long-Term Storage by 2010.” Tech-On! (March 6).

Rydning, John, and Jeff Janukowicz. 2009. “Worldwide Hard Disk Drive Component 2008-2012 Forecast Update.” IDC (February 1).

There is a large research community advancing work on data mining, pattern recognition, and machine learning. Here are just a few starting points:

Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. New York: Springer.

Kargupta, H., et al. (eds.). 2009. Next Generation of Data Mining. London: Chapman and Hall.

ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

International Conference on Data Mining (ICDM).

SIAM International Conference on Data Mining.

Total Recall predicts the future based on technology trends. A similar book in this genre is Being Digital, which did a wonderful job in 1995 of predicting our digital lives today.

Negroponte, Nicholas. 1995. Being Digital. New York: Alfred A. K nopf.

2. MY LIFE BITS

Million Books Project (also called the Universal Library Project) Web site. http://www.ulib.org

Project Gutenberg Web site. http://www.gutenberg.org/wiki/Main_Page

Foer, Jonathan Safran. 2006. Extremely Loud and Incredibly Close. New York: Mariner Books.

Bill Gates and Jim Gray were inspirations to us.

Gates, Bill. 1996. The Road Ahead. New York: Penguin Books.

Gray, Jim. 1999. “What’s Next? A Dozen Information-Technology Research Goals.” Journal of the ACM 50:41-57.

About the Aaron painting program:

Cohen, Harold. 1995. “The Further Exploits of AARON, Painter.” Stanford Humanities Review 4, issue 2 (July): Constructions of the Mind. http://www.stanford.edu/group/SHR/4-2/text/cohen.html

Memex was proposed by Bush in his Atlantic Monthly article.

Bush, Vannevar. 1945. “As We May Think.” Atlantic Monthly (July). Reprinted in Life magazine, September 10, 1945. http://www.theatlantic.com/doc/194507/bush

This book tells you much more about Bush, his life, and his amazing technological vision.

Nyce, James M., and Paul Kahn (eds.). 1992. From Memex to Hypertext: Vannevar Bush and the Mind’s Machine. Boston: Academic Press.

In this report, Bush proposes the National Science Foundation (NSF) and more.

Bush, Vannevar. 1945. “Science The Endless Frontier. A Report to the President by Vannevar Bush, Director of the Office of Scientific Research and Development, July 1945.” Washington, D.C.: United States Government Printing Office. http://www.nsf.gov/about/history/vbush1945.htm; also available as ACLS Humanities E-Book (August 1, 2008).

Some believe that Paul Otlet, not Bush, ought to get the credit for the concept of hyperlinks for his 1934 “réseau” idea.

Wright, Alex. 2008. “The Web Time Forgot.” The New York Times (June 17).

In the 1960s, Ted Nelson took Bush’s ideas and extended them to support a new paradigm for literature in a networked world. He coined the term hypertext and proposed ideas that are current today, like virtually including one work inside another and using micropayments.

Nelson, Theodor Holm. 1993. Literary Machines. Sausalito, Calif.: Mindful Press.

Nelson, Theodor Holm. 1999. “Xanalogical Structure, Needed Now More Than Ever: Parallel Documents, Deep Links to Content, Deep Versioning, and Deep Re-Use.” Computing Surveys (ACM) 3, issue 4es (December).

Another pioneer in the 1960s who was inspired by Bush was Douglas En glebart, who founded a research lab with the goal of “augmenting human intellect.” His lab developed a hypermedia groupware system called Augment (originally called NLS). Augment supported bookmarks, hyperlinks, recording of e-mail, a journal, and more.

Engelbart, Douglas C. “Augmenting Human Intellect: A Conceptual Framework. Summary Report AFOSR-3223 Under Contract AF 49(638)- 1024,” SRI Project 3578 for Air Force Office of Scientific Research. Menlo Park, Calif.: Stanford Research Institute, October 1962.

———. “Authorship Provisions in AUGMENT.” COMPCON ’84 Digest: Proceedings of the COMPCON Conference, San Francisco, California, February 27-March 1, 1984, 465-72.

Many others besides us have noted the inadequacy of conventional computer file systems. Here are a few representative works.

Adar, Eytan, David Karger, and Lynn Andrea Stein. “Haystack: Per-User Information Environments,” 1999 Proceedings of the Conference on Information and Knowledge Management, Kansas City, Mo., 1999, 413-422.

Dourish, Paul, Keith Edwards, Anthony LaMarca, John Lamping, Karin Petersen, Michael Salisbury, Douglas Terry, and Jim Thornton. 2000. “Extending Document Management Systems with User-Specific Active Properties.” ACM TOIS 18, no. 2: 140-70.

Gifford, David K., Pierre Jouvelot, Mark A. Sheldon, and James W. O’Toole Jr. “Semantic File Systems.” Thirteenth ACM Symposium on Operating Systems Principles, October 1991, 16-25.

Maier, David. “Care and Feeding of Your PetDB.” NSF Workshop on Context-Aware Mobile Database Management (CAMM), January 24, 2002.

Olsen, Michael A. “The Design and Implementation of the Inversion File System,” 1993 Winter USENIX, San Diego, California, January 25-29, 1993.

You can find nearly everything about MyLifeBits at the Web site www.MyLifeBits.com. Here are some of the key papers:

Bell, Gordon, and Jim Gemmell. 2007. “A Digital Life.” Scientific American (March).

Gemmell, Jim, Gordon Bell, and Roger Lueder. 2006. “MyLifeBits: A Personal Database for Everything.” Communications of the ACM 49, issue 1 (January): 88-95.

Gemmell, Jim, Gordon Bell, Roger Lueder, Steven Drucker, and Curtis Wong. “MyLifeBits: Fulfilling the Memex Vision.” ACM Multimedia ’02, Juan-les-Pins, France, December 1-6, 2002, 235-38.

And here are some of the best articles written about MyLifeBits:

Cherry, Steven. 2005. “Total Recall.” IEEE Spectrum (November). Thompson, Clive. 2006. “A Head for Detail.” Fast Company (November). Wilkinson, Alec. 2007. “Remember This.” The New Yorker (May 28).

Look for the latest on SenseCam at research.microsoft.com—here are some published papers about it:

Berry, E., N. Kapur, L. Williams, S. Hodges, P. Watson, G. Smyth, J. Srinivasan, R. Smith, B. Wilson, and K. Wood. 2007. “The Use of a Wearable Camera, SenseCam, as a Pictorial Diary to Improve Autobiographical Memory in a Patient with Limbic Encephalitis.” In research.microsoft. com: Psychology Press, 582-681.

Harper, R., D. Randall, N. Smyth, C. Evans, L. Heledd, and R. Moore. 2007. “Thanks for the Memory.” Human-Computer Interaction Conference, Lancaster, UK, 2007.

Laursen, Lucas. A Memorable Device, Science 13 March 2009: 1422-23.

Sellen, A., A. Fogg, S. Hodges, and K. Wood. “Do Life-Logging Technologies Support Memory for the Past? An Experimental Study Using SenseCam.” Conference on Human Factors in Computing Systems, CHI ’07, Irvine, California, 2007, 81-90.

Berry, E., M. Conway, C. Moulin, H. Williams, S. Hodges, L. Williams, K. Wood, and G. Smith. 2006. “Stimulating Episodic Memory: Initial Explorations Using SenseCam.” Abstracts of the Psychonomic Society, 47th Annual Meeting 11:56-57. Oxford: Oxford University Press.

Gemmell, Jim, Lyndsay Williams, Ken Wood, Roger Lueder, and Gordon Bell. “Passive Capture and Ensuing Issues for a Personal Lifetime Store.” Proceedings of the First ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE ’04), New York, October 15, 2004, 48-55.

Hodges, Steve, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan, Alex Butler, Gavin Smyth, Narinder Kapur, and Ken Wood. 2006. “SenseCam: a Retrospective Memory Aid.” In Dourish and A. Friday, eds., Ubicomp 2006: Ubiquitous Computing. Lecture Notes in Computer Science 4206: 177-193. Berlin, Heidelberg: Springer-Verlag 2006.

StartleCam is another sensor-triggered wearable camera. It uses multiple skin conductivity sensors worn on the fingers. The sensors are used to detect the startle response in the wearer, and save the recently captured images, which will presumably be of events that aroused the user’s attention.

Healey, J., and R. W. Picard. “StartleCam: A Cybernetic Wearable Camera.” Proceedings of the Second International Symposium on Wearable Computing , Pittsburgh, Pennsylvania, October 19-20, 1998.

A system called u-Photo extends the action of picture taking to also include capturing the state of devices and sensor values in the camera’s view. LED tags are placed on objects to determine if they are in view. For instance, a picture of a room may also record the information that the lights are on, that the temperature is seventy-four degrees, and that a movie is playing at a certain point.

Iwamoto, Takeshi, Genta Suzuki, Shun Aoki, Naohiko Kohtake, Ka zunori Takashio, and Hideyuki Tokuda. “u-Photo: A Design and Implementation of a Snapshot Based Method for Capturing Contextual Information.” Pervasive 2004 Workshop on Memory and Sharing of Experiences, Vienna, Austria, April 20, 2004.

The remote control in your home—or perhaps that you carry with you—can even be part of lifelogging.

Abe, M., Y. Morinishi, A. Maeda, M. Aoki, and H. Inagaki. 2009. “A Life Log Collector Integrated with a Remote-Controller for Enabling User Centric Services.” IEEE Transactions on Consumer Electronics 55, no. 1.

Cathal Gurrin’s Web page is http://www.computing.dcu.ie/~cgurrinand here are some of his papers about e-memories.

Doherty, A., C. Gurrin, G. Jones, and A. F. Smeaton. “Retrieval of Similar Travel Routes Using GPS Tracklog Place Names.” SIGIR 2006—Conference on Research and Development on Information Retrieval, Workshop on Geographic Information Retrieval, Seattle, Washington, August 6-11, 2006.

Gurrin, C., A. F. Smeaton, D. Byrne, N. O’Hare, G. Jones, and N. O’Connor. “An Examination of a Large Visual Lifelog.” AIRS 2008—Asia Information Retrieval Symposium, Harbin, China, January 16-18, 2008.

Lavelle, B., D. Byrne, C. Gurrin, A. F. Smeaton, and G. Jones. “Bluetooth Familiarity: Methods of Calculation, Applications and Limitations.” MIRW 2007—Mobile Interaction with the Real World, Workshop at the MobileHCI07: 9th International Conference on Human Computer Interaction with Mobile Devices and Services, Singapore, September 9, 2007.

Lee, H., A. F. Smeaton, N. O’Connor, G. Jones, M. Blighe, D. Byrne, A. Doherty, and C. Gurrin. 2008. “Constructing a SenseCam Visual Diary as a Media Process.” Multimedia Systems Journal, Special Issue on Canonical Processes of Media Production (in press) 14, no. 6:341-49.

Smeaton, A. F., N. O’Connor, G. Jones, G. Gaughan, H. Lee, and C. Gurrin. “SenseCam Visual Diaries Generating Memories for Life.” Poster presented at the Memories for Life Colloquium 2006, British Library Conference Centre, London, UK, December 12, 2006. [BibTex] Memories For Life Web site.

We started the CARPE (Capture, Archival and Retrieval of Personal Experiences) research workshop. Many interesting papers about lifelogging were presented at these workshops.

CARPE Web page. http://www.sigmm.org/Members/jgemmell/CARPE

The Microsoft Research Digital Memories program gave funding, MyLifeBits software, and SenseCams to fourteen universities. You can find information about the projects, including the published articles.

Digital Memories program Web site. http://research.microsoft.com/en- us/collaboration/focus/cs/memex.aspx

3. THE MEETING OF E-MEMORY AND BIOMEMORY

Some books that explain how our brains remember, misremember, and forget. Daniel Schacter’s seven sins of memory are: transience (the loss of memories over time), absentmindedness (forgetting due to inattentive-ness, or forgetting what you meant to do), blocking (the temporary inability to recall something, like someone’s name), misattribution (assigning a memory to the wrong source), suggestibility (memories planted by suggestions or leading questions), bias (using current knowledge to revise past memories), and persistence (unwanted recall of a memory).

Kandel, Eric. 2007. In Search of Memory: The Emergence of a New Science of Mind. New York: W. W. Norton and Co.

Schacter, Daniel L. 2001. The Seven Sins of Memory: How the Mind Forgets and Remembers. New York: Houghton Mifflin.

Wang, Sam, and Sandra Aamodt. 2008. Welcome to Your Brain: Why You Lose Your Car Keys but Never Forget How to Drive and Other Puzzles of Everyday Life. New York: Bloomsbury USA.

Forgetfulness tends to get worse as we age, and midlife is often marked by a sharp increase in absentmindedness, as delightfully described by Cathryn Jakobson Ramin in Carved in Sand. She takes the reader on a midlife quest for improved memory, covering the gamut from synthetic estrogen to mental aerobics.

Ramin, Cathryn Jakobson. 2007. Carved in Sand: When Attention Fails and Memory Fades in Midlife. New York: HarperCollins.

We quote Joseph LeDoux answering the Edge’s 2008 question “What have you changed your mind about?” LeDoux is the author of The Synaptic Self: How Our Brain Becomes Who We Are.

LeDoux in Edge. http://www.edge.org/q2008/q08_1.html

David Allen’s Getting Things Done advocates a “logical and trusted system outside of your head.” His insight that “if your reference material doesn’t have a nice clean edge to it, the line between actionable and nonaction able items will blur” points out how important an e-memory of reference material, in addition to action items, can be. In addition to the Getting Things Done book, there are conferences, seminars, software tools, and a host of Web articles devoted to helping you implement the Getting Things Done methodology.

Allen, David. 2003. Getting Things Done: The Art of Stress-Free Productivity. New York: Penguin Books.

Allen’s Web site. http://www.davidco.com

Regarding false memories of sexual abuse, Elizabeth Loftus writes, “Now, nearly two decades later, there are hundreds of studies to support a high degree of memory distortion. People have recalled nonexistent broken glass and tape recorders, a clean-shaven man as having a mustache, straight hair as curly, and even something as large and conspicuous as a barn in a bucolic scene that contained no buildings at all.” “The repressed memory cases,” she observes, “are another outlet for women’s rage over sexual violence. Although women’s anger is certainly justified in many cases, and may be justified in some repressed memory cases too, it is time to stop and ask whether the net of rage has been cast too widely, creating a new collective nightmare.”

Loftus, Elizabeth F. 1993. “The Reality of Repressed Memories.” American Psychologist 48:518-37.

Danitz, Tiffany. 1997. “Making Up Memories?” Insight on the News (December 15).

Researchers at Dublin City University have done some really interesting work on dealing with thousands of SenseCam pictures (see also the work under Cathal Gurrin from the previous chapter).

Blighe, M., H. Le Borgne, N. O’Connor, A. F. Smeaton, and G. Jones. “Exploiting Context Information to Aid Landmark Detection in SenseCam Images.” ECHISE 2006—2nd International Workshop on Exploiting Context Histories in Smart Environments—Infrastructures and Design, 8th International Conference of Ubiquitous Computing (Ubicomp 2006), Orange County, California, September 17-21, 2006.

Byrne, D. 2007. “SenseCam Flow Visualization for LifeLog Image Browsing.” BCS IRSG Informer, no. 22 (Spring).

Byrne, D., B. Lavelle, A. Doherty, G. Jones, and A. F. Smeaton. “Using Bluetooth and GPS Metadata to Measure Event Similarity in SenseCam Images,” IMAI ’07—5th International Conference on Intelligent Multi media and Ambient Intelligence, Salt Lake City, Utah, July 18-24, 2007.

Doherty, A., A. F. Smeaton, K. Lee, and D. Ellis. “Multimodal Segmentation of Lifelog Data.” Eighth RIAO Conference—Large-Scale Semantic Access to Content (Text, Image, Video and Sound), Pittsburgh, Pennsylvania, May 30-June 1, 2007.

Lee, Hyowon, Alan F. Smeaton, Noel E. O’Connor, and Gareth J. F. Jones. “Adaptive Visual Summary of LifeLog Photos for Personal Information Management.” AIR 2006—First International Workshop on Adaptive Information Retrieval, Glasgow, UK, October 14, 2006.

O’Conaire, C., N. O’Connor, A. F. Smeaton, and G. Jones. “Organizing a Daily Visual Diary Using Multi-Feature Clustering.” SPIE Electronic Imaging—Multimedia Content Access: Algorithms and Systems (EI121), San Jose, California, January 28-February 1, 2007.

Smeaton, A. F. “Content vs. Context for Multimedia Semantics: The Case of SenseCam Image Structuring.” SAMT 2006—Proceedings of the First International Conference on Semantics and Digital Media Technology. Lecture Notes in Computer Science (LNCS), Athens, Greece, December 6-8, 2006.

Smeaton, A. F., D. Diamond, and B. Smyth. “Computing and Material Sciences for LifeLogging.” Presented at the Memories for Life Network Workshop 2006, British Library Conference Centre, London, UK, December 11, 2006. Memories for Life Web site.

There is lots of other work on automatic summarization; for example, this paper on video summarization:

Sundaram, Hari, and Shih-Fu Chang. “Condensing Computable Scenes Using Visual Complexity and Film Syntax Analysis.” Second IEEE International Conference on Multimedia and Expo (ICME-2001), Tokyo, Japan, August 2001.

We mention a wearable system from the University of Tokyo that includes a brain wave sensor in a baseball cap. Their system also continuously captures video, GPS, gyroscope, and accelerometer data.

Aizawa, Kiyoharu, Datchakorn Tancharoen, Shinya Kawasaki, and Toshi hiko Yamasaki. “Efficient Retrieval of Life Log Based on Context and Content.” Proceedings of the First ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE ’04), New York, October 15, 2004, 21-31.

Hori, Tetsuro, and Kiyohara Aizawa. “Context-Based Video Retrieval System for the Life-Log Applications.” Proceedings of the 5th ACM SIGMM International Workshop on Multimedia Information Retrieval, Berkeley, California, 2003.

reQall has a Web site, and there are a number of good articles about reQall, including the one in Forbes, below.

reQall Web site. http://www.reQall.com

Woyke, Elizabeth. 2008. “You Must Remember This.” Forbes (September 30).

reQall cofounder Sunil Vemuri previously developed a PDA system to record audio and location. Calendar, e-mail, and common Web site and weather reports could also be captured. Speech-to-text was performed, and regular text search was augmented with phonetic “sounds-like” search. A speaker identification algorithm was also used, and text was colored according to speaker.

Vemuri, S., C. Schmandt, W. Bender, S. Tellex, and B. Lassey. “An Audio-Based Personal Memory Aid.” In Proceedings of Ubicomp 2004: Ubiquitous Computing, Nottingham, UK, September 7-10, 2004, 400-17.

Supermemo:

Supermemo Web site. http://www.supermemo.com

Wolf, Gary. 2008. “Want to Remember Everything You’ll Ever Learn? Surrender to This Algorithm.” Wired (April 21).

Rank Xerox EuroParc had a number of projects involving capture in aid of human memory. Pepys was a diary automatically created and mailed to the user each day. It was based on location information derived from Active Badges worn by the users. Episodes were recognized (e.g., “Meeting with Joe” or “Working in office”), false positives were rejected (e.g., people sharing an office should not be presumed to be constantly in a meeting), and the level of detail was reduced to a workable summary. A “video diary” (actually a series of images captured at a rate of around ten frames per minute) was also captured by cameras in the building. Marcel was a paper document tracking system that used a video camera mounted over the user’s desk. Documents on the desk were compared with the database of documents to identify them. Forget-me-not was a memory aid system. It logged e-mail, file sharing, printing, and telephone calls and supported browsing of a user’s diary, with events filtered by when, where, or who to aid recall. A study compared three conditions: (1) no computer support, (2) Pepys, (3) “video diary.” They found Pepys improved recall, and video did even more. People and objects were the most common memory cues.

Eldridge, Margery, Michael Lamming, and Mike Flynn. 1992. “Does a Video Diary Help Recall?” In A. Monk, D. Diaper, and M. D. Harri son (eds.), People and Computers VII. Cambridge: Cambridge University Press, pp. 257-69, also published as Technical Report EPC-1991-124, Rank Xerox Research Center, Cambridge, UK, 1992.

Lamming, M., and M. Flynn. “Forget-me-not: Intimate Computing in Support of Human Memory,” Proceedings of FRIEND21,’94 International Symposium on Next Generation Human Interface, Meguro Gajoen, Japan, 1994.

Lamming, M., P. Brown, K. Cater, M. Eldridge, M. Flynn, G. Louie, P. Robinson, and A. Sellen. 1994. “The Design of a Human Memory Prosthesis.” The Computer Journal 37, no, 3:153-63.

Lamming, M. G. “Using Automatically Generated Descriptions of Human Activity to Index Multi-Media Data.” IEEE Colloquium on Multimedia Communications and Applications, February 7, 1991, pp. 5/1-5/3.

Lamming, M. G., and W. M. Newman. 1992: “Activity-based Information Retrieval: Technology in Support of Personal Memory.” Personal Computers and Intelligent Systems: Information Processing ’92.Amsterdam: North-Holland, 68-81.

Newman,W. M., M. A. Eldridge, and M. G. Lamming. 1991. “Pepys: Generating Autobiographies by Automatic Tracking.” Proceedings of the Second European Conference on Computer-Supported Cooperative Work—ECSCW ’91. Dordrecht, Netherlands: Kluwer, 175-88.

The SPECTOR project looks at how to use Pepys-like automatic diaries to develop a model of the user and perform machine learning to help the user.

Kröner, Alexander, Stephan Baldes, Anthony Jameson, and Mathias Bauer. “Using an Extended Episodic Memory Within a Mobile Companion.” Pervasive 2004 Workshop on Memory and Sharing of Experiences, Vienna, Austria, April 20, 2004.

The Infinite Memory Multifunction Machine (IM3) was a system to automatically capture every document that a user copies, faxes, or prints. It was able to automatically detect duplicates, and was quite successful in automatically filing new documents into a user’s existing file hierarchy based on word counting and image analysis. A study with twenty users over two years showed that the average age of a retrieved document was forty-four days, with 10 percent of all accesses being for documents older than six months. This debunked a common conjecture that old documents would virtually never be needed.

Hull, Jonathan J., and Peter Hart. “The Infinite Memory Multifunction Machine (IM3).” Pre-Proceedings of the Third IAPR Workshop on Document Analysis Systems, Nagano, Japan, November 4-6, 1998, 49-58.

Hull, Jonathan J., Dar-Shyang Lee, John Cullen, and Peter Hart. “Document Analysis Techniques for the Infinite Memory Multifunction Machine.” Proceedings of the 10th International Workshop on Database and Expert Systems Applications, Florence, Italy, September 1-3, 1999, 561-65.

Hull, Jonathan J., and Peter E. Hart. 2001. “Toward Zero Effort Personal Document Management.” IEEE Computer 34, no. 3 (March).

Here are several works that discuss expanding our definition of mind to encompass e-memories. For instance, David Chalmers, who says in an interview with The Philosophers’ Magazine,

When bits of the environment are hooked up to your cognitive system in the right way, they are, in effect, part of the mind, part of the cognitive system. So, say I’m rearranging Scrabble tiles on a rack. This is very close to being analogous to the situation when I’m doing an anagram in my head. In one case the representations are out in the world, in the other case they’re in here. We say doing an anagram on a rack ought be regarded as a cognitive process, a process of the mind, even though it’s out there in the world.

. . . A whole lot of my cognitive activities and my brain functions have now been uploaded into my iPhone. It stores a whole lot of my beliefs, phone numbers, addresses, whatever. It acts as my memory for these things. It’s always there when I need it.

Baggini, Julian. “A Piece of iMe: An Interview with David Chalmers.” 2008. The Philosophers’ Magazine, Issue 43 (4th Quarter).

Clark, Andy. 2008. Supersizing the Mind. Introduction by David J. Chalmers. New York: Oxford University Press.

Noë, Alva. 2009. Out of Our Heads: Why You Are Not Your Brain, and Other Lessons from the Biology of Consciousness. New York: Hill and Wang.

Jill Price has an astonishing memory, and it seems to be a burden to her. The human mind seems to improve memory at the price of unwanted recall—this won’t be the case for e-memory.

Price, Jill, and Bart Davis. 2008. The Woman Who Can’t Forget: The Extraordinary Story of Living with the Most Remarkable Memory Known to Science. New York: The Free Press.

Stephen Wiltshire uses a photographic memory to amaze people with his art.

Adams, Stephen. 2008. “Stephen Wiltshire, the Human Camera Who Drew London from Memory.” Telegraph (April 3).

The Stephen Wiltshire Gallery Web site. http://www.stephenwiltshire.co.uk

4. WORK

Regarding DARPA’s LifeLog and ASSIST:

Archived copy of the LifeLog Proposer Information Pamphlet, http://web.archive.org/web/20030603173339/http%3a//www.darpa.mil/ipto/Solicitations/PIP_03-30.html

Safire, William. 2003. “Dear Darpa Diary.” The New York Times (June 5).

Shachtman, Noah. 2004. “Pentagon Kills LifeLog Project.” Wired (February 4).

———. “Pentagon Revives Memory Project.” Wired (September 13).

The soldier patrol story is based on Thad Starner’s talk at CARPE 2006, but also borrow’s from other publications related to ASSIST. Special thanks to Daniel Ashbrook.

Minnen, David, Tracy Westeyn, Peter Presti, Daniel Ashbrook, and Thad Starner. “Recognizing soldier activities in the field.” Proceedings of International IEEE Workshop on Wearable and Implantable Body Sensor Networks (BSN), Aachen, Germany, March 2007.

Schlenoff, Craig, et al. “Overview of the First Advanced Technology Evaluations for ASSIST.” Proceedings of Performance Metrics for Intelligent Systems (PerMIS) 2006, IEEE Press, Gaithersburg, Maryland, August 2006.

Stevers, Michelle Potts. “Utility Assessments of Soldier-Worn Sensor Systems for ASSIST.” Proceedings of the Performance Metrics for Intelligent Systems Workshop, 2006.

Starner, Thad. “The Virtual Patrol: Capturing and Accessing Information for the Soldier in the Field.” Proceedings of the 3rd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences, Santa Barbara, California, 2006.

Glass Box:

Cowley, Paula, Jereme Haack, Rik Littlefield, and Ernest Hampson. “Glass Box: Capturing, Archiving, and Retrieving Workstation Activities.” Proceedings of the 3rd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences, Santa Barbara, California, 2006.

The Microsoft Research VIBE team, led by Mary Czerwinski, has developed a number of excellent visualizations, including time spent on your computer, and browsing your e-memories.

VIBE Web page. http://research.microsoft.com/en-us/groups/vibe

Brush, A. J., Brian Meyers, Desney Tan, and Mary Czerwinski. “Understanding Memory Triggers for Task Tracking.” In Extended Abstracts at CHI 2007 Conference on Human Factors in Computing Systems,Association for Computing Machinery, Inc., April 2007.

Smith, Greg, Mary Czerwinski, Brian Meyers, Daniel Robbins, George Rob ertson, and Desney Tan. 2006. “FacetMap: A Scalable Search and Browse Visualization.” In IEEE Transactions on Visualization and Computer Graphics.

The wearable Remembrance Agent is hardware consisting of a one-handed chording keyboard with a heads-up display, along with radio and infrared receivers. Location is detected using radio location beacons and people are detected using infrared name badges. It runs note-taking software that selects old notes to show you based on your current location, people it detects around you, and the text of notes that you are presently writing. A desktop version of the Remembrance Agent operates within a text editor and brings up relevant items in a window based on what you are typing. A contextual retrieval application called Margin Notes has also been developed for Web browsing.

Rhodes, Bradley. 2003. “Physical Context for Just-in-Time Information Retrieval.” IEEE Transactions on Computers 52, no. 8 (August): 1011-14.

———. 1997. “The Wearable Remembrance Agent: A System for Augmented Memory.” Special Issue on Wearable Computing, Personal Technologies Journal 1:218-24.

Rhodes, Bradley J. “Margin Notes: Building a Contextually Aware Associative Memory” (html), to appear in The Proceedings of the International Conference on Intelligent User Interfaces (IUI ’00), New Orleans, Louisiana, January 9-12, 2000.

Rhodes, Bradley, and Pattie Maes. 2000. “Just-in-Time Information Retrieval Agents.” Special issue on the MIT Media Laboratory, IBM Systems Journal 39, nos. 3 and 4: 685-704.

Rhodes, Bradley, and Thad Starner. “The Remembrance Agent: A Continuously Running Automated Information Retrieval System. The Proceedings of the First International Conference on the Practical Application of Intelligent Agents and Multi Agent Technology (PAAM ’96), London, UK, April 1996, 487-95.

DARPA is funding work toward a vision that “a cognitive computer system should be able to learn from its experience, as well as by being advised. It should be able to explain what it was doing and why it was doing it, and to recover from mental blind alleys. It should be able to reflect on what goes wrong when an anomaly occurs, and anticipate such occurrences in the future. It should be able to reconfigure itself in response to environmental changes. And it should be able to be configured, maintained, and operated by nonexperts.” CALO (Cognitive Assistant that Learns and Organizes) and RADAR are projects funded by DARPA toward this end.

CALO Web site. http://caloproject.sri.com

RADAR Web site. http://www.radar.cs.cmu.edu

DevonThink:

DevonThink Web page. http://www.devon-technologies.com/products/devonthink

Oakes, Chris. 2004. “Software Makes a Tiger of Panther.” Wired (July 6).

DSpace is “an open-source platform for accessing, managing, and preserving scholarly works. Developed by MIT Libraries and HP Labs, DSpace preserves and enables easy and open access to all types of digital content including text, images, moving images, mpegs, and data sets in an institutional repository.” “A university-based institutional repository is a set of services that a university offers to the members of its community for the management and dissemination of digital materials created by the institution and its community members. It is most essentially an organizational commitment to the stewardship of these digital materials, including long-term preservation where appropriate, as well as organization and access or distribution.”

Dspace Web site. http://www.dspace.org

Naone, Erica. 2007. “DSpace Goes Olympic.” Technology Review (Nov ember/December).

Sharing information about repairing Xerox copiers:

Bobrow, D. G., and J. Whalen. 2002. “Community Knowledge Sharing in Practice: The Eureka Story.” Reflections, the SOL Journal 4, issue 2 (Winter): 47-59.

To be productive with your e-memories you must employ Personal Information Management (PIM). There is some great research being done on the topic.

Jones, William. 2008. Keeping Found Things Found: The Study and Practice of Personal Information Management (Interactive Technologies). Burlington, Mass.: Morgan Kaufman.

Jones, William, and Jaime Teevan, eds. 2007. Personal Information Management. Seattle: University of Washington Press.

http://pim.ischool.washington.edu

An overview and primer about the nature and organization of information includes descriptions of the Dewey Decimal systems and facets:

Wright, Alex. 2007. Glut: Managing Information Through the Ages. Ithaca N.Y.: Cornell University Press.

5. HEALTH

A number of facts cited in this chapter came from presentations at the 2007 New Paradigms in Using Computers (NPUC) workshop. We used information from Dr. Paul Tang, chief medical information officer for Sutter Health, Elizabeth Mynatt of the Georgia Institute of Technology, and Peter Miller of Vanderbilt HealthTech Laboratory,

NPUC 2007 Web page. http://www.almaden.ibm.com/cs/user/npuc2007

Some other sources on electronic health records:

“VA’s Electronic Patient Records Are a Model to Industry.” United States Department of Veterans Affairs press release, http://www1.va.gov/opa/pressrel/pressrelease.cfm?id=1277

“Medication Errors Injure 1.5 Million People and Cost Billions of Dollars Annually; Report Offers Comprehensive Strategies for Reducing Drug-Related Mistakes.” National Academies press release, July 20, 2006, http://www8.nationalacademies.org/onpinews/newsitem. aspx?RecordID=11623 (refers to the following reference)

Committee on Identifying and Preventing Medication Errors (author), Philip Aspden, Julie Wolcott, J. Lyle Bootman, Linda R. Cronenwett (eds.). 2007. Preventing Medication Errors. Washington, D.C.: National Academies Press.

“Health Information Technology: Can HIT Lower Costs and Improve Quality?” http://www.rand.org/pubs/research_briefs/RB9136/index1.html

Amarasingham, Ruben, MD, MBA; Laura Plantinga, ScM; Marie Diener-West, Ph.D.; Darrell J. Gaskin, Ph.D.; Neil R. Powe, MD, MPH, MBA. 2009. “Clinical Information Technologies and Inpatient Outcomes: A Multiple Hospital Study.” Archives of Internal Medicine 169, no. 2 ( January 26): 108-14.

Carter, Jerome H. 2008. Electronic Health Records: A Guide for Clinicians and Administrators. Second edition. Philadelphia: American College of Physicians Press.

The UK “spine”:

NHC connecting for health Web site. http://www.connectingforhealth. nhs.uk

http://www.connectingforhealth.nhs.uk/resources/systserv/spine- factsheet

Cross, Michael. 2006. “Getting Hospital Data to Connect to the NHS ‘Spine.’ ” The Guardian (August 10).

The European Union e-Health action plan:

European Commission. “The Right Prescription for Europe’s eHealth.” http://ec.europa.eu/information_society/activities/health/policy_action_plan/index_en.htm

Sherriff, Lucy. 2004. “European Healthcare ‘Online by 2008.’ ” The Register (May 5).

Articles about implementing electronic health records:

McGee, Marianne Kolbasuk. 2007. “Why Progress Toward Electronic Health Records Is Worse Than You Think.” Information Week (May 26).

Darcé, Keith. 2007. “Unhealthy Records.” San Diego Union-Tribune (May 20).

Merlin, Bruce. 2007. “What Killed the Santa Barbara County Care Data Exchange?” iHealthBeat (March 12).

Molecular imaging agents have been developed by companies like CellPoint:

http://cellpointweb.com

Comprehensive blood sampling by companies like BioPhysical:

http://www.biophysicalcorp.com/biomarker-research

Philips has some home health-care devices, including scales, blood pressure cuffs, pulse oximeters, and glucose meters:

http://www.medical.philips.com/us/homehealth/index.wpd

Also by Philips: “The world’s first large-scale, randomized prospective telemonitoring trial showed that homebased telemonitoring reduced the number of days spent in hospital by 26% and led to an overall 10% cost savings compared to nurse telephone support. Home Telemonitoring also significantly improved survival rates relative to usual care and led to high levels of patient satisfaction.”

http://www.medical.philips.com/main/products/telemonitoring/assets/docs/TEN-HMS_White_Paper_FINAL.pdf

More devices and approaches to biometrics:

Brady, S., S. Coyle, and D. Diamond. “Smart Shoes for Healthcare and Security.” In pHealth 2007, Chalkidiki, Greece.

Chan, K. W., Hung, K., Zhang, Y. T. “Noninvasive and Cuffless Measurements of Blood Pressure for Telemedicine.” Engineering in Medicine and Biology Society, 2001. Proceedings of the 23rd Annual International Conference of the IEEE, 2001.

Coyle, S., Y. Wu, K. Lau, J. H. K im, S. Brady, G. Wallace, and D. Diamond. “Design of a Wearable Sensing Platform for Sweat Analysis.” In pHealth 2007, Chalkidiki, Greece.

Jaimes, Alejandro. “Sit Straight (and Tell Me What I Did Today): A Human Posture Alarm and Activity Summarization System.” Proceedings of the 2nd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE ’05), November 2005.

Oliver, Nuria, and Fernando Flores-Mangas. “HealthGear: A Real-time Wearable System for Monitoring and Analyzing Physiological Signals . Proceedings of the International Conference on Body Sensor Networks (BSN’06). MIT, Boston, Massachusetts, April 2006.

Ooe, Yasuhiko, Kentaro Yamasaki, and Tsukasa Noma. “PWS and PHA: Posture Web Server and Posture History Archiver.” Proceedings of the First ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE ’04), New York, October 15, 2004, 99-104.

Tan Ee Lyn. 2007. “HK Invents Pain-Free Device to Measure Blood Sugar,” Reuters (May 7).

Teller A., and J. I. Stivoric. “The BodyMedia Platform: Continuous Body Intelligence.” Proceedings of the 1st ACM Workshop on Continuous Archival and Retrieval of Personal Experiences, New York, 2004.

Ya-Ti Peng, Ching-Yung Lin, and Ming-Ting Sun. “Multi-Modality Sensor for Sleep Quality Monitoring and Logging.” eChronicles Workshop, ICDE, 2006.

BodyMedia Web site. http://www.bodymedia.com

“Needle-Free Blood and Tissue Measurements,” http://sci.rutgers.edu/forum/archive/index.php/t-1470.html

BiacaMed Web site. http://www.biancamed.com

Omron health care Web site. http://www.omronhealthcare.com

Oregon Scientific sports and fitness products. http://www2.oregonscientific.com/

Smartex Wealthy system. http://www.smartex.it/garment_en.html

It appears that the cell phone will be the hub for a lot of health information. Allen Cheng at the University of Pittsburgh is working on HeartToGo, for mobile ECG. Daniel Fletcher at UC Berkeley converts the cell phone into a portable microscope for disease diagnosis with CellScope. Portable ultrasound is under development by Richard and Zar at Washington University.

Zhanpeng Jin, Joseph Oresko, Shimeng Huang, and Allen C. Cheng. “HeartToGo: A Smartphone-based Mobile Platform for Continuous and Real-Time Cardiovascular Disease Monitoring.” In Proceedings of the Microsoft External Research Symposium 2009 (invited paper), Redmond, Washington, March 2009.

Richard, W. D., D. M. Zar, and R. Solek. 2008. “A Low-Cost B-Mode USB Ultrasound Probe.” Ultrasonic Imaging 30:21-28.

“Mobile phone microscopes: Doctor on call.” 2008. The Economist (May 15).

Storing your health information:

Quicken Health Web site. http://quickenhealth.intuit.com

Microsoft HealthVault Web site. http://www.healthvault.com

Google Health Web site. http://www.google.com/health

10,000 Steps a Day:

Isaacs, Greg. 2006. 10,000 Steps a Day to Your Optimal Weight: Walk Your Way to Better Health. Santa Monica, California: Bonus Books, Inc.

“Laura” the e-nurse:

Elton, Catherine. 2007. “‘Laura’ Makes Digital Health Coaching Personal.” Boston Globe (May 21).

6. LEARNING

Deb Roy’s Speechome:

Biever, Celeste. 2006. “Watch Language Grow in the ‘Baby Brother’ House.” New Scientist (May 15).

Roy, Deb, et al. “The Human Speechome Project.” Proceedings of the 28th Annual Cognitive Science Conference, 2006.

Study comparing lectures with Web-based learning and activities:

Wallace, David R., and Suzanne T. Weiner. 1998. “A Comparison of a Lecture-Style Second Coverage of Materials vs. Limited-Coverage Guided Experiential Activity. ASEE Journal of Engineering Education.

On the Web:

http://scholar.google.com

http://books.google.com

http://www.archive.org

http://citeseerx.ist.psu.edu

http://www.questia.com

ePortfolios:

Batson, T. 2002. “The Electronic Portfolio Boom: What’s It All About?” Syllabus (Dec 1).

Mason, R., C. Pegler, and M. Weller. 2004. “E-Portfolios: An Assessment Tool for Online Courses.” British Journal of Educational Technology 25, no. 6:717-27.

On tablet PCs for education, IEEE Computer ran a special issue on “Tablet PC Technology: The Next Generation” that included articles on “Ink, Improvisation, and Interactive Engagement: Learning with Tablets,” “Classroom Presenter: Enhancing Interactive Education and Collaboration with Digital Ink,” and “Facilitating Pedagogical Practices Through a Large-Scale Tablet PC Deployment.”

IEEE Computer, September 2007.

The Gray paradigm of science:

Gray, Jim, Alexander S. Szalay, Ani R. Thakar, Christopher Stoughton, and Jan vandenBerg. 2002. “Online Scientific Data Curation, Publication, and Archiving.” Microsoft Technical Report MSR-TR-2002-74 ( July).

The Computer History Museum has some online exhibits, as well as some transcribed oral histories.

http://www.computerhistory.org

There are many books, articles, and even think tanks on lifelong learning, for example:

Academy of Lifelong learning Web site. http://www.academy.udel.edu

Cohn, E., and J. T. Addison. 1998. “The Economic Returns to Lifelong Learning in OECD Countries.” Education Economics 6, no. 3 (December): 253-307.

Field, John. 2006. Lifelong Learning and the New Educational Order. Stoke-on-Trent, Staffordshire, UK: Trentham Books.

7. PERSONAL LIFE, AND AFTERLIFE

Organic light emitting polymer (OLEP) and Organic light emitting diode (OLED) displays:

Russell, Terrence. 2008. “Samsung Gearing up for OLED Push in 2009/2010.” Wired (April 22).

Shinar, Joseph. 2003. “Organic Light-Emitting Devices: A Survey.” New York: Springer.

TripReplay from MyLifeBits is described here:

Aris, Aleks, Jim Gemmell, and Roger Lueder. 2004. “Exploiting Location and Time for Photo Search and Storytelling in MyLifeBits.” Microsoft Research Technical Report MSR-TR-2004-102 (October).

Adding summarization to visualization for geolocated photos:

Ahern, Shane, Mor Naaman, Rahul Nair, Jeannie Yang. “World Explorer: Visualizing Aggregate Data from Unstructured Text in Geo-Referenced Collections.” In Proceedings, Seventh ACM/IEEE-CS Joint Conference on Digital Libraries ( JCDL 07), June 2007.

The Stuff I’ve Seen project did some experiments that showed how displaying milestones alongside a timeline may help orient the user. Horvitz et al. used statistical models to infer the probability that users will consider events to be memory landmarks.

Ringel, M., E. Cutrell, S. T. Dumais, and E. Horvitz. 2003. “Milestones in Time: The Value of Landmarks in Retrieving Information from Personal Stores.” Proceedings of IFIP Interact 2003.

Horvitz, Eric, Susan Dumais, and Paul Koch. “Learning Predictive Models of Memory Landmarks.” CogSci 2004: 26th Annual Meeting of the Cognitive Science Society, Chicago, August 2004.

Pondering digital immortality with Jim Gray back in 2001:

Bell, G., and J. N. Gray. 2001. “Digital Immortality.” Communications of the ACM 44, no. 3 (March): 28-30.

MyCyberTwin:

MyCyberTwin Web site. www.mycybertwin.com

Roush, Wade. 2007. Your Virtual Clone. Technology Review (April 20).

The Turing test:

Turing, A. 1950. “Computing Machinery and Intelligence.” Mind 59, no. 236: 433-60.

Creating biographical and family histories:

LifeBio: www.lifebio.com, formed in 2000, has a process, tools, and software to enable a person, family, or groups to create stories and documents that can be printed or displayed on the Web.

8. REVOLUTION

Dear Appy:

Bell, Gordon. 2000. “Dear Appy” ACM Ubiquity, 1, no. 1 (February).

Bannon argues in favor of forgetting:

Bannon, Liam. 2006. “Forgetting as a Feature, Not a Bug: The Duality of Memory and Implications for Ubiquitous Computing.” CoDesign 2, no. 1 (March): 3-15.

Management guru Drucker on managing yourself:

Drucker, Peter. 1999. “Managing Oneself.” Harvard Business Review (March-April): 65-74.

Google street views:

Derbyshire, David, and Arthur Martin. 2008. “Google ‘Burglar’s Charter’ Street Cameras Given the All Clear by Privacy Watchdog.” Mail Online (July 31).

Weeks, Carly. 2007. “Google’s Detailed Streetscapes Raise Privacy Concerns. National Post (September 11).

“Google Street Views, Cool or Creepy?” New York Post (June 7, 2007).

E-memories in court:

“Electronic storage devices function as an extension of our own memory,” Judge Pregerson wrote, in explaining why the government should not be allowed to inspect them without cause. “They are capable of storing our thoughts, ranging from the most whimsical to the most profound.” The magistrate judge, Jerome J. Niedermeier of Federal District Court in Burlington, Vt., used an analogy from Supreme Court precedent. It is one thing to require a defendant to surrender a key to a safe and another to make him reveal its combination.

Liptak, Adam. 2008. “If Your Hard Drive Could Testify.” The New York Times (January 7).

The idea of purposely falsifying some of your records to avoid them being used in court is examined in

Cheng, William, Leana Golubchik, and David Kay. “Total Recall: Are Privacy Changes Inevitable?: A Position Paper.” Proceedings of the First ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE ’04), New York, October 15, 2004, 86-92.

An overview of Privacy, Ownership, Search, Cryptography, and many of the critical aspects of our lives in Cyberspace:

Abelson, Hal, Ken Ledeen, and Harry Lewis. 2008. Blown to Bits: Your Life, Liberty, and Happiness After the Digital Explosion. Boston: Addison Wesley.

We only touched on a little of what Steve Mann has done in wearable computers and mediated reality. His work runs the gamut from technical issues to legal issues to art.

Mann, S. “ ‘ Sousveillance’: Inverse Surveillance in Multimedia Imaging.” Proceedings of ACM Multimedia 2004, New York, October 2004, 620-27.

Mann, Steve. 2001. Intelligent Image Processing. New York: John Wiley and Sons.

Mann, Steve, Anurag Sehgal, and James Fung. “Continuous Lifelong Capture of Personal Experience Using Eyetap.” Proceedings of The First ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE ’04), New York, October 15, 2004, 1-21.

Mann, Steve, with Hal Niedzviecki. 2001. Cyborg: Digital Destiny and Human Possibility in the Age of the Wearable Computer. New York: Random House Doubleday.

Mann’s “EyeTap” glass can record what he sees—now Canadian filmmaker Rob Spence plans to turn his prosthetic eye into a video camera.

“Anti-Surveillance Filmmaker Plans Eye-Socket Camera.” Reuters, March 5, 2009.

http://www.eyeborgblog.com

Hasan M. Elahi is an artist exploring the bounds of surveillance and sousveillance by being a more extreme and consistent life blogger. He captures nearly everything that happens in his waking life in photos with time and location. Hasan’s tracking site provides an image of exactly where Hasan is at any time.

Hasan Elahi’s Web site. http://elahi.org

Hasan’s tracking site. http://www.trackingtransience.net

9. GETTING STARTED

Aimee Baldridge has written a complete and excellent how-to for digitizing everything in your life. She describes how you go about estimating, planning, and digitizing your collection of documents, photos, cassettes, videotapes, vinyl records, et cetera. This two-hundred-page book makes it clear just how daunting digitizing your entire past can be. Our recommendation is to start now by accumulating everything that is born digital and go back in time based on need.

Baldridge, Aimee. 2009. Organize Your Digital Life: How to Store Your Photographs, Music, Videos, and Personal Documents in a Digital World. Washington, D.C.: National Geographic.

The end of paper is nowhere in sight. Copiers and computer printers continue to accelerate the growth in the use of paper and printing. The first step of going paperless is not storing or transmitting paper—the convenience of paper as a write-once, high quality, portable display will continue until display technology advances quite a bit more.

Paper digitizers, aka scanners, are used to eliminate paper storage and transmission. An ideal scanner would be small, fast, and low cost; it would be able to handle stacks of one- or two-sided items in arbitrary formats and sizes from business cards and photos to books and blueprints, and to scan black-and-white as well as color at arbitrary resolutions, would have a shredder as its last stage, and would be as easy to use as the shredder. In 2010 it takes a half-dozen scanners plus a shredder to realize the ideal. Office copiers with scanning capability are increasingly approaching the ideal because they are fast and require the high resolution for printing that is inherent with an office copier.

The following scanners are in order of my own personal preference:

• Small units that fit nicely at the back of a desk. Usually this will be a feed-through format to save space. The Fujitsu ScanSnap models are very nice, handling both sides of the paper in a single pass, both color and black-and-white, at five to fifteen pages per minute.

• Digital cameras suffice as scanner alternatives. A high-resolution digital camera can digitize almost anything. An A-size 8½” x 11” page can be resolved at 150 dpi with 2 megapixels or 300 dpi with 8 megapixels. A copy stand or tripod and proper lighting are essential.

• All-in-one personal print-scan-copy-fax. A single device scans and prints documents and photos. Quality, reliability, and speed are often unimpressive.

• Personal flatbed with document feeder. These are typically inexpensive, large, and bulky, with a relatively slow feeder (five to eight pages per minute) yet able to scan almost anything, including 3-D objects. Professional flatbed scanners operate at up to twenty-five double-sided pages per minute.

• Business card scanner. Special scanner of business cards. Desktop scanners usually handle this, making them redundant.

• Photo scanner (slides, negatives, positives, and prints). Photo scanners require the ability to handle multiple items at high resolution (>1,000 dpi), including slides. Photo scanning services are an excellent alternative to owning a photo scanner.

• Book-specialized flatbed scanners cost a few hundred dollars but require setup time per page scanned. At the high end of the spectrum, a fully automatic book scanner sells for tens of thousands of dollars.

• Large format, e.g., for blueprints. There are a variety of special scanners for scanning large items such as plans and blueprints. Most people will not have very many of these to scan, and will better off using a service to scan large items. An alternative is stitching multiple scans together using image-editing software.

Photo- and slide-scanning services are everywhere, from drugstores to photo stores. The Web lists thousands of them. Document scanning services also abound. Some scanning services will also come to your facility to do the scanning. VendorSeek and RecordNation will get you several quotes from different vendors for scanning your documents.

http://vendorseek.com

http://recordnation.com

If you are handling scanned documents, PDF is likely the format you will settle on, and PDF tools for creation, OCR, and editing will be essential.

Adobe Acrobat Web site. https://www.acrobat.com

Nuance Web site. www.nuance.com

CutePDF Web site. http://www.cutepdf.com

Tools for recording family and small business finances:

Microsoft Money. http://www.microsoft.com/MONEY/default.mspx

Quicken. http://www.quicken.com

Mind Your Own Business. http://myob.com

Tools for music ripping and management:

Apple iTunes Web site. http://www.itunes.com

Microsoft Zune Web site. http:// www.zune.net

WinAmp Web site. http://www.winamp.com

Photo editing and management software:

Adobe Elements Web site. http://www.adobelements.com

Adobe PhotoShop Web site. http://www.photoshop.com

Apple iLife Web site. http://www.apple.com/iLife

Picassa Web site. http://picassa.google.com

Microsoft Live Photo Gallery. http://download.live.com/photogallery

Video editing and management software:

Adobe Premier Web site. http://www.adobe.com/products/premiere

Apple iMovie Web site. http://www.apple.com/ilife/imovie

Microsoft Movie Maker Web site. http://www.microsoft.com/windowsxpusing/moviemaker/default.mspx

Magix Movie Edit Web site. http://www.magix.com/us/video

Pinnacle Studio Web site. http://www.pinnaclesys.com

Ourpix Web site. http://www.ourpix.com

Movie Story Web site. http://ourpix/dvd-slid-show.html

See the Health section below for devices, especially by Philips, Bian caMed, BodyMedia, Oregon Scientific, and Omron. See that section also for health software, including Google Health, Microsoft HealthVault, and Quicken Health.

Microsoft OneNote uses a notebook metaphor for putting “everything,” e.g., handwritten notes, voice comments, e-mail messages, writing, Web pages, photos, et cetera, into a searchable hierarchy of notebooks, sections, and pages.

One Note Web page. http://office.microsoft.com/onenote

Evernote, patterned after OneNote and inspired by MyLifeBits, is a cloud service that includes the use of a cell phone for transferring images, video, and voice into the notebooks where it is stored and recognized for access.

Evernote Web site. http://www.evernote.com

Livescribe’s Pulse Smartpen is a pen with audio recorder enabling handwritten notes and text input. When handwritten notes become part of computer documents, the voice recordings can be retrieved by pointing to the text. The pen has numerous applications; one can use it as a calculator, recorder, and to record meetings. IOgear has the competing Digital Scribe pen.

Livescribe Web site. http://www.livescribe.com

IOgear Digital Scribe pen. http://www.iogear.com/product/GPEN100C

IBM Pensieve:

Aizenbud-Reshef, Neta, et al. “Pensieve: Augmenting Human Memory.” Conference on Human Factors in Computing Systems. Florence, Italy, 2008.

Oregon Scientific and Hammacher Schlemmer have body-mounted cameras. Hammacher Schlemmer even has a diving mask with built-in camera.

Oregon Scientific. http://www2.oregonscientific.com

Hammacher Schlemmer. http://www.hammacher.com

Audio recording is easily accomplished with small recorders from Sony, Philips, Olympus, and others.

Martin, James A. 2008. “The Best Digital Voice Recorder.” PC World (April 23).

Digital Voice Recorders, Consumer search report. http://www.consumersearch.com/digital-voice-recorders

For GPS logging, I am currently using a Semsons i-Blue 747. Garmin, Magellan, and others also make devices.

Semsons Web site. http://www.semsons.com

Garmin Web site. http://www.garmin.com

Magellan Web site. http://www.magellangps.com

Geo-tagging:

Microsoft ProPhoto Tools Web site. http://www.microsoft.com/prophotodownloads/tools.aspx

HoudahGeo Web site. http://www.houdah.com/houdahGeo/

Yahoo Research has some excellent people working on geo-tagging. Their ZoneTag lets you upload you photos to Flickr with location tags.

ZoneTag Web site. http://zonetag.research.yahoo.com

Eisenberg describes cloud storage and sharing services of files, including Syncplicity and Dropbox.

Eisenberg, Anna. 2009. “Digital Storage Options for Workers on the Go.” The New York Times (January 17).

Dropbox Web site. http://www.getdropbox.com

Syncplicity Web site. http://www.syncplicity.com

Tools for home storage and media serving:

Microsoft Windows Home Server Web page. http: //www.microsoft.com/windows/products/winfamily/windowshomeserver/default.mspx

Apple’s Time Machine Web page. http://www.apple.com/macosx/features/timemachine.html

Apple’s Time Capsule Web page. http://www.apple.com/timecapsule

Microsoft Windows Media Center Web site. http://www.microsoft.com/windows/windows-vista/features/media-center.aspx

Apple TV. http://www.apple.com/appletv

Photo frames increasingly offer the ability to distribute photos throughout your home and include the ability to display photos, slide shows, video, and music.

Jacobowitz, P. J., and Zach Honig. 2008. “How to Buy a Digital Photo Frame.” PCMag.com (December 12). http://www.pcmag.com/article2/0,2817,2300977,00.asp

Digital Photo Frames, cnet reviews. http://reviews.cnet.com/digital-picture-frames

Family Genealogy databases, family-tree making, and time lines are available on the Web, e.g., FamilyTreeMaker.

FamilyTreeMaker Web site. http://www.FamilyTreeMaker.com

The top levels of Gordon Bell’s folder hierarchy, circa 2009. While there is no “right” organizing principle, it is a big mistake to not have an organizing principle, or to apply one inconsistently. The full Bell folder hierarchy contains more than 2,000 folders, holding more than 130,000 items. The top division is between active and archive items.

1. My Documents holding active content

a. Administrative and Systems

b. CyberAll aka MyLifeBits Research

i) Papers, patents

ii) Presentations

iii) Project plans

iv) Hardware (eBook, SenseCam, et cetera)

v) Conferences and other papers

vi) Classification, facets, metadata

vii) Database

viii) This book

c. Media-in-the-home research

d. Telepresence research

e. Systems of all kinds, chips to supers

f. Other active company and organizational (companies, government, schools)

g. GB in process books and papers

h. GB family financial and legal

i) CYxx or FYxx: Yearly bank, brokerage, expense, receipt, detailed tax, statements

ii) Investments

iii) Money and historical transactions

iv) Property

1. Real estate (folder per property)

2. A/V, cameras, electronics, phones

3. . . .

v) Start-ups (100+ company folders with business plan, history, ownership and official signed docs, stock certificate TIFFs)

vi) Tax for all years

vii) Wills, trusts, and family legal

viii) Family member(s) financial

i. GB personal and family members

i) X’s (for all X family members), including: Certificates (birth, . . . passport, et cetera)

ii) Health X (for all X family members)

iii) Society memberships

iv) Vitae

2. My Documents-Archive holding archival content

a. A Big Shoebox for most personal photos and video snippets

i) Family by year and albums

ii) Person(s) x

iii) Places

iv) Trips

v) SenseCam (o(1000) sequences of birthdays, celebrations, conferences, show, walks, et cetera)

vi) Things

vii) Food dishes, restaurants, et cetera

viii) Ephemera, memorabilia

ix) Animals

x) Underwater photos

b. Profession-specific photos for artifacts, computers, robots, people

c. My bio including articles, events, interviews, patents

d. My publications papers and reports

e. My talks and presentations

f. Other publications papers and reports

g. People, references, recommendations, vitae

h. Archived company and organizational folders (X)

i) Digital Equipment Corp. . . .

ii) NSF

i. Archived calendars and correspondence (t)

j. Archived files (e.g., DEC WPS, e-mail)

3. My Books books authored, books scanned

4. My Voice Conversations and Notes (telephone conversations are held in MyLifeBits database)

5. My Media, i.e., song collections from ripped CDs

6. My Videos including c. 1950s 8mm movies and lectures

Psychologists have identified “lifetime periods” as an important way that autobiographical memories work. Lifetime periods are thematic and include work or jobs, educational institutions, and relationships that exist over an extended period of time. These lifetime periods are an important part of my hierarchy, above.

Conway, M. A. 2005. “Memory and the Self.” Journal of Memory and Language 53:594-628.

Conway, M. A., and C. W. Pleydell-Pearce. 2000. “The Construction of Autobiographical Memories in the Self-Memory System.” Psychological Review 107, no. 2:261-68.

McNeely, I. F., and L. Wolverton. 2008. Reinventing Knowledge: From Alexandria to the Internet. New York: W. W. Norton and Company.

Hierarchies can be useful, but are sometimes too constricting. Many times you want to just access items by some attribute. For instance, you can probably access your e-mail by the attributes of date, subject, and sender, without designating that each of them occupies a particular level of a hierarchy. You can also see this when you shop at, say, Amazon, where you can sift cameras by brand and number of pixels. In the world of librarians, using attributes to organize things is called “faceted classification.” The Flamenco Search Interface project from the UC Berkeley School of Information has some good demos on faceted organization.

Flamenco Web page. http://flamenco.berkeley.edu

Hearst, Marti A. “UIs for Faceted Navigation: Recent Advances and Remaining Open Problems, in the Workshop on Computer Interaction and Information Retrieval,” HCIR 2008, Redmond, Washington, October 2008.

For those of you interested in trying a start-up:

Bell, C. Gordon, and John E. McNamara. 1991. High-Tech Ventures: The Guide for Entrepreneurial Success. New York: Perseus Book Publishing.

Nesheim, John L. 2000. High Tech Start Up, Revised and Updated: The Complete Handbook For Creating Successful New High Tech Companies. New York: The Free Press.

10. THE FUTURE

Nathan Myhrvold said, “On the hardware side, I’m pretty confident there’ll be another twenty years at least, which is another factor of a million. A factor of a million reduces a year into thirty seconds. Twenty years from now, a computer will do in thirty seconds what one of today’s computers would take a year to do. So, for particularly big computational problems there’s no point in starting. You should wait, and then do it all in thirty seconds twenty years from now! That is the hardware side. The growth of software is certain, because it’s only limited by human imagination.”

Brand, Stewart. 1995. “The Physicist.” Wired (September).

Bell’s law:

Bell, G. 2008. “Bell’s Law for the Birth and Death of Computer Classes.” Communications of the ACM 51 (1) (January): 86-94.

Sensors:

Warneke, Brett, Matt Last, Brian Liebowitz, and Kristofer S. J. Pister. 2001. “Smart Dust: Communicating with a Cubic-Millimeter.” Computer 34: 44-51.

http://robotics.eecs.berkeley.edu/~pister/SmartDust/

http://ceng.usc.edu/~anrg/SensorNetBib.html

Zhao, Feng, and Leonidas Guibas. 2004. Wireless Sensor Networks: An Information Processing Approach. San Francisco: Morgan Kaufmann.

ACM Transactions on Sensor Networks. http://tosn.acm.org

Sensors will be everywhere, and so will computational power. The ubiquitous and pervasive computing research communities are driving this forward. There are journals and magazines, including IEEE Pervasive Computing, Springer’s Personal and Ubiquitous Computing, and Elsevier’s Pervasive and Mobile Computing journal.

Weiser, Mark. 1991. “The computer for the 21st century.” Scientific American 265, no. 3 (September): 94-104.

Ubiquitous computing conference Web site. http://ubicomp.org

Pervasive conference Web site. http://www.pervasive-conference.org

Pattie Maes’s lab at MIT, which previously did the Remembrance Agent work, has recently unveiled a system called “sixth sense” that includes a wearable projector and camera. Any surface becomes a possible computer display, and you control the system using hand gestures (your hands are tracked by the camera). A hand gesture takes a snapshot. A virtual keyboard can be displayed to type on. Bar codes can be scanned while shopping and extra information about products projected on them. When you read the newspaper, extra video footage for a story can be played. When you read a book, annotations and extra information can be projected into the margins. Sixth Sense is a great illustration of the technological climate that Total Recall will live in: constant recording possible with a wearable camera, and your e-memory ready to consult and enjoy at any moment on any surface.

Sixth Sense demo video. http://www.ted.com/talks/pattie_maes_demos_the_sixth_sense.html

Sixth Sense Web page. http://www.pranavmistry.com/projects/sixth-sense.html

Mistry, P., P. Maes, and L. Chang. “WUW—Wear Ur World—A Wearable Gestural Interface.” To appear in the CHI ’09 Extended Abstracts on Human Factors in Computing Systems, Boston, Massachusetts, 2009.

Storage trends: See Chapter 1.

Unified communications: “Forrester Research said recently that the unified communications (UC) market in North America, Europe, and Asia Pacific will reach $14.5bn (£10.5bn) in 2015.”

Bell, Gordon, and Jim Gemmell. 2002. “A Call for the Home Media Network. Communications of the ACM 45, no. 7 ( July): 71-75. Association for Computing Machinery, Inc.

Montalbano, Elizabeth. 2008. “IBM Pledges $1 Billion to Unified Communications.” PC World (March 11).

O’Reilly, Paul. 2009. “Managing Unified Communications Performance.” CRN (March 9).

Semantic Web:

Berners-Lee, T., and J. Hendler. 2001. “Scientific Publishing on the Semantic Web.” Nature (26 April).

W3C Semantic Web Frequently Asked Questions. http://www.w3.org/RDF/FAQ

British Library Digital Lives Project and conference:

Digital Lives Research Project Web page. http://www.bl.uk/digital-lives

First Digital Lives Research Conference: Personal Digital Archives for the 21st Century. British Library, St. Pancras, London, February 9-11, 2009.

Randy Hahn helped us craft the story about him. The details are fictitious, but the scenario is completely realistic.