Deb Roy

Leading Expert on Technology & Human Cognition, MIT Media Lab

BIG IDEAS

• Innovating the Customer Experience
  How can emerging technologies transform the consumer experience? As virtual and physical realities become increasingly intertwined, new ways for technology and analysis to reveal fresh perspectives on customer behavior, interactions, and preferences are evolving. By gathering longitudinal video from real-world settings, MIT Professor Deb Roy has invented new ways to reveal unprecedented new levels of insight into human behavioral and social patterns.
  
  Predictive models grounded in observational video will drive automated redesign of architectural spaces. Retailers will have the ability to reconfigure physical retail spaces based on consumer usage patterns. Marketers and sales managers will better understand customer service experiences, one-on-one interactions, anticipate the needs and desires of customers down to the level of the individual. Through the results of his research, Roy shares examples, data and projections for the future of the retail experience and design.
• The Rise of Human Data
  In the past decade we have witnessed the birth of novel forms of human data–data about people and created by people–enabled by new imaging and networking technologies. When combined with advances in Artificial Intelligence and data mining, we are in a position to reveal sweeping new views into the human condition that will leave few areas of business and government untouched. From genome sequences and brain images to longitudinal video and cell phone traces, new data about people are revealing insights into how we develop, think, behave, and interact. At the same time, billions of people have come online and are expressing themselves in forms that simply did not exist just ten years ago–from blogposts and podcasts to YouTube videos and twitterfeeds. By analyzing and cross-pollinating these rich streams of human data, our ability to understand, predict, and effect human behavior will be expanded dramatically with deep implications for health, finance, retail, government, and beyond.
• Human-Machine Intelligence
  Many of the most powerful ideas in Artificial Intelligence (AI) have yet to be realized because early practitioners in the field underestimated the difficulty of programming machines with detailed knowledge of how the physical and social world works. As a result, first generation AI systems were often brittle and unable to handle the rich diversity of situations that confront humans in everyday life. The rise of human data changes everything. We now have access to a flood of information of how people communicate and connect with one another that can be harnessed to create a new breed of AI that assimilates human interests and experience into hybrid forms of human-machine intelligence. Google’s search algorithm, which is grounded in human preferences and judgments, is an early example of the transformative effect of human-machine intelligence. As the capabilities of AI technologies and forms of human data evolve, competitive organizations must continuously adapt in shifting landscapes to find optimal forms of human-machine intelligence.

SNAPSHOT BIO

Deb Roy is an entrepreneur, innovator, and an expert on data analysis and interpretation. He is the founding director of the Center for Future Banking at MIT, which, in collaboration with Bank of America, explores how emerging technologies and insights into human behavior can transform customers' experience.  In this effort, he is joined by a multidisciplinary team of researchers and students with a passion for invention who are developing new ideas for the banking industry, and building and testing new working prototypes.

A pioneer in cognitive modeling, communication theory, and human-machine interaction, Roy is the AT&T Associate Professor at MIT and chair of the academic program in Media Arts and Sciences.  In this role he oversees the academic program of 140 masters and Ph.D. students at the MIT Media Lab.  In addition, he directs the Cognitive Machines group, a research team of 15 PhD students and staff working on several projects, including: the Human Speechome Project, a pioneering effort to understand how children develop language grounded in extensive longitudinal video; collaborative work with Autism researchers and clinicians to better understand the developmental course of the disorder in young children; and The Restaurant Game, a research project that will harness the power of the Internet and capture rich behavior and language by algorithmically combining the gameplay experiences of thousands of people playing an identical scenario.

In 2008 he co-founded his first start-up company in the consumer media space based on research in his lab.  A native of Canada, Roy received his bachelor of computer engineering from the University of Waterloo in 1992, his PhD in the Cognitive Sciences from MIT in 1999, and joined the MIT faculty immediately after in 2000. He has authored numerous scientific papers in the areas of artificial intelligence, cognitive modeling, human-machine interaction, data mining and information visualization.

A Closer Look at Deb

FOCUS AREAS
What's on Deb's current research agenda?

Human Speechome Project: Data-Driven Analysis of Child Language Development

Deb directs the Human Speechome Project, a pioneering effort to understand how children develop language grounded in extensive longitudinal video. In the pilot phase of the project, Deb’s team designed an embedded video capture system to record a 250,000 hour audio-video archive of his son’s first years of life at home. The resulting corpus provides an ecologically-valid foundation for exploring new scientific questions about child development. The team is developing data analysis techniques and cognitive models to uncover the patterns of social life in his home that guided his son’s assent into the world of words. Preliminary speech analysis suggests that caregivers may be capable of more sophisticated “fine tuning” of language to fit a child’s stage of development than previously suspected. The media has called the Speechome project “the most ambitious account of human language development ever attempted”  (Globe & Mail) and compared it to “humankind's first experiments with flight” (WIRED).

New Methods for Understanding Autism

Deb has begun a collaborative effort with Autism researchers and clinicians to better understand the developmental course of the disorder in young children. Technology derived from the Human Speechome Project provides ecologically-valid observational data of the child’s behavior in the home, providing a new way to study whether treatments translate into sustained and general effects in a child’s natural life.

Human-Machine Collaboration for Rich Media Analysis

Motivated by the need to analyze millions of hours of audio and video generated from home and retail settings, Deb’s team is developing a new approach to fuse machine and human processing for the tasks of large-scale speech transcription and large-scale video annotation. A key emerging concept is the creation of an operating system for human-machine teams to optimally leverage the complementary strengths of human and AI resources to solve a common task.

The Restaurant Game: Machine Learning of Social Intelligence

The goal of the Restaurant Game, now in its third year of operation, is to create socially intelligent AI agents that learn to use language meaningfully by watching people interact in online virtual worlds. If successful, this research could have transformative effects on the video game industry and in training systems that incorporate social simulators. To gather social interaction data, an online game (http://web.media.mit.edu/~jorkin/restaurant/) has been created in which two humans play the roles of waiter and customer in a virtual restaurant. Players can move and speak freely in a 3-dimensional restaurant to serve or consume a meal, leaving detailed behavioral traces in the process. Over 8,500 detailed game traces of virtual meals have been recorded to date. In initial experiments, a computer-controlled agent that has mined these traces and learned how to greet, seat, take orders, and serve the first course of a meal in convincingly human fashion.

Longitudinal Data-Driven Retail Behavioral Analysis

The long term vision guiding this project is to fluidly reconfigure physical retail spaces based on consumer usage patterns akin to dynamically reconfigurable online web sites that evolve content based on dwell times and click-through rates. Working with industrial partners, Deb’s team is gathering longitudinal video from real-world retail settings. Video analysis algorithms are being developed to mine recurrent patterns of activity that provide detailed insight into behavioral patterns and interaction effects with elements of the environment. The team is also developing predictive models grounded in observational video that may ultimately drive automated redesign of architectural spaces.