I. EXECUTIVE SUMMARY

Paul Ekman and Terrence J. Sejnowski

I-A. Introduction

Grand challenges like space exploration and weather prediction are expanding human frontiers, but the grandest challenge is the exploration of how we as human beings react to the world and interact with each other. Faces are accessible "windows" into the mechanisms which govern our emotional and social lives. The technological means are now in hand to develop automated systems for monitoring facial expressions and animating artificial models. Face technology of the sort we describe, which is now feasible and achievable within a relatively short time frame, could revolutionize fields as diverse as medicine, law, communications, and education.

In this report we summarize the scientific and engineering issues that arise in meeting those challenges and outline recommendations for achieving those goals. First we provide background on the importance of facial expression.

I-B. Background About the Importance of Facial Expression

Facial expressions provide information about:

• affective state, including both emotions such as fear, anger, enjoyment, surprise, sadness, disgust, and more enduring moods such as euphoria, dysphoria, or irritableness;
• cognitive activity, such as perplexity, concentration, or boredom;
• temperament and personality, including such traits as hostility, sociability or shyness;
• truthfulness, including the leakage of concealed emotions, and clues as to when the information provided in words about plans or actions is false;
• psychopathology, including not only diagnostic information relevant to depression, mania, schizophrenia, and other less severe disorders, but also information relevant to monitoring response to treatment.

In basic research on the brain, facial expressions can identify when specific mental processes are occurring, periods that can be examined with new imaging technologies (e.g., Magnetic Resonance Imaging or Positron Emission Tomography) too expensive for continuous monitoring. Facial expressions also hold promise for applied medical research, for example, in identifying the role of emotions and moods in coronary artery disease.

In education, the teacher's facial expressions influence whether the pupils learn and the pupil's facial expressions can inform the teacher of the need to adjust the instructional message.

In criminal justice contexts, facial expressions play a crucial role in establishing or detracting from credibility.

In business, facial expressions are important in negotiations and personnel decisions.

In medicine, facial expressions can be useful in studies of the autonomic nervous system and the psychological state of the patient.

In international relations, analysis of facial expressions of world leaders can be used for assessing reliability and change in emotional state.

In man-machine interactions, facial expressions could provide a way to communicate basic information about needs and demands to computers.

I-C. Progress and Obstacles in Measuring Facial Expression

Development of methods for measuring the movement of the facial muscles, which produce the facial expressions, has taken great strides recently. Research using these new methods has shown promise, uncovering many findings in the diverse areas listed above. There now is an international community of more than two hundred facial researchers, but their progress is slowed because the facial measurement methods available are very labor intensive -- requiring more than one hour to measure each minute of facial expression.

I-D. The Opportunity

Recent developments in computer vision and neural networks, used primarily in lip reading, and the recognition of specific individuals from static faces, indicate the possibility of being able to automate facial measurement. Work is underway at a number of laboratories both in the United States and abroad, in university settings and in businesses, to take the next steps in developing semi or fully automated procedures for extracting information from faces.

The National Science Foundation has a crucial role to play in maintaining the U.S. technological lead in this area, in light of well funded efforts in other countries. There is more work to be done than can be funded only by NSF and other federal agencies. Foundations and the private sector will need to be involved.

I-E. Recommendations

This report contains a list of the major unresolved research issues and needed technological innovations related to efforts to measure the face automatically. Many basic science questions on how to interpret the messages of the face, how messages function in communication, and how they are produced and interpreted by the neuro-muscular system, remain open. Answering these questions is prerequisite for developing and applying automated measurements fully, but answering these question would, in turn, be expedited by automated measurement tools. Development of these tools is an interdisciplinary project requiring input from many areas of engineering, computer science, neuroscience, and behavioral science. Such efforts require both an infrastructure, such as multimedia databases and image processing computers, and training and education, such as post-doctoral programs and electronic communications, to support cooperation among investigators. Recommended tools, infrastructure elements, and educational resources are also listed below in the Recommendations, pages 53 to 57.

I-F. Organization of the Report

First the origin, planning, and goals of the Workshop are described. Then the Workshop's schedule and participants are listed.

Following this overview of the Workshop is a summary of each of the tutorial sessions. (The separate tutorials on Neural Networks and on Finding, Organizing and Interpreting Faces have been combined into one report).

Next are reports from each of the breakout work groups assigned to plan the steps for progress in areas relevant to this endeavor: basic science, sensing and processing, and databases and modeling.

A list of recommendations, integrated from across the different sections, follows. A discussion of the benefits of research and development in this field is in the final section.