Personal tools

Christoph Mayer

Recent research aims at enabling machines to utilize communication channels natural to human beings, such as gesture or facial expressions. Humans interpret emotion from video and audio information and heavily rely on this information during every-day communication. Therefore, knowledge about human behavior, intention, and emotion is necessary to construct convenient human-machine interaction mechanisms.

A preprocessing step often includes the segmentation of the face from the background. Skin color has been proven useful for this task. We developed an algorithm that adapts on image properties and therefore performs robust even in heavy light changes. The video below shows the original image data, the adaptive approach and a static approach that does not adapt to the specific image content.

Adaptive Skin Color Estimation

 

We rely on a model-based approach for facial expression recognition. Models represent knowledge about real-world objects, such as position, shape or texture via a model parameter and therefore provide an abstraction for the interpretation task. In case of facial expression recognition, the model reflects face related information like the position and rotation in 3D space, the rising level of the eyebrows or the opening degree of the mouth. In a sequence of images the model parameters have to be updated for every image in order to reflect the image content. For visualization, the video below shows a simple 3D tracking of a human face.

3D Face Tracking

 

In order to extract higher level information, the model parameters have to match the image content. The computational challenge of determining these model parameters is called model fitting and is tackled with a fitness measurement function between the image content and the model parameterization. These functions are either minimized to determine a good model fit or calculate a parameter update directly. The process of model fitting is visualized in the video below. In the beginning, the model parameters do not match the image content. However, by changing the model parameters the model fitness is increased. The fitness function is visualized by the color of the model, changing from red to green.

Face Model Fitting

 

From this information, machine learning techniques are applied to determine the facial expression. The image data in the movie below is taken from a standard database for facial expression recognition, the Cohn-Kanade facial expression database. Please see this reference for further information on the database: Kanade, T., Cohn, J. F., & Tian, Y. (2000). Comprehensive database for facial expression analysis. Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (FG'00), Grenoble, France, 46-53.

Facial Expression Recognition with 3D Models



Research Topics

  • Model-Based Image Interpretation
  • Model Fitting
  • Objective Functions, Learning Objective Functions
  • Face Detection, Face Model Fitting, Head Tracking
  • Facial Expression Recognition, Emotion Recognition
  • Color Detection, Adaptive Skin Color Classification

Document Actions