Joint Audio-Visual Speech Processing

Dynamic Bayesian Networks for Audio-Visual Speech Recognition

The use of visual features in audio-visual speech recognition (AVSR) is justified by both the speech generation mechanism, which is essentially bimodal in audio and visual representation, and by the need for f...

Authors: Ara V. Nefian, Luhong Liang, Xiaobo Pi, Xiaoxing Liu and Kevin Murphy

Noise Adaptive Stream Weighting in Audio-Visual Speech Recognition

It has been shown that integration of acoustic and visual information especially in noisy conditions yields improved speech recognition results. This raises the question of how to weight the two modalities in ...

Authors: Martin Heckmann, Frédéric Berthommier and Kristian Kroschel

Statistical Lip-Appearance Models Trained Automatically Using Audio Information

We aim at modeling the appearance of the lower face region to assist visual feature extraction for audio-visual speech processing applications. In this paper, we present a neural network based statistical appe...

Authors: Philippe Daubias and Paul Deléglise

On the Relationship between Face Movements, Tongue Movements, and Speech Acoustics

This study examines relationships between external face movements, tongue movements, and speech acoustics for consonant-vowel (CV) syllables and sentences spoken by two male and two female talkers with differe...

Authors: Jintao Jiang, Abeer Alwan, Patricia A. Keating, Edward T. Auer Jr. and Lynne E. Bernstein

Editorial

Authors: Chalapathy Neti, Gerasimos Potamianos, Juergen Luettin and Eric Vatikiotis-Bateson

A Support Vector Machine-Based Dynamic Network for Visual Speech Recognition Applications

Visual speech recognition is an emerging research field. In this paper, we examine the suitability of support vector machines for visual speech recognition. Each word is modeled as a temporal sequence of visem...

Authors: Mihaela Gordan, Constantine Kotropoulos and Ioannis Pitas

Separation of Audio-Visual Speech Sources: A New Approach Exploiting the Audio-Visual Coherence of Speech Stimuli

We present a new approach to the source separation problem in the case of multiple speech signals. The method is based on the use of automatic lipreading, the objective is to extract an acoustic speech signal ...

Authors: David Sodoyer, Jean-Luc Schwartz, Laurent Girin, Jacob Klinkisch and Christian Jutten

Automatic Speechreading with Applications to Human-Computer Interfaces

There has been growing interest in introducing speech as a new modality into the human-computer interface (HCI). Motivated by the multimodal nature of speech, the visual component is considered to yield inform...

Authors: Xiaozheng Zhang, Charles C. Broun, Russell M. Mersereau and Mark A. Clements

Moving-Talker, Speaker-Independent Feature Study, and Baseline Results Using the CUAVE Multimodal Speech Corpus

Strides in computer technology and the search for deeper, more powerful techniques in signal processing have brought multimodal research to the forefront in recent years. Audio-visual speech processing has bec...

Authors: Eric K. Patterson, Sabri Gurbuz, Zekeriya Tufekci and John N. Gowdy

Joint Audio-Visual Tracking Using Particle Filters

It is often advantageous to track objects in a scene using multimodal information when such information is available. We use audio as a complementary modality to video data, which, in comparison to vision, can...

Authors: Dmitry N. Zotkin, Ramani Duraiswami and Larry S. Davis

Audio-Visual Speech Recognition Using MPEG-4 Compliant Visual Features

We describe an audio-visual automatic continuous speech recognition system, which significantly improves speech recognition performance over a wide range of acoustic noise levels, as well as under clean audio ...

Authors: Petar S. Aleksic, Jay J. Williams, Zhilin Wu and Aggelos K. Katsaggelos