Yoav Y. Schechner*
Dept. Electrical Eng.
Technion - Israel Inst. Technology
http://webee.technion.ac.il/~yoav/

Widespread current cameras are part of multisensory systems with an integrated
computer (smartphones). Computer vision thus starts evolving to cross-modal
sensing, where vision and other sensors cooperate. This exists in humans and
animals, reflecting nature, where visual events are often accompanied with
sounds. Can vision assist in denoising another modality? As a case study, we
demonstrate this principle by using video to denoise audio.
Unimodal (audio-only) denoising is very difficult when the noise source is
non-stationary, complex (e.g., another speaker or music in the background),
strong and not individually accessible in any modality (unseen). Cross-modal
association can help: while the audio is noisy, the video is rather clean and
can direct the audio estimator. We demonstrate this in challenging experiments,
both in speech and music.

--------------------
* Joint work with Dana Segev and Michael Elad