Real-Time Physically Based Sound Synthesis and Application in Multimodal Interaction

Public Deposited
Analytics

Downloadable Content

Download PDF
Request Version for Screen Reader
Last Modified
  • March 19, 2019
Creator
  • Ren, Zhimin
    • Affiliation: College of Arts and Sciences, Department of Computer Science
Abstract
  • An immersive experience in virtual environments requires realistic auditory feedback that is closely coupled with other modalities, such as vision and touch. This is particularly challenging for real-time applications due to its stringent computational requirement. In this dissertation, I present and evaluate effective real-time physically based sound synthesis models that integrate visual and touch data and apply them to create richly varying multimodal interaction. I first propose an efficient contact sound synthesis technique that accounts for texture information used for visual rendering and greatly reinforces cross-modal perception. Secondly, I present both empirical and psychoacoustic approaches that formally study the geometry-invariant property of the commonly used material model in real-time sound synthesis. Based on this property, I design a novel example-based material parameter estimation framework that automatically creates synthetic sound effects naturally controlled by complex geometry and dynamics in visual simulation. Lastly, I translate user touch input captured on commodity multi-touch devices to physical performance models that drive both visual and auditory rendering. This novel multimodal interaction is demonstrated in a virtual musical instrument application on both a large-size tabletop and mobile tablet devices, and evaluated through pilot studies. Such an application offers capabilities for intuitive and expressive music playing, rapid prototyping of virtual instruments, and active exploration of sound effects determined by various physical parameters.
Date of publication
Keyword
Subject
DOI
Identifier
Resource type
Rights statement
  • In Copyright
Advisor
  • Bishop, Gary
  • Manocha, Dinesh
  • Raghuvanshi, Nikunj
  • Lin, Ming
  • Klatzky, Roberta
Degree
  • Doctor of Philosophy
Degree granting institution
  • University of North Carolina at Chapel Hill Graduate School
Graduation year
  • 2014
Language
Publisher
Place of publication
  • Chapel Hill, NC
Access right
  • There are no restrictions to this item.
Date uploaded
  • April 22, 2015

Relations

Parents:

This work has no parents.

Items

Thumbnail Title Date Uploaded Visibility Actions
file details Ren_unc_0153D_14846.pdf 2019-04-09 Public Download