Portable lab to study audio-visual perception invirtual reality

Abstract

Workshop 7 Tuesday Helium - Audio-visual perception in VR: Noise and its multifarious effects on productivity, wellbeing, and health within the built environment are general knowledge. Within the continuum of ecological validity, transferring the ‘in the wild’ noise scenarios to laboratory conditions offers a tremendous degree of control that is generally offset by the accompanying lack of ‘realness’ of laboratory simulations. At our institute (IHTA), we have been pushing the boundaries of what is possible in laboratory conditions, in terms of representing the complexities of human interactions with noise in certain contexts.
This is made possible by integrating state-of-the-art audio-visual simulation methods. The visual models run on game engines, e.g., Unreal Engine, while the acoustic simulation is performed with the open-source software Virtual Acoustics (VA), which has been developed at our institute. Regarding the visuals, the game engine can apply physics-based renderings (PBR), which allow photorealistic rendering of materials, direct and indirect illumination, and weather effects. From the acoustic perspective, VA includes the main effects of sound propagation in the atmosphere as well as reflection, absorption, and diffraction due to ob-jects. By using a head-mounted display and headphones for the immersive audio-visual playback and by running the software on a portable computer, the whole laboratory becomes highly portable.

Participants of the workshop will be able to interactively explore the lab, e.g.:
Urban planning: Urban planners and architects are interested in the effects the environment they design has on inhabitants, clients or future users. The lab can be used to collect subjective evaluations of different designs of an urban area, e.g., a public space or a park surrounded by buildings. For this purpose, study participants are shown a graphical user interface (GUI) within the virtual scene which allows for interaction, e.g., change the design or weather conditions, and to answer embedded questionnaires.
Classroom / open-plan office acoustics: In the context of indoor scenarios, we focus on characterizing the effect of noise and other indoor environmental quality factors on the cognitive development and mental health of children in classroom conditions in Germany. Our approach is to characterize auditory distraction and speech communication for office employees within audio-visual simulations of the respective contexts. Using the lab, virtual classroom or open-plan office representations can be used to put study participants in the position of a plausible learning/working environment. In such an environment, the performance of cognition tasks (via a GUI) under multiple auditory stimuli can be studied in detail.