Towards Sound-Integrated Digital Twin for Hazard Detection and Response

Abstract

Auditory feedback is commonly used in virtual reality (VR) to support user performance, particularly when visual attention is highly engaged. However, most VR applications rely on event-based or spatially triggered sounds, leaving limited understanding of how eye fixations can trigger auditory feedback. This study investigates the effects of eye fixation triggered sounds on task performance in a VR categorization task. Participants classified virtual objects while auditory feedback was triggered by eye fixations. Task performance was evaluated using task completion time as well as eye fixation duration and haptic interaction time. Results showed that eye fixation triggered auditory feedback significantly reduced task completion time, eye fixation duration, and haptic interaction time compared to the no-audio condition. These findings demonstrate that eye fixation triggered sound can serve as an effective mechanism for supporting performance in visually demanding VR tasks and highlights its potential for future training and hazard-related applications.