Mixing sound and vision: The interaction of auditory and visual information for earwitnesses of a crime scene

Purpose. Previous research has shown that visual information impairs the perception of the sound of individual syllables, often called the McGurk effect. In everyday life sounds are seldom heard as individual syllables, but are embedded in words, and these words within sentences. The purpose of this research is to see whether auditory and visual information interact in the perception of a contextually rich scene that is of forensic importance. Methods. Participants were shown a video of a man following a woman. The man either says ‘He's got your boot’ or ‘He's gonna shoot’. Half the participants saw the actor say the same phrase as they heard, and half saw a different phrase than they heard. Results. When the visual and acoustic patterns did not match, people made mistakes. Many reported the fusion: ‘He's got your shoe’. Conclusions. This is the first demonstration of the interaction of auditory and visual information for complex scenes. The scene is one of forensic importance and therefore the findings are of importance within the emerging field of earwitness testimony.