Sign Language in VR

A 360-degree silent video demonstration showing a sign language interpreter embedded within a virtual classroom environment. The interpreter's position remains static relative to the digital world as the camera pans around. Click and drag your mouse or rotate your phone to move the camera.

Hello, my name's Craig, I’ve been investigating sign language interpreter rendering and guiding methods in virtual reality 360-degree content. For many Deaf individuals, sign languages are their primary language, meaning text captioning may be inaccessible. Auditory accessibility is vital to ensure digital inclusion in technology and reduce the digital divide caused by inaccessible design. Virtual reality is a growing area of interest, with inclusive guidelines still actively emerging. The primary objective for this research was to establish presentation and guiding guidelines for sign language in three degrees of freedom 360-degree content.

Overall, research has provided no clear conclusions about VR subtitle presentation preferences. Sign languages have been explored more thoroughly in augmented reality, with findings indicating for example preferences for outward facing direction cues due to higher levels of glanceable understanding. The ImAc player was designed with customisable sign language presentation offered. However, recommendations for presentation of sign language are currently entirely based upon adopting subtitle guidelines.

Unreal Engine 5 was used to create the virtual environments, with the testing design following a within-subject two-factor independent variable design. Five environments were created, including the acclimation environment pictured here. This was designed to introduce all participants to the Oculus Rift head-mounted display and the testing procedure. The testing itself consisted of eight videos, with each participant viewing four testing videos, with condition and environment ordering both counterbalanced. Two environments were designed for rendering, containing non-visible narrators to isolate the rendering variable by eliminating the need for guiding. The two guiding environments meanwhile had two virtual characters placed in the environment opposite one another, with the participant placed in the middle to ensure that participants would be guided to speakers outside of their field of view. The following two slides showcase panorama screenshots of the environments, firstly consisting of a train station and a classroom for the rendering testing, and secondly consisting of a saloon and an office for the guiding testing.

The dependent variables were user experience, presence, sickness, and usability, all measured with separate questionnaires for each variable immediately following every video. User experience questions were based upon the UX framework, exploring issues such as sign language understanding and blocking. Measuring presence is vital for VR design, as inducing presence is the defining characteristic for many modern virtual environments. Literature indicates that non-visually integrated elements, such as 2D videos, may have the potential to exacerbate sickness due to mismatches between the moving background and the static foreground element. Finally, SUS was used as it provides reliable and highly consistent feedback on system usability, essential for understanding how usable the accessibility features are. Due to the use of Likert-scale data, semi-continuous total scores were examined with SPSS.

User experience results showed no significant overall differences. Individual user experience question differences were found between fixed-position and always-visible in terms of blocking, with the static placement of the interpreter used with fixed-position rendering having a strong negative impact on blocking. In terms of presence, significant differences were found between always-visible and fixed-position. It appears that the static placement of the interpreter on the video sphere led to this difference, with qualitative feedback supporting these statistical results. For sickness, all conditions showed low results. The testing procedure likely influenced these findings, with designs aimed at minimising sickness symptoms. Usability findings showed no significant differences between the rendering modes. There was a large difference in usability for guiding however, with arrows scoring A plus, whilst radar scored C minus.

For rendering modes, the two main feedback themes were presence and blocking, with fixed-position receiving mostly negative blocking comments, whilst always-visible received positive. Meanwhile, the majority of presence comments for always-visible were negative, with multiple participants mentioning the interpreter HUD presentation made the environment feel more artificial. This is opposed to fixed-position, where the majority of presence comments were positive, with comments that the static positioning meant the interpreter appeared more like she was present with them. For guiding, learnability was highlighted for both methods, with all participants quickly understanding how to interpret the arrows, whilst multiple participants took time to understand the radar. An issue highlighted by the arrows was timing, with participants missing parts of the sign language as they turned their head following arrows.

Numerous potential research avenues were highlighted, largely from semi-structed interview suggestions and think-aloud thematic analysis. Distancing appears to have an impact for fixed-position rendering, for example with potential personal space issues, highlighting the need for further testing. Arrow testing highlighted the issue of timing in VR, with testing needed for example to explore adding pauses to allow for head turn. Finally, sign language rendering and guiding must be explored in complex virtual environments such as 6 degree of freedom content.

In conclusion, fixed-position rendering appears to provide participants with a greater sense of presence than always-visible rendering due to the static placement of the interpreter on the video sphere, whilst always-visible rendering produces less of a blocking effect, with participants able to move the interpreter with head movements to view the environment. As both methods had similar high usability and UX scores, both rendering methods are viable for sign language presentation, with the best choice depending on content goals. For guiding, arrows appear to be the preferred visual guiding method due to being more usable than radar, including offering greater levels of enjoyment and understanding. Further testing is needed to explore the suitability of these requirements in six degree of freedom content.