Hand gesture-based interaction in an immersive cinematic environment

Abstract

With cheaper head-mounted displays (HMDs) available in the consumer market, new virtual reality (VR) applications are being explored, one such being interactive VR movies. In conventional movies, the user passively watches a movie on a 2D screen and the level of immersion depends on the narrative of the story. In VR, the user's field of view is covered by the digital content, thus interaction with the digital content can affect the user's immersion in the virtual environment. This thesis explores how hand gesturebased interaction and hand appearance affect the user's immersion and embodiment in a 360 VR movie.

A prototype was developed by combining a Leap Motion controller and a SoftKinetic RGB-D camera. It captures the user's real hands and blends them into the virtual environment. The prototype also supports natural interaction with the virtual environment. A user experiment was conducted using the prototype to investigate the effect of handbased interaction and hand appearance on the user's immersion and embodiment. There are two conditions for the hand appearance: the real hand, and the virtual hand. There are also two conditions for the hand-based interaction: with interaction, and without interaction.

Results showed that the real hand increased the user's embodiment in a 360 VR movie. However, hand appearance did not have any effect on the user's presence. It was found that having hand-based interaction increases the user's embodiment in the 360 VR movie. Similarly, it also displayed no effect on the user's presence. User feedback collected from the study identifies the limitations of the study and the system.