Grasp-Shell vs Gesture-Speech: A Comparison of Direct and Indirect Natural Interaction Techniques in Augmented Reality

Type of content
Conference Contributions - Other
Thesis discipline
Degree name
University of Canterbury. Human Interface Technology Laboratory
Journal Title
Journal ISSN
Volume Title
Piumsomboon, T.
Altimira, D.
Kim, H.
Clark, A.
Lee, G.
Billinghurst, Mark

In order for natural interaction in Augmented Reality (AR) to become widely adopted, the techniques used need to be shown to support precise interaction, and the gestures used proven to be easy to understand and perform. Recent research has explored free-hand gesture interaction with AR interfaces, but there have been few formal evaluations conducted with such systems. In this paper we introduce and evaluate two natural interaction techniques: the free-hand gesture based Grasp-Shell, which provides direct physical manipulation of virtual content; and the multi-modal Gesture-Speech, which combines speech and gesture for indirect natural interaction. These techniques support object selection, 6 degree of freedom movement, uniform scaling, as well as physics-based interaction such as pushing and flinging. We conducted a study evaluating and comparing Grasp-Shell and Gesture-Speech for fundamental manipulation tasks. The results show that Grasp-Shell outperforms Gesture-Speech in both efficiency and user preference for translation and rotation tasks, while Gesture-Speech is better for uniform scaling. They could be good complementary interaction methods in a physics-enabled AR environment, as this combination potentially provides both control and interactivity in one interface. We conclude by discussing implications and future directions of this research.

Piumsomboon, T., Altimira, D., Kim, H., Clark, A., Lee, G., Billinghurst, M. (2014) Grasp-Shell vs Gesture-Speech: A Comparison of Direct and Indirect Natural Interaction Techniques in Augmented Reality. Munich, Germany: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 10-12 Sep 2014. 73-82.
Augmented reality, natural interaction, multi-modal interface
Ngā upoko tukutuku/Māori subject headings
ANZSRC fields of research
Field of Research::08 - Information and Computing Sciences::0801 - Artificial Intelligence and Image Processing
Field of Research::08 - Information and Computing Sciences::0806 - Information Systems::080602 - Computer-Human Interaction