This research aimed to assess whether a virtual instructor and visual feedback combination displayed on a Mixed Reality (MR) mirror can be used to teach a beginner a simple dance routine, replacing the traditional instructor and mirror methods. A prototype was developed using a camera and projector that displayed a digital mirror image of the participant as they learned dances, with the system able to overlay computer graphics onto the image. The camera used to capture the image and motion of the participants was a Microsoft Azure Kinect camera.

Three visual feedback types were developed and used as randomized conditions in the user study based on input from expert interviews and an online survey. These were Spheres, Rubber Bands, and Arrows. Three simple dance routines were developed, motion captured, and presented in random order in the user study. During the user study participants learned the dances by following a virtual instructor in the MR mirror (present for each condition), with the MR mirror providing a different form of visual feedback for each dance. After practicing a dance three times with the feedback, participants then performed the dance in front of the MR mirror following the virtual instructor without any feedback, and the system measured the accuracy of their performance by comparing the amount of time that the user’s joints, such as shoulders and elbows, were within desired bounds for each pose.

Participants filled out an AttrakDiff Questionnaire describing their experience for each form of feedback, and gave comparative opinions of the different forms of visual feedback in a final interview.

The results showed that participants performed best with the Arrows feedback variant which were a directional feedback showing their depth difference, however they ranked this variant the lowest based on their own preference. The most preferred form of feedback was Spheres, which were the simplest feedback, not providing any guidance into the correct pose, but participants performed poorest with them.