Interaction in augmented reality based on hand detection with stereo-vision technique

Abstract

Recently there has been a wide range of research in the application of Augmented Reality. Advanced AR technology allows users to experience augmented objects together with the real world simultaneously. Augmented objects are able to overlay physical elements in real-world environment according to how the users interact with them. Both marker-less and marker-based AR can achieve this sort of interactive manipulation. In this Final Year Project, a novel interaction framework for marker-less Augmented Reality is built. Modelbased detection is one of the solutions for marker-less Augmented Reality. A marker is used to be the label of the position, on which the augmented object is placed on. Instead of a marker, human hand is used as a distinctive object on which the augmented object placed in this interaction framework. Leap Motion, a Virtual Reality (VR) hand tracking device, is used to detect the hand in our interaction framework for marker-less Augmented Reality. 3D hand position, gesture and direction can be obtained by using Leap Motion and passed to Unity 3d game engine. The main objective of this final year project is to provide a cheaper solution for hand-based interaction in Augmented Reality. Instead of using a seethrough display, a VR head mounted display is used to illustrate what users are able to see through a AR head set by displaying the real-world image as the background of the VR head mounted display. Thus, the main task for this framework is to calibrate the actual hand displayed as the background and the virtual hand generated by computer so that they can overlay each other. With the help of marker-less Augmented Reality Framework, a user can experience intuitive interaction with virtual object and natural occlusion using their hands which will be the core functionality for next generation game, education, user interface and industrial. Meanwhile, hand based interaction with the virtual objects for Virtual Reality and Augmented Reality devices will make these devices more usable and interesting since user will not need to hold controllers and directly interact with virtual objects using their hands