The system utilises Augemented Reality (AR) Technology to provide visual instructions overlaid in the real environment for the trainee. The trainer is allowed to create these AR instructions using the editor provided by the application. A user just needs to point their device to the area of interest to view those instructions. The application is deployable on Android devices and it utilises ARToolKit as its tracking technology.
In this work, we present a practical system which enable mobile devices to be used as interactive manuals. In particular, there are two modes provided in the system, namely, expert/trainer and trainee modes. Given the expert/trainer editor, experts design the step-by-step interactive manuals. For each step, the experts capture the images by using phones/tablets and provide visual instructions such as regions of interest, text, and action animations. In the trainee mode, the system utilizes the existing object detection and tracking algorithms to identify the step scene and retrieve the respective instructions to be displayed on the mobile device. The trainee then follows the displayed instructions. Once each step is performed, the trainee prompts the device to proceed to the next step.
The technology may be used in the area of repair and maintenance in industries where processes and people are constantly changing. The trainer can create and edit the AR instructional steps for operating equipment.
The trainee can use these AR Instructions without the need to interpret instructions from thick and cumbersome manuals which may lead to errors. These just-in-time instructions will also help to reduce effort and time required to train new staff. The trainer is also allowed to create and edit these AR Instructions without any programming knowledge.