Machine Vision, Machine Learning, and Mixed Reality for Worker Training

We use machine vision, machine learning, and augmented reality (AR) in order to be able to give real-time feedback if a process or a component has been done correctly or not. This is achieved by modelling through our algorithms a process performed by an expert and then asked to monitor it while it is done by a trainee or by another user on the production line. Alternatively, the verification can be done straight from a CAD model. Thus, either with a process done by an expert, or taking design specifications from Computer-Aided Design (CAD), it is possible to guide a user through a procedure, verify, and provide real-time feedback if they are doing it correctly or not as they are building it.

This is without the need to wait until they have finished the process before being able to inspect it. At the end of the process an automated report can be created with the results of the build, who worked on it, under what conditions was it built, and the inspection results. We build our own cameras and software. Thus, it can be customised to required specifications. 

We have also developed our own AR indoor and outdoor navigation system. Thus, by combining this system with the platform mentioned above it is possible to guide a user to a certain place (i.e. where a specific part is located), track, monitor, inspect, and verify a process, material and people flow, real-time production updates and live inventory tracking.

The technology consists of our own hardware and software. The hardware consists of our proprietary cameras which are customised specifically for the tasks required. These can be a combination of normal cameras, infrared for depth sensing, or even thermographic cameras. It all depends on the task at hand, and accuracies needed. We also provide our own servers if the application requires to have an on-site server which can give extra processing power and robustness.

The software is composed of machine learning algorithms, and the required interfaces in order to allow the communication between all the different software and hardware such as cameras, sensors, phones, tablets, laptops, headsets, servers, CAD software, etc. This is what then enables the possibility to do real-time verification with AR as well as AR indoor and outdoor navigation and tracking.

• Manufacturing (Automotive, Aerospace, Shipbuilding, etc)
  • The platform can be used to guide new or experienced users through a process as well as provide real-time inspection and verification if a process is being executed correctly or not. Users will be able to detect errors earlier in the process or even avoiding them. It is possible to have an automated reporting system of production at the end if required.
• Construction
  • It is possible to guide a worker through a build process and notify them in real-time during the building of a certain section, to check if what they are doing is correct or not. This can be verified straight to the CAD and the design specifications. Furthermore, our cameras can be placed on drones or robots at the end of a day and be asked to do a full scan and inspection of the construction site. Then a report can be produced highlighting progress done that day, what is missing, what could potentially be wrong or how far off from design specifications something is, etc.
• Medical
  • Ability to provide real-time assistance, verification, and feedback to a surgeon while operating on a patient. It can also be used to train future doctors, or as an assistive tool for the new doctors which are conducting their first procedures, to ensure that they are not missing any steps, and that they are carrying out the procedure correctly.
• Leak Detection
  • If our platform is paired with a thermographic camera it is possible to conduct leak detection procedures for pipelines and tanks. This could be in automotive, aerospace, oil & gas, etc.
• Sports and rehabilitation
  • Our platform can be used to record a professional perform certain exercises, then have a user perform those same movements, capture them, and analyse them against the expert’s movements. This then allows for the possibility of giving a breakdown to the user through AR on how well they performed compared to the professional, and what they did right or wrong. This can be used by professional sports as engagement with fans, or professional teams can use it to monitor and improve performance. It could also be used by medical staff to provide therapy to patients, and they can monitor themselves at home to ensure they are doing the rehabilitation exercises correctly.

There is guidance on how to carry out procedures through AR. However, at this moment in time, a platform which can allow a user to verify if an activity has been performed correctly in real-time as he or she is executing it is rare. The current practice is that the worker builds a product first and then a supervisor inspects it. By that time, it may be too late to address product issues.

Extra time and money will be required to rectify the problem and perform the required re-works and modifications. Furthermore, the current procedures rely heavily on the user confirming if they have done something or not, and how well they performed. With our platform, the monitoring and inspection can be automated. In this way, it can immediately be ascertained if they have done certain tasks correctly and if workers have completed the tasks.

This platform can also function as a knowledge transfer technology, providing users tutorlege opportunities with skills, tools, and capabilities which they previously did not have. Users will be able to go through different camera lenses to see the environment in different ways such as infrared, thermographic, or a normal camera feed, all of which perspectives will be augmented with valuable information that can better enable users to make decisions.

Is there something wrong, is it a go or no go, is there anything missing, or that we should be aware about? All of these is possible without the user needing to be experts in the areas or procedures.