AR Maintenance Case Study

Hackathon Hackvention VR/AR 2017
(Hannover, Germany) 24-26.11.2017

The application has won: Best industrial App award.

The task was to create an interactive holographic repair manual that would allow the user to perform simple repair tasks without prior knowledge of the device.

Challenge Brief

The scenario was based on real world need that was presented by our friends from BTC Consulting.

There are a lot of different machines on remote oil platform that break from time to time. The malfunctions can be easily discovered but the repair process normally requires presence of a specialized expert. The challenge here was to develop a solution that would allow a non-expert person to easily repair the damaged unit. The additional challenge here was that the application would have to work totally offline so there won’t be a possibility to contact the expert or external support anyhow.

Microsoft HoloLens was the desired hardware because of its advanced augmented reality capabilities.

Our test machine to “repair” was an old boiler.

“Exploded view” all the important parts highlighted (HoloLens Screenshot)

Application’s main features

  • Precise positioning of the Hologram over an real life object.
  • Interactive user interface that would show the description about each step
  • Interactive manual with detailed description of each step required to be made in order to complete the repair.
    Each step consisted of:
    • text description
    • visual animation overlaid on real life model
    • voice over instruction
    • safety precautions if needed
  • 3D minimap hologram acting as an navigation help
  • Advanced visualizations for the repaired device
    • “exploded view” of all parts
    • see- through model
    • highlight of hazardous machine elements
  • All UI elements were easily manually adjustable (position and size)

The application usage is pretty straightforward. The user starts the program on HoloLens and then has to follow the app narration:

  1. Setup the correct position of the hologram so it correctly overlays the real life device.
  2. Acknowledge the information about safety precautions and tools required to complete the task.
  3. Manually adjust the position of UI elements if the starting position of these elements is not optimal for the user (different user height, size of the room etc.)
  4. Follow the interactive manual showing what has to be done in each step in order to fix the machine.
Manual Adjustment of 3d “minimap” model

Biggest challenges and solutions

The two biggest challenges in this project were the positioning the hologram in relation to the real object and developing intuitive user interface.

Calibrating the hologram on real world object

1. Precise positioning of the Hologram over an real life object.

There were 3 approaches to solve this task. We could use 3d object recognition, a special marker and manual positioning. The first two options would also require manual adjustments afterward.

After briefly experimenting with marker solution we’ve quickly come to conclusions that this solution wasn’t precise enough and would not be possible to develop it in such short time.

We’ve developed a simple interface that would allow the user to position and calibrate the 3d model of the boiler in the correct relation to the real one.

Complete user interface overview (HoloLens Screenshot)

2. Intuitive UI and good user experience.

In this case one of the most important aspects was that we were designing not for a screen but for the space.

HoloLens has relatively small field of view. That’s something to bear in mind when defining the size and the position of the UI elements. We’ve also head to develop an simple system of arrows and pointers that would point the user important elements that were currently out of the HoloLens FOV.

Due to specific rendering in HoloLens it’s really important to constantly test the design on the device. The size of the text, it’s colour and contrast vary greatly between computer screen and HoloLens projection. It’s also worth mentioning that the UI elements also had to be optimized so rounded edges, transparencies and additional ornaments may impact the overall performance of the application.
That was also the main reason why wanted to keep our UI simple and clean.

Our biggest achievement was that our application turned out to be really easy to use and understand. The competition judges were able to finish the demo repair process easily and without any help or instruction from our side.

Piotr wearing technicians uniform, using HoloLens while testing application before the final pitch.

Summary

We’ve approached this challenge mainly because we were curious if HoloLens would be the right tool for this kind of job. We wanted to check how well the Microsoft's device would perform in real-life problem scenario and how it will perform.

After successfully developing the prototype we are now able to confirm that HoloLens is capable of fulfilling such tasks, and despite its limited field of view it can be successfully used in work environment.

We’ve had a lot of fun creating a “Ghost in the shell”-like experience for repairing a simple boiler ;-)

The everyday usage of AR interfaces might be closer to reality than we all think.

Want more details?

Contact us!

Recent works