About the customer

Zakłady Automatyki KOMBUD SA is the leading supplier of solutions in rail traffic control. The company employs over 240 employees, more than 150 of which are automation and electronics engineers. They have operated in Poland for 30 years and have expanded their business to other European countries.

The challenge / Business requirements

The company was struggling with providing high-quality expert support for their demanding customers, which operate in all different areas of Poland and require to resolve issues in a very short time frame that usually is eight or even four hours. Quite often, the time to get to the incident for the technical engineer was longer than when they promised to resolve the issue. Therefore, they cooperate with local service companies with less experience but can come faster to fulfill those demanding requirements. This approach requires providing them remotely (often via phone call) support to quickly resolve the issue and keep high-quality standards.

Field engineer and remote expert has been using regular smartphones for audio connection and sending pictures or videos from the place of incident. However, this simple solution has a significant disadvantage: the field engineer must hold a smartphone in one hand and have only one hand available for operation. Because of that, video recording is not possible when both hands are busy.

Kombud asked us if we could provide them a solution that can utilize smart glasses with video cameras that allow field engineers to have both hands free and allow others to view in real-time and help when needed.

The solution

We suggested using HoloLens 2 Mixed Reality glasses with video cameras and microphones build-in. It doesn’t require holding in hand and adds the ability to use voice commands for faster interactions. Engineers and remote experts can see in real-time video or take pictures for further analysis or audit purposes.

In the beginning, we had recommended the Remote Assistance app from Microsoft. Still, soon we faced the problem that streamed video resolution was too low (is limited at 720p) to read small text or distinguish electronics components. HoloLens has a 4K camera and can record video with better quality. Our internal tests show that it is possible to record video with a resolution of up to 2272×1278 pixels that satisfies support and maintenance scenarios. So we decided to create a custom application that will stream high-quality video and add a few other features that are missing with the Remote Assistance app, such as modifying quality or taking pictures by the remote expert without disturbing the field engineer.

We have used WebRTC technology and a custom server to communicate from places with a slower internet connection to optimize video quality. We have discovered a few observations from the field. For example, there is no need to keep stable 30 FPS all the time. It is very bandwidth expensive, but we can lower it and have a clear video with lower network usage. We also separate audio and video streams and allow the remote expert to decide if only one or both channels are needed. We also enable field engineers to have complete control of when and what could be shared for privacy reasons and because there are some places where recording is not allowed for security reasons. We don’t want our users to feel that someone can spy on them, and in our opinion, it is crucial to explain to them that they have complete control over the device.

Although we didn’t utilize Azure communication services for this application, we used Azure hosting for our Linux routing server. As the application is used now as Proof-of-Concept and is not resource demanding now, it can be scaled up with Azure’s services when needed. We would also like to test Azure’s Mesh services for managing audio and video transitions soon when it will be available.

The partnership between supplier and customer

Our project started as a research project, but as soon as we did the first tests and had field results, KOMBUD was very interested in using this technology in their everyday maintenance, support scenarios, and as a learning tool for their service partners. As a result, a modified version of our app was publicly available at the KOMBUD’s booth during the fair trade TRAKO 2021 to showcase the benefits of this technology.

We are planning to continue our collaboration with KOMBUD as throughout the project, we’ve discovered together with new ideas worth implementing with Mixed Reality and HoloLens.