Skip to main content

Contact InformationContact Information


    As new devices enter medical use, doctors, nurses, and other medical practitioners are continually being called upon to quickly familiarize themselves with new equipment and skills. To facilitate this, Hitachi Solutions, Ltd. offers a service that uses MR technology from Microsoft Corporation. Based on consultations with experts from Medtronic Japan Co., Ltd., a Japanese subsidiary of the world’s largest medical device company, it was suggested that this service could be of benefit to theater nurses in operating theaters. Hitachi Solutions has now partnered with Hitachi Solutions Create, Ltd. to develop a training tool called HoloMe that shows where an experienced theater nurse directs their attention while working. This article describes the development of this tool, which became available from Medtronic Japan in March 2022, as well as the technologies used and the benefits anticipated.

    Table of contents

    Author introduction

    Mikiyo Yamaki

    Mikiyo Yamaki

    • Global Promotion Center, Global Division, Industrial Innovation Business Division, Hitachi Solutions, Ltd. Current work and research: Proposal and marketing activities focusing on mixed reality technology and Microsoft business applications to achieve digital transformation mainly for manufacturing companies. Certifications: METI Certified IT Strategist.


    Along with aging demographics and a low birthrate, the challenges facing community healthcare in Japan include increased workloads for doctors, nurses, and other medical practitioners and a shortage of resources relative to the number of patients. Meanwhile, advances in medical technology mean that these doctors and nurses also need to find time amid their busy daily schedules to familiarize themselves with the correct ways of using new devices. In operating theaters in particular, theater nurses, the nurses who assist the surgeon by passing instruments, also work with them to ensure that the surgery proceeds safely and smoothly, visiting the patient prior to surgery, conducting assessments, and making meticulous preparations, including the instruments and whatever else is needed to be ready for whatever situations may arise during an operation. During the surgery itself, they need to be able to anticipate how the operation is proceeding so as to assemble the right instruments and have them ready to pass to the surgeon when needed. Past practice has been to use on-the-job training to upskill inexperienced theater nurses, having a more experienced colleague alongside during operations to supervise and help them acquire the requisite skills. Unfortunately, valuable opportunities for inexperienced nurses to acquire these skills have been lost due to factors such as the COVID-19 pandemic, which has required the number of people present during operations to be kept to an absolute minimum while also taking away many learning opportunities such as participation in intensive training programs at the hospital or off site. This has created a need for more training opportunities for inexperienced theater nurses and new and efficient ways of teaching them how to handle medical devices correctly.

    Combining Nurse Upskilling with Workload Reduction

    The skill gap between experienced and inexperienced theater nurses manifests not only in their proficiency at selecting and arranging medical instruments, but also in their “watchfulness,” meaning their ability to anticipate what will come next during the course of a surgical operation. This watchfulness enables a theater nurse to know which instruments they need to have ready so that they can be handed to the surgeon as needed. Unfortunately, much as inexperienced nurses may wish to acquire this skill of watchfulness, it is hard for them to see where an experienced nurse is directing their attention during an operation. In response, Medtronic Japan Co., Ltd.; Hitachi Solutions, Ltd.; and Hitachi Solutions Create, Ltd. have developed a training tool that uses mixed reality (MR)*1 to provide a spatial representation of the direction of gaze of an experienced theater nurse who has good knowledge of how to use medical instruments and when to pass them to the surgeon. Inexperienced nurses can then learn these skills by watching them. Used in practice, this can reduce the amount of time experienced nurses spend on supervision and give the inexperienced nurses the ability to train during their spare time, requiring only that they have a suitable device available. This makes it possible to upskill theater nurses while also reducing their workload.

    A technology for viewing through a device where virtual objects can be displayed against a real-world backdrop just as if they were physically present.

    Use of MR in Work of Theater Nurses

    Advances in Hardware and Choice of HoloLens 2

    Figure 1 — HoloLens 2 Hardware SpecificationsFigure 1 — HoloLens 2 Hardware SpecificationsThe hardware specifications for the HoloLens 2 are listed above together with a frontal view of its component parts.

    A number of new virtual reality (VR) goggles have come on the market since 2019 when work on developing the MR-based training tool first got underway. Along with their external design, the cost of these goggles has progressively decreased while also featuring lighter weight and enhanced functionality compared to their predecessors. The HoloLens*2 2 from Microsoft Corporation comes with the Mixed Reality Toolkit (MRTK) for development and is equipped with infrared cameras in the lenses that track head movement. Using these, it is possible to track where the user is looking and to represent this within the virtual space. While this tracking and display of the user’s line of sight has typically required expensive specialist equipment in the past, the arrival of the HoloLens 2 means it can now be achieved for a much lower price. Furthermore, a device like the HoloLens 2, with its contactless operation, is particularly well-suited to medical environments where there are restrictions on which objects can be touched. Figure 1 lists the specifications of the HoloLens 2 used in this development.


    Figure 2 — Use of HoloLens 2 Eye Tracking to Show Sight LineFigure 2 — Use of HoloLens 2 Eye Tracking to Show Sight LineThe indicator object displayed mid-screen next to the medical instruments shows where the user is looking.

    Figure 3 — How HoloLens 2 Shows where User is LookingFigure 3 — How HoloLens 2 Shows where User is LookingThe IR cameras inside the HoloLens 2 lens are used to determine where the user is looking.

    1. HoloLens 2 eye tracking function
      The HoloLens 2 uses the infrared cameras in its lenses to track the user’s line of sight, providing this information as coordinates based on which indicator objects can be projected and visualized in real space (see Figure 2).
      This function was implemented using MRTK 2.4 and the Unity*3 development platform. Figure 3 indicates how the eye tracking function of HoloLens 2 is used to show the user’s line of sight.
      The line-of-sight information presented by this function shows more than just the main direction in which the user is looking, also providing details such as the speed of eye movement, length of gaze, and when they shift their gaze from one point to another. Showing the user’s line of sight in this way can be thought of as a form of human enhancement, with technology being used to augment human physical capabilities and senses.
    2. Design of spatial UI
      Using the HoloLens 2 to implement the eye tracking feature of this system also required the design of a user interface (UI) that is simple enough for anyone to use. The way the HoloLens 2 is designed to be used involves displaying menus or buttons in virtual space and allowing the user to interact with them by pointing, voice, or line of sight. In other words, it is very different from using a keyboard or mouse. To allow the theater nurses who are the intended users of this system to keep both hands free, it mainly uses voice commands, with simple menus designed to be easy to read and uncluttered by too much information. Likewise, the optimal display position is chosen so as to not impede the user’s field of view. They are also designed to feel much like pressing a real button, featuring visual cues and audio feedback.
      Utilizing these features, the HoloMe*4 training application was successfully equipped with a UI that even first-time users of the HoloLens 2 will find comparatively straightforward. Figure 4 shows a nurse using a HoloMe menu displayed in virtual space.
    3. Offline use
      As healthcare applications need to allow for cases where Wi-Fi*5 is unavailable due to the potential impact on other medical devices, HoloMe was designed to work offline.

    Figure 4 — HoloMe in Action in Medical PracticeFigure 4 — HoloMe in Action in Medical PracticeThe image on the left shows a voice-invoked menu and the right shows a nurse using a HoloLens 2.

    How Learning from Sight Lines of Experienced Theater Nurses Works in Practice

    Use of HoloMe is divided into two steps: STEP-1 and STEP-2. STEP-1 is already implemented. It involves having an experienced nurse wear a HoloLens 2 and record how they go about preparing for a surgical operation, including assembling the instruments that will be needed. The resulting video, which also shows their sight line, can then be viewed on a PC or other device. Trainee nurses can watch this to learn about the steps an experienced nurse goes through to prepare for an operation and where they are directing their attention (see Figure 5). Furthermore, a trainee can then undertake the same surgery preparation while themselves wearing a HoloLens 2, recording the steps they take and where they are looking. By replaying their own sight-line video on a PC alongside that of the experienced nurse, they can learn by observing how the two differ over time.

    STEP-2 is to be developed at some point in the future. Among the possibilities being considered is to use the virtual space to show the pre-recorded sight line of an experienced nurse (synchronized with the progress of the operation) alongside information about the real-time sight line of the trainee. The aim is to provide trainees with a more visceral and on-the-spot sense of how their own “watchfulness” differs from that of an experienced nurse, including the timing of actions. By taking full advantage of extended reality, it should be possible in the future to overcome physical and time constraints, for example by replicating an operating theater anywhere and at any time. It is hoped that MR will continue to deliver new value to the medical workplace in the future.

    Figure 6 shows the expected benefits of using this tool. Through the use of HoloMe, Medtronic Japan is targeting a 20% reduction in training times for both experienced and inexperienced nurses. It is anticipated that this will also help address workforce shortages as well as the long working hours of medical practitioners by shortening the time it takes for them to be able to work without supervision.

    Figure 5 — STEP-1 Process for Skills Training by Showing Sight Lines of an Experienced NurseFigure 5 — STEP-1 Process for Skills Training by Showing Sight Lines of an Experienced NurseThe figures show how trainees can learn by observing the sight lines of an experienced theater nurse captured along with video of a surgical operation.

    Figure 6 — Expected Benefits of Using HoloMeFigure 6 — Expected Benefits of Using HoloMeIt is estimated that use of HoloMe will reduce the time taken for training a medical practitioner by 20% (based on a target of reducing the number of 60-to-90-minute training sessions each week from five to four). It will also help address workforce shortages by shortening the time it takes for medical practitioners to be able to work unsupervised.

    HoloLens is a registered trademark of Microsoft Corporation.
    A game engine with a built-in integrated development environment (IDE) that can also be used to create MR, VR, and augmented reality (AR) content. Unity is a registered trademark of Unity Technologies or its affiliates in the U.S. and elsewhere.
    The name of the Medtronic Japan training tool.
    Wi-Fi is a registered trademark or trademark of the Wi-Fi Alliance in the USA and other countries.


    This exercise in collaborative creation (co-creation) with Medtronic Japan has involved the use of MR for medical training. Given the complexity of selecting and organizing medical devices, the vendors of this equipment strive to ensure that it is used correctly and safely. Along with detailed written user manuals, existing practices also include video manuals that can be viewed on a PC. The reason for using MR in this training is that it can explain the use of medical equipment in three dimensions (3D), an entirely new medium. When accompanying Medtronic Japan staff on visits to healthcare workplaces, Hitachi saw instances of medical staff coming in one after another at the end of a long day’s work to take turns using the latest equipment for training on suturing techniques. Medtronic Japan staff would likewise help out by bringing along a number of devices to coincide with times when medical staff could make themselves available. Were advances in MR to make possible training that is both realistic and able to overcome the constraints of time and place, it would help to reduce the burden on medical practitioners.

    For its part, Hitachi intends to continue developing and supplying technologies that can help to reduce the workloads of healthcare workers.


    Considerable assistance was received during the development of the HoloMe training tool described in this article from Go Ikeda, Kanae Hasegawa, and Satomi Miura of Medtronic Japan Co., Ltd. and Tetsu Muranaka and Shinji Hayashi of Microsoft Corporation. The author would like to take this opportunity to express her deep gratitude.