Co-creation with Medtronic Japan
As new devices enter medical use, doctors, nurses, and other medical practitioners are continually being called upon to quickly familiarize themselves with new equipment and skills. To facilitate this, Hitachi Solutions, Ltd. offers a service that uses MR technology from Microsoft Corporation. Based on consultations with experts from Medtronic Japan Co., Ltd., a Japanese subsidiary of the world’s largest medical device company, it was suggested that this service could be of benefit to theater nurses in operating theaters. Hitachi Solutions has now partnered with Hitachi Solutions Create, Ltd. to develop a training tool called HoloMe that shows where an experienced theater nurse directs their attention while working. This article describes the development of this tool, which became available from Medtronic Japan in March 2022, as well as the technologies used and the benefits anticipated.
Along with aging demographics and a low birthrate, the challenges facing community healthcare in Japan include increased workloads for doctors, nurses, and other medical practitioners and a shortage of resources relative to the number of patients. Meanwhile, advances in medical technology mean that these doctors and nurses also need to find time amid their busy daily schedules to familiarize themselves with the correct ways of using new devices. In operating theaters in particular, theater nurses, the nurses who assist the surgeon by passing instruments, also work with them to ensure that the surgery proceeds safely and smoothly, visiting the patient prior to surgery, conducting assessments, and making meticulous preparations, including the instruments and whatever else is needed to be ready for whatever situations may arise during an operation. During the surgery itself, they need to be able to anticipate how the operation is proceeding so as to assemble the right instruments and have them ready to pass to the surgeon when needed. Past practice has been to use on-the-job training to upskill inexperienced theater nurses, having a more experienced colleague alongside during operations to supervise and help them acquire the requisite skills. Unfortunately, valuable opportunities for inexperienced nurses to acquire these skills have been lost due to factors such as the COVID-19 pandemic, which has required the number of people present during operations to be kept to an absolute minimum while also taking away many learning opportunities such as participation in intensive training programs at the hospital or off site. This has created a need for more training opportunities for inexperienced theater nurses and new and efficient ways of teaching them how to handle medical devices correctly.
The skill gap between experienced and inexperienced theater nurses manifests not only in their proficiency at selecting and arranging medical instruments, but also in their “watchfulness,” meaning their ability to anticipate what will come next during the course of a surgical operation. This watchfulness enables a theater nurse to know which instruments they need to have ready so that they can be handed to the surgeon as needed. Unfortunately, much as inexperienced nurses may wish to acquire this skill of watchfulness, it is hard for them to see where an experienced nurse is directing their attention during an operation. In response, Medtronic Japan Co., Ltd.; Hitachi Solutions, Ltd.; and Hitachi Solutions Create, Ltd. have developed a training tool that uses mixed reality (MR)*1 to provide a spatial representation of the direction of gaze of an experienced theater nurse who has good knowledge of how to use medical instruments and when to pass them to the surgeon. Inexperienced nurses can then learn these skills by watching them. Used in practice, this can reduce the amount of time experienced nurses spend on supervision and give the inexperienced nurses the ability to train during their spare time, requiring only that they have a suitable device available. This makes it possible to upskill theater nurses while also reducing their workload.
Figure 1 — HoloLens 2 Hardware Specifications
The hardware specifications for the HoloLens 2 are listed above together with a frontal view of its component parts.
A number of new virtual reality (VR) goggles have come on the market since 2019 when work on developing the MR-based training tool first got underway. Along with their external design, the cost of these goggles has progressively decreased while also featuring lighter weight and enhanced functionality compared to their predecessors. The HoloLens*2 2 from Microsoft Corporation comes with the Mixed Reality Toolkit (MRTK) for development and is equipped with infrared cameras in the lenses that track head movement. Using these, it is possible to track where the user is looking and to represent this within the virtual space. While this tracking and display of the user’s line of sight has typically required expensive specialist equipment in the past, the arrival of the HoloLens 2 means it can now be achieved for a much lower price. Furthermore, a device like the HoloLens 2, with its contactless operation, is particularly well-suited to medical environments where there are restrictions on which objects can be touched. Figure 1 lists the specifications of the HoloLens 2 used in this development.
Figure 2 — Use of HoloLens 2 Eye Tracking to Show Sight Line
The indicator object displayed mid-screen next to the medical instruments shows where the user is looking.
Figure 3 — How HoloLens 2 Shows where User is Looking
The IR cameras inside the HoloLens 2 lens are used to determine where the user is looking.
Figure 4 — HoloMe in Action in Medical Practice
The image on the left shows a voice-invoked menu and the right shows a nurse using a HoloLens 2.
Use of HoloMe is divided into two steps: STEP-1 and STEP-2. STEP-1 is already implemented. It involves having an experienced nurse wear a HoloLens 2 and record how they go about preparing for a surgical operation, including assembling the instruments that will be needed. The resulting video, which also shows their sight line, can then be viewed on a PC or other device. Trainee nurses can watch this to learn about the steps an experienced nurse goes through to prepare for an operation and where they are directing their attention (see Figure 5). Furthermore, a trainee can then undertake the same surgery preparation while themselves wearing a HoloLens 2, recording the steps they take and where they are looking. By replaying their own sight-line video on a PC alongside that of the experienced nurse, they can learn by observing how the two differ over time.
STEP-2 is to be developed at some point in the future. Among the possibilities being considered is to use the virtual space to show the pre-recorded sight line of an experienced nurse (synchronized with the progress of the operation) alongside information about the real-time sight line of the trainee. The aim is to provide trainees with a more visceral and on-the-spot sense of how their own “watchfulness” differs from that of an experienced nurse, including the timing of actions. By taking full advantage of extended reality, it should be possible in the future to overcome physical and time constraints, for example by replicating an operating theater anywhere and at any time. It is hoped that MR will continue to deliver new value to the medical workplace in the future.
Figure 6 shows the expected benefits of using this tool. Through the use of HoloMe, Medtronic Japan is targeting a 20% reduction in training times for both experienced and inexperienced nurses. It is anticipated that this will also help address workforce shortages as well as the long working hours of medical practitioners by shortening the time it takes for them to be able to work without supervision.
Figure 5 — STEP-1 Process for Skills Training by Showing Sight Lines of an Experienced Nurse
The figures show how trainees can learn by observing the sight lines of an experienced theater nurse captured along with video of a surgical operation.
Figure 6 — Expected Benefits of Using HoloMe
It is estimated that use of HoloMe will reduce the time taken for training a medical practitioner by 20% (based on a target of reducing the number of 60-to-90-minute training sessions each week from five to four). It will also help address workforce shortages by shortening the time it takes for medical practitioners to be able to work unsupervised.
This exercise in collaborative creation (co-creation) with Medtronic Japan has involved the use of MR for medical training. Given the complexity of selecting and organizing medical devices, the vendors of this equipment strive to ensure that it is used correctly and safely. Along with detailed written user manuals, existing practices also include video manuals that can be viewed on a PC. The reason for using MR in this training is that it can explain the use of medical equipment in three dimensions (3D), an entirely new medium. When accompanying Medtronic Japan staff on visits to healthcare workplaces, Hitachi saw instances of medical staff coming in one after another at the end of a long day’s work to take turns using the latest equipment for training on suturing techniques. Medtronic Japan staff would likewise help out by bringing along a number of devices to coincide with times when medical staff could make themselves available. Were advances in MR to make possible training that is both realistic and able to overcome the constraints of time and place, it would help to reduce the burden on medical practitioners.
For its part, Hitachi intends to continue developing and supplying technologies that can help to reduce the workloads of healthcare workers.
Considerable assistance was received during the development of the HoloMe training tool described in this article from Go Ikeda, Kanae Hasegawa, and Satomi Miura of Medtronic Japan Co., Ltd. and Tetsu Muranaka and Shinji Hayashi of Microsoft Corporation. The author would like to take this opportunity to express her deep gratitude.
