Projection Mapping Is Revolutionizing Operating Rooms: Tracking Changes in Shape and Location of Organs in Real-Time

Solving Issues in Resection of the Liver with Projection Mapping

Animal organs are roughly divided into two categories. Those that have a cavity on the inside like the stomach and intestines are called "hollow organs", whereas those that are packed with tissue on the inside like the brain, liver and kidneys are known as "solid organs". Generally speaking, surgical procedures are more complicated with solid organs than hollow organs because of the profuse bleeding.

When performing, for example, a resection due to liver cancer, it is necessary to control bleeding and maintain liver function. Though the surgeon knows where the tumor is because of preliminary CT scans and so forth, the actual procedure is complicated because there are no markers on the outside of the liver indicating where to start cutting.

Recently, to make resections more efficient, simulations have been done prior to surgery by watching 3D images of the liver. Another technique that has drawn attention in the last few years is "Fluorescence Image-Guided Surgery" in which the doctor injects a fluorescent dye known as indocyanine green (ICG) into the body and performs the resection while watching the ICG image in real-time through an infrared camera. The ICG glows in near-infrared light.

Despite these advanced technologies, there are a number of remaining issues regarding resections of the liver as Dr. Satoru Seo, Department of Surgery, Kyoto University, explains.

"Preoperative simulation uses very high definition 3D imagery. However, the liver is an extremely soft organ and readily changes shape during surgery, so simulations lack a real-time component. Then, with fluorescence image-guided surgery, the doctor has to constantly look back at the monitor in order to locate the tumor, so frequently shifting vision can be stressful. Moreover, to prevent any interference by other light, surgical lamps have to be turned off and the procedure performed in the dark. Assistance is also needed to operate the infrared camera, as it is heavy."

Knowing how useful fluorescence image-guided surgery is as a procedure, Dr. Seo turned to Panasonic's projection mapping technology to solve the issues that prevent it from being even better.

Visualizing the Target by Tracking Changes in Organ Shape in Real-Time

Integrated with projection mapping technology from Panasonic and medical microscope manufacturing technology from Mitaka Kohki, MIPS projects the ICG image captured by an infrared camera directly on the patient's organ using a projector. The optical axes of the camera and projector are perfectly aligned, so any movements or shape changes of the organ are tracked in real-time.

Using MIPS, the surgeon can focus on the surgery without shifting vision back and forth between the liver and monitor. Moreover, MIPS shines a white light outside of the fluorescent area, therefore surgery can be performed without surgical lights, as just the light from MIPS suffices.

Katsuyuki Nakamura of Mitaka Kohki explained the role that his company played in developing MIPS in the following words.

"MIPS was designed to house Panasonic's imaging devices like their infrared camera and projector. At Mitaka Kohki, we manufacture surgical microscopes for neurosurgery, plastic surgery and other applications, equipped with a suspension arm that can be positioned with one finger on one hand. That know-how has been poured into MIPS to lessen the stress placed on doctors during long surgical procedures."

Hopes and Expectations of Using MIPS for Surgical Procedures on Other Parts of the Body Than the Liver

At Kyoto University, doctors are looking to use MIPS for sentinel lymph node biopsies, a procedure for verifying whether or not breast cancer has metastasized. Dr. Masahiro Takada, Breast Surgery Department, Kyoto University Hospital, explains how MIPS is useful in that.

"Searching through subcutaneous fat for the sentinel lymph nodes can take a considerable amount of time during surgery. But, we can inject ICG from the areola and easily trace its movement with an infrared camera until it reaches the sentinel lymph nodes. Conventional fluorescence image-guided surgery ran into this same issue as liver surgery. But, MIPS solves it."

Both liver resections and sentinel lymph node biopsies are highly specialized procedures. But, the way MIPS enables doctors to efficiently perform them is not the only merit it has to bear, because, if it can lower the associated surgical hurdles, it will give regional hospitals the means to offer these complicated procedures. Moreover, the less time needed to complete the procedures additionally reduces the physical stress placed on the patient.

As Dr. Seo described it, the diffusion of MIPS throughout the medical world will also help surgeons to enhance their skills.

"To a surgeon, the most important thing is anatomy. And, the basis of anatomy is in the blood vessels. If MIPS can visualize in real-time whether there is blood flow or not, it can be applied to a wide variety of surgical procedures. MIPS may very well revolutionize the medical world in the future."

Clinical Prototype Development Adopted by AMED

What got Kyoto University, Panasonic and Mitaka Kohki to develop MIPS in the first place? Looking back, Soichiro Kawamura of Panasonic's Connected Solutions Company recounted, "Back around 2013, our Advanced Development Department was searching for themes where our projection mapping technology could help solve problems in various fields. As good fortune would have it, about that time, some doctors from Kyoto University Hospital saw a projection mapping demonstration we did with a working prototype that combined an infrared camera and projector. They liked it and expressed to us that the technology might have surgical applications."

Repeated testing and demonstrations with Kyoto University got to a point where a working prototype seemed practical. To take development to the next phase, a request to conduct clinical trials was made to the Japan Agency for Medical Research and Development (AMED) in order to verify usability through their Acceleration Transformative Research for Medical Innovation (ACT-M) program. AMED welcomed the idea and a joint development project was launched. Over the ensuing two years of clinical trials at Kyoto University, MIPS was used in about 80 cases as a prelude to its practical application. Then, in 2017, the Security Systems Division took over the project from the Advanced Development Department with the goal of bringing MIPS to market. At the same time, the decision was made to bring Mitaka Kohki onboard because of their experience with manufacturing and selling surgical microscopes and other high-end instruments.

Panasonic delivers to Mitaka Kohki the key components they developed and manufacture for MIPS to process images. As Mr. Kawamura explained, "Panasonic's role in this project was to visualize things that were not visible by developing and integrating infrared camera and projector modules, a control unit, an image analysis program and so forth. Moreover, for the infrared camera, Panasonic used technology it had been using already to make cameras for medical devices, such as high-precision bonding on the submicron order."

Panasonic's Broad Technological Reach Was the Crux of Developing MIPS

Speaking about the development struggles, Yuhi Sasaki, who headed up the MIPS project, pointed out, "The Security Systems Division first and foremost develops commercial-grade cameras. But, MIPS not only needed to take pictures with a camera but also to output the pictures by projecting them without any misalignment. Fortunately, the Media Entertainment Division at Panasonic made projector units that performed better than products of other manufacture and, being in the same company, were easy to share information with. So, for this project, we decided to go with a projector unit of ours. However, the projector unit had not been made from the get-go for MIPS, so getting it to fit inside the MIPS head was tough. It was particularly hard to align the optical axes of the infrared camera and projector. But, we learned a lot through trial and error that can be utilized to create projection mapping products in other fields down the road."

As trying as they were, these struggles have paved the way to Panasonic's superiority in image processing solutions.

Mr. Kawamura added, "Because we were able to procure various units and parts from within the Panasonic Group, the project was a success. The idea of applying a projector made by the Media Entertainment Division to a medical device is something only Panasonic would come up with. This is a perfect example of how Panasonic can bundle the latest technologies from diverse fields into a working solution."

Cooperative Development Between Industry and Academia Will Help to Solve Issues in Other Fields

Throughout the development process, Kyoto University, Panasonic and Mitaka Kohki worked very closely together. Panasonic developed the first prototype, but engineers from Panasonic and Mitaka Kohki visited each other's plant for meeting after meeting in order to turn it into a more complete product. Kyoto University also chimed in for the regular bimonthly progress reports and frequent videoconferencing on Skype, so the three parties were constantly bouncing ideas off each other as to what features might be good to include.

Because of this development structure, Mr. Nakamura noted, "We had never worked so closely with another company before this. I came away very impressed by the high level of Panasonic's technological prowess, especially their basic technologies for infrared cameras and how they use projection mapping to accurately position and synchronize images, as well as their stable product supply system." Speaking as a physician, Dr. Takada added, "We all learned a lot from working with experts from other fields of endeavor. I was particularly amazed by how quickly Panasonic would come up with solutions to problems encountered in the field."

Talking about the future prospects of MIPS, Dr. Seo expressed his high hopes in Panasonic. As he put it, "Looking ahead, I'd like to see projection mapping with different colors for blood vessels, nerves, tumors and so forth instead of just visualizing blood flow monochromatically. If that becomes available, it will be easy to differentiate between what stays and what goes not just in liver surgery but procedures for other parts of the body as well. Knowing what Panasonic is technologically capable of, I'm sure that day will come."

- Reproduced from the website "Nikkei xTECH (Nikkei CrossTech)," by courtesy of Nikkei Business Publications, Inc.

# # #