Adroit Visuals That Track an Athlete's Movements - A New Age in Experience Value Engineered by Sports x Technology

A spectacular performance opens the ceremony

With the Olympic Games Tokyo 2020 finally just a year away, an absolutely spectacular "1 Year to Go!" ceremony was held on July 24, 2019.

Already before anything was happening, the hall was teeming with enthusiasm. Some 4,000 people between guests and those tied to the event in one way or another packed Hall A of the Tokyo International Forum, waiting anxiously on the edges of their seats for the show to begin. The junior high and high school kids seated in the front rows could be seen larking about posing for commemorative photos. Bearers of the next generation, these boys and girls will experience the Olympic Games Tokyo 2020 during their most impressionable years of their lives. So, what's not to get excited about?

Zap! Suddenly, the stage went dark and suspense filled the air. All breaths held, the audience looked on as the Yoshida Brothers from the Tsugaru region in northern Japan raised the curtain with an energizing duet on the shamisen. Their opening instrumental done, the brothers were swiftly whisked to opposites sides of the stage to make way for a mesh screen that floated in center-front. Then ...

Behind the screen, rhythmic gymnast Honami Tsuboi began dancing across the stage like a nimble fairy to the musical accompaniment of the Yoshida Brothers. The tip of the wand she held in her right hand painted a glowing trail of light that seemed alive as it followed her unbridled fluid motions. The dynamic and overtly "Japanese" imagery was fused into the dance using a technology known as "projection mapping." Very high-definition and 3D, it looked like Tsuboi would jump out of the screen at any moment. All eyes were glued to the stage in awe by this never-seen-before performance. The loud applause at the end told just how thrilled and amazed the audience was.

Because it makes perfectly good sense, most people probably thought that Tsuboi "danced beautifully in tune with the images." Yet, it was actually the opposite: the images followed Tsuboi's motions in real-time. The technology that does that was developed by Panasonic, an Official Worldwide Partner of the Olympic and Paralympic Games Tokyo 2020. This article goes behind the scenes - so to speak - to discover how real-time tracking and projection mapping was born and honed into the artistic tool it proved to be that day.

Technology has the power to change sports

This was not the first time that Panasonic showcased its real-time tracking and projection mapping technology and high luminance projectors. Just in relation to the Olympic Games, the new technology had already been announced in March 2019 at the "TOKYO VISION: 500 Days to Go! Night," a series of events planned 500 days out from the Olympic and Paralympic Games Tokyo 2020.

Hiroyuki Kobayashi of the Tokyo Organising Committee of the Olympic and Paralympic Games described the opening performance for the "1 Year to Go!" ceremony as follows.

"As is laid out in our vision of the Olympic and Paralympic Games Tokyo 2020, we want to be the most innovative in the history of the Games and bring about positive change in this world. By that, I mean our goal is to create new values beyond just the sports component, by adding technology and culture into the mix."

"When we thought about how to fuse sports, technology and culture, we decided to try something new using real-time tracking and projection mapping compatible projectors."

Having actually witnessed the performance, Mr. Kobayashi rated it as "powerful on a whole nother level in terms of the imagery, acoustics and movement." The impression he walked away with was that new content had been created by fusing together the distinctively separate elements of the Tsugaru shamisen, rhythmic gymnastics and the latest in imaging technology. He added, "This opening act was designed to energize those watching."

"Technological advances are creating opportunities for people who are not that familiar with sports to appreciate them more. That goes not just for playing but spectating as well. It is also an important role of ours to stir latent interests by conveying more and more the fun and attraction of sports through the Olympic and Paralympic Games Tokyo 2020."

"We see sports as capable of changing the future and technology as capable of changing sports. So, we expect something unimaginable to come from combining the two."

Real-time tracking at less than 2/1000 sec in latency

The system was developed by the Media Entertainment Business Division at Panasonic's Connected Solutions Company. The division makes projectors, flat panels, broadcasting equipment, sound equipment, conference systems and other BtoB solutions.

Kenji Fujiune, who hails from that division, recounted, "We started developing the system in 2015, bearing in mind that it was a very new challenge in the imaging industry and that we would be pushing speeds as far as they could go. This is meant to enhance the user's experience."

Projectors have been improved along the tracks of both picture quality and brightness. In terms of picture quality, there are SD, HD, Full HD and 4K products out there, while some high-end commercial projectors have reached 30,000 lm in brightness. Panasonic's focus on "speed," as opposed to the industry's standard approach to development, was because it wanted to go with something different.

Hirotatsu Asai proudly explained, "With projection mapping everywhere these days, people have gotten tired of looking at just images projected on the wall of a building. Real-time tracking and projection mapping integrates performances and imagery into something life-like."

The system consists of a projector for projecting the images, an infrared light for sensing and a high-speed camera. The infrared light is radiated on the marker at the tip of the performer's wand and picked up by the camera. The location of the marker in space is then pinpointed within a virtually negligible time lag by feeding the captured image into a processor built into the projector. Based on this positional information, the projector instantly projects the images at 1,920 fps (frames per second). This configuration allows real-time tracking with just a lag of 0.0016 sec from when positional information is detected until the content is projected.

According to Mr. Fujiune, most people who see the performance for the first time are likely to say something like, "The dancer went all out to keep up with the images." Mr. Asai noted, "You cannot detect the time lag with your eyes. Since a typical projector is 60 fps, the smoothness of the images deserves particular mention."

"At the 'TOKYO VISION: 500 Days to Go! Night' event, dancer Kento Mori performed 3 days in a row. But, what struck me was when he said it was a lot of fun to change up his dancing. Since the images track his movements, I gather he slightly changed his routine," added Mr. Fujiune.

Though the same technology as the "TOKYO VISION: 500 Days to Go! Night" event was used for the opening performance of the "1 Year to Go!" ceremony, the screen for this latter was four times larger. And, a total four projectors (including 1 backup) were on-site to enhance the brightness. As Mr. Fujiune put it, "The system blurs the boundaries between the real and virtual worlds. It augments the 3D effect and blending sensation by tracking high-definition, high-brightness images as an added component, whereby enabling a higher level of performance. The opening performance of the '1 Year to Go!' ceremony perfectly demonstrated that power of expression."

Looking to the future, Mr. Asai said, "If further evolved, the system could possibly project images on the walls and floors in someone's home so that they could feel like they are inside the games they play. Moreover, advances in technology may spawn sports that don't yet exist. This system takes us one step closer to realizing those kinds of dreams."

With virtual reality, one immerses him/herself alone because goggles are required, but with real-time tracking and projection mapping, multiple persons can share the "experience." Already, a positive response has been obtained from users who tried using the tracking feature with ping-pong, soccer and other sports. So, the fusion of sports and technology has already begun via the interaction between high-speed tracking and human motions.

Events will pick up even more speed next year and, before you know it, the Olympic Games will be upon us.

The "1 Year to Go!" ceremony was a great success, but it was just one stop along the way to the big show. Next year is the Olympic Year and come April the Olympic Torch Relay will make its way to Japan and the "TOKYO 2020 NIPPON FESTIVAL" will start.

As Mr. Kobayashi explained, "The TOKYO 2020 NIPPON FESTIVAL will be a compilation of all of the participatory events promoted to date. The main purpose is to acquaint people with the Olympic and Paralympic Games Tokyo 2020 by getting more of them to participate. With the Tokyo 2020 Games, we want to get as many people as possible interested in sports."

Official Worldwide Partner Panasonic thinks likewise. As Mr. Fujiune put it, "The Olympic Games are one opportunity for us to continue developing sports and entertainment. We can firmly support those foundations with our technology." Mr. Asai noted, "These will be the first Olympic Games since Panasonic feted its 100th anniversary. On the technology front, we want to make a bigger impact than all of the Games before these."

New events like karate, skateboarding and surfing have been added to the Olympic Games Tokyo 2020. Who knows? Maybe, "new technology-derived sports" will be adopted as new events two or three Games from now. Just the thought thereof conjures up exciting images of the future Olympic Games.

- Reproduced from the website "Mirai-kotohajime," by courtesy of Nikkei Business Publications, Inc.

# # #