Rapid and Accurate Facial Recognition: Even Faces Covered by Sunglasses or a Mouth Mask

Facial Recognition Server Software FacePRO

Needs for better security are growing each year, including a rise in social unrest caused by various incidents. In the area of facial recognition technology, other companies have already established a strong reputation. However, Panasonic is suddenly now on the offensive thanks to this highly accurate security solution with integrated Panasonic surveillance cameras.

Talk 1: Moving beyond Surveillance by Humans
Create Panasonic's Unique Face Recognition System with High-Precision Engine and Camera Technologies

Works with images taken at an angle or for individuals wearing sunglasses or a mouth mask

Your highly accurate facial recognition system has generated a stir in the security equipment industry. What was the key to developing it?

Sotodate: It began with an image matching engine developed by the Innovation Center. In 2017, it was rated the best in tests by the National Institute of Standards and Technology in the US.* FacePRO combines this high-precision engine with a surveillance camera. Using the camera's Best Shot function, poorer images are discarded, and the good ones are sent to the verification engine. This technology has led to great improvements in facial recognition.

Tsuneno: The important thing for software applications is to enhance accuracy while minimizing the burden on the data-receiving server. Subjects are continually moving in surveillance camera footage. Furthermore, the cameras are installed in places that are difficult to spot, such as ceilings. We performed repeated error identification and problem setting, and investigated the level of robust control that could be achieved. This means control that maintains stability even if the dynamic characteristics of the control target change slightly.

Takase: My job was system evaluation, and I was most focused on the kind of images that could be obtained. External factors vary depending on the camera location. It was difficult to respond to each case and give feedback to the development team.

Sotodate: I have been working on facial recognition since 2010, so I was familiar with the weaknesses of the previous engine. It was very poor at handling faces viewed from an angle and small faces. Those were development challenges for many years. Also, as development went on, identifying faces partially covered by a mouth mask became increasingly important. As is clear from viewing people on city streets in recent years, the number of people wearing masks is growing. The power of our technology is that even if someone partially hides their face, the system can still identify them.

Takase: I got a sense of the reaction to our product when we did a demonstration test at Haneda Airport. We performed onsite facial recognition in a location with a limited field of view. However, the system produced almost no false or erroneous reports. The system proved to be highly accurate after checking more than 60,000 people a day during a two-week test running 24 hours a day.

Sotodate: The trial also involved companies already established in facial recognition market. It was a big turning point because our slogan is, "Accuracy That No One Can Beat."

Engineers love to take on challenges

FacePRO has already been installed in many stores and facilities, so tell us about what you are working on currently.

Tsuneno: The system can be expanded to include up to 2,000 cameras and 100 servers. If it is installed in a large facility with a dedicated surveillance room, the operator can remotely tell the staff on the scene to chase a certain individual. Such a link to frontline staff is one of the keys to accelerating client operations, which is an essential requirement.

Takase: The most important task involves adding functions the client wants and making it even easier to use. We enhance the interface through repeated discussions with system engineers working at client sites. Through ongoing interaction, we get a sense of issues that are difficult to understand intuitively, enabling us to make improvements.

Sotodate: We want to enable the system to recognize certain individuals in real time when they enter the premises. Our strength lies in being able to modify the features of the camera. This is a big advantage because the entire system can be customized according to the type of verification the client wants.

Tsuneno: We can offer true value in the way the camera captures clear images and how the server matches them at the back end. This is a strength that our competitors lack. Going forward, we will take on the challenge of installation in spaces with even more visual noise.

Sotodate: If facial recognition can be performed not only in stores, but also in other public spaces, its application will be greatly expanded. Even in the darkness of night or with backlighting from the morning sun, the camera can see things that are not visible to the human eye. Our goal is to match and exceed the abilities of human surveillance. It is technically very difficult, but the challenge motivates us.

Takase: At previous product exhibits, clients asked us to compare our system with those of other companies. That changed with the launch of FacePRO. Now people remark on our system's ability to recognize even faces that are partially covered by a mouth mask. Like Mr. Sotodate and Mr. Tsuneno, who have been driving the project, as a facial recognition engineer I want to continue enhancing the value of FacePRO.

Talk 2: For Stable Operations
Value Is Determined by Client's Feeling. How Much Can It Be Improved?

Choosing the best, even in restricted situations

Please tell us the features of FacePRO from the standpoint of the system engineer, who makes the system operate at client sites.

Saito: Clients who operate retail stores are especially interested in the ability to identify faces that have aged over time. Customers who have visited the shop numerous times are of particular concern. Longtime clients can be recognized, so they are "amazed," saying "this could truly help us." We can never get such information or feedback until we visit client sites.

Miyata: Basic facial features do not change with age, and FacePRO can identify them. When the product was first released, its ability to identify partially masked faces also attracted a lot of attention. Once the product is well received, then the client's expectations will increase even further to the point where they want it to be 100% effective and never give a false report. The challenge for a system engineer is to ensure accuracy when the product is installed under restrictive and adverse conditions.

Saito: For example, when a camera is positioned at the glass entrance of a shopping center, light reflects off the front of the building and walkway surface, making faces in the captured images appear dark. Panasonic cameras are able to cope with this kind of backlighting, but it is still a challenge. We have to adjust the recognition parameters according to camera's installation location and tune the system to the site. There are ideal locations for setting up each camera, but we can't dictate to clients where the cameras should be placed (laugh).

Miyata: Sometimes we can't find a proper location to install a camera. When we get stuck at a client site, we call our colleague Mr. Akihiko Fukada, who has a great deal of installation experience. He is very familiar with the cameras and his experience is far greater than ours. We explain the situation to him, and he considers all the details, and immediately comes up with a solution.

Saito: We also share information with our colleagues about what we have learned and what issues we have faced. When we are out in the field, we make full use of the know-how we have developed as a team.

After operation begins, enhancement proposals are offered

What is the most important thing for popularizing facial recognition systems and expanding sales of FacePRO?

Miyata: It is important to put yourself in the client's shoes and solve problems facing the client. Even with FacePRO's high accuracy rate, face recognition is not correct every time. It is the client's feeling that determines its value. Whether it feels useful.

Saito: Even in the same retail chain, the manager's requirements can be completely different for each store. We can solve this problem by increasing the number of cameras, but the client has a budget. We have to consider the best possible system within their budget.

Miyata: The number of FacePRO installations has increased, but challenges still remain. This is especially true when customer expectations are very high. In such past cases, our system needed to be tested, improved, and checked by the clients several times before they were finally satisfied. Through this experience, I realized that the same procedure does not always work.

Saito: It is also key to be involved in the client's operations. This is because even if FacePRO issues an alert, it's not much use if no one is there to receive it. For example, at a client site, security patrols are not conducted every day, and on certain days of the week, security control is implemented only from a control center. Going over those details with clients can bring a different perspective, which has actually led to the creation of another security product.

Miyata: There are problems that customers themselves are not aware of. In the case of a retirement home, can a nurse be notified or can the alert be sent to a nurse's station or someone nearby? The approach will change after learning about the client's operation. If we can make an effective proposal on solutions to the problems, it will increase their satisfaction.

Saito: We and other facial recognition solution providers are in the stage of building up a new market. First, we want the world to see the value of facial recognition. We will then give our all to meet clients' needs so that they will choose Panasonic solutions.

Miyata: I think the worst thing is to lose a client due to an unsatisfactory proposal, rather than due to the price or the accuracy of the system. Therefore, we will continue to improve both our sales system and skills. After-sales service is also important. That is the key to selling system products.

Talk 3: Living up to the Great Expectations
FacePRO Facial Recognition Can Be Used Globally

I became convinced that FacePRO can be successful globally after we showed it to VIPs from the United States. They were very surprised by the accuracy of its facial recognition and became very interested in the system. The engine, which was developed jointly by the Innovation Center and the National University of Singapore, has been verified using facial images that include people from a wide variety of ethnic groups spanning the globe. In Japan, non-Japanese are an unfamiliar sight, and FacePRO is often better at identifying foreign faces than a Japanese observer, who might be swayed merely by the person's impression as a foreigner.

In July 2019, FacePRO was installed for a soccer club in a Danish stadium. The stadium is packed with fans during matches with rival teams. FacePRO was installed to prevent nuisance behavior and keep fans safe. With this major contract, Panasonic's facial recognition technology is now recognized by the public even outside Japan, and its benefits are also appreciated.

Also as a camera manufacturer, we have the potential to further expand our market share. Our sales companies outside Japan have very high expectations for FacePRO, and business talks are increasing. Of course, whenever there is an issue with recognition accuracy, we receive a call and provide support from Japan. Thanks to the system engineers and the development team, we have been able to troubleshoot and keep the systems operating properly. Although it takes time to develop the B2B business, we want to increase awareness of our facial recognition technology while earning the support of our clients.

* Evaluation points from the official evaluation report of the National Institute of Standards and Technology (NIST) as of May 9, 2017: https://www.nist.gov/programs-projects/face-challenges (The IJB-A Face Verification Challenge Performance Report, The IJB-A Face Identification Challenge Performance Report). This technology has been published as a paper. https://arxiv.org/abs/1704.00438

# # #