Panasonic Develops Hyperspectral Imaging Technology with the World's Highest^*1 Sensitivity

Osaka, Japan - Panasonic Holdings Corporation today announced it has developed the world's highest sensitivity^*1 in hyperspectral imaging technology^*2 by applying a compressed sensing technology^*3, which has been used in medical care and space exploration. This technology makes it possible to identify slight color differences-small differences hard to discern with the naked eye-maintaining the usability of a conventional color camera^*4, and enables improvement of image analysis and recognition accuracy. As the world's first demonstration of such efficient hyperspectral imaging, it was published on January 23, 2023 in the online version of the British scientific journal Nature Photonics under a joint name with imec, a Belgian research institute.

With the evolution of image recognition technology, the application of machine vision is expanding, including industrial use of image data for efficiency, labor saving, and energy saving. Since machine vision recognizes images on a computer, information that humans cannot perceive, such as continuous color changes (spectral information), can be used to make analyses. Images with spectral information, known as hyperspectral images, are expected to play a role in expanding the application fields of machine vision.

In conventional hyperspectral imaging, optical elements such as prisms and filters that selectively pass light of a specific color (i.e., wavelength) have been used. However, since these methods detect light separately in each wavelength, there is a physical restriction in that light utilization efficiency (i.e., the sensitivity) decreases in inverse proportion to the number of wavelengths. Therefore, illumination with a brightness comparable to that of the outdoors on a sunny day (illuminance of 10,000 lux or more) was required to shoot, which decreases usability and versatility.

The newly developed hyperspectral imaging technology employs compressed sensing, which efficiently acquires observation data by "thinning out" and reconstructing data to restore data to what it was before thinning out through image post-processing. The compressed sensing technology is also used in MRI examinations in the medical field as well as black hole observations. A special filter that transmits multiple wavelengths of light to appropriately thin out data is implemented on an image sensor (Fig. 1), and the image reconstruction is carried out by a uniquely optimized algorithm for digital image processing. By leaving a part of the color-separating functions to the software, Panasonic Holdings have overcome the trade-off between the number of wavelengths and sensitivity, the fundamental issue of conventional technology. This approach has made it possible to capture hyperspectral images with the world's highest sensitivity (Fig. 2) and video (Fig. 3) under indoor illumination (550 lux).

Utilizing the developed technology, Panasonic Holdings will collaborate with our partners to provide new spectral sensing solutions, including highly accurate image analysis and recognition, and to expand machine vision applications with highly sensitive hyperspectral imaging technology.

In conventional hyperspectral imaging, optical elements such as prisms and filters that selectively pass light of a specific wavelength are used to detect light of the wavelength assigned to each pixel of the image sensor. However, these methods have the physical restriction that light of the non-assigned wavelength is not detected in view of signal detection at each pixel, and the sensitivity decreases in inverse proportion to the number of wavelengths.

Therefore, Panasonic Holdings developed a special filter employing a distributed Bragg reflector^*6 (DBR) structure, utilizing the wave nature of light, and implemented it on an image sensor (Figs. 1a and 1b). The special filter is designed to transmit incident light with randomly changing the intensity for each pixel and wavelength (Fig. 1c). The thinning out of data corresponds to change the intensity for each pixel and wavelength. The data pre-thinning out can be obtained after the data reconstruction process as long as the data is detected with thinning out in appropriate way. In Panasonic's newly developed technology, color separation is partly performed in the reconstruction process on software, and thus the physical restriction of sensitivity, which has been a fundamental issue in conventional hyperspectral imaging, can be overcome.

The improvement of sensitivity can be achieved by using the special filter since multiple wavelengths of light are transmitted and detected. Specifically, a developed special filter transmits about 45% of the incident light. This sensitivity is about 10 times higher than conventional technology (light use efficiency less than 5%), being the world's highest sensitivity of hyperspectral imaging. For example, the objects under room illumination (550 lux) are clearly able to be captured by the developed technology (Fig. 2a), while it is hard to recognize using the conventional one (Fig. 2b). The developed hyperspectral imaging technology enables clear shooting without extremely bright lighting (more than 10,000 lux), which has been required in conventional hyperspectral imaging.

Hyperspectral imaging is experimentally achieved using the developed special filter and reconstruction process on software, where wavelength range is from 450 nm to 650 nm and divided into 20 wavelengths. As shown in an example using color samples (Fig. 2c), the spectral information is correctly obtained via accurate color separation based on appropriate data thinning out by the special filter. More accurate image analysis and recognition are enabled using 20-wavelegth spectral information, which is much richer than that visible by the naked eye and color cameras that can recognize three colors (red, green, and blue).

Conventional hyperspectral imaging techniques have suffered from poor usability owing to a low frame rate coming from a low sensitivity. The low frame rate significantly deteriorates the usability because the images are displayed frame-by-frame, making it difficult to perform focusing and alignment.

On the contrary, the developed hyperspectral imaging technology enables a fast shutter speed because of the high sensitivity. High-speed (more than 30 fps) hyperspectral imaging is achieved by using uniquely developed algorithm enabling fast data reconstruction. Frame rates more than 30 fps are perceived as smooth movie by humans, making it easy to perform focusing and alignment.

1. Recognition of whitish tablets
   Although it is hard to distinguish by the naked eye, whitish tablets have a slightly different color depending on the components contained, being distinguishable from the hyperspectral image depending on the color difference. Figure 4a shows three kinds of whitish tablets distinguished using hyperspectral images.
2. Inspection of fresh foods
   The ingredients of fresh foods, for example, the sugar content of tomatoes, correlate with spectral information and can be inspected using hyperspectral images (Fig. 4b). The developed imaging technology, which does not require bright lighting, is preferable to the inspection of fresh foods and medicines without increasing their temperature. 
3. Inspection of coating unevenness
   Even the slightest color differences that are invisible to the naked eye can be discerned using hyperspectral images. Figure 4c shows an example in which color differences (ΔE^*7) as small as ΔE = 0.4 and 1.0 are discerned using hyperspectral images.

*1: As of January 26, 2023
*2: The images acquired for each wavelength of light are known as multispectral images when the number of wavelengths is four or more, and as hyperspectral image when the number of wavelengths is about 10 or more.
*3: A method to acquire rich signals by reconstructing observed data of smaller number of signals. This method uses the fundamental property (sparsity) that the observed data has a biased distribution in a certain dimension (for example, frequency space). It has been applied to MRI technology, and recently applied to observe black holes.
*4: A camera that has three types of filters, red, green, and blue, mounted on an image sensor to express colors as the ratio of the three colors. Digital cameras and smartphone cameras mostly fall into this category.
*5: An index showing how many images are displayed in a video per second, expressed in units of fps (frames per second). A general television broadcast is developed with 30 fps (in Japan).
*6: A periodically arranged structure containing mediums (thickness d, refractive index n) such that d = λ/4n, where λ is the wavelength of light. It can control the reflectance of light at wavelength λ.
*7: An index that quantitatively indicates the difference between two colors. In general, when ΔE ≤ 1.0, the difference cannot be discerned with the naked eye even if the two colors are compared side by side.

Panasonic Holdings Corporation, Technology Division, Public Relations Section
Email: crdpress@ml.jp.panasonic.com