image:Overview of the developed hyperspectral imaging technology

Jan 26, 2023

Company / Press Release

Panasonic Develops Hyperspectral Imaging Technology with the World's Highest*1 Sensitivity

- Based on compressed sensing technology used in medical care and space exploration -

Osaka, Japan - Panasonic Holdings Corporation today announced it has developed the world's highest sensitivity*1 in hyperspectral imaging technology*2 by applying a compressed sensing technology*3, which has been used in medical care and space exploration. This technology makes it possible to identify slight color differences-small differences hard to discern with the naked eye-maintaining the usability of a conventional color camera*4, and enables improvement of image analysis and recognition accuracy. As the world's first demonstration of such efficient hyperspectral imaging, it was published on January 23, 2023 in the online version of the British scientific journal Nature Photonics under a joint name with imec, a Belgian research institute.

Outline

With the evolution of image recognition technology, the application of machine vision is expanding, including industrial use of image data for efficiency, labor saving, and energy saving. Since machine vision recognizes images on a computer, information that humans cannot perceive, such as continuous color changes (spectral information), can be used to make analyses. Images with spectral information, known as hyperspectral images, are expected to play a role in expanding the application fields of machine vision.

In conventional hyperspectral imaging, optical elements such as prisms and filters that selectively pass light of a specific color (i.e., wavelength) have been used. However, since these methods detect light separately in each wavelength, there is a physical restriction in that light utilization efficiency (i.e., the sensitivity) decreases in inverse proportion to the number of wavelengths. Therefore, illumination with a brightness comparable to that of the outdoors on a sunny day (illuminance of 10,000 lux or more) was required to shoot, which decreases usability and versatility.

The newly developed hyperspectral imaging technology employs compressed sensing, which efficiently acquires observation data by "thinning out" and reconstructing data to restore data to what it was before thinning out through image post-processing. The compressed sensing technology is also used in MRI examinations in the medical field as well as black hole observations. A special filter that transmits multiple wavelengths of light to appropriately thin out data is implemented on an image sensor (Fig. 1), and the image reconstruction is carried out by a uniquely optimized algorithm for digital image processing. By leaving a part of the color-separating functions to the software, Panasonic Holdings have overcome the trade-off between the number of wavelengths and sensitivity, the fundamental issue of conventional technology. This approach has made it possible to capture hyperspectral images with the world's highest sensitivity (Fig. 2) and video (Fig. 3) under indoor illumination (550 lux).

Utilizing the developed technology, Panasonic Holdings will collaborate with our partners to provide new spectral sensing solutions, including highly accurate image analysis and recognition, and to expand machine vision applications with highly sensitive hyperspectral imaging technology.

Main features

1. Improve sensitivity beyond physical restrictions through data acquisition with thinning out using a special filter randomly transmitting multiple wavelengths of light.

2. Achieves clear images even in indoor illumination (550 lux) through hyperspectral imaging with the world's highest sensitivity (about 10 times higher than conventional ones).

3. Achieves hyperspectral imaging with color camera-comparable usability by improving frame rates*5 to levels as high as video-rates (over 30 fps) via a uniquely developed algorithm.

Technological Information

1. Development of a special filter that randomly transmits multiple wavelengths of light

In conventional hyperspectral imaging, optical elements such as prisms and filters that selectively pass light of a specific wavelength are used to detect light of the wavelength assigned to each pixel of the image sensor. However, these methods have the physical restriction that light of the non-assigned wavelength is not detected in view of signal detection at each pixel, and the sensitivity decreases in inverse proportion to the number of wavelengths.

Therefore, Panasonic Holdings developed a special filter employing a distributed Bragg reflector*6 (DBR) structure, utilizing the wave nature of light, and implemented it on an image sensor (Figs. 1a and 1b). The special filter is designed to transmit incident light with randomly changing the intensity for each pixel and wavelength (Fig. 1c). The thinning out of data corresponds to change the intensity for each pixel and wavelength. The data pre-thinning out can be obtained after the data reconstruction process as long as the data is detected with thinning out in appropriate way. In Panasonic's newly developed technology, color separation is partly performed in the reconstruction process on software, and thus the physical restriction of sensitivity, which has been a fundamental issue in conventional hyperspectral imaging, can be overcome.

image:Figure 1: Overview of the developed hyperspectral imaging technology.

Figure 1: Overview of the developed hyperspectral imaging technology.
(a) A photograph of an image sensor with a special filter. (b) An optical microscope image of the special filter. (c) A schematic image of light detection with thinning out using the special filter.

2. Hyperspectral imaging with the world's highest sensitivity (about 10 times higher than conventional ones)

The improvement of sensitivity can be achieved by using the special filter since multiple wavelengths of light are transmitted and detected. Specifically, a developed special filter transmits about 45% of the incident light. This sensitivity is about 10 times higher than conventional technology (light use efficiency less than 5%), being the world's highest sensitivity of hyperspectral imaging. For example, the objects under room illumination (550 lux) are clearly able to be captured by the developed technology (Fig. 2a), while it is hard to recognize using the conventional one (Fig. 2b). The developed hyperspectral imaging technology enables clear shooting without extremely bright lighting (more than 10,000 lux), which has been required in conventional hyperspectral imaging.

Hyperspectral imaging is experimentally achieved using the developed special filter and reconstruction process on software, where wavelength range is from 450 nm to 650 nm and divided into 20 wavelengths. As shown in an example using color samples (Fig. 2c), the spectral information is correctly obtained via accurate color separation based on appropriate data thinning out by the special filter. More accurate image analysis and recognition are enabled using 20-wavelegth spectral information, which is much richer than that visible by the naked eye and color cameras that can recognize three colors (red, green, and blue).

image:Figure 2: Comparison of sensitivity and hyperspectral data obtained by developed technology

Figure 2: Comparison of sensitivity and hyperspectral data obtained by developed technology
(a) An image taken by developed technology with 550 lux illumination. (b) An image taken by conventional technology with 550 lux illumination. (c) Spectral information of color samples obtained by developed technology.

3. Realization of high usability with high frame rate

Conventional hyperspectral imaging techniques have suffered from poor usability owing to a low frame rate coming from a low sensitivity. The low frame rate significantly deteriorates the usability because the images are displayed frame-by-frame, making it difficult to perform focusing and alignment.

On the contrary, the developed hyperspectral imaging technology enables a fast shutter speed because of the high sensitivity. High-speed (more than 30 fps) hyperspectral imaging is achieved by using uniquely developed algorithm enabling fast data reconstruction. Frame rates more than 30 fps are perceived as smooth movie by humans, making it easy to perform focusing and alignment.

image:Figure 3: A typical frame of a hyperspectral movie acquired through the developed special filter and algorithm, and spectral information extracted from the frame (characteristic spectrum of LED lighting is shown). 

Figure 3: A typical frame of a hyperspectral movie acquired through the developed special filter and algorithm, and spectral information extracted from the frame (characteristic spectrum of LED lighting is shown). 
The original movie can be found in the presented paper (Supplementary Video). 

Practical applications of hyperspectral imaging

  1. Recognition of whitish tablets
    Although it is hard to distinguish by the naked eye, whitish tablets have a slightly different color depending on the components contained, being distinguishable from the hyperspectral image depending on the color difference. Figure 4a shows three kinds of whitish tablets distinguished using hyperspectral images.
  2. Inspection of fresh foods
    The ingredients of fresh foods, for example, the sugar content of tomatoes, correlate with spectral information and can be inspected using hyperspectral images (Fig. 4b). The developed imaging technology, which does not require bright lighting, is preferable to the inspection of fresh foods and medicines without increasing their temperature. 
  3. Inspection of coating unevenness
    Even the slightest color differences that are invisible to the naked eye can be discerned using hyperspectral images. Figure 4c shows an example in which color differences (ΔE*7) as small as ΔE = 0.4 and 1.0 are discerned using hyperspectral images.
image:Figure 4: Examples of analysis/inspections using hyperspectral images.

Figure 4: Examples of analysis/inspections using hyperspectral images.
(a) Recognition of whitish tablets. (b) Estimation of sugar content of tomatoes. (c) Inspection of coating quality.

Notes:

*1: As of January 26, 2023
*2: The images acquired for each wavelength of light are known as multispectral images when the number of wavelengths is four or more, and as hyperspectral image when the number of wavelengths is about 10 or more.
*3: A method to acquire rich signals by reconstructing observed data of smaller number of signals. This method uses the fundamental property (sparsity) that the observed data has a biased distribution in a certain dimension (for example, frequency space). It has been applied to MRI technology, and recently applied to observe black holes.
*4: A camera that has three types of filters, red, green, and blue, mounted on an image sensor to express colors as the ratio of the three colors. Digital cameras and smartphone cameras mostly fall into this category.
*5: An index showing how many images are displayed in a video per second, expressed in units of fps (frames per second). A general television broadcast is developed with 30 fps (in Japan).
*6: A periodically arranged structure containing mediums (thickness d, refractive index n) such that d = λ/4n, where λ is the wavelength of light. It can control the reflectance of light at wavelength λ.
*7: An index that quantitatively indicates the difference between two colors. In general, when ΔE ≤ 1.0, the difference cannot be discerned with the naked eye even if the two colors are compared side by side.

Media Contact:

Panasonic Holdings Corporation, Technology Division, Public Relations Section
Email: crdpress@ml.jp.panasonic.com

About the Panasonic Group

A global leader in developing innovative technologies and solutions for wide-ranging applications in the consumer electronics, housing, automotive, industry, communications, and energy sectors worldwide, the Panasonic Group switched to an operating company system on April 1, 2022 with Panasonic Holdings Corporation serving as a holding company and eight companies positioned under its umbrella. Founded in 1918, the Group is committed to enhancing the well-being of people and society and conducts its businesses based on founding principles applied to generate new value and offer sustainable solutions for today's world. The Group reported consolidated net sales of 7,388.8 billion yen for the year ended March 31, 2022. Devoted to improving the well-being of people, the Panasonic Group is united in providing superior products and services to help you Live Your Best. To learn more about the Panasonic Group, please visit: https://holdings.panasonic/global/

The content in this website is accurate at the time of publication but may be subject to change without notice.
Please note therefore that these documents may not always contain the most up-to-date information.
Please note that German, French and Chinese versions are machine translations, so the quality and accuracy may vary.

Issued:
Panasonic Holdings Corporation

Downloads (Images)

Featured news