ANDY LAMBRECHTS
Hyperspectral (HS) cameras divide light reflected by an object into many narrow spectral bands, which are all registered and processed separately. As a result, they capture a spectral signature for each pixel in an image. For many applications, HS imagers can be built at a price much closer to a standard camera than to an expensive research instrument, thanks to CMOS-based chip technology (see Fig. 1). Moreover, new camera prototypes are in the works using shortwave-infrared (SWIR) sensors.
The cameras designed around these chips are light and compact, making them suitable for a wide range of applications, such as precision agriculture and medical instrumentation. The main requirements driving HS imaging commercial markets are form factor (mobile, lightweight), broad-spectrum capabilities (visible, NIR, and SWIR ranges), and snapshot acquisition at video-rate or faster speeds. The Integrated Imaging group at imec, together with colleagues from imec USA, have demonstrated the versatility of HS imaging in new applications.
The Da Vinci of the low countries
Leonardo Da Vinci’s famous fresco “The Last Supper” can be admired in Milan, but American art historian Jean-Pierre Isbouts believes that Da Vinci may have had a hand in painting a copy of the relic that lies at the abbey of Tongerlo in Belgium. Over the last 500 years, art restorers have preserved the painting’s detail by ensuring that any added paint remains invisible to the naked eye, making evidence to support this theory difficult to prove.
So, Isbouts contacted imec in search of technology that could analyze the chemical composition of the paint strokes to see past paint additions, enabling inspection of the oldest layers of paint. Because hyperspectral images are combined with detailed images from a traditional camera, they reveal exceptional painting technique and unique characteristics such as how the artist was holding the brush while painting. Additionally, using IR or deeper spectral bands allows a more in-depth look at underlying layers or cracks that may have been repaired. The painting was imaged in 150 spectral bands, resulting in a large dataset on which the researchers could perform their analysis (see Fig. 2).
Overcoming challenges of the traditional HS camera
Although hyperspectral cameras have offered much promise for such applications, they have traditionally been too bulky to bring out of the lab. However, for this case and other applications, moving artifacts to a lab is expensive and could cause damage during transport, creating more harm than good.
Updates to imec’s SNAPSCAN visible and near-infrared (VNIR) range hyperspectral imaging camera made it lightweight (780 g without optics), compact (13 × 9 × 7 cm), and efficient enough to be taken to the abbey, leaving the painting safely in place. The camera has an imaging rate of 100 to 200 or more frames per second (fps), flat signal-to-noise ratio, and high spatial (up to 3650 × 2048 pixels with 7 Mpixel RAW per band) and spectral resolution (100+ bands for the NIR version or 150+ bands for the VNIR version).
As a result, the right images of the Da Vinci painting were available for analysis in just a few hours, since the camera operates from about 200 ms to 20 s speeds, depending on acquisition parameters, lighting, and object. Isbouts was especially interested in imaging the heads of the figures depicted. His theory was that if Da Vinci had worked on the painting, he would probably have painted the details of the most important figures. The camera can record an image in just a few minutes, but the challenge was the right calibration, setup, and focus. Compared to existing scientific HS cameras, which are much larger, the procedure went very smoothly. This experiment marked the first time that researchers used the new SWIR sensors in the field.
Catching pythons
Another experiment based on HS imaging took place in Florida, where exotic pythons are threatening the native fauna of the Everglades. The animals have no natural enemies; current population estimates are between 30,000 to 300,000 pythons, resulting in a severe decline in Everglade mammals, with 90% reductions in raccoons, opossums, bobcats, and foxes, and complete eradication of the marsh rabbit.
Hunters currently go on foot or by a vehicle along roads and levees to track down the snakes, but pythons are very well camouflaged and almost invisible in vegetation to the human eye. Moreover, they are cold-blooded animals, so thermal cameras are not a solution. Ronald Driggers, professor at the University of Central Florida (UCF) and imec scientist, brought this problem to imec’s attention and proposed a collaboration between imec, UCF, Extended Reality Systems (ERS), and the University of Florida. ERS is a small, local company that was actively looking for a solution to detect pythons in the Everglades with optical sensors.
Orges Furxhi, head of the Camera Systems & Computational Imaging group at imec USA, and his team measured the spectral fingerprint of the pythons—first in captivity, and then in the wild with vegetation in the background. They then used the measurements to design a single-band camera and algorithms for detecting pythons; since then, the team has shown that the camera and algorithms are effective in detecting pythons (see Fig. 3). The imec team in Belgium also provided multispectral cameras with real-time images to record additional images of the snakes and the backgrounds. The team used this data to build up a deep-learning network for automatic python detection.
The spectral measurements showed that the pythons blend in very well with the Florida Everglades’ natural backgrounds in the visible band (400–700 nm). In the NIR band (700–1000 nm), however, most backgrounds become highly reflective while the python skin reflectance stays low. This difference in reflectance provides sufficient contrast for detection.
Precision agriculture with Flemish technology
Precision agriculture is a topic of global interest as experts seek sustainable methods of feeding the world’s growing populations. One project addressing this need features several Flemish research centers that have teamed up to create a common testing ground to work on ‘smart crop protection’ in potato and fruit cultivation (see Fig. 4). By installing HS cameras on drones and tractors, plant diseases such as fire blight (a bacterial plant infection, especially in pears) can be detected more quickly and in a more targeted and precise manner.
In addition to the HS cameras themselves, attention is paid to calibration to deliver high-quality data to researchers. The software is, to a certain extent, application-agnostic. For example, the software used in fire-blight recognition is not specifically designed for this purpose; instead, the software’s primary goal is simply to ensure the quality of the data produced by the camera. It reads out individual images and combines them into one HS image. Simultaneously, the sensor’s sensitivities are removed, and the difference in lighting is eliminated.
The right HS camera for the job
For all applications, the first choice was a generic camera. Especially when the type of camera needed is not known in advance, it is advisable to use the camera with the highest number of spectral bands. Starting with the richest dataset ensures that researchers do not have to go back on location and instead can use software to simulate how the image with less spectral bands would have appeared. Afterward, researchers can develop an application-specific sensor, including only the bands of interest.
For still images, researchers can use a snapscan camera (see Fig. 5). The sensors in these cameras scan and generate very high-resolution images (up to 7 Mpixels with 150+ spectral bands). Furthermore, that means for the HS camera not only high resolution based on form information (the number of pixels in an image as with an ordinary photo), but also in the number of spectral bands: for cameras that record VNIR light, there are 150 bands and 100 in SWIR. The snapscan camera is not the preferred approach when the object moves too frequently. In this case, researchers use other sensors that alternate between 16 different spectral bands on individual pixels of the image sensor, allowing a direct video recording at a lower resolution.
Andy Lambrechts is program director for hyperspectral and lens-free imaging at imec, Leuven, Belgium; imec-int.com.