Infrared imaging learns from camera industry

Amosaicked sensor is a monolithic array of many sensors, arranged in a geometric pattern. In the commercial digital-color-camera market, the mosaicked color-filter-array (CFA) technique is widely used. Instead of using three charged-coupled devices (CCDs) to capture the red, green, and blue spectral bands for each pixel, each pixel is associated with just one device covered by an optical filter sensitive to a specified wavelength. In this way, only one spectral component is sensed at each pixel and the other spectral components must be estimated from neighboring pixels. The CFA overlayer is fabricated in a sequence of conventional steps that places a filter of a wavelength-selective material over each cell in a pattern according to a specific sensor design.

The Bayer array is one of two popular CFAs currently used in the industry. Patented in 1976, it covers each pixel with a red, green, or blue filter, and the number of green filters is two times the number of red or blue filters.¹ Many CFAs designed later have their origin in the Bayer array. Another popular CFA is Sony's RGBE filter array announced in July 2003, which adds a fourth band, emerald, that replaces half of the green channels of the Bayer array to reduce color reproduction errors and record natural images closer to the natural sight perception of the human eye (see Fig. 1).²

In recent years, considerable work has been conducted in multispectral imaging, in which more than three bands can be acquired—visible and infrared, for example—as many objects of interest possess unique signatures in the IR band. With available technologies, multispectral image acquisition now requires either an imaging spectrometer, which is very expensive; an assembly utilizing one or more prisms or mirrors and multiple sensor arrays, which are relatively delicate; or quantum-well imaging arrays, a technology with years of basic research required before maturity.

Encouraged by the advantages offered by CFAs over full-color cameras, we have investigated the potential of using multispectral filter arrays to generate multispectral images. The advantages of multispectral mosaicked focal-plane arrays (MMFPA's) can be summarized from cost, registration, and robustness perspectives. From the cost perspective, MMFPA's are potentially much cheaper to manufacture than any of the other technologies mentioned above because they require only one substrate and the technology for applying the filter material is mature. With regard to registration, the images are monolithic and therefore their registration is exact. As for robustness, cameras with MMFPA-based sensors exhibit unprecedented mechanical ruggedness. Because they can withstand enormous forces, monolithic imaging sensors can be used as artillery payloads in cargo rounds and in missile seekers.

Mosaic technology achieves multispectral capability at the cost of reduced resolution. However, in the color-camera industry, a considerable part of this resolution has been effectively recovered with sophisticated interpolation schemes and with advanced mosaic patterns. In the multispectral domain, more information must be estimated at each pixel and the increased number of spectral bands makes thefilter-array design more complicated.

To realize MMFPA, three issues must be resolved: the selection of spectral bands, the choice of tessellation mechanism, and the design of array patterns in which different spectral bands are arranged for optimal performance.

Selection of spectral bands

The selection of spectral bands is application oriented. Different types of targets would have different identification bands that can be selected either through experimental study or by solving an optimization problem. When solving the optimization problem, there are two criteria that can be used to find the optimal subset of spectral bands—for best reconstruction or for best classification performance.³

Selection of tessellation mechanisms

The selection of tessellation mechanisms concerns two aspects of study— the shape of the sensor and the arrangement of these sensors in the array. Most existing CFA techniques use rectangular sensors arranged in rectangular arrays because 2-D signals are normally digitized and stored as rectangularly sampled arrays. Over the years, the number of sensors in conventional half-inch CCDs has increased dramatically. To further increase the resolution over the limit (around three million), the size of individual sensors must be reduced, which could affect the sensitivity, dynamic range, and signal-to-noise ratio.⁴ To overcome this barrier, research is exploring sensors of new shapes and sampling schemes other than rectangular.

Hexagonal sampling, for example, has received a lot of attention since the 1970s (see Fig. 2, left). Researchers have noted that the hexagonal grid is more efficient than the rectangular grid and requires 13.4% fewer sampling points to obtain the same high-frequency information.⁵ This property of hexagonal sampling provides substantial savings in storage and computations. In addition, the hexagonal grid can model the human visual system more precisely because the cones on the retina are hexagonal in shape. Hexagonal sampling also eliminates the connectivity ambiguity in the rectangular grid. Each pixel in the rectangular grid has eight directly connected neighbors; however, its distance from these eight neighbors differs (see Fig. 2, right). On the other hand, in the hexagonal grid, each pixel has six directly connected neighbors and the distance between the central pixel and its neighbors is exactly the same (see Fig. 2, center).

Fujifilm announced its new product "super CCD" at the Photo Marketing Association Show 2000 (Las Vegas, NV) and the fourth-generation super CCD made its debut in 2003. Super CCD uses unique octagonal-shaped sensors to form the CFA (see Fig. 3). This design can achieve considerable improvements in sensitivity, dynamic range, and signal-to-noise ratio. It is also claimed that it improves spatial efficiency.

Design of filter-array pattern

Although there have been many studies in the development of interpolation algorithms, to our knowledge few studies actually analyze the design requirements for filter arrays and all of the discussions are based on CFAs. Here, we summarize our findings and extend the design requirements from CFA to multispectral filter arrays (MSFAs).⁶

First, consider the frequency/probability of appearance requirement. One criterion used in the design of a visual-band CFA (such as a Bayer array) is that the pattern has to match the sensitivity response of the human visual system. Because the human visual system is more sensitive to changes in the green spectral band, most CFAs have more pixels sensitive to green than to red and blue. The design of MSFAs, on the other hand, is not oriented toward the human visual system. Rather, the filters are arranged such that the array would have the best recognition performance on a certain kind of target. Therefore, the frequency of appearance must be application oriented as well. We assume this parameter is given a priori.

Second, consider the spectral consistency requirement. For an image sensor with a filter array, it is possible for an incoming photon to go through one kind of filter element and enter the adjacent pixel well, which will contaminate the adjacent pixel's charge packet and generate artifacts in the output image. Because this kind of artifact cannot be eliminated using image processing, some manufacturers introduce barriers between pixels to counter the contamination. In the design of MSFAs, to present the same reconstruction performance across the whole image plane, pixels should always have the same number of neighbors of a certain spectral band and the relative position of each band should be the same.

Third, consider the uniform distribution requirement. In a mosaicked pattern, the unmeasured spectral components of a given pixel are estimated from its neighbors. This requires that the filter array for each spectral band can sample the entire image as evenly as possible. If the pixels distribute densely in some regions while sparsely in other regions, serious degradation might occur in the final reconstructed images.

Experimental results

Following the above design philosophy, we have developed a generic algorithm to automatically generate the MSFA given the spectral bands and the probability of appearance of each spectral band. Three design options have been derived from the algorithm (see Fig. 4). We observe that for each center pixel, the first and the third design patterns (left and right) always have the same number of a certain spectral band in a six-neighbor adjacency, although the arrangement might vary. The second design pattern (center) must use a 12-neighbor adjacency to have the same property.

REFERENCES

1. E. B. Bayer, Color Imaging Array, U.S. Patent 3,971,065 (1976).
2. Sony, Sony Press Release, July 2003, www.sony.net/SonyInfo/News/Press/200307/03-029E/.
3. R. Ramanath, W. E. Snyder, H. Qi, SPIE Defense and Security Symp., Orlando (Kissimmee), FL (April 12–16, 2004).
4. Fujifilm, Press Release - Super CCD, PMA 2000, Las Vegas, NV (February 2000), home.fujifilm.com/pma2000/sprccd.html.
5. R. Mersereau, Proc. IEEE 67(6) (1979).
6. L. Miao, H. Qi, W. E. Snyder, IEEE Intl. Conf. Image Processing, Singapore (October 2004).

HAIRONG QI is an assistant professor in the ECE Department, University of Tennessee, Knoxville, TN 37996; e-mail: [email protected]. WESLEY E. SNYDER is a professor in the ECE Department, North Carolina State University, Raleigh, NC 27695: e-mail: [email protected].