MICHAEL W. KELLY and MEGAN H. BLACKWELL
Many emerging infrared (IR) imaging applications demand high performance in a small package. Low noise, large dynamic range, wide field of view, high resolution, and a fast readout rate are often simultaneous requirements placed upon an imaging sensor. Traditional focal plane arrays (FPAs), based on analog electronics, perform well in many circumstances, but satisfying the most aggressive requirements presents challenges and can lead to expensive, large, and complex system solutions. The digital-pixel focal plane array (DFPA), developed at MIT Lincoln Laboratory, provides a solution that enables low-cost, compact, and simple system solutions.
The 250 Mpixel aerial thermal image of Boston shown in Fig. 1 was obtained at night using a single 256 × 256 pixel LWIR DFPA sensor coupled to a fast scanning mirror. The sensor was packaged in a 26 in. PV-Lab lookdown gimbal weighing 205 lb and provided a proof-of-principal demonstration of DFPA technology. Second-generation airborne sensors, now in development, will utilize a single 640 × 480 pixel DFPA to produce the same imagery; however, the sensor will be packaged into a 15 in. turret weighing less than 100 lb.
A DFPA is formed when a detector array is mated to a silicon CMOS readout integrated circuit (ROIC). MIT Lincoln Laboratory DFPA ROICs contain an array of unit cells that convert detected photons directly to digital counts and read out digital image data at a very high rate. Unit cells are arrayed on the ROIC in a matching pattern to the detector array. Following hybridization of the detector array and ROIC, each unit cell makes an electrical connection to a corresponding detector in the detector array.
ROICs are fabricated at a commercial foundry (e.g., IBM), and detector arrays are procured from industrial partners. DFPA ROICs are nearly detector agnostic and may be used across a broad range of the electromagnetic spectrum. Although initial efforts have optimized mid- and longwave-IR (LWIR) spectral bands, applications in the shortwave-IR (SWIR), near-IR (NIR), and visible bands are under development.
DFPA design
Every unit cell in a DFPA ROIC contains a low-power analog-to-digital converter and a simple computer. This unique feature of the DFPA can significantly improve system performance over conventional approaches because data can be processed while they are being collected, rather than after they are read from the imager. Using in-pixel processing and an all-digital readout, the DFPA provides low noise, wide dynamic range, wide field of view, high resolution, and fast readout rate. DFPAs can outperform other FPAs in existing imaging applications and can expand the realm of future applications.
Current commercial IR imagers can be categorized into analog and digital FPAs. Many digital FPAs are actually based on analog pixels and multiplexers. Most of these conventional devices store signal charge at each pixel and then sample the capacitor voltage for digitization and/or readout.
This approach requires a large integrating capacitance at each pixel site for storing signal charge. MIT Lincoln Laboratory's solution instead digitizes the detector signal using a very small capacitance that integrates photocurrent to a predefined threshold charge level. Once the threshold charge level is reached, the capacitor is automatically reset and starts to accumulate charge again. A pulse generator that is triggered on every reset forms a current-to-frequency converter when coupled with reset circuitry and drives a digital counter contained within each pixel.
This counter can be configured to increment (add) or decrement (subtract) values on each reset pulse, providing a low-power computational capability at the pixel level. Signal processing can be achieved without the use of traditional digital adders or multipliers; rather, the DFPA manipulates the integration time and sign of the digital counters to achieve the desired processing functionality.
Image readout from a DFPA is analogous to that of a CCD converter. Rather than shifting charge like CCDs, DFPA readout is achieved entirely in the digital domain. Following an integration period, digital data stored in every pixel are shifted to neighboring pixels sequentially until all data are transferred to the edge of the array and read out through high-speed digital I/O structures.
DFPAs are configured to shift data in four orthogonal directions across the rectangular array. The ability to transfer data between pixels coupled with in-pixel computations facilitates the real-time application of many image-processing algorithms, such as filter functions, convolution, and background subtraction, as illustrated by the following examples.
Some DFPA applications
Combining the FPA detection, digitization, and image-processing electronics into a massively parallel architecture simplifies the overall system design while providing greatly enhanced capability.
Spatio-temporal filtering. By shifting data between adjacent pixels and adding and subtracting pixel counts, the DFPA can convolve an image with a predefined filter kernel while the image is being acquired. Low-pass, high-pass, and texture filter kernels are applied in two dimensions (see Fig. 2). No additional system resources are required to filter the image data.
Background subtraction. In IR imaging, there is typically a small difference in photon flux between an object of interest and the background. For example, the difference in mean flux between two objects radiating at equivalent blackbody temperatures of 302K and 300K, respectively, is approximately 3.3% in the LWIR band (8–12 μm) and 7.4% in the MWIR band (3–5 μm). The large background pedestal is difficult for analog FPAs to overcome because of limited well capacity. Often, frames must be read out at a very high rate simply to accommodate the high background, even though the background information is ultimately disregarded.
Alternatively, the simple computer in every pixel of a DFPA can be used to implement background subtraction. The effective charge well depth for the DFPA can be many orders of magnitude higher than conventional technologies and offers significant dynamic-range advantages.
Real-time processing enables change detection. Image differencing for change detection can be achieved by accumulating signal while incrementing counters for an integration period and then accumulating signal for the same period while decrementing counts prior to reading out the image. Only dynamic areas in the scene—caused by object motion or signal modulation—produce a signal in the resulting image.
For example, the fast temporal processing of the DFPA can detect the track of a bullet in flight by sequentially collecting many image differences prior to data readout. Figure 3 displays a shooter holding a pistol within a 50 ft range from the DFPA imager and a single raw image of the highlighted field of view showing a bullet fired at a speed of approximately 350 m/s. High-speed, in-pixel processing removes stationary clutter from the image while retaining the signal of interest (i.e., the bullet trajectory).
Fast-moving objects such as the bullet and the muzzle flash (hot gases escaping the gun barrel) produce a black-and-white striped pattern; seven consecutive increment/decrement cycles are demonstrated in the data. Stationary and slowly moving objects such as the gun barrel are not detected. The in-pixel processing allows clean imagery of the bullet trajectory to be read out at data rates between one and two orders of magnitude lower than what would be required using conventional imagers.
Electronic image stabilization. Image blurring caused by camera motion during exposure degrades the benefits of long integration times offered by the DFPA. For example, an integration time of 0.1 s, which is approximately 1000 times longer than integration times offered by conventional LWIR FPAs, produces an image that is no higher in quality than an image produced using an integration time of 0.0001 s (see Fig. 4).
By leveraging the orthogonal-transfer architecture of the DFPA, the digital image can be appropriately transferred within the array while accumulating counts to null the effects of pointing jitter. An inertial measurement unit monitors the camera pointing and automatically controls data transfer between pixels on the DFPA to compensate for image motion. Using on-chip stabilization, an integration time of 0.1 s is now sufficient to resolve the image. Image stabilization can be achieved without the use of mechanisms to control a steering mirror, the lens, or imager position. It is not necessary to read out images at a high rate and process the data off-chip.
Future DFPA development
Over the past decade there has been renewed interest in CMOS imagers replacing CCDs as the preferred sensor technology for many applications. A CCD transfers signal charge from each pixel to a common output structure for charge-to-voltage conversion; most processing functionality, including A/D conversion, is implemented on a camera's printed circuit board. In contrast, a CMOS imager typically converts measured charge to voltage within each pixel. CMOS imagers, especially the system-on-a-chip variety, include more processing on the imager chip itself, permitting smaller systems that presumably use less power.
Historically, CCDs are known to offer a greater dynamic range than CMOS imagers as well as significant noise advantages, in part because of less on-chip circuitry and a greater tolerance to circuit component variations. Furthermore, CMOS image sensors are typically specialized for one or a few applications and may not easily adapt to other uses. CCDs can support scanning applications using time-delay and integration (TDI) techniques and currently are the best choice for many scientific, industrial, and medical applications requiring large-coverage-area scanning with low noise.
Large-format CMOS imagers can be produced in volume at low cost and currently are the best choice for many consumer-market applications. With the emergence of high-performance backside-illuminated (BSI) CMOS imagers and through-silicon via (TSV) 3-D packaging, CMOS imagers may yet dominate all markets. These techniques enable stacked CMOS imagers with large pixel-detection fill factor and low noise. Recently, Sony introduced a line of stacked CMOS imagers that promises improved image quality, imaging speed, and sensor compactness.
The DFPA architecture combines the benefits of both CCD and CMOS imagers. Functionally, DFPAs read out and operate in a way that is analogous to CCDs, except in the digital domain, where data can be read out at a very high rate. Because the DFPA ROIC is fabricated using a low-cost CMOS manufacturing process, computational and operational functionality can easily be integrated, even at the pixel level.
The advantages of the DFPA architecture can now be brought to bear in many medical, industrial, or scientific applications using existing VGA-format ROICs integrated to the appropriate detector array—-i.e., with sensitivity in the required spectral band. Near-term development of visible-, NIR-, and SWIR-band DFPAs are required for these applications.
Future widespread adoption of the DFPA approach requires device scaling to achieve larger array formats and smaller pixels. To date, 20 μm, 14 bit pixels have been fabricated using the 90 nm CMOS process node. Future small-pixel DFPAs, with 5 μm pitch or less, could leverage a 22 nm or smaller CMOS process node and 3-D stacked fabrication techniques. If successful, DFPA technology could one day become the standard approach for every application with the need to extract information from a scene quickly, efficiently, and with high performance.
ACKNOWLEDGMENTS
This work was sponsored by the Department of the Army under Air Force contract FA8721-05-C-0002. Opinions, interpretations, conclusions, and recommendations are those of the authors and not necessarily those of the United States Government.
Michael W. Kelly is assistant leader and Megan H. Blackwell is on the technical staff of the Advanced Electro-Optical Systems Group at MIT Lincoln Laboratory, 244 Wood Street, Lexington, MA 02420-9108; e-mail: [email protected], [email protected]; www.ll.mit.edu.