Single-photon avalanche diode (SPAD) arrays are emerging as an exciting class of sensor for imaging applications. These sensor arrays can use the same optics as conventional complementary metal-oxide semiconductor (CMOS) image sensors, but are able to count individual arriving photons at each pixel.

Large-resolution SPAD arrays have the potential to produce an enormous amount of data, and thus require good bandwidth reduction techniques for practical applications. But unlike conventional CMOS sensors, SPADs digitize incoming photon counts immediately, which makes near-pixel compute feasible for compression and other advanced signal processing techniques.

It presents an exciting new paradigm that can overcome existing tradeoffs present in conventional CMOS sensors, particularly in low light or high dynamic range scenes where shot noise and motion blur present significant challenges.

SPAD pixels and arrays

SPADs are reverse-biased diodes that operate above their breakdown voltage—a single incoming photo-electron can trigger an avalanche that is easily detected by surrounding circuitry. The electronics surrounding the SPAD charge the pixel, detect and quench the avalanche, and recharge the pixel. The detected avalanche can be timestamped with a time-to-digital converter (TDC), using systems known as time correlated single-photon counters (TCSPC).

Time-correlated counting is primarily used in SPAD-based LiDARs, in which a time-synchronized laser pulse is used for time-of-flight (ToF) depth measurements. Many solid-state LiDARs use vertical-cavity surface-emitting laser (VCSEL) arrays in combination with SPAD arrays to generate ToF depth measurements using optics to project the 2D VCSEL source onto the scene and image the resulting reflected pattern onto the 2D SPAD sensor. This configuration effectively turns the 2D SPAD array into a ToF camera.

While SPAD arrays have been widely used for LiDAR, the SPAD avalanche can also be detected and used to increment a counter on digital logic near the SPAD pixel. Many 2D SPAD arrays are now produced using three-dimensional (3D) stacked wafer technology so the photosensitive area of each SPAD pixel is maximized, and much of the detection and recharge circuitry can be located beneath the pixel itself. 3D stacking increases the area available for digital logic beneath each SPAD pixel, which enables larger digital counters and even simple arithmetic operations. By placing imaging optics in front of SPAD arrays, it’s clear SPADs can be used as cameras by reading out photon counts over time.

While we may consider CMOS sensors as photon counting devices because they digitize photo-electron counts at the time of readout using an analog-to-digital converter (ADC), they don’t have the same level of accuracy as SPADs. Critically, the ADCs found in CMOS sensors add noise, known as “read noise,” and introduce uncertainty in photon count measurements. In effect, SPADs can be considered photon counting sensors that exhibit zero read noise. Both CMOS and SPAD sensors are produced using silicon, so their spectral response and thermal noise properties are similar (in CMOS sensors thermal noise is known as “dark current,” while in SPADs it’s “dark counts”).

SPADs for imaging

The zero read-noise property makes SPAD arrays particularly interesting for imaging applications. In the simplest case, a SPAD array can be configured with a N-bit counter associated with each SPAD pixel. At a given exposure time, these counters are reset, and then detected photons increment this N-bit counter. Unlike CMOS sensors, this counter can be read-out repeatedly during the exposure without incurring any noise penalty. And more complex operations such as differential counting (alternating between incrementing and decrementing the counter) can produce photon count information that would be infeasible due to read noise limitations in conventional CMOS sensors.

Another distinguishing characteristic of SPAD pixels is their ability to only count a single photon at a time. Once an avalanche is triggered, it takes some time to recharge (~ order of nanoseconds) before it becomes sensitive to another incident photon. In effect, this may appear to be a problem for SPADs, because they are unable to count every photon. But this recharge time introduces a nonlinear response to incoming photon flux (photon arrival per unit time), producing an effect known as “soft saturation” that effectively increases their dynamic range.

A CMOS sensor signal-to-noise ratio (SNR) increases proportional to flux, while SPAD SNR increases proportional to the number of photons counted. A SPAD pixel’s nonlinear response can be adjusted dynamically to enable SNR to be dynamically allocated across an extremely large dynamic range. This is possible because photon counts are digitized at the time of detection and produce zero read noise, which enables the full potential of digital logic and signal processing. While SPAD cameras can be used to produce high dynamic range images, they can also be read out at very high rates and produce an enormous amount of information about the visual scene.

Scaling SPAD sensor resolution

SPAD array resolution and pixel pitch has advanced rapidly during the past few years, with a growing number of SPAD arrays either currently available or announced by a variety of upcoming and established image sensor manufacturers. SPAD arrays have been announced with 2MP resolution, 5-µm pixel pitch, and numerous academic papers showcasing high-resolution sensors with 3-µm and even 1-µm pixel pitch sensors.

While these resolution and pixel sizes make SPAD arrays potentially feasible as a drop-in replacement for conventional CMOS in many applications, the bandwidth required to read out sensor data remains a significant challenge for further resolution scaling. Consider a 1MP SPAD array that counts photons at 100 kHz, this produces 100 Gbps of data. As SPAD array resolutions scale up to 8MP (4K) and above, bandwidth requirements quickly approach 1 Tbps.

By using the additional area made available by 3D stacked sensors and digital logic to process incoming photon counts, these bandwidth requirements can be drastically reduced by counting and transmitting photons that contain only the most important information. At Ubicept, we’ve developed adaptive closed-loop algorithms that can effectively compress photon count information in the most optimal way, given bandwidth and power constraints. As such, SPADs enable new operating domains and engineering tradeoffs that were previously inaccessible when considering conventional CMOS image sensors.

SPADs for computational imaging and computer vision

As an increasing number of imaging systems are used for machines, we can design image processing algorithms that preserve the most salient information for a given task. For example, in most computer vision applications low light is a significant challenge because increasing exposure time produces more motion blur. Perception loops used in mobile robot applications, for instance, may operate at 10 Hz, but are restricted to exposure times of 10 ms or less. This means that in low light scenarios, each frame used by the system immediately discards 90% of the light available.

With SPAD cameras, we can integrate light for the full time available without introducing motion blur by dynamically compensating for motion. Optical flow can be tracked in real time, and streaming photon count updates can be used to generate imagery that reduces or entirely eliminates the effect of motion blur. From the perspective of the computer vision algorithm, it still receives an image frame, but the input image is much higher quality and facilitates the use of the same models or downstream processing systems. But information like optical flow can also be included in this frame data to unlock further efficiencies for downstream applications.

SPAD arrays have advanced significantly during the past few years. While often associated with LiDAR, SPAD arrays can also be used as cameras for general-purpose imaging and perception using the ambient visible light around us. SPADs digitize light in a way that can make use of digital logic near the pixel, so dynamic adaptive signal processing can be applied in ways that were infeasible using conventional CMOS. This SPAD based perception paradigm is poised to provide a “digital retina” for a wide range of computer vision and perception applications during the next few years.