PHOTONICS APPLIED: UAV VISION: Rigors of unmanned aerial vehicles challenge vision technology

Vision for an unmanned aerial vehicle is critical to that vehicle performing its intended task, which is sometimes a covert imaging operation in a hostile setting that is unsuitable for manned aircraft.

Jan 1st, 2009
Th Rigors 01

Vision for an unmanned aerial vehicle is critical to that vehicle performing its intended task, which is sometimes a covert imaging operation in a hostile setting that is unsuitable for manned aircraft.


Unmanned aerial vehicles (UAVs) are used for both commercial and industrial surveillance and reporting, as well as covert military operations in an oftentimes hostile environment. These UAVs need to be small, lightweight, and highly maneuverable, placing rigorous demands on the image sensors, optics, and communications equipment and software algorithms that provide their intelligence.

Unmanned flight

The history of UAVs reaches all the way back to the U.S. Civil War, when the opposing North and South armies equipped balloons with explosives and attempted to launch them over enemy territory where they would hopefully fall and explode.2 Of course, these earliest of UAVs were entirely blind, lacking any sort of vision capability or intelligence. According to Pete Lewis, U-systems program manager at SRA International (Arlington, VA) and director of the UAVforum (, an online community for all things UAV related, it wasn’t until 1924 that the British RAE made the first successful radio-controlled flight without a pilot.

Today, no less than 45 countries fly hundreds of different UAV models; in the U.S. alone, Lewis estimates that there are approximately 80 companies, academic institutions, and government groups developing more than 200 different UAV designs ranging in price from $1000 to $26 million dollars (see also Japan, says Lewis, leads in commercial UAV use with approximately 2000 unmanned helicopters spraying fields for pest abatement or seeding crops. Worldwide, these UAVs range in size from the palm-size Black Widow “micro” UAV spy plane (6 in. in diameter, 1.5 oz) to Northrop Grumman’s RQ-4/Global Hawk (weighing in at 25,600 lb).

FIGURE 1. The MQ-1 Predator, which has seen combat over Bosnia/Serbia and the Middle East, can carry a dual-missile payload (top). Visual information including video is transmitted to a ground station for analysis (bottom). Here, U.S. Air Force Captain Richard Koll, left, and Airman First Class Mike Eulo, right, perform function checks at Balad Air Base, Iraq, after Predator launch. Koll and Eulo control the Predator in a radius of approximately 25 miles around the base before handing it off to controllers stationed in the U.S. to continue its flight. (Courtesy of United States Air Force)
Click here to enlarge image


Click here to enlarge image

One of the most well-known and successful UAVs is the Predator, which has been used by the U.S. Air Force (USAF) since 1995 in combat operations over Bosnia, Serbia, Yemen, Afghanistan, and Iraq (see Fig. 1). Manufactured by General Atomics Aeronautical Systems (San Diego, CA), it serves in a reconnaissance role and can also fire two Hellfire missiles. The USAF calls the Predator a “system” consisting of four air vehicles with sensors, a ground-control station, and a staff of 55 people. Sensor communications–from its multispectral targeting system, color nose camera (used primarily for flight control), variable-aperture day and infrared cameras, and its laser designator–are relayed by either a line-of-sight or satellite data link. In addition to dangerous combat operations, the Predator has also been used for civilian search and rescue and scientific research.

Dull, dirty, and dangerous

“Trite but true, UAVs can perform those missions considered dull, dirty, or dangerous for their manned counterparts,” says Lewis. “Examples include orbiting endlessly over a point for communications relay or jamming (dull), collecting air samples to measure pollution or toxin levels (dirty), and flying reconnaissance over hostile territory (dangerous).”

FIGURE 2. A flight trial at Yuma Proving Ground shows the launched-by-hand carbon-fiber Zephyr UAV, which flies by day on solar power from amorphous silicon solar arrays and by night on charged lithium-sulfur batteries. As flight times increase, so does the requirement for more sophisticated vision systems for both day (visible spectrum) and night (infrared spectrum) surveillance. (Courtesy of Qinetiq)
Click here to enlarge image

But whatever the application, UAVs must have built-in intelligence to maneuver and to perform their required task: imaging sensors must gather the desired wavelength and intensity information and send it back to the control station for analysis. And even though UAVs do not require seating or oxygen-servicing equipment for a pilot, cost-reduction pressures for civilian applications and maneuverability requirements for defense applications demand that the UAV vision payload–including sensors, optics, illumination, and communications gear, as well as flight instrumentation–be as small and lightweight as possible (see

Smaller and lighter sensors

In the case of larger UAVs, such as the massive RQ-4/Global Hawk, vision sensors can have the luxury of larger size and greater function. But for the palm-size Black Widow from AeroVironment (Monrovia, CA), for example, imaging sensors are immediately constrained in terms of size, function, and weight. A slightly larger but still “micro” UAV from AeroVironment called Wasp III (with a wingspan of 29 in. and weighing 1 lb), for example, carries a miniature infrared camera along with two integrated color cameras that transmit streaming video directly to a handheld ground controller.

Along with small size comes limited flight duration because of the simple fact that UAVs are typically powered by heavy batteries. Currently, the world record for flight endurance is 82 hours, 37 minutes, set by QinetiQ’s (Hampshire, England) solar-powered Zephyr UAV (see Fig. 2). But as flight duration increases, so too does the need for both day (visible) and night (infrared) imaging requirements.

“When we first went public in 2004 the only vision offerings available were exceedingly heavy and cumbersome for our small helicopters,” said Jei Wheeler, director of unmanned aircraft systems for Tactical Aerospace Group (TAG; Corozal, Belize), which offers four different UAV helicopter models with payload capacity from 60 up to 600 lb. “Realizing that the majority of UAVs would be in the smallest payload categories, vision suppliers quickly reduced form factors and new smaller systems began appearing on the market almost quarterly throughout 2005 and beyond. Now in late 2008 we have some remarkable systems coming to market, in particular a new 4 in. system from Controp (Israel) with full visible and infrared capability and optional laser targeting/tracking. Cost is also being reduced from the $250,000 range to near $100,000, keeping the same or better resolution and capabilities in these much smaller packages.”

FIGURE 3. A window is tracked during a visual servoing task in which the UAV’s vertical and lateral displacements are controlled by a visual control loop that fixes the window in the center of the image as the approaching movement is controlled by a GPS position controller (top). The UAV vertical and lateral positions are plotted during the vision-controlled flight (bottom). (Courtesy of Universidad Politécnica Madrid)
Click here to enlarge image

Another company at the forefront of this balancing act of sensor size/weight versus capability is Sensors Unlimited (Princeton, NJ; a Goodrich company). “A very interesting part of the whole UAV imaging story has to do with the famous U-2 spy plane,” said Ed Hart, general manager of Sensors Unlimited. “About five years ago, the secret was let out that Goodrich made the cameras for the U-2–cameras using sensors that can see in the short-wave-infrared (SWIR) part of the spectrum, enabling the U-2 to see as far as 130 miles through clouds, fog, high humidity, and even dust when visible cameras cannot.” Hart explains that the U-2 camera does this with cryogenically cooled indium antimonide (InSb) technology to span both the mid-wave and short-wave parts of the spectrum.

Because cryogenic cooling and its large lens system makes the U-2 camera too large for all but the largest UAVs, Goodrich–through its acquisition of Sensors Unlimited (Princeton, NJ) in 2005–developed indium gallium arsenide (InGaAs) sensors that allow imaging in the SWIR band without cooling, enabling exceptionally small cameras. The Sensors Unlimited SU640KTSXInGaAs SWIR camera, for example, weighs less than 90 g, operates at room temperature, and uses less than 2.5 W at 20°C. The 640 × 512-pixel camera gathers full-size images at 30 frames/s and features a 25-µm-pixel-pitch focal-plane array responsive from 0.7 to 1.7 µm. And because InGaAs technology detects reflected light rather than heat signatures, this SWIR camera captures images both day and night and during the dawn and dusk thermal crossover periods when mid- and long-wave cameras are unable to distinguish targets against backgrounds at similar temperatures.

New challenges

The war on terror has flooded defense companies with funding to develop more-sophisticated UAVs (see Using the proceeds of an Office of Naval Research (Arlington, VA) contract award, Imperx (Boca Raton, FL) developed a prototype UAV that integrates vision, global positioning, and infrared (IR) ranging technology. Designed to operate within the payload compartment of an Aerosonde (Notting Hill, Australia) UAV, the system captures images using Imperx’s own IPX-4M15C sensor–a 2048 × 2048-pixel color camera with 8- or 10-bit digital output and programmable frame rate and integration time.

According to Petko Dinev, CEO of Imperx, PC-104 frame grabbers were not powerful enough to handle the MJPEG compression required by the system for the massive amounts of data collected. As a result, images are sent to Imperx’s FrameLink-JFLEX frame grabber that is integrated with a JRex-PM CPU board from Kontron (San Diego, CA) through a proprietary PC-104-like interface; the frame grabber/CPU combination is also being offered separately by Imperx and Kontron as a product for quality assurance, production automation, materials flow, and machine-control applications. Image and position data are captured by a ground station using a radio-frequency data transceiver; ground-station operators configure the camera’s operating parameters in real time. The UAV is targeted for deployment in Iraq to search for roadside munitions.

Vision capabilities such as target tracking for UAVs are improved, not only through better sensors and data transmission architectures, but also through sophisticated image-processing algorithms (see Researcher Pascual Campoy and colleagues in the Computer Vision Group at Universidad Politécnica Madrid (Madrid, Spain) are applying visual servoing techniques–based on the displacement in the image plane of a visual patch or a characteristic extracted from an image–to control the position of a UAV (see Fig. 3).3 Global-positioning-system technology guides the UAV to the general region of a target and then selected features in the image plane are tracked using such algorithms as the Harris Corner detector, Hough transform, the Lukas-Kanade algorithm, or the scale-invariant feature transform to provide references for flight control.

Vision-system development for UAVs will only grow in complexity as the value of unmanned flight is further recognized for both military and civilian operations. “Rotor F/X (Van Nuys, CA) just partnered with a Las Vegas sensor technology company to combine a UAV with vision sensors and a data-analysis package developed by JPL for use on the Mars Rover,” says Jei Wheeler. “This new system will also be used in open-pit mining for high-resolution measurement of pit slopes and tailings, and for the oil and gas industries for exploration and pipeline leak detection,” adds Wheeler.

“UAVs are taking on more and more sophisticated tasks–watching day and night over several days,” says Ed Hart. “If you’re used to seeing a stable market, UAV vision is not that. Rapid, continuous improvement is the order of the day.”


  1. Http://
  3. P. Campoy et al., J. Intelligent and Robotic Systems, DOI 10.1007/s10846-008-9256-z (June 30, 2008).

Tell us what you think about this article. Send an e-mail to

More in Detectors & Imaging