Previous Abstract Return to Session F6

Session F6: PNT for Uncrewed Systems

Relative Pose Estimation from Monocular Flight Imagery of Aerial Refueling
Stephanie Hanson, Liam Weinfurtner, Derek Worth, Scott Nykl, Air Force Institute of Technology
Location: Room 6-8
Alternate Number 2

The mission statement of the United States Air Force declares “airpower anytime, anywhere.” To achieve this goal, USAF aircraft must have the ability to remain airborne in areas of interest to support our warfighters. To keep aircraft in the air indefinitely, automated aerial refueling has been a topic of interest across the Department of Defense as a force. To reconcile human limits, remotely piloted tankers and other aircraft have found increased use around the world. However, latency issues make manual refueling efforts dangerous and ineffective, especially in degraded or denied operational environments. In the pursuit of real-time automated air refueling, researchers at the Air Force Institute of Technology have developed computer vision techniques that compute the relative pose between a tanker and receiver using monocular vision sensors. These algorithms leverage convolutional neural networks (CNNs) to recognize aircraft features and then localize a relative pose between the tanker and receiver.
To enable automated air refueling, a 4-stage method was developed that uses monocular camera vision to capture an image, find 2D features within the image on the aircraft, and analytically solve for the aircraft’s pose, enabling the calculation of a vector from the tip of the probe to the drogue. With this vector, another Artificial Intelligence agent can control the aircraft and plug the probe into the drogue. This novel approach requires no extrinsic camera calibrations; dual object detection is used to identify both the end of the probe and the net of the drogue, effectively allowing arbitrary camera placement on the receiving aircraft. Currently, simulation of this method in a 3D virtual environment has been extremely successful, showing the method is reliable, accurate within 3-centimeters at probe contact, and efficient, calculating at 45.5 frames per second.
Following the success of simulations in a virtual world, several flight tests were performed in December of 2023 to record footage of an in-flight Learjet trailing a drogue mounted behind a Gulfstream G3. The recordings were performed in varied visibility and lighting conditions such as clear skies during the day, direct sunlight into the camera, evening, and nighttime. We present an analysis of the estimated probe-to-drogue vectors through a hybrid of synthetic and real imagery. We apply our aforementioned relative dual object detection algorithm to the flight data and show that our can produce significant results in real-world situations, as well as in simulation.

Previous Abstract Return to Session F6