Previous Abstract Return to Session B9 Next Abstract

Session B9: Complementary PNT: Vision Aided/Optical Ground

Ground Vehicle Vision Navigation Map Matching Test Results Using Tactical Army Hardware and Satellite-Derived 3D Geospatial Data
Matthew Castleberry, Steven Gambino, Troy Mitchell, Chris Rose, Velislav Stamenov, Kyung Su, and Kevin Betts, Leidos, Inc.
Location: Ballroom B
Date/Time: Thursday, Jun. 15, 10:55 a.m.

Leidos demonstrated real-time high accuracy GPS-denied navigation at the Army 2022 PNT Assessment Exercise (PNTAX) using tactical computers and imaging systems that are currently installed on military vehicles. The Leidos Street Level Image Matching (SLIM) algorithm system used deep convolutional neural networks (DNNs) to match images from the vehicle’s Driver’s Vision Enhancer (DVE) long wavelength infrared (LWIR) camera image to globally available 3D models derived from satellite imagery to provide accurate geodetic position updates. SLIM measurements provide an RF-independent absolute position source that is available regardless of the current electronic warfare threat environment. The SLIM vision measurements were fused with an onboard MEMS IMU and wheel speed sensor in the Leidos Assured Data Engine for Position and Timing (ADEPT) navigation filter to provide a reliable GPS-denied navigation solution while testing against realistic live GPS threats at White Sands Missile Range (WSMR) in New Mexico. Performance was significantly improved compared to last year’s testing through the use of optimized 3D rendering software and an updated DNN matching network.

Background and Objectives:
The primary testing objective was to demonstrate accurate SLIM algorithm positioning in a feature sparse environment using a narrow FOV tactical DVE camera and a tactical processing unit. Previous results presented at JNC in 2018, 2019, and 2020 demonstrated the utility of SLIM in urban environments, but the PNTAX desert test environment is more challenging with fewer few man-made structures along the test route. Our PNTAX 2021 test results presented at JNC 2021 demonstrated that SLIM could still generate useful measurements for navigation aiding based on road curvature and natural features in the environment, but the system used a wide field of view DVE camera system that is not yet in widespread use on operational vehicles. Our PNTAX 2022 testing used the standard DVE camera (which has a 3x reduction in horizontal field of view compared to the 2021 camera system) that is currently used on fielded vehicles. The system performance goal was to validate that the updated SLIM algorithm could provide improved accuracy compared to previous test results even while using a camera with a narrower field of view.

The accuracy of the integrated navigation solution will inherently vary depending on the frequency of visual map matches, but even intermittent SLIM position fixes lead to significant error reduction compared to a drifting GPS-denied inertial solution. The testing would also prove that processing of the DVE camera and execution of the vision algorithms were possible on currently fielded tactical Army hardware.
A key enabler for military use of this capability is the availability of 3D models that are derived from satellite imagery, making them available for use anywhere on the globe without requiring a pre-survey from a local ground or aerial asset in the forward operating area. This aiding technique was successfully demonstrated for the first time on the Army Visual-Based Localization (VBL) program for mounted vehicles in 2017.

Methodology and Key Innovations:
Unlike other vision navigation such as Visual Odometry and Simultaneous Localization and Mapping (SLAM) which provide relative navigation updates that can only bound inertial drift, the SLIM algorithm provides absolute geodetic position updates through its map matching process. SLIM compares the current DVE thermal camera image to multiple synthetic images rendered from the satellite derived 3D databases to create an absolute position measurement which is used to correct the drifting inertial solution in the ADEPT filter. A grid of synthetic ground-level images is rendered using an emulated DVE camera placed in the 3D geospatial database at candidate locations based on the navigation filter’s current position estimate and uncertainty. The DNN vision algorithm predicts the probability of a possible match between the camera image and each synthetic image in the grid based on the similarity of the scene geometries and semantic content. The SLIM position measurement location and uncertainty is then calculated and sent to the navigation filter based on the distribution of similarity scores across the synthetic image grid.

The image comparison algorithm which is at the heart of SLIM must be robust to distractors (such as vehicles and pedestrians present in the live image who don’t exist in the virtual world) as well as occlusions from foliage and other nearby objects that sometimes block the camera view. An additional challenge is that the satellite-derived 3D models do not include crisp edges or high-resolution textures that are present in 3D datasets collected with terrestrial assets such as the Google Street View car that use LiDAR and multiple cameras. Finally, the problem is fundamentally a cross-band matching problem since the live input is a thermal image while the rendered images come from color visible band 3D models. These challenges motivated the use of DNNs for the image matching task. Classical computer vision algorithms struggle to match traditional features (such as corner keypoints or extracted line edges) from the live to the synthetic image in light of the challenges described above.

Two key innovations added for 2022 PNTAX testing were the use of the optimized renderer as well as the updated matching network structure. The 3D rendering software was tailored to make better use of the tactical processor's graphical processing unit (GPU). These updates allowed more synthetic images to be rendered with lower latency, thus increasing the number of images that could be used in the candidate image grid. A larger grid of candidate locations enabled SLIM to determine the location of the vehicle more accurately since the spacing between images was finer. The matching network itself was retrained to perform image comparisons at higher resolutions while still executing in real-time, thus enabling feature comparisons at a finer level of detail. Additionally, the SLIM algorithm was updated to explicitly extract semantic categories such as road, building, ground, and tree from thermal images to provide additional information content to the matching process.

Results, Conclusions, and Significance:
Results from PNTAX 2022 and local Huntsville testing in other environments will be shared at the CUI level quantifying system performance with and without the SLIM vision navigation algorithm. The test results demonstrate that the ability to perform vision map matching using existing Army tactical hardware installed on deployed vehicles provides a significant enhancement to current military capabilities by allowing ground vehicles to continue their mission with accurate navigation when GPS is threatened or completely denied. The SLIM software can run as a software application or plug-in on the existing vehicle hardware, thus providing additional capability with no additional size and weight penalty and only a small additional power draw to execute the algorithms. SLIM outputs are generated in the All-Source Positioning and Navigation (ASPN) message format to provide Modular Open System Architecture (MOSA) benefits when integrated as a measurement source for third party navigation filters.

Previous Abstract Return to Session B9 Next Abstract