Neural City Maps: A Case for 3D Urban Environment Representations Based on Radiance Fields

Mira Partha, Shubh Gupta, Grace Gao

Peer Reviewed

Abstract: Neural Radiance Fields (NeRFs) have recently emerged as a breakthrough technique for 3D reconstruction. While NeRFs have shown impressive results in synthesizing novel views of complex 3D scenes, their suitability as a map representation for urban environments has not yet been systematically explored. In this paper, we develop Neural City Maps, compact NeRF-based representations of urban environments that can be used for applications such as visual localization for autonomous vehicles. We propose a system for the construction, storage, and updating of Neural City Maps using both publicly sourced and user-captured image data. As one possible application of Neural City Maps, we evaluate their potential for visual localization in urban settings by assessing the ability of NeRFs to disambiguate between different locations, using real-world camera images as references. We do so by training NeRFs at three locations selected from the UrbanNav (Hsu et al., 2021) dataset, and comparing matching between real images and NeRF renders captured at diverse poses. While the NeRF-rendered images at near poses successfully match the real-world views more closely than NeRF renders taken at farther poses, classic metrics for image consistency prove to be poorly suited for this setting. We suggest that image segmentation serve as the basis for a better performance metric that can meaningfully evaluate image consistency in the context of 3D reconstructions of urban environments for autonomous driving applications.
Published in: Proceedings of the 36th International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS+ 2023)
September 11 - 15, 2023
Hyatt Regency Denver
Denver, Colorado
Pages: 1953 - 1973
Cite this article: Partha, Mira, Gupta, Shubh, Gao, Grace, "Neural City Maps: A Case for 3D Urban Environment Representations Based on Radiance Fields," Proceedings of the 36th International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS+ 2023), Denver, Colorado, September 2023, pp. 1953-1973. https://doi.org/10.33012/2023.19324
Full Paper: ION Members/Non-Members: 1 Download Credit
Sign In