Test and Evaluation of Autonomous Surface Vehicles: A Case Study

Brian C. Reitz and Joshua L. Wilkerson

Peer Reviewed

Abstract:	The rigorous test and evaluation of autonomous systems is often a challenging process. Specifically, the transformation of qualitative assessments often used in the development phase to unbiased quantitative assessment can be difficult. Additionally, few standardized metrics exist for the test and evaluation of autonomous systems, as applications/approaches vary so widely that specialized metrics must be developed in the majority of cases. This paper presents a case study for this process, describing a multi-year, iterative test and evaluation effort undertaken by the authors in support of the development of an autonomous surface vehicle system. The case study details the development of quantitative performance metrics (including the rationale behind each one), revisions/additions that were made to the metrics between testing and evaluation events, impacts the metrics had on the autonomy development process observed through successive tests, and lessons learned throughout the process.
Published in:	2020 IEEE/ION Position, Location and Navigation Symposium (PLANS) April 20 - 23, 2020 Hilton Portland Downtown Portland, Oregon
Pages:	839 - 850
Cite this article:	Reitz, Brian C., Wilkerson, Joshua L., "Test and Evaluation of Autonomous Surface Vehicles: A Case Study," 2020 IEEE/ION Position, Location and Navigation Symposium (PLANS), Portland, Oregon, April 2020, pp. 839-850. https://doi.org/10.1109/PLANS46316.2020.9110129
Full Paper:	ION Members/Non-Members: 1 Download Credit Sign In