Session B2b: Autonomous Navigation and Safety-Critical Applications

Investigation of New Fuzzing Techniques to Address Navigation System Testing
Nina Haag, Collins Aerospace & Ecole Nationale de l'Aviation Civile (ENAC); Christophe Ouzeau, Collins Aerospace; Antoine Blais, Daniel Prun, Ecole Nationale de l'Aviation Civile (ENAC); Lotfi Fejri, Collins Aerospace
Location: Beacon B

Worldwide reports from air navigation service providers have confirmed that commercial airborne equipment has been exposed to various threats, including interferences, jamming, and spoofing. Ensuring communication, navigation, and surveillance equipment robustness against evolving threats is critical. These feared events can significantly undermine navigation integrity and safety.
From the standpoint of certification and standardisation, those responsible for developing industry standards believe it is important for manufacturers to assess the resilience of their equipment. Furthermore, they should provide guidance to air operators on detecting and addressing feared events that may lead to partial or complete loss of performance in safety-critical operations. Moreover, they should offer instructions on operating and maintaining products during the occurrence of these events. In addition, Minimum Operational Performance Standards (MOPS) serve as a basis for manufacturers to ensure their equipment complies with established standards. Furthermore, they provide guidance for testing procedures in addition to specifying the minimum requirements. Basically, analysis, test, inspection and demonstration are standardized verification methods. Manufacturers can propose more stringent requirements and test procedures on top of the MOPS ones. Relevancy of testing procedures is a key point to certify products for two major reasons: the increasing complexity of equipment taking credit of new systems on various frequency bands and the increasing number/variety of threats that may cause loss of performance and services for safety-critical operations.
For instance, spoofing involves broadcasting counterfeit satellite signals to deceive GNSS receivers, causing incorrect position, navigation and timing data. A successful spoofing attack can result in navigation errors, unauthorized deviations from flight paths, or even accidents, all of which can have devastating implications. Conversely, jamming disrupts legitimate GNSS signal processing, rendering receivers incapable of obtaining accurate positional information. Such disruptions can sever communication with air traffic control, leading to misinterpretations of positional data and potentially causing flight delays or emergencies. Both spoofing and jamming represent substantial risks to GNSS receivers and the aviation systems that depend on reliable and timely information to ensure flight safety. Given the life-critical nature of GNSS in aviation, addressing these threats is essential for maintaining safety and operational integrity.
Given the nature of these sophisticated threats, it becomes evident that neglecting their complexities results in certified products that may perform adequately under ideal conditions but remain vulnerable when faced with real-world challenges. For instance, the ED-259/DO-401 MOPS aims to address some of these shortcomings; however, their slow implementation and limited scope highlight a significant gap in testing exhaustiveness. Traditional deterministic testing methods fail to simulate these threats' unpredictable and dynamic nature, exposing receivers to potential exploitation in critical situations. Furthermore, existing standards tend to be reactive rather than proactive, focusing on known threats rather than anticipating future vulnerabilities.
Fuzz testing is a method used in software testing that involves inputting random or unexpected data into a system to identify vulnerabilities. Unlike deterministic methods, which test performance under controlled and predictable conditions, fuzz testing introduces variability to uncover hidden issues. This variability simulates real-world scenarios, uncovering weaknesses that might otherwise remain unnoticed. For instance, fuzz testing can effectively reveal how GNSS receivers respond to rapid signal fluctuations and other anomalous behaviours, situations often overlooked by standard tests. Unlike traditional methods that rely on predefined inputs, Collins Aerospace works on a new fuzz testing framework for GNSS, which employs advanced techniques such as automated input generation and real-time response monitoring. This approach not only facilitates a comprehensive assessment of receiver resilience but also allows for the dynamic adaptation of test scenarios in real-time, ensuring that a wide range of operational conditions is explored. The navigation equipment minimum testing procedures must be defined and need scenarios definitions as well as test steps and pass/fail criteria to provide minimum guidance to manufacturers for future equipment certification. The limitations of current testing methods further highlight the necessity of adopting fuzz testing. These methods predominantly rely on deterministic approaches, which do not effectively simulate the unpredictable nature of real-world signal degradation or complex interference scenarios posed by advanced spoofing techniques. As technology advances, the techniques utilised by malevolent actors likewise evolve, emphasising the necessity for adaptive testing methodologies capable of responding to these changes. By introducing randomness and variability, fuzz testing plays a critical role in bolstering the reliability and operational integrity of GNSS systems by rigorously assessing their ability to withstand both known and unknown threats. The anticipated results from this fuzz testing framework are expected to identify vulnerabilities and enhance the resilience of GNSS receivers, suggesting that fuzz testing can play a transformative role in GNSS validation.
Despite its numerous advantages, fuzz testing is not without challenges. The computational costs associated with generating and processing a diverse array of random inputs can be substantial, particularly when simulating complex scenarios over extended periods. To address these challenges, optimisation techniques that reduce the resource footprint of the testing process are planned to be implemented while ensuring comprehensive coverage. Additionally, fuzz testing may produce false positives, complicating the assessment of a system's reliability or performance. The diverse nature of inputs can also make result analysis complex, requiring significant effort and expertise to identify the root causes of unexpected behaviours. However, these limitations highlight the need for careful implementation and thorough post-test analysis, ensuring that the benefits of fuzz testing outweigh the challenges.
This research aims to explore the applicability and potential effectiveness of fuzz testing in GNSS validation, focusing on the following objectives:
1. Investigate various fuzzing techniques to enhance receiver robustness against interference, particularly in the context of jamming and spoofing scenarios.
2. Assess the feasibility of applying fuzz testing to augment traditional deterministic methods, expanding test coverage to rare conditions and edge cases.
3. Propose a framework for integrating fuzz testing into GNSS validation, focusing on configuration, input generation, and system behaviour evaluation.
4. Analyze the strengths and limitations of fuzz testing compared to traditional methods, considering computational resources and post-test analysis requirements.
5. Apply fuzz testing to specific use cases, comparing results with deterministic methods to highlight vulnerabilities.
As part of this project, Collins Aerospace is developing a comprehensive Fuzz Testing Framework for GNSS receivers, employing both black-box and white-box testing strategies. Black box testing involves designing lab scenarios that randomly modify specific GNSS messages, assessing how well the receiver maintains accurate navigation solutions despite unforeseen signal modifications. For instance, fuzz testing can reveal a receiver’s vulnerability to abrupt signal loss by simulating an environment where signals intermittently disappear, potentially leading to critical failures in position calculation. While this approach effectively uncovers the system's performance under varied external conditions, it may not fully address internal logic flaws or state transitions. Therefore, it serves as a crucial complement to white box testing, which leverages formal models, such as finite state machines and stochastic models, to conduct an in-depth examination of the system's internal workings. This approach includes assessing the transition coverage of the state machine to ensure that all possible states and transitions are evaluated during testing. It not only identifies potential weaknesses that may not surface during traditional black box testing but also assesses the coverage of the tests and adapts the fuzzing process accordingly. Additionally, this approach involves monitoring the system's response to various different input sequences, which helps identify corner cases and logic flaws. To support this process, the project aims to develop user-friendly tools that facilitate model-driven testing and improve the usability of testing environments, allowing developers to observe and manage the fuzzing process efficiently. By integrating both strategies, this framework provides a unique assessment of GNSS receiver resilience, providing a holistic evaluation of vulnerabilities from both external and internal perspectives.
This paper underscores the urgent need for innovative testing methods to combat the increasing threats to GNSS receivers in aviation. Introducing fuzz testing aims to highlight potential avenues for bridging the critical gaps between existing certification standards and the complex, real-world challenges these systems face, thereby enhancing resilience and performance. Ultimately, this research aspires to significantly improve the safety and reliability of GNSS technology in aviation and other critical sectors, fostering trust and confidence in navigation systems for years to come.