Enhanced EGNOS NOTAM Predictions Through Machine Learning Techniques

Román Rodríguez Pérez, Victor Castro Moreno, Jorge Morán García, Elena Rodríguez Rojo, Miguel Ángel Sánchez

Abstract:	The EGNOS NOTAM proposals provision is a key enabler for the publication of RNP approaches down to LPV minima in Europe, as it provides the different AIS (Aeronautical Information Service) the required information to establish a NOTAM service covering this type of procedures, in line with the ICAO standards. Currently, more than 400 APV-I and 150 LPV-200 SBAS procedures are published in Austria, Belgium, Croatia, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Lithuania, Luxembourg, Malta, Montenegro, Netherlands, Norway, Poland, Portugal, Romania, Serbia, Slovak Republic, Spain, Sweden, Switzerland, UK and the Bailiwick of Guernsey. The EURONOTAM tool is responsible for generating EGNOS NOTAM proposals, implementing an analytical service volume that propagates the orbits of the satellites (based on the GPS almanacs) and computes the satellites in view for each configured airport. Once the satellite visibility is computed, the prediction module estimates the residual errors for each Line Of Sight (LoS), taking into account the number of EGNOS Ranging Integrity Monitoring Stations (RIMS) in view (for the User Differential Range Error —UDRE— computation) and the number of Ionospheric Pierce Points surrounding the neighbouring Ionospheric Grid Points (for the Grid Ionospheric Vertical Error —GIVE— estimation). With these estimations, the tool calculates the Protection Levels for each airport and determines if a new NOTAM proposal is to be issued in case the service is not available to support APV-I or LPV-200 operations in a specific location. Thus, the accuracy of the UDRE and GIVE estimations is essential in order to precisely predict the EGNOS availability events. Taking this into account, the present study is aimed at analysing the feasibility and the potential improvement that could be obtained, by using Machine Learning (ML) or Deep Learning (DL) techniques to enhance or replace the current analytical model. More specifically, the different “classic” ML models which will be addressed are decision trees, random forests, support vector machines and extreme gradient boosting (XGBoost). On top of these ML models, DL techniques are also covered by this study. The outputs of these models will be used to estimate the Protection Levels, and its applicability to the current EURONOTAM software will be discussed. The study will start with the analysis of the input dataset used to train the model. Aimed at analysing the impact of EGNOS station outages on the system performance and the capacity of the model to predict such impact, the input dataset will be composed of a minimum of 41 scenarios, including single and multiple station outages. For the UDRE estimation, more than 1.1 million samples, including the following features, will be considered: satellite coordinates, satellite dilution of precision (DOP) (based on the number of station in view), number of RIMS-A and RIMS-C on sight, satellite family, slot number and RIMS status. Similar features will be considered for the GIVE, with more than 3.5 million samples, including as well information on the number and distribution of the IPPs surrounding each IGP. Feature importance will be evaluated using SHAP values. SHAP values are calculated for each input feature and for each sample, and they measure the contribution of that feature to the predicted value for the sample. Summing the absolute SHAP values for all samples gives a measure of how the output is driven by that feature globally. This will be used to identify which are the most important features (and eventually, consider updating EURONOTAM model to consider these features – e.g. DOP) and discard the unimportant ones from the ML model. The real UDREi and GIVEi values used for the model training and testing will be obtained from the real EGNOS messages downloaded from the EDAS FTP. After preparing the input data set and splitting it into the training and testing datasets, the model hyperparameters will be tuned through a grid search with 5 fold cross-validation (for each set of values, 4/5 of the original training set will be used for actually training the model and the remaining 1/5 will be used for assessing the performance). The parameters resulting in the best averaged model performance will be selected. The resulting model will be evaluated with the testing dataset, using the following metrics: - Binary Accuracy: ratio between number of samples correctly classified as monitored or not-monitored vs the total number of samples - Total Accuracy: ratio between the number of correctly classified samples (considering the 16 possible labels) and the total number of samples. - Matthews correlation coefficient (MCC): A coefficient of +1 represents a perfect prediction and 0 represents an average random prediction. - Mean & RMS error on UDRE or GIVE value (UDREi / GIVEi converted to UDRE /GIVE for the assessment). The results obtained so far seem to indicate that Machine learning models can significantly improve the estimates obtained from the current analytical model (geometric-based). For instance, for one of the ML models tested, the system is able to correctly classify the 86% of the UDREi labels versus the 76% obtained with the current model, meaning a 13% improvement. In terms of binary accuracy, the ML model properly estimates the monitored / not-monitored status for the 99% of the samples tested. The RMS and mean absolute error are decreased by a 45% and 55% respectively. For the GIVE estimation, the improvement in the classification problem is more remarkable, with a 30% enhancement with respect to the current value. The RMS and mean absolute errors decrease as well by a 36% and a 50% factor respectively. The study will determine if Machine learning models can significantly improve the estimates obtained from the current analytical model (geometric-based). The study will also evaluate Protection Levels predicted with the usage of ML-estimated UDRE and GIVE values, where an important improvement in terms of prediction accuracy is expected. The analysis will be completed with a brief discussion on the available software libraries enabling the embedding of ML trained models in existing code and the capability of the model to predict situations that are very different from the ones actually trained. This is particularly challenging in this study, due to the lack of a full set of samples covering all the non-nominal conditions, since potential future situations might have never happened before and cannot be therefore, trained. Future steps will focus on this topic, and aim at finding more robust machine learning models that are able to deal with non-nominal conditions while still outperforming traditional methods under nominal conditions.
Published in:	Proceedings of the 34th International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS+ 2021) September 20 - 24, 2021 Union Station Hotel St. Louis, Missouri
Pages:	364 - 384
Cite this article:	Pérez, Román Rodríguez, Moreno, Victor Castro, García, Jorge Morán, Rojo, Elena Rodríguez, Sánchez, Miguel Ángel, "Enhanced EGNOS NOTAM Predictions Through Machine Learning Techniques," Proceedings of the 34th International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS+ 2021), St. Louis, Missouri, September 2021, pp. 364-384. https://doi.org/10.33012/2021.17885
Full Paper:	ION Members/Non-Members: 1 Download Credit Sign In