Mosaic brought its deep expertise in aviation to design and test effective machine learning models that would predict the arrival and departure assignments of flights, helping facilitate automated testing and deployment of advanced predictive decision-support tools.
Take Our Content to Go
Assigning flights to runways at an airport is a crucial function influencing airport operations and aircraft performance. Air traffic controllers develop these plans and set the runway a flight must land or depart from. As with any operational decision-making process, there are many contributing factors, constraints, and events to account for, including airport configuration, the configuration of neighboring airports, aircraft size and engine type, noise/environmental regulations, and traffic volumes. Controllers must consider requests from flight crews for specific runways out of operational necessity and preference. The assignment rules evolve as new procedures and routes are implemented.
There was a need for runway assignments to be more efficient while considering these factors, which could help airports reduce runway delays and improve the stakeholder experience. Mosaic Data Science developed an ML-driven approach to predict the runway assignments that controllers would make. The project was part of a larger effort between Mosaic’s team of skilled data scientists and NASA to address inefficiencies in today’s air transportation system, known as the NASA-led ATD-2.
Throughout the project, Mosaic brought its specific expertise in aviation to design and test effective machine learning models that would predict the arrival and departure assignments of flights, which would help facilitate automated testing and deployment of advanced predictive decision-support tools. Mosaic’s deep roots in the industry enabled our team to address such legacy problems with creative analytics solutions.
Previous research on runway assignments has focused on optimizing the process (e.g., to improve throughput). While that body of work suggests potential benefits, there are significant barriers to implementation. Other work, including earlier in ATD-2, has focused on interviewing operational personnel to learn the underlying heuristic they use to make runway assignments. This approach is costly because of the time and effort involved, makes capturing exceptions and interactions in the rule-based approach difficult, and can be difficult to update as the operation evolves.
The approach in this project assumes that controllers will continue to use their expertise and available automation tools. The machine learning models replicated the runway assignment heuristics used by controllers. By developing this data-driven approach and proving its effectiveness against the existing static approaches already developed for some airports, runway assignment models can then be easily developed for additional airports at low cost.
To develop an accurate machine learning model, Mosaic’s aviation data scientists needed to model the decision heuristics used by controllers in their runway assignments. One of the critical objectives in this research is to have the ability to deploy the model in a real-time system to compare performance against existing ATD-2 approaches to evaluate future automation and decision support features.
Mosaic wanted to apply a genuine data-driven approach, meaning it does not require significant adaption data maintenance. These objectives were selected to allow the models to be tested against dozens of airports across the U.S., provided that the data formats matched the training data. Mosaic generated data formats using the ATD-2 data fuser to cut down on feature and data engineering efforts.
The modeling process for the arrival and departure runway assignment problem was developed to be generic and quickly deployable to several U.S. airports. The overall machine learning flow followed standards set out by CRISP-DM and ASUM-DM. The CRISP-DM can be seen in the figure below.
Mosaic identified a target value as the historical runway identifier used by the flight. Feature engineering was developed based on literature review, brainstorming, and collaboration with available air traffic controllers. After exploratory data analysis, additional filtering was conducted to prepare the data for the model training algorithm. This filtering process picked up on anomalies, which skew results and imputed missing values deemed critical to the algorithm. The training data was now ready to be run into a machine learning algorithm.
|Aircraft engine class||x||x|
|Wake turbulence category||x||x|
|Time since airport configuration changed||x|
|Time until estimated operation||x|
Feature engineering is critical to any machine learning effort. The features are summarized in the table above, which are included in the current version of the model.
Machine Learning Prototyping
The problem lends itself to multinomial classification algorithms. There are several classification algorithms, and Mosaic’s aviation data scientists tested several, including a classic logistic regression (logit), and XGBoost. These results must be tested against a baseline to determine performance metrics, and Mosaic reported the most common target value in the training dataset as the prediction. More sophisticated baseline heuristics and classification algorithms will be tested in future iterations.
Preliminary Predictive Results
The preliminary results indicate the potential benefits of a machine learning approach to the problem. It is important to note these results were generated during the COVID-19 pandemic, which led to significant disruptions in aviation operations. Mosaic observed significant differences in the data and took these effects into the presentation of the results.
Predicting Runway Assignments | Arrivals
Initial models focused on Dallas-Fort Worth International Airport, as it is a test site for the systems being developed on the ATD-2 project. The accuracy can be represented in terms of a percentage of correct predictions made by the models. The initial results for the two ML models are listed below.
|Airport Identifier||Logistic Regression Accuracy||XGBoost Classifier Accuracy|
Predicting Runway Assignments | Departures
The departure models were validated against airports listed in the table below. Again, the accuracy metrics can be represented as a percentage of how well the model projected the departing runway.
Mosaic explored additional airports using pre-pandemic data and found similar accuracy scores. Several improvements are planned to expand and solidify these preliminary results.
Predicting Runway Assignments vs. ATD-2 Baseline
The air traffic controller runway assignments are used in the ATD-2 systems at Charlotte Douglas International Airport and Dallas-Fort Worth. As part of the project, these models are regularly updated with controller feedback and provided a great baseline to compare the ML values. The performance is displayed below.
|Airport Identifier||Departure Accuracy||Arrival Accuracy|
Several trends are apparent from the baseline results. The departure runway accuracy exceeds what is achieved with current ML models, meaning the ML models need to include more features, model types, and further tuning. The arrival accuracy performance at Dallas-Fort Worth indicates that the decision heuristics employed by controllers have likely changed, but the baseline models used in the ATD-2 system have not been updated to reflect these new practices.
With increasing air traffic demands, efficient runway use has become very important. Future implementation of arrival management with optimal runway assignments is being examined by NASA. While a nominal runway is basically assigned, an optimal runway assignment can potentially increase runway capacity. On the other hand, optimal assignment could increase the workload of air traffic controllers (ATCs). Therefore, an optimal runway assignment strategy must consider both capacity and workload for operational feasibility.
Machine learning shows initial promise for learning how controllers make runway assignment decisions when aircraft land or depart, which was the goal of this research effort. The overall approach enables broad deployment across U.S. airports by following careful data preparation and modeling flows. Fused with other models to predict NAS operations, Mosaic and NASA believe this data-driven ML approach will facilitate automated testing and deployment of advanced predictive decision-support tools.