Fuel planning optimization with machine learning: the KLM challenge
By Jeroen Nelen
In aviation, fuel planning is an essential but complex task that requires optimizing efficiency while ensuring the aircraft carries the required amount of fuel. At KickstartAI, we believe that AI holds great potential to improve this process, and we put it to the test in our recent project with KLM. This project, called Daidalos, aims to optimize contingency fuel: the reserve fuel planes carry for unforeseen circumstances, such as delays, rerouting, and adverse weather.
Part of this project is a challenge, a two-week structured hackathon organized by KickstartAI and KLM to improve the baseline of the initial proof of concept (PoC). During the challenge, the participants looked at new data sources, different models, and explainability techniques. This blog dives into the problem that sparked the project and details the challenge setup and its preliminary results.
The problem
As mentioned above, contingency fuel is the reserve fuel planes carry to handle unforeseen events during the flight. Every airline is required to carry contingency fuel, but calculating the exact amount varies between different companies.
KLM, in particular, has a statistical approach that is generally performing well. However, one limitation of traditional statistical models is their challenge in anticipating rare outliers which can be particularly significant when it comes to contingency fuel planning. Recognizing this, KLM saw an opportunity to investigate ML including the incorporation of additional data sources and features, to predict a flight-specific amount of contingency fuel required.
By harnessing these innovations, the goal is to better tailor contingency fuel loads, adding more when unusual conditions are detected, and safely optimizing the amount carried when possible. Overestimating the amount of contingency fuel adds more weight on board, increasing the overall fuel consumption, a concept known as “cost-to-carry.”
The challenge
For two weeks, 12 data scientists from Ahold Delhaize, ING, KLM, and KickstartAI worked on the problem at hand. We grouped the participants into three teams, each asked to build a complete machine learning solution.
We broke the ML pipeline down into three work packages: validating new data sources, testing different ML models, and exploring explainability techniques. Each team member had to focus on one of these components and work together with other teams to share knowledge. To make sure the hackathon was about improving the baseline, KLM together with KickstartAI provided an evaluation script that automatically calculated the two important axes and key performance indicators (KPIs); cost-to-carry excess fuel and contingency efficiency.
The work packages:
The challenge focused on three objectives, which were:
Validating new data sources: Participants were given unexplored data to perform additional analysis and develop features. The goal was to identify new features that had a positive effect on the performance of the model. The data sources included weather data, airport-specific information, and historical flight data.
Test different ML models: This work package focused on challenging the baseline by testing different models and improving performance.
Explore explainability techniques: Understanding the model as a data scientist and being able to communicate the rationale behind it to the pilots is important for the final solution. This is key to increasing trust in the solution and therefore encouraging adoption by key stakeholders such as pilots and dispatchers.
Results:
WP1: Data analysis & features
One of the goals of the challenge was to investigate unexplored data sources. The first source was TAF forecasted weather data and the second source was METAR actual weather data. We provided this data because weather is one of the main reasons why flights divert from their route. After the raw data was parsed, it was possible to transform it into features, such as wind direction, rainfall, or visibility on the departure and arrival airport. Surprisingly, these features did not improve the model significantly during the challenge.
The second data source consisted of historical records of all scheduled international flights of the last two years. We used this data to estimate how busy an airport is at a specific moment and if there might be congestion at the arriving airport. Based on an inferred capacity, we could calculate the likelihood of congestion by comparing the expected number of movements against different capacity proxies. The different proxies that were tried during the challenge were: the total number of runways of an airport, the historical hourly maximum movements on that specific hour of the day on that airport, and lastly, how the current traffic deviates from a dynamic baseline. This baseline was calculated by taking a 4-week rolling average of that specific airport, day of the week, and hour.
The third feature we added was a set of lagged features, feeding the model with additional data points from the previous 7 days. The rationale was that recent events could indicate underlying causes for delays or deviations in trip fuel consumption. This lagged feature had a positive influence on the performance of the model.
Lastly, we provided a lot of features extracted from the Operational Flight Plan (OFP). This included distance, flight time, airplane details, and more. A technique called Reverse Feature Ablation was used to filter out all features that did not have a significant effect on the output. This final set of features made it possible to beat the baseline.
WP2: Modeling
For the baseline model we used LightGBM, which proved to be the best performing model during the challenge as well. Nevertheless, we were able to make two notable improvements that boosted LightGBM final performance.
The first one was removing extreme outliers from the training dataset, which resulted in better performance on the KPIs. This filter was not applied on the test set, as these outliers are an important part of the evaluation. Secondly, we post-processed the predictions by adding a safety margin to each prediction. This simple step made the model significantly more reliable, while not increasing the amount of unused fuel. We then used hyperparameter tuning to find the optimal balance between these two parameters.
It must be noted here that while these solutions contributed to the final KPIs, they didn’t increase the model performance when looking at the evaluation metrics such as the pinball loss. This might be solved with a custom loss function, but this isn’t something we tried during the challenge.
WP3: Explainable AI
The last workpackage focused on explaining the output of the model to the pilots and the dispatchers. To achieve this, we explored multiple explainability techniques and visualizations. The final prototype was built using Streamlit and included a couple of interesting features.
The first technique was local feature importances using SHAP values. This technique can be used to show which features were contributing the most to the output and how. Pilots could use this information to further understand whether the contingency fuel was sufficient for the circumstances of that flight. On top of this logic, we built another feature we called “what-if.” This tries to answer the question: “If I change feature X, what happens to the final output?”.
Lastly, we included additional information about the current flight and the previous flights from that same route. Together, these explainability features and visuals can provide pilots with a clearer view into model decisions, and therefore encourage adoption of the ML solution.
Conclusion and next steps
During the KLM & KickstartAI challenge, 12 participants took a fresh look at the problem and built an improved solution. They turned new data sources into interesting features, tested different models, and took the first step toward explaining the model’s results.
In the next phase, the team will validate and further discuss the solution with the business stakeholders, before moving to implementation. Once the ML model is approved, KLM will compare it in practice with the current solution and continue testing. The pilots will also receive the model’s output along with clear explanations — an important step toward full adoption of this machine learning model.
