02 AI-supported inspection drone in base maintenance

Intro

Jan-Christopher Knufinke is Lean Innovation Manager with a strong focus on digital applications. He will share with you how AI-supported inspection drones might be used at Lufthansa Technik for base maintenance checks. In this issue he shares his experiences in the research project he conducted together with partners to explore the possibilities of AI inspection drones for base maintenance.

Jingle

Did you know that during base maintenance checks mechanical experts thoroughly inspect the outer skin of the aircraft? That means they spend a lot of time examining the aircraft in a crouched or crawling position, secured by harnesses to stay safe at great heights. What if this process could be performed by inspection drones supported by Artificial Intelligence, that eliminates poor ergonomics, ensures job safety and digital data output?

Ideally, in such a way, the application specifically supports the competence of the mechanical experts and the drone and recording system pre-selects surface damage, so that the actual damage repair - the so-called defect rectification - can be directly prepared and performed.

That is what we were eager to find out in a research project, funded by the German Federal Ministry for Economic Affairs and Energy. Together with many partners we worked to explore the possibilities that the AI inspection drone can bring to our MRO business during base maintenance checks.

Advanced Air Mobility and our research focus

Of course, as an aviation company, the Lufthansa Group stays up to date on Advanced Air Mobility. In the future, we may see air taxis flying to and from hubs and destinations. And the development of service offerings for MROs or IT services has already started, so that an infrastructure can be established - drones are not a thing of the future anymore!

But back to our research project: We will focus on how drones can be used primarily within a base maintenance operation as a tool to increase efficiency and further developing our vision into a holistic Tech-Ops provider with data-driven production.

But back to our research project: Let’s get started - Setup of our research project

But back to our research project: For us at Lufthansa Technik, we always strive to improve and digitalize production processes that lead to reduced turn-around-times and cost savings through efficiency for our customers.

But back to our research project: Together with the Helmut Schmidt University Hamburg, the Technical University Munich, the University of Applied Science Würzburg-Schweinfurt, Lufthansa Industry Solutions and zeroG we teamed up to explore possibilities to research innovative operation processes in aircraft inspection through the use of drones with inspection cameras and Artificial Intelligence for data analytics. Ultimately leading to the automation of inspection processes – e.g. reporting, evaluation and report generation.

Through the funding of the German Federal Ministry for Economic Affairs and Energy, we were given the opportunity to explore the following four fields of action that we identified:

Process optimization: Drone-based inspection process for visual verification of surface damage with focus on common damage types such as lightning strikes, dents and scratches.

Evaluation of existing solutions on the market: Check existing drone companies and start-ups on the market

Innovation: Application of Artificial Intelligence for classification and analysis of recorded damage images as well as contextual understanding in maintenance relevant documentation such as Aircraft Maintenance Manuals, Structural Repair Manual or customer specific work packages

Strategy: Support for prototypical development in digital projects such as automated readout of work packages with regard to inspection tasks, Aircraft Maintenance Manuals structure for automated reading of all relevant information or automated readout of defect Job Cards regarding instructions for damage repair.

 We defined our work packages based on the core competencies of each partner and structured the project into three clusters:

 We defined our work packages based on the core competencies of each partner and structured the project into three clusters: maintenance concepts,

 We defined our work packages based on the core competencies of each partner and structured the project into three clusters: AI architecture,

 We defined our work packages based on the core competencies of each partner and structured the project into three clusters: and operation & application scenarios.

 We defined our work packages based on the core competencies of each partner and structured the project into three clusters: In this way, we were able to work our way up from concept phase to demonstration and testing to evaluation and utilization.

 We defined our work packages based on the core competencies of each partner and structured the project into three clusters: Contributing our expertise

As one of the world's leading MRO providers, we were able to contribute specialist knowledge of the General Visual Inspection process as part of recurring functional inspections from our global maintenance network:

As one of the world's leading MRO providers, we were able to contribute specialist knowledge of the General Visual Inspection process as part of recurring functional inspections from our global maintenance network: The objective was to verify whether visual inspection of aircraft exterior surfaces focusing on damage, defects or irregularities can be optimized by innovative solutions such as drones or robots at touching distance.

As one of the world's leading MRO providers, we were able to contribute specialist knowledge of the General Visual Inspection process as part of recurring functional inspections from our global maintenance network: A prototypical development in digital projects as part of the digitalization strategy of the Aircraft Maintenance segment for automated reading of work packages with regard to inspection tasks, structure based on Aircraft Maintenance Manuals for automated reading of all relevant information as well as automated reading of instructions for damage repair.

Fast forward: After three years of research - the results are in!

Fast forward: It was a long way but we were able to accomplish challenging tasks and gather important information during our three-year project together with our partners.

Fast forward: Our use case for the development of an autonomous flying drone with one meter flight distance to the object and automatic collision avoidance, including obstacle detection, capable of capturing high-resolution damage images from a size of 5x5 mm in full HD resolution was successfully fulfilled. Furthermore, the 3D imaging of hangar infrastructure, aircraft and ground support equipment enables the drone system its orientation during flight.

Fast forward: Artificial Intelligence

Fast forward: In the field of Artificial Intelligence, we accomplished the simulation of surface damages in a real-time environment and in HD resolution for generating the data material. Through a mobile device app, defects can be captured with prototypical classifications and analysis of damage patterns. In addition, we used Natural Language Processing to analyze historical damage documentation and created an aviation-specific “MRO dictionary” for automated extraction of relevant information from the maintenance documents. This includes as well the automated reading of workpacks regarding inspection tasks and Aircraft Maintenance Manuals-based structure for automated extraction of all relevant information.

Fast forward: Technological concept development and prototypical integration into adjacent MRO operating processes

Fast forward: In order to include adjacent MRO operational processes in the technological concept and prototypical integration, the required data points for selected use cases, such as job card creation or defect management, were identified. This includes automated readout of workpacks related to inspection tasks and the structure based on Aircraft Maintenance Manuals for automated readout of all relevant information (unified master data sets regarding planning attributes for routine and non-routine tasks).

Fast forward: What’s next?

Fast forward: The AI inspection drone research project proves that the use of an AI inspection drone in base maintenance checks is a concept that supports our mechanical experts in detecting and classifying surface damage on aircraft. This can lead to increased efficiency, as turnaround time can be significantly reduced by allowing the experts’ attention to focus on the damage detected during the drone flight.

However, implementation cannot be accomplished overnight, as infrastructural adjustments are also necessary for integration into routine processes. And, of course, all legal requirements must be taken into account. So the next step will be to identify further use cases based on the findings from this research project. It will be interesting to follow how further data, including historical data, can be integrated into the processes. One thing is certain, however: Drones are no longer a thing of the future and will also bring benefits to the MRO industry!

However, implementation cannot be accomplished overnight, as infrastructural adjustments are also necessary for integration into routine processes. And, of course, all legal requirements must be taken into account. So the next step will be to identify further use cases based on the findings from this research project. It will be interesting to follow how further data, including historical data, can be integrated into the processes. One thing is certain, however: Three years of research project with important insights, great colleagues and partners

However, implementation cannot be accomplished overnight, as infrastructural adjustments are also necessary for integration into routine processes. And, of course, all legal requirements must be taken into account. So the next step will be to identify further use cases based on the findings from this research project. It will be interesting to follow how further data, including historical data, can be integrated into the processes. One thing is certain, however: During the three-year project period, we gained insights into scientific high-tech projects and potential process improvements through the application of innovative solutions and market trends. Our multiple hangar visits with the project consortium during hardware and software testing and meetings in Munich and Hamburg served as an interesting platform for technology and knowledge exchange between industry and science.

The intensive internal exchange with our mechanical and structural expert colleagues was very valuable to me to learn more about the General Visual Inspection process, regulatory requirements, process potentials and a possible integration of drone solutions – but most importantly: It was a great opportunity to expand my personal knowledge in research project work and as well as in my professional network.