Peng Mun Siew, Massachusetts Institute of Technology; Haley Solera, Massachusetts Institute of Technology; Giovanni Lavezzi, Massachusetts Institute of Technology; Thomas Roberts, Massachusetts Institute of Technology; Daniel Jang, MIT Lincoln Laboratory; David Baldsiefen, Julius-Maximilians-University of Würzburg; Binh Tran, Millennial Software Solutions Inc. & Indiana University of Pennsylvania; Christopher Yeung, Millennial Software Solutions Inc. & Indiana University of Pennsylvania; Kurtis Johnson, Millennial Software Solutions Inc. & Indiana University of Pennsylvania; Nathan Metzger, Millennial Software Solutions Inc. & Indiana University of Pennsylvania; Francois Porcher, University of California, Berkeley & BNP Paribas; Isaac Haik, University of California, Berkeley & BNP Paribas; Victor Rodriguez-Fernandez, Universidad Politécnica de Madrid; Jeffrey Price, United States Air Force; Jonathan How, Massachusetts Institute of Technology; Richard Linares, Massachusetts Institute of Technology
Keywords: Artificial Intelligence, Space Situational Awareness, Satellite Pattern-of-Life Characterization, Anomaly Detection
Abstract:
In recent years, the application of artificial intelligence (AI) techniques to solve complex problems has received significant attention within the scientific community. However, there remains a notable lack of adoption of these techniques within the space situational awareness (SSA) community. Despite the wealth of historical data available on Earth-orbiting objects, including orbital state and light curve measurements, the disconnect between the AI and SSA research communities has hindered interdisciplinary progress.
Over the last year, the Massachusetts Institute of Technology (MIT) has launched the 2024 MIT ARCLab Prize for AI innovation in Space focusing on leveraging AI algorithms to characterize satellite pattern-of-life (PoL) within the Geostationary Earth Orbit (GEO). GEO satellites play a critical role in various applications such as communication, weather monitoring, guidance and navigation, and Earth observation. These satellites, throughout their operation lifetime, undergo various behavioral modes, including station-keeping, longitudinal shifts, and end-of-life behaviors, all of which are carefully planned and executed by satellite operators. Satellite PoLs can be used to describe on-orbit behaviors that are typically composed of sequences of both natural and non-natural behavior modes.
This unique challenge combines complex multivariate time-series data analysis, behavioral pattern recognition, and anomaly detection to contribute to a safe and sustainable space environment. One of the key challenges is the variability in GEO satellite behaviors, which can be attributed to factors such as the lack of standard operational guidelines, mission objectives, and propulsion systems. This variability makes it challenging for analysts to manually identify and classify satellite behaviors accurately.
AI algorithms are particularly adept at recognizing and learning from patterns, making them well-suited for processing SSA data. The increasing number of space objects and the vast amount of data generated daily make manual analysis increasingly challenging. AI algorithms can alleviate this burden by automating repetitive and mundane tasks, freeing up human resources for more critical decision-making. Additionally, advanced AI algorithms can predict future behaviors and assist in proactive decision-making. AI algorithms can also help uncover complex patterns and non-intuitive relationships that are obscured within the noisy time series data.
The challenge dataset consists of 2402 satellite trajectories spanning six months with a two-hour temporal resolution. These trajectories include 2000 simulated data points, 162 generated from Vector Covariance Message (VCM) data, and the remaining 296 generated from two-line elements (TLE) data. The synthetic data is generated using an in-house satellite simulation tool developed by the MIT Lincoln Laboratory. This tool utilizes a high-fidelity satellite propagator to simulate the trajectories of satellites under various operating objectives and propulsion capabilities. Each simulated satellite trajectory features satellites with unique physical parameters, initial orbital elements, and propulsion systems. The simulated satellites can freely change their behavioral modes independently of each other. The dataset is divided into training, public test, and private test sets. A private test set is used to prevent the participants from overfitting to the public test set. Furthermore, trajectories from different periods are included in the test data to evaluate the generalization capability of the algorithms. Accompanying the challenge dataset is a development toolkit that includes basic utility functions for data parsing, manipulation, and visualization, as well as baseline heuristic and machine learning solutions.
The challenge was hosted on the EvalAI platform, where participants submit trained models and algorithms to be evaluated on a hidden test dataset. Submissions are assessed based on a combination of their precision and recall scores using the F2 metric, along with the quality of their technical write-up, with an 80:20 weightage. The technical write-ups are judged on clarity, novelty, technical depth, reproducibility, and insights by a panel of expert judges. This comprehensive evaluation approach ensures that both the algorithm’s performance and the value of the technical insights are recognized and rewarded in the competition.
The AI SSA challenge has been a success, attracting over 100 teams from diverse backgrounds and regions worldwide. These teams have collectively submitted more than 350 entries on our evaluation platform. The participants included a mix of students, seasoned machine learning practitioners, aerospace engineers, and representatives from academic institutions and companies. Each team brought a unique approach to solving the challenge, from novel algorithmic developments to creative problem-solving strategies. A particularly encouraging outcome of the challenge was the high level of performance achieved by many teams, with the majority of submissions surpassing the baseline solution provided and the top 3 teams achieving an F2 score of 0.92, 0.90, and 0.77 respectively.
We also examined the trends and patterns observed among the submissions, providing valuable insights into the current state-of-the-art in AI techniques for behavioral pattern recognition in SSA. Additionally, we discuss the implications of these findings for future research and development efforts in the field, including the upcoming 2025 MIT ARCLab Prize for AI Innovation in Space Challenge.
Date of Conference: September 17-20, 2024
Track: Space Domain Awareness