Imène Goumiri, Lawrence Livermore National Laboratory; Luc Peterson, Lawrence Livermore National Laboratory; Ashley Cocciadiferro, Lawrence Livermore National Laboratory; Ryan Lee, Lawrence Livermore National Laboratory; Jason Bernstein, Lawrence Livermore National Laboratory
Keywords: Space domain awareness, test and evaluation, machine learning, conjunction, RPO
Abstract:
The Space Domain Awareness (SDA) Tools, Applications, and Processing Lab (TAP), or SDA TAP Lab, is an initiative of the U.S. Space Force / Space Systems Command to efficiently and effectively transfer technology from industry, academia, and federally funded research and development centers (FFRDCs) to Space Force guardians and SDA operators. A primary goal of the SDA TAP Lab is to accelerate the development and implementation of innovative solutions to real mission-relevant SDA problems. Challenged with such problems as determining whether a rocket launch poses a threat to a satellite or detecting pattern of life violations, participants are tasked with rapidly developing software to be used by actual operators. Testing and evaluation (T&E) of this software is critical to ensure that it solves the technical problem, operates as required, and can be objectively benchmarked against other software solutions. Lawrence Livermore National Laboratory is performing T&E for the SDA TAP Lab. By drawing from best practices across the software development, SDA, and machine learning and artificial intelligence (ML/AI) communities, our goal is to ensure the process is quantifiable, objective, and rigorous, all while spurring innovation. In this work, we outline the general approach we have adopted for T&E, namely the Common Task Framework (CTF) that has driven innovation in the AI/ML communities, and focus on a particular benchmarking problem we have developed involving classification of rendezvous and proximity operations (RPOs) from uncorrelated tracks (UCTs).
The TAP Lab’s’ general approach to T&E is called the Common Task Framework (CTF) in the ML/AI community. In brief, the idea is to clearly specify a challenge dataset and a well-defined data-centric objective, such as classifying launches as potential threats or not, and score submissions on their performance against a hold-out dataset via a small number of meaningful but quantifiable metrics such as precision, recall, and accuracy. The progress of the community on these tasks can then be easily monitored over time and the state of the art solution is readily identifiable as the solution that scored best according to the defined metrics. Well-known examples of this approach in the ML/AI communities are image recognition applied to the MNIST and CIFAR-10 data sets. In the SDA TAP Lab context, there is additional complexity due to the scarcity of real-world, labeled data, that can be mitigated in part with the use of modeling and simulation or novel AI tools such as large language models. Our T&E framework accommodates other desirable features such as the ability to submit anonymously and a thoughtful user-interface to lower the barrier to entry.
One benchmark we are developing with the CTF approach is to classify a batch of data as an RPO, conjunction, or neither from a set of UCTs. This task is challenging for several reasons, including class imbalance between RPO and conjunction events and the need to precisely define an RPO event. We report statistics on this data set and metrics used to benchmark submissions, including miss distance for quantifying orbit determination quality and categorical cross-entropy for assessing data association. The recent literature includes several benchmark SDA problems, but this to the best of our knowledge is the only one involving RPOs. Access to this data is available to participants in the Project Apollo initiative of the SDA TAP Lab and other affiliated individuals or groups.
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.
Date of Conference: September 17-20, 2024
Track: Machine Learning for SDA Applications