Kimberly Kinateder, Applied Optimization; Brandon Baucher, Applied Optimization; Joseph Timm, Applied Optimization; Alexander Crabtree, Previously with Applied Optimization; Richard Linares, Massachusetts Institute of Technology ; Anil Chaudhary, Applied Optimization
https://doi.org/10.64861/NSRF1949
Keywords: Space object characterization; light-curves; Machine Learning; SSA; Bayesian Neural Networks; Space Object Classification
Abstract:
In this paper, we present a Bayesian neural network (BNN) for classification of light curve feature data into debris, rocket body, and payload classes. Our method achieves state-of-the-art performance for learning-based light curve feature classification on real observational data, while accounting for fundamental noise in the data. The accuracy of the classification method is measured by multiple performance metrics, as well as the overall classification success rate. Our BNN classifies vectors of numerical features that are computed from the light curve signatures, after an optimal set of features is determined. Using feature vectors as inputs to the BNN instead of a full light curve reduces the dimensionality of the input data, which reduces the number of parameters that need to be learned by the BNN. In addition, using feature vectors as inputs provides a shortcut to the model’s learning, since the BNN no longer needs to use convolution and max pooling layers to locate and extract the features.
We define a customized loss function that increases classification performance as measured by multiple metrics, even in the presence of severe class size imbalance in the training data set. Typical multi-class classification BNN models utilize the categorical cross entropy and Kullback-Leibler divergence loss terms, which optimize overall classification performance but can be highly influenced by the accuracy of the model on the largest class of the training data. To overcome this problem, we replaced the categorical cross entropy with a new customized loss function. The customized loss function aims to maximize accuracy for each of the classes and hence overall accuracy, as well as positive predictive value for all classes. Positive predictive value evaluated for the test data is the proportion of class assignments that are correct, while accuracy for each class individually is measured as sensitivity, the proportion of objects within a class that are assigned to the correct class.
We trained the BNN on a set of 3849 real light curve tracks. The data set was comprised of 233 debris, 2679 payload, and 937 rocket body tracks. Once the model was trained, we demonstrated performance success of the BNN on a set of real testing data using metrics for predictive success, object type discovery rate, credible intervals for individual object type classes, and uncertainty reporting for overall object type classification. Moreover, the object type classification for uncertainty reporting can be further decomposed into uncertainty from the model and uncertainty in the data. We recognized that fundamental noise in the data is an expected component of any real data set. For our application, fundamental noise consists of the set of testing data samples that do not have enough information to produce an accurate classification with sufficiently high certainty, as measured by BNN model uncertainty measures computed the testing data. Our model achieves an overall accuracy rate of 85% classification success when fundamental noise in the data is identified and removed. Moreover, the positive predictive values for our three classes ranged from 67% to 90%, while sensitivity values ranged from 66% to 90%.
Our results provide three notable achievements compared to recent neural network results in the literature for light curves: our customized loss function improves both predictive and discovery rates for all classes individually, as well as overall; we use the BNN to identify fundamental noise in the testing data that could negatively affect the classification of new data; and our data was both trained and tested on real data, resulting in a higher success rate than other recent models trained only on simulated light curve data and tested on real data. This paper concludes that the proposed BNN and its customized loss function for classifying light curve data is useful for object type classification as measured by multiple metrics and by its ability to identify fundamental noise in the data set, and it therefore has potential to expand SDA and add valuable information to the growing population of RSOs.
Date of Conference: September 15-18, 2026
Track: Satellite Characterization