Amit Jain, Massachusetts Institute of Technology ; Richard Linares, Massachusetts Institute of Technology
https://doi.org/10.64861/PWXN8166
Keywords: Space Domain Awareness (SDA), Multi-agent sensor tasking, Flow matching, Reinforcement learning, Policy optimization
Abstract:
Rapid advances in computer vision have produced substantial improvements in object detection, segmentation, pose estimation, localization, image enhancement, and other areas. While these state-of-the-art algorithms have demonstrated remarkable capabilities in terrestrial applications, their adoption in space operations has lagged significantly. This delay stems from challenges such as hardware limitations, risk tolerance, and the unique demands of space environments. Evaluating the suitability of these methods for space applications remains difficult because studies often use different datasets, metrics, and experimental procedures. This limits objective comparison and complicates technology selection for missions involving Space Domain Awareness (SDA), Rendezvous and Proximity Operations (RPO), and autonomous spacecraft navigation.
This paper presents Foresight, a modular benchmark framework for evaluating computer vision algorithms on space-relevant imagery using standardized datasets, metrics, and evaluation procedures. The framework currently supports five capability areas: object detection, instance segmentation, simultaneous localization and mapping, spacecraft pose estimation, and super resolution. Dataset versions, software environments, model configurations, and evaluation results are tracked within a common experiment-management infrastructure, enabling consistent comparison and reproducible results across algorithms and computing environments.
Foresight combines physics-based synthetic imagery with real mission data. The current benchmark includes more than 139,000 visible and long-wave infrared (LWIR) images of satellites generated using Space Dynamics Laboratory’s SynGen software, together with several thousand annotated images from NASA’s Raven experiment aboard the International Space Station. An initial benchmark campaign evaluated 57 object-detection and instance-segmentation models across synthetic and operational imagery.
Results show that model rankings vary substantially across sensing modalities, datasets, and task types: newer YOLO architectures performed strongly on the Raven benchmarks, while transformer-based and two-stage architectures led the synthetic LWIR segmentation benchmark. Several compact models also achieved performance comparable to substantially larger alternatives, highlighting opportunities for resource-constrained onboard processing.
These results demonstrate the value of evaluating algorithms across multiple datasets, sensing modalities, and task types rather than relying on performance from a single benchmark. Foresight provides a reusable foundation for systematically assessing emerging computer vision technologies, identifying promising algorithms, quantifying accuracy-compute tradeoffs, and prioritizing candidates for further testing on flight-representative hardware.
Date of Conference: September 15-18, 2026
Track: AI for SDA Applications