Ryan Herchig, IERUS Technologies, Inc; Ted Green, Lockheed Martin Corporation; John Burger, IERUS Technologies, Inc; Mary Stewart, IERUS Technologies, Inc
https://doi.org/10.64861/KKEH7827
Keywords: multi-agent reinforcement learning, space domain awareness, dynamic tasking, satellite constellation operations, satellite maintenance planning, multi-objective reinforcement learning, autonomous space operations
Abstract:
Operating a space-based sensing constellation requires balancing two competing imperatives: executing the observation taskings the mission exists to serve, and safekeeping the on-orbit assets execution depends on. With the rapid growth in launches, orbital debris, and on-orbit activity, the management of space-based sensing assets is becoming increasingly complex, compounding the difficulty of sustaining continuous tasking performance. Every maintenance period spent preserving a satellite’s sensor is observation capacity taken from the tasking queue; every period of aggressive tasking degrades the fleet that must execute tomorrow’s queue. These decisions are coupled across satellites and across time, with the task side itself being temporally structured: taskings arrive dynamically, carry priority grades and perishable collection windows, and include custody-class requirements paying off only when a multi-observation sequence is completed. Because the queue persistently exceeds fleet capacity, every planning period is an exercise in triage: which taskings to execute now, which to defer, and which to let expire. Current operational practice typically addresses scheduling with combinatorial optimizers that plan each period independently. This single-period optimization forfeits exactly the multi-period reasoning the balance requires: it cannot anticipate stochastic sensor degradation, adapt to dynamic tasking arrivals, coordinate fleet-level maintenance timing, or weigh present execution against future capacity._x000D_
_x000D_
Reinforcement Learning (RL) has shown promise for satellite scheduling, with single-agent approaches demonstrating fast policy execution suitable for on-board autonomy. However, applications to multi-satellite coordination that integrate task execution with fleet-level operational control (maintenance, energy, and dynamic tasking together) remain limited. The problem is the space-based counterpart of the sensor-network orchestration challenge that has motivated recent operational programs with the added complication that collecting assets themselves require ongoing management in an increasingly congested and contested operating environment. Hybrid RL-optimizer architectures have emerged as a promising paradigm where machine learning guides classical solvers rather than replacing them. However, existing approaches lack the strategic, multi-period coordination needed for fleet-level operations under uncertainty, and also lack multi-objective formulations that would allow operators to move along the execution-versus-fleet-health trade which remain largely unexplored._x000D_
_x000D_
Our work addresses three gaps: (1) extending strategic-guidance architectures to multi-agent settings with centralized training and decentralized execution (CTDE), enabling fleet-level coordination while respecting operational communication constraints; (2) identifying the credit-assignment structure this balance requires, showing that under naive shared credit, agents learn to refuse maintenance precisely when fleet health is lowest, and that counterfactual (marginal-contribution) credit corrects this failure, a mechanism we predicted from offline analysis and confirmed in training; and (3) enabling multi-objective doctrine discovery, where operators select policies along the execution-versus-stewardship front based on evolving mission priorities rather than fixed optimization weights._x000D_
_x000D_
We present a two-layer autonomous operations framework pairing learned strategic reasoning with classical combinatorial optimization. A multi-agent RL system (one agent per satellite) outputs strategic parameters each planning period: maintenance decisions, collection priority shaping across the tasking queue, and per-satellite energy envelopes. These parameters are consumed by a downstream combinatorial solver. During training we use integer linear programming (ILP) for provably optimal, deterministic schedules, eliminating solver noise so the reward signal reflects only the quality of the agents’ strategic choices; the guidance object is solver-agnostic by design for deployment flexibility. The architecture functions as a serial shield: the solver guarantees constraint satisfaction regardless of the agents’ proposals, so the learning layer can be enabled, disabled, or overridden without affecting safety. Task execution proceeds at the tempo of the planning cycle. Each period, the feasibility stage generates hundreds to over a thousand candidate sensor-target-window opportunities. The policy prices this menu, and the solver selects a joint fleet plan satisfying access, energy, and health constraints, replanning every period as new taskings arrive and conditions evolve._x000D_
_x000D_
Training follows CTDE with multi-agent proximal policy optimization (MAPPO) on simulation rollouts. The training environment, COSMO (Constellation Operations Simulation for Multi-Objective Optimization), is a medium-fidelity simulator. It models stochastic sensor degradation via Ornstein-Uhlenbeck processes with health-to-quality coupling, weather fields from ERA5 reanalysis, and coupled onboard energy dynamics with battery carryover, with per-satellite resource envelopes enforced by the solver. Tasking arrivals follow Poisson processes and include custody-class multi-observation requirements with completion-threshold credit. The constellation is heterogeneous (optical and SAR), and a pluggable solver is embedded in the environment dynamics with parallelized rollout collection. We additionally report findings from investigating model-based gradient augmentation through a physics-informed learned dynamics model in solver-in-the-loop training._x000D_
_x000D_
From reward alone, the policy acquires fleet-management judgment that single-period optimization cannot represent. It pulls satellites offline for maintenance at a sharp learned health threshold closely tracking the dynamic-programming optimum. It operates the battery through deep charge cycles matched to eclipse and demand rather than holding a static reserve. It triages a dynamic, oversubscribed tasking queue by priority, realizing a large, provably-bounded fraction of the theoretically achievable observation value. The policy also makes the central balance of constellation operations explicit: near-term task execution trades against fleet longevity through an operator-controllable dial. Evaluated against a deliberately strong baseline ladder, the learned policy approaches the task execution of an expertly hand-tuned composite policy while surpassing its fleet-health outcomes. While approaches that are task-execution-only are near-optimal when decisions are decoupled, they fail outright once degradation and energy couple those decisions across satellites and time. Multi-objective extensions maintain competing goals as separate reward dimensions. Doctrine-conditioned training then discovers a front of operationally distinct doctrines, from collection-maximizing to fleet-preserving, selectable by operators as mission priorities evolve._x000D_
_x000D_
_x000D_
Date of Conference: September 15-18, 2026
Track: AI for SDA Applications