Edward Morgan, Raft, LLC; Steve Nino, Raft, LLC; Angela Sheffield, Raft, LLC
Keywords: SDA, Cislunar, Kubernetes, Scalable, Data Fabric, Data Transformation, Space Data, Space Data Awareness, Data Mesh
Abstract:
Raft, a digital consultancy focusing on data management problems in the Department of Defense, has partnered with the Department of Air Force Command, Control, Communications and Battle Management (C3BM) Program Element Office (PEO) (formerly the DAF Chief Architects Office), through a Small Business Innovation (SBIR) Phase 3 contract, to provide a Data Fabric solution meeting the needs of Space Domain Awareness (SDA) operators and analysts.
As part of its work to architect and implement the DAF Data Fabric, Raft is working with the USSF 18th and 19th Space Defense Squadrons (SDS) to use the DAF Data Fabric to measurably improve how data is ingested, transformed, and disseminated to operational users.
Currently, for some of their orbital object observational data, space defense operators receive the collection data via email, receiving data files in multiple formats as attachments at a routine interval. Then, they manually ascertain what kind of data was sent before manually converting the single data file into a common output format. Next, they manually transcribe the transformed data to a location where it is retrieved by an intermediary and finally to the operational users. This is a time-consuming process that takes valuable time away from more pressing space domain awareness (SDA) and space situational awareness missions. Further, this process presents a challenge to operations in the strategically relevant area of cislunar space. Data from cislunar space does not conform to the historical SDA data formats; as a result, cislunar data cannot be processed and disseminated according to standard processes.
Key requirements and research goals included: automatic ingest, transformation, and upload of files; scalability in both the number and types of observations processed; automated inference of file type and format to reduce burden on users; capability to handle both discrete uploads and continuous, real-time streaming workloads.
Over the past year, Raft conducted end user research to determine project requirements and executed research and development to apply the DAF Data Fabric to the 19th operational mission. As a result of this effort, the 19th no longer relies on emailing SDA data files, instead uploading them to the Data Fabric via a browser or utilizing it with streaming data repositories like the Unified Data Library (UDL) to automatically ingest and upload data.
After developing the initial capability for Space Fence observational data provided in XML format, Raft has deployed the Data Fabric in an operational capacity on the VAULT data platform utilizing Platform Ones BigBang architecture on top of Kubernetes. Because of Data Fabrics modular, microservices-based architecture, additional processing capabilities can be added simply by developing the appropriate data processing code; no additional infrastructure needs to be created. In this way, new processing capabilities for other types of LEO, MEO, GEO, and Cislunar data can be developed and operationalized as soon as they are needed. Particularly for Cislunar observations, as efforts like Project Rocket develop and mature, the Raft Data Fabric can mirror those changes by being adaptable and dynamic. While research is still ongoing, data types and formats may change quickly and pipelines for streaming real-time data may not exist. Rafts Data Fabric provides for these use cases by supporting ad-hoc file
uploads for timely ingestion and processing via a web UI. As observational sources are integrated into data lakes like the UDL, Data Fabrics existing infrastructure allows it to connect out-of-the-box and apply the same processing functions to data streams in real time. In this way, Data Fabric has been developed to meet data providers where they are and in whatever stage of development, they are to provide actionable results as soon as possible.
Date of Conference: September 19-22, 2023
Track: Space Domain Awareness