Phillip Cunio, ExoAnalytic Solutions; Ben Corbin, IDA/STPI; Brien Flewelling, ExoAnalytic Solutions
Keywords: Data lakes, proliferated systems, pervasive space situational awareness, big data
Abstract:
As the importance of Space Situational Awareness (SSA) for both national security and commercial enterprise continues to increase, more attention will be paid to collecting SSA data by national, corporate, academic, and amateur actors, resulting in ever-growing volumes of data available. In addition, the ongoing development of proliferated, automated collection networks will create growth in the collection rate per network, generating overall an exponentially-expanding amount of available data.
This data volume will need to be transferred, stored, organized, and catalogued in order to be of maximal utility to stakeholders in the SSA domain. However, managing a large data volume can be challenging, particularly when the data are sourced from multiple providers with varying levels of capability and tasking criteria. This paper addresses a few of the challenges of dealing with large volumes of SSA data, by examining the inherent features of lakes (general open storage constructs) of data, including the effects of inadvertently incorporating low-quality or superfluously-duplicative data, and comparing them to the features of waterworks (pre-routed and extensively-labeled warehouse constructs) of data.
The differences between lakes and waterworks are examined in terms of estimated relative costs to instantiate, maintain, and operate; probability of being amenable to various classes of stakeholders in the SSA domain; and general deployment challenges. This paper also investigates the probable rate of growth in available SSA data, and assesses the scaling of data lakes and waterworks that may result if said available data is aggregated rather than being left partitioned and usable only by its original collectors. From this, a general assessment of the value of aggregated data is made, and recommendations for future data architecture guidelines are given such that a wave of SSA data will not accidentally swamp the capacity of future users to digest it and extract valuable meaning.
Date of Conference: September 11-14, 2018
Track: Poster