Fast, Ad Hoc Query Evaluations over Multidimensional Geospatial Datasets
Abstract: Networked observational devices and remote sensing equipment continue to proliferate and contribute to the accumulation of extreme extreme-scale scale datasets. Both the rate and resolution of the readings produced by these devices have grown over time, exacerbating the issues surrounding their storage and management. In many cases, the sheer scale of the information being maintained makes timely analysis infeasible due to the comp computational utational workloads required to process the data. While distributed solutions provide a scalable way to cope with data volumes, the communication and latency involved when inspecting large portions of an overall dataset limit applications that require freq frequent uent or rapid responses to incoming queries. This study investigates the challenges associated with providing approximate or exploratory answers to distributed queries. In many situations, this requires striking a balance between response times and error rates r to produce meaningful results. To enable these use cases, we outline several expressive query constructs and describe their implementation; rather than relying on summary tables or pre pre-computed computed samples, our solution involves a coarse-grained global index dex that maintains statistics and models the relationships across dimensions in the dataset. To illustrate the benefits of these techniques, we include performance benchmarks on a real real-world world dataset in a production environment.