Star Flow: Accelerating On-Line Processing of Astronomic Data on Heterogeneous Processors
Several new astronomy projects, including the Large Synoptic Survey Telescope (LSST) in the US, and the
Ground-based Wide-Angle Cameras (GWAC) in China, are building either a single, large telescope or an
array of smaller cameras to continuously observe a considerable portion of the sky. Once in operation,
these instruments will produce observation images every few seconds. Furthermore, complex processing
is required to catalog raw images online, and transient objects, such as stars that change positions
or brightness, should be detected from the data instantly to enable timely follow-up observation.
Consequently, accelerating the online processing of these data is crucial to meet the demands of
these projects and to facilitate astronomic discoveries.
There have been initial studies on utilizing commodity computer processors such as multicore CPUs
(Central Processing Units) and GPUs (Graphics Processing Units) to parallelize computation on
astronomical data. However, the pipeline processing of camera data during normal observations is
complex, involving image processing, astrometry computation, and database operations. As a result,
parallelizing this pipeline remains a challenging open problem, especially in a heterogeneous
environment with both multicore CPUs and GPUs. Moreover, recent developments in heterogeneous
processors, such as the Intel Phi co-processors for the CPU, and the dynamic scheduling capabilities
in NVIDIA Kepler GPUs to involve multiple CPU cores, pose interesting opportunities to parallel computing.
We propose Star Flow, an open-source software project that utilizes state-of-the-art many-core CPUs
with their co-processors as well as GPUs to parallelize the processing pipeline of astronomical observation
data. In Star Flow, we will tackle three major components in the processing pipeline: (1) source
extraction, (2) image subtraction, and (3) cross match. In source extraction, the tabular attributes of
hundreds of thousands of stars and galaxies are extracted from each observation image. Image subtraction
and cross match are two alternatives of finding transient objects with the difference being whether the
input are observation images or tabular data.
Xiaoying Jia, Qiong Luo:
Multi-Assignment Single Joins for Parallel Cross-Match of Astronomic Catalogs on Heterogeneous Clusters. SSDBM 2016: 12:1-12:12
- Xiaoying Jia, Qiong Luo, Dongwei Fan:
Cross-Matching Large Astronomical Catalogs on Heterogeneous Clusters. ICPADS 2015: 617-624
- Accelerating astronomical source extraction with graphics processors. Baoxue Zhao, MPhil Thesis, HKUST, Dec 2014.
- Parallelizing Astronomical
Source Extraction on the GPU. Baoxue Zhao, Qiong Luo, and Chao Wu.
IEEE 9th International Conference on e-Science, Beijing, China, Oct 2013, pp 88-97.
- Accelerating Astronomical
Image Subtraction on Heterogeneous Processors. Yan Zhao, Qiong Luo, Senhong
Wang, and Chao Wu. IEEE 9th International Conference on e-Science, Beijing, China, Oct 2013, pp 70-77.
- Accelerating In-Memory Cross Match of Astronomical Catalogs. Senhong Wang, Yan Zhao, Qiong Luo,
Chao Wu, and Yang Xv. IEEE 9th International Conference on e-Science, Beijing, China, Oct 2013, pp 326-333.
- GPU-SExtractor. The source code package of our GPU-based version of
the original SExtractor. Jan 2015.
- P-HOTPANTS. The source code package of our GPU and multicore-CPU
parallelized version of the original HOTPANTS
image subtraction tool by Andrew Becker. May 2014.
The license is a free non-exclusive, non-transferable license to reproduce, use, modify and display the source
code version of the Software, with or without modifications solely for non-commercial research, educational or
evaluation purposes. The license does not entitle Licensee to technical support, telephone assistance, enhancements
or updates to the Software. All rights, title to and ownership interest in Software, including all intellectual
property rights therein shall remain in HKUST.
We thank our collaborator the National Astronomical Observatories of China for providing us application
requirements, access to their data sets, and sharing their domain knowledge. Funding for this project
is provided by grants 616012, 617509, and 16206414 from the Hong Kong Research Grants Council and MRA11EG01 from Microsoft
SQL Server China R&D.