Championing reproducible science

BY THE SKAO SCIENCE TEAM

The SKAO Science team has announced the results of the Science Data Challenge 2 (SDC2) reproducibility awards. The SKAO Science Data Challenges are designed to prepare the astronomical community – and the SKAO – for the novel, yet challenging, nature of SKA data.

SDC2 saw over 100 worldwide participants tackle a 1 TB dataset to find and measure galaxies in a simulation of a deep SKA-Mid neutral hydrogen survey. The challenge demonstrated the value of complementary methods and diverse collaborations, and full findings from the challenge will be published in a paper soon.

The SDC2 reproducibility awards sat alongside the main challenge, the results of which were announced in October 2021. Developed with the help of the Software Sustainability Institute (SSI), awards were made to all teams whose pipelines demonstrated best practice in the provision of reproducible results and reusable methods.

“Reproducibility leads to better, more efficient science,” says Dr Philippa Hartley, SKAO Scientist and co-lead of SDC2. “When we talk about reusable methods, we mean creating software that can be adapted by others, allowing previous work to be built upon for the future: that’s a key feature of Open Science.”

Six teams took part in the reproducibility part of the challenge, each submitting their analysis pipeline to an expert panel for evaluation against a set of pre-defined criteria

The team HI-FRIENDS, comprising members across eight institutions in six countries, received the top award for providing an excellent gold-standard solution containing many examples of best practice.

“Reproducibility is a fundamental principle of the scientific method and the best practices defined by Open Science, such as having precise documentation and metadata, are our ally to enable our results to be reproduced,” says Dr Lourdes Verdes-Montenegro, researcher at the Instituto de Astrofísica de Andalucía (IAA) and co-leader of the HI-FRIENDS team.

“Pipelines designed with reusability and reproducibility in their core can be used and improved by the community so we can all benefit from prior work and be better equipped to do great science with the SKAO,” says Javier Moldon, Operations Scientist of the SKA Regional Centre (SRC) Spanish prototype and HI-FRIENDS co-leader.

“We appreciate that preparing analysis software for sharing publicly can take time, and we very much appreciate the involvement of the teams in this aspect of the challenge,” says Dr Hartley. “The challenge results highlight the value of community-built software and underscore the need for that software to receive the funding to be well-maintained.”

The reproducibility awards will be included and discussed in the upcoming SDC2 results paper, and the SKAO Science team welcomes feedback on how best to support collaborative science through reproducibility best practice.

The SKAO is now gearing up for the start of SDC3, which focuses on peering into the so-called Epoch of Reionisation: a time in the very early Universe when the very first stars were forming. Registration has now closed, with more than 230 participants worldwide having signed up.

Stay tuned for updates when the challenge begins.