The SC20 Transparency and Reproducibility Initiative undertook a survey in August of this year to study perceptions within the community of the measures implemented by SC to make results from the Technical Program and Workshops more transparent and reproducible. SC’s measures for increased reproducibility take the form of additional information collected at the time of submission of papers to the Technical Program about the experimental activity that supports the results in the manuscript. This information is submitted as an appendix to the manuscript and published as an appendix to the final published version in the SC Conference Proceedings.

The study is timely, as the reproducibility initiative has been in effect since SC15, when authors of SC15 papers were invited to submit an Artifacts Description (AD) Appendix after the conference.

While analysis of the data is ongoing, early findings speak to the strength and value of the sustained effort towards transparency and reproducibility at SC.

After six years of the initiative, a reassuring result of the survey is that a full 90% of the community respondents are aware of the issues around reproducibility. Only 15% think the concerns about reproducibility in science are exaggerated, and only 7% think that the concerns about reproducibility in science do not apply to computer and computational science.

An important impact of the SC reproducibility initiative is whether the reproducibility measures result in published papers having greater impact. The results suggest this to be true. Thirty five percent (35%) of the respondents have used the appendices information from papers in the SC Conference Proceedings in their research.

Food for thought for the future of the SC reproducibility effort: 75% of respondents strongly agreed or somewhat agreed with the statement that transparency (and not reproducibility) is the goal of the SC AD/AE appendices. This result is somewhat surprising and suggests that the visibility that has come through the additional appendix information is overwhelmingly seen as a benefit of the SC reproducibility measures.

The survey was open August 17–31, 2020 and had 204 survey respondents. A manuscript is in preparation that discusses the study, analysis, and detailed conclusions. The study was conducted under Indiana University protocol #2005780098, “Assessing Reproducibility Initiative of IEEE/ACM SC Conference,” led by Beth Plale and Line Pouchard.

Contact the Transparency and Reproducibility Committee for more details.

Beth Plale, SC20 Transparency and Reproducibility Initiative Chair

Line Pouchard, SC20 Transparency and Reproducibility Initiative Evaluation Chair

SC is an unquestioned pioneer in improving scientific rigor of the works published in its conference proceedings. Over the past 5 years, the SC reproducibility initiative has strived to enhance reproducibility, offering appendices through which the community is given additional detail (beyond what the typical paper structure allows) to build upon a research result. This post is intended for authors and the general community, to call your attention to a few new developments. Reproducibility is a work in progress, and responsiveness to community needs is an ongoing priority.

Beth Plale
SC20 Transparency and Reproducibility Initiative Chair

Q: The name of the initiative is now the “Transparency and Reproducibility Initiative”, how does that affect authors?

A: The new naming is recognition of the growing prevalence of AI-based innovation in HPC, and the awareness that the established norms in HPC paper structuring often impede key decisions. For instance, research results should include evidence for why one algorithm worked and others do not, or why one software architecture is chosen over another. Also, ad hoc methods (e.g., to tune an AIs’ “learning rates”—how much an algorithm corrects itself after each mistake) are sometimes used without justification for why one ad hoc method is better than others.

The call for participation for SC20 has been edited to address the need for greater transparency in submitted manuscripts. Papers making fundamental, non-performance related research contributions to algorithms, or their application within a software architecture, should address in the body of their manuscript the questions of transparency raised above. If additional resources need to be included, they should be linked to in the AD appendix.

Q: How are the Appendices (AD/AE) reviewed?

A: Since its inception the initiative has striven for appendices that improve scientific rigor. Thus, and because appendix information will frequently reveal the identity of the author, the SC AD/AE Appendices Committee (SC AC) is independent of the SC Papers Committee (SC PC). The SC AC works under a double-open arrangement where authors and committee members are known to each other, and the SC AC team is poised to help authors improve their appendices.

SC20 further clarifies that the results of SC AC review of appendices is in the form of recommendations to the SC PC chairs, and that these recommendations are to be provided to the SC PC at each of the multiple points of SC PC review of submitted manuscripts.

Q: Any other important information to share?

A: We’re excited about a new initiative that we are working on this year to take the temperature of the community. SC has nurtured the reproducibility activity over the last 5 years, and the community has had this long to not only respond as authors, but to benefit from the potential the initiative provides for new discovery. The transparency and reproducibility team is building a community survey that should be coming out soon. We are also looking forward to supporting the student reproducibility challenge as in years past. Consult with the SC20 Transparency and Reproducibility Leadership Team, members listed below, and stay tuned!

SC20 Transparency and Reproducibility Leadership Team:

• Beth Plale, Chair, Indiana University
• Stephen Harrell, Purdue University
• John Linford, ARM
• Line Pouchard, Brookhaven National Lab
• Andrew Younge, Sandia National Lab

More Information:

• Transparency and Reproducibility Initiative
• AD/AE Appendix FAQ & Author Kit

—

Rebecca Hartman-Baker
SC20 Communications Team