The SC20 Transparency and Reproducibility Initiative undertook a survey in August of this year to study perceptions within the community of the measures implemented by SC to make results from the Technical Program and Workshops more transparent and reproducible. SC’s measures for increased reproducibility take the form of additional information collected at the time of submission of papers to the Technical Program about the experimental activity that supports the results in the manuscript. This information is submitted as an appendix to the manuscript and published as an appendix to the final published version in the SC Conference Proceedings.

The study is timely, as the reproducibility initiative has been in effect since SC15, when authors of SC15 papers were invited to submit an Artifacts Description (AD) Appendix after the conference.

While analysis of the data is ongoing, early findings speak to the strength and value of the sustained effort towards transparency and reproducibility at SC.

After six years of the initiative, a reassuring result of the survey is that a full 90% of the community respondents are aware of the issues around reproducibility. Only 15% think the concerns about reproducibility in science are exaggerated, and only 7% think that the concerns about reproducibility in science do not apply to computer and computational science.

An important impact of the SC reproducibility initiative is whether the reproducibility measures result in published papers having greater impact. The results suggest this to be true. Thirty five percent (35%) of the respondents have used the appendices information from papers in the SC Conference Proceedings in their research.

Food for thought for the future of the SC reproducibility effort: 75% of respondents strongly agreed or somewhat agreed with the statement that transparency (and not reproducibility) is the goal of the SC AD/AE appendices. This result is somewhat surprising and suggests that the visibility that has come through the additional appendix information is overwhelmingly seen as a benefit of the SC reproducibility measures.

The survey was open August 17–31, 2020 and had 204 survey respondents. A manuscript is in preparation that discusses the study, analysis, and detailed conclusions. The study was conducted under Indiana University protocol #2005780098, “Assessing Reproducibility Initiative of IEEE/ACM SC Conference,” led by Beth Plale and Line Pouchard.

Contact the Transparency and Reproducibility Committee for more details.

Beth Plale, SC20 Transparency and Reproducibility Initiative Chair

Line Pouchard, SC20 Transparency and Reproducibility Initiative Evaluation Chair

SC is an unquestioned pioneer in improving scientific rigor of the works published in its conference proceedings. Over the past 5 years, the SC reproducibility initiative has strived to enhance reproducibility, offering appendices through which the community is given additional detail (beyond what the typical paper structure allows) to build upon a research result. This post is intended for authors and the general community, to call your attention to a few new developments. Reproducibility is a work in progress, and responsiveness to community needs is an ongoing priority.

Beth Plale
SC20 Transparency and Reproducibility Initiative Chair

Q: The name of the initiative is now the “Transparency and Reproducibility Initiative”, how does that affect authors?

A: The new naming is recognition of the growing prevalence of AI-based innovation in HPC, and the awareness that the established norms in HPC paper structuring often impede key decisions. For instance, research results should include evidence for why one algorithm worked and others do not, or why one software architecture is chosen over another. Also, ad hoc methods (e.g., to tune an AIs’ “learning rates”—how much an algorithm corrects itself after each mistake) are sometimes used without justification for why one ad hoc method is better than others.

The call for participation for SC20 has been edited to address the need for greater transparency in submitted manuscripts. Papers making fundamental, non-performance related research contributions to algorithms, or their application within a software architecture, should address in the body of their manuscript the questions of transparency raised above. If additional resources need to be included, they should be linked to in the AD appendix.

Q: How are the Appendices (AD/AE) reviewed?

A: Since its inception the initiative has striven for appendices that improve scientific rigor. Thus, and because appendix information will frequently reveal the identity of the author, the SC AD/AE Appendices Committee (SC AC) is independent of the SC Papers Committee (SC PC). The SC AC works under a double-open arrangement where authors and committee members are known to each other, and the SC AC team is poised to help authors improve their appendices.

SC20 further clarifies that the results of SC AC review of appendices is in the form of recommendations to the SC PC chairs, and that these recommendations are to be provided to the SC PC at each of the multiple points of SC PC review of submitted manuscripts.

Q: Any other important information to share?

A: We’re excited about a new initiative that we are working on this year to take the temperature of the community. SC has nurtured the reproducibility activity over the last 5 years, and the community has had this long to not only respond as authors, but to benefit from the potential the initiative provides for new discovery. The transparency and reproducibility team is building a community survey that should be coming out soon. We are also looking forward to supporting the student reproducibility challenge as in years past. Consult with the SC20 Transparency and Reproducibility Leadership Team, members listed below, and stay tuned!

SC20 Transparency and Reproducibility Leadership Team:

• Beth Plale, Chair, Indiana University
• Stephen Harrell, Purdue University
• John Linford, ARM
• Line Pouchard, Brookhaven National Lab
• Andrew Younge, Sandia National Lab

More Information:

• Transparency and Reproducibility Initiative
• AD/AE Appendix FAQ & Author Kit

—

Rebecca Hartman-Baker
SC20 Communications Team

Meet State of the Practice Chair Brett M. Bode and get ready to submit your work to this new venue at SC20. Submissions for State of the Practice Talks open February 17, 2020.

Brett M. Bode
State of the Practice Chair

Brett is an Assistant Director at the National Center for Supercomputing Applications (NCSA) and a co-Principal Investigator on the Blue Waters project. Prior to the current operational role with Blue Waters, Brett was the lead of the Advanced Digital Services division at NCSA, and managed the system software development projects for the Blue Waters project. Before joining NCSA in 2008, Brett spent nearly ten years as a scientist with the Department of Energy’s Ames Laboratory at Iowa State University, researching ways to deliver better performance to scientific applications from HPC clusters. Brett holds a BS in chemistry and physics from Illinois State University and a PhD in physical chemistry from Iowa State University.

Why focus on state of the practice at SC20?

BB: The State of the Practice theme spans the entire range of SC topics and is already a sub-area under Papers, accounting for three papers at SC19. However, there are many more potential state of the practice advances than can be accommodated under the currently available offerings.

The goal of State of the Practice Talks is to provide a venue within the SC technical program for reporting of advances and best practices of interest to the SC community. State of the Practice Talks allow for submissions of late-developing advances and improvements that do not necessarily meet the high bar of a full SC paper.

What topical areas will State of the Practice Talks cover, and who should submit?

BB: Submissions are encouraged from any area of interest to the SC attendee. Possible topics include (but are not limited to) systems, storage and facility management, HPC project management, networking and cybersecurity, and HPC training and education. We expect submissions particularly from operations-focused staff such as system administrators, operations managers, cybersecurity specialists, and training and outreach coordinators.

State of the Practice Talks, like Papers, require an Artifact Description. What kind of artifacts do you foresee from submissions to this area?

BB: The State of the Practice Talks will include the extended abstract, presentation, and optionally additional artifacts in the digital library. The artifacts will be as varied as the topical areas, and may range from scripts and code snippets to risk management templates to sample materials for demonstrating best practices in education. We encourage the submission of artifacts that, combined with the abstract and presentation, make the submission complete and reproducible.

—

Rebecca Hartman-Baker
SC20 Communications Team