James et al. (2014) — Standing Together for Reproducibility in Large-Scale Computing#
A collaborative workshop report from XSEDE14 on reproducibility challenges in supercomputing — connecting to the Evolvix BEST Names vision of semantic reproducibility.
Download the original document (PDF)
James et al. (2014) — Reproducibility@XSEDE — PDF (492 KB) — 16 pages, Jonah License with CC0 Public Domain
Filename: james-2014-study-reproducibility-challenges-in-supercomputing-xsede-final-report-16page.pdf
— Overview AI-generated by dv_ClaOp46_ExhB_2026m04d14 —Start—
Abstract#
This is the final report of a one-day workshop held at XSEDE14 (Atlanta, GA, 2014m07d14) focused on reproducibility in large-scale computational research. The report represents the collaborative output of 35 authors including 12 position papers presented at the workshop.
Key themes and recommendations include: (1) supercomputer centers should actively promote reproducible research as part of their mission, (2) researchers should conduct research as though someone will replicate it, (3) documentation and training are essential for reproducibility, (4) system-level tools should capture build-time and run-time environment information automatically, (5) best practices for collaborations should be established and shared, and (6) science gateways can serve as platforms for reproducible workflows.
The report was finalized on 2014m12d17 and represents a community consensus on the state of reproducibility challenges in high-performance computing.
Broader Significance (Claude’s Assessment)#
This workshop report connects to LLoL’s broader scientific vision:
Community engagement on reproducibility. LLoL participated in this 35-author workshop, contributing to the community discussion on computational reproducibility at a time when the “reproducibility crisis” was becoming a major concern across scientific disciplines.
Direct connection to Evolvix BEST Names. The reproducibility challenges identified in this report — naming ambiguity, environment documentation, semantic drift — are precisely the problems that the Evolvix BEST Names concept (published 2017) was designed to address. The workshop experience likely informed the BEST Names approach.
ResearchCity vision. The report’s recommendations for system-level tools, science gateways, and reproducible workflows align with the ResearchCity vision of infrastructure that makes reproducible science the default rather than the exception.
Collaborative scientific culture. The workshop format — 35 authors, 12 position papers, community consensus report — exemplifies the collaborative approach to science that LLoL advocates through the epiocracy concept and the Good News Pack’s cooperative framework.
Who This Document Is For#
Audience |
Why This Document Matters |
|---|---|
Computational scientists |
Practical recommendations for making large-scale computational research reproducible, from documentation practices to system-level tooling. |
HPC center administrators |
Guidance on how supercomputer centers can promote reproducibility as part of their mission, including specific tool and policy recommendations. |
Science policy makers |
Community consensus on the state of reproducibility in computational science circa 2014, with actionable recommendations for funding agencies and institutions. |
Reviewers of LLoL’s scientific credentials |
Documents LLoL’s participation in the broader scientific community’s efforts to address reproducibility, connecting to the later Evolvix BEST Names work. |
Key Concepts at a Glance#
Reproducibility |
The ability for others to independently replicate computational results; the central challenge addressed by the workshop |
XSEDE |
Extreme Science and Engineering Discovery Environment — the US national cyberinfrastructure for scientific computing |
Science gateways |
Web-based platforms that provide reproducible access to computational tools and workflows |
Build/run-time capture |
System-level tools that automatically record the software environment used to produce computational results |
Position papers |
12 contributed papers from workshop participants presenting different perspectives on reproducibility challenges |
Cyberinfrastructure |
The shared computing, data, and networking resources that support scientific research at scale |
Document Information#
Document ID |
Key Paper 16 (Dusty Deep Data, loewe-researchcity-key-papers/) |
Full title |
Standing Together for Reproducibility in Large-Scale Computing: Report on reproducibility@XSEDE |
Principal Editors |
Doug James, Nancy Wilkins-Diehr, Victoria Stodden, Dirk Colbry, Carlos Rosales |
Type |
Workshop report (collaborative, 35 authors, 12 position papers) |
Venue |
XSEDE14 Workshop, Atlanta, GA (2014m07d14) |
Finalized |
2014m12d17 |
Pages |
16 |
License |
Jonah License with CC0 Public Domain |
Part of |
Good News Pack MMv3, Dusty Deep Data / Key Papers collection |
PDF size |
492 KB |
WebP size |
180 KB |
Related documents in the Good News Pack:
Loewe et al. (2017) — Evolvix BEST Names (semantic reproducibility via naming conventions)
— Overview AI-generated by dv_ClaOp46_ExhB_2026m04d14 —End—