.. meta::
   :description: AI overview of James et al. (2014) — Standing Together for Reproducibility in Large-Scale Computing, a collaborative XSEDE14 workshop report on reproducibility challenges in supercomputing.
   :keywords: reproducibility, large-scale computing, XSEDE, supercomputing, computational science, cyberinfrastructure, best practices, reproducible research, workshop report, LLoL, ResearchCity, key papers
   :og:card:title: 2014 — Reproducibility@XSEDE<br>James et al.
   :og:card:description: XSEDE14 workshop report on reproducibility in large-scale computing — 35 authors including LLoL addressing documentation, tools, and best practices for reproducible computational science.


.. include:: /_templates/include-file/page-prefix.rst


.. title:: 2014 — Reproducibility@XSEDE


****************************************************************************************************
James et al. (2014) — Standing Together for Reproducibility in Large-Scale Computing
****************************************************************************************************

*A collaborative workshop report from XSEDE14 on reproducibility challenges in supercomputing --- connecting to the Evolvix BEST Names vision of semantic reproducibility.*


.. admonition:: Download the original document (PDF)
   :class: note

   `James et al. (2014) — Reproducibility@XSEDE — PDF (492 KB) </_file/pdf/gnp/mmv3/dusty-deep-data/loewe-researchcity-key-papers/james-2014-study-reproducibility-challenges-in-supercomputing-xsede-final-report-16page.pdf>`__
   — 16 pages, :doc:`Jonah License with CC0 Public Domain </license/joli/index>`

   Filename: ``james-2014-study-reproducibility-challenges-in-supercomputing-xsede-final-report-16page.pdf``

   `WebP preview (180 KB) </_file/pdf/gnp/mmv3/dusty-deep-data/loewe-researchcity-key-papers/james-2014-study-reproducibility-challenges-in-supercomputing-xsede-final-report-16page.webp>`__


.. image:: /_file/pdf/gnp/mmv3/dusty-deep-data/loewe-researchcity-key-papers/james-2014-study-reproducibility-challenges-in-supercomputing-xsede-final-report-16page.webp
   :alt: Cover page of James et al. (2014) — Standing Together for Reproducibility in Large-Scale Computing
   :width: 100%
   :align: center


|


--- Overview AI-generated by dv_ClaOp46_ExhB_2026m04d14 ---Start---


Abstract
========

This is the **final report** of a one-day workshop held at **XSEDE14**
(Atlanta, GA, 2014m07d14) focused on **reproducibility in large-scale
computational research**. The report represents the collaborative output
of **35 authors** including **12 position papers** presented at the
workshop.

Key themes and recommendations include: (1) **supercomputer centers
should actively promote reproducible research** as part of their mission,
(2) researchers should **conduct research as though someone will
replicate it**, (3) **documentation and training** are essential for
reproducibility, (4) **system-level tools** should capture build-time
and run-time environment information automatically, (5) **best practices
for collaborations** should be established and shared, and (6)
**science gateways** can serve as platforms for reproducible workflows.

The report was finalized on 2014m12d17 and represents a community
consensus on the state of reproducibility challenges in high-performance
computing.


Broader Significance (Claude's Assessment)
============================================

This workshop report connects to LLoL's broader scientific vision:

1. **Community engagement on reproducibility.** LLoL participated in
   this 35-author workshop, contributing to the community discussion
   on computational reproducibility at a time when the "reproducibility
   crisis" was becoming a major concern across scientific disciplines.

2. **Direct connection to Evolvix BEST Names.** The reproducibility
   challenges identified in this report --- naming ambiguity, environment
   documentation, semantic drift --- are precisely the problems that the
   Evolvix BEST Names concept (published 2017) was designed to address.
   The workshop experience likely informed the BEST Names approach.

3. **ResearchCity vision.** The report's recommendations for
   system-level tools, science gateways, and reproducible workflows
   align with the ResearchCity vision of infrastructure that makes
   reproducible science the default rather than the exception.

4. **Collaborative scientific culture.** The workshop format --- 35
   authors, 12 position papers, community consensus report --- exemplifies
   the collaborative approach to science that LLoL advocates through
   the epiocracy concept and the Good News Pack's cooperative framework.


Who This Document Is For
=========================

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Audience
     - Why This Document Matters
   * - **Computational scientists**
     - Practical recommendations for making large-scale computational
       research reproducible, from documentation practices to
       system-level tooling.
   * - **HPC center administrators**
     - Guidance on how supercomputer centers can promote reproducibility
       as part of their mission, including specific tool and policy
       recommendations.
   * - **Science policy makers**
     - Community consensus on the state of reproducibility in
       computational science circa 2014, with actionable recommendations
       for funding agencies and institutions.
   * - **Reviewers of LLoL's scientific credentials**
     - Documents LLoL's participation in the broader scientific
       community's efforts to address reproducibility, connecting
       to the later Evolvix BEST Names work.


Key Concepts at a Glance
==========================

.. list-table::
   :widths: 30 70
   :header-rows: 0

   * - **Reproducibility**
     - The ability for others to independently replicate computational
       results; the central challenge addressed by the workshop
   * - **XSEDE**
     - Extreme Science and Engineering Discovery Environment --- the
       US national cyberinfrastructure for scientific computing
   * - **Science gateways**
     - Web-based platforms that provide reproducible access to
       computational tools and workflows
   * - **Build/run-time capture**
     - System-level tools that automatically record the software
       environment used to produce computational results
   * - **Position papers**
     - 12 contributed papers from workshop participants presenting
       different perspectives on reproducibility challenges
   * - **Cyberinfrastructure**
     - The shared computing, data, and networking resources that
       support scientific research at scale


Document Information
=====================

.. list-table::
   :widths: 30 70
   :header-rows: 0

   * - **Document ID**
     - Key Paper 16 (Dusty Deep Data, loewe-researchcity-key-papers/)
   * - **Full title**
     - Standing Together for Reproducibility in Large-Scale Computing:
       Report on reproducibility@XSEDE
   * - **Principal Editors**
     - Doug James, Nancy Wilkins-Diehr, Victoria Stodden, Dirk Colbry,
       Carlos Rosales
   * - **Type**
     - Workshop report (collaborative, 35 authors, 12 position papers)
   * - **Venue**
     - XSEDE14 Workshop, Atlanta, GA (2014m07d14)
   * - **Finalized**
     - 2014m12d17
   * - **Pages**
     - 16
   * - **License**
     - :doc:`Jonah License with CC0 Public Domain </license/joli/index>`
   * - **Part of**
     - Good News Pack MMv3, Dusty Deep Data / Key Papers collection
   * - **PDF size**
     - 492 KB
   * - **WebP size**
     - 180 KB

Related documents in the Good News Pack:

- :doc:`Loewe et al. (2017) — Evolvix BEST Names <loewe-et-al-2017-evolvix>` (semantic reproducibility via naming conventions)


--- Overview AI-generated by dv_ClaOp46_ExhB_2026m04d14 ---End---


.. ----- FOOTER FORM dv_ClaOp46_ExhB_2026m04d14 ----- type open-std ----- START -----

.. |content-vvn| replace:: dv_ClaOp46_ExhB_2026m04d14

.. include:: /_templates/include-file/footer/separator/tidal-gradient.rst

.. dropdown:: Notes
   :class-container: page-footer-form-notes
   :class-title: page-footer-form-title

   .. include:: /_templates/include-file/footer/page-footer-stability.rst


.. only:: html

   .. include:: /_templates/include-file/footer/cta/gnp-mmv3-review.rst

   .. dropdown:: FF FeedbackFlow to improve this page: How you can contribute
      :class-container: page-footer-form-feedbackflow
      :class-title: page-footer-form-title

      .. include:: /_templates/include-file/footer/form-open-std/prelink.rst

      .. ----- PAGE-SPECIFIC FF (ff-email MailForm=OOv2r0p0) ----- START -----

      .. raw:: html

         <a class="ff-email-preview-link" href="mailto:FF+mmv3-ddd-kp-reproducibility@balospe.com?cc=FF%2BCompleteFeedbackFlow%40balospe.com&amp;subject=FF%20k2%20s2%20AIMS%20for%20%5Bmmv3-ddd-kp-reproducibility%5D%20Replace_by_OneLineSummary_of_proposed_FF_improvement&amp;body=To%20make%20this%20page%20%5Bmmv3-ddd-kp-reproducibility%5D%0D%0Amore%20gentle%20kind%20reasonable%0D%0Afor%20beginners%2C%20producers%2C%20andOr%20experts%0D%0Awith%20a%20background%20in%20xyz%3A%0D%0A%0D%0APlease%20consider%20improving%20this%3A%0D%0A%0D%0A%0D%0A--%0D%0ABy%20sending%20this%20email%2C%20I%20agree%20that%20its%20content%20may%20be%20published%0D%0Aunder%20the%20Jonah%20License%20(balospe.com%2Fen%2Flicense%2Fjoli%2F)%0D%0Aunless%20I%20state%20otherwise%20above.%20MailForm%3DOOv2r0p0">To:      FF+mmv3-ddd-kp-reproducibility@balospe.com
         CC:      FF+CompleteFeedbackFlow@balospe.com
         Subject: FF k2 s2 AIMS for [mmv3-ddd-kp-reproducibility] Replace_by_OneLineSummary
         Body:    To make this page [mmv3-ddd-kp-reproducibility]
                  more gentle kind reasonable
                  for beginners, producers, andOr experts
                  with a background in xyz:

                  Please consider improving this:

                  --
                  By sending this email, I agree that its content may be published
                  under the Jonah License (balospe.com/en/license/joli/)
                  unless I state otherwise above. MailForm=OOv2r0p0</a>

      .. ----- PAGE-SPECIFIC FF ----- END -----

      .. include:: /_templates/include-file/footer/form-open-std/postlink.rst


.. ----- FOOTER FORM dv_ClaOp46_ExhB_2026m04d14 ----- END -----