Reproduction Challenge
Important Dates
| Paper Abstracts Due | Monday 10 June 2024 |
| Papers (all materials) Due | Monday 17 June 2024 |
| Posters Due | Monday 1 July 2024 |
| Author Notification for Papers | Monday 5 August 2024 |
| Revised Papers Due | Monday 12 August 2024 |
| Notification for Papers and Posters | Friday 16 August 2024 |
| Camera-Ready Papers and Posters Due | Monday 19 August 2024 |
| Conference | Wednesday 9 – Friday 11 October 2024 |
Overview
In the pursuit of robust scientific knowledge, the importance of replication cannot be overstated. Reproducing research findings is not only a fundamental step in the scientific method but also a cornerstone for building a trustworthy knowledge base. Despite this, replication efforts are often limited, primarily due to the lack of incentives in our highly competitive scientific landscape.
In response to this challenge, we are excited to introduce the first “VRST 2024 Reproduction Challenge,” aiming to promote and recognize the crucial role of replication in scientific research in XR. Submissions will be accepted both in the form of replication papers, in which published XR work is replicated and discussed, and replication posters, in which considerations and designs for future replication studies are shared and discussed with the XR research community. Outstanding replication papers and posters will be recognized with prestigious awards at the conference.
Accepted design abstracts will receive an invitation from the Diamond open-access Journal of Visualization and Interaction (JOVI) to submit a full paper based on their proposed study design for a special issue. We will strive to maintain reviewer continuity throughout the process. Contributions can be submitted via two different tracks: either directly as an article or through a two-stage process (registered report), consisting of initial preregistration followed by a full paper submission based on it. The design abstract could be extended for this purpose based on the discussion at the conference. Additionally, the journal offers an experimental track where interactive submissions, such as those created in HTML, can be made. These latter submissions can also be included in the special issue.
Submission Formats
We invite authors to submit their replication research in the following two formats:
(1) Replication Papers: Papers can be submitted to the Reproduction Challenge if they make contributions to the reproducibility of published research in XR. As there is a lack of consensus in the scientific literature on a definition of replication research, here are a few examples of acceptable forms of submissions:
- Full replication studies may comprehensively replicate and validate published research in XR, e.g., with a larger or more diverse sample size, including detailed methodologies, results, and comparisons with the original work.
- Partial replication studies may replicate and validate a significant portion of the published research in XR, e.g., when details in the original publication were insufficient to fully replicate the work, or in situations where parts of the original research are considered unimportant from today’s perspective.
- Replication-extension studies may either fully or partially replicate published research in XR but with the extension of new conditions that were not part of the original research, e.g., by evaluating additional factors or new variants of the original methods.
- Methodology or review papers may focus on the methodology or scholarship of replication, discussing best practices, challenges encountered, lessons learned, or how replication has been applied.
(2) Replication Posters: Posters submitted to the Reproduction Challenge afford researchers the opportunity to contribute to replication research without the requirement to perform a full study or present mature research. Examples of acceptable types of submissions include:
- Identify or highlight studies needing replication in a particular field of XR. A well-structured argument for the demand for replication research in a particular field of XR will support the community in identifying foundational work that needs to be revisited, e.g., if the original studies were run with dated hardware and software or involved small sample sizes or narrow participant demographics.
- Describe a specific replication study design. Highlighting the reasons for replicating a particular study and the authors’ plans for their replication study will afford the community insights into ongoing replication efforts and allow the authors to gain feedback on their replication design from the community, refine their study plans, or find new collaborators interested in the same topic.
- Present and discuss general methodologies or insights that may be valuable for XR researchers in the pursuit of designing, performing, or analyzing replication studies, or generally improving the replicability of their original research.
Note on authorship of replication papers and posters: Researchers working on a replication study are free to contact and collaborate with the authors of the original publication, especially when crucial information for a successful reimplementation and reproduction is missing in the publication. Further, we impose no limitations on authorship that would prevent researchers from revisiting their own prior research.
We look forward to receiving your contributions to the Reproduction Challenge and the opportunity to strengthen the scientific knowledge base in the field of XR!
Submission Instructions
Please use the Precision Conferences System (PCS) to submit your work. After clicking the link below, please choose “SIGCHI” from the “Society” drop-down list, then “VRST 2024” from the “Conference/Journal” drop-down list, and choose the corresponding “Papers” or “Posters” track for your submission.
https://new.precisionconference.com/vrst2024
- Replication Papers should be submitted as a “Paper” following the submission details and due dates in the main Call for Submissions. Note that the submission form in PCS includes the flag “Reproduction Challenge” that authors should check to indicate that their submission is a replication paper.
- Replication Posters should be submitted as a “Poster” following the submission details and due dates in the main Call for Submissions. The submission form in PCS includes the flag “Reproduction Challenge” that authors should check to indicate that their submission is a replication poster.
Review Process
Submissions to the Reproduction Challenge will undergo a peer review process in line with that of regular conference papers and posters. However, reviewers will be cognizant of these particular submission types and receive detailed instructions on how to review replication papers. Each submission will be evaluated based on its quality, adherence to replication standards, and contributions to the field.
Replication Awards
Outstanding submissions to the Reproduction Challenge will be recognized with awards for the “Best Replication Paper” and the “Best Replication Poster.” Awards will be decided by an awards committee and presented during an awards ceremony at the conference.
Frequently Asked Questions (FAQs)
Why should I perform a replication study?
Reproducing research findings is an often overlooked but fundamental step in the scientific method to ensure that the knowledge base in our discipline is valid, trustworthy, and up to date. In particular, in a fast-moving field like XR, where we have seen tremendous advances in technologies over the last years, e.g., related to fields of view, tracking latency, ergonomics, and rendering quality, it is imperative for our research community to regularly fact check and update the support for some of the accepted notions in our field to ensure we have a strong scientific foundation to build upon.
Why should I submit my “Replication Paper” to the VRST 2024 Reproduction Challenge instead of some other venue?Anecdotally, we have seen that replication papers submitted to the regular papers tracks at the premier XR conferences and journals are often rejected due to reviewers and program committee members valuing the novelty of first-time studies higher than the contributions made by papers that replicate prior studies. Submissions to the Reproduction Challenge will be reviewed by cognizant reviewers and program committee members with detailed instructions to ensure that this review bias is reduced.
Is a submission to this Reproduction Challenge appropriate for Bachelor and Master students as a thesis topic?
Most certainly! Replicating prior studies is a great way for emerging researchers to revisit the material and methods of a previously published study and to provide contributions to the research community in the course of trying to replicate the work, sometimes with surprising new findings along the way. With the Reproduction Challenge, students can support the research community by replicating specific works and have a publication goal through the challenge.
Why should I submit a “Replication Poster” to the VRST 2024 Reproduction Challenge?
Replicating original research can be a difficult undertaking. The best format for researchers to discuss options or specific plans for future replication studies is to discuss them with the community. Replication posters are 2-page non-archival submissions that allow researchers to present their planned replication research to the community in the form of a poster presentation at the conference, which does not limit their ability to later publish a full paper on the same topic. Replication posters allow the authors to discuss with other conference attendees which study in a particular field may need replicating, refine their plans for a replication study, find and involve other collaborators in a replication project, learn about ways to preregister studies (e.g., COS), and potentially plan larger studies across multiple lab sites.
Is the goal of a replication study to invalidate prior findings? Isn’t this a publication bias?
Yes and no. Yes, anecdotally, we have seen that replication papers in the past were often only accepted in regular conference paper tracks if the replication studies claimed new results that conflicted with prior findings, whereas studies that confirmed prior results were rejected due to a lack of novelty. However, as we outlined above, replication papers can make a wide range of nuanced contributions. Submissions to the VRST 2024 Reproduction Challenge will be reviewed by cognizant reviewers to minimize such publication biases.
Contacts:
Reproduction Challenge Chairs: reproducibility2024@vrst.acm.org
Gerd Bruder, University of Central Florida, USA
J. Edward Swan II, Mississippi State University, USA