When you hear supporting the research community and the peer review process, what do you think? Our recent CRA whitepaper published in July outlined key points for conferences to consider. Here’s what you should know.
Encouraging good documentation practices and promoting the sharing of research materials, conferences can create a culture of transparency. This makes research more reliable and helps researchers collaborate to verify each other’s work. Commitment to reproducibility protects the credibility of individual studies and strengthens entire fields of research.
Accurate documentation is crucial for ensuring reproducibility and replicability, which help maintain the accuracy and objectivity of research. Reproducibility refers to obtaining consistent results using the same data and methods, while replicability involves achieving consistent results across different studies addressing the same question. Failures in reproducibility can also undermine entire fields of research. In computing, a study found that only two-thirds of machine learning papers were fully reproducible, regardless of whether authors shared their code.
To promote best practices in reproducibility, conference organizers should encourage authors to document their experimental processes, share research artifacts, and have reviewers assess these artifacts for reproducibility.
Aside from reproducibility best practices, conferences should address abuse of the peer review process. According to a 2022 the National Academies report, there is a need for incorporating ethical and societal implications into the research publication review process, referred to as “ethics review.”
Currently, conferences typically handle ethics review through:
- Asking reviewers to consider ethics as part of the regular peer review
- Asking senior members like area chairs to conduct the ethics review
- Having a dedicated ethics review committee to evaluate flagged submissions, allowing for a focused, multidisciplinary approach to responsible research guidelines.
While the third approach is recommended for its potential for increased effectiveness, it raises concerns about delays in the review process. If some papers require extra steps, it might prolong feedback and decision timelines, potentially causing a disadvantage to some authors.
Therefore, it’s crucial to carefully manage these timelines and understand the reasons behind objections to ensure a fair and efficient review process.
While conferences are essential for growth in research, they must prioritize integrity, reproducibility, and fairness in the review process to protect researchers and the credibility of their work.
