TMLR guidelines for reviewers
As a reviewer, you are expected to hold the following responsibilities.
Double blind. The TMLR review process is meant to be double blind. The identities of the authors of a submission will be kept from the reviewers, and vice versa. You are expected not to take any actions that would violate this double blind state. This includes not actively seeking to find out the identity of the authors of a submission you are reviewing (e.g. by searching for related presentations online or preprints on arXiv). Generally, we suggest you do searches on related work after doing a preliminary read of the submission. If you accidentally discover the identity of the authors, and believe it may influence your judgment, then contact the Action Editor (AE) so they may determine whether you should be unassigned from the submission.
Open Reviewing. As soon as all reviews are submitted for a submission, they will become publicly visible (but anonymous). The authors' response to your review and subsequent discussion will also be immediately visible. You are expected to actively engage with the authors until the issues you have raised for discussion have been sufficiently explored. During this process, you are expected to be respectful and abide by our Code of Conduct.
Annual Quota. Reviews are assigned on a rolling basis, as submissions are received. By default, TMLR will not assign you more than six reviews per year. The default quota is 6 papers per year, but it is possible to change it through the openreview console, as well as to mark yourself temporarily unavailable for periods such as illness, vacation or work leave. On the other hand, you are expected to accept all requests for review of submissions that fall within your expertise and quota. Acceptable exceptions include having an unsubmitted review for another TMLR submission.
Review deadline. TMLR is committed to a reviewing process with fast turnaround. Therefore, reviews must be submitted within 2 weeks of their assignment by the AE for submissions up to 12 pages (of main content, before references), and 4 weeks for submissions over 12 pages. Two weeks after all reviewers have submitted their review and the discussion with the authors has started (and no later than 1 month), each reviewer will then be allowed to submit a final decision recommendation to the AE.
Conflict of interest. You are responsible for making sure your affiliation and education/career history information is up to date in your OpenReview profile, so we can properly enforce our conflict of interest policy. We consider in conflict any individual or institution with which you have shared a collaboration within the past three years. After being assigned a submission, if you suspect or believe that you have a conflict of interest with any of the authors, consult with the AE for next steps.
Submission Acceptance Standard
The Editorial Policies page provides a brief description of TMLR's acceptance criteria, which we expand further here.
Acceptance of a submission to TMLR should be based on positive answers to the following two questions.
Are the claims made in the submission supported by accurate, convincing and clear evidence?
This is the most important criteria. This implies assessing the technical soundness as well as the clarity of the narrative and arguments presented.
Any gap between claims and evidence should be addressed by the authors. Often, this will lead reviewers to ask the authors to provide more evidence by running more experiments. However, this is not the only way to address such concerns. Another is simply for the authors to adjust (reduce) their claims.
Would some individuals in TMLR's audience be interested in the findings of this paper?
This is arguably the most subjective criteria, and therefore needs to be treated carefully. Generally, a reviewer that is unsure as to whether a submission satisfies this criteria should assume that it does.
Crucially, it should not be used as a reason to reject work that isn't considered “significant” or “impactful” because it isn't achieving a new state-of-the-art on some benchmark. Nor should it form the basis for rejecting work on a method considered not “novel enough”, as novelty of the studied method is not a necessary criteria for acceptance. We explicitly avoid these terms (“significant”, “impactful”, “novel”), and focus instead on the notion of “interest”. If the authors make it clear that there is something to be learned by some researchers in their area from their work, then the criteria of interest is considered satisfied. TMLR instead relies on certifications (such as “Featured” and “Outstanding”) to provide annotations on submissions that pertain to (more speculative) assertions on significance or potential for impact.
Here's an example on how to use the criteria above. A machine learning class report that re-runs the experiments of a published paper has educational value to the students involved. But if it doesn't surface generalizable insights, it is unlikely to be of interest to (even a subset of) the TMLR audience, and so could be rejected based on this criteria. On the other hand, a proper reproducibility report that systematically studies the robustness or generalizability of a published method and lays out actionable lessons for its audience could satisfy this criteria.
A review should have the following content.
Summary of contributions Brief description, in the reviewer's words, of the contributions and new knowledge presented by the submission.
Strengths and weaknesses List of the strong aspects of the submission as well as weaker elements (if any) that you think require attention from the authors.
Requested changes List of proposed adjustments to the submission, specifying for each whether they are critical to securing your recommendation for acceptance or would simply strengthen the work in your view.
Broader impact concerns Brief description of any concerns on the ethical implications of the work that would require adding a Broader Impact Statement (if one is not present) or that are not sufficiently addressed in the Broader Impact Statement section (if one is present).
Immediately after submitting your review, the crucial period of discussion with the authors starts . You should engage with the authors by responding to their requests for clarifications, acknowledging the changes they make to their submission and signal to them as soon as you are willing to recommend acceptance.
Once two weeks have passed since all reviews have been submitted, reviewers will be able to submit their official recommendations to the AE . Specifically, they will be asked for the following.
Decision recommendation (accept, leaning accept, leaning reject or reject) Whether or not you recommend accepting the submission, based on your initial assessment and the discussion with the authors that followed.
The AE will monitor the discussion and consult the reviewers' feedback and recommendations. When they are ready to make a decision on the submission, they will also evaluate and rate your review.
Certification recommendations Certifications are meant to highlight particularly notable accepted submissions, and act as public endorsements of the paper in a number of categories. Notably, it is through certifications that we make room for more speculative/editorial judgement on the significance and potential for impact of accepted submissions. Certification selection is the responsibility of the AE, however you are asked to submit your recommendation.
- Outstanding Certification. The editorial board of TMLR jointly awards this certification to papers which are deemed to be exceptionally high quality and broadly significant for the field. The certification may be given well after the paper’s initial publication in TMLR (a year or more), and is equivalent to a best paper award at a top-tier conference.
The remaining four certifications are recommended by reviewers, nominated by action editors and must be approved by at least one editor-in-chief:
- Featured Certification. This certification may be awarded to papers that are very high quality. These papers present significant contributions which are novel, clearly explained, and well supported with evidence, theory, or analysis. If this paper was submitted to a top-tier conference, it would likely be presented as an oral/spotlight.
- Expert Reviewer Certification. This is awarded to papers that are authored by highly-rated TMLR reviewers, to recognize their contributions and expertise.
- Reproducibility Certification. This is awarded to papers whose primary purpose is reproduction of other published work. Beyond simple verification, the paper must contribute significant added value through additional baselines, analysis, ablations, or insights.
- Survey Certification. Papers that not only meet the criteria for acceptance but also provide an exceptionally thorough or insightful survey of the topic or approach may be awarded this certification.
|© TMLR 2022.|