TMLR guidelines for reviewers

This document provides instructions for reviewers. Before reading this section, make sure to also read and be familiar with the Editorial Policies, the Ethics Guidelines, and the Code of Conduct.

Reviewing Responsibilities

As a reviewer, you are expected to hold the following responsibilities.

Double blind. The TMLR review process is meant to be double blind. The identities of the authors of a submission will be kept from the reviewers, and vice versa. You are expected not to take any actions that would violate this double blind state. This includes not actively seeking to find out the identity of the authors of a submission you are reviewing (e.g. by searching for related presentations online or preprints on arXiv). Generally, we suggest you do searches on related work after doing a preliminary read of the submission. If you accidentally discover the identity of the authors, and believe it may influence your judgment, then contact the Action Editor (AE) so they may determine whether you should be unassigned from the submission.

Open Reviewing. As soon as all reviews are submitted for a submission, they will become publicly visible (but anonymous). The authors' response to your review and subsequent discussion will also be immediately visible. You are expected to actively engage with the authors until the issues you have raised for discussion have been sufficiently explored. During this process, you are expected to be respectful and abide by our Code of Conduct.

Annual Quota. Reviews are assigned on a rolling basis, as submissions are received. By default, TMLR will not assign you more than six reviews per year. The default quota is 6 papers per year, but it is possible to change it through the openreview console, as well as to mark yourself temporarily unavailable for periods such as illness, vacation or work leave. On the other hand, you are expected to perform all review assignments of submissions that fall within your expertise and quota. Acceptable exceptions include having an unsubmitted review for another TMLR submission. A couple days after being assigned to review a submission, you will be asked by OpenReview to acknowledge the review assignment, so that the Action Editor has a confirmation that you are aware of the assignment and will be submitting it in time.

Review deadline. TMLR is committed to a reviewing process with fast turnaround. Therefore, reviews must be submitted within 2 weeks of their assignment by the AE for submissions up to 12 pages (of main content, before references), and 4 weeks for submissions over 12 pages. Two weeks after all reviewers have submitted their review and the discussion with the authors has started (and no later than 1 month), each reviewer will then be allowed to submit a final decision recommendation to the AE.

Conflict of interest. You are responsible for making sure your affiliation and education/career history information is up to date in your OpenReview profile, so we can properly enforce our conflict of interest policy. We consider in conflict any individual or institution with which you have shared a collaboration within the past three years. After being assigned a submission, if you suspect or believe that you have a conflict of interest with any of the authors, consult with the AE for next steps.

Submission Acceptance Standard

Please see the page Acceptance Criteria.

Review Format

A review should have the following content.

Summary of contributions Brief description, in the reviewer's words, of the contributions and new knowledge presented by the submission.

Strengths and weaknesses List of the strong aspects of the submission as well as weaker elements (if any) that you think require attention from the authors.

Requested changes List of proposed adjustments to the submission, specifying for each whether they are critical to securing your recommendation for acceptance or would simply strengthen the work in your view.

Broader impact concerns Brief description of any concerns on the ethical implications of the work that would require adding a Broader Impact Statement (if one is not present) or that are not sufficiently addressed in the Broader Impact Statement section (if one is present).

Immediately after submitting your review, the crucial period of discussion with the authors starts . You should engage with the authors by responding to their requests for clarifications, acknowledging the changes they make to their submission and signal to them as soon as you are willing to recommend acceptance.

Official Recommendation

Once two weeks have passed since all reviews have been submitted, reviewers will be able to submit their official recommendations to the AE . Specifically, they will be asked for the following.

Decision recommendation (accept, leaning accept, leaning reject or reject) Whether or not you recommend accepting the submission, based on your initial assessment and the discussion with the authors that followed.

The AE will monitor the discussion and consult the reviewers' feedback and recommendations. When they are ready to make a decision on the submission, they will also evaluate and rate your review.

Certification recommendations Certifications are meant to highlight particularly notable accepted submissions, and act as public endorsements of the paper in a number of categories. Notably, it is through certifications that we make room for more speculative/editorial judgement on the significance and potential for impact of accepted submissions. Certification selection is the responsibility of the AE, however you are asked to submit your recommendation.

Outstanding Certification. The editorial board of TMLR jointly awards this certification to papers which are deemed to be exceptionally high quality and broadly significant for the field. The certification may be given well after the paper’s initial publication in TMLR (a year or more), and is equivalent to a best paper award at a top-tier conference.

The remaining four certifications are recommended by reviewers, nominated by action editors and must be approved by at least one editor-in-chief:

Featured Certification. This certification may be awarded to papers that are very high quality. These papers present significant contributions which are novel, clearly explained, and well supported with evidence, theory, or analysis. If this paper was submitted to a top-tier conference, it would likely be presented as an oral/spotlight.
Expert Reviewer Certification. This is awarded to papers that are authored by highly-rated TMLR reviewers, to recognize their contributions and expertise.
Reproducibility Certification. This is awarded to papers whose primary purpose is reproduction of other published work. Beyond simple verification, the paper must contribute significant added value through additional baselines, analysis, ablations, or insights.
Survey Certification. Papers that not only meet the criteria for acceptance but also provide an exceptionally thorough or insightful survey of the topic or approach may be awarded this certification.