Frequently Asked Questions

General

Q: I think TMLR is great, and I'd like to be involved! How can I sign up to be a reviewer (or Action Editor)?

A: Great! We have filled our reviewer and AE pools for now, but you can volunteer to review any submission via OpenReview (a "Volunteer to Review" button should be visible if you are logged in). You can also ask a current Action Editor that can vouch for your qualifications to add you as a reviewer. If you would like to be an Action Editor and believe that you are qualified through your academic credentials as a professor or through your experience (e.g. as an Area Chair or Editor), please contact the Editors-in-Chief.

Q: Will there be multiple back-and-forth rounds between reviewers and authors, as in some journals, or just one round as with most conferences?

A: There is one discussion phase, but during that phase there can be as much (and as frequent) back and forth, including revisions to the manuscript, as the authors and reviewers need. The discussion phase will be at least 2 weeks and up to 4 weeks long.

Q: How are reviewers assigned? Will it be entirely up to the action editors, or will there be a bidding process?

A: Assignments will largely be up to action editors. However, because submissions are shown publicly on OpenReview, a user who is not in conflict with the submission may request to be a reviewer (and it will be up to the action editor to accept the request).

Q: How does the double blind policy work? Would papers already on arXiv be allowed? How about papers that are visible on authors' websites, etc.?

A: You can still submit, even if a preprint of your work already is available online somewhere (as long as it's not in official proceedings for some other publishing venue). We follow the same approach as double blind conferences, asking that reviewers don't actively search online to figure out who the authors are.

Q: Will other artifacts of a paper be available, such as software, data, talk recordings or videos, slides, public discussions?

A: Code and videos may be uploaded and associated with camera ready papers. This may expand over time.

Q: Are you using Toronto paper matching in the review system?

A: OpenReview has its own paper matching system to rank similarity of reviewers and papers, and we will be leveraging it.

Q: Will accepted TMLR papers have the opportunity to be presented in ML conferences like NeurIPS, ICML, and ICLR?

A: We hope that TMLR will eventually partner with workshops and conferences (see blog post). We don't have such agreements with these conferences, but we'd like to down the road.

Q: Will non-anonymous OpenReview users be able to comment on the paper during the review process?

A: Yes.

Q: Can I request to anonymously review a submission?

A: Yes. For this, log in to OpenReview, visit the page of the submission and click on button "Volunteer to Review". The AE in charge of the submission will receive a notification asking them to accept or deny your request.

Q: Does TMLR have a preference for methodological/theoretical work or will application papers be welcome?

A: TMLR does not have a preference. As long as the paper is within scope and we have an action editor who is qualified to handle a submission, we should be open to receiving it.

Q: Does TMLR accept survey papers?

A: Yes. Authors should make sure to emphasize the contributions made by the survey. Ideally, we want survey papers that draw new, previously unreported connections between several pieces of work in an area, and/or that clearly highlight trends in the area and/or suggest currently open problems. It should also be noted that if a submission has more than 12 pages of main content, then TMLR's normal short review timeline will not be enforced.

Q: How are submissions stored?

A: All submitted papers, including those accepted, rejected, withdrawn, or retracted, are made publicly accessible on OpenReview. An exception is made for desk-rejected papers, which are never made public. OpenReview intends to archive all submissions by keeping them publicly accessible indefinitely.

Q: How often is TMLR published?

A: TMLR is a continuous publication, and every paper is immediately published upon completion of the review process, including submission of the camera-ready version of the paper and its verification by the assigned action editor.

Q: Does TMLR have a ISSN code?

A: Yes, it is 2835-8856.

Q: Are authors and/or reviewers allowed to use LLMs when writing submissions and/or reviews?

A: LLMs may be used as general-purpose assistive tools. Whichever tools are used, authors and reviewers are fully responsible for content on which they are listed as (co-) authors or reviewers. This includes, but is not limited to, content generated by LLMs that could be construed as plagiarism or scientific misconduct (e.g., fabrication of facts). Low-quality contributions (be they submissions or reviews) that appear to be largely LLM-generated will be closely examined for evidence of the issues mentioned previously, such as scientific misconduct. LLMs are not eligible for authorship. We will periodically revise this policy as new information about the use of LLMs in the scientific process becomes available.

Authors

Q: Can I use ChatGPT or other large language model systems to assist in writing my TMLR submission?

A: Yes, but TMLR also considers that it is the authors' responsibility to avoid any unforeseen consequences that might come about from using such tools, such as the later discovery that a significant part of the submission's content has been plagiarized (in which case we'd have to reject the paper). Similarly, our expectation is that the ideas, claims and results found in submissions are human-sourced, not produced by such tools.
Additionally, transparency is always good, so we also require that authors mention explicitly in their submission that they have used this tool, as a footnote on the first page.

Q: When are authors expected to submit responses to reviews?

A: We strongly recommend that authors submit responses, including any revisions, no later than 2 weeks after the submission of the 3rd review.

Q: Can authors submit more than one revision during the discussion period?

A: Yes. It is also recommended to leave a comment on the OpenReview page, visible to reviewers and the AE, to alert them to the revision.

Q: Is it possible to make changes to the camera ready version of an accepted paper?

A: In general, camera ready versions of papers are considered to be final, and changes are not allowed. In exceptional cases, post-publication modifications may be considered at the discretion of the Editors-in-Chief. To request a change, send an email to tmlr-editors@jmlr.org.

Q: The decision for my submission is “Accept with minor revision”. How should I submit that revision for consideration by the Action Editor?

A: The required revisions should be made in the camera ready submission. After you submit the revised camera ready version, the Action Editor will review it and determine whether the appropriate minor revisions were made.

Q: Who sees the author's names on a submitted paper?

TMLR adopts a double-blind reviewing system. On papers that are submitted but not yet accepted for publication, the author names are hidden Only the action editor assigned to that paper and the editors-in-chief can see the author's.

Q: How is TMLR different from other machine learning journals such as JMLR?

A: TMLR provides a similar experience to submitting to a conference – double-blind, open access reviews, fast turnaround, and novel, unpublished content – but with a rolling, year-round review process and evaluation criteria that focuses on technical correctness of claims rather than more subjective, editorial measures. Consider submitting shorter manuscripts to TMLR, and submitting extended works which incorporate previously published material to JMLR.

Q: How long will it take to review my manuscript?

A: TMLR does not make any guarantees, and manuscripts whose main body exceeds 12 pages are subject to longer timescales, but in general the rolling review process aims to deliver a final decision approximately 9 weeks after submission.

Q: Are replications of previous research welcome?

A: Yes.

Q: Can I include the appendix in the main pdf so that hyperlinks are preserved, or should it go in the supplementary zip file?

A: Appendices can be included in the main pdf. The supplementary zip file should contain additional material such as videos or code.

Q: Can I use the TMLR style and template if I upload my paper to arXiv?

A: We don't have a policy proscribing the use of our template on arXiv. However, make sure to use the "preprint" option of the stylefile if you do.

Q: Upon resubmission of a paper I need to specify the differences with the previous version. Won't this violate the anonymization of my latest submission?

A: Only the action editor (not the reviewers) will see the previous submission URL, so including it will not violate our guidelines.

Reviewers

Q: Is it possible to see the changes made between revisions of a submission?

A: Yes. If you click “Show Revisions”, you are directed to a page showing the list of “Revision History”. If we want to compare two versions from the revision history, click “Compare Revisions” button on the top right corner of this page. See here for more details.

Q: How long will reviewers have to complete their review?

A: We ask for reviews to be submitted within 2 weeks for submissions of up to 12 pages (main content, before references). For submissions over 12 pages, we ask for reviews to be submitted within 4 weeks. While the timeframe will be tight, we're trying to compensate by ensuring a low workload at any given time (no more than 1 paper per reviewer).

Q: How can I avoid receiving assignments during a certain period?

A: If you enter a period of unavailability to receive any new assignments (e.g. you are on vacation or on work leave), you may now enter that information explicitly in OpenReview. To do so, simply go to your reviewer console at https://openreview.net/group?id=TMLR/Reviewers and, next to Assignment Availability, select Unavailable.

Please don't forget to change that back to “Available” as soon as you are back to being able to receive assignments.

Q: Can I adjust my annual quota of assignments?

Yes, you can adjust your annual quota of assignments higher or lower than the default quota (6 assignments per year) through the openreview console. To do so, simply go to your reviewer console at https://openreview.net/group?id=TMLR/Reviewers and, next to Custom Max Paper, select the number you wish for your quota.

Q: When submitting my review, the form isn't asking for a decision recommendation or a score. Is that normal?

A: Yes. Initially, we don't ask for a formal recommendation from you, so you can keep your mind open during the discussion period with the authors. However, two weeks after the 3rd review for a submission has been submitted, you will be asked to separately submit to the Action Editor an official recommendation for a decision on the submission.

Q: What if I have seen the paper at a workshop, on arXiv, on Twitter, etc, and I know who the authors are?

A: You should not try to de-anonymize any submission by actively searching for it. However, you may come across it by chance. If this happens, and you believe that it will influence your judgment, please contact the action editor, who will help decide whether you should be replaced with a new reviewer.

Q: The submission has an appendix and supplementary material. Am I obligated to consult it?

A: Much like for reviewing at conferences, consulting appendices and supplementary material is left to the discretion of the reviewer.

Q: Can I discuss the paper privately with the other reviewers and/or the Action Editor?

A: Yes, but only once all 3 reviewers have posted their review. Simply post a comment that is only visible to them to start the discussion.

Q: How can I influence what type of papers I'll be asked to review?

A: Go in your openreview account profile and click on Edit (https://openreview.net/profile/edit) and then if you scroll down you'll see your imported publications. You can remove those that you don't want to influence which papers are assigned to you.

Action Editors

Q: When are reviews visible to other reviewers and the public?

A: Reviews become visible to other reviewers and the public as soon as 3 reviews have been submitted. For authors, there is an exception to this rule: authors see each review as soon as it is submitted, so they can start to address them immediately.

Q: If the authors of a submission want more time than the standard 2-week period to submit a response to reviews or a revised submission, can I give them an extension?

A: Generally, authors should not expect an extension to be granted. However, if the reviewers and yourself are in agreement that an extension is warranted, you may informally provide them with more time.

Q: I would like to move to the final review process phase for a submission (rating reviews and submitting a decision), but some of the reviewers are late on submitting their official recommendation. What should I do?

A: First, email directly the reviewers to remind them that they must submit their official recommendation. If after some time you still don't hear from them, but you feel that you can still make an informed decision on the submission, you can email the Editors-in-Chief (tmlr-editors@jmlr.org) to move forward without all official recommendations.

Q: A reviewer I have assigned is significantly late/unresponsive. Is there a way to report their inappropriate behavior?

A: Yes. In the AE console, there is a link “Reviewers Report” at the top, leading to a section where you can submit a report. The report will only be visible by you and the Editors-in-Chief.

Q: Is it possible to see the changes made between revisions of a submission?

Q: Can I adjust my annual quota of assignments?

A: Yes, you can adjust your annual quota of assignments higher or lower than the default quota (12 assignments per year) through the openreview console. To do so, simply go to your AE console at https://openreview.net/group?id=TMLR/Action_Editors and, next to Custom Max Paper, select the number you wish for your quota.

Q: How can I avoid receiving assignments during a certain period?

A: If you enter a period of unavailability to receive any new assignments (e.g. you are on vacation or on work leave), you may now enter that information explicitly in OpenReview. To do so, simply go to your AE console at https://openreview.net/group?id=TMLR/Action_Editors and, next to Assignment Availability, select Unavailable.

Please don't forget to change that back to “Available” as soon as you are back to being able to receive assignments. Your status will automatically be switched back to "Available" after two months.

Q: Can I invite someone to join TMLR's pool of reviewers?

A: Yes, this can be done through the AE Console on OpenReview, by clicking on link "Recruit Reviewer" and then on button "Reviewer Recruitment by AE".

Q: Can I invite someone to review a specific submission I'm in charge of, without them joining TMLR's general pool of reviewers?

A: Yes. There are two ways to do this. One is to simply ask that person to visit the submission's OpenReview page, where they can request to review the submission by clicking on button "Volunteer to Review". You'll then receive a notification asking you to accept their request. Once you do, they will become an official reviewer for that submission.

The other approach is to search for the person's name on the submission's reviewer assignment page and, if you don't find them, to use the dedicated text box below (which appears when no reviewer with a searched name is found) to enter that person's email and send them an invitation. Assuming the candidate reviewer doesn't have a conflict with the authors of the submission, then they'll be able to accept the assignment, while not joining the regular pool of TMLR reviewers.

Q: I'm in charge of a resubmission that I previously rejected, with encouragement to resubmit. I'd like to assign the same reviewers I used in the first round of reviews. Can I do that?

A: Yes. If the reviewers in question are currently available (e.g. they don't have a pending review for another submission), go ahead and assign them as you would normally. However, if the reviewers aren't available, consider reaching out by email directly to them, asking them if they could still take on the review assignment. If they agree, then ask them to click on the "Volunteer to Review" button on the OpenReview page of the submission.

Q: I am concerned that a submission, which is otherwise acceptable, violates the TMLR ethical guidelines; how should I handle this?

A: Consider whether the submission would be acceptable if the authors added or improved a broader impact statement, and if so, this can be required for the camera-ready version. If the submission carries significant risk of harm, contact the Editors-in-Chief.

Q: Three reviewers have now submitted their official recommendations. I want to submit my decision, but I don't seem able to do it on OpenReview. What should I do?

A: Before you can submit your decision, you must rate the quality of the reviews submitted by the reviewers, on OpenReview. Once you have submitted these ratings, the submission of your decision will be enabled on OpenReview.

Q: I tried adding reviewers to the paper, but I get the error Error: Can not add assignment, invitation is not active yet.

A: After the action editor submits the review approval, there is a small delay of time during which affinity scores between the paper and reviewers are computed and before which they can start assigning reviewers. Once the computation is done, you will receive a confirmation email announcing that you can now start assigning reviewers.