AISTATS 2022 Reviewer Guidelines

How to Review?

The purpose of the review process is twofold. First, to identify papers which offer significant contributions to the fields of artificial intelligence and statistics, for attendees and readers. Second, to provide constructive feedback to authors that they can use to improve their work. Your role as a reviewer is vital to both goals. When reviewing a paper, always think about the impact the work may have on the community in the long run: out-of-the-box ideas, novel problems, and “bridging fields” contributions are crucial for the successful development of the field, so do not neglect the importance of papers with this type of contributions. Novel or interdisciplinary works are often very easy to criticize, because, for example, the assumptions they make or the models they use are not yet widely accepted by the community. Yet, such works may be of high importance for the progress of the field in the long run, so please try to be aware of this bias, and avoid dismissive criticism.

Acknowledgments: The guidelines in this document are partially adopted from the NeurIPS 2020 reviewer guidelines, which in turn utilize reviews written for some NeurIPS, ICML, and ICLR papers.

Review Form and Guidelines for Writing a Good Review

The review form will ask you for the following:

Summary and contributions: Briefly summarize the paper and its contributions

Summarize the paper motivation, key contributions and achievements in a paragraph. Although this part of the review may not provide much new information to authors, it is invaluable to ACs and program chairs, and it can help the authors determine whether there are misunderstandings that need to be addressed in their author response. There are many examples of contributions that warrant publication at AISTATS. These contributions may be theoretical, methodological, algorithmic, empirical, connecting ideas in disparate fields (“bridge papers”), or providing a critical analysis (e.g., principled justifications of why the community is going after the wrong outcome or using the wrong types of approaches.).
Strengths: Describe the strengths of the work. Typical criteria include: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution, and relevance to the AISTATS community.

List the strengths of the submission. For instance, it could be about the soundness of the theoretical claim or the soundness of empirical methodology used to validate an empirical approach. Another important axis is the significance and the novelty of the contributions relative to what has been done already in the literature, and here you may want to cite these relevant prior works. One measure of the significance of a contribution is (your belief about) the level to which researchers or practitioners will make use of or be influenced by the proposed ideas. Solid, technical papers that explore new territory or point out new directions for research are preferable to papers that advance the state of the art, but only incrementally. Finally, a possible strength is the relevance of the line of work for the AISTATS community.
Weaknesses: Describe the limitations of this work according (but not limited) to the following criteria: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution, and relevance to the AISTATS community.

This is like above, but now focussing on the limitations of this work.

Your comments should be detailed, specific, and polite. Please avoid vague, subjective complaints. Think about the times when you received an unfair, unjustified, short, or dismissive review. Try not to be that reviewer! Always be constructive and help the authors understand your viewpoint, without being dismissive or using inappropriate language. Remember that you are not reviewing your level of interest in the submission, but its scientific contribution to the field!
Correctness: Are the claims and method correct? Is the empirical methodology correct?

Explain if there is anything incorrect with the paper. Incorrect claims or methodology are the primary reason for rejection. Be as detailed, specific and polite as possible. Thoroughly motivate your criticism so that authors will understand your point of view and potentially respond to you.
Clarity: Is the paper clearly written? Does it clearly state its contributions, notation and results?

Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)
Relation to prior work: Is it clearly discussed how this work differs from or relates to prior work in the literature?

Explain whether the submission is written with due scholarship and suitably relates the proposed work to prior work in the literature. The related work section should not just list prior work, but explain how the proposed work differs from, builds upon, and/or improves on it. Note that authors are not expected to know about all non-peer-reviewed work (e.g, preprints such as on arXiv). Other works (whether peer-reviewed or not) that appeared less than two months before the submission deadline are considered concurrent to AISTATS submissions; authors are not obligated to make detailed comparisons to such papers (though, especially for the camera ready versions of accepted papers, authors are encouraged to).
Additional Comments. Add your additional comments, feedback and suggestions for improvement, as well as any further questions for the authors. (Optional)

Add here any additional comment you might have about the submission, including questions and suggestions for improvement.
Reproducibility. Are there enough details to reproduce the main results of this work?

Please assess whether, with the information provided by the authors in the paper and the supplementary material, the main results of this work are reproducible or not. Lack of reproducibility should be listed among the weaknesses of the submission.
Assumptions and limitations. Does the paper explicitly and clearly state the main assumptions and limitations of the work?

Please assess whether, with the information provided by the authors in the paper and the supplementary material, the assumptions and limitations of this work are clearly stated. If they are not, consider listing this as a weakness of the paper.
Societal impact: Does the paper discuss the societal impact of the work, including the impact that may arise from the misuse of the paper’s contribution?

Please assess whether the paper discusses the potential societal impact of the work. If not, please assess if the paper should include this point due to the nature of the research topic or ideas.
Code release. Do the authors promise to release code for this submission?

If the authors have promised to release code in their paper (either as an anonymized link or by including the code as part of the supplementary material) or during the rebuttal phase, please mark “Yes”. This year, accepted papers that promise to release code are only accepted conditionally on the authors releasing their code publicly (e.g. on GitHub) by the camera ready deadline. You help the Program Chairs to identify such papers by answering this question.
Score:

You should NOT assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. The “Overall Score” for each submission should reflect your assessment of the submission’s contributions.
- 1 : Trivial or wrong or already known results
- 2 : Strong reject (I would be very upset if accepted)
- 3 : Clear reject (I vote and argue for rejection)
- 4 : Reject (an ok submission but not good enough)
- 5 : Marginally below the acceptance threshold (I lean towards rejection but would not be upset if accepted)
- 6 : Marginally above the acceptance threshold (I lean towards acceptance but would not be upset if rejected)
- 7 : Accept (a good submission)
- 8 : Clear accept (a very good submission, top 50% of accepted AISTATS papers)
- 9 : Strong accept (an excellent submission, top 15% of accepted AISTATS papers)
- 10 : Must accept (top 5% of accepted AISTATS papers)
.
Confidence score:
- 5: You are absolutely certain about your assessment. You are very familiar with the related work.
- 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
- 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
- 2: You are willing to defend your assessment, but it is quite likely that you did not understand central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
- 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
.
Does the submission raise potential ethical concerns? Does this submission raise potential ethical concerns? These include methods, applications or data that create or reinforce unfair biases and/or that have a primary purpose of harm or injury.

Yes or No. Note that your rating should be independent of this. Your duty here is to flag papers that might need additional review from the ethical perspective.
Ethical Concerns Details.

If you answered “Yes” to the previous question, please briefly explain the potential ethical concerns. If the submission might raise any potential ethical concern, please briefly explain the potential concerns.
Code of conduct.

Agree to abide by the AISTATS code of conduct The AISTATS code of conduct can be found here: AISTATS Code of Conduct
Confidentiality agreement.

Reviewers must keep the paper and supplementary materials (including code submissions and LaTeX source), as well as the reviews, confidential. This includes deleting any submitted code at the end of the review cycle to comply with confidentiality requirements.
Confidential comments for the Area Chair (optional).

If you have comments that you wish to be kept confidential from the authors, you can use the “Confidential Comments to Area Chair” text field. Such comments might include explicit comparisons of the submission to other submissions and criticisms that are more bluntly stated. If you accidentally find out the identities of the authors, please do not divulge the identities to anyone.

Best Practices

Be thoughtful. The paper you are reviewing may have been written by a first year graduate student who is submitting to a conference for the first time and you don’t want to crush their spirits.
Be fair. Do not let personal feelings affect your review.
Be useful. A good review is useful to all parties involved: authors, other reviewers, and area chairs.
Be specific. Do not make vague statements in your review, as they are unfairly difficult for authors to address.
Be flexible. The authors may address some points you raised in your review during the discussion period. Make an effort to update your understanding of the paper when new information is presented, and revise your review to reflect this.
Be timely. Please respect the deadlines and respond promptly during the discussion. If you cannot complete your review on time, please let the Area Chair know as soon as possible.

If someone pressures you into providing a positive or negative review for a submission, please notify the Program Chairs right away at aistats2022+pc@gmail.com.

If you notice unethical or suspect behavior, please notify your Area Chair right away.

How to Access and Download Your Assigned Papers

Login to the CMT portal using your reviewer CMT email address.
Select “Reviewer” role by clicking next to “Select Your Role” at the top of the page. You will then see the submissions that are assigned to you in your console. Note that the reviewing process is double-blind, so you will not see author information.
Download all your assigned papers by selecting “Actions -> Download Files” in the upper right of the page.

Other Remarks

Minor formatting issues. You may find papers with minor formatting issues such as different citation styles. Moreover, a large percentage of papers uses the font Times New Roman, which is slightly smaller than the default Computer Modern font from the template. We have decided not to desk-reject these papers and ask you to disregard such minor formatting issues in your reviews (please flag them but do not change your score/evaluation). We will take them into account and ensure the correct style is used if the paper gets accepted.
Supplement. Some authors forgot to separate the pages for supplementary materials from the main paper, e.g., by splitting the PDF. We are keeping these papers in the review process. If you find such extra pages, you can treat them as supplementary materials. As stated in the CfP, looking at the supplementary materials is optional to evaluate the submission.
Links in submission. Some submissions may contain links (e.g., URL) to external materials such as code. We wish to emphasize that we cannot guarantee the safety of such external links. Our general recommendation is that you not follow such links and simply take it as a statement from the authors that they are ready to make such material available if the paper is accepted. If you decide to access a link, be aware it might reveal that you have consulted the linked site, and thus potentially reveal that you are involved in the reviewing of the paper.
Code in Github. Some of the papers assigned to you may have submitted code. Note that, as for any supplementary material, reviewing such additional information (e.g., code) is optional. When accessing anonymous Github links submitted by authors, please make sure to directly download the code or clone the repository after logging out of your own github account; please do not fork the repository as this might reveal your identity to the authors. Note also that any submitted code may contain security vulnerabilities.