[edit]

AISTATS 2024 AC Guidelines

This page provides guidelines for Area Chairs (ACs) for AISTATS 2023. The guideline has referenced previous AISTATS conferences’ practises and the AC guideline of NeurIPS 2023.

Responsibilities of an Area Chair (AC)

• Identify desk-rejection cases (e.g., papers with unacceptable formatting issues, dual submissions)
• Help reviewer assignments by recruiting expert reviewers and adjusting reviewer assignments
• Monitor the review process and ensure reviewers are responsive
• Ensure the submissions have high-quality reviewers
• Organize reviewer discussions after author feedback
• Make acceptance/rejection recommendations based on reviews, author feedback, discussions, and their own understanding of the submission
• Recommend accepted submissions for oral talks
• Resolve any issue during the review process, e.g., by recruiting emergency reviewers, addressing concerns from both reviewers and authors, and/or raising concerns with Senior ACs and Program Chairs.

Important Dates

Dates may be subject to change. All deadlines are AoE timezone.

• AC bid on papers: Sat, Oct 7 2023 - Sun, Oct 22 2023
• Reviewers bid on papers: Sat, Oct 14 2023 – Sun, Oct 22 2023
• Reviewer assignment adjustments: Mon, Oct 23 2023 - Sun, Oct 29 2023
• Reviewing period: Mon, Oct 30 2023 - Fri, Nov 17 2023
• Checking and soliciting emergency reviews: Fri, Nov 17 2023 - Thur, Nov 23 2023
• Reviews released to authors: Mon, Nov 27 2023
• Author feedback: Mon, Nov 27 2023 - Tue, Dec 5 2023
• Reviewer-AC Discussions: Wed, Dec 6 2023 - Fri, Dec 15 2023
• AC-SAC discussions: Fri, Dec 15 2023 - Wed, Dec 20 2023
• AC meta reviews due: Wed, Dec 20 2023
• SAC decisions: Mon, Jan 8 2024
• Author Notifications: Fri, Jan 19 2024

Main Tasks

Below is a list of tasks (and the detailed explanations) that an AC is expected to conduct.

1. Registrations on CMT

   • Make sure your profile on CMT is up-to-date. Adding your conflict of interests as well as selecting your expertise topics. Import papers to your CMT account.
   • Read and agree to abide by the AISTATS code of conduct.

2. Paper bidding

   • We will assign papers to ACs and reviewers using a combination of bids and a matching algorithm supported by the TPMS paper matching system.
   • After receiving the initial assignment, quickly check the submissions and identify desk-reject and/or conflict-of-interest cases if applicable.
   • It is possible that some of the submissions in your batch might not be closely related to your expertise. As we have a large number of submissions, your flexibility is greatly appreciated. When making decisions in later stages, contact Senior AC if you need help for the submissions that are too far away from your expertise area

3. Reviewer assignments adjustments

   • Make sure that every submission in your batch is matched with 5 suitable reviewers whom you can trust on this submission, ideally with a diverse set of opinions.
   • Do invest time for this before the reviewing process starts to ensure that your batch has expert reviewers - this will make your life much easier in reducing the number of borderline and mis-judged cases.

4. Monitor the review process and ensure reviewers are responsive

   • Send reminders to reviewers for submitting reviews on time.
   • Recruit emergency reviewers if not having enough high-quality reviews, or having reviews with diverging opinions, 1 week before the review release date.
   • Ask reviewers to rewrite and resubmit their review if appropriate.
   • Please ensure reviewers provide their comments with evidence. E.g., if a reviewer selects “combinatorial approach” in their answer to the question regarding novelty, ensure that in the next question that reviewer has provided the corresponding references to justify this choice.

5. Handle author feedback

   • Make sure the submitted author feedback document adheres to the instructions of the provided template (mirror).
   • Check if the reproduction checklist is submitted if applicable. Note that availability of this information is not used in acceptance decisions, just remind the authors to attach the reproduction checklist in camera ready if applicable.
   • Read confidential comments from both reviewers and authors.

6. Organize reviewer discussions after author feedback

   • Ensure reviewers have read the author feedback document. Ask them to update their review with e.g., a new paragraph starting “Post author feedback comments”.
   • Initiate and lead discussions between reviewers. At this point you should read both reviews and author feedback, to understand the pros and cons of the submission, as well as the major disagreements between reviewers and authors. List these points in your discussion post. If applicable, explicitly ask reviewers with e.g., very high/low scores to explain their opinions.
   • Read confidential messages from reviewers and authors. If applicable, contact the corresponding reviewers to address authors’ concerns (without revealing the authors’ confidential comments).

7. Decision making and meta-review

   • Be on time in writing your meta-review and making decisions – we have a tight schedule regarding final decisions!
   • You should write the meta reviews based on your reading of the paper, the reviews, the author feedback, and the reviewer discussions. Keep in mind that the meta-review needs to provide a convincing justification about your decision (especially in rejection cases).
   • The meta-review should augment the reviews, and explain how the reviews, the author response, and the discussions were used to arrive at your recommendation. Do not dismiss or ignore a review or author feedback unless you have a good reason for doing so; in such case you should also explain this in your meta review.
   • If uncertain about your decision, reach out to your SAC as early as you can. For papers where the reviewers cannot come to a consensus, you should read the paper carefully and write a detailed meta-review. You are expected to discuss such difficult cases with your SAC.
   • For submissions with rejection decisions, you are encouraged to also provide comments regarding potential ways to improve the paper.

Best Practices

• Please respect deadlines and respond to emails as promptly as possible.
• It is okay to be unavailable for part of the review process (e.g., on vacation for a few days). Contact SAC if you will be unavailable for more than that—especially during important windows such as the last week of reviewing and/or meta review.
• Be professional and listen to both reviewers and authors. However, your responsibility is not just to facilitate reviewer discussions and calculate average scores. We expect you to be familiar with all the submissions in your batch and to be able to justify your meta review with technical arguments.
• Be kind and considerate: personal situations may lead to late or unfinished work among reviewers. In the event that a reviewer is unable to complete their work on time, you might have to find emergency reviewers or even provide your own review. In all communications, exhibit empathy and understanding.
• If you notice unethical or suspicious behavior involving either authors or reviewers, please notify your SAC and the Program Chairs.

Writing the meta-review

Adapted from NeurIPS 2023 AC guidelines. Original source: Chris Williams and John Lafferty.

• Don’t focus too much on the scores. Instead, look carefully at the comments.
• Judge the quality of the review rather than taking note of the reviewer’s confidence score; the latter may be more a measure of personality.
• Compare the reviewers’ answers regarding their provided keywords for the submission and their expertise. Also compare it with the keywords specified by the authors. A mismatch between these keywords may indicate clarity issues and/or misunderstandings.
• Indicate that you have read the author feedback and explicitly write about whether the rebuttal has addressed (some of) the reviewers’ concerns or not.
• If you use information that is not in the reviews (e.g., new arguments in reviewer-AC discussions), tell the authors (a) that you have done so and (b) what that information is.
• If you find yourself wanting to overrule a unanimous opinion of the reviewers, the standards for your summary should be at the level of a full review. In these cases, we also suggest soliciting an auxiliary review if possible.
• Please attempt to take a decisive stand on borderline papers – we know these are tough decisions and we need your judgment here. Much of our work will involve borderline papers where no one confidently expresses excitement, nor are any major problems identified.
• Try to counter biases you perceive in the reviews. Unfashionable subjects should be treated fairly but often aren’t, to the advantage of the papers on more mainstream approaches. It is important to encourage risk and recognize that new approaches can’t initially yield SOTA results. Nor are they always sold according to the recipes we are used to.

Confidentiality

You must keep everything relating to the review process confidential. Do not use ideas, code and results from submissions in your own work until they become publicly available (e.g., via a technical report or a published paper for ideas/results, via open source for code, etc.). Do not talk about or distribute submissions (whether it is the code, or the ideas and results described in them) to anyone without prior approval from the program chairs. Code submitted for reviewing cannot be distributed. DO NOT talk to other ACs and/or SACs about submissions that are assigned to you unless you have a justified reason to do so, in such case before contacting other ACs/SACs, consult with your SAC and/or the Program Chairs.

Frequently Asked Questions

1. How do I contact the Senior Area Chair (SAC) and/or the Program Chairs?

   CMT provides contacting functions to do so once you’ve been assigned to an SAC. Also you can contact the Program Chairs via aistats2024conference@gmail.com.

2. What if the paper does not have the reproducibility checklist?

   This does not count for desk-reject, although you should check whether the authors have submitted the checklist during author feedback. Remind the authors of accepted submissions to include the checklist in camera ready.

3. What is the page limit?

   In the review period, the main part of the submission should be 8 pages long. After that, only the references are allowed. If that is not the case, please mark it as a format violation.

4. What else counts as format violations?

   Anything that breaks anonymity, violates dual submission policy, changes the format (e.g., margins or vertical spaces, or submitting the main paper and appendix in the same pdf). If in doubt, please mark it as a violation, which will be looked after by the Program Chairs.