[edit]

See also the detailed reviewer instructions if you are a reviewer.

We're making some changes to the typical review process for AISTATS.  The main changes are: (1) a two-stage reviewing process with more time for the authors to revise their papers, which will hopefully improve the quality of the final versions; and (2) awards for an increased number of notable papers, along with a process for giving these papers increased visibility and discussion, hopefully motivating authors to submit their best work.

Expanded review interaction

We will expand the author response phase: instead of the typical very-short plain-text author response, the authors will have an opportunity to revise their paper as they see fit with the benefit of the initial reviews.  In addition, in most cases, the revised version of the paper will be reviewed by at least one completely fresh reviewer, who has not seen the original paper or reviews.  (The new reviewer will of course still be able to discuss with the SPC and original reviewers after their review is submitted.) This feature is a reaction to author frustration with the low bandwidth of the response channel, and with authors' sense that the reviewers may already have made their decision before reading the author feedback.  The authors will still be responsible for summarizing their changes, to keep the original reviewers from having to re-read the entire paper.

In addition, we will conduct reviews in two phases: an initial review by two reviewers, and a second phase with zero or more additional reviewers.  This feature is motivated, first, to be compatible with the expanded author response phase described above, and second, to try to reduce the total effort spent by our volunteer reviewers without reducing the quality of the review process, by focusing reviewer effort on papers that are likely to be accepted.  We expect that the quality of reviews will even be increased in some cases: first, by not overloading the reviewers, and second, by having the benefit of the initial reviews when selecting additional reviewers.  (Note: some recent conferences have tried to reduce reviewer effort by having initial “quick reviews” to determine which papers get full reviews.  Our impression is that such experiments have typically not worked as well as hoped: it requires some time to fully understand a paper and its strengths and weaknesses, so quick reviews are even noisier than regular ones.  So, our thought is that the best way to shortcut the process is by having a smaller number of full reviews.)

Finally, for all accepted papers, we will make drafts available well ahead of the conference, and facilitate online discussion.

Notable papers

To encourage authors to submit their best work, we will expand the number of paper awards.  Currently, the selection process for best paper awards often seems somewhat arbitrary, and because of this perception, the possibility of getting an award doesn't necessarily contribute much to authors' motivation to submit their best work.  Instead, we will give a larger number of “notable paper” awards.  We will keep the number of awards small enough so that they are still exclusive and confer an honor on the recipients, but large enough that submitting a really good paper should result in a significant chance of an award.  Notable papers will be given a longer oral presentation time then regular accepted papers, and the plan will be to publish these papers in a special issue of a journal.

In addition, we will invite a “discussant” for each notable paper.  The discussant will write a short summary discussion of the paper's most significant benefits, caveats, and relationships to the broader literature.  We will publish these summaries as invited papers in the journal special issue, and make initial drafts of the summaries available online before the conference.  Each discussant will have access to the reviewer discussion and to the later online discussion of the paper, and will be expected to work without breaking the anonymity of the original reviewers.  The goal is to keep the community as a whole from losing the benefit of the effort that reviewers have put in to understand the paper and place it within the larger context of related work.

Additional points

Reviews are only advisory.  The senior program committee, and ultimately the program chairs, will have full authority to accept or reject papers by criteria other than sorting the list by average numerical score.  Instead, the job of the SPC will be to synthesize the issues brought up in the reviews to make and explain an informed acceptance decision.  This feature has been more or less true of many conferences for a while now, but we thought it best to put it out in the open.  This feature is a reaction to the well-known observation that numerical scores are only very loosely correlated with paper quality.