Reviewing Processes

Deadlines

As a program chair, you need to be as clear as possible with PC members about your expectations. To give a specific example, when I served on the OSDI'00 PC, I made the bad assumption that OSDI was more like the ATC than SOSP in the way the PC was run. I expected to review 20-30 papers, and then go to a PC meeting. Only as the papers came in did I find out the number of papers we were each expected to review was significantly higher, and there would be another round to provide more reviews of the papers that made the first cut. Providing workloads and due dates when inviting PC members can go a long way toward ensuring that only people available to satisfy those demands will accept your invitation.

In addition to the final deadline, consider explicit intermediate deadlines. For instance, the first time I served on an ATC PC, for the 1997 conference, I was told that there were multiple deadlines: we should get 1/3 of our reviews in by the first deadline, another 1/3 by the next, and the rest by the end just before the PC meeting. Some conferences seem to follow this approach, and some don't. As a chair, I have usually used this approach and it has been a great help in identifying any PC members who need a little extra urging. When I haven't done it I have usually regretted it.

-- FredDouglis - 01 Apr 2008

Assigning papers to reviewers

I used the CRP software which I needed an assistant with database experience to use well, and still I encountered challenges. Having a new person run the software at every conference instantiation is probably a bad choice. It probably would be a good investment to use one of the conference submission services for SIGARCH and IEEE conferences. (SIGPLAN is already using these services successfully.) We borrowed or added several features to the software to make the following changes.

Conflicts. The authors specified their PC conflicts and institution conflicts with a check box, and others in a list which greatly eased paper assignation.

Matching. For ASPLOS, I made all the paper assignments by matching paper topic areas to PC member topic areas. I found that I did not do a great job (wrt PC interest and expertise) and for 169 papers and 18 PC members. Because of the unexpected quantity of submissions, I applied two rounds of reviewing. In the first round, I assigned each paper two committee member reviewers (about 20-25 hours of work for me), and each of them requested one or more reviews. For the 112 papers with a significant number of positive reviews or conflicting reviews, I then assigned an additional committee member reviewer (another 10 hours of work) and in a few cases, external reviews. Thus, each paper had at least 4 reviews and all papers discussed at the committee meeting had 5 reviews, with 3 from committee members. Most of these reviews came in before the rebuttal period. In addition, each PC member had a higher density of papers in the top group. I recommend this two tier strategy for better focus with a large number of submissions, but it is more work for the PC chair.

For PACT, I instead gave the PC members a list of all their non-conflict papers, and let them indicate their preferences. With this information, I achieved a 100% match of submissions to one or more PC members that wanted to review the submission, but it only took me about 10 hours. I assigned each submission 3 PC members and they choose an additional 3 outside reviewers. Since the ratio between the PC members and number of submissions was low, I did not do two tiers of reviewing. On the PC side, I assigned PC members from 70% to 100% of the submissions in which they had high interest. As a result, I believe we had better and more qualified reviewing than we would have otherwise.

-- excerpted (with permission) from Notes on Chairing Program Committees by Kathryn S McKinley

External reviewers

Getting all external reviews (2 per paper). I personally solicted non-PC reviews for most papers and, for papers where I had a conflict of interest, designated two PC members to act as PC chairs to solict reviews (indirectly, with me sending the actual email). With some nagging I got all external reviews. The reviews were less correlated with PC reviews (but were correlated with me). I believe they carried more weight at the PC meeting, since they were mostly from known people rather than junior students. This process cost me two-person weeks (but I am good at nagging).

-- excerpted (with permission) from Some Advice for Program Committee Chairs Based on my ISCA 2005 Experience by Mark D. Hill

Software

Refer to the ReviewingSystems page.

Other recommendations

Some Things That Worked (Reasonably Well):

• Asking for three PC reviews. This seemed to balance PC burden with a sufficient representation of papers at the meeting.
• Reducing review count on a few weak papers (15 papers). This somewhat reduced PC and external reviewer burden, but was not a big factor.
• O'Hare Hilton. It worked well to have a one-day meeting near Chicago O'Hare, because PC members could leave after 6:00 PM and still reach most of the United States.
• No PC Grading. I did not have PC members "grade" papers before the PC meeting, because I feared partial participation. Based on "re-scoring" experience below, I think this was the correct decision.

Things That Did NOT Work (Well):

• Re-Scoring Papers. Before the PC meeting, I encouraged PC members to read all reviews, interact with each other, and, based on this new input, optionally change their overall recommendation on a paper. Some PC members took this seriously; others did not. People are busy. I wish I knew a way to do this better. Nevertheless, even partial re-scoring helps improve the paper ranking used at the PC meeting.
• Small PC (23 members). I chose a smaller PC than recent ISCAs. This may have helped discussion some, but was quite a burden on PC members (26-ish papers each). I might increase the size by several members, but still worry about the group dynamics of 35 people.
• PC Information. PC members were not good about listing their areas of interest or conflicts of interest. You need to know or learn these by other means.
• Paper Potential. On the review form, after "Overall Recommend," I asked, At what level would you recommend this paper if you consider only its POTENTIAL (e.g., ignoring methodological flaws) and DIRECTION (e.g., Could it expose new areas of research for ISCA?)?. This question did not work well. Reviewers answered it inconsistently, and, to my surprise, sometimes lower than the overall recommendation. I asked a few reviewers about this, and they responded that this was their intention.
• Ranking papers with 2/3*(Recommendation) + 1/3*(Potential). This did not work much better than using "Recommendation" only. I had a secondary ranking that removed the single lowest recommendation and then averaged the rest. I recommend this ranking function to you, as I removed the initial effect of a single harsh reviewer (a common occurrence).

-- excerpted (with permission) from Some Advice for Program Committee Chairs Based on my ISCA 2005 Experience by Mark D. Hill