Tips for running a PC meeting

This page collects advice (mostly for PC chairs) about how to run a PC meeting. PC members might want to read this, to know what to expect and to be able to help the chairs.

Goals

It usually helps to get some agreement at the beginning of the meeting about goals, including issues such as:

• how many papers are we hoping to accept?
• what characteristics are we going to favor?
• are we planning to shepherd all, some, or no papers?

(Personal opinion, JeffMogul): There is a lot of evidence that, no matter what scoring system is used, the actual scores have too much noise to be of much use. That is, once the scores have been used (with some corrections) to decide which papers should be discussed at all, the PC meeting should essentially ignore the scores, and discuss the reasons for accepting or rejecting a paper. The PC chair(s), in my view, should strive to remind people at the meeting that they should ignore the scores. However, this is probably something that the PC as a whole should agree about, at the start of the meeting, rather than treating it as a point of contention all through the meeting.

-- JeffMogul - 15 Apr 2008

Paper discussion process

I have a few points of possible interest about how to run the meeting itself smoothly.

• Select the PC member with the highest score on the paper to lead the discussion. I borrowed this technique from Mark Hill's ISCA 2005 PC meeting, where it also worked well. The point of the meeting is to accept papers, and this selection means each discussion starts with the most positive review.
• Keeping the PC on task. Post the next ten papers and their PC lead for discussion to help the PC members refresh their memories (borrowed from ISCA 2005). Post the lists of accepted, rejected, and tabled papers as you go (widely used).
• Watch the clock. I tried to limit discussion to five minutes, and actually achieved 10 for the vast majority of papers (widely used).
• Ordering. I used the following ordering in both my meetings: top down until bogged, bottom up (reject 10 at time until the PC saved more than one paper in the group), PC papers, top down, and in the end, the advocate system (widely used).
• Closing. When there was only about two hours left, I moved to a system in which a PC member must volunteer to advocate for the paper in order for it to be considered further. We quickly listed one-by-one all the remaining "middle" papers, and asked for a PC advocate volunteer. The remaining discussion was limited to these papers led by the advocate (used successfully at ASPLOS 2004, ISCA 2005, and PACT 2004).

-- excerpted (with permission) from Notes on Chairing Program Committees by Kathryn S McKinley

Here are some things that worked well, in order of importance:

• Seek reasons to accept papers. At the PC meeting, I had the PC member who gave the highest recommendation begin the discussion by summarizing the paper and then critiquing it. Next, other PC members who read the paper spoke. Finally, we had a discussion. This set a positive tone, and, in the long run, helped focus PC members on looking to what is good (since some PC members summarized many more papers than others). I also kept mentioning 42 papers as a target, reminding members to look for papers to accept. In the end, we took 45 papers.

• Keep PC meeting discussion focused on making decisions. I was ruthless about focusing on decisions. I cut people off when they rambled. I cut off discussion when it appeared consensus was reached. See "egg timer" below. We covered the top 50 papers explicitly and then moved to a model where PC members could "save" papers (to cover 30 more).

• Being very clear on conflict of interest rules. Doing so makes the process more fair and, equally important, appear more fair. In the PC meeting, for example, I had two other researchers run the discussion for papers for which I had a conflict of interest. This was critical, since on the rest of the papers, I would often cut off discussion. If I did that for a conflicting paper, it might look like I was influencing the decision (because I would be). I found that, with reminders, authors were good at listing conflicts, but PC members were not.

-- excerpted (with permission) from Some Advice for Program Committee Chairs Based on my ISCA 2005 Experience by Mark D. Hill

Logistics

PC meetings usually end up recording paper-by-paper decisions on a whiteboard, as they are made, to give PC members a view of how the program is coming together. (It's a good idea to also immediately record all decisions in the online review system, so that they are properly captured.) One useful technique is to prepare, in advance, a set of sticky notes with paper ID numbers and titles -- then during the meeting, you can move these around on the whiteboard, rather than having to write/erase/rewrite whiteboard entries. It might also be useful to mark the notes corresponding to papers with PC-member authors or conflicts, in case you don't want to expose the decisions on these papers too early in the meeting.

-- JeffMogul - 15 Apr 2008

You need help. At the meeting, I highly recommend that you ask a graduate student, colleague, or assistant to help you. This person can keep records, enter decisions in the software and/or on the board, track conflicts, fetch people who leave the room for conflicts, and handle unforeseen requests. This assistance will enable you to move quickly from one paper to the next, instead of pausing one or more minutes between papers.

-- excerpted (with permission) from Notes on Chairing Program Committees by Kathryn S McKinley

Other recommendations:

• Egg Timer. An egg timer was set to beep after five minutes of initial discussion. I did not explicitly cut off discussion, but it effectively reminded people to keep things moving.
• PostIt Notes. PostIt notes were used to display accepted, rejected, and deferred papers. "Deferred" meant "could accept depending on what else we had at the end of the day." This was a great visual way to see progress.
• Easel. The next ten or so papers to discuss were displayed on an easel. This let PC members "pipeline" refreshing their memories with discussion of papers they were less interested in. As you know, pipelining can greatly improve throughput.

-- excerpted (with permission) from Some Advice for Program Committee Chairs Based on my ISCA 2005 Experience by Mark D. Hill

Etiquette and Ethics

I suggest the following meeting and post-meeting etiquette:

• Make all conflicts (advisor, advisee, friend, family, co-author or grant collaborator in past 5 years) leave the room.
• Keep all specific PC comments private.
• I assigned a secondary chair for the papers with which I had a conflict.

-- excerpted (with permission) from Notes on Chairing Program Committees by Kathryn S McKinley

PC members should never reveal to outsiders the specifics of who said what during a PC meeting, even if they disagree themselves. Also, PC members should never reveal anything at all about the discussion of a paper except to its authors. I think it is generally OK to tell authors the gist of the PC's discussion of their own paper, and probably OK to explain the general tone of a PC ("this year, we seemed reluctant to accept papers on distribute shared memory").

-- JeffMogul - 21 May 2008

I strictly followed Guidelines for SIGARCH Sponsored Conferences. I found it helpful to build on the wisdom of others. In particular, I fould it valuable to follow the guideline on restricting each PC member to at most two submissions. This reduced the number of PC papers that we would likely accept (reducing appearance and reality of inbreeding) and saved time at the PC meeting (since PC paper discussions take longer than others).

-- excerpted (with permission) from Some Advice for Program Committee Chairs Based on my ISCA 2005 Experience by Mark D. Hill