HotCRP

HotCRP is a new conference reviewing tool developed by Eddie Kohler at UCLA. This page is a placeholder to provide documentation and other resources. Eddie's WOWCS paper on HotCRP is "Hot Crap!" (PDF) (HTML)

• Software and documentation are at http://www.cs.ucla.edu/~kohler/hotcrp/

Suggestions and Feedback

Unless Eddie offers an external site to collect comments, this page can serve that purpose. After using HotCRP for OSDI'08 I had a few reactions. Overall it is very very nice. But some thoughts for changes include:

• Do a better job at preserving privacy of people who submit feedback on reviews.
• Make it easier to find such feedback, especially negative feedback.
  • Reviewers can now search for "rate:-" to find reviews that have received negative feedback. -- EddieKohler - 24 Aug 2008
• As a PC member, I might be interested in discussions even if I am not a reviewer who is emailed a comment by default. It would be nice if we could search on "commented after 7/23" or display and sort on things like the number of comments and the last time a comment was entered.
• Don't leak information about conflicted papers unnecessarily. If I look at order:discuss and don't see paper N, then I know already that a paper I have a conflict with is ranked N. If I only have one or two conflicts, I can pin down that info. This is similar to leaving the room in a PC meeting, but releases the info to more people, potentially, as well as earlier.
  • The fix is to randomly add gaps in the discussion order, making it impossible to tell whether a gap is due to a conflict. As of 2.24. -- EddieKohler - 24 Aug 2008

-- FredDouglis - 24 Jul 2008

Some additions:

• This might be more work than Eddie wants to do, but HotCRP does not currently support multi-track PCs (e.g., mapping subsets of papers to subsets of the PC) or hierarchical PCs.
• It would be nice to be able to extract all of the metadata for the papers into something like an XML or spreadsheet file
  • You can already extract a lot of metadata via text files, and of course there's always the database! -- EddieKohler - 24 Aug 2008
• The submission form setup allows one to add new questions for each submission; it would be nice to have a way to automatically convert these into tags, since the tags feature of HotCRP is often more precise than the free-text search feature.
  • It's possible to search specifically for options: "opt:shadow" finds papers that checked "Shadow PC," for example. This doesn't do everything you'd want, but it's a start. -- EddieKohler - 24 Aug 2008

-- JeffMogul - 25 Jul 2008