The OopslaProgramCommitteeMeeting(s) I've sat on had complex rating schemes for papers :
- your recommendation
- your confidence in your recommendation
- your familiarity with the paper's topic
All of this went into creating a numerical rating for the paper. I found this approach too pseudo-scientific for my taste.
The whole program committee meeting comes down to thumbs up/thumbs down for every paper. I wanted the committee members to be thinking about that binary decision from the first, so I only accepted binary decisions as recommendations. These recommendations went into creating the SortedPaperList. -- KentBeck