Proposal Review Procedure
(Some redundancy in the instructions below is intentional.)
Contents
Program Committee Membership
The conference will benefit from as diverse a program committee as possible. Members should be recruited from major projects to ensure that they are represented.
AT THIS TIME WE ARE NO LONGER LOOKING FOR NEW PROGRAM COMMITTEE MEMBERS
Can I be a reviewer and submit a talk?
Yes! You won't be asked to review your own talk however...
AT THIS TIME WE ARE NO LONGER LOOKING FOR NEW PROGRAM COMMITTEE MEMBERS
I'm not a Python expert, can I still help?
Yes. We want as diverse a range of reviewers as possible, so if you are interested in PyCon you can help review talks.
AT THIS TIME WE ARE NO LONGER LOOKING FOR NEW PROGRAM COMMITTEE MEMBERS
How do I become a reviewer?
Create a login on this website and then send an email to the PyCon Program Committee Mailing List ( pycon-pc@python.org ) introducing yourself.
AT THIS TIME WE ARE NO LONGER LOOKING FOR NEW PROGRAM COMMITTEE MEMBERS
Reviewer Instructions
Instructions for authors (and reviewers) can be found in the proposal submission instructions and in the FAQ.
Reviewers must join the pycon-pc mailing list, which is for program committee discussions.
Assignment of Proposals to Reviewers
As a reviewer proposals will be assigned to you randomly, as proposals are submitted. You will receive an automated e-mail notification when this happens. This random assignment ensures that each proposal is assigned to at least three reviewers. You may choose to "opt-out" of any proposal assigned to you before you vote, and another reviewer will be assigned in your place. You are also encouraged to vote or comment on other proposals, even those not assigned to you. All communication with authors should be done via the comments.
Requesting Information or Changes
If you see a proposal that you think could be good, but it's lacking in some way and there is not enough material for you to judge, please ask the author to flesh out their proposal, via comments. Be specific about which parts of their proposal are lacking. Suggest additions they could make that would improve the proposal.
Identify the Champion
Proposals will be assessed using the "Identify the Champion" process. Reviewers vote using a +1/+0/-0/-1 scale:
+1: Good proposal. I will champion it at the PC meeting.
The reviewer is strongly in favor of the proposal, and is willing to argue for the proposal during the accept/decline period (November 24 to 30). To be accepted, a proposal must have at least one +1 vote (i.e. the proposal must have a champion).
A +1 vote means that the reviewer is willing to put their reputation on the line in favor of the proposal. So if you're not sure about a proposal, don't champion it.
+0: OK proposal, but I will not champion it.
-0: Weak proposal, though I will not fight strongly against it.
+0 & -0 register opinions without affecting the voting, except in marginal cases.
-1: Serious problems. I will argue to reject this proposal.
Reviewer Notifications
You will receive e-mail notifications for every proposal that is assigned to you. You will also receive e-mails for comments and reviews for proposals you watch.
In addition to email, there are RSS feeds on just about every page in the proposal system. These will not work in readers that don't support usr/login for rss feeds, google reader for example.
Comments
All communication with authors should be done via the comments.
As soon as a comment is added to the system, the comment is sent to the submitter by e-mail. The reviewer's name is included (i.e. no anonymity).
Reviewers also receive the comments of the submitter and other reviewers by e-mail.
Reviews (Votes)
You are encouraged to vote or comment on any proposal, whether or not it is assigned to you.
You may choose to "opt-out" of any proposal assigned to you before you vote, and another reviewer will be assigned in your place.
You can revise your review/vote at any time until the system has been locked down for final selections.
The author will not see your review until after the process is done, and will only ever see the 'final' version after the system has been locked down. Other reviewers will have access to the entire history of your edits.
Please go ahead and enter interim reviews. We get a FLOOD of proposals at the end, so any reviewing we can get done early will only help. Another job of reviewers is to find more information about the speaker/subject/etc. and include that in the review where applicable (or ask the submitter to provide such information).
Often people will give reviews like:
"+0 The talk looks interesting but I can not champion this talk because XYZ. Once the author answers my comments I will update my review."
Change History
The change history at the bottom of the review will allow other reviewers to put that review into a proper timeline. If the author addresses the issues mentioned in a review, but the review has not been updated yet, that situation can easily be figured out.
Program Committee Review
Once all the proposals have received votes from three different reviewers, we'll schedule an IRC meeting to make the final selections. Not all the committee members show up for this meeting, though all members are certainly welcome to participate.
All votes should be reviewed before the meeting. Many votes will have been made before authors made edits to their proposals, and some votes may need to be updated by the reviewers.
Proposals that have at least one +1 vote and no -1 votes are provisionally accepted, and proposals with only -1 and -0 votes are almost invariably rejected. Most of the discussion is spent looking at proposals with a mix of positive and negative votes, or proposals without a +1/champion vote.
Decisions are provisional at this point only because among the accepted proposals we might find essentially the same proposal submitted three times by different people, and we might be forced to reject one or two in the service of diversity.
Finally we come to the meat of the process: proposals with counterbalancing +1/-1 votes or that have several +0 without champion. Generally speaking, all such proposals should be considered biased against accepting, and if the proposals provisionally accepted fill the schedule, these gray area proposals should only be mined for critical diversity needs.
For background, please see the "Identify the Champion" paper.
High-Profile Talks
Independently of the program committee review described above, any "high profile" talks should be identified and may be accepted by the Conference Chair or the Program Committee Chair (with recommendations from reviewers). "State of the project"-type talks for major projects and implementations should be identified and may be treated specially.
Voting Statistics Page
On the voting statistics page ("voting" link), proposals are broken down into categories:
- Needs champion
Proposals which need a champion are those with no +1 or -1 votes. +0 & -0 are weak votes, and not enough for a good decision.
- Needs votes
Each proposal requires a minimum of 3 votes before an accept/decline decision can be made.
- All votes positive
Proposals which have at least 3 votes that are all postitive. These are usually accepted without discussion.
- All votes negative
Proposals which have at least 3 votes that are all negative. These are usually declined. Proposals that are not beyond hope may be worked on with the authors (via comments) to make them better.
- Mixed votes
Proposals with a mixture of positive and negative votes (at least 3 total). These proposals are on the fence. Reviewers should work with authors before the final accept/decline phase to get these propsals all positive. Many will remain mixed, and the champions will help make the final accept/decline decision.
.