Doug's Program Committee Rant from 2007
Contents
Who We Are
With most professional technical conferences, proposals and papers are peer-reviewed, and often presenters need to be known and published elsewhere to be recognized.
PyCon is a volunteer-run "unconference", of, by, and for the community. In addition to the core developers, project heads, corporate sponsors, and published luminaries, the Python community is made up of all the people who use Python for everyday tasks, to solve real problems.
PyCon is designed to connect, strengthen, support, and build that community. This is done in many ways:
- Tutorials provide accelerated learning.
- Open Space (the real soul of the conference in many ways, including the "hallway track"), builds the connections, and provides a crucial means of exchanging knowledge and deep learning withing the community.
- Lightning talks provide a means for a rapid dissemination of information, and are a lot of fun.
- The social events build the community.
- The development sprints keep the community strong, many times even reinventing parts of it.
- The keynotes punctuate and frame the conference.
- The scheduled talks form the core supporting all the rest.
Actually only about 25% of the conference, the scheduled talks are the most visible part, and the most crucial. The talks are the voice of the community. They are the skeleton which supports and connects all the other events.
Tutorials give people foundations in areas where they lack knowledge. The talks build on those foundations, or make people aware that new areas of knowledge exist. The talks are kept short to focus the information. It is next to impossible to keep an entire audience interested and engaged in a subject for long. Keeping the talks short keeps the audience hungry. This leads into the open space talks which focus the discussion on areas which are really important to all involved. There the communication is completely interactive, versus mostly-unidirectional lectures. This exchange of ideas and knowledge leads into the sprints where this new knowledge is put to use.
We are the ones who make sure that skeleton is strong. We are the ones who ensure the voice of the community is heard. We are the Program Committee.
Proposal Process
The process is about taking a monumental task, breaking it into successively smaller parts, and processing those parts in successive refinements.
- Phase 0: Building the Program Committee.
- Phase 1: Proposal Submissions - Divide the proposals among reviewers.
- Phase 2: Review Period - Help elevate the standard of proposal, and identify the talks which do not have consensus.
- Phase 3: Accept/Decline Period - Resolve the contentious talks, and make the acceptance decisions.
- Phase 4: Feedback & Publication - Allow authors to voice concerns, catch potential misunderstandings, make the decisions public.
- Phase 5: Schedule - Figure out who, when and where.
Phase 0: Build the Program Committee
We need people from the community to review the proposals, and decide which are truly representative of the community. This can be very hard to do. It is best to get people with as broad a range of Python experience and knowledge as possible. It is also important to make sure you have enough people. The magic number is to have 1 reviewer for every 5-6 proposals, with a minimum of 3 reviewers and a maximum of 30. This means each reviewer has about 15 proposals to review and manage.
Phase 1: Proposal Submissions
This is the phase where we accept talk proposals. This is also when proposals are divided among the program committee members, to make things manageable. It will take time for the assignments to gel and become final.
The program committee needs to nurture the proposals at this point, helping to explain the process, answering questions, and resolving problems. Proposals need not be final, as the authors are allowed to modify and flesh out their proposals. The program committee should help authors in this process.
Comments
The way the program committee communicates with the authors is via the proposal comments. To keep initial reviews and impressions objective, the reviewer and author names are masked from each other. Reviewers will be able to see other reviewer names, but authors will see 'Reviewer #42'. Reviewers will see 'Author #52'. As such names should not be used in the proposal, or in comments. Later in the process, the names will be made public. All comments are immediately sent to all other people associated with a proposal (authors, reviewers, and commenters). You do not need to be a reviewer for a proposal to make a comment. You just need to be part of the program committee.
NOTE: the name masking was only for the first phase of reviewing in past years. For 2009, all the masking has been turned off
Reviewers
3 Reviewers will be randomly assigned to talks. This is a true random process, so assignments might feel lopsided at first. Reviewers should opt-out of any talks they are not comfortable with, or feel there is a conflict of interest. Another reviewer will be randomly assigned. You will not be re-assigned to a talk that you have opted-out of. Reviewers may also opt-in and review any proposal, even those they are not assigned to. If you previously opted out of a proposal, you may still opt-in later. The assignment is just to ensure that every proposal has at least 3 reviewers and at least 3 votes. A proposal can have as many votes as there are reviewers.
Reviewers should vote early and vote often! Reviews are not set in stone, and should be changed over time. Getting proposals with initial reviews and information up front is crucial to a smooth process.
Voting
The votes do not determine if a talk is accepted or declined. They are just a means of breaking down the talks into categories later to help organize the end discussion in an organized and meaningful way.
The review details should include information to help the program committee as a whole make a decision on a proposal. If you are an domain expert on the topic of the proposal, please say so. If you are an outsider, please say so. This is important information. Even if you have no prior knowledge of a topic, your review is still important, especially for basic level talks, but the program committee as a whole needs to know this.
- Reviewers vote using a +1/+0/-0/-1 scale:
+1: Good proposal. I will champion it at the PC meeting. +0: OK proposal, but I will not champion it. -0: Weak proposal, though I will not fight against it. -1: Serious problems. I will argue to reject this proposal.
The +1 and -1 votes are special, you are agreeing to be a talk Champion (or nemesis). A talk champion is someone who will speak up for a talk during the final vote review meeting at the end of the process. More on this later.
Phase 2: Reviewing Period
No more proposal submissions are accepted. Reviewers should be set with their list of proposals they are working on.
The program committee switched gears a bit here. the idea behind this period is to help the authors refine their proposals and make them as good as they can. Many people have never given a talk before, and will need help refining their proposal. The program committee's job here is really to help the community produce the best talks possible. The ideal is to get all the proposals the be +1 worthy (i.e. a proposal anyone would be willing to champion).
Being part of the program committee is not just about deciding which talks get accepted. It is about building the community as well.
During this phase, the program committee should be attempting to get all their votes in, with a minimum of 3 votes per talk. At the end of this phase, the program committee members should switch from looking at their list of associated proposals, to the vote breakdown.
Phase 3: Accept/Decline Period
Authors are locked out of the system. No more changes will be made to the proposals. The comment system will also be disabled. Reviewers get to see the author names and proposal associations (many authors submit multiple proposals).
Who the author is and their history can be very important in the final decision making process. We want to accept proposals by authors who have not presented before over those who have. We may know that a certain presenter is not good, or is exceptionally good. This can change a reviewers vote decision. Reviewers need to factor this new information, and update any of their reviews appropriately. This information will also be crucial for the acceptance meeting at the very end of this phase.
In this phase the program committee switched from looking at their individual proposal assignments over to the proposals which have a lack of consensus or contention. And then make the acceptance decisions.
Each proposal is discussed and the Champions speak up and arguments are made. At the end of the meeting, there should not be any proposals still 'Under Review'.
- Proposals are broken down into 5 vote categories:
- Needs champion
Proposals which need a champion are those with no +1 or -1 votes.
- Needs votes
Each proposal requires a minimum of 3 votes before an accept/decline decision can be made.
- All votes positive
Proposals which have at least 3 votes that are all postitive. These are usually accepted without discussion.
- All votes negative
Proposals which have at least 3 votes that are all negative. These are usually declined.
- Mixed votes
Proposals with a mixture of positive and negative votes (at least 3 total). Many will remain mixed, and the champions will helpmake the final accept/decline decision.
The proposals in Category #1 need more voting before the final decision meeting. All reviewers are asked to look at these proposals again, to try to find people to champion them (+1 OR -1).
At this point there should not be any votes in category #2.
Those in Category #4 are usually talked about next to make sure they really are not worth having.
Those in Category #3 are usually left for the end of the discussion. They are most likely all worth having, but most likely have overlap.
This leaves those proposals in category #5. These are the talks which will be discussed at the acceptance decision meeting. this is a meeting where the fate of these proposals is decided.
Usually more than one pass is given to the proposals. When looking at talks on a subject, talks from all categories are considered. The point is to consider everything, but in an organized manner which helps to make the best choice.
Phase 4: Feedback & Publication
The authors are given back access to the system. The proposals are still locked and can not be edited. The comment system is re-enabled. The reviews are made 'public' to the authors, and the authors can now see the names of the reviewers, as well as the reviews (but not the review change histories). The authors are sent the decision e-mail messages letting them know about the program committee decisions.
This is a feedback period where the authors get to post comments and give feedback. At the end of this period, the final list of accepted talks is made public with their summaries.
Phase 5: Schedule
This phase actually starts during phase 2, and is refined throughout the process. What talks happen when will need to be determined. Some speakers might have time restrictions. This can be quite the rubix cube problem.
Tutorials need to be fit into the schedule as well. In the software, the accepted proposals need associated events, speakers, and scheduling objects created. times tables, rooms, etc also need to be created, and events assigned.
.