Answer 1 of 10
As discussed in program committee:
caching for speed in IRC meetings,
posting scores for reviewers also on the index
alllowing submission of short reviews without displaying what it will look like
separate reviewer feedback form
better if clearer on dates sooner.
Answer 2 of 10
Answer 3 of 10
A minor quibble: The wiki-style used in the submission process was poorly documented and I ended spending a LOT of time trying to get my submission to look decent. That par tof hte process was very frustrating, which is why the interface scores so lowly.
More seriously, I think when you create this revamp this process for PyCon 2009, you really *must* make sure to confirm the user's email as part of the registration process. Standard stuff, send them a confirmation link to activate the account.
I know it s a pain to code (as I've had to do 3 such systems in the last 6 months) and to use. However, I think my experience shows that it is necessary.
My email was entered incorrectly when I registered. This was discovered when an email was sent to all submitters indicating a delay in judgment.
Ok, the mistake was mine and I thought it was corrected.
However, when I got my rejection, the information provided indicated that some of the reviewers were willing to champion my submittal, and attempted to contact me to request some changes. According to the notes I did not respond. As far as I can tell, this is due to the email typo. I've checked my through my email and I can find no attempt to contact me at the email address I registered with.
So, from a real world example, the importance of verifying the user's email address is shown.
[NOTE FROM SOFTWARE AUTHOR: this is exactly how the system works. It has always required an activation url sent to the user. How did you activate the account w/o that URL? I have no clue what happened here, but something went seriously wrong, and I am very sorry that this happened. --Doug Napoleone]
Answer 4 of 10
I think one of my talks was "Accepted" but I did not receive an email. I'm not sure if this was by design or not.
Answer 5 of 10
IIRC, the directions seemed to indicate that the review process would be more interactive. Under this direction, I chose to disclose what I felt were weak points, thinking that I might receive help with overcoming these weaknesses. Instead, I received only one comment prior to submission of final reviews (where I was ask to show where my code was -- which more or less broke anonymity), and the primary critic of my work used my own points against me. This disposes me to basically be "dishonest" in the future, and use indirection to cover any weaknesses in my work and presentations.
On a more personal note: I understand this is not LL(1) or the dynamic languages symposium (DLS), but I have been continually frustrated by the PyCon community denying me the opportunity to speak about language evolution in front of a wider audience (and since DLS is in Crete this next year, I doubt I can afford to speak there). If you consult Larry Wall's most recent state of the onion, extensibility is a feature going into some of Python's competitors. I strongly feel overlooking next generation language features because they may explode heads (unless Armin or Guido are speaking) is a mistake that we can not afford to keep making. I'm very glad we find ourselves with more proposals than time this year, but in the past, this same bias (against language issues) has led previous PyCon organizers to schedule empty rooms instead of granting time for raising language issues. Three or four years ago (the last time I tried to talk at PyCon), language extensibility in popular scripting languages was science fiction. Now it is timely. Perhaps by PyCon 2009, we'll be asking ourselves how we can catch up with Perl and Ruby.
I trust open space and lightening talks will ensure I am not completely shut out, but I wanted to make sure that what I see as multiple opportunity costs (to the community) have been brought to your attention. Thank you for organizing this, and all the hard work that entails. I did find the automation and process useful -- I just wish the reviewers were willing to use it more than once on my submission.
Answer 6 of 10
I was unable to view the reviewers' comments until after the review was complete. Was this intentional? It wasn't spelled out anywhere. I'd thought, from looking at the provided examples, that there would be some back-and-forth communication, rather than walled-off silence followed by a decision announcement. That was my expectation, anyway. Part of the learning process, I suppose, and good experience for next year.
By the way, the fact that the question on the review interface is required, even though I was not on the Program Committee, and specified such in question 2 above, is something that could/should be improved for next year.
Answer 7 of 10
1.How are the decisions to accept a proposal made?
2.More comments on rejecting proposal would be good. (example: the proposal #xxx is better then yours, or we try to concentrate more on python language vs YYY package)
3. Pre-decision popularity survey would be nice (which proposal people interested in attending digg?, +1 -1) Give a little counter on the web interface.) This would allow a little feedback of what is hot and what people want to see before decision is made.
Thanks.
Lucas
Answer 8 of 10
While I have no problem with one of my proposals being declined, I do have a problem with the review process, as it applied to that proposal. For reference, I'm referring to #20, "Building on Django".
One of the reviewers specifically asked questions, and another outline two hopes for things I hadn't laid out specifically in the outline. Unfortunately, since I was unable to see these reviews, I couldn't respond to the concerns raised, and better flesh out the proposal. I was worried that this exact situation might happen, and it frustrates me that it did.
If the proposal system has a comments feature, why weren't the reviewers using it? If they had, I could've explained more about what I had planned, which might have improved their opinions of it. As a first-time proposal to PyCon, I didn't know how much detail was expected at this stage, and there was no feedback to help me provide the detail that was apparently necessary.
Again, I understand that with so many proposals, many had to be cut, and I'm not offended that one of mine was declined (I'm glad the other one was accepted, of course). I just wish that it had been declined on virtue of it truly not being worthy, rather than due to reviewers not having the information they needed to make a better decision. That could've been easily remedied given proper feedback.
I don't know if the reviewers weren't told about comments, or if they thought I would be able to see the reviews, or if they assumed my proposal was already as complete as it could possibly be, or if they just didn't feel like asking the one person who could explain it in more detail. Whatever the reason, I think additional education for reviewers on how to interact with authors would help the process run more smoothly, and ultimately, it would produce better proposals, and thus better presentations.
Answer 9 of 10
The software interface was very good. Only problem is that the HTML text boxes put hard breaks in wrapped lines which screws ReST formatting.
Oh - and in this form "If you were on the Program Committee how would you rate the review interface?" is compulsory even if you weren't on the review interface...
Answer 10 of 10
Overall the process was easy to understand and ran smoothly. I was disappointed that the reviewers could not meet their deadline - but I understand the reasons for this.
.