Change the future

Packaging and distribution tools mini-summit

What will be happening?

Occurring on Friday evening, this packaging mini-summit is aimed at getting developers of various tools and services related to building, packaging, distributing and installing Python software around a table where we can review the unfortunate failure of the distutils2/packaging project in the lead up to Python 3.3, and look at alternative ways forward. The intent is similar to that of the Python language summit before the conference, which aims to bring together developers of multiple Python implementations and related tools.

We'll be starting with a post-mortem of distutils2/packaging from ɉric Araujo, then hopefully hearing in detail from multiple different people on things they're working on and why they think they're necessary (see the Lightning Talks section below).

This mini-summit is being coordinated by Nick Coghlan (CPython core developer). Questions can be sent to me directly, or passed to distutils-sig.

When and where is this happening?

The mini-summit is scheduled in Open Space room 206, on Friday night from 7-9 pm.

Why is it happening?

Packaging and distribution are big enough problems that they aren't all going to be solved in a single night, even when limited to the needs of just the Python community (if things were that simple, everything would have been sorted years ago). Rather, the goal, at least for me (Nick) is to end the night with a clearer idea of what python-dev can do to better enable a more open ecosystem of interoperable tools, with less reliance on implementation defined behaviour in specific tools (as happened with distutils and setuptools). distutils is now in its 15th year (and setuptools isn't much younger): design choices that seemed reasonable in the late 90s aren't standing up well to the growth of the Python community and the increased complexity in build system requirements and the variation in distribution models.

The discussions at the open space should also affect the answers given at the Directions for Packaging panel on Saturday afternoon.

Who should come along?

This session is aimed primarily at those already actively involved in the development of Python packaging and distribution tools and resources, including, but not limited to:

  • distutils
  • distutils2/packaging
  • distlib
  • setuptools & distribute
  • pip & virtualenv
  • zc.buildout
  • bento (alternate build system from the scientific Python community)
  • wheel (cross-platform binary distribution format)
  • PyPI (and related services like python-packages.com, crate.io, pypi-mirrors.org)
  • pyp2rpm (repackaging for RPM based systems)
  • dh_python2/dh_python3 (repackaging for Debian and Debian-based systems)
  • best practices guides (such as the Hitchhiker's Guide to Python or the Hitchhiker's Guide to Packaging)

For anyone just interested in learning more about what the future may have in store for Python packaging and distribution tools (rather than wanting to hack directly on code and documentation in order to help make that future happen), then the Directions for Packaging panel on Saturday afternoon will likely be a better option.

On the other hand, for anyone considering contributing to some of these tools and resources (especially those that are attending the sprints in the days following PyCon US), the discussions here may provide some interesting background.

Lightning Talks

Note: rather than a "sign-up if you want to talk for 5 minutes" model, I created a list of projects I wanted to hear more about at this session, and found project leaders and/or contributors attending PyCon US that are willing to speak about them:

  • distutils/Python stdlib/meta-packaging: Nick Coghlan
  • PyPI: Richard Jones
  • distutils2/packaging: Éric Araujo
  • distlib: Nick Coghlan (on behalf of Vinay Sajip, who unfortunately can't attend)
  • wheel: Daniel Holth
  • pip/virtualenv: Marcus Smith
  • bento: Daniel Holth (on behalf of David Cournape, who unfortunately can't attend)
  • zc.buildout: Jim Fulton
  • setuptools: Jason R. Coombs! (unfortunately, PJ Eby can't attend)
  • distribute: Jason R. Coombs
  • TUF (The Update Framework): Monzur Mohammed (unfortunately, Justin Cappos can't attend)
  • (Ana)conda: Ilan Schnell

The aim is for each of these talks to provide a (very!) short summary of:

  • why does this project exist?
  • what is the current status of the project?
  • where does development and discussion of the project primarily occur?
  • what interoperability standards are needed for the project to thrive?

With a dozen or so talks to get through, we will need to be strict on the 5 minute time limit! :)

User and Goals

Often, when people have discussions like this, people argue about solutions, without clearly stating problems. Here's an attempt to identify some user types and their goals, in hopes of making the discussions more productive --Jim Fulton

I've identified some users (actors, personas) below to try to highlight different and often conflicting concerns. This does not imply that real people fit neatly into these boxes.

Scripter

A scripter creates solutions very quickly. These may be one-off scripts or scripts small enough to repair quickly if Python or a dependency changes. "Batteries included" is very valuable, as is rapid evolution in language or libraries that provide new tools for their toolbox. They really appreciate it when system packaging tools (such as RPMs and debs) make it easy to install packages that would be difficult to install otherwise. And they appreciate tools, like pip and easy_install, that make it easy to add new tools their environment.

For scripters, scripts are a means to an end, not a primary focus.

Application Developers

An application is bigger, often much bigger than a script. It's built from multiple Python projects (aka packages or distributions). It may be made up of multiple Python programs/scripts as well as third-party applications, such as databases.

Because of the complexity of applications and, these days, the need to iterate quickly, control is critical, both of the application components, and of the environment. To deploy applications automatically, it's critical that the developer knows precisely what their deploying and that it's the same application that was tested in their continuous integration environment. For this reason, sharing dependencies with other applications or users is unattractive.

Package (/project/distribution) Developers

Package developers create distributions and upload them to PyPI to:

  • Make it easy to install packages into their applications,
  • Share their code with other people,
  • Get credit for their work.

They often don't want to provide support, or be "on the hook" for packages they upload. They may feel (rightly or wrongly) the need to remove distributions they've uploaded for legal reasons or to try to prevent propagation of bugs.

Proprietary Software Developers

Developers may provide solutions for Python users that they charge for. They don't upload their software to PyPI, but they provide meta data so that people looking for solutions, who might be willing to pay, can find them. This is of value to software consumers, who might otherwise not find good solutions to their problems.

System packagers

System packagers provide system packages (such as RPMs and debs) that make it easy to install Python and add ons. They primarily serve the needs of scripters and their own application developers. (OK, what I said just now is controversial. :)

They are concerned with:

  • making installation easy,
  • providing automated updates,
  • avoiding duplication,
  • protecting users by making it easy to provide security fixes quickly.

It's worth noting that the ability to get security updates quickly is important, but a feature that application developers often trade off to maintain control of their dependencies. It might be helpful if there a way for applications to notify a system of dependencies they're using so that they can be notified of security issues without having their dependencies automatically changed.