Support for binary extensions is an exceptional advantage of Python that is too
often avoided for smaller packages with low developer resources. Binary
extensions are used to achieve high performance for libraries like PyTorch,
MyPy, and many thousands more. Binary extensions also allow access to a wealth
of existing compiled libraries. Building your own binary extension is plagued
by historically poor documentation, bad common practices, and many
misconceptions. But it is actually easy to write extensions today that work
seamlessly on all common developer platforms using modern libraries and
continuous integration.
We will take a look at packaging a binary extension from start to finish. This
starts with pybind11 for C++ bindings, providing simple, header only builds
and avoiding the need for a new language or pre-processor step. We will look at
scikit-build for building, providing powerful CMake based builds with library
search, multithreaded builds, and more. We will use PyPA's build to produce
SDists. And we will use PyPA's cibuildwheel to produce binaries for all
common platforms with minimal setup and simple CI code in GitHub Actions (but
trivially movable to any other CI system). We will talk about how to automate
common tasks, like using GitHub's Dependabot to keep cibuildwheel up-to-date
while also ensuring reproducible builds.
After this talk, it is our hope that you will no longer shy away from using
compiled code in libraries, but will feel comfortable writing extensions to
accelerate or advance your libraries functionality.