Over the last few years, improving performance has been one of the main areas of focus for CPython. This talk focuses on how CPython's performance is measured, and what it means when we say that one version is X% faster than another. Many of these techniques apply in general to all kinds of software.

This includes:

• A history of performance measurement in Python, and how measuring an interpreter can be different from other kinds of software.

• An overview of CPython's pyperformance benchmark suite, and what makes a benchmark more or less useful. Why is the concept of the "typical Python workload" so elusive? An evaluation of the breadth of the benchmark suite and a discussion of ways in which the Python community can help to improve it.

• Why measuring time is hard, and how we improve that situation with software, hardware, and statistics.

• An overview of our continuous benchmarking system, based on Github Actions.

• Lastly, a discussion of how these techniques can be applied to software in general.